Andrii Bidochko

• Updated: June 14, 2026
• 6 min read

Continual Model Routing in Evolving Model Hubs – A Deep Dive

Direct Answer

The paper Continual Model Routing in Evolving Model Hubs (arXiv) introduces Continual Model Routing (CMR), a framework that dynamically selects the most appropriate expert model from a rapidly expanding hub without retraining or exhaustive search. By anchoring models with contrastive embeddings and replaying a curated subset of past routing decisions, CMR enables scalable, low‑latency inference even as the hub grows to thousands of models.

Background: Why This Problem Is Hard

Modern AI ecosystems increasingly rely on model hubs—central repositories that host thousands of specialist models (e.g., language, vision, multimodal experts). While these hubs promise “plug‑and‑play” capabilities, they also create a combinatorial explosion in the model‑selection problem:

• Scale. New checkpoints are added daily, pushing the total count beyond 2,000 in many public hubs.
• Continual drift. Models evolve, deprecate, or specialize for emerging tasks, making static routing tables quickly obsolete.
• Latency constraints. Real‑time applications cannot afford to query every model or run heavyweight similarity searches for each request.

Existing solutions—zero‑shot retrieval, fine‑tuning on a fixed pool, or adapter‑based merging—either assume a static model set or require costly per‑task adaptation. Consequently, they falter when the hub is both large and fluid, limiting the practical deployment of AI services that need to stay current with the latest research breakthroughs.

What the Researchers Propose

The authors present two tightly coupled contributions:

1. CMRBench. A benchmark that simulates a living model hub by continuously injecting new models, retiring old ones, and diversifying downstream tasks. It contains over 2,000 models spanning multiple families (e.g., GPT‑4‑style, LLaMA, BLOOM) and domains (code, dialogue, vision‑language).
2. CARvE (Contrastive Anchored Routing with Embeddings). A routing algorithm that treats each model checkpoint as an anchor in a contrastive embedding space. When a new query arrives, CARvE computes a lightweight embedding, finds the nearest anchors, and then refines the choice using a structured replay buffer that remembers past successful routes.

CARvE’s design deliberately separates two concerns: (a) a static, low‑cost similarity index that can be updated incrementally as new models appear, and (b) a dynamic, task‑aware replay mechanism that mitigates catastrophic forgetting by re‑exposing the router to representative past queries.

How It Works in Practice

The operational workflow of CARvE can be broken down into three stages, illustrated in the diagram below.

1. Model Anchoring

When a new model checkpoint is added to the hub, a short “anchor” embedding is generated by feeding a curated set of probe inputs (e.g., generic prompts, image patches) through the model and extracting the final hidden state. These anchors are stored in a high‑dimensional index (FAISS or HNSW) that supports sub‑millisecond nearest‑neighbor lookups.

2. Query Embedding & Initial Retrieval

For each incoming request, the router computes a query embedding using a lightweight encoder (often a distilled version of the hub’s base model). The embedding is matched against the anchor index, returning the top‑k candidate models.

3. Structured Replay & Final Selection

The replay buffer holds a balanced sample of past queries and their successful model assignments, grouped by task type and difficulty. CARvE scores the top‑k candidates against this buffer, applying a contrastive loss that favors models that have historically performed well on similar queries. The highest‑scoring model is then dispatched for inference.

What sets this approach apart is its continual nature: new anchors can be inserted without rebuilding the entire index, and the replay buffer is refreshed incrementally, ensuring the router adapts to both model additions and task drift.

Evaluation & Results

Using CMRBench, the authors evaluated CARvE against three strong baselines:

• Zero‑shot retrieval. Direct similarity search without replay.
• Fine‑tuned specialist. Per‑task fine‑tuning of a single model.
• Adapter‑merging ensemble. Combining adapters from multiple models via weighted averaging.

Key findings include:

• Accuracy boost. CARvE achieved a 7.4% absolute improvement in task‑level success rate over zero‑shot retrieval, narrowing the gap to fine‑tuned specialists by only 2.1%.
• Scalability. In hubs with >2,000 models, CARvE’s average routing latency remained under 15 ms, whereas the adapter‑merging approach exceeded 120 ms due to larger memory footprints.
• Robustness to drift. When 30% of the models were replaced mid‑evaluation, CARvE’s replay buffer allowed it to retain 92% of its original performance, while baselines dropped by 15‑20%.
• Family‑level generalization. CARvE correctly routed queries to the appropriate model family (e.g., code‑oriented vs. dialogue‑oriented) 94% of the time, demonstrating strong semantic awareness.

These results collectively demonstrate that CARvE can serve as a practical “router” for living model hubs, delivering near‑expert performance with minimal computational overhead.

Why This Matters for AI Systems and Agents

For engineers building AI‑powered products, the ability to tap into the latest specialist models without manual curation is a game‑changer. CARvE’s low‑latency routing enables:

• Dynamic agent orchestration where a single meta‑agent can delegate sub‑tasks to the most suitable expert on the fly.
• Seamless integration with UBOS platform overview, allowing enterprises to plug CARvE into existing workflow automation pipelines.
• Cost‑effective scaling, because the router avoids loading multiple heavyweight models simultaneously, reducing GPU memory pressure.
• Future‑proofing of AI services: as new models appear in the hub, they become instantly reachable without redeploying the entire stack.

In practice, a customer support chatbot could automatically switch between a dialogue‑optimized LLM, a code‑generation model, and a vision‑language model based on the user’s request, all orchestrated by CARvE. This level of adaptability is essential for enterprises seeking to maintain a competitive edge in the rapidly evolving AI landscape.

What Comes Next

While CARvE marks a significant step forward, several open challenges remain:

• Memory‑bounded replay. As the hub grows beyond tens of thousands of models, the replay buffer itself may become a bottleneck. Research into hierarchical or meta‑learning‑based replay could keep memory usage in check.
• Cross‑modal routing. Extending the contrastive anchor space to jointly embed text, image, and audio modalities would enable truly multimodal hubs.
• Security and provenance. Ensuring that routed models are trustworthy and free from malicious backdoors is critical for production deployments.

Addressing these issues will likely involve tighter integration with data‑centric platforms. For example, coupling CARvE with a Chroma DB integration could provide a persistent, vector‑search‑backed store for anchors and replay samples, while a ChatGPT and Telegram integration could expose the routing service as a conversational assistant for developers.

Beyond research, we anticipate commercial products that bundle CARvE with Enterprise AI platform by UBOS, offering turnkey model‑hub management, continuous routing, and monitoring dashboards. Such solutions would empower startups and SMBs alike to leverage the full breadth of the AI model ecosystem without the overhead of manual model selection.

In summary, Continual Model Routing redefines how we interact with ever‑growing model hubs, turning a scalability nightmare into a manageable, high‑performance service. As AI research continues to produce specialized checkpoints at an unprecedented pace, frameworks like CARvE will be indispensable for turning that abundance into actionable intelligence.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.