Andrii Bidochko

• Updated: July 23, 2026
• 6 min read

Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement

Direct Answer

Director is a new distributed serving system that proactively moves Mixture‑of‑Experts (MoE) components across GPUs based on real‑time predictions of which experts will be needed next. By combining a lightweight activation predictor with a relaxation‑based optimizer, Director cuts end‑to‑end latency by 11‑55 % for large‑scale models such as Mistral, DeepSeek, and Qwen.

Background: Why This Problem Is Hard

Mixture‑of‑Experts models achieve state‑of‑the‑art performance by routing each input token to a small subset of specialized “experts” that live on different GPUs. This expert parallelism promises linear scaling, but in practice the latency of a request is dominated by two intertwined factors:

• Communication latency: When an input activates experts spread across many devices, data must travel over PCIe or NVLink links, incurring costly round‑trips.
• Computation latency: Even if the experts are on the same GPU, the GPU’s compute pipeline can become a bottleneck if too many experts compete for the same resources.

Existing placement strategies rely on historical activation logs to co‑locate frequently co‑activated experts. While this works for relatively stable traffic patterns, modern AI services experience highly dynamic workloads—think personalized chat, real‑time recommendation, or multi‑tenant inference clusters. In such environments, past patterns quickly become stale, leading to sub‑optimal placements, increased cross‑GPU traffic, and higher tail latency.

Three technical challenges make an online, proactive solution difficult:

1. Uncertainty of future requests: The system must predict which experts a new request will need before the request arrives.
2. Migration cost: Moving an expert’s parameters between GPUs consumes bandwidth and can stall ongoing inference if not handled carefully.
3. Optimization complexity: Finding the optimal placement under capacity constraints is NP‑hard, so a fast, near‑optimal algorithm is required for real‑time operation.

What the Researchers Propose

Director tackles the three challenges with a three‑layer architecture:

• Prediction Layer: A cascaded predictor (or a low‑bit quantized replica) estimates the expert activation pattern of each incoming request in microseconds.
• Migration Layer: An online migration engine schedules expert moves during compute‑bound phases, ensuring that migrations have “near‑zero downtime.”
• Placement Optimizer: A relaxation‑based optimizer converts the NP‑hard placement problem into a polynomial‑time linear program, then rounds the solution while guaranteeing a (1 + ε) approximation of the optimal latency.

In essence, Director continuously forecasts where the next wave of traffic will go, reshuffles experts just‑in‑time, and does so with provable performance bounds.

How It Works in Practice

Conceptual Workflow

The end‑to‑end flow for a single inference request looks like this:

1. Request Arrival: The front‑end receives a token batch.
2. Activation Prediction: The Prediction Layer runs a fast model that outputs a binary vector indicating which experts are likely to be activated.
3. Placement Decision: The optimizer takes the predicted vector, current GPU capacities, and migration costs, then solves the relaxed placement problem.
4. Migration Execution: If the optimizer suggests moving an expert, the Migration Layer enqueues the move to the next compute‑bound window (e.g., after the current token’s matrix multiplication finishes).
5. Inference Execution: The request is routed to the newly‑placed experts, minimizing cross‑GPU hops.
6. Feedback Loop: Actual activation patterns are logged and fed back to improve the predictor.

Component Interaction

Director’s components communicate through a lightweight shared‑state service:

• The Predictor writes predicted activation masks to a distributed cache.
• The Optimizer reads these masks, combines them with a capacity matrix, and writes placement directives.
• The Migration Engine monitors the directives, schedules non‑blocking parameter copies, and updates a placement registry once migration completes.

What sets Director apart is the timing of migrations. Traditional systems pause inference to move experts, causing spikes in latency. Director’s “compute‑bound migration” runs concurrently with GPU kernels that are already waiting on memory, effectively hiding the transfer cost.

Why This Approach Differs

• Proactive vs. Reactive: Instead of reacting after a request suffers high latency, Director anticipates the need and reshapes the topology ahead of time.
• Relaxation‑Based Guarantees: By solving a relaxed linear program, Director achieves a provable (1 + ε) approximation, a rarity in online placement literature.
• Near‑Zero Downtime Migration: The system exploits natural compute stalls, turning a traditionally disruptive operation into a background task.

Evaluation & Results

To validate Director, the authors built a prototype on a 16‑GPU cluster and benchmarked three popular MoE models: Mistral‑7B‑MoE, DeepSeek‑MoE, and Qwen‑MoE. The evaluation covered two dimensions:

Scenarios Tested

• Static workload: Request patterns drawn from a fixed distribution, mimicking a steady‑state service.
• Dynamic workload: Rapidly shifting activation distributions, emulating personalized chat sessions or time‑of‑day traffic spikes.
• Baseline comparisons: Director was pitted against (a) a static placement strategy, (b) a history‑based optimizer, and (c) a naïve round‑robin scheduler.

Key Findings

• Under dynamic workloads, Director reduced 99th‑percentile latency by 11‑55 % across the three models, with the largest gains observed on Qwen‑MoE where expert activation variance was highest.
• The relaxation‑based optimizer solved placement instances in under 2 ms, well within the sub‑second budget required for online operation.
• Migrations incurred less than 0.5 % of total request time, confirming the effectiveness of compute‑bound scheduling.
• Prediction accuracy of the cascaded predictor exceeded 92 % for top‑2 expert selections, directly correlating with latency improvements.

These results demonstrate that proactive, prediction‑driven placement can close the latency gap that has long limited MoE serving at scale.

Why This Matters for AI Systems and Agents

Low‑latency inference is a decisive factor for any production AI service—whether it powers a conversational agent, a recommendation engine, or a real‑time analytics pipeline. Director’s ability to keep expert traffic local translates into tangible business benefits:

• Improved user experience: Faster response times reduce churn in chat‑based agents and increase conversion rates for recommendation systems.
• Higher GPU utilization: By minimizing cross‑GPU traffic, clusters can serve more requests per watt, lowering operational costs.
• Scalable multi‑tenant environments: Proactive placement isolates noisy tenants, preventing one user’s bursty traffic from degrading others.
• Simplified orchestration: System operators no longer need to manually rebalance experts; Director handles it automatically.

For teams building AI agents on platforms like ChatGPT and Telegram integration, the reduced latency directly improves conversational fluidity. Similarly, enterprises leveraging the Enterprise AI platform by UBOS can now host larger MoE models without over‑provisioning hardware.

What Comes Next

While Director marks a significant step forward, several avenues remain open for exploration:

• Heterogeneous hardware support: Extending the optimizer to handle CPUs, TPUs, and emerging accelerators could broaden applicability.
• Adaptive ε tuning: Dynamically adjusting the approximation factor based on workload volatility may yield even tighter latency bounds.
• Cross‑cluster coordination: In multi‑region deployments, coordinating expert placement across data centers could further reduce tail latency.
• Security and privacy: Ensuring that expert migration does not leak sensitive model parameters is an emerging concern for regulated industries.

Future research may also investigate tighter integration with model‑parallel training pipelines, allowing a single system to both train and serve MoE models with the same placement intelligence.

References

Director paper on arXiv

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.