Andrii Bidochko

• Updated: July 4, 2026
• 7 min read

MoECodec: Image Compression for joint human and machine perception via Mixture-of-Experts

Direct Answer

MoECodec introduces a token‑aware image compression framework that simultaneously serves human visual quality and downstream machine‑vision tasks. By embedding a Mixture‑of‑Experts (MoE) routing mechanism inside a transformer‑based codec, the model dynamically allocates compute to image regions based on their semantic relevance, delivering better reconstruction and higher task performance without proliferating separate task‑specific models.

Background: Why This Problem Is Hard

Modern AI pipelines increasingly treat images as both visual media for humans and raw data for machines—think autonomous vehicles, surveillance analytics, and e‑commerce visual search. Traditional codecs (JPEG, HEIC) are optimized for human perception; they ignore the feature hierarchies that convolutional or transformer‑based vision models rely on. Conversely, “compression for machines” research has produced two divergent paths:

• Task‑specific end‑to‑end designs: Each downstream vision task (e.g., object detection, segmentation) receives a bespoke encoder‑decoder pair. While performance can be strong, the approach multiplies parameters, complicates deployment, and forces engineers to maintain a zoo of models.
• Transfer‑based adapters: A generic codec is trained first, then a lightweight adapter is fine‑tuned per task. This reduces model count but often treats the codec as a black box, applying uniform transformations across all image tokens regardless of content complexity.

Both strategies suffer from a static computation pattern: every token—whether a smooth sky patch or a densely textured street sign—receives the same amount of processing. In reality, machine perception assigns vastly different importance to regions; a tiny traffic sign can dominate detection accuracy, while a uniform background contributes little. Capturing this heterogeneity without exploding model size or inference cost is the core challenge.

What the Researchers Propose

MoECodec reframes image compression as a token‑level decision problem. The key idea is to replace the conventional feed‑forward network (FFN) layers inside a transformer‑based codec with a Mixture‑of‑Experts module that routes each token to one of several lightweight experts. The routing decision is conditioned on two factors:

1. Input content: Visual complexity, texture, and semantic cues extracted from the token itself.
2. Task objective: A task identifier (e.g., classification, detection) that biases the routing toward experts that have learned task‑relevant transformations.

In this architecture, the codec retains a single shared backbone for all tasks, but the MoE layer dynamically selects the most appropriate expert per token, enabling fine‑grained allocation of compute and parameters.

How It Works in Practice

The MoECodec pipeline can be broken down into four conceptual stages:

1. Tokenization and Initial Encoding

An input image is split into non‑overlapping patches (tokens) and projected into a latent space using a shallow convolutional stem. This yields a sequence of token embeddings that preserve spatial ordering.

2. Mixture‑of‑Experts Routing

Each token passes through a routing network that computes a score for every expert in the MoE pool. The scores are transformed into a sparse selection—typically the top‑1 expert per token—using a stable routing strategy that blends expert‑choice routing with a spatial total variation regularizer. The regularizer encourages neighboring tokens to share the same expert, preserving spatial coherence and preventing noisy, checkerboard assignments.

3. Expert Processing (Group Shuffle MLP)

Selected experts are lightweight Group Shuffle MLPs (GShMLP). A GShMLP first groups token channels, shuffles them across groups, and then applies a small MLP per group. This design limits parameter growth while still offering expressive non‑linear transformations. Because each expert operates on a distinct subset of tokens, the overall compute scales with image complexity rather than image size.

4. Decoding and Reconstruction

After expert processing, the token sequence is fed into a standard transformer decoder that reconstructs the compressed image. The decoder is shared across tasks, ensuring that the final bitstream remains compatible with conventional image viewers while also preserving task‑specific feature maps for downstream models.

The entire system is trained end‑to‑end with a multi‑objective loss that balances pixel‑wise reconstruction error, perceptual quality (e.g., LPIPS), and task‑specific performance (e.g., detection mAP). This joint optimization forces the routing network to learn which regions matter most for each downstream objective.

Evaluation & Results

To validate MoECodec, the authors conducted extensive experiments across two axes: traditional image quality metrics and downstream vision tasks. The evaluation suite included:

• Standard reconstruction benchmarks (PSNR, SSIM, LPIPS) on the Kodak and DIV2K datasets.
• Three downstream tasks: image classification (ImageNet), object detection (COCO), and semantic segmentation (ADE20K).
• Baseline comparisons against a vanilla transformer codec, a task‑specific MoE‑free model, and leading compression‑for‑machines methods such as DVC and BPG‑ML.

Key findings:

1. Higher visual fidelity: MoECodec achieved a 0.4 dB gain in PSNR over the vanilla transformer codec at comparable bitrate, while also improving LPIPS by 12 %.
2. Task performance boost: For object detection, MoECodec raised mAP by 3.2 points relative to the best non‑MoE baseline at the same compression ratio. Similar gains were observed for classification (+1.8 % top‑1 accuracy) and segmentation (+2.5 % mIoU).
3. Parameter efficiency: Despite adding multiple experts, the GShMLP design kept the total parameter count within 1.2× of the baseline, far lower than the 2.5× increase seen in task‑specific models.
4. Spatially coherent routing: Visualizations of expert assignments revealed that high‑frequency regions (edges, text) consistently selected specialized experts, while smooth backgrounds were handled by a generic expert, confirming the effectiveness of the total variation regularizer.

These results demonstrate that MoECodec does not merely trade off visual quality for task performance; it improves both simultaneously, confirming the hypothesis that token‑aware, dynamic computation is a viable path forward for unified image codecs.

For a deeper dive into the experimental setup, see the MoECodec paper.

Why This Matters for AI Systems and Agents

Unified codecs like MoECodec simplify the engineering stack for any organization that processes visual data at scale. Instead of maintaining separate pipelines for human‑focused compression (e.g., streaming video) and machine‑focused feature extraction (e.g., edge analytics), a single model can serve both, reducing storage costs and latency.

For AI agents that rely on visual perception—autonomous drones, retail recommendation bots, or surveillance analysts—the ability to retrieve a compressed image that still retains task‑critical cues means faster inference on edge devices. The token‑level routing also opens the door to adaptive bandwidth allocation: an agent can request higher fidelity for regions it deems important while keeping the rest highly compressed.

From an operational standpoint, MoECodec aligns with the UBOS platform overview, which emphasizes modular, reusable AI components. By exposing the codec as a service, developers can plug it into existing workflow automation studios, combine it with Chroma DB integration for vector search, or pair it with OpenAI ChatGPT integration to build conversational agents that can discuss compressed visual content without sacrificing accuracy.

What Comes Next

While MoECodec marks a significant step forward, several open challenges remain:

• Scalability of expert pools: As the number of downstream tasks grows, managing expert specialization without redundancy will require more sophisticated gating mechanisms.
• Cross‑modal extensions: Integrating audio or text streams into the same token‑aware routing framework could enable truly multimodal compression for agents that process video, speech, and captions simultaneously.
• Hardware acceleration: Efficiently executing sparse expert selection on GPUs or specialized AI chips is an engineering hurdle that will determine real‑world latency.
• Robustness to distribution shift: Ensuring that routing decisions remain optimal when encountering out‑of‑distribution scenes (e.g., night‑time driving) is critical for safety‑critical agents.

Future research may explore hierarchical MoE structures, where coarse‑level experts first decide macro‑regions and fine‑level experts handle sub‑patches, further reducing compute while preserving detail. Another promising direction is self‑supervised routing, allowing the model to discover task relevance without explicit task identifiers.

Enterprises interested in deploying such technology can start by experimenting with the Enterprise AI platform by UBOS, which offers built‑in support for custom model serving, monitoring, and integration with existing data pipelines.

As visual data continues to dominate AI workloads, token‑aware, dynamic compression frameworks like MoECodec will become essential building blocks for next‑generation AI agents, enabling them to see more, think faster, and operate cheaper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.