Andrii Bidochko

• Updated: June 17, 2026
• 7 min read

AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers

Direct Answer

AdaMerge introduces a salience‑aware, training‑free token‑merging framework that dynamically adjusts how many visual tokens are collapsed at each layer of a Vision Transformer (ViT). By weighting token similarity with importance scores and adapting the compression intensity per input, AdaMerge preserves high‑impact visual information while delivering up to 2× speed‑up with minimal accuracy loss.

Background: Why This Problem Is Hard

Vision Transformers have reshaped computer‑vision pipelines by treating an image as a sequence of patches and applying self‑attention across the entire token set. The self‑attention operation scales quadratically with token count, so a 224×224 image split into 16×16 patches already yields 196 tokens, and deeper models quickly reach thousands of tokens when intermediate feature maps are flattened. This quadratic growth translates into high FLOPs, memory pressure, and latency—critical bottlenecks for real‑time inference on edge devices, cloud‑cost‑sensitive services, and large‑scale batch processing.

Researchers have tackled this issue through three broad strategies:

• Token pruning: discarding low‑importance tokens based on attention scores or learned masks.
• Low‑rank approximations: factorizing attention matrices to reduce computation.
• Token merging (ToMe): pairing similar tokens and averaging their representations, a method that requires no retraining.

Among these, token merging stands out because it can be applied to any pre‑trained ViT without fine‑tuning. However, the original ToMe algorithm assumes that all tokens are equally important—a premise that clashes with extensive evidence that self‑attention distributes focus unevenly across an image. Salient regions (e.g., faces, objects) generate strong attention, while background patches contribute little. When ToMe merges aggressively, it often collapses high‑salience tokens with unrelated neighbors, causing noticeable drops in classification accuracy, especially at high compression ratios.

What the Researchers Propose

The AdaMerge framework addresses the equality assumption by introducing two complementary mechanisms:

1. Salience‑Weighted Similarity: Instead of relying solely on raw feature similarity, AdaMerge computes a centrality‑based salience score for each token. This score reflects how much a token participates in the attention graph, effectively measuring its importance. The salience scores are then incorporated into the bipartite matching process that decides which tokens to merge, ensuring that high‑salience tokens are either preserved or merged with similarly important partners.
2. Adaptive Merging Intensity: Compression is not a one‑size‑fits‑all operation. AdaMerge pre‑computes layer‑wise similarity statistics on a calibration set and uses these statistics at inference time to decide how many token pairs to merge for each specific input. Images with redundant backgrounds trigger more aggressive merging, while cluttered scenes retain more tokens.

Together, these mechanisms create a token‑merging pipeline that is both aware of visual importance and flexible enough to adapt to the redundancy profile of each image, all without any additional training.

How It Works in Practice

The AdaMerge workflow can be broken down into three sequential stages that run inside the forward pass of a standard ViT:

1. Salience Estimation

For each token, AdaMerge computes a column‑wise affinity vector by summing its attention weights across all heads and layers. Tokens that receive or distribute large attention values obtain higher centrality scores, which are normalized to produce a salience weight between 0 and 1.

2. Salience‑Weighted Bipartite Matching

Tokens are represented as nodes in a bipartite graph. Edge weights combine two factors: (a) cosine similarity of token embeddings and (b) the product of their salience scores. A Hungarian algorithm finds the optimal pairing that maximizes the combined weight, preferentially matching high‑salience tokens with other high‑salience tokens.

3. Adaptive Reduction Count

Before matching, AdaMerge consults a lookup table derived from offline statistics. The table maps the average similarity distribution of a layer to a recommended number of merges. If the current image exhibits low average similarity (i.e., diverse content), the table suggests fewer merges; if similarity is high (e.g., large uniform sky), more merges are allowed.

After the pairs are selected, each pair is merged by a weighted average that respects the salience scores, producing a new, shorter token sequence that proceeds to the next transformer block. The process repeats at designated layers (commonly after the 3rd, 6th, and 9th blocks in a ViT‑B/16), progressively shrinking the token set while preserving critical visual cues.

The following diagram illustrates the end‑to‑end flow:

What distinguishes AdaMerge from prior token‑merging methods is the explicit incorporation of token importance into the matching score and the data‑driven decision of how many tokens to drop per layer. This dual awareness eliminates the “one‑token‑fits‑all” pitfall that caused earlier approaches to degrade performance under aggressive compression.

Evaluation & Results

To validate AdaMerge, the authors conducted a comprehensive suite of experiments on the ImageNet‑1k benchmark using the ViT‑B/16 architecture. The evaluation focused on three dimensions:

• FLOPs reduction: Measured the total floating‑point operations required for a forward pass after token merging.
• Top‑1 accuracy impact: Compared the classification drop relative to the uncompressed baseline.
• Robustness across compression levels: Tested multiple target FLOPs budgets ranging from 13.4 G to 30 G.

Key findings include:

1. At a 13.4 G FLOPs operating point (≈2× speed‑up), AdaMerge incurred only a –1.06 % Top‑1 accuracy loss, outperforming PiToMe (–1.45 %) and DSM (–4.62 %).
2. The accuracy gap widened monotonically as compression intensified, confirming that AdaMerge scales more gracefully than its competitors.
3. Qualitative visualizations showed that AdaMerge retained fine‑grained details around salient objects (e.g., animal faces) while aggressively merging background patches, aligning with its salience‑aware design.

These results demonstrate that AdaMerge not only pushes the accuracy‑FLOPs Pareto frontier but also does so without any retraining, making it immediately applicable to existing ViT deployments.

Why This Matters for AI Systems and Agents

For practitioners building AI‑powered products—whether autonomous agents interpreting visual streams, recommendation engines that incorporate image embeddings, or edge‑deployed vision services—the trade‑off between latency and accuracy is a daily concern. AdaMerge offers a plug‑and‑play acceleration layer that can be inserted into any pre‑trained ViT pipeline, delivering:

• Lower inference cost: Reduced FLOPs translate directly into cheaper cloud GPU usage and longer battery life on mobile devices.
• Predictable performance: Because the merging decisions are deterministic and data‑driven, system engineers can model latency budgets with confidence.
• Preserved task fidelity: Salience‑aware merging ensures that downstream agents—such as object detectors, visual question answering modules, or multimodal chatbots—receive high‑quality embeddings even under tight compute constraints.

Integrating AdaMerge into a broader AI workflow can be as simple as swapping the token‑merging hook in the UBOS platform overview with the AdaMerge module. Once in place, developers can orchestrate visual preprocessing, language generation, and decision‑making steps within the Workflow automation studio, achieving end‑to‑end pipelines that respect both speed and accuracy requirements.

What Comes Next

While AdaMerge marks a significant step forward, several avenues remain open for exploration:

• Cross‑modal salience: Extending the salience metric to incorporate textual or audio cues could enable joint token merging for multimodal transformers.
• Hardware‑aware scheduling: Coupling AdaMerge’s adaptive intensity with real‑time GPU/CPU load signals could further optimize latency under variable workloads.
• Fine‑grained control: Allowing users to specify domain‑specific importance maps (e.g., medical ROI masks) would tailor merging to specialized applications.
• Robustness to distribution shift: Investigating how salience estimation behaves on out‑of‑distribution images will be crucial for safety‑critical deployments.

Future research may also explore integrating AdaMerge with emerging sparse‑attention architectures, creating a hybrid that leverages both token reduction and attention sparsity for maximal efficiency.

For organizations interested in experimenting with AdaMerge, the AdaMerge paper on arXiv provides the full algorithmic details and code snippets. Implementations can be wrapped as a microservice on the UBOS homepage, enabling rapid prototyping across teams.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.