Andrii Bidochko

• Updated: July 13, 2026
• 7 min read

SHAP‑Weighted Cross‑Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

Direct Answer

The paper introduces SHAP‑Weighted Cross‑Modal Expert Fusion (XGAF), a tree‑based mixture‑of‑experts framework that uses TreeSHAP attribution magnitudes to assign sample‑level weights to unimodal and cross‑modal experts. By carefully choosing a sum‑absolute reduction for SHAP values, XGAF matches or exceeds traditional early‑fusion performance while preserving the modularity of late‑fusion systems.

Background: Why This Problem Is Hard

Emotion and sentiment recognition in real‑world applications—such as customer‑service bots, video analytics, and social‑media monitoring—relies on multiple data streams: facial expressions, speech prosody, textual content, and sometimes physiological signals. Combining these modalities is non‑trivial for three reasons.

• Heterogeneous feature spaces. Visual embeddings can have thousands of dimensions, while textual embeddings may be a few hundred. Simple concatenation (early fusion) creates a monolithic model that is difficult to interpret and costly to retrain when a modality changes.
• Loss of cross‑modal interaction. Late fusion treats each modality independently and merges predictions at the decision level. This modularity is attractive for system design, but it discards subtle cues that emerge only when modalities are examined together (e.g., sarcasm detected through tone and facial tension).
• Explainability demands. Deployments in regulated domains (healthcare, finance, HR) require transparent reasoning. Traditional deep‑fusion models are black boxes, making it hard to justify why a particular emotion label was chosen.

Existing approaches either sacrifice interpretability for accuracy (early fusion) or sacrifice interaction for modularity (late fusion). The research community has therefore been searching for a middle ground that retains the best of both worlds while providing clear, per‑sample explanations.

What the Researchers Propose

The authors propose SHAP‑Weighted Cross‑Modal Expert Fusion (XGAF), a hierarchical mixture‑of‑experts architecture guided by Explainable AI (XAI) at inference time. The key ideas are:

• Expert pool. A collection of specialized predictors: unimodal experts (audio‑only, video‑only, text‑only) and cross‑modal experts that ingest two or three modalities simultaneously.
• TreeSHAP attribution. For each input sample, a decision tree (trained to predict the target) computes SHAP values for every feature across all experts. The magnitude of these values reflects how much each expert contributed to the final decision.
• SHAP reduction strategy. Because experts differ in dimensionality, the raw SHAP vector must be reduced to a single weight per expert. The paper compares three reductions—mean‑absolute, median‑absolute, and sum‑absolute—and shows that the sum‑absolute method preserves the total attribution mass, preventing high‑dimensional cross‑modal experts from being unfairly down‑weighted.
• Sample‑level routing. The reduced SHAP magnitudes are normalized into a probability distribution that weights each expert’s prediction for the current sample. The final output is a weighted sum of all expert predictions.

How It Works in Practice

The XGAF pipeline can be broken down into four conceptual stages.

1. Feature extraction. Each modality is processed by a pre‑trained encoder (e.g., a ResNet for video frames, a wav2vec model for audio, a BERT variant for text). The resulting embeddings feed into the corresponding experts.
2. Expert inference. Unimodal experts receive a single embedding, while cross‑modal experts concatenate the relevant embeddings and pass them through a lightweight classifier (often a shallow MLP or a transformer block).
3. TreeSHAP attribution. A shallow decision‑tree model, trained on the same training set, evaluates the full feature set (all embeddings). TreeSHAP computes a per‑feature contribution score for the sample. These scores are grouped by expert and reduced using the sum‑absolute rule to produce a weight vector w = (w₁,…,wₙ).
4. Weighted aggregation. Each expert’s softmax output is multiplied by its corresponding weight, and the results are summed to produce the final emotion or sentiment label.

What distinguishes XGAF from earlier fusion schemes is the dynamic, data‑driven routing that is both transparent (thanks to SHAP) and sensitive to the relative importance of high‑dimensional cross‑modal experts. The system remains modular: new experts can be added or removed without retraining the entire fusion layer, because the routing logic depends only on the SHAP attribution of the current sample.

Evaluation & Results

The authors validated XGAF on two benchmark datasets that are standard in multimodal affective computing.

Datasets and Metrics

• MELD (Multimodal EmotionLines Dataset). Contains 7 emotion classes (e.g., joy, anger) derived from TV‑show dialogues, with synchronized video, audio, and text.
• CMU‑MOSEI (Multimodal Opinion Sentiment and Emotion Intensity). A 3‑class sentiment task (positive, neutral, negative) with longer video clips and richer textual content.
• Performance is reported using the weighted‑F1 (WF1) score, a standard metric that balances class imbalance.

Key Findings

• Using the sum‑absolute reduction, XGAF achieved a WF1 of 0.5983 on MELD with a transformer‑based cross‑modal expert, virtually identical to the early‑fusion baseline (0.6018) and far above probability‑average late fusion (0.4598).
• On CMU‑MOSEI, XGAF reached a WF1 of 0.6519, slightly surpassing early fusion (0.6485) and dramatically outperforming late fusion (0.5696).
• McNemar’s test confirmed that the difference between sum‑abs XGAF and early fusion on MELD was not statistically significant (p = 1.000), while both were significantly better than late fusion (p < 0.0001).
• Ablation studies revealed that the primary performance boost came from adding the trimodal expert; the routing mechanism itself contributed only marginal gains.
• Weight distribution analysis showed that mean‑abs and median‑abs reductions produced nearly uniform expert weights, diluting the influence of powerful cross‑modal experts. In contrast, sum‑abs weights concentrated on the trimodal expert, aligning with the observed performance improvements.

Why This Matters for AI Systems and Agents

For practitioners building conversational agents, sentiment‑aware recommendation engines, or real‑time video analytics, XGAF offers a pragmatic path to high‑accuracy multimodal perception without sacrificing system modularity.

• Plug‑and‑play modularity. Teams can develop or replace individual experts (e.g., swapping a speech encoder for a newer model) without retraining the entire fusion network.
• Transparent decision‑making. The SHAP‑derived weights can be logged and visualized, satisfying compliance requirements in regulated sectors.
• Resource efficiency. Because the routing logic is lightweight, XGAF can run on edge devices or within latency‑sensitive pipelines, unlike monolithic early‑fusion transformers that demand large GPU memory.
• Scalable orchestration. The expert‑centric design aligns well with workflow automation platforms. For example, the Workflow automation studio can orchestrate the feature extraction, expert inference, and SHAP routing as discrete steps, enabling rapid experimentation.

What Comes Next

While XGAF demonstrates that SHAP‑guided routing can bridge the early‑late fusion gap, several open challenges remain.

• Scalability to many modalities. Adding sensors (e.g., physiological signals) will increase feature dimensionality, potentially re‑introducing attribution dilution. Future work could explore hierarchical SHAP reductions or attention‑based weighting.
• Dynamic expert selection. Currently, all experts run for every sample, incurring unnecessary compute for easy cases. A cascade architecture that prunes experts early based on preliminary SHAP scores could improve efficiency.
• Robustness to missing modalities. Real‑world deployments often encounter dropped video frames or noisy audio. Extending XGAF with imputation strategies or confidence‑aware SHAP weighting would increase resilience.
• Integration with commercial AI stacks. Embedding XGAF into end‑to‑end platforms—such as the Enterprise AI platform by UBOS—could provide out‑of‑the‑box multimodal sentiment services for enterprises.
• User‑centric evaluation. Beyond benchmark scores, measuring how SHAP explanations improve human operator trust in sentiment‑aware agents remains an open research direction.

Developers interested in experimenting with XGAF can start by leveraging existing UBOS integrations. For instance, the OpenAI ChatGPT integration can serve as a textual backbone, while the ElevenLabs AI voice integration supplies high‑quality audio embeddings. Combining these with a video encoder and a trimodal expert built on the UBOS platform overview would recreate the XGAF pipeline in a production‑ready environment.

References

• Alihodzic, A., & Hubljar, S. S. (2026). SHAP‑Weighted Cross‑Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits. arXiv preprint.
• Hazarika, D., et al. (2020). MELD: A Multimodal Multi‑Speaker Dataset for Emotion Recognition in Conversation.
• Zadeh, A., et al. (2018). CMU‑MOSEI: A Dataset for Multimodal Sentiment Analysis.

Call to Action

Ready to bring explainable multimodal perception to your AI products? Explore the UBOS homepage for tutorials, templates, and a free trial of the UBOS templates for quick start. Join the conversation on our blog and stay ahead of the next wave of AI‑driven emotion intelligence.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.