An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion

Direct Answer

The paper introduces Mixed Fusion, a dual‑Mamba architecture that couples a visual state‑space model with a tabular state‑space model to classify cancer lesions using both image data and clinical or sociodemographic information. By leveraging the interpretability of SHAP explanations, the approach delivers higher recall on challenging medical datasets while offering a transparent decision pipeline for clinicians.

Background: Why This Problem Is Hard

Healthcare AI increasingly relies on multimodal inputs—radiology scans, pathology slides, electronic health records, and patient demographics. Combining these heterogeneous sources raises three intertwined challenges:

• Data heterogeneity: Images are high‑dimensional tensors, whereas clinical variables are low‑dimensional, often categorical, and may contain missing values.
• Model alignment: Traditional deep networks excel at a single modality but struggle to fuse disparate representations without losing salient information.
• Interpretability: In oncology, a false negative can be fatal. Regulators and clinicians demand clear, patient‑level explanations for every prediction.

Current state‑of‑the‑art solutions typically employ transformer‑based multimodal encoders (e.g., ViT‑BERT hybrids). While powerful, these models are computationally heavy, require massive pre‑training, and often produce opaque attention maps that do not translate directly into clinically actionable insights. Moreover, transformer architectures can under‑perform on recall‑sensitive tasks where missing a malignant case is far more costly than a false alarm.

What the Researchers Propose

The authors present a Mixed Fusion framework built on two specialized instances of the Mamba architecture, a recent state‑space model (SSM) that replaces self‑attention with a linear‑time recurrence. The key components are:

• Visual Mamba: Processes raw lesion images, outputs class‑probability vectors for each cancer type.
• Tabular Mamba: Consumes the probability vector from the visual branch together with structured clinical or sociodemographic features, producing the final diagnosis.
• Mixed Fusion Layer: A lightweight concatenation followed by a feed‑forward block that aligns the two modalities before the tabular Mamba ingests them.
• SHAP Explainability Module: Applies Shapley Additive Explanations to the tabular Mamba’s output, attributing importance to both image‑derived probabilities and each tabular feature.

By delegating image understanding to a dedicated visual SSM and reserving the tabular SSM for decision synthesis, the system preserves modality‑specific inductive biases while remaining computationally efficient.

How It Works in Practice

The end‑to‑end pipeline can be visualized as a three‑stage flow:

1. Image Ingestion: A dermoscopic or histopathological image is resized and fed into the Visual Mamba. The model extracts hierarchical features via its state‑space recurrence and emits a softmax probability distribution over target classes (e.g., benign vs. malignant).
2. Fusion & Enrichment: The probability vector is concatenated with structured inputs—age, gender, lesion location, genetic markers, etc. This combined vector passes through a shallow fully‑connected “fusion” block that learns cross‑modal interactions.
3. Tabular Decision Engine: The enriched vector enters the Tabular Mamba, which processes the sequence of features using its own SSM dynamics. The final layer outputs a binary or multi‑class diagnosis.

After inference, the SHAP module decomposes the Tabular Mamba’s prediction into feature contributions. Clinicians can see, for example, that a high probability from the visual branch contributed 0.42 to the malignant score, while the patient’s smoking history added 0.15.

What distinguishes this approach from transformer‑based multimodal networks is twofold:

• Efficiency: Mamba’s linear‑time recurrence reduces GPU memory consumption, enabling training on modest hardware.
• Explainability by Design: Because the tabular branch directly consumes the visual probabilities, SHAP can attribute importance to the image‑level decision without approximating attention maps.

Evaluation & Results

The authors validated Mixed Fusion on two publicly available cancer datasets:

• PAD‑UFES‑20: Contains skin‑lesion photographs paired with patient metadata (age, sex, lesion site).
• NDB‑UFES: Consists of oral‑cancer histopathology slides together with sociodemographic variables (smoking status, alcohol consumption, socioeconomic index).

Both datasets were split into stratified training, validation, and test sets. The evaluation suite included balanced accuracy, recall, precision, and F1‑score. Key observations:

• On PAD‑UFES‑20, Mixed Fusion achieved a balanced accuracy marginally below that of leading transformer baselines (≈1‑2 % gap) but outperformed them on recall by 4.3 %, indicating fewer missed malignant cases.
• On NDB‑UFES, the Mixed Fusion model surpassed transformer counterparts in every metric, with a 6.7 % lift in recall and a 3.2 % boost in balanced accuracy.
• SHAP analysis revealed that visual probabilities consistently ranked among the top three contributors, confirming that the fusion strategy preserved critical image information.

These results demonstrate that a state‑space‑based multimodal pipeline can match or exceed transformer performance, especially when sensitivity (high recall) is a primary clinical requirement.

Why This Matters for AI Systems and Agents

From an engineering perspective, Mixed Fusion offers a blueprint for building modular, interpretable agents that operate on heterogeneous health data:

• Scalable Architecture: Separate visual and tabular SSMs can be swapped independently, allowing system designers to upgrade the image encoder without retraining the entire multimodal stack.
• Agent‑Level Explainability: SHAP at the decision node provides patient‑specific rationales, a prerequisite for AI‑assisted diagnostic agents that must comply with medical regulations.
• Resource‑Constrained Deployment: Linear‑time recurrence reduces inference latency, making it feasible to embed the model in edge devices or low‑power hospital servers.
• Orchestration Friendly: The clear separation of concerns aligns with workflow automation platforms, enabling seamless integration with existing Workflow automation studio pipelines.

For organizations building AI marketing agents or other domain‑specific assistants, the Mixed Fusion paradigm illustrates how to fuse unstructured and structured inputs while preserving traceability—a pattern that can be replicated in finance, logistics, or customer support.

What Comes Next

While the study establishes a solid foundation, several avenues remain open for exploration:

• Broader Modality Inclusion: Extending the fusion layer to incorporate genomic sequences, radiomics features, or longitudinal time‑series could further improve diagnostic accuracy.
• Self‑Supervised Pre‑training: Pre‑training the Visual Mamba on large, unlabeled medical image corpora may close the remaining accuracy gap on datasets like PAD‑UFES‑20.
• Real‑World Clinical Trials: Deploying the model in a prospective study would validate its recall advantage under true clinical workflow constraints.
• Federated Learning: Because the two branches are modular, they can be trained on separate institutions’ data without sharing raw images, addressing privacy regulations.

Developers interested in prototyping similar pipelines can start with the UBOS platform overview, which provides pre‑built connectors for image storage, tabular databases, and SHAP‑based explainability dashboards. By leveraging the platform’s low‑code environment, teams can iterate on Mixed Fusion variants without deep expertise in state‑space mathematics.

References & Further Reading

Rocha, M. B., Dettogni, G. B., & Krohling, R. A. (2026). An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion. arXiv preprint arXiv:2606.20738v1.

For a deeper dive into state‑space models and the Mamba architecture, see the original Mamba paper (Gu et al., 2023) and related tutorials on the About UBOS page.

Explore our UBOS templates for quick start to accelerate the development of multimodal AI solutions in your organization.