Andrii Bidochko

• Updated: August 5, 2026
• 6 min read

Multimodal fusion of visual and morphometric features for avian bone classification

Direct Answer

The paper introduces a multimodal deep‑learning framework that fuses visual cues from bone photographs with traditional osteometric measurements to classify avian skeletal remains. By combining a state‑of‑the‑art image encoder with standardized morphometric data, the system achieves reliable bone‑type identification and promising family‑level taxonomic predictions, opening a scalable path for AI‑assisted zooarchaeology.

Background: Why This Problem Is Hard

Zooarchaeologists rely on bone fragments to reconstruct past ecosystems, diet, and human‑animal interactions. However, avian bones pose a unique identification challenge:

• Fragmentary nature: Excavated specimens are often broken, weathered, or missing diagnostic landmarks.
• Morphological similarity: Many bird families share overlapping size ranges and bone shapes, making visual discrimination difficult even for experts.
• Limited reference collections: High‑quality, digitized datasets are scarce, and manual measurement is time‑consuming.

Existing computer‑vision approaches in archaeology have focused on large mammals or hominin fossils, where distinctive features are more pronounced. Traditional machine‑learning pipelines for bone classification typically use either:

• Pure image‑based CNNs, which ignore quantitative measurements that centuries of osteology have proven essential.
• Hand‑crafted morphometric classifiers, which lack the ability to capture subtle visual patterns such as surface texture or curvature.

Consequently, current tools either miss the nuanced visual information or discard the rich metric data, leading to sub‑optimal accuracy and limited adoption in real‑world archaeological workflows.

What the Researchers Propose

The authors present a unified multimodal architecture that treats visual and morphometric inputs as complementary signals rather than competing alternatives. The framework consists of three core agents:

1. Segmentation Agent: A two‑stage pipeline that first isolates the bone from its background using BiRefNet, then refines the mask with the Segment Anything Model 2 (SAM2) to handle occlusions and variable lighting.
2. Visual Feature Encoder: A pre‑trained EfficientNet V2 S backbone extracts high‑level image embeddings from the segmented bone region.
3. Morphometric Fusion Layer: Standardized osteometric measurements (e.g., length, width, curvature) are concatenated with the visual embeddings and passed through a shallow fully‑connected network that learns joint representations.

By training this end‑to‑end system on a curated dataset of over 10,000 bone images, the model learns to weigh visual texture against metric dimensions, achieving a balanced understanding of each specimen.

How It Works in Practice

The operational workflow can be broken down into four sequential stages, each of which can be orchestrated by an AI‑driven pipeline:

1. Data Ingestion & Pre‑processing

Raw photographs from museum collections are uploaded to a central repository. An automated script normalizes image resolution, corrects color balance, and extracts any accompanying CSV files containing osteometric measurements.

2. Automatic Segmentation

The BiRefNet model generates an initial binary mask that roughly outlines the bone. SAM2 then refines this mask, correcting for gaps and ensuring that fine structures (e.g., trochlear ridges) are retained. The result is a clean, background‑free image ready for feature extraction.

3. Feature Extraction & Fusion

The segmented image is fed into EfficientNet V2 S, producing a 1280‑dimensional vector that captures shape, texture, and shading cues. Simultaneously, the osteometric data are standardized (z‑scored) and concatenated to the visual vector, forming a multimodal feature tensor.

A lightweight fusion network—comprising two dense layers with ReLU activations—learns interactions between the two modalities, outputting class probabilities for either skeletal element or taxonomic family.

4. Inference & Post‑Processing

During deployment, the system returns a ranked list of predictions (top‑1, top‑3). For taxonomic tasks, the top‑3 list is especially valuable because it surfaces plausible families that can be reviewed by a human expert, reducing false negatives.

What distinguishes this pipeline from prior work is the tight coupling of segmentation and feature fusion. By ensuring that the visual encoder only sees the bone itself, the model avoids background bias, while the fusion layer explicitly learns how metric dimensions modulate visual patterns.

Evaluation & Results

The researchers evaluated the framework on two distinct classification tasks using a held‑out test set (≈20% of the data):

Skeletal Element Identification

• Goal: Distinguish among major bone types (e.g., femur, humerus, tibiotarsus).
• Outcome: The multimodal model achieved 86% top‑1 accuracy, surpassing a vision‑only baseline (≈78%) and a morphometry‑only baseline (≈71%).

Family‑Level Taxonomic Classification

• Goal: Assign each bone fragment to its correct avian family (e.g., Anatidae, Accipitridae).
• Outcome: Top‑1 accuracy reached 51%, while top‑3 accuracy climbed to 75%, indicating that the correct family was frequently among the three most likely predictions.

These results demonstrate two key insights:

1. Visual cues dominate bone‑type discrimination, but morphometric data provide a decisive edge in ambiguous cases.
2. Even when the model cannot pinpoint the exact family, it reliably narrows the search space, offering a practical decision‑support tool for archaeologists.

Why This Matters for AI Systems and Agents

From an AI engineering perspective, the study showcases a reproducible pattern for building multimodal agents that must operate on heterogeneous scientific data:

• Modular design: Each component (segmentation, visual encoder, metric fusion) can be swapped or upgraded without retraining the entire pipeline, supporting continuous improvement.
• Interpretability: By exposing both visual embeddings and raw measurements, the system offers traceable reasoning paths—critical for compliance in heritage domains.
• Scalable orchestration: The workflow aligns with modern Workflow automation studio concepts, where data ingestion, model inference, and result curation can be chained as micro‑services.
• Human‑in‑the‑loop augmentation: The top‑3 taxonomic output enables AI‑assisted agents to suggest candidate families, allowing curators to confirm or reject predictions, thereby accelerating cataloging while preserving expert oversight.

For developers of AI agents in cultural heritage, the paper provides a concrete blueprint for integrating visual perception with domain‑specific numeric descriptors—a pattern that can be generalized to pottery, lithics, or even ancient texts.

What Comes Next

While the proof‑of‑concept is compelling, several avenues remain open for refinement and broader impact:

Limitations

• Dataset bias: The training set, though large, is skewed toward well‑preserved specimens from European museums, potentially limiting transferability to tropical or fragmented assemblages.
• Family granularity: Achieving only 51% top‑1 accuracy at the family level suggests that finer‑grained taxonomic cues (e.g., DNA, micro‑CT) may be needed for higher precision.
• Real‑time constraints: The two‑stage segmentation pipeline adds latency, which could be problematic for on‑site field applications.

Future Research Directions

• Incorporate 3‑D scanning data to enrich the visual modality and capture internal bone architecture.
• Explore contrastive learning techniques that align visual and metric spaces more tightly, potentially boosting family‑level performance.
• Deploy the system within an Enterprise AI platform by UBOS to test end‑to‑end integration with museum databases and collaborative annotation tools.
• Develop a ChatGPT and Telegram integration that lets field archaeologists upload a bone photo via a messaging app and receive instant classification suggestions.
• Leverage Chroma DB integration for vector‑based storage of multimodal embeddings, enabling fast similarity search across global collections.

By addressing these challenges, the community can move from a laboratory prototype to a production‑grade service that democratizes avian bone identification for museums, universities, and heritage NGOs worldwide.

References

Multimodal fusion of visual and morphometric features for avian bone classification

Illustration

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.