Andrii Bidochko

• Updated: August 14, 2026
• 7 min read

COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention

Direct Answer

The COSI‑Lab dataset introduces a large‑scale, multimodal collection of social interaction recordings captured in a controlled “conference living lab” environment, enabling researchers to infer apparent intent from synchronized audio, video, motion, and physiological signals. Its release matters because it fills a critical gap in publicly available data for training and evaluating AI systems that must understand nuanced human intentions in real‑time collaborative settings.

Background: Why This Problem Is Hard

Modeling human intent is a cornerstone of socially aware AI, yet existing datasets fall short in three fundamental ways:

• Modal scarcity: Most public corpora provide only text or video, ignoring the rich cues from speech prosody, body kinematics, and physiological responses that together shape intent.
• Contextual thinness: Intent is highly dependent on the surrounding interaction context—who is speaking, the task at hand, and the shared environment. Traditional datasets capture isolated utterances, stripping away this context.
• Annotation granularity: Human intent is subjective and evolves over seconds. Coarse labels (e.g., “question” vs. “statement”) cannot support fine‑grained inference needed for autonomous agents that must anticipate actions before they occur.

These limitations hinder progress in domains such as collaborative robotics, virtual meeting assistants, and adaptive learning platforms, where misreading intent can lead to breakdowns in trust and efficiency.

What the Researchers Propose

The authors present COSI‑Lab (Conference Social Interaction Lab), a framework that couples a physical conference‑room‑scale testbed with a synchronized data‑capture pipeline. The core contributions are:

1. Multimodal sensor suite: High‑definition RGB‑D cameras, microphone arrays, wearable inertial measurement units (IMUs), and contactless physiological sensors (e.g., remote photoplethysmography) record participants simultaneously.
2. Apparent Intent Annotation (AII): Trained annotators label intent at 0.5‑second intervals using a taxonomy that spans communicative (e.g., “request clarification”), affective (e.g., “express agreement”), and collaborative (e.g., “offer assistance”) dimensions.
3. Living‑lab protocol: Scenarios mimic real conference activities—paper presentations, Q&A sessions, and informal networking—ensuring ecological validity while retaining experimental control.

By releasing both raw sensor streams and the AII labels, the dataset enables end‑to‑end training of models that fuse audio‑visual, motion, and physiological cues to predict intent before it is explicitly verbalized.

How It Works in Practice

Conceptual Workflow

The COSI‑Lab pipeline can be broken into four sequential stages:

1. Capture: Sensors stream synchronized data to a central server with sub‑millisecond timestamp alignment.
2. Pre‑processing: Automated pipelines normalize audio levels, extract facial landmarks, compute skeletal joint trajectories, and derive heart‑rate variability from video‑based photoplethysmography.
3. Annotation: Human annotators view multi‑view playback and assign intent tags using a custom web interface that enforces inter‑rater reliability thresholds.
4. Packaging: Processed streams and annotations are stored in a hierarchical file format (HDF5) with metadata describing participant demographics, scenario type, and sensor configuration.

Component Interactions

Each component communicates via lightweight RESTful endpoints, allowing researchers to replace or augment parts of the pipeline. For example, a research team could swap the default facial landmark extractor with a deep learning‑based model without disrupting the downstream annotation workflow.

What Sets COSI‑Lab Apart

• Temporal resolution: Intent labels are provided at 0.5‑second granularity, supporting predictive modeling.
• Sensor diversity: The inclusion of physiological signals is rare in public datasets, opening avenues for affect‑aware intent inference.
• Scenario realism: By embedding interactions in a conference‑living‑lab, the data captures natural turn‑taking, interruptions, and non‑verbal negotiation cues.

Evaluation & Results

The authors benchmarked three families of models on the COSI‑Lab test split:

• Unimodal baselines: Audio‑only and video‑only convolutional networks.
• Multimodal early‑fusion: Concatenated feature vectors from all sensors fed into a transformer encoder.
• Hierarchical late‑fusion: Separate modality‑specific encoders whose outputs are merged by a gating network that learns to weight cues dynamically.

Key findings include:

• Late‑fusion models outperformed early‑fusion by an average of 7 % in F1‑score, highlighting the benefit of modality‑specific processing.
• Physiological features contributed a 3 % boost in detecting affective intent (e.g., “express agreement”), confirming their complementary value.
• Predictive latency was reduced to under 300 ms for the best model, demonstrating feasibility for real‑time agent deployment.

These results collectively demonstrate that the COSI‑Lab dataset supports the development of intent inference systems that are both accurate and responsive enough for interactive AI agents.

Why This Matters for AI Systems and Agents

For practitioners building conversational assistants, collaborative robots, or meeting analytics platforms, COSI‑Lab offers a turnkey resource to train models that understand not just what users say, but what they intend to do next. The fine‑grained intent labels enable:

• Proactive assistance: Agents can anticipate a speaker’s request for clarification and surface relevant documents before the question is fully articulated.
• Emotion‑aware coordination: By detecting subtle physiological signs of agreement or disagreement, multi‑agent systems can adjust turn‑taking strategies to maintain conversational flow.
• Cross‑modal robustness: In noisy environments where audio degrades, motion or physiological cues can fill the gap, improving reliability of intent detection.

Integrating these capabilities into existing platforms is straightforward thanks to the dataset’s open‑source preprocessing scripts and standardized data schema. For example, developers can plug the pretrained hierarchical fusion model into the UBOS platform overview to enrich AI marketing agents with real‑time intent awareness.

What Comes Next

While COSI‑Lab marks a significant step forward, several avenues remain open for expansion:

• Scalability: Extending the living‑lab to larger groups and diverse cultural settings would improve generalizability across global workforces.
• Label depth: Incorporating hierarchical intent taxonomies that capture long‑term goals (e.g., “build consensus”) could enable strategic planning for autonomous agents.
• Privacy‑preserving pipelines: Research into federated learning on multimodal streams would allow organizations to benefit from the dataset without exposing raw sensor data.

Future research can also explore coupling COSI‑Lab with synthetic data generators to augment rare intent classes, thereby reducing the need for costly manual annotation.

Organizations interested in deploying intent‑aware agents at scale may consider joining the UBOS partner program, which offers dedicated support for integrating multimodal datasets into enterprise AI workflows.

How to Access and Use the Dataset

The COSI‑Lab dataset is released under a Creative Commons Attribution‑NonCommercial license and can be downloaded from the project’s GitHub repository. The repository includes:

• Raw sensor recordings (≈ 1.2 TB) organized by scenario and participant.
• Pre‑processed feature tensors for audio, video, motion, and physiology.
• Annotation files in JSON format aligned with timestamps.
• Python utilities for data loading, synchronization checks, and baseline model training.

To get started quickly, users can clone the repo and run the provided Docker image, which sets up the entire environment—including the Workflow automation studio for custom preprocessing pipelines.

SEO‑Friendly Summary and Call‑to‑Action

In summary, the COSI‑Lab dataset delivers a comprehensive, high‑resolution multimodal benchmark for apparent intent inference, addressing long‑standing gaps in sensor diversity, annotation granularity, and ecological validity. Researchers and AI engineers can leverage this resource to build more anticipatory, context‑aware agents that operate effectively in collaborative environments such as conferences, remote meetings, and hybrid workspaces.

Ready to explore intent‑aware AI? Download the dataset today, experiment with the provided baselines, and consider contributing new annotations to expand the community’s collective knowledge.

For the original research paper, see COSI‑Lab dataset paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.