MAVEN: A Multi-stage Agentic Annotation Pipeline for Video Reasoning Tasks

Direct Answer

MAVEN introduces a multi‑stage, agent‑driven pipeline that automatically converts raw video streams into richly structured training data for Vision‑Language Models (VLMs), complete with chain‑of‑thought reasoning traces. By doing so, it removes the manual bottleneck that has limited large‑scale video reasoning and delivers performance gains that surpass leading proprietary models on real‑world surveillance tasks.

Background: Why This Problem Is Hard

Training VLMs to answer “what happened, when, where, why, and with what consequence” in videos requires annotations that capture temporal order, spatial context, causal links, and outcome semantics. Traditional annotation pipelines rely on human labelers to watch hours of footage, write captions, and then craft question‑answer pairs. This process is:

• Costly: Professional annotators charge $15‑$30 per minute of video, making large‑scale datasets prohibitively expensive.
• Inconsistent: Human judgments vary in granularity, leading to noisy training signals.
• Scalable only for narrow domains: Existing datasets focus on sports or movies, leaving surveillance, traffic, and industrial safety under‑represented.

Current automated approaches either generate shallow captions or rely on pre‑trained models that inherit the same data limitations they aim to solve. Consequently, VLMs struggle with multi‑step reasoning, temporal grounding, and domain‑specific nuance—critical capabilities for safety‑critical AI agents.

What the Researchers Propose

The MAVEN framework reframes video annotation as a hierarchical, agentic workflow. Its core contribution is the Multi‑Scale Spatio‑Temporal Event Description (MSTED), an intermediate representation that fuses three captioning levels:

1. Fine‑grained frame‑level descriptions that pinpoint objects and actions.
2. Mid‑level segment summaries that capture short‑term event arcs.
3. High‑level narrative overviews that articulate causal chains and outcomes.

These layers are generated by specialized agents—each a lightweight language model guided by custom prompts. Once MSTED is assembled, a downstream agent consumes it to produce chain‑of‑thought (CoT) reasoning traces and a suite of question‑answer formats (multiple‑choice, open‑ended, temporal ordering, etc.). The pipeline is fully agent‑driven: when a new video domain appears, a meta‑agent redesigns prompts top‑down, eliminating manual re‑engineering.

How It Works in Practice

The MAVEN workflow can be visualized as a four‑stage assembly line:

1. Ingestion & Pre‑processing: Raw video is split into overlapping clips, and a vision encoder extracts spatio‑temporal features.
2. Agentic Captioning: Three parallel agents generate the fine‑grained, mid‑level, and high‑level captions, each conditioned on the previous level’s output.
3. MSTED Synthesis: A coordinator agent merges the three streams into a single, hierarchical event description, explicitly marking timestamps, locations, actors, and causal links.
4. CoT Q&A Generation: Using the MSTED as the sole input, a reasoning agent produces step‑by‑step thought traces and formats them into diverse question types for downstream VLM fine‑tuning.

What sets MAVEN apart is its hierarchical refinement loop. After an initial pass, an error‑classification agent compares generated annotations against a taxonomy (e.g., temporal misalignment, missing actors, causal ambiguity). Errors are traced back to the originating stage, prompting targeted prompt rewrites or even structural adjustments (e.g., adding an extra captioning agent). This loop iterates until a predefined quality threshold is met, ensuring that the final dataset is both consistent and richly detailed.

Evaluation & Results

The authors applied MAVEN to a corpus of 5,300 traffic‑camera videos, producing over 200,000 question‑answer pairs with CoT traces. They fine‑tuned the Cosmos‑Reason2‑8B model on this data and evaluated on three benchmarks:

• Private CCTV set: MAVEN‑trained Cosmos achieved a 38.8‑point lift in multiple‑choice accuracy over zero‑shot baselines, outperforming Gemini 2.5 Pro and Gemini 3.1 Flash.
• AccidentBench (dash‑cam domain): Training solely on CCTV data raised Cosmos‑Reason2 by 10.7 points, matching Gemini 2.5 Pro despite never seeing dash‑cam footage.
• Domain‑adapted dash‑cam augmentation: Adding MAVEN‑generated dash‑cam annotations narrowed the gap to Gemini 3.1 Flash; subsequent reinforcement‑learning post‑training pushed overall performance past both proprietary baselines.

Qualitative case studies on warehouse surveillance and public‑safety footage demonstrated that the same pipeline could be re‑configured in under an hour to handle new environments, confirming the claim of rapid domain adaptation. The results collectively illustrate that a well‑structured, agentic annotation pipeline can produce training data that not only rivals but exceeds the capabilities of large, closed‑source models.

Why This Matters for AI Systems and Agents

For practitioners building AI agents that must interpret video—whether for security monitoring, autonomous driving, or industrial safety—the MAVEN approach offers a scalable path to high‑quality training data without the expense of manual labeling. The hierarchical MSTED representation aligns naturally with the reasoning architectures of modern LLM‑augmented agents, enabling:

• Fine‑grained temporal grounding, essential for event sequencing and anomaly detection.
• Explicit causal chains that improve explainability and trustworthiness in safety‑critical deployments.
• Plug‑and‑play domain adaptation, reducing time‑to‑market for new surveillance scenarios.

Integrating MAVEN‑style pipelines with existing automation platforms can accelerate the creation of bespoke video‑reasoning modules. For example, the UBOS platform overview provides a unified environment where generated MSTEDs can be stored, indexed, and served to downstream agents. The Workflow automation studio can orchestrate the multi‑stage agents, monitor the refinement loop, and trigger re‑training of VLMs as new footage arrives. Finally, the AI marketing agents illustrate how the same reasoning backbone can be repurposed for non‑visual domains, highlighting the versatility of the underlying agentic design.

What Comes Next

While MAVEN marks a significant step forward, several open challenges remain:

• Scalability of the refinement loop: As datasets grow to millions of hours, the computational cost of hierarchical error analysis may become a bottleneck. Future work could explore lightweight proxy models or active‑learning strategies to prioritize the most informative clips.
• Generalization across modalities: Extending MSTED to incorporate audio cues, sensor metadata, or textual overlays would broaden applicability to multimodal surveillance systems.
• Robustness to adversarial content: In security contexts, actors may deliberately obscure actions. Embedding adversarial detection agents within the pipeline could safeguard annotation quality.

Potential research directions include:

1. Co‑training the captioning agents with reinforcement signals from downstream VLM performance, creating a closed feedback loop.
2. Leveraging foundation models that natively understand spatio‑temporal graphs to replace the three‑stage caption hierarchy with a single, end‑to‑end graph generator.
3. Deploying MAVEN in edge environments where on‑device inference can generate annotations in real time, enabling immediate alerting for critical events.

Practitioners interested in experimenting with MAVEN can start by reviewing the MAVEN paper on arXiv, which provides full implementation details, prompt templates, and the error taxonomy. By integrating the pipeline with platforms like UBOS, organizations can rapidly prototype domain‑specific video reasoning agents, reduce annotation costs, and achieve performance that rivals the most advanced proprietary systems.