Andrii Bidochko

• Updated: July 3, 2026
• 7 min read

Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic

Direct Answer

The paper introduces a novel “task‑arithmetic” framework that merges multiple fine‑tuned vision‑language models to achieve robust zero‑shot generalization for open‑vocabulary action recognition (OVAR). By recombining knowledge from existing public datasets without any target‑domain training, the approach delivers superior performance on out‑of‑distribution video streams, eliminating costly fine‑tuning and privacy‑sensitive data collection.

Background: Why This Problem Is Hard

Open‑vocabulary action recognition aims to identify actions that were never seen during training by leveraging the semantic richness of large‑scale vision‑language models (e.g., CLIP‑based backbones). In practice, two intertwined challenges limit its adoption:

• Domain shift. Video data collected in the wild varies dramatically in lighting, camera motion, background clutter, and cultural context. Models trained on curated benchmarks (e.g., Kinetics, Something‑Something) often collapse when deployed on surveillance feeds, sports broadcasts, or user‑generated content.
• Data‑privacy and regulatory constraints. Many industries (healthcare, finance, public safety) cannot legally share raw video for model fine‑tuning, making it infeasible to adapt a generic OVAR system to their specific domain.

Current work typically addresses these issues by fine‑tuning a pre‑trained backbone on a domain‑specific dataset. While effective, this solution incurs three major costs:

1. Collecting and annotating enough video clips to cover the target action space.
2. Running expensive GPU training pipelines for each new deployment.
3. Potentially violating privacy regulations when raw video leaves the organization.

Consequently, there is a pressing need for a method that can reuse existing public models and datasets to produce a single, robust OVAR system that works zero‑shot on unseen domains.

What the Researchers Propose

The authors propose a two‑step “task‑arithmetic” pipeline:

1. Extract Task Vectors from Publicly Fine‑Tuned Models

Each publicly released OVAR model—fine‑tuned on a distinct dataset such as HMDB‑51, UCF‑101, or AVA—encodes a “task vector” that captures how its parameters deviate from the shared base vision‑language model. By subtracting the base model’s weights from a fine‑tuned model’s weights, the resulting vector isolates the knowledge specific to that dataset’s action distribution.

2. Merge Task Vectors via Arithmetic Operations

Task vectors are combined using weighted addition and subtraction, a process the authors term “task arithmetic.” By carefully selecting coefficients, the merged vector can amplify complementary knowledge (e.g., sports actions from UCF‑101) while suppressing dataset‑specific biases (e.g., background textures unique to a single benchmark). The merged vector is then added back to the base model, producing a single network that inherits the strengths of all source models.

Crucially, this procedure requires no additional training data from the target domain, and it can be executed on a single GPU in under an hour.

How It Works in Practice

The operational workflow can be broken down into four concrete stages:

1. Model Collection. Gather a portfolio of publicly released OVAR models, each fine‑tuned on a different public dataset. The base model is typically a CLIP‑style vision‑language encoder pre‑trained on 400 M image‑text pairs.
2. Task Vector Extraction. For each collected model, compute the difference between its weights and the base model’s weights. This yields a set of task vectors Δ₁, Δ₂, …, Δₙ.
3. Arithmetic Composition. Define a coefficient vector **α** = (α₁, α₂, …, αₙ) that reflects the desired contribution of each source task. The merged vector is Δ* = Σ αᵢ·Δᵢ. The authors explore simple heuristics (equal weighting) and data‑driven strategies (optimizing α on a held‑out validation set of public videos).
4. Model Reconstruction. Add Δ* back to the base model’s weights to obtain the merged OVAR model. This model can be directly deployed for zero‑shot inference: given a textual action label, the model computes a similarity score between the label embedding and the video representation.

What distinguishes this approach from traditional ensembling is that the merged model remains a single, compact network. Inference cost is identical to that of the base model, avoiding the latency penalties of running multiple models in parallel.

Evaluation & Results

The authors evaluate the task‑arithmetic method on three widely used OVAR benchmarks, each treated as an out‑of‑distribution (OOD) target:

• HMDB‑51 OOD. Models fine‑tuned on UCF‑101, Kinetics‑400, and AVA are merged and tested on HMDB‑51 without any HMDB‑51 training data.
• AVA OOD. A merged model built from Kinetics‑400 and Something‑Something V2 is evaluated on the dense spatio‑temporal AVA actions.
• Something‑Something V2 OOD. The reverse scenario, where models trained on Kinetics‑400 and HMDB‑51 are merged and tested on Something‑Something V2.

Key findings include:

1. Zero‑shot boost. The merged model consistently outperforms the raw base CLIP model by 4–7 % top‑1 accuracy across all OOD tests.
2. Better than single fine‑tuned models. Even though each source model was never exposed to the target dataset, the merged model surpasses the best individual fine‑tuned model by 2–3 %.
3. Robustness to domain shift. Qualitative analysis shows that the merged model is less sensitive to background variations and camera motion, indicating that task arithmetic effectively averages out dataset‑specific biases.
4. Efficiency. The entire merging pipeline completes in under 45 minutes on a single RTX 3090, with no additional training epochs required.

These results demonstrate that recombining public knowledge can replace costly target‑domain fine‑tuning while delivering stronger zero‑shot performance.

Why This Matters for AI Systems and Agents

From a systems‑engineering perspective, the proposed framework offers several practical advantages:

• Plug‑and‑play deployment. Organizations can download the merged model and integrate it into existing video analytics pipelines without retraining, reducing time‑to‑value.
• Privacy‑first compliance. Since no raw target video is required, the approach aligns with GDPR, HIPAA, and other data‑protection regulations.
• Scalable agent orchestration. Multi‑modal AI agents that need to interpret user‑generated video (e.g., virtual assistants, security bots) can rely on a single, lightweight OVAR model instead of maintaining a fleet of specialized classifiers.
• Cost‑effective scaling. Cloud providers can host the merged model as a shared inference endpoint, amortizing GPU costs across many tenants.

For teams building AI‑driven products on the UBOS platform overview, the merged OVAR model can be wrapped as a micro‑service and invoked from the Workflow automation studio. This enables rapid prototyping of use cases such as automated video tagging, compliance monitoring, and real‑time sports analytics without the overhead of custom model training.

What Comes Next

While the task‑arithmetic method marks a significant step forward, several open challenges remain:

• Dynamic coefficient learning. Current experiments use static or validation‑set‑derived α coefficients. Future work could explore reinforcement learning or meta‑learning to adapt α online based on streaming video characteristics.
• Extending to multimodal actions. Actions that involve audio cues (e.g., “playing guitar”) are not captured by vision‑only models. Integrating audio‑text embeddings into the arithmetic could broaden the action vocabulary.
• Robustness to adversarial perturbations. Merged models inherit the vulnerabilities of their constituents. Systematic robustness testing is needed before deployment in safety‑critical domains.
• Open‑source ecosystem. A community‑maintained registry of fine‑tuned OVAR models would simplify vector extraction and encourage collaborative improvement.

Developers interested in experimenting with the technique can start by cloning the GitHub repository and following the provided scripts for vector extraction and merging. For enterprises looking to embed the merged model into production, the Enterprise AI platform by UBOS offers managed deployment, monitoring, and scaling capabilities.

References

• Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic – Original arXiv pre‑print.
• Radford, A. et al., “Learning Transferable Visual Models From Natural Language Supervision,” 2021.
• Carreira, J. et al., “Kinetics: A Large-Scale Video Dataset for Action Recognition,” 2018.

Ready to accelerate your video AI projects? Explore the UBOS solutions for SMBs or contact our About UBOS team for a personalized demo.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.