Andrii Bidochko

• Updated: August 23, 2026
• 6 min read

VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

Direct Answer

VICBench is a newly released, multi‑language benchmark that aggregates 100 verified vulnerability‑inducing commits (VICs) across Python, Java, and C++ projects, each tied to a distinct CVE. It matters because it supplies the most realistic, large‑scale ground truth for evaluating automated vulnerability detection tools, exposing the shortcomings of current state‑of‑the‑art methods.

Background: Why This Problem Is Hard

Software security teams rely on automated scanners, static analysis, and increasingly on AI‑driven SZZ‑style algorithms to locate the commit that first introduced a flaw. The core difficulty lies in the scarcity of high‑quality training and test data that faithfully represent real‑world code evolution. Existing datasets suffer from three systemic weaknesses:

• Language narrowness: Most public collections focus on a single language (often Java or C), leaving Python‑heavy ecosystems under‑represented.
• Patch simplicity: Prior benchmarks contain tiny bug‑fix patches—often fewer than ten lines—whereas production‑grade fixes routinely span dozens of lines, intertwining multiple code paths.
• Project homogeneity: Datasets are dominated by a handful of large open‑source projects, limiting the diversity of coding styles, dependency graphs, and build systems that a detection tool must handle.

These gaps translate into an evaluation paradox: a tool that scores well on a narrow benchmark may crumble when faced with the tangled, multi‑language reality of modern codebases. As enterprises adopt AI agents to automate security triage, the need for a robust, cross‑language benchmark becomes a strategic imperative.

What the Researchers Propose

The authors introduce VICBench, a curated collection of 100 vulnerability‑inducing commits, each linked to a unique CVE and verified through a dual‑annotation pipeline:

1. Human expert validation: Security researchers manually confirm that the selected commit indeed introduces the vulnerability, cross‑checking against official CVE descriptions.
2. Agentic workflow assistance: An autonomous annotation agent extracts candidate commits using heuristics, then presents them for expert review, dramatically reducing manual effort while preserving accuracy.

The benchmark spans 88 distinct open‑source projects, covering 48 CWE (Common Weakness Enumeration) types. By deliberately selecting commits with an average size of 252.5 lines and corresponding fixes averaging 38.6 lines, VICBench captures the complexity of real‑world remediation.

How It Works in Practice

The VICBench creation pipeline can be visualized as a three‑stage workflow:

1. Candidate Extraction: An LLM‑augmented crawler scans public vulnerability databases (e.g., NVD) for CVE identifiers, then pulls the associated repository histories.
2. Agent‑Driven Pre‑filtering: An autonomous agent (the “annotation bot”) applies heuristics such as commit message keywords, file‑type filters, and diff size thresholds to shortlist potential VICs.
3. Human Confirmation Loop: Security experts review each shortlisted commit, verify the presence of the vulnerability, and annotate the exact lines that constitute the VIC. The final dataset records both the inducing commit and the subsequent fixing commit.

This hybrid approach differs from prior efforts that either relied solely on manual curation—making scaling impossible—or on fully automated heuristics that often mislabel non‑vulnerable changes. By combining the speed of agents with the judgment of experts, VICBench achieves both breadth (multi‑language, multi‑project) and depth (large, realistic patches).

Evaluation & Results

To demonstrate VICBench’s diagnostic power, the authors benchmarked two leading SZZ‑style detection methods: V‑SZZ and LLM4SZZ. Both tools attempt to trace a vulnerability back to its introducing commit using version‑control metadata and, in the case of LLM4SZZ, large language model reasoning.

Across the 100 VICs, the tools achieved the following F1 scores:

• V‑SZZ: 33.3%
• LLM4SZZ: 40.1%

These numbers are starkly lower than the >80% scores reported on older, simpler benchmarks. The gap highlights two critical insights:

1. Complexity penalty: Larger, multi‑file patches introduce noise that confuses heuristic‑based blame assignment.
2. Language variance: Tools tuned for Java or C++ struggle when confronted with Python’s dynamic typing and idiomatic patterns.

By publishing the full dataset and evaluation scripts, the authors enable the research community to iterate on detection algorithms with a realistic yardstick. The VICBench paper also includes a detailed error analysis, pinpointing failure modes such as “misattributed refactoring” and “indirect dependency injection,” which are fertile ground for future model improvements.

Why This Matters for AI Systems and Agents

For organizations that embed AI agents into their DevSecOps pipelines, VICBench offers a concrete, reproducible testbed to validate security‑oriented models before production rollout. The benchmark’s multi‑language scope aligns with the heterogeneous tech stacks of most enterprises, ensuring that an agent trained on VICBench will not be blindsided by a language it never saw during training.

Specific implications include:

• Model robustness: Training on VICBench forces agents to learn cross‑language patterns of insecure code, reducing overfitting to language‑specific quirks.
• Orchestration confidence: When an AI orchestrator (e.g., a CI/CD security bot) reports a high‑confidence vulnerability, teams can reference VICBench‑derived metrics to gauge the likelihood of false positives.
• Continuous improvement loops: By feeding missed VICs back into the training pipeline, agents can iteratively refine their detection heuristics.

Practically, security teams can integrate VICBench into their UBOS platform overview to benchmark custom LLM pipelines, or leverage the Workflow automation studio to automate the ingestion of new CVEs into the benchmark as they are disclosed.

What Comes Next

While VICBench marks a significant step forward, several limitations remain:

• Language expansion: The current trio—Python, Java, C++—covers a large share of open‑source code but omits Go, Rust, and JavaScript, which are increasingly prevalent in cloud‑native environments.
• Scale of annotation: Even with agentic assistance, manual verification caps the dataset size. Future work could explore semi‑supervised labeling to grow beyond 100 VICs.
• Dynamic analysis integration: VICBench focuses on static diffs; coupling it with runtime exploit traces would enrich the ground truth for fuzzing‑oriented agents.

Potential research directions include:

1. Developing language‑agnostic embeddings that capture vulnerability semantics across syntactic differences.
2. Designing reinforcement‑learning agents that propose candidate VICs and receive expert feedback as a reward signal.
3. Embedding VICBench into Enterprise AI platform by UBOS to enable large‑scale, on‑premise evaluation for regulated industries.

For developers eager to experiment, the benchmark is openly available for download, and the UBOS ecosystem provides ready‑made connectors—such as the OpenAI ChatGPT integration—to plug LLM‑based detectors into existing security workflows.

By establishing a rigorous, multi‑language baseline, VICBench equips AI practitioners, security researchers, and enterprise engineers with the evidence they need to build more trustworthy vulnerability detection agents. Explore the dataset, run your own experiments, and help shape the next generation of secure AI‑driven development pipelines.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.