Andrii Bidochko

• Updated: July 8, 2026
• 6 min read

VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

Direct Answer

VERITAS is a domain‑agnostic, command‑line replication framework that lets researchers feed a paper, a code repository, or both into an automated pipeline, extracts the paper’s claims, runs the original methodology, resolves failures on the fly, and produces a weighted Replication Score with a detailed fix log. It matters because it turns the costly, manual process of reproducing scientific results into a scalable, auditable workflow that can keep pace with the accelerating volume of AI‑driven publications.

Background: Why This Problem Is Hard

Scientific reproducibility has become a bottleneck for both academia and industry. The surge of AI‑generated manuscripts, pre‑prints, and rapid‑release codebases means that reviewers and downstream engineers must verify claims faster than ever. Traditional replication suffers from three intertwined challenges:

• Fragmented tooling: Existing replication efforts are tied to bespoke benchmarks, requiring custom agents that only operate inside a narrow pipeline.
• Manual debugging overhead: When code fails to run—due to missing dependencies, environment mismatches, or undocumented preprocessing steps—human experts spend hours or days fixing the issues.
• Lack of standardized evidence: Even when a replication succeeds, there is no uniform metric that captures how closely the reproduced results align with the original claims, nor a transparent log of what was altered.

These pain points are amplified in interdisciplinary domains such as medicine, astrophysics, and social science, where the methodological stack can span multiple languages, data formats, and hardware requirements. As a result, many published findings remain unverified, eroding trust in the scientific record.

What the Researchers Propose

The authors introduce VERITAS (Verification Engine for Reproducible and Transparent Automated Science), a general‑purpose replication framework built around CLI coding agents. At a high level, VERITAS consists of three logical layers:

1. Claim Extraction Layer: Uses large‑language‑model (LLM) agents to parse the manuscript, identify explicit experimental claims, and translate them into machine‑readable specifications (e.g., target metrics, datasets, hyper‑parameters).
2. Execution & Repair Layer: Spins up isolated containers, runs the original code, and employs a second LLM‑driven agent that monitors logs, detects failures, and automatically applies patches (dependency fixes, path corrections, data preprocessing tweaks).
3. Evaluation & Scoring Layer: Compares reproduced outcomes against the extracted claim specifications, computes an importance‑weighted Replication Score, and generates a severity‑rated fix log that documents every modification.

Crucially, VERITAS does not require any bespoke benchmark harness; it can ingest any publicly available repository or a zip file of code, making it truly domain‑agnostic.

For a deeper dive into the original methodology, see the VERITAS paper on arXiv.

How It Works in Practice

The end‑to‑end workflow can be visualized as a loop of four stages, each orchestrated by a dedicated CLI agent:

1. Ingestion: The user supplies a DOI, PDF, or Git URL. VERITAS clones the repository and extracts the manuscript text.
2. Claim Mining: An LLM agent scans the text, extracts statements like “Model X achieves 92% accuracy on Dataset Y,” and stores them as structured JSON objects.
3. Automated Execution: A sandboxed runtime launches the original training script. If the run crashes, a repair agent reads the error, searches for known fix patterns, and applies a patch (e.g., updating a deprecated library call).
4. Scoring & Reporting: After successful runs, the evaluation agent computes the Replication Score, weighting each claim by its reported importance in the paper, and produces a human‑readable log.

The following diagram illustrates the interaction between these components:

What sets VERITAS apart from prior benchmarks is its self‑healing capability: the repair agent continuously learns from previous fixes, building a knowledge base that reduces future manual intervention. Moreover, because the entire pipeline is driven by CLI agents, it can be integrated into existing CI/CD systems, enabling nightly reproducibility checks for any research codebase.

Evaluation & Results

To validate VERITAS, the authors benchmarked it on two large‑scale suites:

• CORE‑Bench: 35 papers spanning computer‑science subfields, each with a mix of code quality and documentation completeness.
• ReplicationBench: 30 interdisciplinary papers from medicine, social science, and astrophysics, deliberately chosen for their heterogeneous toolchains.

VERITAS was compared against two strong baselines built on Claude Code agents, all running on identical hardware. The key findings were:

• Higher Replication Success Rate: VERITAS reproduced 87% of claims across both benchmarks, versus 71% for the best baseline.
• Improved Score Accuracy: The importance‑weighted Replication Score correlated 0.92 with human expert assessments, a notable jump from the baseline’s 0.78.
• Reduced Human Intervention: The average number of manual patches per paper dropped from 4.3 (baseline) to 1.1 (VERITAS), saving an estimated 12 hours of engineer time per paper.
• Scalable Runtime: End‑to‑end replication of a typical deep‑learning experiment completed in under 45 minutes, demonstrating feasibility for large‑scale nightly audits.

These results indicate that VERITAS not only outperforms existing agent‑based replication tools but also delivers a transparent audit trail that can be trusted by reviewers and downstream developers.

Why This Matters for AI Systems and Agents

For AI practitioners, VERITAS offers a concrete pathway to embed reproducibility into the product development lifecycle:

• Agent‑Centric Orchestration: By exposing each stage as a CLI agent, VERITAS can be composed with existing orchestration platforms, enabling automated verification pipelines that run alongside model training and deployment.
• Continuous Validation: Teams can schedule nightly runs that flag regressions or undocumented changes, reducing technical debt in research codebases.
• Trustworthy Model Cards: The Replication Score and fix log can be attached to model cards, giving customers verifiable evidence of performance claims.
• Accelerated Innovation: Start‑ups can quickly assess the feasibility of adopting a published method without allocating weeks of engineering effort.

Integrating VERITAS with broader AI infrastructure is straightforward. For example, the Workflow automation studio can trigger VERITAS pipelines as part of a larger data‑science workflow, while the UBOS platform overview provides the container orchestration layer needed for isolated execution.

What Comes Next

While VERITAS marks a significant leap, several open challenges remain:

• Domain‑Specific Knowledge: Certain fields (e.g., high‑energy physics) require specialized simulation software that is not yet covered by the generic repair agent.
• Scalability to Massive Datasets: Replicating experiments that involve petabyte‑scale data will demand tighter integration with distributed storage solutions.
• Human‑in‑the‑Loop Feedback: Incorporating expert review of the fix log could further improve the quality of automated patches.

Future research directions include extending VERITAS with a plug‑in architecture for domain‑specific agents, leveraging Chroma DB integration for semantic search over claim repositories, and exploring voice‑enabled reporting via the ElevenLabs AI voice integration. These extensions would make VERITAS a cornerstone of an emerging ecosystem where reproducibility is as automated as continuous integration.

Organizations interested in adopting VERITAS can start by exploring the UBOS templates for quick start, which include pre‑configured pipelines for common scientific stacks.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.