Andrii Bidochko

• Updated: March 22, 2026
• 3 min read

Building a Specialized Test Suite for Detecting, Quantifying, and Mitigating AI Agent Hallucinations with OpenClaw

Problem Statement

AI agents, especially large language models, can produce hallucinations – confident statements that are factually incorrect or nonsensical. In production environments these hallucinations can erode trust, cause downstream errors, and even lead to security risks. Developers need a systematic way to detect, quantify, and mitigate such behaviour before deploying agents.

Why OpenClaw?

The OpenClaw Agent Evaluation Framework provides a flexible sandbox for running agents against a suite of custom‑crafted scenarios. By extending OpenClaw with a dedicated hallucination‑test suite, you gain repeatable, automated insight into where your agent fails.

Step‑by‑Step Implementation

1. Set up the OpenClaw environment

   git clone https://github.com/ubos-tech/openclaw.git
   cd openclaw
   pip install -r requirements.txt
   

2. Define a hallucination test case schema

   Create a JSON file (hallucination_cases.json) that describes prompts designed to trigger hallucinations and the expected factual answer.

   {
     "cases": [
       {
         "id": "HC001",
         "prompt": "What is the capital of Atlantis?",
         "expected": "[No factual answer]",
         "type": "factual_mismatch"
       },
       {
         "id": "HC002",
         "prompt": "List the three laws of robotics as defined by Asimov.",
         "expected": "1. A robot may not injure a human being...",
         "type": "partial_correct"
       }
     ]
   }
   

3. Implement the test runner

   Add a new module tests/hallucination_test.py that loads the JSON, sends each prompt to the agent, and compares the response.

   import json
   from openclaw.agent import Agent
   
   def load_cases(path):
       with open(path) as f:
           return json.load(f)["cases"]
   
   def evaluate(agent: Agent, cases):
       results = []
       for case in cases:
           response = agent.run(case["prompt"])
           match = case["expected"] in response
           results.append({
               "id": case["id"],
               "prompt": case["prompt"],
               "response": response,
               "match": match,
               "type": case["type"]
           })
       return results
   

4. Quantify hallucination severity

   Assign a score (0‑1) per case: 0 = fully correct, 1 = completely hallucinated. Aggregate to a Hallucination Index for the whole suite.

   def compute_index(results):
       total = len(results)
       hallucinations = sum(1 for r in results if not r["match"])
       return hallucinations / total
   

5. Mitigation hooks

   When a case fails, automatically trigger one of the following mitigation strategies:

   • Re‑prompt with chain‑of‑thought prompting.
   • Apply a post‑processing validator that flags unsupported facts.
   • Log the failure for human review.
6. Run the suite and review the report

   agent = Agent(model="gpt‑4o-mini")
   cases = load_cases("hallucination_cases.json")
   results = evaluate(agent, cases)
   index = compute_index(results)
   print(f"Hallucination Index: {index:.2%}")
   # Optionally export to HTML
   

Example Test Cases

Best‑Practice Tips

• Curate diverse hallucination prompts – include factual‑mismatch, ambiguous‑question, and domain‑specific scenarios.
• Version‑control your test suite – store .json and .py files in Git to track changes over time.
• Automate in CI/CD – run the suite on every model update and fail the build if the Hallucination Index exceeds a threshold (e.g., 5%).
• Combine with human review – automated checks catch obvious errors; a sample of edge cases should still be examined manually.
• Document mitigation actions – keep a log of which strategy fixed which case for future reference.

By integrating this specialized test suite into your OpenClaw workflow, you gain continuous visibility into hallucination risks and a repeatable path to improve model reliability.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.