Andrii Bidochko

• Updated: June 18, 2026
• 6 min read

AssertLLM2: A Comprehensive LLM Benchmark for Assertion Generation from Design Specifications

Direct Answer

AssertLLM2 is an open‑source benchmark that evaluates how well large language models can generate SystemVerilog Assertions directly from realistic hardware design specifications. It matters because it moves LLM‑driven verification from toy examples toward the gritty, bug‑prevention and bug‑hunting scenarios that silicon teams face every day.

Background: Why This Problem Is Hard

Assertion‑based verification (ABV) is the de‑facto safety net for modern ASIC and FPGA design. Engineers write SystemVerilog Assertions (SVAs) that formally capture design intent, then use formal or simulation tools to prove that the RTL respects those intents. The manual translation from a high‑level specification—often a mix of functional blocks, timing constraints, and architectural notes—into correct SVAs is labor‑intensive, error‑prone, and a major source of verification debt.

Existing attempts to automate this translation with LLMs have hit three recurring roadblocks:

• Unrealistic inputs. Prior benchmarks feed LLMs only a handful of lines of natural‑language description, ignoring the rich, hierarchical design documents that engineers actually maintain.
• Oversimplified evaluation. Most studies measure only syntactic correctness or surface‑level similarity, which does not guarantee that the generated assertions will catch real bugs.
• Lack of bug‑focused testing. Without a systematic way to inject and detect RTL faults, it is impossible to know whether an LLM‑generated assertion set is useful for bug hunting.

These gaps leave verification teams uncertain about the true ROI of LLM‑assisted ABV, especially as design cycles shrink and the cost of a missed bug skyrockets.

What the Researchers Propose

The authors introduce AssertLLM2, a comprehensive benchmark that mirrors two real‑world verification workflows:

1. Bug‑prevention mode. LLMs receive a structured design specification and must produce a complete, dependency‑aware set of SVAs that will guard the design against future errors.
2. Bug‑hunting mode. LLMs are given a buggy RTL implementation (generated by systematic mutations) alongside the specification and must craft assertions that expose the discrepancy.

Key components of the framework include:

• Design Specification Corpus. A machine‑readable, hierarchical description of each design’s functionality, timing, and interface contracts.
• Golden RTL. A verified, dependency‑complete reference implementation that serves as the ground truth.
• Mutated RTL Variants. A suite of controlled bugs (e.g., off‑by‑one counters, missing resets, swapped signals) that simulate realistic design mistakes.
• Evaluation Engine. A multi‑layered validator that checks syntactic validity, formal provability (via model checking), coverage breadth, and mutation‑based detection rate.

How It Works in Practice

The workflow can be visualized as a three‑stage pipeline:

1. Input Preparation. For each benchmark case, the system extracts the structured specification (JSON‑like) and, in bug‑hunting mode, pairs it with a mutated RTL file.
2. LLM Prompting. A carefully engineered prompt injects the specification, optional buggy RTL, and a concise instruction (“Generate SystemVerilog Assertions that capture the functional intent and expose any mismatches”). The prompt also includes a few-shot example to steer the model toward correct syntax.
3. Post‑Processing & Validation. The raw LLM output is parsed, de‑duplicated, and fed into a verification toolchain:
   • Syntax check with a SystemVerilog parser.
   • Formal provability test using a model checker (e.g., JasperGold) against the golden RTL.
   • Coverage analysis to ensure assertions touch each functional block.
   • Mutation detection where the assertions are run against the buggy RTL; any failure indicates successful bug exposure.

What sets AssertLLM2 apart is the inclusion of buggy RTL as an explicit input for the bug‑hunting scenario. This forces the LLM to reason about *differences* rather than merely restating the specification, a capability that aligns closely with how verification engineers actually use assertions.

The diagram above illustrates the data flow from specification extraction through LLM generation to multi‑level validation.

Evaluation & Results

To assess the benchmark, the authors tested several state‑of‑the‑art LLM families (including GPT‑4, Claude‑2, and LLaMA‑2‑70B) under both bug‑prevention and bug‑hunting settings. The evaluation covered three dimensions:

• Syntactic Validity. Percentage of generated assertions that passed the SystemVerilog parser without errors.
• Formal Provability. Fraction of assertions that could be proved true against the golden RTL using bounded model checking.
• Mutation Detection Rate. Proportion of injected bugs that were successfully caught by the generated assertion suite.

Key findings include:

• All models achieved >90% syntactic validity, confirming that modern LLMs have mastered SystemVerilog syntax.
• Formal provability varied widely: GPT‑4 reached 78% while smaller models hovered around 45%, highlighting the importance of model size and instruction tuning.
• In bug‑hunting mode, the best‑performing model detected 62% of the mutated bugs, a substantial improvement over baseline rule‑based generators (≈30%).
• Coverage analysis revealed that many missed bugs were due to assertions that omitted peripheral interfaces, suggesting a need for more holistic prompting strategies.

These results demonstrate that while LLMs can reliably produce syntactically correct SVAs, achieving high formal provability and bug‑detection performance still requires careful prompt engineering, model selection, and post‑processing.

Why This Matters for AI Systems and Agents

For AI practitioners building verification assistants or autonomous design agents, AssertLLM2 offers a realistic testbed that mirrors production constraints. The benchmark’s dual‑mode design encourages agents to:

• Interpret structured specifications rather than free‑form text, aligning with the data formats used in modern EDA tools.
• Perform differential reasoning—comparing intended behavior against an observed RTL implementation—a skill that underpins many debugging and self‑repair agents.
• Integrate formal verification steps into their pipelines, moving beyond “generate‑and‑hope” toward provably correct outputs.

Adopting AssertLLM2 can accelerate the development of Enterprise AI platform by UBOS that automates verification workflows, reduces time‑to‑market, and lowers the cost of design re‑spins. Moreover, the benchmark’s open‑source nature invites community contributions, enabling a shared repository of edge‑case bugs that can be used to harden future AI agents.

What Comes Next

Despite its breadth, AssertLLM2 leaves several avenues open for exploration:

• Richer Specification Languages. Extending the input format to include timing diagrams, UVM testbench metadata, or high‑level synthesis constraints could push LLMs toward deeper semantic understanding.
• Multi‑Model Ensembles. Combining a code‑generation LLM with a reasoning‑oriented model (e.g., a chain‑of‑thought verifier) may boost provability and bug‑detection rates.
• Continuous Learning Loops. Feeding back mutation detection results into the LLM fine‑tuning pipeline could create self‑improving verification agents.
• Integration with Existing Toolchains. Embedding the benchmark’s validation engine into commercial simulators or formal tools would provide end‑to‑end automation for verification teams.

Practitioners interested in prototyping such extensions can start with the Workflow automation studio, which offers a low‑code environment for stitching together LLM prompts, EDA tool invocations, and result analytics.

Finally, the community should consider expanding the benchmark to cover emerging hardware paradigms—such as RISC‑V custom extensions, neuromorphic accelerators, and quantum control logic—where assertion semantics are still being defined.

References

• AssertLLM2: A Comprehensive LLM Benchmark for Assertion Generation from Design Specifications
• Y. Wu et al., “AssertLLM2,” arXiv preprint arXiv:2605.27472, 2026.
• JasperGold Formal Verification Tool Documentation, Cadence Design Systems.
• OpenAI, “GPT‑4 Technical Report,” 2023.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.