Andrii Bidochko

• Updated: August 23, 2026
• 8 min read

NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation

Direct Answer

NetlistBench is a newly released, structure‑verified benchmark that measures how reliably large language models (LLMs) can read, edit, and reason about SPICE netlists—the textual representation of electronic circuits. It matters because the growing reliance on LLM‑driven design assistants exposes a hidden failure mode: even minor syntax or topology errors can render a circuit simulation useless, jeopardizing the trustworthiness of AI‑augmented hardware design pipelines.

Background: Why This Problem Is Hard

Electronic design automation (EDA) has long depended on deterministic, rule‑based tools to generate and simulate SPICE netlists. These netlists, while plain text, encode a dense graph of components, hierarchical blocks, and precise numerical parameters. A single misplaced character or an incorrect node connection can cause a simulator to crash or produce misleading results.

Recent advances in generative AI have encouraged engineers to use LLMs for high‑level tasks such as schematic suggestion, component selection, and even code generation for hardware description languages. However, most evaluations focus on the model’s ability to discuss circuit theory or propose design strategies, not on its competence in handling the low‑level netlist language that directly feeds simulators.

Existing benchmarks for LLMs—such as CodeXGLUE or HumanEval—treat code as a monolithic string and evaluate correctness through unit tests. They do not enforce the strict structural invariants required by SPICE, nor do they assess the model’s ability to perform incremental edits without breaking the netlist’s topology. Consequently, developers lack a reliable yardstick for measuring the “simulation‑ready” reliability of LLM‑powered agents.

What the Researchers Propose

The authors introduce NetlistBench, a curated suite of 2,342 test cases spanning 24 distinct task families. Each task isolates a specific manipulation or recognition operation—ranging from simple parameter extraction to multi‑step hierarchical restructuring—while a deterministic, structure‑aware oracle validates the model’s output against the ground‑truth netlist.

Key components of the framework include:

• Task Generator: Synthesizes realistic netlists from a library of common analog and mixed‑signal blocks, ensuring diverse topologies and parameter ranges.
• Operation Library: Defines atomic actions (e.g., “change resistor value”, “add a MOSFET”, “merge two subcircuits”) and compound sequences that simulate real design workflows.
• Structure‑Aware Oracle: Parses both the reference and model‑produced netlists into graph representations, then checks for topology preservation, parameter validity, and hierarchical consistency.
• Evaluation Harness: Automates prompt construction, model invocation, and oracle scoring, producing per‑task accuracy as well as aggregated metrics for edit horizon and structural complexity.

How It Works in Practice

When an LLM is tasked with a NetlistBench case, the workflow proceeds as follows:

1. Prompt Assembly: The benchmark formats a concise instruction (e.g., “Add a 10 kΩ resistor between nodes N001 and N002”) together with the original netlist snippet.
2. Model Generation: The LLM produces a modified netlist text. For “non‑thinking” models, this is a single forward pass; for “reasoning‑enabled” variants, the model may emit intermediate thoughts before the final edit.
3. Parsing Stage: Both the original and edited netlists are fed into the oracle’s parser, which builds component graphs and extracts hierarchical blocks.
4. Structural Comparison: The oracle verifies that the edit respects SPICE syntax, maintains connectivity where required, and updates parameters correctly. Any violation—such as a missing semicolon or an orphaned node—results in a failure for that case.
5. Scoring: Accuracy is recorded per task, and aggregated statistics are computed across dimensions like operation type (local edit vs. device addition) and edit horizon (single vs. multi‑step edits).

This pipeline differs from prior code‑generation benchmarks by treating the netlist as a structured object rather than a flat string. The oracle’s graph‑based validation catches subtle topology errors that would otherwise pass a simple diff check.

Evaluation & Results

The study evaluated six publicly available LLMs that lack internal chain‑of‑thought prompting (“non‑thinking” models). Tasks were grouped into three broad categories:

• Local Recognition & Edits: Extracting a parameter value or tweaking a single component.
• Structural Manipulations: Adding or removing devices, re‑wiring connections, and handling hierarchical subcircuits.
• Equivalence & Long‑Horizon Editing: Determining whether two netlists describe the same circuit and performing sequences of edits that span multiple steps.

Key findings include:

• For simple local edits, models achieved between 96 % and 100 % accuracy, indicating that LLMs can reliably parse and modify isolated tokens.
• When asked to add a new device, performance dropped sharply to a range of 41 %–83 %, reflecting difficulty in preserving global connectivity.
• Equivalence judgment—deciding if two netlists are functionally identical—varied from 49 % to 90 %, highlighting the challenge of reasoning about circuit semantics.
• Enabling explicit reasoning (by prompting the model to “think step‑by‑step”) improved weaker models by up to 15 % absolute points, yet did not eliminate failures on complex hierarchical edits.
• Performance degraded proportionally with the edit horizon: multi‑step compound edits saw accuracy fall below 50 % for most models.

These results demonstrate a clear gap: while LLMs excel at surface‑level token manipulation, they struggle to maintain the deeper structural invariants required for trustworthy netlist editing.

Why This Matters for AI Systems and Agents

For AI‑driven design assistants, the reliability of low‑level operations is as critical as the quality of high‑level suggestions. A single malformed netlist can cause a downstream SPICE simulation to fail, leading to wasted compute cycles, delayed time‑to‑market, and potentially costly design re‑work.

NetlistBench provides a concrete, reproducible metric that system architects can use to:

• Benchmark new LLMs or fine‑tuned variants before integrating them into EDA pipelines.
• Identify failure patterns (e.g., hierarchical merging) and target them with specialized prompting or post‑processing validators.
• Design hybrid agents that combine LLM creativity with rule‑based guards—such as a Chroma DB integration for semantic caching of verified netlist fragments.
• Implement safety nets that automatically re‑run the oracle after each edit, rejecting outputs that break topology before they reach the simulator.

In practice, this means that companies building AI‑enhanced hardware design tools can adopt NetlistBench as part of their continuous integration testing, ensuring that each model update does not regress on netlist reliability. The benchmark also informs the design of Workflow automation studio pipelines, where LLM‑generated edits can be orchestrated alongside deterministic verification steps.

What Comes Next

While NetlistBench marks a significant step forward, several limitations remain:

• Scope of Devices: The current dataset focuses on analog and mixed‑signal components; extending to digital standard cells and RF blocks would broaden applicability.
• Real‑World Netlists: Benchmarks are generated from synthetic templates; incorporating proprietary designs from industry partners could surface additional edge cases.
• Model Diversity: Only non‑thinking LLMs were evaluated. Future work should assess chain‑of‑thought, retrieval‑augmented, and multimodal models that can leverage circuit diagrams.
• Feedback Loops: Integrating the oracle’s structural feedback directly into the model’s generation loop (e.g., via reinforcement learning) could improve edit fidelity.

Potential research directions include:

1. Developing a “netlist‑aware” tokenizer that respects component boundaries and hierarchical scopes.
2. Exploring graph‑neural‑network back‑ends that translate LLM textual suggestions into validated circuit graphs.
3. Building a community‑driven repository of challenging netlist transformations, similar to the NetlistBench paper dataset.

Practitioners interested in prototyping such pipelines can start by experimenting with the Enterprise AI platform by UBOS, which offers built‑in orchestration for LLM calls, verification steps, and versioned artifact storage.

Conclusion

NetlistBench shines a light on a previously under‑examined reliability bottleneck: the ability of LLMs to manipulate SPICE netlists without breaking structural integrity. The benchmark’s fine‑grained tasks and deterministic oracle reveal that current models excel at superficial edits but falter on deeper, topology‑preserving operations—especially as edit horizons lengthen.

For AI researchers, EDA engineers, and product teams building AI‑augmented hardware design tools, NetlistBench offers a practical yardstick to measure progress, guide model selection, and design robust safety layers. By integrating these insights early, organizations can avoid costly simulation failures and move toward truly trustworthy AI‑driven circuit design.

We encourage the community to adopt NetlistBench, contribute additional netlist scenarios, and explore hybrid approaches that combine LLM creativity with rigorous structural verification.

References

• Jiarui Ma, Jianghan Wang, Yuheng Ma, Ziyi Zhuang, Xiaoguang Liu. “NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation.” arXiv preprint, 2026.

Illustration

Further Reading & Resources

Explore how AI agents can be integrated into existing workflows:

• Telegram integration on UBOS
• ChatGPT and Telegram integration
• OpenAI ChatGPT integration
• About UBOS

Stay updated with the latest AI‑for‑hardware research by visiting our UBOS homepage and subscribing to the blog.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.