Andrii Bidochko

• Updated: July 22, 2026
• 7 min read

REFORGE: A Method for Benchmarking LLMs’ Reverse Engineering Capabilities in Decompiled Binary Function Naming

Direct Answer

REFORGE introduces a provenance‑tracked benchmarking pipeline that measures how well large language models (LLMs) can reverse‑engineer binary functions by naming them after decompilation. It matters because it exposes hidden uncertainties in compiler‑induced code transformations, enabling fair, reproducible evaluation of LLM‑driven security tools.

Background: Why This Problem Is Hard

Binary reverse engineering sits at the intersection of software security, compiler theory, and machine intelligence. Practitioners need to map low‑level machine code back to high‑level semantics—a process that traditionally relies on human expertise, static analysis, and heuristic decompilers. Recent advances have shown LLMs can suggest function names, infer intent, and even generate patches, prompting a wave of “LLM‑assisted reverse engineering” tools.

However, evaluating those claims is non‑trivial for three core reasons:

• Ground‑truth scarcity: To judge an LLM’s output, you must know the original source‑level name of each compiled function. Compilers routinely inline, split, or reorder code, erasing a clean one‑to‑one mapping.
• Optimization noise: Higher optimization levels (e.g., -O2, -O3) aggressively transform control flow and data structures, making it impossible to guarantee that a decompiled function corresponds to a single source function.
• Survivorship bias: Existing benchmarks often discard functions that cannot be aligned, reporting accuracy only on the “survivors.” This inflates performance numbers and hides the true impact of compiler optimizations.

Because security teams increasingly embed LLMs into live offensive‑security pipelines, an unreliable benchmark can lead to over‑confidence, missed vulnerabilities, and wasted resources.

What the Researchers Propose

The authors present REFORGE, a systematic framework that builds a traceable provenance chain from C source code to decompiled binary functions. The pipeline consists of four logical stages:

1. Compilation with DWARF emission: Source files are compiled with debug information (DWARF) that records exact source‑line and function boundaries.
2. Syntactic extraction: The DWARF data is parsed to extract a canonical list of source‑level functions, preserving their original names.
3. Alignment engine: A multi‑pass matcher aligns each extracted function with its counterpart in the decompiled output, accounting for inlining, splitting, and other optimizations.
4. Confidence funnel: An eight‑gate filter quantifies alignment certainty, stratifying results into three tiers (high, medium, low confidence).

By exposing the alignment uncertainty as a first‑class metric, REFORGE forces evaluators to acknowledge which functions are truly comparable and which are not.

How It Works in Practice

The REFORGE workflow can be visualized as a linear pipeline with feedback loops for error handling:

• Source ingestion: Developers feed a collection of C files into the pipeline.
• Compiler driver: The driver invokes gcc/clang with configurable optimization flags (e.g., -O0 to -O3) and requests DWARF sections (-g).
• DWARF parser: A dedicated parser extracts a mapping table: {source_function_name → address_range}.
• Decompilation stage: An open‑source decompiler (e.g., Ghidra, RetDec) produces pseudo‑C for each address range.
• Alignment engine: The engine runs eight sequential checks—range overlap, control‑flow similarity, symbol table hints, etc.—to decide whether a decompiled snippet reliably corresponds to a source function.
• Confidence stratification: Functions passing all eight checks are labeled “high confidence,” those passing five to seven are “medium,” and the rest fall into “low.”
• Benchmark export: The final artifact is a CSV file containing {function_name, confidence_tier, decompiled_code}, ready for downstream LLM evaluation.

What sets REFORGE apart is its explicit provenance tracking: every transformation step logs its parameters, enabling reproducibility and auditability. Moreover, the confidence funnel is not a binary filter; it provides a graded view that can be tuned per research need.

Evaluation & Results

The authors constructed a controlled micro‑benchmark using 1,200 small C functions drawn from the GNU coreutils suite. They compiled each function at four optimization levels (-O0, -O1, -O2, -O3) and ran the REFORGE pipeline to measure alignment yield.

These numbers illustrate a steep decline in reliably alignable functions as optimizations increase, confirming the authors’ hypothesis that compiler transformations are the primary bottleneck for fair benchmarking.

To demonstrate the impact of survivorship bias, the team performed an “unpaired” comparison: they evaluated LLM naming accuracy only on the high‑confidence subset for each optimization level. The resulting accuracy curve suggested a modest performance drop (≈ 5 %). In contrast, a “paired” analysis—tracking the same source functions across all optimization levels—revealed a much larger decay (≈ 18 %). This discrepancy underscores how discarding low‑confidence functions can mask true capability loss.

Finally, the authors tested seven contemporary LLMs (including GPT‑4, Claude‑2, Llama‑2‑70B, and open‑source alternatives) on the high‑confidence set. Across the board, naming accuracy ranged from 42 % (baseline) to 68 % (top‑performing model) at -O0, but fell below 30 % for most models at -O3. The results suggest that while LLMs can capture simple naming patterns, they struggle with heavily optimized code where semantic cues are obscured.

Why This Matters for AI Systems and Agents

For AI practitioners building autonomous security agents, REFORGE offers a calibrated yardstick to measure “reverse‑engineering intelligence.” Without a trustworthy benchmark, agents may be over‑promised in product roadmaps, leading to costly mis‑deployments.

Key practical takeaways include:

• Confidence‑aware scoring: Agents can be programmed to weigh LLM suggestions by the confidence tier, rejecting low‑confidence names that are likely to be misleading.
• Optimization‑aware pipelines: Security workflows that ingest binaries from diverse build environments should first assess the optimization level and adjust LLM prompting strategies accordingly.
• Data‑driven model selection: The benchmark highlights which LLM families retain more semantic signal under optimization, guiding model licensing or fine‑tuning decisions.
• Integration pathways: REFORGE’s provenance logs can be fed into a UBOS platform overview for automated orchestration, allowing agents to trigger decompilation, alignment, and LLM inference as a single workflow.

By embedding REFORGE into an Workflow automation studio, security teams can create repeatable pipelines that surface only high‑confidence function names, reducing false positives in threat‑intel reports.

What Comes Next

While REFORGE marks a significant step forward, several limitations remain:

• Scope of languages: The current implementation focuses on C; extending to C++, Rust, or Go will require handling richer type systems and name mangling schemes.
• Decompiler dependency: Alignment quality is tied to the decompiler’s fidelity. Future work could incorporate multiple decompilers and ensemble voting to improve robustness.
• Dynamic analysis integration: Runtime traces (e.g., via QEMU or DynamoRIO) could supplement static alignment, especially for heavily inlined code.
• Fine‑tuning LLMs on aligned data: The high‑confidence dataset produced by REFORGE could serve as a specialized training corpus, potentially boosting naming accuracy on optimized binaries.

Looking ahead, researchers might explore “reverse‑engineering agents” that close the loop: an agent proposes a function name, the confidence funnel validates it, and the feedback is used to iteratively refine the model. Such agents could be deployed on the Enterprise AI platform by UBOS, where they would benefit from scalable compute, secure data handling, and built‑in monitoring.

Community involvement is also crucial. By publishing the REFORGE pipeline as open source, the authors invite contributions that add new compilers, support additional architectures (ARM, RISC‑V), and integrate with existing threat‑intel platforms.

Conclusion

REFORGE delivers a provenance‑tracked, confidence‑aware benchmark that reveals how compiler optimizations erode the reliability of LLM‑generated binary function names. Its eight‑gate funnel quantifies alignment uncertainty, exposing survivorship bias in prior evaluations and providing a reproducible foundation for future research.

For security engineers, AI developers, and product leaders, the paper signals that raw LLM accuracy numbers are insufficient; a nuanced, confidence‑driven approach is essential for trustworthy automation. The open‑source pipeline, combined with UBOS’s orchestration tools, offers a practical path to integrate rigorous reverse‑engineering evaluation into real‑world AI agents.

Read the full study on the original arXiv paper and consider contributing to the REFORGE ecosystem to shape the next generation of AI‑enhanced security tooling.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.