Andrii Bidochko

• Updated: July 5, 2026
• 6 min read

Local LLM Agents as Vulnerable Runtimes: A Source-Code Audit of the Agent Runtime Layer

Direct Answer

The paper introduces CLAWAUDIT, a static‑analysis framework that systematically audits the runtime layer of local LLM agents for implementation‑level security flaws. By turning the opaque mediation code—prompt builders, parsers, tool dispatchers, and permission gates—into a measurable surface, the authors expose a previously hidden attack surface that directly impacts the safety of AI‑driven personal assistants.

Background: Why This Problem Is Hard

Local LLM agents such as OpenClaw or Nanobot run on end‑user devices and act on privileged resources: the shell, file system, browsers, credential stores, and messaging apps. Their power stems from a three‑way mediation:

• User intent expressed in natural language.
• Model output that translates intent into actionable commands.
• Runtime layer that parses, validates, and dispatches those commands to host APIs.

While the research community has explored prompt injection, malicious skill libraries, and marketplace risks, the runtime layer itself has remained a blind spot. This omission is critical for three reasons:

1. Privilege escalation: The runtime often runs with the same permissions as the user, giving it direct access to OS‑level operations.
2. Dynamic code loading: Skills and plugins can be fetched at runtime, creating a moving target for static security checks.
3. Complex data flow: Natural‑language prompts are transformed through multiple parsers and builders, each introducing potential injection points that traditional fuzzing or black‑box testing miss.

Existing defenses—sandboxing, permission prompts, or model‑level safety filters—address the symptoms but not the root cause: insecure implementation patterns embedded in the agent’s source code. Auditing these patterns requires a methodology that can reason about code structure, data flow, and security semantics at scale.

What the Researchers Propose

To fill this gap, the authors present CLAWAUDIT, a static‑analysis framework built around two pillars:

• Vulnerability taxonomy derived from the STRIDE model (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege). The taxonomy is tailored to the agent runtime, identifying five concrete categories that map directly to code patterns.
• Rule engines—a collection of 47 Semgrep YAML rules and 30 CodeQL queries—each targeting a specific taxonomy item. The rules capture agent‑specific anti‑patterns such as unchecked command concatenation, insecure deserialization of skill payloads, and missing permission checks before invoking host APIs.

In essence, CLAWAUDIT turns the runtime layer into a set of verifiable contracts: every prompt builder, parser, or dispatcher must satisfy a rule that guarantees a particular security property.

How It Works in Practice

The CLAWAUDIT workflow can be visualized as a three‑stage pipeline:

1. Code ingestion: The framework clones the agent’s repository and extracts the runtime module (typically a Python or JavaScript package).
2. Static analysis: Semgrep and CodeQL run in parallel, applying the custom rule sets. Each rule emits a finding that includes the file path, line number, and a concise description of the violated security property.
3. Result aggregation: Findings are merged, de‑duplicated, and classified according to the STRIDE‑based taxonomy. The output is a structured report that highlights high‑risk categories (e.g., Elevation of Privilege) and provides remediation hints.

What distinguishes CLAWAUDIT from generic static analysis tools is its focus on “agent‑specific” patterns. For example, a rule might flag any instance where a user‑provided string is interpolated directly into a shell command without sanitization—a classic injection vector that is especially dangerous in an LLM‑driven context because the string originates from model output rather than a human.

The framework also supports “semantic filtering”: after the initial recall‑oriented scan, developers can apply higher‑level heuristics (e.g., only flag findings that affect privileged APIs) to reduce false positives before integrating the tool into CI/CD pipelines.

Evaluation & Results

To validate CLAWAUDIT, the authors built OPENCLAWBENCH, a benchmark comprising 446 source‑code advisories harvested from the OpenClaw repository. The dataset is split temporally: 229 advisories serve as the rule‑derivation (training) set, while 217 unseen advisories form the held‑out test set.

Two baseline configurations were used for comparison:

• Pro baseline – the default Semgrep rule set shipped with the tool.
• Security‑extended baseline – the official CodeQL security queries.

Key findings:

• Semgrep recall jumped from 21.7 % (baseline) to 66.8 % with CLAWAUDIT rules.
• CodeQL recall rose from 13.8 % to 75.1 % when augmented by the custom queries.
• Train‑test gaps stayed under four percentage points across all configurations, indicating strong generalization to previously unseen vulnerabilities.
• A live‑code audit on a fresh OpenClaw fork confirmed that the high‑recall rules surface real‑world issues, though manual triage remains necessary to filter out low‑severity noise.

These results demonstrate that a targeted, taxonomy‑driven rule set can dramatically improve the detection of runtime‑specific flaws without sacrificing the ability to discover novel issues.

Why This Matters for AI Systems and Agents

For AI practitioners, the implications are immediate:

• Risk reduction: By catching insecure patterns before deployment, teams can prevent agents from becoming inadvertent backdoors that compromise user data or system integrity.
• Compliance alignment: Many enterprise security standards (e.g., ISO 27001, SOC 2) require evidence of secure software development practices. CLAWAUDIT’s structured reports provide audit‑ready documentation.
• Developer productivity: Integrating the framework into CI pipelines automates a large portion of the security review, freeing engineers to focus on higher‑level design decisions.
• Ecosystem hygiene: Open‑source agent marketplaces can adopt CLAWAUDIT as a gatekeeper, ensuring that contributed plugins meet a baseline security posture.

Practically, teams building agents on the UBOS platform overview can embed CLAWAUDIT checks into their Workflow automation studio, guaranteeing that every new skill passes the same rigorous audit before being exposed to end users. Likewise, startups leveraging the UBOS for startups suite can benefit from pre‑validated security templates, reducing time‑to‑market while maintaining a strong security posture.

What Comes Next

Despite its promise, CLAWAUDIT has limitations that open avenues for future work:

• False‑positive management: The current rule set favors recall, which can overwhelm developers with low‑severity alerts. Incorporating machine‑learning‑based ranking or context‑aware pruning could streamline triage.
• Dynamic behavior coverage: Static analysis cannot capture runtime‑only bugs such as race conditions or mis‑configured environment variables. Hybrid approaches that combine static rules with lightweight instrumentation would close this gap.
• Cross‑language support: The benchmark focuses on Python implementations. Extending the taxonomy and rule libraries to Rust, Go, or compiled languages would broaden applicability.
• Community‑driven rule expansion: Open sourcing the rule definitions and encouraging contributions could accelerate the discovery of new agent‑specific anti‑patterns.

Potential applications beyond security audits include:

• Automated compliance checks for Enterprise AI platform by UBOS deployments.
• Integration with the OpenAI ChatGPT integration to enforce safe prompt handling at the gateway level.
• Embedding into the ChatGPT and Telegram integration to prevent malicious command injection via messaging bots.

In summary, turning the agent runtime layer into a first‑class security artifact reshapes how developers think about LLM‑driven software. As local agents proliferate across enterprises, the need for systematic, source‑code‑level assurance will only intensify.

References

arXiv paper

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.