Andrii Bidochko

• Updated: July 10, 2026
• 9 min read

Agent Step Value: State-Transition Measurement with State-Grounded LLM Evaluators

Direct Answer

Agent Step Value (ASV) is a new diagnostic framework that measures the impact of each individual action in an autonomous agent’s trace by comparing the before‑and‑after state distributions produced by a stateless LLM evaluator. By turning every step into a quantifiable belief shift, ASV lets developers pinpoint exactly which moves advance or derail an agent’s goal, a capability that traditional end‑to‑end scores simply cannot provide.

Background: Why This Problem Is Hard

Modern AI agents—whether they retrieve scientific literature, negotiate contracts, or orchestrate multi‑tool workflows—operate through long, multi‑step reasoning chains. Evaluating such agents has traditionally collapsed the entire trace into a single metric: a final answer correctness flag, a cumulative reward, or a trajectory‑level likelihood score. This aggregation creates two critical blind spots.

• Diagnostic opacity. When an agent fails, engineers cannot tell whether the failure originated from a mis‑retrieved document, a faulty reasoning step, or an ill‑chosen action at the very end of the chain.
• Feedback sparsity. Reinforcement‑style rewards are often delayed until the final answer, making it difficult to assign credit or blame to intermediate actions during training or debugging.

Existing approaches attempt to address these issues with heuristics such as step‑wise entropy tracking or token‑level log‑probability monitoring. However, entropy alone only captures uncertainty, not directional belief change, while token‑level scores are tied to a specific language model’s internal distribution and can be noisy when the model is used as a tool rather than a judge. Moreover, most evaluators require a fully‑grounded environment or a gold‑standard answer, which is impractical for open‑domain tasks where the “right” answer may be unknown or evolving.

What the Researchers Propose

The authors introduce Agent Step Value (ASV), a state‑transition measurement framework that treats each observed action as a pivot point in a belief space defined by a state‑grounded LLM evaluator. The evaluator is a stateless language model that receives a redacted snapshot of the agent’s world state before and after an action, along with a fixed set of candidate outcomes (e.g., possible answers, retrieved documents, or decision options). It then assigns a log‑score to each candidate, effectively producing a probability distribution over the outcome space for both the pre‑action and post‑action states.

ASV computes the difference between these two distributions, yielding a scalar “step value” that quantifies how much the action moved the agent’s belief toward or away from each candidate. The framework also includes a label‑free rationale pass that separates the evaluator’s deliberation (a natural‑language explanation) from the raw token‑level scoring, preserving the integrity of the likelihood numbers while exposing potential leakage or floor‑score events.

Key components of the ASV pipeline are:

• State projector. Generates a concise, redacted representation of the agent’s internal or external state (e.g., the current query, retrieved snippets, or tool parameters).
• Stateless LLM evaluator. A large language model (such as DeepSeek) that processes the before/after state projections and the candidate set, outputting log‑scores for each candidate.
• Step‑value calculator. Subtracts the pre‑action log‑score vector from the post‑action vector, optionally normalizing by entropy or Bayesian surprise to produce interpretable metrics.
• Rationale extractor. Generates a natural‑language justification for the evaluator’s scoring, which can be inspected for leakage (when the evaluator inadvertently sees the gold answer) or floor‑score events (when all candidates receive near‑zero probability).

How It Works in Practice

The ASV workflow can be visualized as a three‑stage loop that runs alongside any existing agent execution pipeline. Figure 1 (placeholder image) illustrates the data flow.

1. Capture the pre‑action state. When an agent is about to take an action—such as issuing a PubMed search, invoking a calculator tool, or selecting a knowledge‑base entry—the system snapshots the relevant context. Sensitive details (e.g., private user identifiers) are redacted to keep the evaluation privacy‑preserving.

2. Project the state through the evaluator. The redacted snapshot, together with a static list of candidate outcomes (for an open‑question task these might be the top‑k retrieved passages), is fed to the stateless LLM. The model returns a log‑probability for each candidate, forming the before‑distribution.

3. Execute the action and capture the post‑action state. The agent performs the action, updates its internal memory, and possibly obtains new evidence (e.g., a retrieved article). A second snapshot is taken, again redacted, and sent through the same evaluator with the identical candidate set, producing the after‑distribution.

4. Compute the step value. By subtracting the before‑distribution from the after‑distribution, ASV yields a vector of belief changes. Summarizing this vector (e.g., taking the maximum increase, the average gain, or the Bayesian surprise) produces a single scalar that reflects how constructive the action was.

5. Record the rationale. The evaluator also emits a short natural‑language explanation for its scoring. This rationale is stored alongside the step value, enabling developers to audit whether the evaluator relied on unintended cues.

What sets ASV apart from prior step‑wise metrics is its state‑grounded nature: the evaluator’s belief space is anchored to the concrete state before and after the action, rather than to a global model hidden state. This makes the measurement robust to changes in the underlying LLM architecture and eliminates the need for a gold answer during the diagnostic phase.

Evaluation & Results

The authors validated ASV on a suite of 100 open‑question evidence‑seeking tasks drawn from the PubMed domain. Each task involved a live retrieval component, a partially live DeepSeek actor that generated actions, and DeepSeek’s own log‑probability scoring for the evaluator. In total, the experiments covered 1,100 individual steps and 2,200 state projections.

Two families of metrics were reported:

• Gold‑free belief diagnostics. These include the mean gold‑margin gain (the average change in belief toward the correct answer) and the mean Bayesian surprise (a measure of how much the belief distribution shifted). The reported gold‑margin gain was –2.335, indicating that many steps actually moved belief away from the correct answer under the fixed‑layout rationale protocol, while the Bayesian surprise averaged 2.693, confirming that the evaluator detected substantial belief movement even when the direction was negative.
• Offline oracle validation. By comparing ASV’s step values against a post‑hoc oracle that knows the ground‑truth answer, the authors demonstrated that ASV could reliably flag “constructive” versus “destructive” steps. Traditional entropy‑only metrics failed to surface these pivots, often reporting near‑zero change for steps that dramatically altered the agent’s trajectory.

Importantly, the evaluation showed that ASV can surface diagnostic signals without any gold labels, making it suitable for continuous monitoring of production agents where labeled data is scarce or expensive. The authors also released a standalone ASV Eval toolkit, enabling other researchers to reproduce the experiments or apply the framework to new domains.

Why This Matters for AI Systems and Agents

For practitioners building complex, multi‑tool agents, ASV offers a concrete answer to the perennial “where did it go wrong?” question. By turning each action into a measurable belief shift, developers can:

• Accelerate debugging. Instead of replaying entire traces, engineers can focus on steps with the highest negative step value, dramatically reducing time‑to‑resolution.
• Inform reinforcement learning. Step‑wise values can serve as dense rewards for policy‑gradient methods, enabling more sample‑efficient training of agents that learn from their own intermediate outcomes.
• Enable automated orchestration. An orchestration layer can dynamically re‑route or abort actions that exhibit a strong negative belief shift, improving overall system reliability.
• Support compliance and auditability. The rationale pass provides a human‑readable audit trail, satisfying regulatory requirements for explainability in high‑stakes domains such as healthcare or finance.

These capabilities align directly with the needs of enterprises adopting AI‑driven automation. For example, the UBOS platform overview highlights the importance of fine‑grained monitoring for workflow automation, and ASV could be integrated as a native diagnostic module within such platforms.

Moreover, the ability to evaluate agents without gold labels dovetails with the trend toward Enterprise AI platform by UBOS, where large fleets of agents operate on proprietary data streams that lack public benchmarks. ASV’s state‑grounded approach ensures that evaluation remains consistent even as the underlying LLMs evolve.

What Comes Next

While ASV marks a significant step forward, several open challenges remain:

• Scalability of candidate sets. In domains with thousands of possible outcomes (e.g., product recommendation), constructing a manageable fixed candidate list without sacrificing diagnostic fidelity is non‑trivial.
• Evaluator bias. The stateless LLM itself may harbor systematic biases that influence step values. Future work should explore ensemble evaluators or calibration techniques to mitigate this risk.
• Integration with tool‑use loops. Agents that invoke external APIs (e.g., payment processors) generate side effects that are not easily captured by a textual state projection. Extending ASV to handle multimodal or non‑textual state representations is an active research direction.
• Real‑time deployment. Computing before/after distributions for every step adds latency. Optimizations such as caching evaluator embeddings or using lightweight distilled models could make ASV viable for high‑throughput production pipelines.

Addressing these challenges will broaden ASV’s applicability beyond academic benchmarks to real‑world enterprise settings. Potential extensions include:

• Coupling ASV with Workflow automation studio to automatically trigger remedial sub‑workflows when a negative step value is detected.
• Embedding ASV diagnostics into AI marketing agents so that campaign‑generation bots can self‑correct when a content‑creation step reduces relevance to the target audience.
• Leveraging the ASV rationale output to feed into ChatGPT and Telegram integration, enabling real‑time human‑in‑the‑loop oversight for critical decision steps.

In summary, Agent Step Value opens a new diagnostic frontier for AI agents, turning opaque multi‑step traces into transparent, actionable belief maps. As agents become more autonomous and embedded in business processes, tools like ASV will be essential for maintaining reliability, interpretability, and continuous improvement.

For readers who wish to explore the technical details directly, the full pre‑print is available on arXiv.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.