Andrii Bidochko

• Updated: July 7, 2026
• 6 min read

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

Direct Answer

ASK+ introduces an uncertainty‑gated framework that lets small language models (SLMs) act as on‑demand consultants for reinforcement‑learning agents operating under partial observability. By feeding the SLM a trajectory‑aware prompt and a structured chain‑of‑thought, the system can intervene when the agent’s action confidence is low, boosting success rates without scaling to massive models.

Background: Why This Problem Is Hard

Real‑world autonomous systems—robots navigating warehouses, game‑playing bots, or virtual assistants in partially observable environments—must make decisions with incomplete state information. This “partial observability” creates two intertwined challenges:

• State uncertainty: The agent cannot directly perceive the full environment, so it must infer hidden variables from limited observations.
• Action uncertainty: Even with a belief state, the optimal action may be ambiguous, leading to brittle policies.

Large language models (LLMs) excel at reasoning and can provide high‑level guidance, but integrating them into a closed‑loop RL loop has proven difficult. Prior attempts used “uncertainty‑gated” queries—asking the LLM only when the agent’s predictive entropy spikes—but observed near‑zero overwrite rates. In practice, the LLM rarely contributed an independent action, reducing the approach to a redundant sanity check.

Why do existing methods fail?

• Egocentric prompts: Traditional prompts only expose the current observation, leaving the LLM without the broader context needed for genuine reasoning.
• Static reasoning templates: Without a step‑by‑step chain‑of‑thought, the LLM cannot articulate why a suggested action differs from the policy, making its advice hard to trust.
• Scale bias: Researchers assumed that larger models would automatically overcome prompt deficiencies, overlooking the dominant role of prompt engineering.

What the Researchers Propose

The paper presents ASK+, a redesign of uncertainty‑gated assistance that treats the SLM as a contextual consultant rather than a passive verifier. The core ideas are:

• Trajectory‑aware context: The prompt includes a partially revealed map, the agent’s visited positions, and a concise action history, giving the LLM a “memory” of the episode.
• Structured chain‑of‑thought (CoT): The SLM is asked to reason step‑by‑step—first summarizing the known map, then hypothesizing hidden elements, and finally proposing an action with justification.
• Uncertainty‑gated querying: Predictive entropy of the policy’s action distribution still drives the decision to query, but now the signal reflects true action uncertainty, which remains informative in partially observable Markov decision processes (POMDPs).

ASK+ therefore converts the SLM from a “redundancy filter” into a “knowledgeable advisor” that can occasionally override the base policy with a higher‑quality decision.

How It Works in Practice

The ASK+ pipeline consists of four interacting components:

1. Base RL policy: A reinforcement‑learning agent (e.g., PPO) that outputs an action distribution given the current observation and belief state.
2. Uncertainty monitor: Computes predictive entropy over the action distribution. If entropy exceeds a calibrated threshold, the system flags the step for assistance.
3. Context builder: Gathers the partial map, visited cells, and recent actions, then formats them into a concise, human‑readable prompt.
4. SLM consultant (ASK+ prompt): Receives the context and a CoT instruction, generates a reasoned action proposal, and returns both the action and its justification.

When the monitor triggers, the workflow proceeds as follows:

• The agent’s observation and belief are passed to the context builder.
• The built prompt is sent to the SLM (e.g., Qwen3.5‑2B) via an API call.
• The SLM returns a structured response: {"action": "move_north", "reason": "the door is likely behind the unexplored corridor"}.
• The system compares the SLM’s confidence (derived from token probabilities) with the policy’s entropy. If the SLM’s confidence exceeds a secondary threshold, the SLM’s action overwrites the policy’s choice for that step.

This selective overwrite mechanism preserves the efficiency of the base policy while allowing the SLM to intervene only when its contextual reasoning is both needed and reliable.

Evaluation & Results

The authors benchmarked ASK+ on three classic POMDP environments, each emphasizing a different aspect of partial observability:

Key takeaways from the experiments:

• Context matters more than scale: The 2‑billion‑parameter Qwen3.5 model matched or outperformed its 4‑billion‑parameter sibling when equipped with the ASK+ prompt.
• Selective gating remains viable: Predictive entropy successfully identified high‑uncertainty steps even when the underlying state was hidden, confirming that action‑uncertainty signals are robust in POMDPs.
• Chain‑of‑thought boosts trust: The SLM’s explicit reasoning allowed the system to accept its advice more often, leading to measurable performance lifts across all domains.

These results demonstrate that a well‑engineered prompt can unlock the advisory power of modest‑sized language models, reducing the need for costly scaling.

Why This Matters for AI Systems and Agents

For practitioners building autonomous agents, ASK+ offers a pragmatic pathway to augment existing policies without overhauling the entire learning pipeline. The approach delivers several practical benefits:

• Cost‑effective expertise: Small, open‑source LLMs can provide domain‑specific guidance, lowering inference costs compared to deploying massive foundation models.
• Modular integration: The uncertainty monitor and context builder act as plug‑in components that can be attached to any RL policy, whether it’s PPO, DQN, or a custom actor‑critic.
• Improved safety and interpretability: The CoT output supplies a human‑readable justification, which can be logged for audit trails or used to trigger fallback mechanisms.
• Scalable orchestration: In multi‑agent ecosystems, ASK+ can serve as a shared “consultant service,” reducing duplicated reasoning across agents.

Enterprises looking to embed intelligent assistants into workflows can leverage ASK+ through platforms like the Enterprise AI platform by UBOS, which already supports custom policy modules and external LLM APIs.

What Comes Next

While ASK+ marks a significant step forward, several open challenges remain:

• Dynamic thresholding: Current entropy thresholds are static; adaptive schemes that consider episode length or task difficulty could further reduce unnecessary queries.
• Multi‑modal context: Incorporating visual or auditory cues into the prompt may enable richer reasoning for robotics or embodied AI.
• Long‑term credit assignment: Understanding how occasional SLM interventions affect the learning signal over many episodes is an open research question.
• Robustness to adversarial prompts: Ensuring that the SLM does not introduce harmful actions when faced with ambiguous or malicious contexts.

Future work could also explore a “teacher‑student” regime where the SLM’s reasoning is distilled into the base policy, gradually reducing reliance on external calls. For organizations eager to experiment, the UBOS partner program offers early‑access resources and co‑development support.

References

ASK in the Dark: Uncertainty‑Gated LLM Assistance under Partial Observability (arXiv)

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.