Andrii Bidochko

• Updated: July 7, 2026
• 7 min read

ASK in the Dark: Uncertainty‑Gated LLM Assistance under Partial Observability

Direct Answer

ASK+ introduces a context‑rich, uncertainty‑gated interface that lets a small language model (SLM) act as a consultant for reinforcement‑learning agents operating under partial observability. By feeding the LLM trajectory‑aware information and a structured chain‑of‑thought prompt, the system converts the model from a passive redundancy check into an occasional, high‑impact corrector, boosting success rates across classic POMDP benchmarks.

Background: Why This Problem Is Hard

Real‑world autonomous systems—robots navigating warehouses, drones exploring unknown terrain, or software agents handling incomplete user data—rarely enjoy full state visibility. In reinforcement‑learning (RL) terminology, these scenarios are modeled as partially observable Markov decision processes (POMDPs). The agent must infer hidden aspects of the environment from a limited observation stream, which inevitably introduces uncertainty and can lead to sub‑optimal or unsafe actions.

Recent research has explored augmenting RL agents with large language models (LLMs) that carry broad world knowledge and reasoning priors. The intuition is simple: a language model can “fill in the blanks” when the agent’s sensor data is sparse. However, early attempts at uncertainty‑gated assistance—where the agent queries the LLM only when its own policy is uncertain—have consistently failed in POMDP settings. Empirically, vanilla approaches report an overwrite rate near zero, meaning the LLM almost never contributes an independent decision.

The root cause is not a lack of model capacity but a context problem. Traditional prompts give the LLM only the current observation (an egocentric view) without any history or map of the environment. Without this trajectory context, the LLM cannot perform the multi‑step reasoning required to resolve hidden state ambiguities, and it defaults to echoing the agent’s own policy.

What the Researchers Propose

The authors present ASK+ (Ask in the Dark Plus), a redesign of the uncertainty‑gated pipeline that supplies the LLM with three key ingredients:

• Trajectory‑aware context: a partially revealed map, a list of visited positions, and the recent action history.
• Structured chain‑of‑thought (CoT) prompting: the LLM is asked to reason step‑by‑step before proposing an action, mirroring human problem‑solving.
• Selective querying based on predictive entropy: the agent still decides when to ask, but the entropy signal now reflects action uncertainty rather than raw state uncertainty, preserving its usefulness in POMDPs.

ASK+ therefore reframes the LLM from a passive “check‑my‑work” module into an active consultant that can occasionally override the RL policy with a better‑informed move.

How It Works in Practice

Conceptual Workflow

1. Observation & Belief Update: The RL agent receives its current partial observation and updates its internal belief state.
2. Uncertainty Estimation: The agent computes predictive entropy over its action distribution. High entropy signals that the agent is unsure which action maximizes expected return.
3. Gate Decision: If entropy exceeds a predefined threshold, the agent triggers a query to the LLM; otherwise it proceeds with its own policy.
4. Context Assembly: The system builds a prompt that includes:
   • The partially revealed map (e.g., walls, doors discovered so far).
   • A chronological list of visited cells and actions taken.
   • The current observation.
   • A CoT template asking the LLM to “think aloud” before suggesting an action.
5. L​LM Reasoning: The small language model (e.g., Qwen3.5‑2B) processes the prompt, generates a step‑by‑step rationale, and outputs a candidate action.
6. Decision Fusion: The agent compares the LLM’s suggestion with its own distribution. If the LLM’s confidence (derived from token probabilities) exceeds a secondary threshold, the LLM action overwrites the agent’s choice.
7. Execution & Feedback: The selected action is executed in the environment, and the resulting observation feeds back into the next iteration.

Key Differentiators

• Stateful Prompting: By embedding the entire trajectory, the LLM can reason about “where I have been” and “what I still don’t know,” a capability missing from egocentric prompts.
• Chain‑of‑Thought Guidance: The CoT format forces the model to articulate intermediate deductions, reducing hallucinations and improving alignment with the agent’s objectives.
• Model‑Scale Agnosticism: Experiments show that a 2‑billion‑parameter model matches or exceeds a 4‑billion‑parameter counterpart when the prompt is well‑designed, highlighting the primacy of prompt engineering over raw size.

Evaluation & Results

Testbeds

The authors benchmarked ASK+ on three canonical POMDP environments:

• DoorKey: An agent must locate a key, unlock a door, and reach a goal in a grid world with hidden walls.
• FourRooms: A classic navigation task where rooms are connected by narrow passages; the agent only sees a limited field of view.
• HigherLower: A number‑guessing game where the agent receives only “higher” or “lower” feedback, requiring inference over a hidden integer.

Findings

Across all tasks, ASK+ closed the performance gap between the RL baseline and the theoretical SLM‑only ceiling. Notably, the improvement was achieved without increasing the LLM’s parameter count, confirming that prompt design and selective gating dominate the impact of model scale.

The entropy‑based gating remained informative even when the agent’s belief state was highly ambiguous, demonstrating that predictive entropy can serve as a reliable proxy for “when I need help” in partially observable settings.

Why This Matters for AI Systems and Agents

For practitioners building autonomous agents, ASK+ offers a pragmatic pathway to boost reliability without incurring the cost of massive LLM deployments. By leveraging a modest‑sized model and a carefully crafted prompt, developers can:

• Reduce the frequency of catastrophic failures in safety‑critical domains (e.g., warehouse robotics) by allowing the LLM to intervene only when the policy is genuinely uncertain.
• Accelerate iteration cycles: the same LLM can be reused across multiple environments because the prompt encapsulates environment‑specific context rather than requiring fine‑tuning.
• Maintain compliance with latency and compute budgets, since the LLM is invoked selectively and the underlying model can be hosted on edge‑friendly hardware.

These advantages align closely with the capabilities of the UBOS platform overview, which provides modular orchestration for RL pipelines, LLM services, and real‑time monitoring. Integrating ASK+ into such a platform could enable enterprises to build “assist‑by‑LLM” agents that scale across departments—from logistics to customer‑service automation—while keeping operational costs predictable.

What Comes Next

While ASK+ demonstrates that prompt engineering can unlock the latent reasoning power of small LLMs, several open challenges remain:

• Dynamic Thresholding: The current entropy threshold is static. Future work could explore adaptive thresholds that consider task difficulty, recent success rates, or external risk metrics.
• Multi‑Agent Collaboration: Extending ASK+ to coordinate several agents, each with its own partial view, would test the scalability of trajectory‑aware prompting in a distributed setting.
• Robustness to Noisy Maps: Real sensors produce imperfect maps. Investigating how the LLM handles erroneous or contradictory context is essential for deployment in the wild.
• Learning the Prompt: Meta‑learning techniques could automatically discover the most effective CoT structures for a given domain, reducing the manual engineering effort.

Addressing these directions could turn ASK+ from a research prototype into a production‑ready component of enterprise AI stacks. Startups and innovators looking to experiment with uncertainty‑gated assistance can prototype quickly on the UBOS for startups environment, which offers sandboxed RL and LLM integration out of the box.

Conclusion

ASK+ proves that a well‑designed, trajectory‑aware prompt can transform a modest language model into a valuable consultant for RL agents navigating partially observable worlds. By marrying predictive‑entropy gating with chain‑of‑thought reasoning, the framework achieves measurable gains on classic benchmarks without relying on massive model sizes. For AI engineers, the takeaway is clear: invest in prompt architecture and selective querying, and you can reap the benefits of LLM assistance even under tight compute budgets.

References & Further Reading

• Monteiro, J., Gavenski, N., Lima, G., et al. “ASK in the Dark: Uncertainty‑Gated LLM Assistance under Partial Observability.” arXiv:2607.02686v1, July 2026.
• Silver, D., et al. “Mastering the Game of Go with Deep Neural Networks and Tree Search.” Nature, 2016.
• Wei, J., et al. “Chain of Thought Prompting Elicits Reasoning in Large Language Models.” arXiv, 2022.

Call to Action

Ready to explore uncertainty‑gated LLM assistance for your own RL projects? Dive into the full paper for technical details and start prototyping with UBOS’s modular AI stack.

Read the complete arXiv preprint and discover how ASK+ can elevate your agents from “guess‑work” to “guided‑reasoning.”

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.