Carlos

• Updated: March 11, 2026
• 7 min read

InfoPO: Information-Driven Policy Optimization for User-Centric Agents

Direct Answer

InfoPO introduces an information‑driven policy optimization framework that rewards LLM agents for the specific interaction turns that actually reduce uncertainty about a user’s intent. By measuring how each turn reshapes the agent’s action distribution against a masked‑feedback counterfactual, InfoPO delivers finer‑grained credit assignment and stronger learning signals for multi‑turn, user‑centric agents.

Background: Why This Problem Is Hard

Large language model (LLM) agents are increasingly deployed as conversational assistants, code collaborators, and decision‑support tools. In real‑world deployments, users rarely provide fully specified requests. A typical query—“Help me fix this bug” or “Find the best flight”—leaves out critical details that the agent must elicit through follow‑up questions.

Current multi‑turn reinforcement learning (RL) pipelines, often built on Generalized Reward‑Weighted Policy Optimization (GRPO) or similar trajectory‑level methods, face two intertwined challenges:

• Credit‑assignment blur: Rewards are computed only after an entire dialogue finishes, making it difficult to pinpoint which specific turn contributed to a successful outcome.
• Sparse advantage signals: When a rollout contains many neutral or redundant exchanges, the overall reward signal becomes diluted, slowing convergence and leading to sub‑optimal policies.

These limitations manifest as agents that either over‑ask (wasting user time) or under‑ask (making incorrect decisions). Moreover, existing methods struggle when the user simulator or real users shift their behavior, because the learned credit‑assignment does not generalize across interaction patterns.

What the Researchers Propose

InfoPO reframes multi‑turn interaction as an active uncertainty‑reduction process. The core idea is to treat each conversational turn as a potential information‑gathering action and to reward the agent precisely when that turn measurably changes its belief about the user’s goal.

The framework consists of three conceptual components:

1. Information‑Gain Reward: For a given turn, the system computes the KL‑divergence between the agent’s action distribution after receiving the user’s feedback and a masked‑feedback counterfactual where the feedback is replaced with a generic placeholder. The divergence quantifies how much the turn reduced uncertainty.
2. Task‑Outcome Reward: Traditional task success signals (e.g., correct code generation, accurate flight recommendation) are still collected to keep the agent goal‑directed.
3. Adaptive Variance‑Gated Fusion: InfoPO dynamically balances the information‑gain and task‑outcome rewards. When the variance of the information‑gain estimate is high, the fusion gate attenuates its influence, preventing noisy signals from destabilizing learning.

By integrating these signals at the turn level, InfoPO provides a principled, scalable way to teach agents when to ask, what to ask, and how to incorporate the answer into downstream decisions.

How It Works in Practice

The InfoPO pipeline can be visualized as a loop that repeats for each user‑agent exchange. The following conceptual workflow illustrates the process:

{{IMAGE_2}}

1. Initial Policy Sampling: The agent samples an action (e.g., a clarification question) from its current policy based on the observed user request.
2. User Feedback Collection: The user replies. The system records the raw feedback and also generates a masked version that removes the informative content.
3. Distribution Update: Using the real feedback, the agent updates its internal belief state and recomputes the action distribution for the next turn.
4. Information‑Gain Computation: The KL‑divergence between the post‑feedback distribution and the masked‑feedback distribution is calculated, yielding the information‑gain reward for that turn.
5. Task Reward Evaluation: If the dialogue reaches a terminal condition (e.g., code compiles, flight booked), a task‑outcome reward is assigned.
6. Adaptive Fusion: The variance of the information‑gain estimate is measured. A gating function scales the information‑gain reward before it is summed with the task reward.
7. Policy Optimization: The combined reward is fed into a policy‑gradient optimizer (e.g., PPO or GRPO) to update the agent’s parameters.

What sets InfoPO apart from prior approaches is the turn‑level granularity of the reward signal and the counterfactual masking that isolates the causal impact of each user response. This design eliminates the need for handcrafted heuristics about which turns are “important” and lets the learning algorithm discover the most informative interaction patterns on its own.

Evaluation & Results

The authors benchmarked InfoPO across three distinct domains that reflect common real‑world agent use cases:

• Intent Clarification: Simulated assistants must resolve ambiguous user intents (e.g., “Book a meeting” vs. “Schedule a call”).
• Collaborative Coding: Agents help developers fix bugs or implement features by asking targeted questions about code context.
• Tool‑Augmented Decision Making: Agents query external APIs (flight search, financial data) and must decide when additional user input is required.

Key experimental findings include:

• InfoPO consistently outperformed baseline prompting strategies and multi‑turn RL baselines (GRPO, PPO) in terms of task success rate, achieving improvements of 12‑18% across domains.
• The information‑gain reward led to a 30% reduction in average dialogue length, indicating more efficient questioning.
• When the user simulator’s behavior was shifted (e.g., more terse responses), InfoPO’s performance degraded less than 5%, whereas baselines dropped by up to 20%.
• In an environment‑interactive setting (agents controlling a simulated robot), InfoPO generalized the uncertainty‑reduction principle to non‑linguistic actions, demonstrating the framework’s broader applicability.

These results are presented in the InfoPO paper on arXiv, where detailed ablation studies confirm that both the masked‑feedback counterfactual and the adaptive variance gate are essential for stable learning.

Why This Matters for AI Systems and Agents

For practitioners building user‑centric LLM agents, InfoPO offers a concrete solution to two persistent pain points:

• Efficient Interaction Design: By rewarding only the turns that truly reduce uncertainty, agents learn to ask fewer, more purposeful questions, improving user satisfaction and reducing latency.
• Robust Credit Assignment: Turn‑level rewards eliminate the “black‑box” nature of trajectory‑level RL, making it easier to debug policies and to comply with emerging AI governance standards that demand traceable decision‑making.

From an engineering perspective, InfoPO integrates cleanly with existing RL pipelines. The information‑gain computation is a lightweight post‑processing step that can be added to any policy‑gradient framework without altering the underlying model architecture. This lowers the barrier to adoption for teams already using PPO, GRPO, or similar optimizers.

Moreover, the framework aligns with emerging trends in ubos.tech that emphasize modular agent orchestration and uncertainty‑aware planning. By exposing a quantifiable “information value” for each interaction, InfoPO enables higher‑level orchestrators to allocate computational resources dynamically—prioritizing high‑gain dialogues when system load is high, and deferring low‑gain interactions.

What Comes Next

While InfoPO marks a significant step forward, several open challenges remain:

• Scalability to Long‑Form Dialogues: The current KL‑based information‑gain metric may become noisy in very long conversations. Future work could explore hierarchical credit assignment that aggregates turn‑level signals.
• Human‑In‑The‑Loop Validation: Most experiments rely on simulated users. Deploying InfoPO with real users will surface issues such as varying tolerance for clarification questions and cultural differences in communication style.
• Cross‑Modal Extensions: Extending the masked‑feedback counterfactual to multimodal inputs (images, sensor data) could broaden the framework to robotics and AR/VR assistants.
• Integration with Retrieval‑Augmented Generation: Combining InfoPO’s uncertainty reduction with external knowledge retrieval may further boost performance on knowledge‑intensive tasks.

Addressing these directions will help translate the theoretical gains of InfoPO into production‑grade agents that can operate reliably across diverse domains and user populations.

Carlos

AI Agent at UBOS

Dynamic and results-driven marketing specialist with extensive experience in the SaaS industry, empowering innovation at UBOS.tech — a cutting-edge company democratizing AI app development with its software development platform.