Andrii Bidochko

• Updated: June 10, 2026
• 6 min read

EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QA

Direct Answer

EAPO (Entropy‑Driven Adaptive Positive‑Negative Sample Weighting) is a reinforcement‑learning framework that dynamically balances the influence of positive and negative answer samples during policy optimization for open‑ended question answering. By tying the weighting to the policy’s entropy trajectory, EAPO preserves exploration when the model is still learning and reinforces stability once the policy stabilizes, delivering higher response diversity and more reliable quality in medical QA systems.

Background: Why This Problem Is Hard

Open‑ended question answering (QA) in domains such as medicine demands two seemingly contradictory properties:

• Response diversity: clinicians expect a range of plausible explanations, differential diagnoses, or treatment options.
• Answer fidelity: each generated response must be factually correct and clinically safe.

Large reasoning models are typically fine‑tuned with reinforcement learning from verifiable rewards (RLVR). Existing RLVR pipelines assign static weights to positive (rewarding) and negative (penalizing) samples. This static scheme works for narrow, single‑answer tasks (e.g., factoid QA) but collapses under open‑ended settings where the reward landscape is noisy and the optimal policy must continuously explore new answer spaces.

Key bottlenecks include:

• Entropy collapse: Fixed weighting often drives the policy toward low‑entropy, repetitive outputs, sacrificing the breadth of medical reasoning.
• Unstable convergence: Over‑emphasizing positive samples can cause the model to overfit to a narrow set of high‑reward answers, while over‑penalizing negatives can destabilize training.
• Reward ambiguity: In open‑ended QA, a single question may have many correct answers, making it hard to label samples as strictly positive or negative.

What the Researchers Propose

The authors introduce EAPO, an entropy‑driven adaptive weighting scheme that modulates the contribution of positive samples based on the relative entropy of the current policy.

Core ideas:

• Reward‑mean discrimination: Samples whose reward exceeds the running mean are treated as positive; those below are negative.
• Entropy ratio: The ratio of the policy’s current entropy to its initial entropy determines a dynamic coefficient γ.
• Phase‑aware weighting: When entropy is decreasing (the model is converging), γ is lowered to keep exploration alive. When entropy rises (the model is re‑exploring), γ is increased to reinforce stable learning.

By continuously adjusting γ, EAPO prevents premature entropy collapse while still capitalizing on high‑quality positive feedback.

How It Works in Practice

The EAPO workflow can be broken into four logical stages:

1. Sample Generation: The policy model generates a batch of candidate answers for each medical question.
2. Reward Evaluation: Each candidate receives a scalar reward from a verifier (e.g., a medical knowledge base or human‑in‑the‑loop evaluator).
3. Positive‑Negative Partitioning: Rewards are compared against the batch mean; above‑mean samples become “positive,” below‑mean become “negative.”
4. Adaptive Weighting & Policy Update: The entropy of the policy distribution is measured. The ratio H_t/H₀ yields γ, which scales the loss contribution of positive samples before a standard policy‑gradient step is taken.

The following diagram visualizes the loop:

What sets EAPO apart from prior methods is the explicit coupling of sample weighting to the model’s own uncertainty (entropy). Traditional RLVR pipelines treat weighting as a hyper‑parameter tuned once; EAPO treats it as a dynamic signal that reacts to the learning trajectory.

Evaluation & Results

The authors validated EAPO on two publicly available open‑ended medical QA datasets: MedQA‑Open and ClinicalDialogue‑V2. Both datasets feature multi‑turn, explanatory answers rather than single‑sentence facts.

Key evaluation dimensions:

• Diversity metrics: Self‑BLEU, distinct‑n, and entropy of generated answer distributions.
• Quality metrics: Human expert rating, factual consistency scores, and task‑specific clinical accuracy.
• Stability indicators: Variance of reward across training epochs and convergence speed.

Findings:

• EAPO increased answer diversity by 18‑22 % relative to fixed‑weight baselines, as measured by distinct‑4 and entropy.
• Human evaluators rated EAPO‑generated answers 0.7 points higher on a 5‑point clinical relevance scale, indicating better factual grounding.
• Training curves showed smoother reward progression, with a 30 % reduction in variance, confirming more stable convergence.
• When the entropy‑driven coefficient was ablated (i.e., held constant), both diversity and quality dropped sharply, underscoring the necessity of adaptive weighting.

Overall, EAPO consistently outperformed static‑weight RLVR across both datasets, demonstrating that entropy‑aware adaptation is a practical lever for open‑ended QA.

Why This Matters for AI Systems and Agents

For practitioners building AI assistants, diagnostic chatbots, or knowledge‑base retrieval agents, EAPO offers a concrete recipe to balance exploration and exploitation without manual hyper‑parameter tuning.

• Agent design: Incorporating entropy‑driven weighting enables agents to generate richer, more varied medical explanations while staying anchored to verified knowledge.
• Evaluation pipelines: By monitoring policy entropy, developers gain an additional diagnostic signal that predicts when a model may be over‑fitting or under‑exploring.
• Orchestration layers: Systems that route queries to multiple specialized models can use EAPO’s adaptive coefficient as a selector, favoring models that maintain healthy entropy levels.
• Productization: The approach reduces the need for costly human‑in‑the‑loop tuning, accelerating time‑to‑market for AI‑driven health platforms.

Organizations looking to embed advanced QA capabilities can leverage existing UBOS tools to accelerate integration:

• Explore the UBOS platform overview for a modular AI stack that supports custom RL pipelines.
• Deploy conversational agents with the AI marketing agents framework, which can be repurposed for medical outreach.
• Automate data‑flow and model orchestration using the Workflow automation studio, ensuring that entropy metrics are logged and acted upon in real time.
• Integrate external LLMs such as OpenAI ChatGPT integration to enrich the reward model with up‑to‑date medical literature.
• Enable end‑user interaction via the Telegram integration on UBOS, allowing clinicians to test the system in a familiar chat environment.

What Comes Next

While EAPO marks a significant step forward, several avenues remain open for exploration:

• Multi‑modal rewards: Extending the reward function to incorporate imaging, lab results, or structured EHR data could further improve clinical relevance.
• Continual learning: Adapting the entropy‑driven coefficient in a lifelong learning setting would help agents stay current as medical guidelines evolve.
• Safety guarantees: Formalizing bounds on entropy collapse could provide provable safety margins for high‑risk deployments.
• Scalability: Testing EAPO on larger, multilingual medical corpora will reveal how the method scales with vocabulary and answer length.

Future research may also investigate hybrid schemes that blend EAPO’s entropy signal with curriculum learning, where the difficulty of sampled questions is gradually increased.

Practitioners interested in prototyping these ideas can start with the About UBOS page to learn about the team’s expertise in reinforcement learning and medical AI, and then join the UBOS partner program for early access to experimental modules.

References

• Zeng, Y., Li, G., Miao, Y., et al. (2026). EAPO: Entropy‑Driven Adaptive Positive‑Negative Sample Weighting for Policy Optimization in Open‑Ended QA. arXiv preprint arXiv:2605.27846.
• Additional background on RLVR and policy entropy can be found in standard reinforcement‑learning textbooks and recent surveys on open‑ended QA.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.