Andrii Bidochko

• Updated: June 11, 2026
• 6 min read

Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback

Direct Answer

COSE (Confidence‑Orchestrated Self‑Evolution) introduces a lightweight uncertainty signal—model‑generated confidence—to steer self‑training of large language models (LLMs) without external verifiers. By weighting policy updates and replay buffers with confidence, COSE reduces harmful gradient noise and consistently lifts reasoning, math, and code performance across multiple model sizes.

Background: Why This Problem Is Hard

Self‑evolving LLMs aim to close the data‑annotation loop by letting the model create its own training tasks, solve them, and learn from the outcomes. In theory, this reduces reliance on costly human labeling and accelerates domain‑specific expertise acquisition. In practice, the loop collapses when the model must also judge the correctness of its own answers. Erroneous self‑judgments become noisy supervision, leading to “self‑reinforcing” errors that degrade model quality over time.

Current work tackles this dilemma in two ways. First, external verifiers—such as separate LLMs, symbolic solvers, or human‑in‑the‑loop checks—provide a reliable correctness signal but limit scalability and generality. Second, approaches that treat the model’s raw feedback as supervision accept the noise, often requiring elaborate filtering heuristics that still leak errors into gradient updates. Both strategies struggle to balance autonomy with trustworthiness, especially in open‑ended reasoning domains where ground truth is scarce.

What the Researchers Propose

COSE reframes confidence as an intrinsic uncertainty proxy that the LLM can compute cheaply during generation. The framework consists of two tightly coupled mechanisms:

• Confidence‑Weighted Proximal Policy Optimization (PPO): Gradient steps are scaled by the model’s reported confidence, amplifying updates from high‑certainty answers while damping those from doubtful ones.
• Confidence‑Prioritized Replay: Self‑generated experiences are stored in a replay buffer and sampled proportionally to confidence, ensuring that the training loop revisits the most trustworthy examples more often.

These components together orchestrate a self‑evolution cycle where the model continuously refines its own task‑generation, solution, and evaluation pipeline, all guided by a single, low‑overhead signal.

How It Works in Practice

The COSE workflow can be broken down into four stages:

1. Task Generation: The base LLM samples a prompt from a distribution of reasoning problems (e.g., arithmetic, logic puzzles, code synthesis).
2. Solution Synthesis: The model attempts to answer the prompt, simultaneously emitting a token‑level confidence score (often derived from the softmax probability of the chosen token).
3. Self‑Evaluation: Using the same confidence estimator, the model predicts the likelihood that its answer is correct. This confidence value becomes the “trust weight” for the example.
4. Learning Update: The example, together with its confidence weight, is fed into a PPO optimizer. The replay buffer prioritizes high‑confidence samples, and the loss is scaled accordingly.

What distinguishes COSE from prior self‑training pipelines is the exclusive reliance on the model’s own probability distribution as a signal of uncertainty. No external oracle, no additional classifier, and no handcrafted heuristics are required. The confidence scores are computed on‑the‑fly, keeping the computational overhead minimal.

Evaluation & Results

The authors benchmarked COSE on 19 held‑out tasks spanning general reasoning, mathematics, and code generation. Four backbone models—Qwen and Llama variants ranging from 0.6 B to 4 B parameters—served as the base LLMs. Each model was evaluated under three conditions: (1) vanilla fine‑tuning, (2) self‑evolution without confidence weighting, and (3) COSE.

Key findings include:

• Consistent Gains: Across all model sizes, COSE delivered average improvements of 3–5 percentage points on reasoning benchmarks compared to the vanilla baseline.
• Mathematics Edge: On arithmetic and symbolic math suites, COSE achieved the highest average score among all tested methods, narrowing the gap to specialist solvers.
• Code Competitiveness: While code generation saw modest gains, COSE remained on par with the best self‑evolution variants, demonstrating that confidence weighting does not harm domain‑specific performance.
• Stability: Training curves showed reduced variance, indicating that confidence‑orchestrated updates mitigate the destabilizing effect of noisy self‑feedback.

These results validate the hypothesis that an internal confidence signal can act as a reliable proxy for external verification, enabling scalable self‑evolution without sacrificing quality.

Why This Matters for AI Systems and Agents

For practitioners building autonomous agents, COSE offers a practical pathway to continuous improvement without the perpetual need for human‑in‑the‑loop labeling. By embedding confidence‑aware learning loops, developers can:

• Deploy agents that self‑diagnose uncertainty and prioritize high‑confidence experiences, leading to more robust decision‑making.
• Reduce operational costs associated with data annotation pipelines, especially in niche domains where expert labeling is expensive.
• Integrate seamlessly with existing reinforcement‑learning‑from‑human‑feedback (RLHF) stacks, as confidence weighting can be layered on top of standard PPO implementations.

Organizations looking to embed AI into customer‑facing workflows can leverage COSE to keep their models up‑to‑date with evolving user queries. For example, a chatbot that continuously generates and solves new FAQ scenarios can use confidence scores to decide which self‑learned answers to trust, thereby maintaining high service quality.

Explore how confidence‑driven self‑evolution can complement your existing AI stack on the UBOS platform overview, or accelerate marketing content generation with AI marketing agents. The modular nature of COSE also aligns well with the Workflow automation studio, enabling you to orchestrate confidence‑aware pipelines without deep ML engineering effort.

What Comes Next

While COSE marks a significant step forward, several open challenges remain:

• Confidence Calibration: The raw softmax probabilities can be miscalibrated, especially in large models. Future work should investigate temperature scaling or Bayesian approaches to refine the confidence signal.
• Cross‑Domain Generalization: COSE was evaluated on reasoning, math, and code tasks. Extending the framework to multimodal domains (vision‑language, audio) will test its universality.
• Safety and Alignment: Self‑evolution raises concerns about unintended behavior drift. Incorporating safety constraints into the confidence‑weighted update rule could mitigate risk.
• Hybrid Verification: Combining lightweight confidence with occasional external checks (e.g., symbolic solvers) may yield a best‑of‑both‑worlds strategy.

Practitioners interested in experimenting with COSE can start by integrating confidence‑aware replay buffers into their existing pipelines. The OpenAI ChatGPT integration provides a ready‑made endpoint for extracting token‑level probabilities, which can serve as the confidence backbone.

For teams building conversational agents on messaging platforms, the Telegram integration on UBOS offers a low‑friction way to collect real‑world user interactions, feed them into a COSE‑enhanced model, and instantly observe confidence‑driven performance gains.

Overall, COSE opens a new research frontier where uncertainty becomes a first‑class citizen in self‑learning loops, promising more autonomous, efficient, and trustworthy AI systems.

References

• Wei, B., Wang, N., Zhou, Y., Pan, J., & Zhu, Z. (2026). Confidence‑Orchestrated Self‑Evolution against Uncertain LLM Feedback. arXiv preprint arXiv:2605.28010.
• OpenAI. (2023). Advances in Reinforcement Learning from Human Feedback.
• Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.