Andrii Bidochko

• Updated: June 19, 2026
• 8 min read

Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain‑of‑Thought Under Knowledge Conflict

Direct Answer

The paper introduces Introspective Faithfulness, a framework for measuring whether a language model’s chain‑of‑thought (CoT) explanation truly reflects the internal mechanism that decides between trusting its own knowledge and following contradictory external information. It matters because it reveals that most CoT narratives are largely decorative, while a thin layer of self‑rated confidence carries the only reliable signal about the model’s decision‑making.

Background: Why This Problem Is Hard

Large language models (LLMs) excel at generating fluent text, but they remain opaque when faced with knowledge conflict—situations where a user‑supplied document contradicts facts the model has seen during pre‑training. In practice, agents built on LLMs must decide whether to accept the new evidence or defer to their internal knowledge base. Existing interpretability tools (e.g., attention visualizations, probing classifiers) assume that the model’s natural‑language reasoning trace is a faithful window into its internal state. However, prior work has shown that LLMs can produce plausible explanations that are unrelated to the actual computation, especially when the prompt encourages “think step‑by‑step.” This gap creates two risks:

• Deployment risk: System operators may monitor CoT output for warning signs, yet the trace could be a hallucinated story.
• Research risk: Benchmarks that reward longer CoT may inadvertently incentivize models to generate verbose but meaningless rationales.

Because LLMs are increasingly used as autonomous agents—retrieving documents, updating knowledge graphs, or making business decisions—the need for a trustworthy diagnostic of their internal conflict resolution is more urgent than ever.

What the Researchers Propose

The authors define Introspective Faithfulness as the degree to which a model’s self‑generated CoT aligns with the latent factor that actually drives its answer choice under knowledge conflict. Their experimental framework consists of three components:

1. Flip‑Pair Construction: For each factual query, two versions are created—one where the supporting document agrees with the model’s pre‑training knowledge and another where it directly contradicts it. This yields “flip pairs” that force the model to either stay with its prior belief or switch to the document.
2. CoT Capture & Scoring: The model is prompted to produce a step‑by‑step reasoning trace and a self‑rated confidence score (0–100). The trace is compared across the two flips using semantic similarity metrics (ROUGE‑L, embedding‑based cosine similarity) to assess stability.
3. Statistical Linking: Correlations between confidence, knowledge‑fame (how well‑known the entity is), and the final decision are computed to isolate genuine predictive signals.

In essence, the framework treats the CoT as a hypothesis test: if the reasoning truly reflects the decision mechanism, flipping the external evidence should cause a noticeable shift in the narrative. If the narrative stays the same, it suggests the CoT is decision‑invariant and therefore not faithful.

How It Works in Practice

The workflow can be visualized as a pipeline:

• Input Layer: A factual question (e.g., “Who invented the telephone?”) plus a short document that either confirms or denies the known answer.
• Prompt Engine: The same prompt template is used across all models, asking for a CoT explanation followed by a confidence rating.
• Model Execution: Eight state‑of‑the‑art LLMs (including GPT‑4o, Claude Sonnet 4.6, and others) generate two outputs per question—one per flip.
• Analysis Module: Semantic similarity between the two CoTs is computed; confidence scores are correlated with the actual answer choice; internal “thinking tokens” (the hidden token stream before the final answer) are also examined.

What sets this approach apart is the explicit pairing of contradictory contexts and the focus on the *difference* between the two generated rationales, rather than evaluating each rationale in isolation. By holding the prompt constant and only toggling the external evidence, the method isolates the effect of knowledge conflict on the model’s internal reasoning.

Evaluation & Results

Test Scenarios

The authors evaluated 200 carefully curated factual items spanning popular, obscure, and mid‑frequency entities. Each item was presented to eight LLMs under four prompting conditions (plain answer, answer + CoT, answer + CoT + confidence, and answer + confidence only). This yielded a total of 6,400 generated responses.

Key Findings

• CoT Stability: Across flip pairs, the semantic similarity of the CoT remained extremely high (average d = 0.34, ROUGE‑L d = 0.45). In plain terms, the narrative changed only 4 % of the time, indicating that the CoT is largely decision‑invariant.
• Confidence as a Signal: Self‑rated confidence showed a modest but statistically significant correlation with the model’s final decision, even for obscure facts where entity fame offered no guidance (p < 0.001, r = 0.134). This suggests confidence captures a genuine, albeit weak, introspective cue.
• Model‑Specific Behaviors:
  • GPT‑4o displayed the only statistically reliable coupling between confidence and decision, meaning its confidence scores were meaningfully predictive of whether it would follow the document or its own knowledge.
  • Claude Sonnet 4.6 exhibited the widest confidence range (SD = 1.39) but a near‑zero pooled correlation because the confidence‑decision relationship reversed between the “agree” and “contradict” conditions. A temperature ablation confirmed this reversal is intrinsic to the model, not a prompting artifact.
  • Internal “thinking tokens” (the hidden token stream before the final answer) were more sensitive to the flip than the user‑visible CoT (p = 0.033), hinting that the model’s latent computation does reflect the conflict even when the surface explanation does not.

Why the Findings Matter

These results overturn the common assumption that a longer, step‑by‑step explanation automatically grants interpretability. Instead, the study shows that most of the CoT is a static “knowledge display” layer, while a thin “confidence” layer carries the only real insight into the model’s internal arbitration. For practitioners, this means that monitoring confidence scores—rather than the narrative itself—offers a more reliable early‑warning system for misaligned decisions.

Why This Matters for AI Systems and Agents

Agent developers often embed LLMs in pipelines that retrieve external documents, update knowledge bases, or make high‑stakes recommendations. The paper’s insights translate into concrete engineering guidelines:

• Confidence‑Driven Guardrails: Instead of parsing the CoT for red flags, systems can set thresholds on the model’s self‑rated confidence. Low confidence could trigger a fallback to a human reviewer or a secondary verification model.
• Dynamic Prompting Strategies: By explicitly requesting confidence alongside the answer, developers gain a cheap, low‑latency signal that is already part of the model’s output.
• Model Selection: The study highlights that not all LLMs treat confidence equally. GPT‑4o’s reliable confidence‑decision coupling makes it a safer choice for autonomous agents that must self‑monitor, whereas Claude Sonnet 4.6 may require additional calibration.
• Instrumentation of Latent Tokens: The sensitivity of internal thinking tokens suggests that future monitoring tools could tap into the hidden token stream (e.g., via API hooks) to obtain a richer picture of conflict resolution.

For organizations building AI‑driven workflows—such as Workflow automation studio or UBOS platform overview—embedding confidence‑based checks can reduce costly errors without sacrificing the speed benefits of LLMs.

What Comes Next

While the study makes a strong case for confidence as the primary introspective signal, several open challenges remain:

• Granular Confidence Calibration: Current confidence scores are coarse (0–100). Future work could explore calibrated probability estimates or Bayesian confidence measures that better reflect uncertainty.
• Cross‑Domain Generalization: The benchmark focuses on factual trivia. Extending the methodology to domain‑specific knowledge (e.g., medical guidelines, legal statutes) will test whether the same patterns hold.
• Multi‑Model Ensembles: Combining models with complementary confidence behaviors (e.g., GPT‑4o’s reliable confidence with Claude’s diverse reasoning styles) could yield more robust agents.
• Tooling for Latent Token Access: Developing APIs that expose the internal “thinking tokens” would let engineers build real‑time monitors that detect conflict before the final answer is emitted.

Addressing these gaps will help turn the thin confidence layer from a “signal‑plus‑noise” artifact into a full‑featured diagnostic tool. Companies interested in prototyping such capabilities can start by integrating confidence‑aware prompts into their existing bots, for example via the OpenAI ChatGPT integration or the ChatGPT and Telegram integration. Early adopters will gain a competitive edge in building trustworthy, self‑monitoring AI agents.

Visual Insight

The diagram below summarizes the experimental pipeline and highlights where confidence diverges from the static CoT layer.

Conclusion

Introspective Faithfulness reveals that most chain‑of‑thought explanations are decorative, while self‑rated confidence carries the only meaningful introspective cue about how LLMs resolve knowledge conflicts. For AI practitioners, this means shifting monitoring focus from the narrative to the confidence score, selecting models with reliable confidence‑decision coupling, and exploring latent token instrumentation for deeper insight. As autonomous agents become more prevalent, embedding these lessons will be essential for building systems that are both powerful and trustworthy.

References & Further Reading

• Original arXiv paper – Do Models Know Why They Changed Their Mind?
• Enterprise AI platform by UBOS
• AI marketing agents
• UBOS templates for quick start

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.