Andrii Bidochko

• Updated: August 15, 2026
• 7 min read

Synchronizing Beliefs with Second-Order Theory-of-Mind in Human-Autonomy Teams (Extended Version)

Direct Answer

The paper introduces a human‑autonomy team framework that lets a knowledgeable teacher (human) design preference‑learning queries while the learner (autonomous agent) maintains a second‑order Theory‑of‑Mind (ToM‑2) model of the teacher’s beliefs. By synchronizing these belief models, the system extracts richer preference information and learns target rewards faster than traditional learner‑driven query strategies.

This matters because it turns the teacher’s expertise—knowing the true objective—into a systematic advantage, dramatically reducing the number of interactions needed to align robots or AI agents with human intent, especially in high‑dimensional reward spaces.

Background: Why This Problem Is Hard

Preference‑based reward learning has become the de‑facto method for aligning autonomous systems when designers cannot write down an explicit reward function. The typical workflow treats the human as a passive oracle that answers binary queries such as “Do you prefer behavior A or B?” While simple, this approach suffers from three intertwined bottlenecks:

• Query inefficiency: Learners generate queries based on their current uncertainty, ignoring the teacher’s deeper knowledge of the objective. In high‑dimensional feature spaces, random or learner‑driven queries quickly become redundant.
• Model drift: As the learner updates its belief about the reward, the teacher’s mental model of the learner’s knowledge becomes stale, leading to mis‑aligned queries and slower convergence.
• Lack of structured feedback: Binary preferences convey only a relative ordering, discarding richer information the teacher could provide about why a choice is better.

Existing solutions—active preference learning, Bayesian optimization, or inverse reinforcement learning—attempt to mitigate query inefficiency but still assume a one‑way information flow from teacher to learner. They do not exploit the teacher’s ability to anticipate the learner’s knowledge gaps, nor do they provide a mechanism for the learner to signal its understanding back to the teacher.

What the Researchers Propose

The authors recast preference learning as a human‑autonomy team problem with two coupled models:

1. Teacher model: The human maintains an explicit belief about what the learner currently knows. Using this belief, the teacher selects the most informative pairwise comparison (or curriculum) to present.
2. Learner model: The autonomous agent builds a second‑order Theory‑of‑Mind (ToM‑2) representation of the teacher’s belief. After each query, the learner emits a structured “understanding statement” that tells the teacher how its belief has shifted.

These two models are synchronized through a feedback loop:

• The teacher designs queries that target the learner’s uncertainty.
• The learner’s ToM‑2 statements correct the teacher’s mental model, preventing drift.
• When the teacher’s error is directional (e.g., over‑emphasizing a particular feature), ToM‑2 statements outperform simpler mean‑belief updates.

In essence, the framework transforms the teacher from a passive answerer into an active curriculum designer, while giving the learner a voice to keep the teacher’s perception accurate.

How It Works in Practice

The operational workflow can be broken down into four stages, each mapped to a concrete software component:

1. Belief Initialization

The learner starts with a prior distribution over reward parameters (e.g., a Gaussian over feature weights). Simultaneously, the teacher forms an initial estimate of this prior based on domain knowledge.

2. Query Generation (Teacher‑Side)

Using its belief about the learner, the teacher computes the expected information gain for candidate behavior pairs. The pair with the highest gain is presented to the human for a preference decision.

3. Preference Capture & Understanding Statement (Learner‑Side)

After the human selects a preferred behavior, the learner updates its first‑order belief (standard Bayesian update). Crucially, it also generates a ToM‑2 “understanding statement” that encodes:

• The direction and magnitude of the belief shift.
• Confidence about which features drove the change.
• Any residual uncertainty that the teacher should consider in the next query.

4. Teacher Model Synchronization

The teacher ingests the understanding statement, adjusts its internal model of the learner, and repeats the cycle. If the teacher’s model drifts (e.g., after alternating between multiple teachers), the learner’s ToM‑2 statements act as a corrective signal, re‑aligning expectations.

What distinguishes this approach from prior active learning pipelines is the explicit, bidirectional modeling of beliefs. Rather than treating the learner as a black box, the system maintains a shared mental model that evolves with each interaction.

Evaluation & Results

The authors validated the framework in three simulated environments that mimic common robot alignment tasks:

1. Linear reward synthesis: A synthetic reward defined over 20 features, where the teacher knows the true weight vector.
2. Navigation preference: An autonomous vehicle must learn a trade‑off between speed, safety, and passenger comfort.
3. Manipulation task: A robot arm learns to prioritize grasp stability versus execution time.

Key experimental conditions included:

• Informed teacher using the proposed curriculum.
• Learner‑driven selection (baseline active preference learning).
• Alternating teachers to induce model drift.
• Variants of learner feedback: simple mean‑belief statements vs. full ToM‑2 statements.

Results consistently showed that the informed teacher outperformed the learner‑driven baseline, achieving target reward accuracy with 30‑45 % fewer queries. When teacher models drifted due to alternating instructors, performance degraded, but the introduction of ToM‑2 understanding statements restored most of the lost efficiency. Notably, ToM‑2 statements were especially beneficial when the teacher’s error was concentrated along a specific feature axis, confirming the hypothesis that second‑order modeling captures directional bias better than aggregate mean updates.

Why This Matters for AI Systems and Agents

For practitioners building real‑world autonomous agents, the paper offers a concrete pathway to reduce the data‑collection burden that typically hampers alignment projects. The implications span several domains:

• Robot alignment pipelines: By leveraging teacher expertise early, developers can shrink the number of human‑in‑the‑loop iterations, accelerating deployment cycles.
• Multi‑agent collaboration: In teams where several humans or AI supervisors provide feedback, the ToM‑2 mechanism can keep each supervisor’s mental model of the agent synchronized, preventing contradictory training signals.
• Enterprise AI platforms: Systems like the UBOS platform overview can embed this framework as a modular “preference curriculum” service, offering clients a plug‑and‑play alignment layer for custom agents.
• Voice‑enabled assistants: Coupling ToM‑2 feedback with the ElevenLabs AI voice integration enables agents to ask clarifying questions and receive structured understanding statements via spoken language.
• Chat‑based workflows: Integrating the framework with the OpenAI ChatGPT integration allows developers to collect preference data directly from chat sessions, while the agent simultaneously updates its belief about the user’s intent.

Collectively, these capabilities translate into faster time‑to‑value for AI products, lower annotation costs, and more reliable safety guarantees—critical factors for enterprises adopting autonomous technology.

What Comes Next

While the study demonstrates clear benefits, several open challenges remain:

• Scalability to real‑world noise: Human preferences are often inconsistent. Future work must test robustness when teachers provide contradictory or noisy statements.
• Multi‑teacher coordination: In large organizations, multiple experts may teach the same agent. Designing protocols for aggregating divergent ToM‑2 feedback is an open research avenue.
• Extension beyond pairwise queries: Incorporating richer feedback modalities (e.g., ranking, natural language explanations) could further accelerate learning.
• Hardware‑in‑the‑loop validation: Deploying the framework on physical robots, drones, or manufacturing arms will reveal latency and safety considerations not captured in simulation.

Potential application pathways include:

• Embedding the curriculum engine in the Workflow automation studio to orchestrate human‑in‑the‑loop training pipelines.
• Leveraging the AI marketing agents to personalize campaign strategies through preference‑driven reward shaping.
• Connecting to the ChatGPT and Telegram integration for remote, asynchronous preference collection in distributed teams.
• Utilizing the Telegram integration on UBOS to deliver real‑time query prompts to field operators, enabling on‑site alignment without specialized UI.

By addressing these gaps, the community can move from simulated proof‑of‑concepts to production‑grade systems that reliably align autonomous agents with nuanced human values.

For readers interested in the full technical details, the original pre‑print is available on arXiv.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.