Andrii Bidochko

• Updated: July 13, 2026
• 7 min read

A Transdiagnostic Space of Disorder Like Phenotypes in Reinforcement Learning Agents

Direct Answer

The paper introduces a dose‑controllable, appraisal‑guided reinforcement‑learning framework that can induce seven distinct psychological‑disorder phenotypes in a single PPO agent, and it demonstrates that these phenotypes self‑organise into a low‑dimensional affective space. This matters because it offers a scalable, testable platform for computational psychiatry while exposing systematic failure modes of affect‑driven AI agents.

Background: Why This Problem Is Hard

Modeling mental‑health disorders in artificial agents has traditionally been a manual, ad‑hoc exercise. Researchers would tweak reward functions to mimic a single pathology, run a handful of simulations, and then label the resulting behaviour after the fact. Such approaches suffer from three critical limitations:

• Non‑systematic manipulation: Hand‑crafted reward shaping does not map cleanly onto the underlying cognitive processes identified by psychiatry, making it hard to compare across studies.
• Lack of reproducibility: Single‑run demonstrations cannot capture the stochastic nature of reinforcement learning, leading to results that are difficult to verify.
• Fragmented phenotype space: Each disorder is modelled in isolation, preventing exploration of comorbidity, treatment interaction, or emergent affective structures.

These gaps matter for two reasons. First, computational psychiatry needs a reliable sandbox where hypotheses about appraisal, reward, and learning can be tested at scale. Second, modern AI agents—especially those deployed in high‑stakes environments like finance, healthcare, or autonomous systems—must be robust to affect‑related failures, yet current evaluation pipelines lack systematic stress‑testing for such failures.

What the Researchers Propose

The authors recast disorder modelling as a set of seven “knobs,” each representing a weighted appraisal signal grounded in a computational psychiatry theory. By adjusting a knob’s magnitude, the agent’s internal appraisal critic is nudged toward a specific affective bias (e.g., heightened threat sensitivity for anxiety or exaggerated reward expectancy for mania). The core of the system is an appraisal‑guided Proximal Policy Optimization (PPO) agent that:

1. Receives standard state observations from a grid‑world environment.
2. Computes a vector of appraisal signals (e.g., threat, reward, uncertainty) using a lightweight critic network.
3. Applies a scalar dose to one of the seven appraisal dimensions, thereby altering the agent’s perceived value of actions.
4. Updates its policy via PPO while the altered appraisal continuously shapes the reward signal.

Each knob corresponds to a well‑studied disorder:

• Anxiety – threat‑overweighting
• Mania – reward‑overvaluation
• Obsessive‑Compulsive Checking – uncertainty‑driven perseveration
• Depression – reward‑underweighting
• Impulsivity – diminished deliberation horizon
• Addiction – habit‑loop reinforcement
• Post‑Traumatic Stress – persistent threat memory

Crucially, the knobs are continuous, enabling a graded dose‑response analysis rather than a binary “on/off” simulation.

How It Works in Practice

At a conceptual level, the workflow proceeds through three stages:

1. Environment Interaction

The agent navigates a deterministic grid‑world where each cell may contain rewards, penalties, or neutral observations. The environment is deliberately simple to isolate the effect of appraisal manipulation.

2. Appraisal Computation

A parallel critic network ingests the current state and outputs a set of appraisal scores. These scores are analogous to the human affective dimensions of threat, reward, and uncertainty. The researcher selects a target disorder knob and injects a scalar dose into the corresponding appraisal channel.

3. Policy Update via PPO

The altered appraisal vector reshapes the immediate reward signal, which the PPO algorithm then uses to compute advantage estimates and update the policy. Because PPO enforces a clipped surrogate objective, the agent’s learning remains stable even as the reward landscape is perturbed.

What distinguishes this approach from prior work is the separation of appraisal (a cognitive bias) from policy learning. Instead of hard‑coding a reward bias, the system simulates how a mental disorder would skew an agent’s internal evaluation of outcomes, producing more realistic and interpretable behaviours.

Evaluation & Results

The authors conducted over a thousand experimental runs (10 random seeds per condition, four control groups, 95 % confidence intervals) across two domains: a classic 2‑D grid‑world and a 3‑D pixel environment (MiniWorld) using a convolutional PPO agent without an explicit appraisal critic. Each disorder was assessed with a preregistered behavioural assay that mirrors a standard psychological paradigm (e.g., elevated avoidance in an open‑field test for anxiety).

Key findings include:

• Monotonic dose‑response: Every knob produced a graded change in its target assay, and no control condition replicated this pattern.
• Emergent affective space: When visualised via principal component analysis, the seven disorders collapsed onto a two‑dimensional plane where mania and anxiety appeared as opposite poles, suggesting a shared underlying affective axis.
• Selective remediation: Turning off reward‑distortion knobs (mania, checking, addiction) restored normal performance, whereas avoidance‑related knobs (anxiety, PTSD) required a graded exposure curriculum to recover.
• Non‑additive comorbidity: Simultaneous activation of two knobs produced interaction effects that could not be predicted by summing individual impacts, offering testable hypotheses about real‑world comorbid presentations.
• Cross‑domain transfer: The depression, addiction, and anxiety knobs transferred to MiniWorld with consistent assay dissociation, demonstrating that the framework is not tied to a specific environment or architecture.

These results collectively validate the framework as a robust, scalable platform for probing affect‑driven failure modes and for generating falsifiable predictions about disorder interactions.

Why This Matters for AI Systems and Agents

For practitioners building autonomous agents, the paper offers a concrete methodology to stress‑test systems against affective biases that could manifest as unsafe or sub‑optimal behaviour. By exposing an agent to graded anxiety or impulsivity, developers can observe how policy degradation unfolds and can design mitigation strategies—such as adaptive exposure curricula or dynamic appraisal recalibration—before deployment.

From a product perspective, the ability to simulate comorbid disorders opens a new avenue for AI marketing agents that need to adapt to fluctuating user emotional states, or for workflow automation studios that must remain robust under stress‑induced decision noise. Moreover, the framework’s modular appraisal layer can be integrated into existing reinforcement‑learning pipelines without redesigning the entire architecture, making it a low‑overhead addition for teams seeking to future‑proof their agents.

Finally, the research bridges a gap between computational psychiatry and practical AI safety, providing a shared language (appraisal weights) that can be used by clinicians, neuroscientists, and engineers alike.

What Comes Next

While the study establishes a powerful baseline, several open challenges remain:

• Rich sensory domains: Extending appraisal‑guided RL to high‑dimensional inputs such as language or multimodal video will test the scalability of the critic network.
• Real‑world validation: Aligning the simulated phenotypes with clinical data (e.g., neuroimaging or behavioural logs) could turn the platform into a diagnostic aid.
• Dynamic treatment protocols: The paper hints that the same knobs used to induce a disorder can model its remediation. Future work could formalise adaptive treatment policies that automatically tune appraisal doses based on performance feedback.
• Integration with enterprise AI stacks: Embedding the appraisal layer into the Enterprise AI platform by UBOS would allow large‑scale organisations to run systematic affective stress tests across fleets of agents.

Developers interested in experimenting with the framework can start by cloning the open‑source implementation, swapping in their own environments, and leveraging UBOS’s platform overview for orchestration and monitoring. As the community iterates, we can expect a richer taxonomy of affective phenotypes, more nuanced comorbidity models, and ultimately, safer, more human‑aligned AI agents.

For a deeper dive into the methodology and to explore the full set of experimental details, read the original arXiv paper. Stay tuned to the UBOS blog for updates on how this research is being integrated into next‑generation AI products.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.