Andrii Bidochko

• Updated: June 10, 2026
• 7 min read

Auditable Decision Models with Learned Abstention and Real-Time Steering

Direct Answer

The paper introduces EvaluatorDPT, a bounded decision‑control model that can output YES, NO, or TBD (deferral) as a learned outcome rather than a post‑hoc confidence filter. By making deferral an integral part of the model and exposing auxiliary semantic signals, the approach enables real‑time steering, auditability, and policy‑driven routing of uncertain cases in production AI systems.

Background: Why This Problem Is Hard

Modern AI deployments—whether in fraud detection, medical triage, or autonomous agents—must often act on incomplete, contradictory, or noisy evidence. Traditional classifiers force a hard label even when the underlying data do not support a confident decision. Generative models, on the other hand, can produce plausible outputs that are difficult to trace back to a concrete decision rule, making compliance and audit trails fragile.

Key challenges include:

• Hidden uncertainty: Confidence thresholds are usually applied after the fact, leaving the model’s internal reasoning opaque.
• Policy mismatch: Business or regulatory policies may require a “defer” action (human review, escalation) that is not represented in the model’s label space.
• Auditability: Without a structured, inspectable output, regulators and internal auditors cannot verify that decisions respect defined thresholds.
• Real‑time steering: Operators need to adjust routing behavior on the fly (e.g., tighten deferral thresholds during a crisis) without retraining the model.

Existing solutions typically address one of these pain points—confidence‑based rejection, rule‑based post‑processing, or separate monitoring pipelines—but they do not provide a unified, learnable deferral mechanism that is both auditable and steerable at inference time.

What the Researchers Propose

EvaluatorDPT (Decision‑Control Transformer) reframes the decision problem as a bounded distribution over three mutually exclusive outcomes: YES, NO, and TBD. The model learns when to defer (TBD) directly from data, guided by domain‑specific policy thresholds supplied at deployment. Its architecture consists of:

• Transformer encoder: Processes heterogeneous evidence (text, numeric features, sensor streams) into a shared latent representation.
• Primary bounded‑decision head: Emits a probability vector constrained to the three outcomes, ensuring the sum equals one.
• Auxiliary channels: Optional heads that predict semantic attributes such as sentiment or emotion, providing extra signals for downstream governance.

The model’s interface is deliberately domain‑agnostic: a deployment layer injects evidence and policy thresholds, while the model returns a bounded distribution that can be inspected, logged, and routed according to real‑time steering commands.

How It Works in Practice

Conceptual Workflow

1. Evidence ingestion: Raw inputs (e.g., transaction logs, patient vitals, chat transcripts) are tokenized and embedded.
2. Contextual encoding: The transformer encoder captures cross‑modal relationships, producing a context vector.
3. Decision head inference: The bounded‑decision head maps the context vector to three logits, which are softmax‑normalized into YES, NO, and TBD probabilities.
4. Policy application: Deployment‑time thresholds (e.g., “defer if TBD ≥ 0.4”) are applied to the output distribution, yielding a final routing decision.
5. Auxiliary signal extraction (optional): If enabled, sentiment or emotion heads emit additional tags that can be logged for audit trails or used to trigger specific mitigation actions.
6. Real‑time steering: Operators can adjust thresholds on the fly via a control dashboard; the model’s bounded output guarantees that any change remains interpretable.

Component Interaction

The system is split into three logical layers:

• Data Layer: Normalizes and batches evidence, ensuring consistent tokenization across modalities.
• Model Layer: Hosts the EvaluatorDPT transformer and its heads. The bounded‑decision head is trained with a custom loss that penalizes mis‑calibration of the TBD class.
• Control Layer: Exposes an API for threshold configuration, logs every inference, and forwards auxiliary signals to downstream compliance modules.

What sets this approach apart is the *learned* deferral. Instead of a static confidence cutoff, the model internalizes when the evidence is insufficient, producing a calibrated TBD probability that aligns with downstream policy.

Visual Overview

Evaluation & Results

The authors evaluated EvaluatorDPT on a held‑out test set of 44,597 instances drawn from a real‑world decision‑making pipeline (the exact domain is anonymized for confidentiality). The evaluation focused on three axes:

• Classification performance: Accuracy and macro‑averaged F1 across the three classes.
• Calibration quality: Expected Calibration Error (ECE) measured on a validation split.
• Stability: Multi‑seed runs to verify that performance does not hinge on random initialization.

Key findings include:

• Overall accuracy of 82.6 % and macro F1 of 0.825, indicating balanced performance across YES, NO, and TBD.
• Per‑class F1 scores of 0.831 (YES), 0.849 (NO), and 0.796 (TBD), demonstrating that the model learns a meaningful deferral behavior without sacrificing the primary decision quality.
• Calibration error (ECE) of 0.0338, confirming that the probability outputs are reliable for threshold‑based routing.
• Consistent results across five random seeds, underscoring robustness.

Beyond raw metrics, the evaluation package includes confusion matrices, threshold‑sweep curves, and reproducibility scripts, enabling external auditors to verify claims and reproduce the results independently.

Why This Matters for AI Systems and Agents

For practitioners building AI‑driven agents, the ability to *audit* every decision is becoming a regulatory prerequisite. EvaluatorDPT offers a concrete pathway to embed auditability directly into the model, rather than as an afterthought.

• Governance compliance: The bounded output can be logged verbatim, satisfying audit trails required by finance, healthcare, and public‑sector regulations.
• Human‑in‑the‑loop workflows: Deferral decisions (TBD) can trigger escalation to a human reviewer, reducing false‑positive risk in high‑stakes environments.
• Dynamic policy enforcement: Real‑time steering lets operations teams tighten or relax deferral thresholds during incidents (e.g., a surge in fraudulent activity), without redeploying a new model.
• Explainability augmentation: Auxiliary sentiment or emotion heads provide contextual cues that can be surfaced in UI dashboards, helping stakeholders understand *why* a case was deferred.

These capabilities align closely with emerging AI governance frameworks and can be leveraged in existing platforms. For example, integrating EvaluatorDPT‑style decision control into a AI marketing agents workflow ensures that promotional content is only auto‑published after passing a calibrated compliance check.

Enterprises that have adopted the Enterprise AI platform by UBOS can embed the bounded decision model as a micro‑service, feeding its TBD signals into the Workflow automation studio to orchestrate downstream human review tasks.

What Comes Next

While EvaluatorDPT marks a significant step toward auditable AI, several avenues remain open for research and productization:

• Domain‑specific extensions: Tailoring auxiliary heads (e.g., risk scores, legal clause detection) to particular industries could enrich audit logs.
• Multi‑modal evidence scaling: Testing the architecture on richer sensor suites (video, audio) will validate its robustness in autonomous‑vehicle or surveillance contexts.
• Feedback loops: Incorporating post‑decision outcomes (e.g., human reviewer verdicts) to continuously fine‑tune the deferral threshold could improve long‑term calibration.
• Open‑source tooling: Providing a plug‑and‑play library for the bounded‑decision head would accelerate adoption across the AI community.

Developers interested in experimenting with the model can explore the UBOS platform overview, which offers a sandbox for deploying custom transformer‑based services. Start‑ups looking to prototype auditable decision pipelines may find the UBOS for startups program particularly supportive, with low‑cost compute credits and integration guides.

Ultimately, the vision is a future where every AI‑driven action is accompanied by a transparent, policy‑driven deferral signal, enabling organizations to balance automation speed with human oversight.

References

Auditable Decision Models with Learned Abstention and Real-Time Steering

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.