Andrii Bidochko

• Updated: July 14, 2026
• 7 min read

A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

Direct Answer

The paper introduces a systematic reliability assessment of Large‑Audio‑Language‑Model (LALM) judges—specifically Gemini 2.5 Flash, 3.5 Flash, and 3.1 Pro—for scoring full‑duplex voice‑agent conversations directly from raw stereo waveforms. It matters because the study provides the first empirical, human‑calibrated evidence that LALM audio judges can replace or augment human raters, cutting evaluation costs by two orders of magnitude while preserving metric fidelity.

Background: Why This Problem Is Hard

Full‑duplex voice agents, such as hands‑free assistants and call‑center bots, exchange speech simultaneously on both channels. Evaluating their performance requires listening to overlapping utterances, detecting turn‑taking glitches, and measuring nuanced acoustic qualities (e.g., intelligibility, latency, prosody). Traditional audio evaluation pipelines rely on human raters who:

• Listen to each stereo session in a controlled environment.
• Score multiple production dimensions (clarity, latency, naturalness, etc.).
• Repeat the process for every new model release or language variant.

This manual loop is expensive, slow, and prone to inter‑rater variability. Automated metrics (e.g., PESQ, STOI) work on single‑channel signals but falter when faced with overlapping speech, speaker‑specific accents, or injected defects. Moreover, existing AI‑based audio judges have been validated only on narrow tasks (speech‑to‑text quality, single‑speaker MOS) and lack rigorous, cross‑model reliability evidence for the full‑duplex scenario.

Consequently, product teams cannot confidently rely on AI judges for continuous integration, A/B testing, or regulatory compliance, creating a bottleneck in the rapid iteration of conversational AI systems.

What the Researchers Propose

The authors propose a three‑pronged reliability framework that treats a Gemini LALM model as an “audio judge” capable of scoring raw stereo waveforms across eight production dimensions. The framework consists of:

1. Ground‑Truth Anchor: Gemini 2.5 Flash is designated as the reference model, calibrated against three independent human raters.
2. Calibration Protocol: Human raters evaluate 209 stereo sessions (152 natural conversations + 57 adversarial defect clips) spanning 13 accent‑and‑condition strata. Their scores establish a statistical baseline for agreement.
3. Cross‑Model Transfer Tests: The same sessions are fed to Gemini 3.5 Flash and 3.1 Pro. The study measures rank correlation, absolute agreement, and defect‑sensitivity relative to the human baseline.

Key components include the LALM audio judge (the Gemini model), a calibrated human rating panel, and a defect‑injection engine that simulates real‑world audio pathologies (e.g., background noise, clipping, echo). By keeping the dataset and scoring rubric constant, the framework isolates model‑specific reliability without confounding variables.

How It Works in Practice

From an engineering standpoint, deploying a LALM audio judge follows a straightforward pipeline:

1. Capture: Raw stereo audio from a full‑duplex interaction is streamed into a storage bucket in lossless format.
2. Pre‑Processing: The waveform is normalized and segmented into fixed‑length windows (e.g., 2 seconds) while preserving channel alignment.
3. Inference: The Gemini LALM model receives the stereo tensor and outputs a vector of eight dimension scores, each on a 1‑5 Likert scale.
4. Post‑Processing: Scores are aggregated per session, optionally smoothed across windows, and compared against predefined thresholds or historical baselines.
5. Feedback Loop: If a score deviates beyond tolerance, the system flags the session for human review or triggers an automated remediation (e.g., model rollback, acoustic tuning).

What distinguishes this approach from prior audio‑metric pipelines is the end‑to‑end handling of overlapping speech without any intermediate transcription or channel separation. The LALM model learns joint acoustic‑linguistic representations directly from the waveform, enabling it to assess conversational dynamics that traditional single‑channel metrics miss.

Evaluation & Results

The researchers evaluated the framework on three fronts:

1. Rank Correlation (Spearman ρ)

Across five of the eight dimensions, the LALM‑human Spearman ρ differed from the human‑human pairwise ρ by no more than 0.07. For seven dimensions, the 95 % bootstrap confidence intervals overlapped, indicating statistically indistinguishable rank ordering.

2. Absolute Agreement

When comparing the LALM’s mean score to the three‑rater human mean, the model fell within a one‑point margin on 60 % to 92 % of sessions for six dimensions. This demonstrates that the LALM can serve as a “fourth rater” without inflating variance.

3. Defect Sensitivity (Newcombe‑Wilson)

Out of 48 (defect, dimension) cells, the LALM matched or exceeded human sensitivity in 45 cases. Although many cells were under‑powered, the trend suggests that the model can reliably detect injected audio defects at least as well as humans.

Cross‑Model Transfer

When the same evaluation was repeated with Gemini 3.5 Flash, simple agreement improved to all eight dimensions, confirming that rank‑correlation gains translate into tighter absolute matches. Conversely, Gemini 3.1 Pro, despite comparable rank correlation, produced systematically lower scores on several dimensions, warning that rank metrics alone cannot guarantee calibration.

Cost Implications

Human rating for the 209‑session corpus required roughly 200 hours of labor, translating to a cost roughly 100× higher than the compute‑only LALM inference. This cost differential underscores the economic incentive for adopting LALM judges in production pipelines.

Why This Matters for AI Systems and Agents

For product managers, engineers, and researchers building full‑duplex voice agents, the study delivers three actionable takeaways:

• Scalable Quality Assurance: LALM audio judges can be integrated into CI/CD pipelines, providing near‑real‑time feedback on conversational quality without waiting for human review cycles.
• Consistent Metric Baselines: By anchoring evaluations to a calibrated LALM, teams can reduce inter‑rater drift and maintain a stable benchmark across model releases, languages, and accent groups.
• Cost‑Effective Defect Detection: Automated sensitivity to injected defects means that regression testing for noise robustness, echo cancellation, and codec artifacts can be fully automated.

These capabilities directly impact the design of voice‑AI products that must meet strict latency and reliability SLAs. For example, a contact‑center bot that mis‑detects turn‑taking can cause customer frustration; an LALM judge can flag such failures before they reach production.

Integrating LALM judges also opens the door to richer orchestration strategies. An Workflow automation studio can route low‑scoring sessions to a human‑in‑the‑loop escalation path, while high‑scoring sessions proceed autonomously, optimizing both user experience and operational cost.

What Comes Next

Despite promising results, the authors acknowledge several limitations that shape future research:

• Domain Generalization: The current dataset focuses on English‑language conversations with a limited set of accents. Extending the evaluation to multilingual, code‑mixed, or low‑resource languages will test the LALM’s cross‑lingual robustness.
• Granular Defect Taxonomy: While the study injected generic defects, real‑world deployments encounter complex acoustic pathologies (e.g., reverberant rooms, packet loss). A more granular defect taxonomy could refine sensitivity analysis.
• Model‑Specific Calibration: The finding that Gemini 3.1 Pro under‑scores despite similar rank correlation suggests that each LALM version requires independent calibration against human baselines before production use.
• Human‑in‑the‑Loop Hybrid Scoring: Combining LALM scores with occasional human audits could create a semi‑supervised evaluation loop that continuously adapts to drift in user speech patterns.

Potential applications extend beyond voice agents. Any system that processes overlapping audio—such as multi‑party video conferences, collaborative robotics, or immersive AR/VR experiences—could benefit from LALM‑driven quality monitoring.

Developers interested in experimenting with LALM audio judges can start by integrating Gemini models through existing OpenAI ChatGPT integration or exploring voice synthesis pipelines via the ElevenLabs AI voice integration. Early adopters can also leverage the Enterprise AI platform by UBOS to orchestrate large‑scale audio evaluation workloads.

In summary, the reliability assessment validates LALM audio judges as a cost‑effective, high‑fidelity alternative to human raters for full‑duplex voice agents. By adopting this approach, organizations can accelerate iteration cycles, enforce consistent quality standards, and allocate human expertise to the most challenging edge cases.

For a deeper dive into the methodology and raw data, consult the original arXiv paper. To explore how UBOS can help you embed reliable audio evaluation into your AI stack, visit the UBOS platform overview or reach out via the UBOS partner program.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.