Andrii Bidochko

• Updated: July 16, 2026
• 7 min read

Best-of-$N$ TTS Evaluation is Confounded by ASR Family Alignment

Direct Answer

The paper introduces a systematic analysis of how the choice of automatic speech recognition (ASR) family used as a verifier can dramatically skew Best‑of‑$N$ (BoN) text‑to‑speech (TTS) evaluation, and it proposes cross‑family rank ensembles that neutralize this bias while delivering state‑of‑the‑art word error rate (WER) reductions.

By exposing the “ASR family alignment” confound and offering a practical, reproducible solution, the work reshapes how researchers and product teams benchmark zero‑shot TTS systems for real‑world deployments.

Background: Why This Problem Is Hard

Zero‑shot TTS models such as F5‑TTS generate speech without task‑specific fine‑tuning, which makes them attractive for rapid product iteration. However, their output quality varies widely across generated candidates, prompting the community to adopt Best‑of‑$N$ inference: synthesize $N$ utterances, run each through an ASR verifier, and pick the one with the lowest transcription error.

While BoN improves perceived consistency, it also inherits a hidden dependency on the verifier’s underlying ASR architecture. Existing pipelines typically pair a single ASR family (e.g., Whisper) with the BoN selector, assuming that a lower WER directly reflects higher synthesis fidelity. This assumption is problematic for three reasons:

• Family‑specific acoustic biases: Different ASR models are trained on distinct corpora and feature sets, causing them to favor certain prosodic or spectral patterns.
• Representation overlap is deceptive: Even when two ASR families produce highly correlated embeddings (linear CKA ≈ 0.98), their ranking behavior can diverge, indicating lineage‑level coupling rather than pure representational similarity.
• Evaluation reproducibility: Researchers reporting BoN improvements without disclosing the verifier family risk presenting results that are not comparable across studies.

These issues matter because TTS quality directly influences user experience in voice assistants, audiobooks, and accessibility tools. An evaluation framework that unintentionally rewards ASR‑specific artifacts can lead product teams to ship sub‑optimal voices that perform poorly under alternative recognizers or real‑world noise conditions.

What the Researchers Propose

The authors propose two complementary strategies to eliminate the ASR family alignment confound:

1. Cross‑family rank averaging: Each candidate utterance receives a rank from multiple, heterogeneous ASR verifiers (e.g., Whisper, wav2vec 2.0, HuBERT). The final score is the arithmetic mean of these ranks, smoothing out family‑specific preferences.
2. Conjunctive max‑rank selection: Instead of averaging, this method takes the worst (maximum) rank assigned to a candidate across all verifiers. The candidate with the lowest worst‑case rank is chosen, guaranteeing that no single ASR family can dominate the decision.

Both ensembles operate at the ranking level rather than raw WER values, preserving the relative ordering while mitigating systematic bias. The framework is deliberately model‑agnostic: any ASR that can produce a transcription and a confidence‑based ranking can be plugged in, enabling seamless integration with existing BoN pipelines.

How It Works in Practice

The practical workflow can be broken down into four stages, illustrated in the diagram below:

1. Candidate Generation

A base TTS model (e.g., F5‑TTS) synthesizes $N$ audio samples from the same input text, each using a different random seed or sampling temperature. This step is unchanged from traditional BoN pipelines.

2. Multi‑ASR Verification

Each audio sample is fed into a suite of ASR systems representing distinct families:

• OpenAI Whisper – a transformer‑based encoder‑decoder model trained on a massive multilingual corpus.
• wav2vec 2.0 – a self‑supervised model that excels at low‑resource languages.
• HuBERT – a hidden‑unit BERT model that captures fine‑grained acoustic patterns.

Every ASR returns a transcription and a confidence score, which is converted into a rank (1 = best, $N$ = worst) for that candidate.

3. Rank Ensemble Computation

Depending on the chosen ensemble strategy, the system either averages the ranks (rank‑averaging) or selects the maximum rank (conjunctive max‑rank) for each candidate. The candidate with the lowest ensemble score is promoted as the final output.

4. Output Selection and Post‑Processing

The selected audio is optionally passed through a lightweight post‑processor (e.g., denoising, volume normalization) before delivery to downstream applications such as voice assistants or audiobook pipelines.

What distinguishes this approach from prior BoN methods is the explicit triangulation across heterogeneous recognizers, which forces the TTS system to produce speech that is universally intelligible rather than merely “Whisper‑friendly.”

Evaluation & Results

The authors evaluated the framework on the LibriSpeech‑PC test‑clean benchmark using F5‑TTS as the synthesis backbone. Three independent ASR families served as both verifiers and evaluators, creating a matrix of same‑family and cross‑family pairings.

Key findings include:

• Rank reversal across families: A candidate ranked first by Whisper could fall to last place under wav2vec 2.0, demonstrating the severity of the alignment confound.
• Same‑family advantage: When the verifier and evaluator belong to the same ASR family, the observed oracle headroom (the gap between the best possible and the selected candidate) was 2–3× larger than in cross‑family settings, despite near‑identical embedding similarity (CKA ≈ 0.978).
• Cross‑family ensembles outperform single verifiers: Rank‑averaging achieved a mean WER of 1.61 % at $N=10$, a 12 % relative improvement over the baseline F5‑TTS (2.06 %). The conjunctive max‑rank method yielded comparable gains with even tighter worst‑case performance.
• No degradation on perceptual metrics: Automatic speech quality scores (SIM‑o, UTMOS) remained statistically unchanged, confirming that the ensembles improve intelligibility without sacrificing naturalness.

These results were consistent across all three evaluators, indicating that the ensembles generalize beyond any single ASR family. The authors also released the full codebase and evaluation scripts, encouraging reproducibility.

For readers interested in the original study, the full arXiv paper provides detailed methodology and statistical analysis.

Why This Matters for AI Systems and Agents

From a product engineering perspective, the findings have immediate, actionable implications:

• Robust voice agents: Deploying TTS voices that pass multiple ASR checks ensures that downstream conversational agents can reliably understand user feedback, even when the user’s device runs a different recognizer.
• Evaluation standardization: Cross‑family triangulation can become a new benchmark norm, reducing the risk of “over‑fitting” to a single verifier and making published results more comparable across research groups.
• Integration with existing AI stacks: The ensemble approach can be wrapped as a micro‑service that sits between a TTS engine and any downstream application, requiring only API endpoints for candidate audio and ASR transcription.
• Cost‑effective quality control: By selecting the best candidate early in the pipeline, developers can avoid expensive human listening tests while still meeting strict quality thresholds.

These benefits translate directly into better user experiences for solutions built on the UBOS platform overview, where voice synthesis is often combined with conversational AI. For startups looking to prototype quickly, the UBOS for startups offering includes pre‑configured pipelines that can ingest multiple ASR services, making the cross‑family ensemble a plug‑and‑play component.

Moreover, teams that already leverage ElevenLabs AI voice integration can augment their workflow with the ensemble logic to guarantee that the generated voice sounds natural across diverse listening environments and recognizer back‑ends.

What Comes Next

While the paper makes a compelling case for cross‑family rank ensembles, several avenues remain open for exploration:

• Expanding verifier diversity: Incorporating non‑neural ASR systems, language‑specific recognizers, or even end‑to‑end speech‑to‑text APIs could further reduce family‑specific bias.
• Dynamic ensemble weighting: Instead of uniform averaging, learning a weighting scheme based on verifier confidence or historical performance might yield even lower WER.
• Real‑time constraints: Evaluating the latency impact of running multiple ASR verifiers in production and exploring lightweight approximations (e.g., phoneme‑level scoring) would be valuable for interactive applications.
• Human‑in‑the‑loop refinement: Combining the ensemble with occasional human listening checks could create a semi‑automated quality assurance loop, especially for high‑stakes domains like healthcare.

From an ecosystem standpoint, the Workflow automation studio can orchestrate these multi‑ASR verification steps, allowing engineers to define custom routing rules (e.g., fallback to a faster ASR under high load). The AI marketing agents could also benefit by automatically selecting the most universally intelligible voice for promotional videos, ensuring that the message reaches the widest audience regardless of the playback device.

Finally, as the field moves toward multimodal agents that combine speech, vision, and language, the principle of cross‑modal verification—testing a generation against multiple downstream consumers—may become a standard design pattern. The insights from this study provide a concrete blueprint for that broader vision.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.