Kyutai Unveils Hibiki‑Zero: A 3‑Billion‑Parameter Simultaneous Speech‑to‑Speech Translation Model

What Makes Hibiki‑Zero Different?

Hibiki‑Zero is built as a decoder‑only transformer that processes three synchronized streams: the source audio tokens, the target audio tokens, and an internal “inner monologue” of padded text tokens. This multistream design, powered by the Mimi neural audio codec, enables the model to handle non‑monotonic word dependencies—a common challenge in real‑time translation where the target language may reorder phrases.

The most striking innovation is the removal of any word‑level aligned data. Traditional S2ST systems rely on painstakingly curated alignments between source and target speech, limiting scalability to high‑resource languages. Hibiki‑Zero sidesteps this by using a two‑stage training regime that starts with coarse sentence‑level alignment and then refines the policy with Group Relative Policy Optimization (GRPO) reinforcement learning.

In practical terms, developers can now train new language pairs with far fewer hours of speech data, opening the door to truly global, low‑resource translation solutions.

Technical Deep‑Dive: Architecture & Training

Model Architecture

• Total Parameters: 3 B (A3B configuration)
• Temporal Transformer: 28 layers, latent dimension 2048
• Depth Transformer: 6 layers per codebook, latent dimension 1024
• Context Window: Up to 4 minutes of continuous audio
• Audio Codebooks: 16 hierarchical levels for high‑fidelity speech synthesis
• Codec: Mimi, a causal streaming codec operating at 12.5 Hz token rate

Training Pipeline

Stage 1 – Coarse Alignment Training

The model first learns from sentence‑level aligned corpora. By inserting artificial silences into the target audio, the system learns to delay its output appropriately, establishing a rough temporal mapping without needing fine‑grained word alignments.

Stage 2 – GRPO Reinforcement Learning

After the coarse stage, Hibiki‑Zero undergoes GRPO, a reinforcement‑learning algorithm that optimizes a combined reward of BLEU score (translation quality) and latency. The reward is computed at multiple checkpoints during generation, allowing the model to learn when to “listen” and when to “speak.” A hyper‑parameter α balances speed versus accuracy; lower α values push the model toward lower latency at a modest quality trade‑off.

Zero‑Shot Multilingual Capability

Because the model is trained on a shared multilingual token space, it can translate between language pairs it has never seen during fine‑tuning—a true zero‑shot capability. This is especially valuable for emerging markets where parallel corpora are scarce.

Benchmark Results: How Hibiki‑Zero Stacks Up

Kyutai evaluated Hibiki‑Zero on the Audio‑NTREX‑4L long‑form benchmark and the Europarl‑ST short‑form tasks. The model consistently outperformed Meta’s Seamless system across key metrics.

On short‑form Europarl‑ST, Hibiki‑Zero achieved an ASR‑BLEU of 34.6 with an average lag of only 2.8 seconds. Human evaluators also rated its speech naturalness and voice fidelity significantly higher than competing baselines.

Real‑World Applications: From Call Centers to Global Events

The ability to translate speech in real time without language‑specific alignment data unlocks a host of use‑cases:

• Customer Support: Integrate with Customer Support with ChatGPT API to provide instant multilingual assistance.
• Live Broadcasts: Stream conferences, webinars, or sports events with on‑the‑fly translation for global audiences.
• Enterprise Collaboration: Power multilingual meetings in platforms like Microsoft Teams or Zoom via the Enterprise AI platform by UBOS.
• Travel & Hospitality: Deploy on‑device translation in kiosks or mobile apps, leveraging the AI for Turn‑by‑Turn Directions template.
• Education: Offer real‑time lecture translation, enabling inclusive learning for non‑native speakers.

Because the model does not depend on language‑specific alignment, new language packs can be rolled out quickly—an advantage for startups aiming to enter emerging markets. The UBOS for startups program can accelerate such deployments by providing ready‑made pipelines and hosting.

Expert Opinions

“Hibiki‑Zero demonstrates that reinforcement learning can replace the painstaking manual alignment process, dramatically lowering the barrier to multilingual speech AI,” says Dr. Aiko Tanaka, Lead Scientist at Kyutai.

“From a product perspective, the latency improvements mean we can finally think about real‑time multilingual voice assistants without sacrificing user experience,” notes Marco Silva, VP of Product at a major telecom provider.