Andrii Bidochko

• Updated: June 21, 2026
• 7 min read

Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts

Direct Answer

PlanAudio introduces a unified, autoregressive framework that can generate coherent audio containing speech, environmental sounds, and their seamless composites directly from unrestricted natural‑language prompts. By leveraging a large language model (LLM) as both reasoning engine and acoustic planner, the system eliminates the need for separate pipelines or rigid input schemas, opening a path toward truly conversational audio creation.

Background: Why This Problem Is Hard

Audio generation has traditionally been split into two silos: text‑to‑speech (TTS) systems that excel at producing intelligible speech, and sound synthesis models that focus on non‑verbal effects such as footsteps, rain, or engine noise. When an application demands a scene where a narrator speaks while a door slams, developers must stitch together multiple models, manually align timestamps, and often resort to post‑processing tricks. This fragmented approach suffers from three core limitations:

• Temporal misalignment: Independent generators lack a shared notion of timing, leading to jittery or overlapping audio artifacts.
• Semantic disconnect: TTS models do not understand contextual cues like “the speaker whispers as the wind howls,” so they cannot modulate prosody in response to surrounding sounds.
• Rigid prompting: Existing pipelines require structured inputs (e.g., separate transcripts and sound tags) or external rewriting steps, which defeats the convenience of free‑form natural language.

These challenges matter because modern AI agents—virtual assistants, interactive games, and immersive training simulators—need to convey information through rich, multimodal audio that feels natural and context‑aware. A single, end‑to‑end model that respects both linguistic intent and acoustic composition would dramatically simplify product development and improve user experience.

What the Researchers Propose

The authors present PlanAudio, a unified framework that treats the entire audio generation problem as a single autoregressive sequence prediction task. Instead of feeding a prompt into a dedicated text encoder, PlanAudio feeds the raw prompt directly into a large language model, allowing the model’s intrinsic reasoning abilities to parse intent, plan the audio structure, and drive acoustic synthesis.

Key components of the system include:

• LLM Reasoning Core: A pre‑trained transformer that interprets the free‑form prompt, extracts high‑level semantic goals (e.g., “a calm voice narrating while a distant train passes”), and produces a step‑wise plan.
• Semantic Latent Chain‑of‑Thought (CoT): An implicit planning layer that translates the LLM’s textual plan into a series of latent audio tokens, preserving the logical flow from semantics to sound.
• Unified Acoustic Decoder: A diffusion‑style or autoregressive decoder that consumes the latent chain and renders the final waveform, handling speech synthesis, sound effect generation, and their temporal blending in one pass.

By collapsing the traditional multi‑stage pipeline into a single, LLM‑driven loop, PlanAudio eliminates the need for external text‑to‑speech engines, sound libraries, or hand‑crafted alignment scripts.

How It Works in Practice

The operational flow of PlanAudio can be broken down into three conceptual stages:

1. Prompt Ingestion: The user supplies an unrestricted text prompt, such as “A friendly guide explains the museum exhibit while soft jazz plays in the background.” The raw string is tokenized and fed directly to the LLM.
2. Semantic Planning via Latent CoT: The LLM generates a “chain of thought” in latent space—a sequence of abstract tokens that encode high‑level actions (e.g., start speech, introduce background music, fade out after sentence three). This latent CoT acts as an internal storyboard, bridging meaning and acoustics without explicit textual intermediate steps.
3. Acoustic Decoding: The unified decoder consumes the latent storyboard and produces a continuous audio waveform. Because the decoder is trained on a mixed corpus of speech and environmental sounds, it learns to blend them naturally, respecting timing cues embedded in the latent chain.

What sets this approach apart is the elimination of a separate “planning” module. The LLM’s reasoning is repurposed as the planner, and the latent CoT serves as a lightweight, differentiable interface between language understanding and sound synthesis. This design yields a compact architecture that can be trained end‑to‑end on a single multi‑scenario dataset.

Evaluation & Results

To benchmark the unified capability, the authors built PlanAudio‑Bench, a curated suite of tasks that span three categories:

• Speech‑only: Traditional TTS prompts.
• Sound‑only: Descriptions of ambient or event‑based noises.
• Composite: Mixed prompts requiring simultaneous speech and sound (e.g., “The announcer reads the schedule while a crowd murmurs”).

Evaluation metrics combined objective measures (e.g., mel‑spectrogram distance, word error rate for speech) with human preference studies that asked listeners to rate naturalness, coherence, and compositional fidelity.

Key findings include:

• PlanAudio outperformed the strongest pipeline baseline (separate TTS + sound model + alignment) by an average of 18% in human preference scores for composite scenarios.
• In speech‑only tests, the model remained competitive with dedicated TTS systems, showing less than 5% degradation in intelligibility.
• The semantic latent CoT consistently yielded higher alignment quality than alternative CoT strategies that operated purely in textual space.
• Continuous multi‑scenario training (exposing the model to all three categories throughout training) proved essential; ablations that trained on a single scenario suffered noticeable drops in cross‑scenario performance.

These results demonstrate that a single LLM‑centric model can match or exceed specialized systems while offering the flexibility of free‑form prompting.

Why This Matters for AI Systems and Agents

For developers building conversational agents, immersive simulations, or automated content pipelines, PlanAudio offers several practical advantages:

• Simplified Architecture: One model replaces multiple services, reducing latency, maintenance overhead, and integration complexity.
• Dynamic Contextualization: Agents can adjust prosody and background ambience on the fly, reacting to user actions or environmental changes without re‑rendering separate audio tracks.
• Scalable Personalization: Because the LLM understands nuanced prompts, developers can generate personalized audio experiences (e.g., “Your name is spoken in a calm tone while your favorite song plays softly”).
• Enhanced Evaluation: Unified generation enables end‑to‑end testing of audio quality, making it easier to benchmark against human expectations.

These capabilities align closely with emerging enterprise needs for UBOS platform overview, where AI agents orchestrate multi‑modal interactions. Integrating PlanAudio‑style synthesis could power richer voice‑first workflows in the Workflow automation studio, allowing bots to narrate processes while ambient sounds reinforce brand identity.

Moreover, the model’s compatibility with voice‑cloning services such as ElevenLabs AI voice integration means that enterprises can maintain a consistent vocal persona across all generated content, strengthening brand cohesion.

What Comes Next

While PlanAudio marks a significant step forward, several open challenges remain:

• Fine‑Grained Control: Users may eventually demand precise timing or spatial placement of sounds, which the current latent CoT abstracts away.
• Data Diversity: Expanding the training corpus to cover rare languages, dialects, and niche sound categories will improve global applicability.
• Real‑Time Interaction: Adapting the model for low‑latency, streaming generation is essential for live‑chat agents and interactive gaming.
• Evaluation Standards: Developing standardized benchmarks for compositional audio, beyond human preference surveys, will help the community track progress.

Future research may explore hybrid architectures that combine the latent CoT with explicit temporal graphs, or integrate reinforcement learning to fine‑tune audio alignment based on user feedback.

From a product perspective, organizations can experiment with PlanAudio‑style pipelines within the ChatGPT and Telegram integration to deliver voice‑enhanced notifications, or leverage the Telegram integration on UBOS for real‑time audio alerts in collaborative workspaces.

Developers interested in prototyping these ideas can start by reviewing the arXiv paper and experimenting with the open‑source components released alongside the study.

Ready to bring unified audio generation into your AI products? Visit the UBOS contact page to discuss partnership opportunities and explore how PlanAudio‑inspired solutions can accelerate your roadmap.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.