Andrii Bidochko

• Updated: March 23, 2026
• 8 min read

Building an Audio‑First AI Agent with OpenClaw: Speech‑to‑Text and Text‑to‑Speech Integration

1. Introduction

Audio‑first AI agents are reshaping how users interact with software—think voice assistants, call‑center bots, and hands‑free productivity tools. OpenClaw is an open‑source framework that simplifies the orchestration of LLMs, tool calls, and custom logic. By pairing it with state‑of‑the‑art speech models, developers can create a seamless voice experience without writing a monolithic backend.

In this guide we walk through every step a developer needs to:

• Provision a development environment.
• Deploy OpenClaw locally and on UBOS.
• Integrate Whisper for real‑time speech‑to‑text.
• Choose and configure a text‑to‑speech engine (OpenAI TTS or Coqui).
• Orchestrate the conversation loop.
• Test and ship the audio‑first agent.

Whether you are a solo AI enthusiast or a product manager overseeing a team, the tutorial is written in a step‑by‑step style that you can copy‑paste into your own repo.

2. Prerequisites

Before you start, make sure you have the following tools and accounts ready:

3. Setting up OpenClaw

OpenClaw ships as a Docker image that exposes a RESTful endpoint for LLM calls. Follow these steps to get it running locally:

3.1 Clone the repository

git clone https://github.com/UBOS-OpenClaw/openclaw.git
cd openclaw

3.2 Create a .env file

Store your OpenAI key and any other secrets in a .env file at the project root:

OPENAI_API_KEY=sk-****************
COQUI_API_TOKEN=your_coqui_token
WHISPER_MODEL=base

3.3 Build and run the container

docker compose up -d --build

OpenClaw will be reachable at http://localhost:8000/v1/chat/completions. You can verify the health endpoint:

curl http://localhost:8000/health

For a deeper dive into OpenClaw’s architecture, explore the UBOS platform overview, which explains how the framework integrates with UBOS’s workflow automation studio.

4. Integrating Whisper for Speech‑to‑Text

Whisper is OpenAI’s open‑source speech recognition model. It can run locally (CPU/GPU) or as an API. Below we show the local integration, which keeps latency low for real‑time voice agents.

4.1 Install Whisper Python package

pip install git+https://github.com/openai/whisper.git
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118

4.2 Create a transcription service

Save the following as whisper_service.py inside the OpenClaw repo:

import whisper
import io
from fastapi import FastAPI, File, UploadFile
from pydantic import BaseModel

app = FastAPI()
model = whisper.load_model("base")  # change to "large" for higher accuracy

class TranscriptionResponse(BaseModel):
    text: str
    language: str

@app.post("/transcribe", response_model=TranscriptionResponse)
async def transcribe(file: UploadFile = File(...)):
    audio_bytes = await file.read()
    audio = io.BytesIO(audio_bytes)
    result = model.transcribe(audio)
    return TranscriptionResponse(text=result["text"], language=result["language"])

4.3 Add the service to Docker Compose

services:
  whisper:
    build: .
    command: uvicorn whisper_service:app --host 0.0.0.0 --port 8001
    ports:
      - "8001:8001"
    volumes:
      - .:/app
    environment:
      - WHISPER_MODEL=base

Now the transcription endpoint is available at http://localhost:8001/transcribe. You can test it with a short audio clip:

curl -X POST -F "file=@sample.wav" http://localhost:8001/transcribe

For a quick visual of Whisper’s capabilities, see the official Whisper GitHub repository.

5. Integrating Text‑to‑Speech (OpenAI TTS or Coqui)

After the LLM generates a textual response, you need to convert it back to audio. Both OpenAI TTS (cloud) and Coqui TTS (self‑hosted) are supported. Choose one based on latency, cost, and licensing.

5.1 OpenAI Text‑to‑Speech (cloud)

OpenAI’s TTS endpoint accepts plain text and returns an MP3 stream.

import requests

def openai_tts(text, voice="alloy"):
    url = "https://api.openai.com/v1/audio/speech"
    headers = {"Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}"}
    payload = {
        "model": "tts-1",
        "voice": voice,
        "input": text
    }
    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()
    return response.content  # binary MP3 data

5.2 Coqui TTS (self‑hosted)

Coqui provides a Docker image that can be run locally. It gives you full control over voice models.

# Pull the official Coqui TTS image
docker pull ghcr.io/coqui-ai/tts:latest

# Run the container
docker run -d -p 5002:5002 ghcr.io/coqui-ai/tts:latest

Once the service is up, call its REST endpoint:

def coqui_tts(text, speaker="en_uk"):
    url = "http://localhost:5002/api/tts"
    payload = {"text": text, "speaker": speaker}
    resp = requests.post(url, json=payload)
    resp.raise_for_status()
    return resp.content  # WAV bytes

Both functions return raw audio bytes that you can stream back to the client or store temporarily for playback.

6. Orchestrating the Audio‑First Agent

The core loop consists of four stages:

1. Capture audio from the user (microphone or phone line).
2. Transcribe the audio with Whisper.
3. Generate a response using OpenClaw (which forwards the text to an LLM).
4. Synthesize speech with the chosen TTS engine and send it back.

Below is a minimal FastAPI orchestrator that ties everything together:

from fastapi import FastAPI, File, UploadFile
import requests, os, json

app = FastAPI()

WHISPER_URL = "http://localhost:8001/transcribe"
OPENCLAW_URL = "http://localhost:8000/v1/chat/completions"
USE_OPENAI_TTS = True   # toggle between OpenAI and Coqui

def synthesize(text):
    if USE_OPENAI_TTS:
        return openai_tts(text)
    else:
        return coqui_tts(text)

@app.post("/voice-chat")
async def voice_chat(audio: UploadFile = File(...)):
    # 1️⃣ Transcribe
    trans_resp = requests.post(WHISPER_URL, files={"file": (audio.filename, await audio.read())})
    trans_resp.raise_for_status()
    user_text = trans_resp.json()["text"]

    # 2️⃣ LLM response via OpenClaw
    payload = {
        "model": "gpt-4o-mini",
        "messages": [{"role": "user", "content": user_text}]
    }
    llm_resp = requests.post(OPENCLAW_URL, json=payload, headers={"Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}"})
    llm_resp.raise_for_status()
    assistant_text = llm_resp.json()["choices"][0]["message"]["content"]

    # 3️⃣ Synthesize
    audio_bytes = synthesize(assistant_text)

    # 4️⃣ Return MP3/WAV
    return Response(content=audio_bytes, media_type="audio/mpeg")

This endpoint can be called from a web front‑end, a mobile app, or even a Telegram bot. For a ready‑made Telegram integration, see the Telegram integration on UBOS and the ChatGPT and Telegram integration examples.

7. Testing the End‑to‑End Flow

Automated testing ensures that each component works in isolation and together. Follow this checklist:

• Unit test Whisper: feed a known WAV file and assert the returned text matches the transcript.
• Mock OpenClaw: use responses library to simulate LLM replies.
• Validate TTS output: check that the audio byte length is > 0 and that the MIME type matches expectations.
• Integration test the orchestrator: spin up Docker Compose, send a multipart request to /voice-chat, and verify the final MP3 can be played.

Here’s a quick pytest example for the Whisper service:

def test_whisper_transcribe():
    with open("tests/fixtures/hello.wav", "rb") as f:
        resp = client.post("/transcribe", files={"file": ("hello.wav", f.read())})
    assert resp.status_code == 200
    assert "hello" in resp.json()["text"].lower()

8. Deployment on UBOS

UBOS provides a one‑click “host‑OpenClaw” service that abstracts away the underlying Kubernetes cluster. Deploying your audio‑first agent on UBOS gives you:

• Automatic SSL certificates.
• Scalable load‑balancing for Whisper and TTS services.
• Built‑in monitoring via the UBOS dashboard.
• Access to the Workflow automation studio for chaining additional business logic.

8.1 Create a new project on UBOS

Log in to the UBOS homepage, navigate to “Projects”, and click “Create New”. Choose “Docker Compose” as the runtime, then paste the docker-compose.yml you used locally.

8.2 Configure environment variables

In the UBOS UI, add the same .env variables you used locally (API keys, model names, etc.). UBOS encrypts them at rest.

8.3 Deploy and verify

Press “Deploy”. UBOS will spin up the containers, expose them under a sub‑domain (e.g., audio-agent.mycompany.ubos.io), and provide health‑check URLs. Test the public /voice-chat endpoint with a tool like Postman.

For pricing details, review the UBOS pricing plans. If you’re a startup, the UBOS for startups tier offers generous free credits.

9. Conclusion

By combining OpenClaw, Whisper, and a modern TTS engine, you can launch a production‑grade audio‑first AI agent in under an hour. The modular architecture lets you swap out components (e.g., replace Whisper with a commercial ASR service) without rewriting the orchestration layer. Hosting on UBOS adds reliability, scaling, and a low‑friction path to enterprise adoption.

Ready to experiment? Start with the UBOS OpenClaw hosting page, clone the repo, and follow the steps above. Your voice‑enabled AI assistant is just a few commands away.

10. Further Resources

Explore more UBOS capabilities that complement audio agents:

• AI marketing agents – automate campaign copy with voice.
• Web app editor on UBOS – build a UI for your voice bot.
• UBOS partner program – get co‑selling and technical support.
• UBOS templates for quick start – jump‑start new projects.
• UBOS portfolio examples – see real‑world deployments.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.