Realtime speech-to-speech API for Hindi voice agents

What does the Realtime API do?

It gives you one turn of a voice conversation as a service. Your process sends the caller’s audio up and plays the agent’s audio back. On Mirai’s side, one voice worker runs speech recognition, turn detection, the language model and speech synthesis next to each other. Compared with calling separate speech-to-text, LLM and text-to-speech APIs, each turn makes one round trip to your server instead of three.

A session is a call on the realtime channel. It has a call ID, appears in GET /v2/calls, sends webhooks if you pass webhook_url, and is billed per minute at your plan’s rate: from ₹1 a minute on T1 with a ₹1,00,000 monthly commitment, to ₹5 a minute pay as you go on T5.

Mirai builds its own speech, recognition and language models and offers them as APIs, and this is the one that bundles all three. Building a full phone agent instead? Agents start at ₹1 a minute at volume; see AI calling.

How do you connect and authenticate?

wss://sandbox.voice.miraiminds.co/v2/realtime

Use the sandbox address with a sandbox key, and wss://prod.voice.miraiminds.co/v2/realtime once your company has gone live. Send your workspace key in the Authorization: Bearer sk_live_… header. Clients that can’t set headers can send it as the WebSocket subprotocol openai-insecure-api-key.sk_live_… alongside realtime. Either way, the Realtime API is for servers only.

Query parameter	What it does
model	Accepted and ignored. Every session reports mira-realtime
agent_id	Start from a single-prompt agent: its prompt, tools, voice and language
variables	URL-encoded JSON with values for the agent’s {{placeholders}}
webhook_url	Where the session’s call.* webhooks go, URL-encoded
max_duration_secs	30 to 1800. Without agent_id the default is 300
metadata	URL-encoded JSON, echoed on the call and every webhook, never shown to the model
mirai_events	1 turns on Mirai’s own mirai.* events. Don’t set it with stock Pipecat

A malformed parameter fails the upgrade with 400 invalid_request, and error.param names it.

What happens in a session?

1. You open the socket. Mirai checks your key and balance and starts the call.
2. Mirai sends session.created.
3. You send session.update with your settings and get session.updated. Mirai waits up to 1.5 seconds for this first update. The language and the audio formats can only be set in it.
4. You stream audio with input_audio_buffer.append. Mirai detects when the caller has finished, transcribes, runs the model and streams the reply as response.output_audio.delta events.
5. You close the socket, or the session reaches its maximum duration. The call ends and is billed.

Every event is a JSON text frame, audio included as base64. The default format is 16-bit PCM at 24 kHz. For telephony audio, set audio/pcmu or audio/pcma at 8 kHz in the first update, which saves two resamples. The agent never greets on its own, so send response.create when you want it to speak first.

A minimal Python client

This client sends one typed line instead of microphone audio and saves the spoken reply, which is enough to see the event flow. It uses the websockets library (pip install "websockets==15.0.1"):

import asyncio
import base64
import json
import os
import wave

import websockets

URL = "wss://sandbox.voice.miraiminds.co/v2/realtime"


async def main():
    headers = {"Authorization": f"Bearer {os.environ['MIRAI_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers, max_size=16 * 1024 * 1024) as ws:
        print(json.loads(await ws.recv())["type"])  # session.created

        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "instructions": "You are Mira, a helpful voice assistant. Reply in one or two short sentences.",
                "audio": {
                    "input": {"transcription": {"language": "hi"}},
                    "output": {"voice": "neha"},
                },
            },
        }))
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": "नमस्ते, मेरा order कब तक आएगा?"}],
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        audio = bytearray()
        async for message in ws:
            event = json.loads(message)
            if event["type"] == "response.output_audio.delta":
                audio.extend(base64.b64decode(event["delta"]))
            elif event["type"] == "response.output_audio_transcript.delta":
                print(event["delta"], end="", flush=True)
            elif event["type"] == "error":
                raise RuntimeError(event["error"])
            elif event["type"] == "response.done":
                print("\n", event["response"].get("metadata"))
                break

    with wave.open("reply.wav", "wb") as w:
        w.setnchannels(1)
        w.setsampwidth(2)
        w.setframerate(24000)  # the default output format: 16-bit PCM at 24 kHz
        w.writeframes(bytes(audio))


asyncio.run(main())

We checked this client against the Realtime reference and the websockets 15.0.1 API, and did not run it against a live session. Billing runs until the socket closes, with a one-block minimum of 10 seconds, so close it as soon as the reply is done. For a real conversation, stream microphone or phone audio with input_audio_buffer.append and let Mirai’s turn detection decide when the caller has finished.

On Pipecat you don’t need to write this. MiraiRealtimeLLMService from pipecat-mirai builds the URL, sends Mirai’s settings and handles its events; see the Pipecat guide. Stock OpenAIRealtimeLLMService from Pipecat 1.8.1 also works with base_url set to the Realtime address.

Which settings can you change?

Field	When it applies
instructions	Any time. Replaces the agent’s prompt; {{placeholders}} in it are not filled
tools, tool_choice	Any time. Function tools only
audio.output.voice	Any time. neha, ashu, shruti or sameer
audio.input.turn_detection	Any time. null means you commit turns yourself
max_output_tokens	Any time. 1 to 4096, or "inf"
audio.input.transcription.language	First update only, for example hi or en
audio.input.format, audio.output.format	First update only

Mirai’s own settings go in session.mirai: temperature, top_p, max_tokens, language, tool_timeout_ms and a few more. That block is validated strictly, and an unknown key returns unknown_parameter. Fields Mirai doesn’t apply, such as audio.output.speed or VAD thresholds, are ignored; with mirai_events=1, the mirai.session.applied event lists each ignored field and why.

How do barge-in and tools work?

When the caller starts speaking over the agent, you receive input_audio_buffer.speech_started, and the reply in progress ends with response.done, status cancelled, reason turn_detected. Stop playback, then send conversation.item.truncate with audio_end_ms, how much of the reply the caller actually heard. Mirai trims the agent’s turn to those words, so the model doesn’t think it said things nobody heard. response.cancel stops a reply the same way.

When the model calls one of your tools, you get response.function_call_arguments.done with the call ID, name and arguments. Run the tool, send a conversation.item.create with a function_call_output item for that call_id, then response.create. Mirai waits 10 seconds for a result by default (mirai.tool_timeout_ms, 1 to 30 seconds), after which the model tells the caller the request couldn’t be completed. With agent_id, the agent’s own tools keep running on Mirai’s side.

Every response.done carries per-turn timing in response.metadata, such as mirai_v2v_ms. The latency guide explains what each value measures. If one of Mirai’s own steps is unavailable during a session, that step moves to a third-party fallback at the same price, and the metadata names the provider that served it.

What can go wrong?

If a session can’t start, the WebSocket upgrade fails with an HTTP status and an OpenAI-style error body:

Status	code	What to do
400	invalid_request	Fix the URL parameter named in param
401	missing_api_key, unauthorized	Send a valid key
402	insufficient_balance	Top up the wallet
403	forbidden, production_workspace_required, contract_unavailable	Revoked key, sandbox key on the production address, or a plan that doesn’t cover the session
404	not_found	The agent_id isn’t in your workspace
409	flow_agent_realtime_unsupported	Use a single-prompt agent, or leave out agent_id
429	concurrency_limit	Sessions count towards your concurrent calls; retry after Retry-After (5 seconds)
503	at_capacity, worker_unavailable, fleet_offline, upstream_unavailable	Retry after Retry-After
504	start_timeout	Setup took longer than about 8 seconds, often slow pre-call tools; try again

During a session, a wrong value in session.update returns an error event naming the field, and nothing in that update is applied. Stock Pipecat 1.8.1 treats most error events as fatal and stops reading, so check your settings before you go live. session_expired means the maximum duration was reached. session_interrupted means the voice worker went away; start a new session.

Not supported yet: text-only replies (every reply is spoken), interim transcripts, settings inside response.create, image or audio parts in conversation.item.create, and rate_limits.updated.