Hindi and Hinglish TTS in Pipecat with pipecat-mirai

How do you install it?

uv add "pipecat-mirai==0.5.1"
# or: pip install "pipecat-mirai==0.5.1"

This guide pins 0.5.1, released on 8 October 2026. The Mirai docs page describes 0.5.0, and 0.5.1 fixes how waiting sockets are shared when many worker processes use one workspace. You need Python 3.11 or newer and Pipecat 1.8.1 or newer, below 2. The package’s own tests cover Pipecat 1.8.1 and 1.12.0.

Get an API key from the sandbox under Developers; no access code is needed. The service reads MIRAI_API_KEY, then MIRA_API_KEY, or takes api_key=.

Mirai builds its own speech, recognition and language models and offers them as APIs, with voice agents built on the same stack. Building a full phone agent instead? Agents start at ₹1 a minute at volume; see AI calling.

How do you add it to a pipeline?

Use MiraiTTSService in place of your current TTS service:

import os
from pipecat_mirai import MiraiTTSService

tts = MiraiTTSService(
    api_key=os.environ["MIRAI_API_KEY"],
    settings=MiraiTTSService.Settings(voice="shruti"),
)

pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

Prompt your LLM to reply with Hindi in Devanagari and English words in Latin script, the way people type Hinglish: आपका order कल deliver होगा। Romanised Hindi is read with English vowels.

By default Pipecat sends the LLM’s reply a sentence at a time, and each sentence goes to Mirai as soon as it is complete. Mirai synthesises the next sentence while the current one plays. To send every token as it arrives and let Mirai find the sentence ends, pass text_aggregation_mode=TextAggregationMode.TOKEN (from pipecat.services.tts_service). Mirai’s sentence splitter knows the danda (।), abbreviations such as “Rs.” and “Dr.”, and numbers like 3.5.

To change the voice mid-call:

await task.queue_frame(TTSUpdateSettingsFrame(delta=MiraiTTSService.Settings(voice="sameer")))

How do you check that it works?

This script speaks one line through a real Pipecat pipeline and saves hello.wav. It has no transport, STT or LLM, so it needs no microphone.

# /// script
# requires-python = ">=3.11"
# dependencies = ["pipecat-ai==1.8.1", "pipecat-mirai==0.5.1"]
# ///
import asyncio
import os
import wave

from pipecat.frames.frames import EndFrame, TTSAudioRawFrame, TTSSpeakFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.frame_processor import FrameProcessor

from pipecat_mirai import MiraiTTSService

SAMPLE_RATE = 16000


class Capture(FrameProcessor):
    """Keeps the audio the TTS service produces."""

    def __init__(self):
        super().__init__()
        self.audio = bytearray()

    async def process_frame(self, frame, direction):
        await super().process_frame(frame, direction)
        if isinstance(frame, TTSAudioRawFrame):
            self.audio.extend(frame.audio)
        await self.push_frame(frame, direction)


async def main():
    tts = MiraiTTSService(
        api_key=os.environ["MIRAI_API_KEY"],
        settings=MiraiTTSService.Settings(voice="shruti"),
    )
    capture = Capture()
    task = PipelineTask(
        Pipeline([tts, capture]),
        params=PipelineParams(audio_out_sample_rate=SAMPLE_RATE),
    )
    await task.queue_frames([TTSSpeakFrame("नमस्ते, आपका order कल deliver होगा।"), EndFrame()])
    await PipelineRunner(handle_sigint=False).run(task)

    with wave.open("hello.wav", "wb") as w:
        w.setnchannels(1)
        w.setsampwidth(2)
        w.setframerate(SAMPLE_RATE)
        w.writeframes(bytes(capture.audio))
    print(f"hello.wav: {len(capture.audio) / 2 / SAMPLE_RATE:.2f} s at {SAMPLE_RATE} Hz")


asyncio.run(main())

Save it as say_hello.py and run uv run say_hello.py. We ran it against the sandbox on 16 October 2026 and it wrote 2.56 seconds of 16 kHz audio. It costs a few paise.

Two log lines are expected. Pipecat 1.8.1 marks PipelineTask and PipelineRunner as deprecated in favour of PipelineWorker and WorkerRunner, which the package’s own examples/foundational/01-say-hello.py uses. And because this pipeline has no output transport, the service warns that it set no output lead; in a phone bot it sets one for you.

What does the service handle for you?

MiraiTTSService opens one WebSocket when the pipeline starts and carries the whole call over it. From the docs and README:

• It asks Mirai for audio at your pipeline’s output rate, so an 8 kHz phone pipeline downloads 128 kbit/s per call instead of 768 kbit/s at 48 kHz, and nothing is resampled on your side.
• When the caller interrupts, it cancels the reply on Mirai, drops audio still arriving, and you are not charged for the cut sentence.
• If Mirai is briefly at capacity and says when to retry (up to 5 seconds), it waits and resends.
• It reopens a dropped socket at once and resends the reply from the first sentence the caller hadn’t started hearing.
• It holds 150 ms of audio before each sentence starts, so a short first chunk can’t start playback and then stall.
• It keeps a quiet socket open through long pauses; Mirai closes a socket after 120 seconds with no traffic.

Sockets open on Mirai’s edge, next to the speech servers, with the key in a header. If the edge can’t be reached within 3 seconds, the socket opens on the API instead, and if WebSockets can’t get through at all, the service speaks the call over HTTP and logs it once. edge=False keeps every socket on the API. If you want HTTP and nothing else, use MiraiHttpTTSService.

What is the recommended phone setup?

For a phone bot on a websocket transport (Twilio, Plivo, Exotel or Telnyx serializers), run the pipeline at 8 kHz and keep the service’s defaults:

import os
from pipecat.pipeline.task import PipelineParams
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams, FastAPIWebsocketTransport
from pipecat_mirai import MiraiTTSService

transport = FastAPIWebsocketTransport(websocket, FastAPIWebsocketParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    add_wav_header=False,
    serializer=serializer,  # TwilioFrameSerializer, PlivoFrameSerializer, ...
))

tts = MiraiTTSService(
    api_key=os.environ["MIRAI_API_KEY"],
    settings=MiraiTTSService.Settings(voice="shruti"),
)

params = PipelineParams(audio_in_sample_rate=8000, audio_out_sample_rate=8000)

Pipecat normally sends audio to the phone line at exactly real time, so the provider holds about 40 ms. A stall on your server longer than that breaks the voice up. The Mirai services let the transport run 0.4 seconds ahead, applied when the pipeline starts. When the caller interrupts, Pipecat still clears the provider’s queue, so at most 0.4 seconds of sent audio is thrown away. One side effect: Pipecat’s “bot stopped speaking” event fires up to 0.4 seconds before the caller stops hearing the bot.

If many calls start at once in one process, call await prewarm(websocket=N) once at startup, in the event loop that runs the pipelines, so the first burst finds sockets already open.

Can Mirai run the whole turn instead?

Yes. MiraiRealtimeLLMService connects to the Realtime API and replaces your STT, LLM and TTS services. Your transport and context aggregators stay:

from pipecat_mirai import MiraiRealtimeLLMService

llm = MiraiRealtimeLLMService(voice="shruti", language="hi")

pipeline = Pipeline([transport.input(), context_aggregator.user(), llm, transport.output(), context_aggregator.assistant()])

To start from one of your agents, pass agent_id and variables. Each reply’s timing arrives through an event handler:

llm = MiraiRealtimeLLMService(agent_id="agt_...", variables={"customer_name": "Rahul"})

@llm.event_handler("on_turn_metrics")
async def on_turn_metrics(service, m):
    print(f"{m.v2v_ms} ms voice to voice")

The service connects to the sandbox by default. Once your company has gone live, pass base_url=PRODUCTION_REALTIME_URL, imported from pipecat_mirai. It resamples caller audio to the protocol’s 24 kHz, so an 8 kHz phone pipeline works unchanged. A Realtime session is billed per minute at your plan’s rate, like a call, not per character.

What can go wrong?

Symptom	Likely cause	Fix
Set MIRAI_API_KEY or pass api_key	No key in the environment	Export MIRAI_API_KEY or pass api_key=
at_capacity errors, or slow first bytes when many bots speak	Your workspace’s limit on sentences in flight, 10 by default	Tell Mirai how many bots will speak at once before a load test; the limit is raised on their side
Voice too fast, too slow or noisy	The pipeline’s rate differs from what the player or transport expects	Set audio_out_sample_rate to the line’s rate; the service asks Mirai for that rate
Gaps on phone calls under load	48 kHz audio over a narrow link, or an old version without the output lead	Run the pipeline at 8 kHz and keep the service defaults
A stock OpenAIRealtimeLLMService session goes silent	mirai_events was turned on, and stock Pipecat stops reading on events it doesn’t know	Use MiraiRealtimeLLMService, which handles them
403 production_workspace_required on the Realtime service	A key from a workspace that hasn’t gone live, sent to the production URL	Keep the default sandbox URL until your company has gone live

For audio that breaks up only on real calls, the docs’ choppy-audio table matches symptoms to fixes, and the latency guide shows how to split the delay into parts.