Glossary · Models & serving
What is a speech-to-speech or realtime API?
Speech-to-speech and realtime APIs
A speech-to-speech API takes the caller's voice in and sends the agent's spoken reply back, on one live connection. You stream audio in and play audio out; listening, thinking and speaking all happen on the provider's side.
On a call
Your Pipecat bot on a phone line streams the caller's "मुझे अपना plan बदलना है" to a realtime API. A moment later the reply comes back as audio, "ज़रूर, आप कौन सा plan लेना चाहेंगे?", and your code just plays it.
AI agents from ₹1 a minute.¹
Explained
Here is how it works, in plain words.
Building a voice agent the classic way means three APIs. Speech to text turns the caller’s audio into words, a language model writes the reply, and text to speech turns it back into audio. Your server sits in the middle of every turn, passing data between them.
A speech-to-speech API collapses that into one socket. You stream the caller’s audio up and play whatever audio comes back. Turn detection, interruptions and the hand-offs between steps happen on the provider’s side.
There are two kinds behind the same idea. Some are native models that take audio in and produce audio out, without writing text in between. Others are cascades, the classic three steps run side by side by one provider and exposed as one connection. Both can feel fast. They differ in what you can see and control.
A cascade keeps a transcript you can read, a prompt you can edit, and a voice you can choose. Each step can be timed on its own, so when a turn is slow you know which step to blame. A native model can carry tone and emotion across more directly, and it can be harder to inspect.
OpenAI’s Realtime API has become a common protocol for this, and frameworks like Pipecat ship clients for it. A provider that speaks the same protocol lets you switch with a new URL and key, keeping your pipeline as it is.
Things to check before you commit: which languages the turn detector was trained on, whether you can bring your own prompt and tools, how interruptions trim the agent’s memory, and how the session is billed. Per-token billing on a chatty call can surprise you.
In Voice Infra
This is what Voice Infra does about it.
Our Realtime API speaks OpenAI's Realtime protocol, so a client built for it only needs a new key and URL. Recognition, turn detection, the model and the voice run next to each other on one of our voice workers, and a session shows up in your call list like any other call. It's billed per minute in the same small blocks as a phone call, with no charge per token.
Questions
People often ask us these questions.
Is Mirai's Realtime API a native speech-to-speech model?
No. It runs speech recognition, turn detection, a language model and speech synthesis as separate steps, next to each other on one worker. You get one connection and one round trip per turn, and each step can be measured.
Still deciding? Ask us.
Fifteen minutes with the team that builds it. Bring your script, or your hardest call.
Book 15 minutes