Realtime speech-to-speech API for Hindi voice agents.

Each turn is one round trip to our voice worker.

Your server sends the caller’s audio up and plays the reply. On our side, one voice worker runs recognition, turn detection, the model and the voice together, so each turn is one round trip, not three.

A session is a call on the realtime channel, with a call ID, a row in GET /v2/calls and webhooks if you pass webhook_url. It bills like one too: ₹1 a minute on our volume plan, ₹5 pay as you go.

Connect with your key over a WebSocket.

wss://sandbox.voice.miraiminds.co/v2/realtime

Switch to wss://prod.voice.miraiminds.co/v2/realtime once your company has gone live. Send Authorization: Bearer sk_live_…, or, if your client can’t set headers, the subprotocol openai-insecure-api-key.sk_live_… alongside realtime. Servers only.

Query parameter	What it does
model	Ignored; every session reports mira-realtime
agent_id	Start from a single-prompt agent
variables	URL-encoded JSON for the agent’s {{placeholders}}
webhook_url	Where call.* webhooks go, URL-encoded
max_duration_secs	30 to 1800; 300 without agent_id
metadata	URL-encoded JSON, echoed on every webhook, never shown to the model
mirai_events	1 adds our mirai.* events. Not with stock Pipecat

Here is how a session goes from start to end.

1. You open the socket. We check your key and balance, start the call and send session.created.
2. You send session.update within 1.5 seconds and get session.updated. Language and audio formats can only be set in this first one.
3. You stream audio with input_audio_buffer.append. We spot when the caller stops and stream the reply as response.output_audio.delta.
4. You close the socket, or the time limit hits. The call ends and is billed.

Events are JSON text frames, audio in base64, 16-bit PCM at 24 kHz by default. For phone audio, ask for audio/pcmu or audio/pcma at 8 kHz and skip two resamples. The agent won’t greet on its own; send response.create if you want it to.

This Python client sends one line and saves the reply.

This sends one typed line and saves the spoken reply. It needs pip install "websockets==15.0.1".

import asyncio
import base64
import json
import os
import wave

import websockets

URL = "wss://sandbox.voice.miraiminds.co/v2/realtime"


async def main():
    headers = {"Authorization": f"Bearer {os.environ['MIRAI_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers, max_size=16 * 1024 * 1024) as ws:
        print(json.loads(await ws.recv())["type"])  # session.created

        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "instructions": "You are Mira, a helpful voice assistant. Reply in one or two short sentences.",
                "audio": {
                    "input": {"transcription": {"language": "hi"}},
                    "output": {"voice": "neha"},
                },
            },
        }))
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": "नमस्ते, मेरा order कब तक आएगा?"}],
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        audio = bytearray()
        async for message in ws:
            event = json.loads(message)
            if event["type"] == "response.output_audio.delta":
                audio.extend(base64.b64decode(event["delta"]))
            elif event["type"] == "response.output_audio_transcript.delta":
                print(event["delta"], end="", flush=True)
            elif event["type"] == "error":
                raise RuntimeError(event["error"])
            elif event["type"] == "response.done":
                print("\n", event["response"].get("metadata"))
                break

    with wave.open("reply.wav", "wb") as w:
        w.setnchannels(1)
        w.setsampwidth(2)
        w.setframerate(24000)  # the default output format: 16-bit PCM at 24 kHz
        w.writeframes(bytes(audio))


asyncio.run(main())

We checked this against the reference and the websockets 15.0.1 API, but didn’t run it live. Billing runs until the socket closes, 10 seconds minimum, so close it when the reply ends.

On Pipecat, MiraiRealtimeLLMService in pipecat-mirai does all this for you; see the Pipecat guide. Stock OpenAIRealtimeLLMService from Pipecat 1.8.1 works too, with base_url set to the address above.

You can change a few of our own settings.

Field	When it applies
instructions	Any time. Replaces the prompt; {{placeholders}} aren’t filled
tools, tool_choice	Any time. Function tools only
audio.output.voice	Any time. neha, ashu, shruti or sameer
audio.input.turn_detection	Any time. null means you commit turns yourself
max_output_tokens	Any time. 1 to 4096, or "inf"
audio.input.transcription.language	First update only, like hi or en
audio.input.format, audio.output.format	First update only

Our own settings go in session.mirai, such as temperature, top_p, max_tokens, language and tool_timeout_ms. Unknown keys there return unknown_parameter. Fields we don’t apply, like audio.output.speed or VAD thresholds, are ignored; mirai.session.applied lists them when mirai_events=1.

Barge-in and tool calls each have their own events.

When the caller talks over the agent, you get input_audio_buffer.speech_started, and the reply ends with response.done, status cancelled, reason turn_detected. Stop playback and send conversation.item.truncate with audio_end_ms, how much the caller heard. We trim the agent’s turn to what was really heard. response.cancel stops a reply the same way.

For a tool call you get response.function_call_arguments.done. Run the tool, send a conversation.item.create with a function_call_output for that call_id, then response.create. We wait 10 seconds by default (mirai.tool_timeout_ms, 1 to 30 seconds), then the model tells the caller it couldn’t finish. With agent_id, the agent’s own tools run on our side.

Each response.done carries timing in response.metadata, like mirai_v2v_ms; the latency guide explains it. If one of our steps goes down mid-session, a third-party fallback takes over at the same price, and the metadata names it.

Each error tells you why a session failed.

If a session can’t start, you’ll see one of these:

• 400 invalid_request: fix the URL parameter named in param.
• 401 missing_api_key or unauthorized: send a valid key. 402 insufficient_balance: top up.
• 403 forbidden, production_workspace_required or contract_unavailable: revoked key, sandbox key on production, or a plan that doesn’t cover sessions.
• 404 not_found: that agent_id isn’t yours. 409 flow_agent_realtime_unsupported: use a single-prompt agent.
• 429 concurrency_limit: sessions count as concurrent calls. Retry after Retry-After (5 seconds).
• 503 at_capacity, worker_unavailable, fleet_offline or upstream_unavailable: retry after Retry-After.
• 504 start_timeout: setup took over about 8 seconds, often slow pre-call tools. Try again.

Mid-session, a bad value in session.update returns an error event naming the field, and none of that update applies. Stock Pipecat 1.8.1 stops on most error events, so test your settings first. session_expired means the time limit hit. session_interrupted means the voice worker went away; start a new session.

Not there yet: text-only replies, interim transcripts, settings inside response.create, image or audio parts in conversation.item.create, and rate_limits.updated.