Hindi and Hinglish TTS in Pipecat with pipecat-mirai.

Install the package and get a key.

uv add "pipecat-mirai==0.5.1"
# or: pip install "pipecat-mirai==0.5.1"

We pin 0.5.1, released on 8 October 2026. The docs still describe 0.5.0; 0.5.1 fixes socket sharing when many worker processes use one workspace. The package is tested on Pipecat 1.8.1 and 1.12.0.

Get a key under Developers in the sandbox. The service reads MIRAI_API_KEY, then MIRA_API_KEY, or takes api_key=.

Put the Mirai service where your TTS was.

Put MiraiTTSService where your TTS service was:

import os
from pipecat_mirai import MiraiTTSService

tts = MiraiTTSService(
    api_key=os.environ["MIRAI_API_KEY"],
    settings=MiraiTTSService.Settings(voice="shruti"),
)

pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

Ask your LLM for Hindi in Devanagari and English words in Latin script, the way people type Hinglish: आपका order कल deliver होगा। Romanised Hindi gets read with English vowels.

Pipecat sends the reply a sentence at a time, and we synthesise the next one while the current one plays. To stream tokens instead, pass text_aggregation_mode=TextAggregationMode.TOKEN (from pipecat.services.tts_service). Our splitter knows the danda (।), “Rs.”, “Dr.” and numbers like 3.5.

To change the voice mid-call:

await task.queue_frame(TTSUpdateSettingsFrame(delta=MiraiTTSService.Settings(voice="sameer")))

One short script checks that it works.

This speaks one line through a real pipeline into hello.wav. No microphone needed.

# /// script
# requires-python = ">=3.11"
# dependencies = ["pipecat-ai==1.8.1", "pipecat-mirai==0.5.1"]
# ///
import asyncio
import os
import wave

from pipecat.frames.frames import EndFrame, TTSAudioRawFrame, TTSSpeakFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.frame_processor import FrameProcessor

from pipecat_mirai import MiraiTTSService

SAMPLE_RATE = 16000


class Capture(FrameProcessor):
    """Keeps the audio the TTS service produces."""

    def __init__(self):
        super().__init__()
        self.audio = bytearray()

    async def process_frame(self, frame, direction):
        await super().process_frame(frame, direction)
        if isinstance(frame, TTSAudioRawFrame):
            self.audio.extend(frame.audio)
        await self.push_frame(frame, direction)


async def main():
    tts = MiraiTTSService(
        api_key=os.environ["MIRAI_API_KEY"],
        settings=MiraiTTSService.Settings(voice="shruti"),
    )
    capture = Capture()
    task = PipelineTask(
        Pipeline([tts, capture]),
        params=PipelineParams(audio_out_sample_rate=SAMPLE_RATE),
    )
    await task.queue_frames([TTSSpeakFrame("नमस्ते, आपका order कल deliver होगा।"), EndFrame()])
    await PipelineRunner(handle_sigint=False).run(task)

    with wave.open("hello.wav", "wb") as w:
        w.setnchannels(1)
        w.setsampwidth(2)
        w.setframerate(SAMPLE_RATE)
        w.writeframes(bytes(capture.audio))
    print(f"hello.wav: {len(capture.audio) / 2 / SAMPLE_RATE:.2f} s at {SAMPLE_RATE} Hz")


asyncio.run(main())

Save it as say_hello.py and run uv run say_hello.py. On 16 October 2026 it wrote 2.56 seconds of 16 kHz audio for us, for a few paise.

Two warnings are normal. Pipecat 1.8.1 calls PipelineTask and PipelineRunner deprecated; the package’s examples/foundational/01-say-hello.py uses PipelineWorker and WorkerRunner. And with no output transport, the service says it set no output lead. A phone bot gets one automatically.

The service handles interruptions and dropped sockets for you.

One WebSocket opens when the pipeline starts and lasts the whole call. On it, the service:

• asks for audio at your pipeline’s rate, so an 8 kHz phone call pulls 128 kbit/s, not 768 kbit/s at 48 kHz,
• cancels the reply when the caller interrupts, and the cut sentence isn’t charged,
• waits and resends if we’re briefly at capacity (up to 5 seconds),
• reopens a dropped socket and resumes from the first sentence the caller hadn’t heard,
• buffers 150 ms before each sentence, so playback can’t start and then stall,
• keeps quiet sockets alive; we close one after 120 seconds with no traffic.

Sockets open on our edge, next to the speech servers. If the edge doesn’t answer within 3 seconds, they open on the API. If WebSockets are blocked, the service falls back to HTTP and logs it once. edge=False skips the edge, and MiraiHttpTTSService is HTTP only.

On a phone line, run at 8 kHz.

For a phone bot on a websocket transport (Twilio, Plivo, Exotel or Telnyx serializers), run at 8 kHz and keep the defaults:

import os
from pipecat.pipeline.task import PipelineParams
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams, FastAPIWebsocketTransport
from pipecat_mirai import MiraiTTSService

transport = FastAPIWebsocketTransport(websocket, FastAPIWebsocketParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    add_wav_header=False,
    serializer=serializer,  # TwilioFrameSerializer, PlivoFrameSerializer, ...
))

tts = MiraiTTSService(
    api_key=os.environ["MIRAI_API_KEY"],
    settings=MiraiTTSService.Settings(voice="shruti"),
)

params = PipelineParams(audio_in_sample_rate=8000, audio_out_sample_rate=8000)

Pipecat normally feeds the line at exactly real time, so the provider holds about 40 ms. A longer stall on your server breaks the voice up. Our services let the transport run 0.4 seconds ahead.

On an interruption, Pipecat still clears the provider’s queue, so at most 0.4 seconds of audio is wasted. The catch: “bot stopped speaking” can fire up to 0.4 seconds before the caller stops hearing it.

Starting many calls at once in one process? Call await prewarm(websocket=N) at startup, in the event loop that runs your pipelines.

Or let us run the whole turn.

MiraiRealtimeLLMService talks to the Realtime API and replaces your STT, LLM and TTS. Your transport and context aggregators stay.

from pipecat_mirai import MiraiRealtimeLLMService

llm = MiraiRealtimeLLMService(voice="shruti", language="hi")

pipeline = Pipeline([transport.input(), context_aggregator.user(), llm, transport.output(), context_aggregator.assistant()])

To start from one of your agents, pass agent_id and variables. Each reply’s timing arrives through an event handler:

llm = MiraiRealtimeLLMService(agent_id="agt_...", variables={"customer_name": "Rahul"})

@llm.event_handler("on_turn_metrics")
async def on_turn_metrics(service, m):
    print(f"{m.v2v_ms} ms voice to voice")

It uses the sandbox by default; after go-live, pass base_url=PRODUCTION_REALTIME_URL from pipecat_mirai. Caller audio is resampled to 24 kHz, so 8 kHz pipelines just work. Full phone agents cost ₹1 a minute on our volume plan; see AI calling.

These are the common problems and their fixes.

• Set MIRAI_API_KEY or pass api_key: no key in the environment.
• at_capacity, or slow first audio with many bots talking: your workspace allows 10 sentences in flight. Tell us before a load test and we’ll raise it.
• Voice too fast, too slow or noisy: set audio_out_sample_rate to the line’s rate.
• Gaps on phone calls under load: run at 8 kHz on the defaults, on the current version.
• A stock OpenAIRealtimeLLMService session goes quiet: mirai_events is on, and stock Pipecat stops at events it doesn’t know. Use MiraiRealtimeLLMService.
• 403 production_workspace_required: your workspace hasn’t gone live yet. Keep the sandbox default.

If audio breaks up only on real calls, see the docs’ choppy-audio table and the latency guide.