Gradium: How to Reduce AI Voice Latency for Live Apps

⏱ 9 Reading Time

Gradium cuts AI voice latency for live apps by combining a streaming Delayed Streams Modeling (DSM) architecture, CUDA Graph-optimized inference, and adjustable audio codebook counts, reaching a 155ms P50 Time-To-First-Audio (TTFA) on the independent Coval benchmark.

What Is Gradium?

Gradium is a voice AI company that builds ultra-low-latency text-to-speech (TTS), speech-to-text (STT), voice cloning, and live translation models delivered through a streaming API. Gradium extends its infrastructure to developers building voice agents, customer service bots, and real-time conversational apps that require sub-300-millisecond response times.

Gradium runs its models on NVIDIA GPUs ranging from the L4 to the H100 and holds NVIDIA Inception program membership, which grants early access to CUDA tooling used to optimize inference speed.

Why Does AI Voice Latency Matter for Live Apps?

Latency above 300 milliseconds breaks the illusion of real-time conversation in voice agents, causing users to talk over the AI or abandon the interaction. Two metrics define voice latency: Time To First Audio (TTFA), the delay between a request and the first audio packet, and Real-Time Factor (RTF), the speed of audio generation relative to playback speed.

An RTF of 2x generates one second of audio in 0.5 seconds. Gradium’s documentation states that an RTF above 1x prevents audio skipping during playback, and a TTFA target of 300 milliseconds keeps voice agents responsive during live calls, according to Gradium’s engineering blog on quality-versus-latency optimization.

How Do You Measure AI Voice Latency Before Optimizing?

Measure TTFA and RTF separately using a load-testing harness that logs the timestamp of the first audio byte and the total generation time against audio duration. Isolating these two metrics identifies whether a bottleneck sits in connection setup, model inference, or network transport.

Gradium built an internal benchmark harness that stress-tests its models under concurrent load and reports TTFA at the P50, P90, and max percentiles rather than a single average, since averages hide tail latency spikes that degrade the experience for a subset of live users.

How Do You Reduce Time to First Audio With Gradium’s Streaming Architecture?

Gradium’s models use Delayed Streams Modeling (DSM), an architecture that generates audio one word at a time as text arrives, instead of waiting for a complete sentence. This eliminates the buffering delay that occurs when a TTS model must receive full text input before starting synthesis.

DSM pairs a temporal transformer backbone with a smaller depth transformer that generates each audio codebook, an architecture introduced in the Moshi research paper and extended in Gradium’s implementation. Because a text LLM streaming its output one token at a time feeds directly into this pipeline, the combined LLM-to-voice latency drops to the sum of each component’s per-token delay instead of a full-response wait.

How Do You Enable CUDA Graphs to Cut Kernel Launch Overhead?

Enable CUDA Graph execution to reduce GPU kernel scheduling overhead, which Gradium’s internal benchmark shows cuts P50 TTFA from 562.4ms to 220.7ms at a batch size of 8 on an NVIDIA RTX 4080 Super. Small audio models spend a disproportionate share of inference time on kernel launch overhead rather than computation, since each model runs hundreds of millions to a few billion parameters instead of the hundreds of billions found in frontier text LLMs.

A CUDA Graph records every kernel in a forward pass once, then replays the full sequence on each new request without re-issuing individual launch calls. Gradium’s benchmark data confirms the effect scales with batch size:

Source: Gradium’s blog post, “Optimizing Quality vs. Latency in Real-Time Text-to-Speech AI Models.”

How Do You Choose the Right Number of Audio Codebooks?

Reduce the number of Residual Vector Quantization (RVQ) codebooks generated per audio frame to lower latency at the cost of audio fidelity, since each 80-millisecond time-slice can use between 8 and 32 codebooks. The first codebook encodes the primary signal, and each additional codebook encodes progressively finer residual detail, so cutting codebook count trims computation without discarding the core waveform.

At a batch size of 8, Gradium’s benchmark recorded a P90 TTFA of 160.3ms with 8 codebooks versus 228.4ms with 32 codebooks, a 68.1-millisecond reduction. Voice agents answering short customer service queries can drop to 8 or 16 codebooks; applications requiring studio-grade narration should keep 24 to 32 codebooks and accept the added latency.

How Do You Integrate Gradium Into a Live Voice Agent Stack?

Connect to Gradium’s WebSocket streaming API using the Python or Rust SDK, or route audio through a pre-built LiveKit or Pipecat integration to avoid writing custom transport code. WebSocket connections keep a persistent bidirectional channel open, removing the HTTP handshake overhead that a REST API would add on every audio chunk.

Developers building on LiveKit or Pipecat configure Gradium as the TTS and STT provider inside those frameworks’ existing agent pipelines, which handle turn-taking, interruption detection, and audio buffering around Gradium’s low-latency model calls. Gradium also offers Phonon, a 100-million-parameter on-device TTS model that runs on CPU across Android, iOS, and browser targets with no network round-trip, eliminating network latency entirely for offline or edge deployments.

How Much Does Gradium Cost for Latency-Sensitive Apps?

Gradium prices access in 6 tiers: Free at $0/month, XS at $13/month, S at $43/month, M at $340/month, L at $1,615/month, and a custom Enterprise tier, with all tiers billed on shared TTS, STT, and translation credits. Pricing figures are published on Gradium’s official pricing page (gradium.ai/pricing).

The Free plan includes 45,000 credits, converting to roughly 1 hour of TTS audio or 4 hours of STT transcription per month, calculated at 1 credit per TTS character and 3 credits per STT second. The M tier at $340/month includes 9 million credits, equal to approximately 200 hours of TTS or 833 hours of STT. Enterprise plans add unlimited credits, on-premise deployment, and zero data retention for compliance-restricted voice applications.

How Does Gradium’s Latency Compare to ElevenLabs, Cartesia, and Deepgram?

Gradium leads the independent Coval 2026 benchmark on TTS latency consistency, recording a 2ms IQR compared to 28ms for ElevenLabs Turbo v2.5, 68ms for Deepgram Aura-2, and 100ms for Cartesia Sonic-3. IQR measures how predictable latency stays across repeated requests, which determines whether a live app’s response time stays stable under real traffic.

Source: Coval independent TTS benchmark, cited on Gradium’s “Best AI Voice Generators in 2026” resource page.

On the STT side, the Coval benchmark reports Gradium STT at 2.4% Word Error Rate with a 1,560ms median Time To First Token, versus Deepgram Nova 3 at 992ms median TTFT but 25.2% to 25.3% WER, and AssemblyAI Universal Streaming at 1,061ms TTFT with 4.2% WER. Voice agents transcribing structured data like order numbers or medical terms need Gradium’s lower error rate; agents handling short, predictable commands can prioritize Deepgram’s faster TTFT instead.

For a full head-to-head breakdown, see our dedicated Gradium vs ElevenLabs: Which Voice AI API Wins in 2026? comparison.

Who Should Use Gradium to Reduce Voice Latency?

Gradium fits developers building production voice agents, customer service automation, and multilingual real-time translation apps where sub-300-millisecond response time determines user retention. Three specific profiles get the most value from Gradium’s latency optimizations:

• Voice agent developers on LiveKit or Pipecat integrating TTS and STT into existing agent orchestration frameworks without building custom streaming infrastructure.
• Enterprise teams handling structured voice data such as order confirmations or account numbers, where Gradium STT’s 2.4% WER reduces transcription-driven failures compared to faster but less accurate alternatives.
• Edge and offline app developers deploying Gradium’s Phonon on-device model to eliminate network latency on Android, iOS, and browser targets entirely.

What Are the Best Alternatives to Gradium?

ElevenLabs, Cartesia, and Deepgram rank as the three most-benchmarked alternatives to Gradium for real-time voice AI in 2026. ElevenLabs Turbo v2.5 offers broader voice library options with a 28ms latency IQR. Cartesia Sonic-3 targets developers prioritizing voice cloning quality over raw latency consistency. Deepgram Nova 3 delivers the fastest median STT response time at 992ms for applications where transcription accuracy is a secondary concern.

Read our full breakdown in Best AI Voice Generators for Developers in 2026.

Frequently Asked Questions

What is a good TTFA target for a live voice app?

Gradium’s engineering documentation sets 300 milliseconds as the TTFA target for AI voice agents to feel responsive during live conversation.

Does reducing audio codebooks noticeably hurt voice quality?

Fewer codebooks reduce audio bitrate and fine acoustic detail. Gradium’s architecture allows dynamic codebook selection per request, so latency-critical turns can use 8 to 16 codebooks while quality-critical turns use 24 to 32.

Can Gradium run without a network connection?

Yes. Gradium’s Phonon model runs on-device on Android, iOS, and browser CPUs with no network dependency, reaching 1.00% WER on the Seed-TTS English benchmark.

Does Gradium support languages besides English?

Gradium supports English, French, Spanish, German, and Portuguese with mid-sentence code-switching, according to Gradium’s product documentation.

Gradium’s 155ms P50 TTFA and 2ms latency IQR on the Coval 2026 benchmark make it the lowest and most consistent latency option among the four TTS APIs tested, at a starting price of $13/month for the XS plan.

Related Reading:

• Best AI Coding Tools in 2026
• Best Free AI Voice Generators (No Watermark)
• Best AI Voice Cloning Tools
• Best AI Voice Generators for Developers in 2026