Building ultra-low-latency voice AI applications on AWS

Conversational AI is transforming how humans interact with technology. From virtual assistants and AI-powered contact centers to real-time translation and accessibility tools, voice-first interfaces are rapidly becoming the default mode of interaction for millions of users worldwide. Yet delivering a truly natural voice experience remains one of the hardest challenges in AI infrastructure. Users expect human-like responsiveness. Excess latency can break conversational flow and erode trust. To feel natural, AI-generated speech must arrive within the same temporal window as human turn-taking. This post introduces how Cartesia , an AWS Partner specializing in real-time voice AI, uses purpose-built State Space Models (SSMs) and AWS infrastructure to deliver sub-100ms text-to-speech and speech-to-text at global scale. Developers can build voice experiences that feel indistinguishable from human conversation. The rise of conversational AI and latency expectations The demand for real-time voice AI is growing rapidly across industries: Contact centers: AI agents handling millions of customer interactions require instantaneous speech generation to maintain natural dialogue. Telehealth: Remote patient consultations demand low-latency voice for accurate, empathetic communication. Gaming and virtual worlds: Immersive experiences require real-time voice synthesis for non-player characters (NPCs) and player interactions. Accessibility: Screen readers and voice interfaces must respond instantly to serve users with visual impairments. Inbound and outbound sales: AI sales agents must respond without awkward pauses to qualify leads, handle objections, and keep prospects on the line. Traditional transformer-based text-to-speech (TTS) models struggle with streaming latency because they process entire sequences before producing output. This architectural limitation creates a fundamental tension: higher quality often means higher latency. A new class of models, State Space Models, resolves this trade-off. AI for voice: State Space Models (SSM) State Space Models (SSMs) take a fundamentally different approach to processing sequences than transformers, the architecture behind most of today’s AI. Transformers use a mechanism called attention, which compares every element in a sequence to every other element. As a result, the amount of computation grows quadratically as the sequence gets longer. SSMs instead treat a sequence as a system that evolves continuously over time, processing it step by step and carrying a compact internal state forward. This can be converted into a form computers handle efficiently, keeping the cost low for certain long sequences. This efficiency makes SSMs well suited to real-time, streaming data like live speech. Key advantages of SSMs for audio processing include: High-quality long-range dependencies: Structured SSMs are a form of linear recurrence, but their state parameterization preserves information over long audio sequences, avoiding the long-range signal decay that limits classic recurrent neural networks (RNNs) such as long short-term memory (LSTM) networks. Constant memory footprint: Unlike transformers that cache all previous tokens, SSMs maintain a fixed-size hidden state. Native streaming support: The recurrent structure processes input token-by-token, so it supports true streaming without architectural modifications. These properties make SSMs uniquely suited for real-time voice applications where both quality and latency are non-negotiable. How it works: SSMs for real-time voice Cartesia puts these SSM advantages to work in two production voice models: Sonic for speech generation and Ink for transcription. Introducing Cartesia Cartesia is a real-time multimodal intelligence company that has built a voice AI application, purpose-built on State Space Model architecture. Founded by researchers who pioneered SSM advances, Cartesia’s application is designed for streaming inference at the edge of human perception. The application features two core models, Sonic for text-to-speech and Ink for speech-to-text, both optimized for sub-100ms first-byte latency while maintaining studio-quality output. Sonic (text-to-speech) Sonic is Cartesia’s flagship streaming text-to-speech model, delivering ultra-realistic speech synthesis with sub-100ms latency: First audio byte in approximately 90ms: Users hear the voice respond almost instantaneously. Ultra-realistic voice quality: Natural prosody, emotion, and expressiveness across over 44 languages. Voice cloning with minimal data: Create natural-sounding custom voices from 10-second audio samples. Streaming-native architecture: Audio begins generating before the full text input is received. Context-aware synthesis: Maintains consistent tone and style across long passages. Sonic is recommended for teams that want the fastest time-to-production with minimal operational overhead. Call the API and receive streaming audio in real time. Ink (speech-to-text) Ink is Cartesia’s streaming speech-to-text model, purpose-built for conversational AI applications: Native turn detection: Automatically identifies when a speaker has finished their turn, alleviating awkward pauses without a separate voice activity detection (VAD) or end pointing model. Context-aware end pointing: Uses semantic understanding to distinguish between mid-sentence pauses and true conversation boundaries. Streaming transcription: Emits partial transcripts in real time as the user speaks, so a downstream large language model (LLM) can begin forming its response before the turn ends. Noise robustness: Maintains accuracy in noisy environments common in real-world deployments. Multi-speaker support: Handles overlapping speech and speaker transitions gracefully. Together, Sonic and Ink provide the complete voice AI stack needed to build conversational agents that feel human. Deploying Cartesia on AWS If you need data sovereignty, latency control, or compliance, you can deploy Cartesia models directly into your own AWS account. Sonic is delivered as an Amazon SageMaker AI model package through AWS Marketplace, which you deploy to a SageMaker AI endpoint running entirely within your virtual private cloud (VPC). Inference data doesn’t leave your account. The deployment patterns in this section cover Sonic. Ink is currently available through the Cartesia API, which you can purchase through the Cartesia Voice AI Platform listing in AWS Marketplace. Two primary patterns are available for Sonic. Option 1: Amazon SageMaker AI (AWS Marketplace model package) Deploying on Amazon SageMaker AI is the fastest path to a managed, in-VPC Sonic endpoint. Sonic ships as an AWS Marketplace model package, so there’s a one-time subscribe step that grants your account an AWS Region-specific model-package ARN. You then create the SageMaker AI endpoint from that ARN, either from the console or with the SageMaker or Boto3 SDK. The following steps follow Cartesia’s official deployment notebook. Prerequisites Run from an Amazon SageMaker Notebook Instance or SageMaker Studio. The AWS Identity and Access Management (IAM) execution role needs AmazonSageMakerFullAccess . To subscribe, your role (and you) need aws-marketplace:Subscribe , aws-marketplace:Unsubscribe , and aws-marketplace:ViewSubscriptions . Otherwise, the account must already be subscribed to the listing. Before deployment, choose a supported instance type from the model package’s SupportedRealtimeInferenceInstanceTypes field: ml.g6e.xlarge or larger ml.g7e.2xlarge or larger ml.g7.2xlarge or larger ml.p5.48xlarge ml.p5en.48xlarge If your chosen instance type’s applied value is below your current usage plus your planned endpoint instance count, you may need to request a service quota increase. See SageMaker AI quotas for details. Deploy Sonic 3.6 to a SageMaker endpoint In Amazon SageMaker JumpStart, you can subscribe to its AWS Marketplace model package, create a SageMaker model, and host it on a real-time endpoi