Speech-to-text inference is computationally intensive. Each audio file triggers a pipeline of deep learning models (acoustic models, language models, and post-processing stages) running on NVIDIA GPUs to meet latency requirements. At scale, this means thousands of GPU instances running continuously.
AssemblyAI needed a way to run GPU-intensive inference workloads cost-effectively at scale, without sacrificing the availability and latency guarantees their customers depend on. Running entirely on On-Demand Instances would be cost-prohibitive. Spot Instances offer significant savings, while Amazon Elastic Compute Cloud (Amazon EC2) can reclaim them with a 2-minute warning when capacity is needed elsewhere. AssemblyAI needed an architecture that absorbs Spot interruptions gracefully while maintaining service level agreement (SLA) commitments. Additionally, customers in regulated industries, including healthcare, finance, and legal, must keep audio data within specific geographic boundaries. They implement strict data residency configurations in European AWS Regions.
Available on AWS Marketplace, AssemblyAI provides an API for transcription, speaker detection, sentiment analysis, chapter detection, and personally identifiable information (PII) redaction. Their models transcribe and analyze speech for high-volume workloads across industries including technology, media, healthcare, and financial services.
In this post, we show you how AssemblyAI runs cost-optimized AI inference at scale for their speech-to-text platform. The architecture uses Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, and Amazon Simple Queue Service (Amazon SQS). Together, these services combine a flexible fleet of NVIDIA GPUs, ECS capacity providers, and a queue-based retry mechanism. The result is significant cost savings while maintaining SLA commitments across European AWS Regions.
Solution overview
AssemblyAI’s inference platform runs on Amazon ECS with compute capacity managed through ECS capacity providers. The fleet mixes Amazon EC2 Spot Instances and On-Demand Instances across multiple NVIDIA GPU instance families, running a single inference model per GPU at a scale of thousands of instances. The architecture is built on four pillars that work together to deliver cost-optimized and resilient inference:
- Amazon ECS capacity providers mixing Spot and On-Demand.
- Multi-family GPU Spot diversification across EC2 instance types (such as g4dn.xlarge, g4dn.2xlarge, g5.xlarge, g5.4xlarge, and g6.2xlarge).
- Queue-based retry architecture using Amazon SQS for interruption recovery.
- Regional deployment across US West (Oregon) us-west-2 and Europe (London) eu-west-2 for data residency compliance.
The following diagram illustrates how this solution is deployed:
Figure 1: AssemblyAI’s inference architecture on Amazon ECS with Amazon EC2 Spot GPU instances and an Amazon SQS-based retry mechanism
An Amazon ECS cluster runs within a virtual private cloud (VPC) across multiple Availability Zones. The cluster hosts GPU-backed ECS tasks (shown as ECS Task, GPU Spot 1 and ECS Task GPU Spot 2) that poll the queue and process inference jobs. Each task retrieves audio files from Amazon Simple Storage Service (Amazon S3) and writes the completed transcription to a results store, which AssemblyAI configures to deliver the output to the end customer.
When a Spot Instance receives an interruption notice, Amazon EventBridge invokes the Spot Interruption Handler (an AWS Lambda function). The handler returns the in-flight job to the Main Job SQS Queue so another healthy ECS task can pick it up. Customers configure their system to route jobs that exhaust retry attempts to a Dead Letter SQS Queue for investigation. Amazon CloudWatch Metrics monitors the health of the fleet and the Spot interruption events, providing observability across the system.
Amazon ECS capacity providers and Spot diversification
To optimize inference costs without compromising reliability, AssemblyAI configures multiple capacity providers in a single capacity provider strategy within each ECS cluster. An On-Demand capacity provider serves as an always-on baseline, and a Spot capacity provider spans different GPU instance families and Availability Zones to serve spikes of demand. Each capacity provider associates with an Amazon EC2 Auto Scaling group. Amazon ECS managed scaling automates the scaling of GPU instances based on the capacity provider weights and the pending inference tasks. When the task queue grows, the Spot capacity provider requests additional instances from the diversified pool. When demand drops, instances scale in gracefully.
One of the most effective Spot best practices is to be flexible across a wide range of instance types to increase the chances of getting the aggregate compute capacity needed. Rather than depending on a single GPU instance family, AssemblyAI’s infrastructure spans three Amazon EC2 GPU instance families:
- g4dn instances – Powered by NVIDIA T4 GPUs, providing a cost-effective baseline for standard speech recognition models.
- g5 instances – Equipped with NVIDIA A10G GPUs, delivering higher throughput for more demanding model architectures.
- g6 instances – Featuring NVIDIA L4 GPUs, offering improved performance-per-dollar for larger models including large language model-based inference.
By spreading demand across instance families and sizes, AssemblyAI draws from a significantly larger pool of available Spot capacity. Using the price-capacity optimized allocation strategy, Amazon EC2 provisions instances from the Spot capacity pools with the highest available capacity at the lowest price, reducing the likelihood of interruptions.
Queue-based retry for interruption resilience
Maintaining quality of service during Spot interruptions was a core design requirement. AssemblyAI decouples job submission from execution using Amazon SQS. When a transcription request arrives, it’s placed into an Amazon SQS queue. Amazon ECS tasks running on GPU instances poll the queue, pull jobs, and execute the inference pipeline. Individual jobs typically take between 15 seconds and several minutes depending on audio length.
When a Spot Instance receives a 2-minute interruption notice, the retry flow works as follows:
- In-flight jobs that can complete within the 2-minute window finish normally. Jobs that need more time are returned to the Amazon SQS queue, where they become immediately available for other workers.
- Another healthy GPU instance picks up the message, potentially on a different instance type or in a different Availability Zone, and completes the inference.
To reduce recovery time, AssemblyAI creates custom Amazon Machine Images (AMIs) containing model artifacts and container images in approximately 2 minutes. For end customers, the impact of a Spot interruption is a small increase in processing time, but the transcription completes successfully and is delivered as expected.
Regional deployment for data residency
Running in multiple AWS Regions allows AssemblyAI to support customers in regulated industries, including healthcare, financial services, and legal services, where audio data must never leave specific geographic boundaries. Each AWS Region runs a fully independent copy of the inference architecture: its own Amazon SQS queues, Amazon ECS clusters, capacity providers, and GPU fleets. Multiple GPU instance families (g4dn, g5, g6) are available in these European Regions with competitive Spot pricing. This instance diversity supports a reliable Spot-based inference service while meeting strict data residency requirements.
Results
By running Amazon EC2 Spot Instances with GPUs in production, AssemblyAI achieved 70 percent cost savings on inference compute compared to On-Demand pricing. As the platform grew nearly 4x year-over-year in compute demand, the Spot-based architecture kept unit economics sustainable. Operating at this scale, AssemblyAI learned that diversifying across instance families and sizes is the most effective lever for minimizing Spot interruption rates. The g4dn, g5, and g6 families tap into independent capacity pools, so when one family experiences capacity pressure, the others remain available. Designing fault tolerance from day one with queue-based retries and fast warm-up turns Spot interruptions into routine events while maintaining SLA commitments.
Conclusion
Teams running batch or near real-time AI workloads with similar characteristics (short-lived, independently retriable jobs on GPU compute) can apply this pattern. Start by diversifying instance families and building queue-based fault tolerance to take advantage of cost-effective Spot infrastructure without compromising availability.
To get started with GPU tasks on Amazon ECS, see Amazon ECS for GPU workloads. To evaluate Spot availability for your workload, explore using Spot placement scores for accelerated compute and Amazon EC2 Spot Instances best practices. AssemblyAI’s Speech-to-Text API is available on AWS Marketplace. To learn more about their platform, visit assemblyai.com.
About the authors
