AWS shows how to set speech AI scaling limits from live measurements

AWS published a measured recipe for scaling self-hosted speech AI on October 22, replacing guessed settings with tests of capacity and recovery time. Its Kokoro-82M deployment on SageMaker AI met its response-time target at 50 simultaneous requests per instance. Planning for additional capacity to become useful required 320.2 seconds, including an allowance for policy detection.

The walkthrough addresses a tradeoff AWS describes in operating text-to-speech services: keeping too much capacity means paying for idle GPUs, while pushing each instance too hard risks queues and slower responses. Rather than choose a scaling target first, the method measures what one instance can handle and how long reinforcement takes.

Find the point where audio starts arriving late

The first measurement counts requests queued or executing on one instance. The test increases that number in steps, looking for the highest tested level that meets a service-level objective, or response-time promise. Here, the target was for the first audio data to arrive within three seconds at the 95th percentile.

At 50 concurrent requests, AWS measured 2,805.7 milliseconds. At 60, the delay reached 3,374.8 milliseconds, breaking the target. The workflow therefore used 50 as its per-instance ceiling. That is a result from this deployment and its tested load steps, not a universal capacity rating for Kokoro-82M.

During this test, the tool freezes automatic scaling and holds the endpoint to one instance, restoring its settings on exit. Developers can first preview the schedule without making AWS calls; AWS estimates roughly 36 minutes for the live run.

uv run tts-bench qmax --model kokoro-82m --dry-run

Extra capacity is not instant relief

The second test forces the service to add capacity, then follows it through provisioning, container startup, model preparation and traffic recovery. It deliberately bypasses the scaling policy’s detection logic. That separates the time needed to supply working capacity from the time a policy takes to recognize that more is needed.

The planning calculation combines both delays. Operators also choose how much traffic growth to withstand during that interval: AWS’s example assumes a 25% surge within one scale-out cycle. The measured ceiling and reaction time feed the scaling thresholds and cooldowns—the pauses between scaling actions.

Make the measurements govern deployment

The deployed policy uses SageMaker’s native ConcurrentRequestsPerModel metric rather than requiring the container to publish its own. There is a calibration step: CloudWatch reports a maximum over a 10-second window, while the test client measures mean concurrency. The tool measures their ratio to translate the threshold; in AWS’s run, that ratio was 1.0 throughout.

The infrastructure code adds another check. It creates scaling resources only when scaling is enabled and a measured, feasible plan exists for the model. Before load testing, a read-only check compares the deployed policies and alarms with the configuration, helping catch settings that could change capacity mid-test and corrupt the measurements.

There is a security caveat before adapting the sample. AWS warns that its container pins an older PyTorch version associated with advisory PYSEC-2026-2286. It says the issue is fixed in torch 2.10.0 and advises reviewing and upgrading the dependency before use beyond a proof of value.