Deepgram Sends SageMaker Billing and GPU Metrics to CloudWatch Without Container Egress

Keeping speech AI inside a customer’s AWS account can support data-residency and compliance work. But network-isolated SageMaker deployments have also left teams with basic endpoint health signals while billing, feature-use and inference-engine details stayed inside the model container. Deepgram now exposes those signals through CloudWatch and SageMaker detailed observability without requiring outbound container connections.

Two routes out of an isolated container

Deepgram Enhanced Metrics is available now for the company’s speech-to-text and text-to-speech deployments on Amazon SageMaker AI. For billing and usage, the container writes CloudWatch Embedded Metric Format records to standard output; SageMaker forwards that output to CloudWatch Logs, which extracts the records into metrics. The design needs no agent, sidecar or additional IAM permissions.

• The Deepgram/SageMakerInference namespace publishes ConsumedUnits, AudioDurationSeconds and CharCount. Deepgram says ConsumedUnits carries the billable-unit values used for AWS Marketplace metered billing, letting customers compare billed volume with traffic by category, model and transport.
• The Deepgram/SelfHosted namespace breaks usage out by deployment, request method, model tier and enabled functions, including diarization, smart formatting, redaction and keyterm prompting. This stream is enabled by default but can be disabled; the billing stream cannot, because it is part of metering.

Cost visibility is not endpoint visibility

The new billing and usage streams trade detail for simpler aggregation. They cover Deepgram endpoints across an AWS account and Region, but do not include endpoint names or instance IDs. A team can identify a model or transport behind its billed usage, but cannot use those streams alone to isolate one endpoint or one machine.

That narrower operational view comes from Deepgram’s Prometheus endpoint and SageMaker detailed observability. The resulting metric store includes Deepgram engine measures such as active requests and estimated stream capacity, alongside host measures and separate GPU series. SageMaker resource labels let operators filter to an endpoint, instance or individual GPU, rather than rely on a fleet-level average.

A configuration boundary for existing fleets

Detailed observability is enabled by default on newly created SageMaker AI endpoints and publishes every 60 seconds unless configured otherwise. Existing endpoints need a new endpoint configuration and an update operation; AWS describes that update as a blue/green deployment that keeps the endpoint serving traffic. The metrics can then be queried with PromQL in CloudWatch, Grafana or another Prometheus-compatible tool.

An added layer, not a single dashboard

The additions complement rather than replace SageMaker’s standard endpoint metrics. Those built-in measures cover request concurrency, first-chunk latency and server errors; Deepgram’s new streams address Marketplace billing, feature adoption, engine load and hardware saturation. The split is consequential for operators: a healthy endpoint is not, by itself, an explanation of what drove cost or whether one GPU is nearing capacity.