Cohere details Ada’s split AI search setup and reported speed and cost gains

Ada’s customer-service agents retrieve information in a median 77 milliseconds, while its offline indexing delivers more work per dollar, according to a Cohere customer case study published October 27. The results followed a decision to stop treating live search and knowledge-base preparation as jobs for the same deployment. Ada instead chose separate infrastructure for each workload.

The distinction matters for Ada’s voice agents. Before answering a customer, an agent must find relevant information in that business’s knowledge base. Ada wanted that step to fit a tighter time budget than its previous retrieval setup could consistently meet. On a phone call, the wait is silence rather than a visible sign that something is happening.

The evaluation overturned a deployment assumption

Ada compared four embedding approaches across 15,000 evaluation queries, testing both online and offline workloads. Its requirements were retrieval quality, latency and the ability to operate across regions at multi-customer scale. Successful resolution of customer inquiries was the guiding outcome, rather than speed alone.

The testing challenged Ada’s initial assumption that one deployment could efficiently handle both workloads. Preparing a knowledge base and searching it during a conversation called for different configurations. Cohere participated in the benchmarking and experiments, so the findings are a vendor-published customer account, not an independent comparison.

Embeddings turn content and queries into numerical representations that make searches for similar meaning possible. Ada uses that capability both to prepare customer knowledge bases for search and to find information while an agent is responding. The evaluation separated where those two jobs should run.

Live conversations and bulk preparation take different routes

• Live retrieval: Ada self-hosts Cohere Embed on dedicated Amazon SageMaker AI instances. SageMaker was geographically closer to Ada’s services, making it the preferred location for time-sensitive conversations. The case study also attributes latency benefits to using dedicated deployments rather than shared services.
• Offline indexing: Ada uses Cohere Model Vault for bulk embedding and knowledge-base ingestion. Cohere manages the dedicated infrastructure. Its managed autoscaler handled spikes when content was republished faster than Ada’s self-managed scaling, according to the case study.

For offline embedding, Ada reports approximately 2.7 times greater throughput per dollar with Model Vault than with its self-hosted setup. That comparison concerns the cost efficiency of preparing searchable information. The online results measure a different job: how quickly and effectively the system retrieves information during a live interaction.

Compression reduced the storage burden

Ada also reports cutting vector storage requirements by a factor of four, with no measurable retrieval degradation. Embed’s built-in compression meant the team did not have to build its own quantization pipeline—the process used here to reduce the numerical representations’ storage footprint.

Those savings apply across thousands of customer knowledge bases, some containing tens of thousands of articles. The deployment changes also fit Ada’s existing AWS environment. Ada could adjust retrieval while preserving the models and infrastructure already supporting its agents, rather than replacing the rest of the system to improve this step.