Octave says shared AI foundation halved deployment time for new use cases

Octave says building shared AI infrastructure before individual features cut deployment time per use case by 50%. An October 20 AWS case study details the system behind its Reliance quality-management platform, where complaint records and audit findings must stay inside each customer’s boundary. Its first application, the Complaint Triage Assistant, is in production, according to Octave.

Reliance supports more than 650 enterprise customers and over 40 applications, Octave says. Those applications hold complaints, corrective actions, operating procedures and audit records. The team chose to build a common foundation rather than recreate AI infrastructure for each workflow, with complaint summarization and triage going first to prove the design.

The first layer turns documents into embeddings: numerical representations that let the system search by similarity. Cohere Embed v4 runs through Amazon Bedrock, and Amazon OpenSearch Service stores the results separately for each customer, or tenant. An Anthropic Claude model handles the agent’s reasoning through Bedrock.

Octave evaluated embedding models against its own documents. Its requirements included connecting a photograph of a cracked product housing with a written description, and matching a German complaint with an English corrective action. The chosen model represents text and images together and supports multiple languages, avoiding separate image-processing and translation pipelines.

The team also tested how much numerical detail each embedding needed. Octave says 512 dimensions maintained its target accuracy on representative production queries while reducing storage roughly 66% against the 1,536-dimension default. That is a result from its own retrieval workload, rather than a general accuracy claim.

Customer separation is enforced beneath the application code. Each user session runs in a dedicated microVM, a small virtual machine with its own processor resources, memory and filesystem. Octave says the session is stopped and sanitized afterward. Credentials are restricted to the agent and remain available only during the session.

Amazon Bedrock AgentCore carries the authenticated user’s identity through execution rather than relying on a shared service identity. Its gateway checks which tools, operations and parameters the agent can use. This puts access decisions outside the model’s reasoning, instead of asking every future feature team to implement every boundary in application code.

The documents themselves pose a different problem: the agent must read material it cannot trust. Complaints can arrive with outside attachments containing hidden instructions. Octave applies Bedrock Guardrails around the workflow and marks attachment-derived content as data rather than instructions. Screening happens at three points, not just when a user submits a request.

Screening is paired with limits on what an agent can do. Document extraction runs in a sandbox without outgoing network access or persistence beyond the session. Personally identifiable information is masked on entry, and Octave tracks detection and masking rates separately for each tenant.

Reliance hosts more than 2 billion documents, making ingestion another constraint. The pipeline distributes batches across concurrent processing jobs, extracts text from scanned pages and preserves document structure. Each chunk carries metadata including its tenant, language, document type and source application, supporting retrieval filters and customer separation.

Octave packed multiple roughly 4,000-token chunks into requests using Embed v4’s 128,000-token context window, reducing API calls by about an order of magnitude. Its internal testing measured 28 million tokens per minute. After batch-size tuning with Cohere, Octave says the pipeline sustained 1.4 million documents per 24 hours per tenant workload.