Loading page…
The newly available Model Vault capability aims to protect data while a model processes it, but customers will need to assess exactly what its attestation evidence covers.
Listen to this story
Cohere has added confidential inference to its single-tenant Model Vault service, extending protection beyond data at rest and in transit to the CPU and GPU memory used during processing. The company says operators cannot read customer prompts, while customers retain keys for returned responses and receive attestation reports for verification. The tier costs existing Model Vault customers no extra, but buyers still must assess hardware and evidence requirements.
Model Vault combines Intel TDX or AMD SEV-SNP confidential VMs with NVIDIA confidential GPUs.
Attestation reports cover the hardware, software, and security policies protecting each inference workload.
The published account does not clarify whether attestation is checked at boot, per request, or on another schedule.
Cohere has made confidential-computing support newly available in Model Vault, its single-tenant inference service. The company says the Encrypted tier protects customer data during the usually exposed moment when an AI model reads and processes a prompt, putting Cohere, cloud providers, and other infrastructure operators outside that boundary.
That is a different proposition from ordinary encryption. Data is commonly protected when stored and sent over a network, but it must normally become readable in system memory for a model to run. Cohere says its new design keeps data encrypted outside a protected execution environment, so its staff, the cloud provider, and cluster operators cannot access it.
Model Vault already gives a customer a dedicated model deployment rather than a shared API. Confidential computing targets a separate concern: whether the people operating the underlying infrastructure can inspect the workload. Cohere’s approach combines a confidential virtual machine using Intel TDX or AMD SEV-SNP with NVIDIA GPUs in confidential-computing mode.
The GPU portion is important because inference does not stay on a CPU. Cohere says its design protects memory and CPU-to-GPU connections, with data decrypted only where processing must occur inside the protected CPU and GPU environment. Customers retain the keys to decrypt returned responses, according to the company.
Cohere’s pitch is not simply that customers should trust the boundary. It says every inference returns an attestation report that customers can use to check the hardware, software, and security policies safeguarding a workload. The company says approved hardware and software configuration values are registered with Intel Trust Authority when a confidential VM starts.
Confidential inference is not a new category. The public account names Tinfoil, Edgeless Systems’ Privatemode, Phala, Maple AI, and Confer as providers offering related privacy-oriented services. Cohere’s distinction is applying the approach to its own enterprise models in its existing dedicated serving product, rather than offering an independent privacy service around other models.
Cohere says the capability carries no added cost for existing Model Vault customers. But price parity does not settle the procurement question. For organizations handling sensitive data, the practical test is whether the attestation evidence, hardware assumptions, and future serving-stack transparency meet their own security requirements.
Story updates
Cohere’s confidential-computing capability for Model Vault is in public beta on NVIDIA H100 GPUs on Microsoft Azure. In an October 22 technical explanation, the company detailed its central safeguard: customer software checks the computing environment before releasing sensitive requests, rather than relying only on the provider’s promise to keep them private.
The design protects data while a model processes it. Cohere says its dedicated model deployment combines hardware-isolated execution, encryption of request contents and remote attestation—cryptographically signed evidence describing the hardware and software handling a request.
The serving software runs inside a confidential virtual machine, whose memory is protected from the infrastructure hosting it. NVIDIA’s confidential-computing technology also protects GPU memory and encrypts communication between the CPU and GPU, according to Cohere.
Model Vault uses Oblivious HTTP, or OHTTP, to encrypt sensitive request contents to a public key associated with the verified environment. Only a gateway inside that environment holds the corresponding private key needed to decrypt them. Responses receive protection on the return path, too.
Intermediary systems can therefore route encrypted messages without reading their protected contents. Applications retain the Model Vault API and familiar request formats; added protection sits in the client and infrastructure layers.
The customer’s client checks attestation evidence against an approved policy. Signed evidence describes what hardware, firmware and software are running; the policy determines whether that configuration is acceptable for the customer’s data.
If a required property is wrong or missing, the client refuses to send the protected payload. Examples include a changed software measurement or a GPU outside the required confidential mode. The sensitive request stays inside the customer’s environment.
Within its stated threat model, Cohere says outside infrastructure and operators cannot read protected workload memory. Customers still trust the confidential-computing hardware, firmware, measured software, approved reference measurements and attestation infrastructure.
Application risks remain: protected hardware does not solve prompt injection, insecure tool use or authorization errors.
Timing, message size and traffic volume can remain visible. API credentials and model identifiers sit outside the OHTTP-protected payload.
The system does not guarantee availability or cover some physical and side-channel attacks, which infer information indirectly from hardware behavior.
Cohere intends to open-source the confidential inference stack so customers and researchers can inspect its policies and reproduce verification measurements. That remains a plan, alongside support for NVIDIA Blackwell infrastructure, including B200 GPUs, as the service moves toward general availability.
cohere.comLoading discussion...
Join the conversation
Explain what proof you would need beyond an attestation report.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.