OpenAI explains why it returned to shared storage at billion-user scale

Keeping one product’s traffic from disrupting another created a different problem for OpenAI: teams had to plan database capacity themselves, sometimes scrambling during launches. In an October 23 engineering disclosure, the company explained why its Habitat storage platform moved back toward shared database containers, with data-movement capability, after trying stronger separation between workloads.

Habitat provides fast access to stored information for OpenAI products. The company says it serves more than 500 petabytes of data and must respond in milliseconds to more than 1 billion users across 40 geographic regions. The disclosure, by engineers Jon Lee, Hrachya Hakobyan and Chaomin Yu, focuses on its underlying Azure Cosmos DB storage.

When a niche feature disrupts login

Habitat initially used one default Cosmos DB container in each account. A container groups data and has an assigned processing budget. Sharing that budget across workloads—known as multi-tenancy—exposed several ways for seemingly separate uses to collide, according to OpenAI.

• Shared keys: Different product uses could choose the same partition key, such as a user or workspace identifier. Those uses then shared the database’s limits for that key, constraining total throughput for a given user.
• Noisy neighbors: Separate keys could still sit on the same physical partition, a backing slice of the database. Because limits were enforced there, niche-feature workloads could interfere with core login and signup flows.
• Growth without more headroom: As stored data grew, physical partitions split. If the container’s total processing budget stayed unchanged, each partition received a smaller share. Storage growth alone could therefore reduce capacity for active workloads, even with steady request rates.

OpenAI responded by giving critical new uses separate containers with independent physical partitions. It also used a script to adjust each container’s provisioned throughput as its partition count changed. Together, those measures largely avoided the problems the team had been encountering.

Isolation moved the burden to product teams

The separation was incomplete. Older critical workloads remained in the original shared container unless their data was migrated. As pressure and production incidents grew there, OpenAI directed new services and major features into separate containers. The company says this prevented many site-wide outages, confining failures instead to a feature or service.

But an isolated workload lost the cushion of a much larger traffic pool. OpenAI illustrates the difference with a use case jumping from zero to 1 million queries per second. That increase might barely shift a large shared container’s overall demand, while representing 1,000 times the baseline load of a dedicated container used only for internal testing.

Teams sometimes launched features without increasing their containers’ capacity beforehand, leaving engineers to scale them during launch-day incidents. Predicting demand also required detailed product knowledge. A basic scaling script could follow growth trends, but could not eliminate the need for teams to anticipate when traffic would shift.

OpenAI says this spread capacity planning across essentially every product team, with widely varying attention to provisioning. That organizational burden drove Habitat back toward multi-tenancy by default, this time with data-movement capability. The company worked closely with Azure on co-managed containers, rather than continuing to make separate containers the default response to growth.