OpenAI says its Habitat platform now serves more than 500 petabytes of data and over 70 million requests per second across nearly 40 regions. The engineering account offers practical lessons in tail latency, overload control and knowing when a fast-moving platform is ready for a rewrite.
NEW DELHI, September 12, 2026, 5:12 PM IST — OpenAI has detailed the storage platform that underpins ChatGPT, Codex, its API and other services, describing how a Python client library evolved into a globally distributed service for products used by more than one billion people each week.
The platform, called Habitat, handles more than 70 million requests per second and serves more than 500 petabytes of data, according to an engineering report published by OpenAI on September 11. The company said Habitat operates across almost 40 geographic regions and sits between product services and storage systems including Azure Cosmos DB, caches, blob storage and change-data-capture services.
For platform and DevOps teams, the important news is not the scale alone. OpenAI’s account shows why rapidly growing AI products can outgrow direct database access, how small defaults can trigger system-wide instability, and why predictable APIs and backpressure become more valuable as traffic and organizational complexity rise.
What OpenAI confirmed
Habitat began about two years ago as a lightweight Python library connected to a single database. As OpenAI added products and its traffic grew by more than tenfold year over year for three consecutive years, the company turned that library into a service that could centralize routing, authorization, encryption, data-residency placement, caching, rate limiting and multi-tenant isolation.
OpenAI separately said on September 8 that its products reach more than one billion weekly active users and 2.5 million businesses, corroborating the user-scale figure in the engineering post. The storage, traffic and efficiency figures remain company measurements and have not been independently audited.
The first version of the service stayed in Python because the team needed to move quickly. OpenAI said that implementation peaked above 20 million requests per second, but CPU-heavy work such as routing, compression, encryption, checksumming, health checks, request shadowing and hedging created event-loop scheduling delays. Under high utilization, those delays could reach hundreds of milliseconds and, in edge cases, several seconds.
A small connection-pool default exposed a larger failure mode
One of the clearest lessons came from an overload incident. OpenAI found that some processes remained degraded after a bursty client had stopped sending excess traffic. The system kept steering more requests toward already overloaded pods, creating what engineers call a metastable failure: the service could not return to a healthy state without intervention even after the original trigger disappeared.
The team traced the behavior to the LIFO, or last-in-first-out, connection-reuse default in Python’s aiohttp connector. Frequently used connections stayed active while connections to less-loaded processes aged out. Capping connection reuse duration reduced the degradation and helped confirm the diagnosis. OpenAI eventually moved Habitat to explicit round-robin load balancing.
The operational takeaway is broader than one Python library. Health checks and load balancers can amplify a skew they are expected to correct. Platform teams should test recovery after a traffic spike, not only behavior during the spike, and monitor load distribution per worker or pod alongside fleet-wide averages.
Backpressure and constrained APIs became reliability features
OpenAI also described why Habitat intentionally offers a simple NoSQL interface instead of allowing arbitrary queries. Predictable, constant-work requests are easier to isolate and scale; unbounded joins, scans or request fan-out can create latency cliffs and allow one workload to harm others.
That is consistent with the company’s January account of scaling PostgreSQL, which said new workloads default to sharded systems and that write-heavy, partitionable traffic was being moved to Azure Cosmos DB. The newer Habitat layer formalizes that boundary across more products and storage resources.
The service also added per-client concurrency limits, bounded queues and adaptive concurrency controls. Those mechanisms reject or slow work before downstream databases are exhausted. Teams building AI platforms can apply the same principle to model gateways, vector stores and agent tool services: make overload behavior explicit, expose retry signals, and ensure a noisy tenant cannot consume shared capacity without limit.
Why OpenAI moved the service to Rust
OpenAI said it delayed a rewrite for about a year while the architecture and product requirements were still changing. In the second quarter of 2026, two engineers, assisted by Codex and GPT-5.5, rewrote the service in Rust. The Rust version now handles 95% of production requests, and the company expects to retire the Python implementation in the coming weeks.
Company measurements put the new service at six times the CPU efficiency and 15 times the memory efficiency of the Python version, with lower average and tail latency. These numbers should not be read as a general benchmark of Rust against Python: they describe one internal workload after its requirements, traffic patterns and failure modes were understood.
The sequencing is the useful part. OpenAI kept the language that supported fast iteration while the platform boundary was unsettled, instrumented the bottlenecks, then rewrote a mature and measurable service. That approach reduces the risk of optimizing the wrong design.
What platform teams should review now
AI applications often multiply ordinary database work. OpenAI said a single user request can produce hundreds of data lookups, making the slowest lookup visible in end-to-end latency. Teams should therefore track high-percentile latency, event-loop or scheduler delay, connection-pool skew, downstream saturation and recovery time after bursts—not just average request duration.
They should also treat data placement, tenant isolation and request shaping as part of the platform contract. GravityDevOps’ guide to LLMOps explains the wider operational lifecycle, while its comparison of CI/CD tools can help teams design staged rollouts, shadow traffic and measurable migrations for infrastructure changes.
OpenAI has promised a second engineering post covering its storage layer, multi-tenant reliability and read optimization in more detail. Until that arrives, important design questions remain open, including the exact consistency trade-offs, failover model, regional recovery objectives and how Habitat divides responsibility between its own service layer and Azure Cosmos DB.
The bottom line
Habitat is a notable AI infrastructure disclosure because it connects product growth to concrete distributed-systems decisions. The report does not prove that every organization needs Rust, a bespoke storage proxy or OpenAI-scale architecture. It does show that as AI traffic grows, reliability often depends on narrow interfaces, disciplined overload control and observability that can expose unstable feedback loops before a fleet-wide incident.
