How ChatGPT Handles 900 Million Users!
900 million people message ChatGPT and everyone gets an answer in seconds. The surprising part is how ordinary the stack is. A global router picks your region, plain Postgres holds the data, and a cache catches almost every read before it reaches the database. If a thousand requests need the same thing, one fetches it and the rest read from memory.
After the data is ready, a scheduler decides which GPU cluster handles the request, then batches it with other users so one pass through the model serves many people at once. Big models get split across several GPUs, and the answer streams back token by token instead of waiting for the full response. One request, traced from the router all the way to the words filling your screen.
Would you have guessed the whole thing runs on Postgres? Tell us in the comments.
#ChatGPT #SystemDesign #Postgres #DistributedSystems #Backend #DevOps #Scalability #OpenAI #Caching #SoftwareEngineering #Databases #CloudComputing