@rick.theengineer: ChatGPT answers you in about a second. Doing that for 800,000,000 people is one of the hardest engineering problems on earth — and almost none of it is the model. Your prompt never touches a GPU first. It hits auth, rate limits, your conversation history, safety checks, and a router that decides whether your question is worth a cheap model or an expensive one. Only then does it reach the compute. And the compute does something most people get wrong. You don't get a GPU. You get a *lane*. One accelerator runs sixteen strangers' conversations at the same time, stepping all sixteen forward by one token, together. Give everyone their own chip and 94% of it sits idle. Then it gets stranger. A frontier model doesn't fit on one chip at all — it's cut into pieces across eight of them, wired at 900 GB/s, talking to each other on every single token. OpenAI wrote a new networking protocol (MRC) because the old one was the bottleneck. Meanwhile the boring half is just as hard: one unsharded Postgres primary and ~50 read replicas doing millions of queries a second. That's the whole idea. ChatGPT is two scaling problems wearing one text box — a normal internet company, and a supercomputer. Which half surprised you more? #systemdesign #aiinfrastructure #gpu #softwareengineering #machinelearning

RickTheEngineer

Open In TikTok:

Region: GB

Saturday 26 September 2026 13:21:47 GMT