The Biggest Chip Ever Built — Why OpenAI Runs On It | Cerebras CEO Andrew Feldman
Summary
This episode focuses on why fast inference has become a central bottleneck in AI, especially as models move from training into real-world production use. Andrew Feldman explains that the most important metric is now "tokens per second per user," because latency determines whether AI feels responsive and useful. The conversation also goes deep into AI infrastructure, including GPUs, Cerebras' wafer-scale chip architecture, memory movement, HBM, SRAM, and manufacturing constraints like advanced packaging and TSMC capacity. Feldman argues that agents and reasoning models increase compute demands, making speed even more important as AI systems begin to act, verify, and iterate autonomously. The episode closes with broader implications for the future of AI products, including potential disruption to SaaS and the rapid obsolescence of today's leading models.
Key Takeaways
- 1Inference speed, not just model quality, is becoming the defining bottleneck in AI product performance.
- 2AI hardware scaling is constrained by a few critical supply-chain chokepoints, especially HBM, advanced packaging, and 3nm wafer capacity at TSMC.
- 3Cerebras' wafer-scale architecture is designed around minimizing memory movement, which Feldman sees as the main limiter for inference workloads.
- 4As AI systems become agentic and reasoning-heavy, CPU demand rises because the system must plan, verify, and take actions around the model itself.
- 5Cerebras' competitive moat comes from architectural uniqueness rather than easily copied features or brute-force spending.
- 6AI model progress is moving so quickly that today's best models may soon look outdated, and younger engineers are already adapting workflows around AI-native tools.
Notable Quotes
""The public markets are voting mechanisms who's most popular. But in the long term, they're weighing mechanisms who's created the most value.""
""The biggest chip that had ever been built before us was 800 square millimeters, 844 to be exact. And this is 46,000.""
""In a little model like a 70-billion parameter block, the weights are about the size of 100 HD movies. So to generate a single word, you move 100 HD movies from memory to compute.""
""The deal is 760 megawatts... We're delivering the full cloud solution. So they connect to us via an API.""
""The number of ways that things can go wrong with supply chain is unbelievable.""
""We know that the model we're using today will be the worst model you ever use.""
""Whatever you think is cool about it right now is going to be boring and backwards in six months.""
""You can ask your AI, build me a tool like Salesforce 30 seconds later.""
Episode questions
Why does speed matter so much for AI products?
Because AI is only useful when people can interact with it quickly enough to stay engaged. Feldman says faster tokens make the product feel real-time, increase usability, and let users tackle harder tasks.
Why are GPUs slower for inference than Cerebras' approach?
Feldman says inference is dominated by moving weights from memory to compute for every token, while GPUs are not optimized for that pattern. Cerebras uses a much larger chip with lots of SRAM to reduce movement and accelerate token generation.
What are the three biggest bottlenecks in AI hardware right now?
He identifies HBM memory, advanced packaging, and 3nm factory capacity as the key constraints. These are all supply-chain issues that limit how fast the AI industry can scale compute.
How do reasoning models change hardware demand?
Reasoning increases compute per request because models do more internal steps before answering. That makes fast inference more valuable, since slow hardware can make the extra reasoning cost feel crippling.