Inference Is the New AI Stack
Issue
Inference Is the New AI Stack
Distilling Deep AI Insights from Leading Podcasts

Buying Tokens vs Building an AI Factory with Sadeepa Wijesekara - #785
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by TWIML · 58 min
Listen to episode →This episode focuses on how enterprises should think about AI inference costs in the age of agentic workflows. Instead of a simple build-vs-buy decision, companies now have to decide call by call whether a request should go to a frontier model, an open-weight model, or an on-prem/private inference setup. The discussion highlights that agents can consume dramatically more tokens than traditional chatbots, making visibility into token usage essential for cost control and ROI measurement. It also covers practical routing patterns, where enterprises use a single internal endpoint to dynamically send work to the most appropriate model. Security, compliance, and data residency concerns are presented as major reasons many organizations are adopting private or hybrid AI infrastructure.
Key Takeaways
- 01
Agentic AI changes the economics of inference because it consumes far more tokens than conventional chatbot usage.
The episode emphasizes that agents may use 20x to 50x more tokens than a standard chatbot, which means lower per-token prices do not automatically reduce total spend. This makes token budgets a central operational concern for enterprises deploying multi-step AI workflows. The concept is especially relevant for organizations building agents that make many model calls per task.
- 02
Enterprises are shifting from a one-time build-vs-buy decision to continuous routing decisions within workflows.
Rather than choosing a single model or deployment strategy for everything, teams increasingly decide per request whether to use a frontier provider, an internal model, or a local GPU-backed service. This call-by-call routing approach allows organizations to balance reasoning quality, latency, cost, and data sensitivity. It reflects a more flexible architecture for enterprise AI systems.
- 03
Routing and workload placement are becoming core enterprise AI design patterns.
A recurring theme is that enterprises need an internal abstraction layer—often a single endpoint—that can intelligently route requests to different backends. This pattern simplifies application development while enabling dynamic optimization across hosted and local models. Technologies and concepts like workload placement, deterministic routing, and model orchestration are central to this approach.
- 04
Granular visibility into token usage is the foundation for cost optimization and ROI proof.
The episode makes clear that organizations cannot optimize what they cannot measure. Teams need visibility into token spend by agent, use case, and sub-call in order to identify expensive workflows and decide where routing changes will matter most. Without this instrumentation, it is difficult to justify AI investments or compare the economics of different inference setups.
- 05
Security and compliance requirements are pushing many enterprises toward private or hybrid inference architectures.
Sensitive data, regulatory constraints, and concerns about prompt logging or reuse make public-cloud-only deployments less attractive for many companies. The discussion highlights safeguards such as secure sandboxes, short-lived scoped tokens, logging, auditing, confidential computing, and network restrictions. These controls are especially important in agentic systems, where models may access more tools and data than traditional chat interfaces.
- 06
Running your own GPUs can make economic sense when usage is predictable and business value is measurable.
The episode suggests that private infrastructure becomes compelling when enterprises have stable workloads, clear cost-per-job visibility, and enough volume to justify ownership. Appliance-style deployments and turnkey private cloud solutions can reduce operational complexity while keeping inference in-house. This is particularly relevant for organizations balancing total cost of ownership against performance, control, and compliance.

How Business AI Use is Changing
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore · 24 min
Listen to episode →This episode focuses on how enterprise AI adoption is evolving from a narrow, coding-heavy use case into a broader business tool. The main story is that companies are using far more AI tokens while spending only slightly more, suggesting falling token prices and improving cost efficiency. It also highlights a major shift in vendor behavior: businesses are increasingly using multiple model providers rather than committing to a single one, which boosts flexibility and bargaining power. Beyond adoption trends, the episode covers the rise of customer service as an AI use case, the slow real-world rollout of autonomous agents, and broader industry issues like U.S. chip export controls and questions about the independence of OpenAI safety evaluators.
Key Takeaways
- 01
Enterprise AI usage is growing much faster than enterprise AI spending, indicating that companies are getting more value per dollar.
Ramp data shows usage rising 36% while spending increased only 2%, which strongly suggests that falling token prices and competitive pressure among providers are improving economics. This matters because it shows the enterprise AI market is becoming more efficient, not just bigger, and that cost per token is declining quickly. The concept of token pricing is central here, along with the broader economics of inference and model access.
- 02
Businesses are increasingly willing to use multiple model providers instead of relying on a single vendor.
Databricks data suggests a clear shift from single-provider dependence to multi-provider usage, which gives buyers more flexibility and leverage. This is significant because it reduces lock-in and makes model selection a procurement and optimization decision rather than a permanent commitment. Relevant concepts include model routing, vendor switching, and multi-model strategy.
- 03
Coding remains the dominant enterprise AI use case, but customer service is becoming a major growth area.
Engineering still accounts for the largest share of AI spending, but customer support spend has risen sharply, showing that AI is spreading beyond technical teams. This matters because it signals a broadening of AI value across the organization, especially in high-volume, repetitive workflows. Technologies and concepts relevant here include copilots, support automation, and workflow-specific AI deployment.
- 04
Autonomous agents are still more aspirational than operational in most enterprises.
The episode emphasizes a gap between excitement around agents and actual deployment, with agents ranking high in interest but low in use. That distinction is important because it suggests the market is still early and that reliability, control, and integration remain barriers. The key concept here is agentic AI, especially autonomous task execution and enterprise readiness.
- 05
The main competitive advantage in enterprise AI may be cost efficiency and flexibility rather than raw model capability.
The episode argues that the winners will be companies that can negotiate well, switch models easily, and optimize for cost per successful task. This is significant because it reframes AI success as an operational and economic problem, not just a model-quality problem. Concepts like procurement strategy, unit economics, and performance-per-dollar are central to this view.

Can AI Learn to Improve Itself? | Recursive Self-Improvement
a16z Podcast by a16z · 17 min
Listen to episode →The episode explores recursive self-improvement (RSI) as a practical research acceleration problem rather than pure sci-fi autonomy. Mark Chen and John Schulman describe how AI is already acting like “research interns,” helping with experiments, coding, automation, and parts of the training pipeline. A major theme is that current models can execute useful scoped tasks, but they still struggle with research taste, long-horizon judgment, and deciding which ideas are worth pursuing. The discussion also highlights how hard it is to measure progress outside of pre-training, where metrics like perplexity are clear, and how compute constraints limit the pace of research. Finally, the episode argues that stronger AI may both increase alignment risk and become one of the best tools for solving alignment itself.
Key Takeaways
- 01
RSI is currently best understood as AI helping accelerate AI research, not as fully autonomous machines improving themselves end-to-end.
The speakers frame the present-day version of recursive self-improvement as models assisting humans with experiments, coding, and workflow automation. This is significant because it shifts the conversation from a distant AGI milestone to an already-visible productivity gain in frontier labs. Technologies and concepts involved include AI-assisted experimentation, automated research pipelines, and model training workflows.
- 02
AI can act like a capable research assistant, but it still lacks research taste and independent judgment.
The episode emphasizes that current systems can complete narrow tasks well, yet they are not reliable at choosing the right problems or recognizing which ideas will matter long term. This matters because research progress depends not just on execution speed, but on selecting high-value directions and learning from failure. The concept of “research taste” is central here, along with scoped task execution and human-in-the-loop oversight.
- 03
Measuring progress in RSI is much harder than measuring progress in pre-training, because downstream research lacks stable benchmarks.
Pre-training offers clearer signals such as perplexity, but research quality is more open-ended and benchmarks can become obsolete quickly. This makes it difficult to know whether AI is truly improving research capability or just optimizing for narrow metrics. The significance is that evaluation frameworks themselves may need to evolve, especially for benchmarking model usefulness in open-ended scientific work.
- 04
Compute is still a major bottleneck, and future gains may depend on AI helping design smaller, more predictive experiments.
The speakers suggest that current progress is limited by the scale and cost of experiments available today. If AI can help researchers choose better experiments, it could reduce wasted compute and speed up discovery. Relevant concepts include compute scaling, experiment design, predictive experimentation, and eventually using AI to help improve compute infrastructure itself.
- 05
Alignment may be both harder and easier in a world of RSI: stronger models could create more risk, but they may also be the best tools for fixing alignment failures.
The episode argues that as models become more capable, they can surface more safety challenges, but they can also help researchers identify and correct them. This is important because it reframes alignment as a problem that may require increasingly capable AI systems to solve. Technologies and concepts mentioned include alignment research, automated debugging of model behavior, and self-reported confession-style training methods.
Questions answered in this issue
Why don't lower model prices automatically translate into lower enterprise AI bills?
Because agentic workflows consume far more tokens than chatbots, often 20x to 50x more. So even if token prices drop, total usage can push the bill upward.
What is the first step an enterprise should take before optimizing AI costs?
It needs granular visibility into token spend by agent, use case, and sub-call. Only then can it decide which workloads should use frontier models, open-weight models, or on-prem inference.
Why are enterprise AI costs rising more slowly than usage?
Because token prices are falling as competition increases among providers. The episode says companies are consuming more tokens while spending only slightly more, which implies better pricing and stronger vendor competition.
What does the episode suggest about model switching behavior in businesses?
Businesses are increasingly comfortable switching providers or using multiple providers at once. Databricks data shows a clear move away from single-vendor dependence, which gives buyers more negotiating power.
What does RSI mean in this episode?
RSI stands for recursive self-improvement, but the speakers use it mainly to describe AI helping improve the AI research process itself. In the narrowest sense, it would mean models independently generating better models, but the current practical version is more like AI-assisted acceleration.
What can AI do in research today?
AI can run experiments, automate parts of the research pipeline, and help with scoped tasks like implementing a specific idea or monitoring experiments. However, humans still need to define the goal, steer the work, and evaluate whether the output is actually valuable.
Explore