Ep 47: Chief AI Officer of Databricks Jonathan Frankle on Why New Model Architectures are Unlikely, When to Pre-Train or Fine Tune, and Hopes for Future AI Policy
Summary
In this episode, Jonathan Frankle, Chief AI Scientist at Databricks, discusses several critical challenges facing AI development and deployment. He emphasizes the difficulty of transitioning AI models from experimentation to reliable production systems that users can trust. A significant point of concern is the lack of clear best practices for fine-tuning and selecting models, with a particular focus on whether to employ Retrieval-Augmented Generation (RAG) or fine-tuning. Despite the existing hurdles, Frankle expresses optimism regarding the potential of AI technologies, particularly generative models, while also stressing the importance of model explainability and reliability. He advocates for an iterative approach to AI testing, suggesting that companies should prioritize learning and development over the pursuit of perfection before deployment. Additionally, the conversation delves into the evolution of evaluation benchmarks and the necessity for human-in-the-loop testing to better align models with real-world complexities. Frankle addresses the ethical implications of AI in sensitive fields like healthcare and law, focusing on the responsibility of developers to ensure accuracy. The discussion also highlights the competitive landscape of AI startups and the need for businesses to make strategic decisions between general-purpose and domain-specific models, as well as the importance of collaboration in the AI ecosystem.
Key Takeaways
- 1Transitioning AI Models to Production Requires Focus.
- 2Iterative Development is Key to AI Success.
- 3Clear Guidelines Needed for Model Fine-Tuning Decisions.
- 4Human-in-the-Loop Evaluation Enhances Model Effectiveness.
- 5The Cost-Quality Trade-Off is a Persistent AI Challenge.
- 6Ethical Implications of AI Need Ongoing Attention.
- 7Industry Collaboration is Essential for Progress.
- 8Understanding AI Model Lifecycles and Sustainability is Critical.
- 9Investment in Domain-Specific AI Models can Yield Competitive Advantages.
Notable Quotes
"You don't try to get your data perfect in advance. It's all kind of one process."
"The first part is knowing it's not going to be perfect."
"Being a little wrong is okay. Sometimes AI is fuzzy. It's its superpower and it's a kill seal."
"I think there's a lot of work to do as a field."
"I don't think we figured out how to help people navigate the space of RAG versus prompting versus fine-tuning yet."
"We're pushing to higher quality, but pushing to perfection or pushing to kind of, you know, getting that, you know, that fifth nine or what have you of quality."
"For a lot of folks, they're getting that use. And then full-on pre-training is the same thing."
"Databricks is not a self-contained universe. We partner with anyone and everyone. You can look at our partner list; it includes a lot of these amazing startups."
"So my view is like, let's make Databricks amazing for the things we're great at. Let's find the best stuff out there and make it as easy as possible for our customers to use it."
"Jumping into this... but as you go on that journey, you'll know if it makes sense because you're getting usage to justify."
"You can't really do AI until your data is perfect... but the measure of your evaluation is— can you build a useful AI system?"
"I would suggest starting small and working your way up, testing what partners can actually do without waiting for every part to be perfect."
"The only way you can figure out if AI is suitable for your needs is to experiment with it."
"I think it's really risky. You can affect someone's life and liberty. You can, even if you don't actually throw them in jail, even if you don't arrest them, you can still turn their home upside down by searching them."
"'I think we should be thoughtful about where we want to use this technology or at least where we really don't want to use this technology.'"
Episode questions
What key strategies does Frankle recommend for organizations just starting their AI journey?
Frankle advises organizations to begin with small, manageable projects and gradually increase complexity. He emphasizes that waiting for perfect data conditions hinders progress. By starting with existing models or simple prompt engineering, companies can learn and grow their capabilities without extensive upfront investment.
How does Frankle suggest organizations evaluate whether to adopt new AI technologies?
He recommends establishing clear metrics that align with business objectives, thus ensuring that any new technologies deployed bring measurable benefits. He underscores the need for ongoing learning and adjustment based on performance outcomes.
How should businesses approach model training and evaluation in an iterative manner?
Businesses are encouraged to embrace an iterative model development cycle, where deploying a preliminary version of a model for real-world testing provides insightful data to make improvements. By not striving for initial perfection, companies can leverage feedback from actual use cases to determine if a model's performance is acceptable or if adjustments to the data or approach are needed. This process allows for timely adaptations that keep pace with real-world applications rather than getting bogged down by extensive pre-launch refinements.
What practices can enhance evaluation benchmarks in AI?
Regularly updating evaluation benchmarks to reflect current state-of-the-art advancements is crucial. AI teams should incorporate broader methodologies, including human-in-the-loop feedback, to ensure benchmarks stay relevant in functional metrics. Continuous dialogue with end-users about practical constraints and needs helps refine the metrics that truly represent model efficacy under real-world stress.