Summary
The podcast episode features an in-depth conversation with Arvind Jain, CEO of Glean and former Google search expert, discussing the rapid evolution and scaling of AI agents within enterprise search and business automation. Jain envisions a future where every knowledge worker is supported by a personalized team of AI assistants that seamlessly integrate into workflows, boosting productivity without replacing human roles. The dialogue explores the current dichotomy between widely adopted closed AI models from providers like OpenAI and Google and emerging open-source foundational models like LLaMA. Enterprises currently favor closed models due to their maturity and reliability, but cost and scalability factors increasingly drive a transition toward fine-tuned open-source variants, especially for large-scale deployments. Jain highlights regulations and data privacy concerns motivating enterprises to deploy AI on-premises or within controlled infrastructure environments. The discussion defines AI agents as task-automating applications leveraging LLMs to reason and take actions in enterprise systems, often supervised by humans to ensure quality and compliance. Practical examples include AI agents that expedite contract review from weeks to minutes and customer service automation. Glean’s product journey is traced from semantic enterprise search powered by transformers in 2019 to retrieval-augmented chat following ChatGPT’s arrival, culminating in their agentic platform executing millions of real-world actions with complex, multi-step workflows. The platform’s horizontal integration across hundreds of enterprise SaaS applications enables permission-aware secure search and agentic interactions, respecting individual access rights and data governance. Jain emphasizes Glean’s LLM-agnostic architecture that dynamically routes queries between different models like Gemini and GPT to leverage strengths and maintain flexibility amidst fast-changing AI landscapes. The episode also addresses the continuing necessity of human-in-the-loop oversight due to current agent limitations and discusses competitive moats in enterprise AI where data flywheels and integrations form barriers in a commoditized LLM market. The vision culminates in AI becoming a natural, trusted collaborator augmenting human capabilities through proactive co-workers embedded within enterprise knowledge and workflow ecosystems.
Key Takeaways
- 1AI agents are transforming the future workplace by becoming embedded, personalized teammates that augment knowledge workers’ effectiveness across industries.
- 2Enterprises currently prefer closed proprietary AI models like OpenAI’s GPT-4 and Google’s Gemini for critical business use cases due to their maturity, reliability, and enterprise readiness, despite the growing ecosystem of open-source models.
- 3Cost and scalability imperatives drive enterprises to transition from large closed AI models to custom fine-tuned open-source models optimized for specific use cases and production volumes.
- 4Open-source AI adoption in enterprises is also motivated by regulatory compliance, data sovereignty, and the necessity to maintain full control over business-critical AI infrastructure and models.
- 5AI agents function as task-specific applications leveraging large language models for reasoning and automation, tightly integrated with enterprise data systems, but still rely heavily on human-in-the-loop supervision for quality assurance.
- 6AI agents represent an advanced evolution beyond traditional Robotic Process Automation, automating complex knowledge-intensive tasks and delivering dramatic efficiency gains, sometimes reducing multi-week processes to minutes.
- 7Glean’s three-phase product evolution—from transformer-powered semantic search, through retrieval-augmented chat interfaces, to complex AI agentic platforms—exemplifies how enterprise AI adoption has matured incrementally alongside NLP breakthroughs.
- 8Glean’s AI platform architecture is designed for horizontal integration across hundreds of SaaS applications, enabling permission-aware, secure access to enterprise data and empowering AI agents to perform authorized actions within these systems.
- 9Glean maintains a model-agnostic, multi-model integration strategy that dynamically routes queries between different large language models, such as Google’s Gemini and OpenAI’s GPT, ensuring platform flexibility and leveraging the strengths of emerging AI capabilities.
Notable Quotes
"The future that I see with AI is that every person who works is going to have this amazing team of assistants, co-workers, and coaches around them that is going to actually make them a lot more effective."
"Today, for most customer use cases, the companies are using the closed models at the moment. Like, you know, for the ones that at least, you know, that we are familiar with, you know, as we go and work with large enterprises and we think about the core business processes that they want to automate with AI. Right now, the closed models are the ones that tend to dominate. And I think it's for the right reason."
"So initially you start your exercise, you work with the state-of-the-art, the largest models and then you start to fine-tune to still to the right model, like as scale, as you start to get to scale."
"Another one is cost, you know, as I mentioned, that, you know, when you have a large-scale application, you have to actually look at the needs that you have from LLMs and use very specific models for them, because they're going to be both better for user experience because they become faster for you. They also become, they will hallucinate less for you because, you know, they are trained to do a very specific thing. So their range of errors actually also reduces. And then finally, they are more cost-effective."
"An agent is simply, you know, it's an application. It's a, you know, it's a unit of work that is being done. You know, it used to be done by a human before. Now it's being done by, with the help of AI."
"That was a two-week activity that if you actually get an AI agent to redline it for you, it's going to do it in one minute. And then, and it won't be perfect, but now I can go in and look at what the work that AI did and potentially spend, like, you know, two hours instead of, you know, a week to actually go and redline that. And that's more than 90% savings for me."
"So, I think that's sort of where things are today, that there is real value being delivered to customers, but agents are still, you know, better run in a supervised, you know, in a supervised manner, where a human is in charge and looking at the work of the agent. All right."
"We saw a big opportunity and we decided to actually, you know, use that to build a really good enterprise search. That's the origins of how Glean started. So that was phase one. And then there was a phase two, I believe, around the ChatGPT moment when you went from AI-powered search to something that's more like RAG. Is that fair? That's right."
"We could read the knowledge inside those systems, but we could also take actions within those applications. And this was a core part of our assistant product. So when you come in Glean assistant, you could actually say that, hey, do I have the following benefit? Can I actually avail of this benefit or that? But you could also actually do things. You could say that, hey, can you file a PTO for me?"
"So the first thing that we did was in February, we renamed like our app applications product to what we used to call AI applications and now call AI agents. So we don't confuse, you know, the market, you know, we have to sort of adopt the terminology that is becoming more standard in the market. But agents are now getting a lot more powerful. They are, you know, they're getting, they're sort of shifting from sort of basic two-step rack kind of application flow where you take a task, you find some information, and then you make AI work on it to generate the right artifact. You know, you're moving from that to actually running very, very complex business processes. You know, there could be a hundred-step business process that you want to automate with AI now. And so the new, you know, version of agents that people expect now in the industry are significantly more complicated. They have things like evaluations, insights, self-reflection, and, you know, things like those, you know, like, you know, built natively into the core sort of agent functionality."
"Since we started as a search product, where our objective was to help people find, you know, any piece of information that they're looking for inside the company, we built this horizontal platform. We built integrations to hundreds of enterprise applications. And we look at data and information inside each one of those systems. So that when somebody is looking for something, we got able to immediately point them to the right pieces of information."
"Number one, enterprise search is fundamentally different from web search in the sense that the information inside your business is protected. Not everybody can see every document inside the company, unlike on the web, where we can all see all the web pages that are out there. So when you build a search product, you have to understand an individual. You have to understand permissioning of content inside your company. For any document that is out there or for any message that's been exchanged in Slack, you need to know who are the people who have the rights to see that information. And that has to become part of your core search stack. You have to build a secure search experience where, knowing who the user is, you only surface information to them that they have permissions for. So that's a big change."
"Like, you built a retrieval system around AI models. You're also LLM agnostic with different models. I think you talked recently about how you just integrated Gemini. I'm curious about how that works in practice, whether you would route certain query to certain models. And as an addition to the question, a lot of people talk about being LLM agnostic. I just wonder what that means in reality because, as you see, those new models come out. And they behave in a very different way than the prior ones. So what does that mean for a company like Glean? Does that mean you need to just stop what you're doing, look at the new model, evaluate it, and figure out how to integrate it so that you don't end up delivering a, you know, sort of unsettling user experience with, like, very different kind of behaviors?"
"Well, that's the reality of an AI company today is you have to fundamentally learn how to work in an unstable environment. This technology is moving so fast. And so, yeah, so you have to do that. Like, you know, as new models come, you don't have the luxury to not look at them. You have to look at them. You have to see the new capabilities. You have to actually build the right evaluation frameworks to quickly see, like, you know, as a new model comes in, like, we need to have a way within, like, you know, 30 minutes to know how well it's going to do on our product. Right? I mean, and you can do those things. You can actually build the right, you know, evaluation frameworks, things like that."
"I'll give you a simple example. So let's say that somebody asks a question in Slack on some technical topic and some other individual answers, you know, with answers that question and attaches a link to a document that contains the answer. So when you, you can actually learn from this interaction. There are a lot of things to learn from this interaction. Number one, like you've learned that that particular document that was linked in the answer contains the answer to that particular question that the user asked. And that association is going to be super helpful to you in the future when somebody asks a similar question. You're going to use this document with higher, you know, with more confidence than you could."