Can Audits Keep Up With Agentic AI?
Issue
Can Audits Keep Up With Agentic AI?
Distilling Deep AI Insights from Leading Podcasts

The Time Savings Era of AI Is Over
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore · 20 min
Listen to episode →The episode reviews results from the AIDB January AI Usage Pulse survey, arguing that AI value is shifting from simple time savings toward increased output and entirely new capabilities. Heavy users are adopting agentic workflows and multi-model portfolios, with Claude emerging as the most common primary model for builder- and agent-oriented use cases. Vibe coding and low-code/no-code creation have spread beyond engineering, enabling executives, operators, and product teams to build their own AI-driven tools. The conversation highlights vendor case studies (e.g., Blitzy) and new product offerings (e.g., Superintelligent's AI Strategy Compass) as evidence that tooling maturity is accelerating enterprise transformation.
Key Takeaways
- 01
The dominant AI benefit has moved from time savings to increased output and new capabilities.
Survey respondents ranked increased output/throughput (38%) and new capabilities (22%) above time savings (20%), indicating that advanced users see AI as transformational rather than merely efficiency-enhancing. This shift matters for how teams measure ROI and position projects—metrics should emphasize throughput, revenue enablement, and capability expansion instead of just hours saved.
- 02
Agentic AI adoption is rising and is concentrated among heavy, builder-oriented users.
More than a third of respondents (37.6%) report agentic usage, with Claude primary users showing especially high agentic adoption—suggesting autonomous agents are moving from experimental to production workflows. Increased agentic use changes design, governance, and safety priorities because agents plan and execute multi-step tasks rather than simply assist.
- 03
Multi-model portfolios are the norm, not the exception.
The average respondent uses about 3.5 models and only ~5% rely on a single model, so organizations should design for interoperability and model-switching rather than one-model lock-in. Product and procurement strategies must account for mixing models (ChatGPT, Claude, Gemini, Copilot, etc.) across different use cases to optimize cost, performance, and capabilities.
- 04
Vibe coding and low-code/no-code tool building have democratized AI development beyond engineering.
Nearly half of coding-like activity is coming from non-engineering roles, and vibe-coding adoption is mainstream among the surveyed audience, enabling executives, product managers, and operators to ship tools. This broadens who can create internal automation and accelerates internal tool development, but it also raises questions about governance, quality control, and maintainability.
- 05
Tooling maturity and vendor platforms are accelerating enterprise transformation with measurable speedups.
Case studies like Blitzy (e.g., delivering a payments app in six weeks vs. a 13-month estimate) illustrate how AI-assisted platforms and agentic tooling can materially compress SDLC and modernization timelines. As vendors productize playbooks (e.g., Superintelligent's AI Strategy Compass), transformation work shifts from bespoke consulting to repeatable, productized workflows—changing cost structures and scaling potential.
Also in this issue

AI incidents, audits, and the limits of benchmarks
Practical AI by Practical AI LLC · 42 min
Listen to episode →The episode examines the gap between research benchmarks and real-world AI safety, drawing on Sean McGregor’s work with the AI Incident Database and the AI Verification & Evaluation Research Institute. It emphasizes that practical AI is defined by systems that produce real-world consequences, and that benchmarks and lab tests often fail to predict brittle failures in deployed systems. The conversation covers sourcing and classifying incidents, challenges of voluntary reporting versus potential mandatory reporting, and the scale trade-offs of indexing many small harms versus focusing on high-impact events. The hosts also discuss the role of third-party audits, lessons from red-teaming (e.g., DEF CON exercises), and the need for new evaluation approaches for general-purpose models and composed systems.
Key Takeaways
- 01
Focus safety work on deployed systems with real-world consequences rather than academic demos.
Practical AI is defined by impact; systems that interact with people or environments create the harms that matter for safety practice. Prioritizing incidents from deployed systems guides evaluation, regression testing, and incident collection efforts toward outcomes that affect users, rather than optimizing for benchmark scores that may not translate to production.
- 02
Benchmarks and lab tests are necessary for research but insufficient as deployment guarantees.
Benchmarks often target controlled prompt spaces and component behavior, so they miss emergent and contextual failure modes that appear in the wild. Relying solely on leaderboard results can produce false confidence—organizations need domain-aware testing, incident-driven regression suites, and meta-evaluation (checking the evidence behind claims) to assess real-world risk.
- 03
Incident collection at scale enables pattern discovery but faces sourcing limits that regulation could address.
The AI Incident Database (>5,000 human-annotated reports across 1,000+ incidents) shows there is a sizable corpus for analysis and safety improvements, revealing common failure modes and motivating mitigations. However, most entries come from journalism or voluntary submissions; mandatory reporting (as proposed in some regulatory drafts) could improve coverage and provide provable evidence of harms.
- 04
General-purpose frontier models make exhaustive verification infeasible, requiring new verification and high-level guarantees.
Because these models operate across diverse contexts and distributions, safety assessments that assume a single operating context break down and exhaustive testing is impossible. The field needs alternative strategies—top-level guarantees, compositional evaluation, continuous monitoring, and new verification science—rather than only context-limited validation.
- 05
Third-party audits are increasingly essential to build trust and validate vendor claims.
Independent audits function like financial audits: they examine evidence, validate documentation, and can catch blind spots internal teams miss—important for investors, partners, and regulators. As AI systems become central to business operations, audits become table stakes for risk management, similar to compliance and financial assurance practices.
- 06
Red-teaming and security-style testing reveal exploitable integration and handoff failures that benchmarks miss.
Events like DEF CON generative red-team exercises show how guard models, handoff strategies (reject/reprompt/pass-through), and composition with LLMs produce unexpected vulnerabilities that can be monetized or weaponized. Incorporating adversarial testing, bug-bounty style disclosure, and operational flaw adjudication helps surface real attack vectors beyond what static benchmarks expose.
Questions answered in this issue
How many survey respondents use AI daily?
97.6% of respondents reported using AI daily, indicating very high engagement in this surveyed group.
Which models had the broadest recent usage and which was chosen most often as primary?
ChatGPT had the broadest reach (87% used last month) but Claude and Gemini each had ~80% recent usage; Claude was chosen as the primary model most often (45.8%).
How does the speaker define 'incident' for the AI Incident Database?
The intergovernmental/workshop definition used is effectively 'an event where a harm has taken place.' The team chose the word 'incident' because it is sufficiently broad to encompass accidents, adverse events, exposures, controversies, and harms while still being meaningful.
What are the primary sources for AI incident entries today?
Most entries currently come from journalistic reporting, supplemented by direct submissions (emails, forms, blog posts). Journalists do substantial validation work, but declining journalistic capacity and proprietary systems limit coverage.