Summary
This episode explores how to build reliable long-horizon AI agents that can perform real work, not just complete coding tasks. Mitch Troyanovsky of Basis argues that accounting is an ideal proving ground because it is structured, high-stakes, and tied to real economic outcomes. The conversation traces the evolution from early agent frameworks like ReAct and BabyAGI to more capable reasoning models such as Opus 3, o1, and o3, which improved long-context state management and self-regulation. A major theme is that agents should be evaluated by process, not only by final output, especially when there is no clean ground truth. The episode also introduces Basis's approach to behavior specs and the broader idea that context, documentation, and ontology function like runtime training data for agents.
Key Takeaways
- 1Accounting is a strong domain for autonomous agents because it combines structure, scale, and direct economic consequences.
- 2Long-horizon agent capability improved significantly once models became better at extended reasoning and self-regulation.
- 3Context should be treated as runtime training data, not just as a prompt.
- 4Reliable agents need process supervision, not just outcome-based evaluation.
- 5Behavior specs provide a practical, markdown-based way to define and evaluate agent behavior across trajectories.
- 6Ontology and documentation function as the agent’s world model, so they must be designed like infrastructure.
Notable Quotes
""Humans are already used to working with non-determined systems. It's just the systems that normally they're co-workers, not their computers.""
""Even if you got it right 100 out of 100 times, if a person is just getting it right because they're going to Wikipedia, they're kind of from when it hire them.""
""The framework I like is thinking about it as training data except you're just training the model at runtime.""
""The thing that someone is buying from us is not this will be the best ever tax return. They're buying that it's going to be consistent and reliable and something that they can trust.""
""not thinking that an agent operating over 10 hours is a black box it's not it has a lot a lot a lot of data and you're probably doing a disservice to your customers if you don't understand like how it's going about the work""
""your context is actually the repo they're working on""
""the English is more precious because the English affects the performance the code does not affect the performance""
""technical modes are not real modes""
Episode questions
Why did Basis choose accounting as its first major autonomous-agent domain?
Because accounting is a huge knowledge-work category, highly structured, and tightly connected to real economic activity. It also has verifiable artifacts like tax returns, trial balances, K-1s, and primary sources that make it a good testbed for long-horizon agents.
Why isn't it enough for an agent to get the right answer at the end of a long task?
Because the same output can come from a bad process, like relying on weak sources or failing to follow professional norms. Mitch argues that in production, especially for accounting, you need confidence in both the result and the path taken to reach it.
What are behavior specs in Basis's system?
They are markdown-written specifications that define how an agent should behave, often across a full trajectory. They act both as a product spec and as a rubric for judges evaluating whether the behavior happened.
Why does Mitch say context is like runtime training data?
Because the agent learns and adapts during execution based on the information it is given. That means the quality and structure of context directly affect behavior, just as training data shapes a model during pretraining or post-training.