Summary
The episode examines the development and implications of the AI Arbitrator, an AI-assisted arbitration platform built by the American Arbitration Association for narrowly scoped, documents-only construction disputes. Bridget McCormack and the host discuss why AAA chose a limited, human-in-the-loop design—grounding agents in domain-specific handbooks and historical case libraries—to reduce risk from hallucinations and credibility assessment errors. The conversation weighs potential benefits, notably expanding access to justice by lowering cost and friction for routine disputes, against concerns about transparency, fairness, and who controls the system. The episode also explores broader institutional questions about trust, the limits of automation (e.g., not suitable for criminal or government actions), and the safeguards needed to audit and de-bias AI-driven decisions.
Key Takeaways
- 1Start narrow and keep humans in the loop to manage risk.
- 2AI-assisted arbitration can materially expand access to justice for routine, high-volume disputes.
- 3Perceived fairness depends on governance, transparency, and who pays for the service.
- 4Technical safeguards are necessary because LLM agents hallucinate and can encode bias.
- 5Not all disputes are appropriate for automation—public-interest and credibility-heavy cases should stay in courts.
Notable Quotes
"For the past several years, Bridget and her team have been developing an AI-assisted arbitration platform they call the AI Arbitrator."
"We built an AI native case management system for the AI arbitrator to operate on...the arbitrator is...a bunch of different agents, probably 20, sometimes more."
"If you were to just take your dispute, all the documents ... and throw it into ChatGPT or into Claude, you could get a result right now...But that's not what we built, and that's why we're moving so narrowly and so slowly."
"We keep a human in the loop to make sure that before an award issues there were no hallucinations...And we're going to be very transparent about all of our audits."
"If we could really resolve every dispute... we'd have a better world."
"You can de-bias a data set a lot easier than you can de-bias a human."
"I'd be very surprised if in 15 years people are still opting for a slow, human-driven... process for documents-only B2B disputes."