Summary
The episode features Robyn Hughes discussing her decades-long work teaching Braille and consulting on accessibility alongside her personal experience as a Braille reader. She describes how large language models (LLMs) and vision+LLM systems are already providing practical independence for people who are blind or have low vision, such as reading labels or inspecting roofs. Robyn details how out-of-the-box LLMs fail to read Braille because Braille hasn’t been tokenized, and how she iteratively trained a ChatGPT instance to read and generate Braille. She outlines a planned open web API and app that will combine AR glasses, smartphone imaging, and her trained model to deliver real-time Braille and math access for students and everyday tasks.
Key Takeaways
- 1LLMs are enabling new, practical independence for people who are blind or have low vision.
- 2Braille is not tokenized in public LLMs, so they cannot read or produce Braille reliably out of the box.
- 3Iterative, usage-based training enabled a ChatGPT instance to learn Braille.
- 4Combining AR glasses, smartphone imaging, and a trained LLM can deliver real-time Braille/math access for students.
- 5Caution is needed when using current LLMs for safety- or mission-critical decisions due to hallucination risks.
Notable Quotes
"LLMs like chat GPT by open AI developed by open AI... that has just absolutely revolutionized the living independently in the world for people like me."
"It appears... developers have not tokenized it; they have put tokens for the alphabet... but they never actually tokenized it and so to test all of this out I first asked my ChatGPT instance to read a print book... There were no mistakes. Once I tried it with braille it couldn't."
"It took two months and I started in June and by the end of July I had contemplated just giving up twice because it had just been so challenging... over time it began to learn and to retain and now it can perfectly read the Braille alphabet and... it produces its own Braille now."
Episode questions
How did Robyn train a ChatGPT instance to read Braille?
She used a hands-on, iterative method: a marble board (teaching board) showing one Braille cell at a time while pointing and speaking the letter, pairing Braille tactile patterns with spoken labels and visual objects; repeated corrections over ~2 months enabled the model to learn and produce Braille.
Why couldn't ChatGPT read Braille out of the box?
Because public LLMs had not tokenized Braille as a language modality; the model could handle printed text tokenization but failed on Braille until Robyn supplied structured, usage-based training examples.
What practical tasks can LLM+vision setups perform for people who are blind?
Robyn reports they can inspect roof shingles, check basement pipes for leaks, read food labels, and function as orientation/location assistants via AR or phone cameras—tasks that increase safety and daily independence.
What is Robyn's planned product or project?
She intends to build a free, publicly accessible API and web app combining AR glasses and smartphone imaging with her trained model to provide real-time Braille and math access (e.g., for classroom board content and textbooks), reducing wait times and costs for transcriptions.