Lil-Vro Model

Inspiration

Kids can easily turn to a chatbot for an answer or get distracted by a screen. We wanted to build something different: a study buddy that makes kids explain their thinking out loud and guides them toward the answer instead of doing the work for them. Lil-Vro is designed to feel like a slightly older classmate—someone who asks the right questions, catches mistakes, and helps you work through a problem without simply giving you the solution.

What it does

Lil-Vro is a voice tutor for school. Its home turf is STEM: it understands spoken math and chemistry and talks equations back like a person. It also helps with history, psychology, and other subjects, and it's a friendly voice a kid can talk to about things outside of class. Its focus stays on helping kids get through their schoolwork. Instead of dumping answers, it asks one leading question at a time, in three modes: -Walkthrough: it guides you through your problem, one step at a time. -Teach-back: you explain a concept, and Lil-Vro plays the curious friend who asks the hard follow-ups. -Mistake mode: Lil-Vro hides an error in a problem, and your job is to catch it. It reads frustration and confusion and changes its tone, keeps every reply age-appropriate, and refuses to tell a kid a wrong answer is right. Parents get a dashboard with streaks, topics covered, and per-topic mastery.

How we built it

Lil-Vro is a full voice-to-voice pipeline we built ourselves. A child speaks, and faster-whisper transcribes the audio locally so raw audio stays on the laptop. Our normalizers convert spoken math and chemistry into structured notation, then an OpenAI model uses Socratic prompts, mood tracking, and mastery tracking to create a guided reply. A content filter checks the response, our math-to-speech renderer converts equations back into natural English, and Deepgram speaks it aloud. An ESP32-S3-BOX-3 streams 16 kHz audio over Wi-Fi, creating a real screen-free talking device. The Nordic nRF54LM20 DK acts as a physical controller, sending button events to the Python server over Bluetooth LE to pause or resume listening. Its LEDs show whether Lil-Vro is idle, listening, thinking, speaking, or paused, giving the child clear feedback while helping prevent unwanted microphone input and echo. Session activity can be stored in Supabase and displayed through our Next.js and TypeScript parent dashboard.Individual Contributions

Chai: Built the core pipeline and agent loop. Wrote the spoken-math normalizer (99-symbol lexicon, context-aware parsing, set-builder notation, three speech modes) and a chemistry normalizer (15/15 tests passing). Set up local speech-to-text with RealtimeSTT and faster-whisper, plus noise filtering, echo cancellation, speech styling, the mood tracker, the content filter, and the answer-verification and honesty guardrails. Kyle: Built the symbol-aware math-to-speech renderer that turns notation into natural English. Wrote the kid-safety guardrails: a rule-based check first, an AI classifier as fallback, and a fail-closed default that blocks anything unclear. Added mood detection, the OpenAI and Deepgram integration, cross-platform audio fixes, and unit tests for the renderer and guardrails. Andy: Designed the Supabase Postgres schema (8 tables covering profiles, children, devices, sessions, turns, mastery events, mastery scores, and child stats) with row-level security and streak-updating session storage. Built the Next.js 15, React 19, TypeScript, and Tailwind dashboard with Supabase Auth. Wrote the utterance gate, tool routing, teach-back and mistake modes, and barge-in. Sam: Built the ESP32-S3-BOX-3 transport server and Nordic development gateway. It takes 16 kHz mono PCM audio streamed over Wi-Fi, forwards it to Deepgram over a WebSocket, passes the transcript through the normalizer and agent, then synthesizes the reply with OpenAI TTS and streams it back to the device. Also wrote the teach-back integration and the first Deepgram microphone transcription.

Challenges we ran into

Building a voice tutor introduced problems that don't exist in traditional text-based applications. Students frequently pause in the middle of a thought, so we had to distinguish between a natural pause and the end of a response. We also had to prevent Lil-Vro from transcribing and responding to its own voice. Spoken mathematics was another major challenge. Expressions such as "5 squared" and "3x" could easily be transcribed incorrectly or lose their mathematical structure. We built a custom symbol lexicon and context-aware normalization system to handle these cases. Most importantly, we had to make the tutor honest. An AI tutor that confidently tells a student their incorrect answer is correct can be worse than having no tutor at all. We added answer verification and safety guardrails so Lil-Vro can acknowledge uncertainty rather than confidently guessing.

Accomplishments that we're proud of

-It works end to end. A kid talks, and Lil-Vro talks back, with math that sounds like a person said it. -It runs on real hardware, an ESP32-S3-BOX-3, so it's something you can put on a desk rather than just a program on a laptop. -It's a tutor that won't hand over the answer, and won't tell a kid a wrong answer is right. -We built the math and chemistry speech handling from scratch, with a 99-symbol lexicon and automated tests. -The parent dashboard runs on a real database, so parents can see streaks, topics, and mastery.

What we learned

-Voice is harder than text in ways we didn't expect. Interruptions, echo, and awkward pauses are problems a screen never has. -When you're building for kids, safety and honesty have to come first, not get added at the end. -A tutor that agrees too easily is broken, so we treated it like a bug. -Splitting the project into clear pieces with shared contracts let all four of us work at the same time without breaking each other's code.

What's next for our project

Next, we're making Lil-Vro truly independent: nRF54LM20 DK & a standalone BOX-3 with an on-device wake word, tutoring that reaches beyond STEM into history, psychology, and writing, and new Mistake Mode topics like fractions, derivatives, and chemistry. Each session will end with a spoken recap so students and parents can see progress, and we'll build the parental consent and sensitive-topic safeguards that real families deserve.