The Oracle: a kinetically augmented device for reflection

I saw that there was a competition on Hackster.io around the new Arduino Uno Q. What caught my eye was being able to do inference locally, off-line.

My projects tend towards tangible experiences that relate to concepts or make folks think. One that was banging around my head was some object that would aid in narrative generation, to prompt reflection.

In brief, the random laying of cards of a Tarot deck helps the reader generate a narrative that can be instructive to the reader.

The idea banging around was similar, but with rolling dice.

My initial thought was to cram electronics into a 8-sided printed die, and that would send positional info to the Q for interpretation and display.

But then I realized I could flip things (literally) and have a dice cup with the electronics reading the rolled dice.

The Oracle

The Oracle is a reflective physical-computing construct with an instrumented dice-rolling cup, a pair of dice, and an Arduino Uno Q.

In short, the user shakes the dice in the cup and inverts it to make their roll. In the base of the cup is a battery-powered Seeed Studio XIAO ESP32S3 Sense with a camera and speaker (I'll call it just the Sense). At the end of the roll, the Sense captures the image of the dice and sends it to the Q, which then records the position and counts the number of pips on the dice. Then, the LLM on the Q interprets the dice and outputs a poem that plays on the speaker in the cup. The poem is the thought-provoking tool to inspire the user to formulate a helpful narrative, to prompt a reflection.

How we built it

For the cup electronics, the key features I needed were small size, a camera, wireless connectivity, SD card, battery-powered, and a speaker. The Sense ticked most of those boxes. I added an Adafruit I2S Audio BFF to connect a mini oval speaker.

Note: before soldering the BFF to the Sense, I soldered two wires to the battery pads, which would eventually be inaccessible once the BFF was added. And as I wanted to turn this thing on and off, I added a switch to the battery circuit.

Note: the BFF is powered by the 5V pin, NOT the 3V3 pin. So when on battery power, the Sense will not power the BFF. I put a Schottky diode to safely feed the 5V pin with power from the 3V3 pin.

Note: I made the battery pack and switch with connectors so I could easily assemble and disassemble.

Two other things added to the Sense were a simple wire, attached to a touch pin, to interact with the Sense to take the picture, and an LED that would illuminate the inside of the cup. Spoiler: that single LED was woefully too dim. So I maybe overdid it and swapped it out for a 10-Neopixel strip (inside a silicone sleeve) that would be on the inside of the cup, for illumination.

Note: the strip also has a connector, so that I would not be stuck with a soldered wire going thru the cup base.

For the cup, I bought one with a small compartment in the bottom, normally used to store dice. I drilled holes for the camera and the Neopixel cable.

Note: one of the most annoying things was the ragged felt edge of the camera hole - I needed to make sure it didn't interfere with image. And, yep, it occasionally did.

Camera testing

I like to test things in units rather than go full build and then test things. So, with the basics of the dice electronics working, we coded up the image capture and transfer and info return.

I used Val Town as glue between the Sense, Claude (API service), and Google Text-to-Speech (API service). The Sense captured a dice image, sent it to Claude, who then read the dice and sent back some text, which was passed through Google TTS, which sent a.wav file to be played on the speaker.

Really cool. Concept working.

First two readings:

Your stillness holds more than you know.
What stirs beneath the surface asks to be felt.
Listen, there may be an answer in the waiting.

Your shadow falls on cloth,
a stranger's gaze turns outwards
the door swings both ways.

Note: the code on the Sense is in CircuitPython. Because I insist. Haha. The libraries we used were espcamera, audiobusio, audiomixer, audiocore, neopixel, touchio, sdcardio, adafruit_requests. And, as I discussed later on, we made heavy use of the Web Workflow for wireless code dev.

Note: We did run into some physical issues with the Sense. The back and forth of putting the Sense in and out of the dice cup would often jostle the SD loose and put the Sense in safe mode. Also, you can use Web Workflow OR use USB, not both. This popped up whenever I was charging the battery and trying to do some code over WiFi.

Attacking the Arduino Uno Q

The whole point of the contest was to show off the AI capabilities of the Q. So we needed to shift the on-line workload (glued thru Val Town) off-line and onto the Q.

We knew we wanted to use the inbuilt LLM brick. Gemma proved (sorta) better than the other included LLM. And we knew we'd need to build an Edge Impulse pip-reading model for counting the position and pips of the dice. App Lab supposedly was well integrated into Edge Impulse.

Reading the dice

To build the vision model, we needed a training set. I wanted to use training based on the view the camera would have inside the cup. And, the tedious part of AI, that folks don't realize, is the need to annotate the images.

For identifying, cropping out, and prepping the dice images for Edge Impulse, we used OpenCV. OpenCV was sorta OK making the bounding boxes around the dice. With the ones it messed up, Claude Code hand-tweaked them, sometimes with spot-checks from me (together, they were better at cropping than individually). Neither Claude Code, nor OpenCV, were good at calling the pips for annotation. So I did that myself clicking thru the individual images and recording their pips. This was not too onerous. As we did multiple training runs, we did pip-calling in batches and I could quickly flip thru in my MacOS finder and dictate (I'm not a masochist) to capture the info.

Note: A challenge we had was to know if the image was being captured properly by the Sense, and that all the images we took were consistent, to avoid capturing dozens of images and then find out they are all bad. At the start, I kept having to disassemble the Sense to get to the SD storing the images. But after a bit of exploration, we started using the Web Workflow. We were able to edit code and download the images. Alas, the Sense was slow and finicky, but with some patience, we were able to complete the project mostly wirelessly, not plugged into computer. What's more, Claude Code worked up a local html page that would make the network calls to browse the contents of the SD card. In that way I could inspect the images as I took them to make sure the camera was working as intended.

Because I was using Claude Code, I was able to use the Edge Impulse APIs to work with Edge Impulse. All the uploading of annotated images, running and reviewing the model generation, and testing the image detection model were thru this API. This made the process fast and responsive for us.

Note: I only noticed later that Edge Impulse actually offers a tool to help annotate images. I did not test it, as I only noticed that at the end.

I was trying to use as few images as possible. So we ended up having to repeat the model-making (very easy once we figured it out) until we had enough images to make a robust pip counting model. That took about 300 images. Along the way, we made sure the die faces were equally represented, and they were in various orientations and lighting shading. Edge Impulse works its magic to get the most out of the data set and returns the best fit model. Also, we were able to test the model in Edge Impulse before deploying it to the Q.

Note: Edge Impulse is supposed to be well connected to the bricks in App Lab. In the App Lab object-detection model-picker, I did get the option to make a custom model, which took me to Edge Impulse. And then in Edge Impulse when I deployed the model, it let me click back to App Lab, to then use the model. But for some reason, this automated transfer didn't work. Because of this, we ended up not using the object-detection brick. We had to load the model manually onto the Q, bypassing the object-detection brick entirely, using custom Python that loads the model based on Edge Impulse's own SDK. Tho interestingly, we had to fix a bug in that SDK that was silently truncating large payloads, making the process hang. We spent an inordinate amount of time figuring out these usability bugs. Crossing fingers for next versions.

Where are we at?

At this stage we have the Sense in the cup, taking pictures. We have a pip-counting model. Now we needed to glue that together thru the Q.

Introducing, SuziQ

Whimsically, I called my Q 'SuziQ'. So, I'll call it that thru the rest of this piece. Haha.

We needed to do a few things with SuziQ. To start, we built an Oracle app in App Lab to tie the various code pieces together. One major bit of code was the pip-counting model from Edge Impulse (not a brick). The other was the App Lab LLM brick running Gemma. We also included Piper TTS for local speech synthesis (a Python library, not a brick). As the glue (much like Val Town) we used the FastAPI (from the web_ui brick) to manage the HTTP endpoint thru which the Sense and SuziQ communicated.

Those were the core parts for the whole round trip: Sense captures the image, posts it to SuziQ, which then uses the model to detect both dice, then uses the LLM to generate a poem, which gets sent back to the Sense as an audio file to speak the poem on the speaker.

Note: At the start, we had a weird issue where the LLM kept returning versions of the same poem. When we looked, we saw the local LLM brick enables conversation memory by default. So each exchange was adding to the history, which is why roughly the same poem kept coming back regardless of the dice rolled. We had to explicitly change this per request, to forget history.

For debugging purposes, we added a few features. We built a webpage using the FastAPI, so that we could see what images were sent, what poems were written, and listen to the poem audio. Under that, we have SQLite-backed history with replayable readings (using the dbstorage_sqlstore brick), saved photos for both successful and failed detections, some diagnostics, and a SuziQ-side image browser.

Furthermore, there's that 8×13 LED-matrix just sitting there on SuziQ, so I used that for immediate feedback: breathing on idle, dice number when pips are read (it shows two numbers if the dice were read, or two X if it failed), and a cancellable scrolling of the poem. There is some simple glue code in App Lab that allows the CPU side of the SuziQ to send info to the MCU side of SuziQ. This shows off a bit the ease in bridging the CPU and MCU worlds. Yeah, we did it because we could. But also because it was great for feedback.

Note: When I realized that the Oracle app on SuziQ had to be started in App Lab each time I rebooted SuziQ, we set the app to auto-start on boot. The other thing we noticed was first-use of the LLM after boot had a huge delay that made the Sense time out. So we added a warmup step to prevent the model’s first-load delay from affecting the first image capture.

The Oracle in action

On the Oracle reading process

Tarot cards are famous for their use in narrative construction (or call it 'divination,' if you will). There are 78 cards in a deck, split between the 22 Major Arcana (the impressive ones they usually show in movies) and 56 Minor Arcana (the suit cards similar to a traditional card deck). For interpretations, Major cards are one aspect and Minor Arcana cards another aspect; suits have meaning, with numbers modifying the meaning; and there can be different layouts. The idea is that together they form a basis for a narrative that can then help whatever the reader wishes to explore. Unlike what most people think, there is no 'answer,' but just some interpretation leading to a prompt for reflection.

In our case, I have significantly simplified things to two 6-sided dice. For interpretation, I chose position right-left or upper-lower to provide one dimension, and, of course, numbers the other.

So we instructed the LLM that the upper-most and left-most die would be for aspects of reality and lower-most or right-most die would be for spiritual aspects. And then the pips, 1-6, would be different aspects of Reality, where the reflection is anchored (Me, Other, Loved One, Nature, Work, Quiet); and Spiritual, how the reflection is looked at (Inside, Outside, Sublime, Past, Future, Now).

Yes, that's all touchy-feely, but the idea is that thru this framework we can generate an evocative piece of text that would prompt the reader to craft a narrative to help them think thru something.

The initial idea was to generate a haiku (because haikus are cool). But both included LLMs in AppLab never seemed to figure out what a haiku was, so we just let Gemma generate its poems. But the process is generative, there is no look up table for the interpretations, just a guide on the feeling of what the pips and position mean. Again, the goal is a reflection prompt, not a scripted answer.

I also initially wanted to infuse other aspects into the formation of the haiku. For example, measure the intensity and duration of the shake, or read other data, such as time of day, weather, location, even status of the person's wearable (heart rate, stress, readiness or health level).

When we ended up with the Sense, which has a mic, I wondered if we could use the mic to sense the energetics and duration of the roll, or the energetics of a whisper as the person asked a question (or maybe even speech to text to ask the questions?). The idea is that adding the energetics could further modulate the reading (low = quiet, tentative, just-emerging; medium = present, active, engaged; high = urgent, charged, overflowing).

There's room to grow there. Tho, as the SuziQ LLMs are not so capable, I'll leave these other extra inputs for a more powerful LLM.

Notes on Claude Code

Claude Code has had a multiplier effect on my ability to build this. We form a pair where I formulate what I want and it tries to make it happen. SuziQ and the Sense all can be programmed over the network. As can using Val Town, Google APIs, Claude APIs, and Edge Impulse. And credentials, code, files, and the like were all on my computer. That meant that in the background, Claude Code could connect to everything, upload or download what it needed, back things up along the way.

My role was to concept the whole thing, solder things together, provide the hands for interaction with the physical parts, make sure the hardware was ready for work, and define the features.

I am not a coder, but understand a lot about code. Claude Code is a coder, but can't do anything without a vision. Together we worked on this project step by step, working magic as it added features I requested, made design choices, monitored the model building, made changes to the Sense and SuziQ.

Where do we go from here?

We have the whole flow working. And it is pretty cool to hear the poem come out the speaker.

For the code, I worked with Claude Code to annotate and put the current code on GitHub. If I do any further tweaks, I'll push them to the repo.

For the hardware, I would like to pretty it up. The touch sensor and switch are hanging out. I had originally envisioned some copper tape on the outside to sense touch. Also, I mentioned the energetics part. That might be easy to add, using the mic.

But I think we have a very workable and interesting build. Very much what I wanted. I know this is not the usual fare for Hackster.io, but it sure was fun.

Lemme know what you think!