SignAlly

What it is

SignAlly is a small box that watches you sign in Indian Sign Language and shows the words on its own screen. Everything runs on the Arduino UNO Q inside it. No cloud, no phone app, no network. Unplug it from the wall and it still works.

Right now it knows six signs. That's a limit of the data, not the idea. There is very little good ISL video to train on, so we recorded the signs ourselves and trained on those, and six is how far that got us. Adding a sign means recording it, and the pipeline is built for exactly that.

The whole thing is one UNO Q, a USB webcam, a 2.8" CrowPanel display with a Start/Stop button, and a battery pack, all meant to live in a single 3D-printed enclosure.

Why build it as a device

If you use Indian Sign Language and you walk up to a counter, a clinic desk or a ticket window, the person on the other side almost never signs back. You end up writing on your phone or pointing. An interpreter is a luxury most places don't have.

We wanted something small enough to sit on that counter. You sign at it. It shows what you said in plain English on a screen, so the person across from you can just read it. No phone, no internet, no account.

That is SignAlly. This is how it works and what we learned building it.

What's in the box

There isn't much in it, on purpose.

• An Arduino UNO Q. It has two brains: a Linux side that runs the camera and the model, and a small microcontroller that talks to the screen.
• A USB webcam.
• A 2.8" CrowPanel display (an ESP32 with a screen). It has two push buttons: Start/Stop, and mute.
• A 3-cell 18650 battery pack with a protection board and a 5 V regulator, so it runs off the wall.

The board does all the thinking. The panel only draws what it's told. That split turned out to matter a lot, and we'll come back to it.

What happens when you press Start

Here is the whole trip, from your hands to the screen.

1. The camera finds your body. We use MediaPipe Holistic, which finds 543 points on you every frame: 33 on the body, 21 on each hand, 468 on the face. That's 1, 662 numbers per frame. If you feed all of that to a model, 85% of what it sees is your face mesh, and your hands are 7%. So we don't.

2. We keep 183 numbers. Each hand's shape, measured from its own wrist so it doesn't matter how far you stand from the camera. Where each wrist is, measured from your shoulders. Eleven upper-body points. Four face numbers: eyebrows raised, eyebrows furrowed, mouth open, mouth wide. And a small list of "did we actually see this part" flags. If a hand is hidden, it's marked missing, not drawn at zero. That distinction sounds fussy. It's the difference between a model that works and one that doesn't.

One thing we learned the hard way: MediaPipe's "left hand / right hand" label flips constantly when your hands cross, which in ISL is most of the time. So we ignore it. Whichever hand is closer to your dominant shoulder is hand one. Always.

3. It waits for you to finish. The device doesn't guess on every frame. It watches your wrists. When they lift, it starts recording. When they drop back to rest, it stops and classifies that one clip. One sign in, one answer out. Our first ten recordings were all wrong because we started signing the instant we pressed the button, so the clips had no "rest" at the start, and the model had never seen a clip like that. Now the recorder keeps a little rolling buffer, so every take begins at rest.

4. A small model reads the clip. It's a temporal convolutional network with two branches: one looks at hand shapes, the other at how the hands move through space. About half a million parameters. It runs on the UNO Q's CPU in a few tens of milliseconds.

5. A lookup table turns the sign into a sentence. The model says doctor. A plain JSON file says doctor means "I need a doctor". That line goes to the screen. Changing the wording, or adding Hindi later, means editing a text file, not retraining anything.

Teaching it to say "I don't know"

This was a rule from day one. If the model isn't sure, the screen says "Didn't catch that. Please sign again." It does not pick its best guess.

A confident wrong answer is worse than no answer. So below a confidence threshold the answer is simply "unclear", and that's treated as a normal outcome, not an error. In practice it is the most common thing the device says, and that's fine.

The part that nearly didn't work

We trained the model on INCLUDE, a public ISL dataset: 4, 284 clips of 262 signs, recorded at about 25 frames a second on good cameras. On a laptop it looked great.

Then we put it on the board. The UNO Q runs MediaPipe at about 3 frames a second. Same model, same code, and it was useless. We measured it: 100% correct at 5 frames a second, 2.5% at 2. The model had learned the fine texture of how a sign moves, and at 3 fps that texture is just gone.

Two other things were different on the board too. The camera is close, so your hips are out of shot, and MediaPipe makes up hip positions anyway. And the tracker loses your hands about half the time.

The fix was to stop pretending the board is a laptop. We took the training clips and made them look like board footage: dropped frames down to 2 to 5 per second at random, hid the lower body, deleted the hands in short runs the way the tracker does. Then we trained a new final layer on that. It learned to read the sign it will actually see.

The last step was recording ourselves. Right now the device knows six signs from our own recordings: hello, thank you, washroom, doctor, happy, sad. A 17-sign version trained on INCLUDE alone got 2 out of 24 right on us. The six-sign one trained on our own hands works. Small and honest beat big and wrong.

One more lesson, because it cost us a day. A lot of hand-tracking demos mirror the camera image so it feels like a selfie. We did too. Accuracy went from 100% to 28%. Mirroring turns a right-handed signer into a left-handed one, and the model has never seen that. The camera frame goes in exactly as the camera produced it.

Three small programs and one screen

On the board, three separate programs are running.

• Recognition owns the camera and the model. It knows nothing about screens. It just says things like {"e":"recognised","gloss":"hello","conf":0.92}.
• The orchestrator is the only thing that decides what the screen shows. It turns "hello" into "Hello", numbers every message, and plays the audio file for it. It's written with nothing but Python's standard library, so there is nothing in it that can fail to install.
• A tiny relay on the UNO Q's microcontroller carries lines of text to the panel over a serial wire.

The screen gets one line of JSON per message, 256 bytes at most, and answers each one with an acknowledgement. That ack is the only way we can tell a message actually crossed the wire, because the microcontroller has no console to read.

{"t":"state","seq":1,"s":"listening"}
{"t":"result","seq":2,"id":"doctor","text":"I need a doctor","conf":0.92}

Why bother splitting it up? Because things die. If the camera unplugs, recognition reports a fault and the screen says so. If recognition crashes, the orchestrator keeps running and shows "Recognition offline" until it comes back. If the screen unplugs, recognition carries on. Nothing takes anything else down with it. On a demo table, that is the feature you want most.

A small detail that we only found on real hardware: the model re-arms the instant a sign finishes, so the "Listening" screen was wiping the answer 100 milliseconds after it appeared. Now a result holds the screen for three seconds. You'd never notice it's there. You'd definitely notice if it wasn't.

Where it stands

Being straight about it:

Works today

• Camera to sentence on the screen, end to end, on battery
• Six signs, recorded by us
• Start/Stop and mute buttons on the panel
• "Didn't catch that" instead of guessing

Not yet

• Sound. The UNO Q has no audio output we can reach yet, so the speaker is drawn in the schematic but silent
• A bigger vocabulary. Adding signs means recording them
• The battery gauge on screen — the divider is wired, nothing reads it yet

That last one is the real gap. Every number we have comes from one or two people signing at it. The next thing this project needs is a room full of ISL users and an afternoon.

From Sketch to Enclosure

The box took three forms before it ever became an enclosure.

It started as a rough pencil sketch to work out where the camera, screen, and buttons belonged. Next came a loose desk mock-up—the UNO Q, display panel, and battery wired together in the open just to confirm the layout made sense. Only then did we commit it to CAD, built around the real hardware and wiring.

You can grab the printable files right from here.

Build it yourself

Everything is on GitHub, including the schematics, the panel firmware, and the two Python services:

• SignAlly, the device: https://github.com/jbrathwa/SignAlly
• islkit, the sign-recognition library we pulled out of it, on PyPI: https://github.com/jbrathwa/islkit

The docs folder has the wiring, the message protocol, and a guide to training your own vocabulary. If you record a set of signs and get it working for someone who signs differently from us, we'd really like to hear about it.