Hardware components | ||||||
| × | 1 | ||||
Software apps and online services | ||||||
 |
| |||||
| ||||||
What if your camera could not only see, but also listen, understand, and talk back?
That's exactly what this project turns the reCamera into.
Under the hood, the reCamera does all the heavy lifting of orchestration: it runs YOLO11n Pose on-device to detect your gesture in real time, then coordinates a full STT → LLM → TTS pipeline through Node-RED. The actual "thinking" — speech recognition, reasoning, and voice synthesis — is handled by the iFlytek cloud via WebSocket APIs.
Because the reCamera has a built-in microphone and speaker, no PC, no extra dev board, and no tethering is required — just power it on, connect to Wi-Fi, and you have a complete, hands-free voice interaction device.
About the demo video above:The video shows an early proof-of-concept where the STT and TTS pipelines were temporarily offloaded to a PC for rapid prototyping.This project goes much further: I've ported the entire STT → LLM → TTS orchestration entirely onto the reCamera itself (running inside Node-RED), eliminating any PC dependency. On top of that, I've added a custom Dashboard UI within the reCamera's Node-RED environment — so you can monitor trigger status, system logs, and API call results in real-time directly from the browser.”
Most edge AI cameras are great at seeing, but terrible at communicating. They flash results on a screen — which is useless if you're standing across the workshop, holding tools in both hands, or covered in grease.
I wanted a camera that listens and speaks back. No keyboard, no phone app, no staring at a monitor. Just one simple gesture (crossing your wrists) to wake it up, and then pure voice interaction: you ask a question out loud, and the camera answers you verbally, like a smart assistant with eyes.
Under the hood, it keeps the sensitive stuff (pose detection) on the edge for instant response, while tapping into cloud AI for heavy-lifting reasoning and speech. But as a user, all you feel is a hands-free, self-contained device that turns any environment into a voice-controllable space — no PC required.
How It Works (System Architecture)Architectural note: This project follows an "edge orchestration + cloud models" split. On-device Node-RED handles the gesture trigger and pipeline logic; iFlytek cloud APIs process speech recognition, LLM reasoning, and voice synthesis. This keeps latency low and privacy intact on the camera, without forcing it to run massive generative models locally.
The entire workflow orchestration runs on the reCamera via Node-RED (no PC server needed). The visual gesture runs on-device; STT/LLM/TTS call the iFlytek cloud APIs over WebSocket.
Pipeline at a glance:
1. Camera node captures frames → YOLO11n Pose estimates 17 keypoints.
2. Action determination Function node computes left-wrist-to-right-wrist distance vs. left-shoulder-to-right-shoulder distance. When wrist distance < shoulder distance (wrists together), the gesture is triggered (with a 10-second cooldown).
3. Trigger → blue LED on → `arecord` records 5 seconds → blue LED off.
4. The WAV file is read and the 44-byte header is stripped to get raw PCM.
5. STT Function node sends PCM to iFlytek via WebSocket → returns recognized text.
6. LLM Function node sends text to Spark Lite via WebSocket → returns a short reply.
7. TTS Function node launches a child Node process to call iFlytek TTS via WebSocket → writes `/tmp/reply.wav`.
8. `aplay` plays back the reply through the speaker.
Step 1 — Set Up the reCamera
1. Follow the official reCamera Getting Started guide to complete basic configuration.
2. Power the device and connect it to your network (same network as the iFlytek cloud API can reach).
3. Open a browser and go to `http://192.168.42.1`, then log in to the Node-RED workspace.
Step 2 — Import the Pre-Built Node-RED Workflow
This demo provides a pre-configured workflow(`voice_interaction.json`) that contains all nodes and connections — including the STT, LLM, and TTS Function nodes.
1. Create a new application on the SenseCraft AI platform.
2. Download the workflow (or import `voice_interaction.json` from githhub directly).
Download voice_interaction.json from GitHub repo.
In the reCamera Node-RED editor, click the Menu (☰) → Import → Clipboard, paste the JSON content, and click Import.
3. Import it into the reCamera's Node-RED editor. (See Access reCamera preview dashboard.)
If the workflow appears in your Node-RED editor, import succeeded.
Step 3 — Configure the Workflow Parameters
After importing, adjust the following parameters to match your environment and credentials.
3.1 Model Node — Pose Model
The Model node ships with several trained models. For this demo, select YOLO11n Pose for human pose detection.
3.2 Action Determination — Gesture Trigger Logic
The "Action determination" Function node decides when to trigger voice interaction:
- It extracts the left shoulder (keypoint 5), right shoulder (keypoint 6), left wrist (keypoint 9), and right wrist (keypoint 10).
- It requires all four keypoints to have confidence > 0.4 to avoid false positives.
- Core logic: when the distance between the two wrists is less than the distance between the two shoulders (wrists together), the gesture is triggered.
- A 10-second cooldown prevents repeated triggers during recording and playback.
You can edit this Function node to change the gesture or cooldown.
3.3 Exec Nodes — LED Control & Recording
Two Exec nodes control the blue LED. Update the reCamera `root` password to match your device:
LED on:
```bash
echo "your_Password" | sudo -S sh -c 'echo 1 > /sys/class/leds/blue/brightness'
```LED off:
```bash
echo "your_Password" | sudo -S sh -c 'echo 0 > /sys/class/leds/blue/brightness'
```Microphone recording (5 seconds, 16 kHz mono):
```bash
arecord -D hw:0,0 -r 16000 -f S16_LE -c 1 -d 5 /tmp/test.wav
```3.4 STT Function Node
The "iFlytek STT (WebSocket)" Function node sends raw PCM audio to the iFlytek IAT API and returns recognized text. Open the node and fill in your iFlytek credentials:
```javascript
const APPID = "your_APPID";
const APISecret = "your_APISecret";
const APIKey = "your_APIKey";
```This node requires the `ws` and `crypto` libraries (declared in the node's Setup tab).
Full STT Function node code:
```javascript
// ===== iFlytek STT: PCM → text =====
const APPID = "your_APPID";
const APISecret = "your_APISecret";
const APIKey = "your_APIKey";3.5 LLM Function Node
The "iFlytek LLM (WebSocket)" Function node sends the recognized text to Spark Lite and receives a short intelligent reply. Fill in the same iFlytek credentials:
```javascript
const APPID = "your_APPID";
const APISecret = "your_APISecret";
const APIKey = "your_APIKey";
```Full LLM Function node code:
The system prompt is preset to: "You are a voice assistant deployed on the smart camera reCamera..." (asks for short, friendly answers of no more than 50 characters). You can edit this prompt inside the `system` message in the code to customize the assistant's personality.
3.6 TTS Function Node
The "TTS (execSync child process)" Function node generates speech from the LLM reply. Because the Node-RED sandbox restricts long-lived WebSocket connections, this node writes a small standalone Node.js script to `/tmp/_tts_runner.js` and launches it via `child_process.execSync`. The script:
1. Connects to the iFlytek TTS WebSocket endpoint.
2. Sends the reply text.
3. Collects the returned PCM audio chunks.
4. Wraps them in a WAV header (16 kHz, mono, 16-bit) and writes `/tmp/reply.wav`.
Fill in your iFlytek credentials inside this node as well. This node requires the `crypto`, `fs`, and `child_process` libraries.
Full TTS Function node code: Code chapter
Step 4 — Run the Demo
1. Click Deploy in Node-RED to activate the workflow.
2. Stand in front of the reCamera and bring your wrists together (wrist distance < shoulder distance).
3. The blue LED lights up → 5-second recording starts.
4. Speak your question into the microphone.
5. The LED turns off → Node-RED sequentially calls cloud STT → LLM → TTS (orchestration on-device, models in the iFlytek cloud) → the reply is played through the speaker automatically.
- A 10-second cooldown is enforced after each trigger to prevent repeated triggers during the STT → LLM → TTS cycle. You can change this in the "Action determination" Function node.
- Recording is fixed at 5 seconds. Adjust the `-d 5` parameter in the `arecord` Exec node if you need longer/shorter capture.
- The gesture trigger requires all four keypoints (shoulders + wrists) to have confidence > 0.4. Raise this threshold in the Function node to reduce false positives.
- The LLM system prompt caps replies at ~50 Chinese characters to keep TTS processing time short. Adjust as needed.
- Always use the correct `aplay` parameters (`-f S16_LE -c 1 -r 16000`); otherwise playback may fail. Match them to the TTS output format.
This flow is completely decoupled. You can easily swap out the iFlytek LLM node for a local Ollama endpoint, or change the arecord duration to 2 seconds for short command words. The Node-RED visual pipeline means you don't need to recompile C++ code for every tweak.
Comments