NOVA - The Adaptive Robot

The Story

NOVA started with a simple question:

Can technology adapt to people, instead of always expecting people to adapt to technology?

Children do not always express their needs through words. A child may lose focus, become frustrated, or have difficulty continuing a conversation without being able to clearly communicate what is happening. This can be particularly relevant when designing technology for children with conditions such as ADHD, autism, or Down syndrome, where communication and interaction may need to be adapted to the individual.

This idea became the foundation of NOVA, an adaptive social robot designed to listen, communicate, observe, and adapt.

What is NOVA?

NOVA is built as three cooperating systems, each with a clearly separated responsibility.

The NOVA Handset, based on an Arduino Nano ESP32, is the part the user interacts with directly. It contains an ILI9341 touchscreen, an INMP441 I2S MEMS microphone, and the interface used for conversations, games, settings, and other activities.

The Arduino Uno Q acts as the main on-device processing unit. Its Linux environment runs the NOVA voice server (FastAPI, port 5000) and handles speech-to-text, text-to-speech generation, adaptive-behavior tracking, and computer vision.

The NOVA Language Server is a separate FastAPI service that hosts the conversational logic. It runs Ollama locally and uses the llama3.1:8b model to generate NOVA's replies. It communicates with the Uno Q over the local network on a different address and port (:8000/chat) than the voice server (:5000). This separation exists because language-model inference is significantly more resource-intensive than speech and vision processing, so isolating it keeps the Uno Q responsive for real-time interaction.

An additional AI-Thinker ESP32-CAM provides the visual input used by NOVA's computer vision system.

The overall architecture can be summarized as:

NOVA Handset → Wi-Fi → Uno Q (STT, TTS, Vision, Difficulty)
                          ↓
                   Wi-Fi → NOVA Language Server (LLM)

and, when visual feedback is required:

ESP32-CAM → Bluetooth (RFCOMM) → Uno Q → Computer Vision → Adaptive System

The NOVA Interface

The handset's touchscreen is organized into four screens, all sharing the same visual style and navigated through a bottom or side menu.

Home Screen

The Home screen displays NOVA's animated avatar — a simple robot face whose expression (happy, sad/angry, or neutral) mirrors the emotion currently detected by the vision system, along with the current communication level. This is the default screen the user sees when idle, and where NOVA's inactivity reminders appear.

Chat Screen

The Chat screen is where the user talks to NOVA, either by typing or by holding a button to record a voice message. It shows the conversation history — the user's message and NOVA's text reply — while the spoken version of NOVA's reply plays through the speaker connected to the Uno Q.

Games Screen

The Games screen offers a menu with two cognitive mini-games: the Memory Game (find matching pairs of cards) and the Reaction Game (tap as fast as possible when the color changes). Each game screen shows the current score, number of attempts (for Memory) or reaction time (for Reaction), and the active difficulty level.

Settings Screen

The Settings screen gives a transparent view into NOVA's adaptive state. It shows the current LLM communication level, the current game level, and the global adaptive score (0–100) as progress bars, along with a label indicating whether difficulty is about to increase, decrease, or stay stable. It also displays detailed statistics for each game: games played, best score, and average score or reaction time.

Building the NOVA Handset

The handset was designed around the Arduino Nano ESP32.

The ILI9341 TFT touchscreen provides the graphical interface through which the user can navigate between Chat, Games, Settings, and other parts of the system.

The voice input is provided by an INMP441 digital MEMS microphone connected through I2S. The microphone is powered from 3.3 V and wired to the following Nano ESP32 board pins:

The microphone is configured as an I2S input at 16 kHz, reading 32-bit samples (the INMP441 outputs 24-bit data MSB-aligned inside a 32-bit frame) which are then scaled down to signed 16-bit PCM with saturation. The recorded samples are assembled into a WAV file before being sent to the Uno Q.

The handset does not perform any speech recognition, language processing, or text-to-speech itself. It captures audio, packages it, sends it to the server, and displays the result to the user. This division of responsibilities keeps the handset lightweight and suitable for a small, embedded form factor.

Communication between the devices

Communication between the Nano ESP32 and the Uno Q happens over Wi-Fi using HTTP.

The Uno Q runs the NOVA Voice Server (FastAPI, port 5000), which exposes endpoints used by the handset for chat, speech-to-text, text-to-speech, settings, games, and vision control:

• POST /api/chat – sends user text and the current communication level, returns NOVA's reply
• POST /api/stt – uploads a WAV recording and returns its transcription
• POST /api/tts – generates a spoken audio file from text
• POST /api/vision/start / POST /api/vision/stop – controls the camera subsystem
• GET /api/settings – returns game and communication settings, polled periodically by the handset to keep its Settings screen up to date

Internally, when the Uno Q receives a chat request, it forwards the conversation to the separate NOVA Language Server over the network, which runs the LLM and returns the generated text. The Uno Q then relays that reply back to the handset and, in parallel, generates a spoken version of it using the TTS engine.

Keeping communication HTTP-based makes each service easy to test independently — for example, the chat endpoint can be exercised directly with a request such as POST /api/chat, without needing the physical handset connected.

Both the Wi-Fi credentials and the server address are configurable per deployment rather than fixed values baked permanently into the project; each build is expected to set its own network name, password, and server IP before flashing.

How voice interaction works

NOVA's voice interaction is implemented as a multi-stage pipeline:

• The Nano ESP32 captures audio from the INMP441 microphone and encodes it as a WAV file.
• The recording is transferred over Wi-Fi to the Uno Q, where Whisper.cpp (using the Whisper tiny model, configured for Romanian) converts speech to text.
• The transcribed text, together with the current communication level, is sent to the NOVA Language Server, which builds a system prompt for that level and queries Ollama running llama3.1:8b to generate a response.
• The generated reply is sent back to the Uno Q, which relays it to the handset, where it is displayed on the Chat screen.
• In parallel, the Uno Q uses Piper with a Romanian voice model (ro_RO-mihai-medium.onnx) to synthesize the reply into speech, played through a speaker connected to the Uno Q via a USB DAC and amplifier.

The complete pipeline is:

INMP441 → Nano ESP32 → Wi-Fi → Whisper.cpp → Text → LLM (llama3.1:8b) → Text response → Displayed on handset
                                                                        ↳ Piper TTS → USB DAC → Amplifier → Speaker

Each stage can be tested independently: the microphone can be verified on its own, Whisper can be tested against an existing WAV file, the language server can be queried directly through its API, and Piper can generate and play an audio file without involving the handset at all.

The same speaker is also used for NOVA's inactivity reminders (see below), so both are routed through a single, shared audio path to avoid overlapping playback.

Adaptive communication

One of the central ideas behind NOVA is that the system should not communicate with every user in exactly the same way. This adaptation happens on two independent tracks, both managed by a single DifficultyManager: conversation difficulty and game difficulty.

Conversation difficulty (emotion-driven)

NOVA maintains a communication level from 1 to 3, included with every request sent from the handset to the language server. This level directly changes the system prompt given to the LLM:

• Level 1 — short sentences, one idea per sentence, no figurative language, everything explained step by step.
• Level 2 — medium-length sentences, gradual explanations, simple examples, common vocabulary.
• Level 3 — natural, adult-level conversation, allowing more complex explanations when appropriate.

The level changes automatically based on the emotion detected during the conversation: if the system detects a happy expression with at least 60% confidence sustained for 20 continuous seconds, the communication level increases. If it detects sad or angry expressions with the same confidence and duration, the level decreases. This time-based threshold avoids reacting to a single noisy frame and instead reflects a sustained emotional trend.

Game difficulty (score-driven): local score vs. global score

NOVA's games use a separate adaptive mechanism, based on performance rather than emotion, and it is important to distinguish between two different scores:

• The local score belongs to a single game session. In the Memory Game it starts at 0 and increases by 5 points for every correctly matched pair; in the Reaction Game it is recalculated on every attempt, from 100 points (reaction under 200 ms) down to a minimum of 50 points for slower reactions. The local score resets every time a new game round starts.
• The global score, tracked by the DifficultyManager, persists across games and rounds. It starts at 50 and is always kept between 0 and 100. Whenever the local score changes — a new pair matched in Memory, or a new attempt finished in Reaction — that change (or, for Reaction, the new score itself) is added to the global score.

The global score is what actually drives difficulty:

• When the global score reaches 80 or above, the game is allowed to increase its difficulty level (up to a maximum of level 3).
• When the global score drops to 20 or below, the game is allowed to decrease its difficulty level (down to a minimum of level 1).
• Every time the difficulty level changes (up or down), the global score is reset back to 50, so the system starts evaluating the new trend from a neutral point rather than carrying over an extreme value.

This two-score design keeps each game session simple to reason about (a clean, resettable local score for immediate feedback) while still letting NOVA track the user's broader performance trend over time through the global score, independently of any single round.

Conversation difficulty and game difficulty are intentionally not linked — a user can be at communication level 3 while playing games at level 1, or vice versa. Both are exposed transparently on the Settings screen, described above.

Adding visual feedback

After the basic communication system was functional, computer vision was added as a separate subsystem.

The visual system uses an AI-Thinker ESP32-CAM. When a Chat session becomes active, the Nano ESP32 sends a request to the Uno Q to start the vision system (POST /api/vision/start).

The Uno Q then establishes a Bluetooth RFCOMM connection with the ESP32-CAM. The camera captures JPEG images and streams them to the Uno Q over that connection, roughly every 0.2 seconds. Images are kept in memory rather than saved to disk.

On the Uno Q, each JPEG frame is decoded with OpenCV and passed through two stages:

Face Detection → Emotion Recognition

Face detection uses OpenCV's Haar Cascade classifier (haarcascade_frontalface_default.xml) to locate faces in the frame.

The detected face is cropped, converted to grayscale, resized to 48×48, normalized, and passed to a TensorFlow Lite emotion-recognition model, which estimates one of seven categories:

• Angry
• Disgust
• Fear
• Happy
• Sad
• Surprise
• Neutral

The emotion with the highest score, along with its confidence value, is passed to the DifficultyManager, which uses it to drive the conversation-difficulty logic described above:

ESP32-CAM → JPEG → OpenCV → Face Detection → Emotion Recognition → Emotion + Confidence → DifficultyManager

The emotion recognition system is not intended to determine someone's actual emotional state with certainty. It provides an additional signal that NOVA can use when deciding whether to adjust an interaction.

When the camera is active

The camera is deliberately not designed to run continuously.

Vision starts when the user begins an actual Chat interaction — the handset calls POST /api/vision/start, which starts the vision thread and opens the Bluetooth connection to the camera.

When the user leaves the Chat interface, the handset calls POST /api/vision/stop, which stops the vision thread and closes the camera connection.

This gives the vision system a clear lifecycle:

Enter Chat → Start Vision → Capture and analyze frames → Leave Chat → Stop Vision

This was chosen both because visual feedback is only meaningful during an active conversation, and to avoid keeping the camera and its processing pipeline running when they are not needed.

How the project was developed

NOVA was not built as one large system from the beginning. It was developed incrementally, with each subsystem tested before the next one was added:

• User interface and handset — the Nano ESP32 was first used to build the touchscreen interface and basic interaction.
• Hardware integration — the microphone, display, and other handset components were integrated and tested individually.
• Wi-Fi communication — the Nano ESP32 and Uno Q were connected over the local network.
• NOVA Voice Server — the Uno Q was configured as the main processing platform, and its FastAPI server was built.
• NOVA Language Server — a separate service was built to host the conversational logic and the LLM, isolated from the voice/vision workload.
• Voice interaction — audio capture, WAV generation, HTTP transfer, Whisper.cpp, LLM querying, and reply display were integrated into a complete pipeline.
• Games and adaptive interaction — the local/global scoring system, games, settings, and activity management were added once the basic pipeline was working.
• Computer vision — the ESP32-CAM, Bluetooth image transfer, face detection, and emotion recognition were added as their own subsystem.
• Hardware integration — the electronics were finally integrated into the custom PCB and enclosure.

This approach matters for reproducibility: someone building a similar system does not need to debug the complete robot at once. Each stage can be built and verified independently before moving to the next.

Challenges during development

Developing NOVA involved several problems that shaped the final architecture.

Integrating computer vision on the Uno Q caused early compatibility issues: the installed OpenCV build did not expose the expected CascadeClassifier functionality, so the vision environment had to be isolated and its dependencies investigated separately before integration.

Bluetooth communication with the ESP32-CAM also required low-level testing. Standard discovery tools were not reliable in the development environment, so the RFCOMM connection was tested directly from Python to confirm the Uno Q could connect and receive JPEG data from the camera.

Running the language model on the same machine as the voice and vision pipelines proved impractical, which is why the LLM was moved into its own server rather than being embedded directly in the Uno Q process — a decision that also made it possible to test and improve the conversational logic independently from the rest of the system.

Getting reliable speaker output also required troubleshooting at the hardware level: the USB-C DAC connected through a hub was not recognized at all by the Uno Q, which turned out to be a known Qualcomm platform issue where the USB-C port defaults to peripheral ("device") mode instead of "host" mode when the board is powered through its VIN pins. Forcing host mode at boot (via a small systemd service) resolved this permanently, allowing the DAC, amplifier, and speaker chain to work as intended.

Because HTTP requests, Wi-Fi communication, camera processing, and UI interaction can all involve blocking operations, the system had to be structured carefully so that background processing did not prevent the handset from staying responsive. Long-running tasks such as TTS generation and vision analysis were separated into background threads and dedicated modules for this reason.

Hardware integration

The final NOVA prototype is not intended to remain a collection of development boards connected by loose wires.

The NOVA Handset uses a custom PCB designed specifically for the project, integrating the required connections into a compact and reliable platform. A dedicated enclosure was designed and manufactured alongside the PCB, allowing the electronics, display, and microphone to be assembled into a real physical device rather than a breadboard prototype.

The hardware and mechanical design were developed using KiCad and Fusion 360, while the firmware and software were developed using the Arduino IDE and Visual Studio Code.

Why NOVA was built this way

The architecture of NOVA is based on a simple principle: each component should have a clear responsibility.

• The Nano ESP32 handles interaction.
• The Uno Q handles speech, vision, audio output, and adaptive tracking.
• The NOVA Language Server handles conversation.
• The ESP32-CAM provides visual information.
• The DifficultyManager decides how NOVA should adjust both its conversation style and its game difficulty.
• The custom PCB and enclosure turn these components into a physical product.

This separation makes the system easier to understand, test, modify, and reproduce. It also makes it possible to build NOVA progressively: a beginner could start with the handset and display, add the microphone, establish Wi-Fi communication, build the voice pipeline, connect a language server, add adaptive behavior, and finally introduce computer vision.

The goal behind NOVA

NOVA is still a prototype, and there are many aspects that can be improved. Emotion recognition can become more accurate, the conversational system can become more sophisticated, and the adaptive behavior can be expanded with additional activities and strategies.

The larger goal remains simple:

Technology should not only become better at processing information. It should also become better at adapting to the people who use it.

NOVA is an exploration of that idea. It brings together embedded systems, artificial intelligence, computer vision, audio processing, wireless communication, PCB design, and mechanical design into one platform built around adaptive interaction.

The project was also developed with reproducibility in mind. Rather than treating NOVA as a single complicated system, its architecture is divided into understandable stages and independent subsystems, allowing someone starting from the basics to follow the same general path — from a simple embedded device to a complete adaptive companion.

Acknowledgements

This project would not have been possible without the support of my sponsors.

PCBWay supported NOVA by sponsoring the manufacturing of the custom PCB and enclosure. Their support helped transform the electronic and mechanical designs into physical components that could be integrated into the final prototype.

HubCo supported the project by providing access to its laboratory facilities, where the soldering, hardware assembly, and other physical work could be carried out.

I am grateful to both organizations for supporting NOVA and helping make the development of this project possible.