Make Home Assistant Voice Fully Local with RK3576

Local Home Assistant Voice on RK3576

Cloud voice assistants are convenient—until the internet drops, latency creeps in, or you start wondering where every command is processed. For a smart home, turning on a light or asking a quick question should feel instant, reliable, and private.

This project turns a Rockchip RK3576 board into a fully local AI voice backend for Home Assistant. Using Docker Compose, it combines wake word detection, speech-to-text, text-to-speech, and a lightweight local language model into a single deployment. Home Assistant continues using its standard Assist pipeline while the RK3576 accelerates the AI workloads behind the scenes.

The result is a responsive voice assistant that keeps everyday interactions on the local network without changing how Home Assistant works.

Project Highlights

• Fully local Home Assistant voice backend
• Rockchip RK3576 NPU acceleration
• Docker Compose deployment
• openWakeWord for wake word detection
• Whisper for speech-to-text
• Piper for text-to-speech
• Qwen 2.5 1.5B served through an OpenAI-compatible RKLLM API
• Wyoming Protocol integration
• Prebuilt ARM64 container images
• No cloud required for everyday voice interactions

How It Works

Once deployed, the experience feels like a native Home Assistant voice assistant.

When the wake word is detected, Home Assistant starts its Assist pipeline. Whisper running on the RK3576 transcribes the audio into text. Home Assistant either executes the requested intent directly or forwards the conversation to a local Qwen 2.5 1.5B model through an OpenAI-compatible RKLLM API. Piper then generates the spoken response and sends it back to the user.

Everything happens on the local network.

The RK3576 exposes four services that Home Assistant connects to through the Wyoming Protocol:

• Piper Text-to-Speech (Port 10200)
• Whisper Speech-to-Text (Port 10300)
• openWakeWord (Port 10400)
• RKLLM OpenAI-Compatible API (Port 8001)

Because Home Assistant communicates with standard Wyoming services, it never needs to understand RKNN models, Rockchip runtimes, or NPU-specific implementation details.

Why I Built It

Home Assistant already has an excellent local voice ecosystem, so replacing its architecture was never the goal. Instead, I wanted to provide a compact backend capable of running the heavier AI workloads on hardware designed for edge inference.

The RK3576 occupies an interesting middle ground. It is powerful enough to accelerate modern AI models while remaining small, quiet, and energy efficient enough to run continuously as part of a smart home.

The goals of this project were straightforward:

• Keep the entire voice pipeline local
• Use RK3576 hardware acceleration where it provides real benefits
• Make deployment simple with Docker Compose
• Keep Home Assistant completely hardware-agnostic

Those goals influenced every design decision. Hardware-specific complexity stays inside the containers while Home Assistant continues interacting with familiar, standards-based services.

Why I Built It

Home Assistant already has a strong local voice ecosystem, so I did not want to replace its architecture. I wanted to give it a compact local backend that could run the heavier AI workloads on hardware designed for edge inference.

The RK3576 is interesting because it sits in a useful middle ground. It is small enough to behave like a home appliance, but it includes a Rockchip NPU that can accelerate real models when they are prepared correctly.

My goals were simple:

• Keep the voice pipeline local
• Use RK3576 acceleration where it matters
• Make deployment practical with Docker Compose
• Keep Home Assistant hardware-agnostic

That last point shaped the whole project. Hardware-specific projects often become difficult to reproduce because every layer knows too much about the board. I wanted the opposite: Home Assistant should see ordinary services, while the containers handle the RK3576 details.

System Architecture

The complete voice pipeline looks like this:

• openWakeWord detects the wake word.
• Home Assistant activates the Assist pipeline.
• Whisper transcribes the audio on the RK3576.
• Home Assistant executes the requested intent or forwards the conversation to the local LLM.
• Piper generates speech.
• Audio is played back to the user.

Wyoming acts as the interface between Home Assistant and the accelerated services. It provides a clean protocol while hiding model loading, RKNN execution, runtime management, and hardware access inside the containers.

This separation means users simply add Wyoming integrations inside Home Assistant and select them in an Assist pipeline. The underlying hardware remains invisible.

Why RK3576?

This project is designed for smart-home control and short local conversations—not cloud-scale AI workloads.

For that use case, the RK3576 is an excellent fit. Its integrated Rockchip NPU accelerates RKNN models while keeping power consumption low enough for always-on operation.

The default language model is Qwen 2.5 1.5B, chosen because it provides a good balance between responsiveness and conversational ability for Home Assistant tasks.

From an engineering perspective, the board also presents interesting challenges. Running models on an NPU requires different runtimes, model formats, and execution paths than running Python applications on a CPU. One of the project’s primary objectives was hiding that complexity behind familiar Home Assistant integrations.

Performance

Current measurements on the RK3576 are encouraging:

• Whisper transcription: 0.626seconds for a typical room-control command
• LLM response: 2.82 seconds for LLM response.
• Piper synthesis: 0.474 seconds from text response to playable audio

These figures represent individual stages of the pipeline. End-to-end latency measurements will be published in a future update.

For a responsive voice assistant, overall experience matters more than raw inference speed. Wake word detection should trigger immediately, transcription should complete quickly, and speech synthesis should begin without noticeable delay.

Deployment and Setup

The full setup lives in the GitHub repository, including Docker Compose files, model packaging notes, and troubleshooting steps. The short version is: prepare the board, start the Compose stack, then add the local services to Home Assistant.

Prebuilt ARM64 images are what make this practical. Users should not need to convert models, build containers on the board, or manually arrange large model files before trying the project.

1. Prepare the RK3576 board

Install a Linux ARM64 image, connect the board to the network, then install Docker Engine and the Docker Compose plugin. The RKNN-based containers need access to the Rockchip accelerator, so the Compose file gives the speech and LLM services the required device access.

docker --version
docker compose version

2. Clone the project

git clone https://github.com/Hanzo-Huang/rk3576-home-assistant-voice.git
cd rk3576-home-assistant-voice

3. Start the voice services

sudo docker compose up -d --pull always
sudo docker compose ps
sudo docker compose logs -f

Use this path if Home Assistant already runs elsewhere on the network. You should see containers for Whisper, Piper, openWakeWord, and RKLLM.

4. Optional: run Home Assistant on the same board

sudo docker compose --profile homeassistant up -d --pull always

Then open:

http://RK3576_BOARD_IP:8123

5.Add the Wyoming services in Home Assistant

In Home Assistant, go to Settings -> Devices & services -> Add integration -> Wyoming Protocol, then add:

• Whisper speech-to-text: RK3576_BOARD_IP:10300
• Piper text-to-speech: RK3576_BOARD_IP:10200
• openWakeWord: RK3576_BOARD_IP:10400

6. Create the Assist pipeline

Go to Settings -> Voice assistants, then create or edit an Assist pipeline and select the RK3576 Whisper, Piper, and openWakeWord services.

7. Add the local LLM

Install the Local LLM integration through HACS, then configure it as an OpenAI-compatible endpoint:

Backend: OpenAI Compatible Conversations API
API hostname: RK3576_BOARD_IP
API port: 8001
API path: /v1
API key: sk-local
Model name: rkllm-model

Select this conversation agent in the Assist pipeline.

8. Test it

Try commands such as:

• "Turn on the office light."
• "What lights are on?"
• "Set the desk lamp to 30 percent."
• "What time is it?"

Watch logs while testing:

sudo docker compose logs -f whisper piper openwakeword llm

Engineering Challenges

Integrating Whisper with the Rockchip NPU was the first major challenge.

Home Assistant produces standard audio streams, while the accelerated RKNN model requires a different execution path. The service needed to receive Wyoming audio, prepare tensors for RKNN inference, execute the model on the NPU, and return transcripts without exposing any hardware-specific details.

Piper presented a different challenge. Although much of its processing remains similar to the standard implementation, the decoder runs through RKNN. Wrapping the accelerated path while continuing to stream standard audio back to Home Assistant required careful integration.

The final challenge was designing an interface that felt native. Rather than creating a custom demonstration, the project was built around Wyoming so it integrates seamlessly with Home Assistant’s existing voice ecosystem.

Packaging proved just as important as inference. Large AI models don’t belong in the source tree, but asking every user to assemble model packages manually would make the project difficult to adopt. The build workflow downloads release assets, verifies them, prepares the required files, and publishes prebuilt ARM64 images ready for Docker Compose deployment.

What I Learned

The most useful design choice was keeping the boundary strict. Home Assistant remains the orchestrator. The RK3576 board provides local AI services. Wyoming connects the two without leaking hardware details across the line.

That separation also makes the project easier to change. A different RK3576-ready Whisper model, another Piper voice, or a different RKLLM image can be swapped behind the same service shape. The user-facing integration stays the same.

The other lesson was that packaging is part of the engineering, not an afterthought. A working prototype is interesting. A Compose stack with prebuilt containers is something another maker can actually try.

Result

The RK3576 board becomes a local voice backend for Home Assistant: wake word, STT, TTS, and local conversation all run on the edge, while Home Assistant continues to use standard integrations.

The takeaway is the interface. Home Assistant stays clean and hardware-agnostic, while the RK3576 does the specialized acceleration behind Wyoming services. That is the balance I wanted: a voice assistant that feels like part of the home, but still remains open, local, and hackable.

Future Improvements

• Add more Piper voices
• Add more RK3576-ready Whisper model sizes
• Benchmark latency across the Assist pipeline
• Build a dedicated voice satellite enclosure
• Add microphone and speaker wiring notes
• Compare RKLLM models for smart-home commands and short conversations