VR StachChan

1. What is your project about?

This system connects "Stack-chan, " a palm-sized robot powered by M5Stack CoreS3, directly with an Android smartphone (Flutter app) over local Wi-Fi, enabling an immersive VR experience from the robot's perspective. It syncs the robot's head movement with the smartphone's tilt in real-time, and renders the camera feed in Side-by-Side VR with barrel distortion correction on the phone's screen.

2. Why did you decide to make it?

Traditional remote-controlled robots and camera streaming systems often rely on external cloud or relay servers, which brings the following challenges:

* Latency: Relaying through a server introduces lag in video and controls, easily causing "VR sickness" when viewed through a VR headset.

* Setup Complexity: Account registration, server setup, and Wi-Fi routing configurations are often tedious.

* Privacy Concerns: Camera footage passes through external servers, making it difficult to use casually in private spaces like at home.

We wanted to solve these problems by building a VR robot system that is "internet-free, ultra-low latency, secure, and ready for anyone to experience instantly."

3. How does it work?

By completely eliminating the relay server, the M5Stack CoreS3 firmware itself operates as a "local WebSocket server."

1. Easy Connection: Simply scan the QR code displayed on the M5Stack screen using the smartphone app. The IP address is automatically recognized and the connection is established.

2. Ultra-Low Latency Control: The smartphone's gyro and accelerometer sensors detect head movement, transmitting angle data directly to Stack-chan's servo motors via WebSocket at a 100ms interval.

3. VR Video Streaming: The M5Stack's camera feed is streamed directly to the phone as JPEG binary data at a 350ms interval. The phone app uses GPU shaders to apply real-time barrel distortion correction and renders the feed in Side-by-Side format tailored to each eye.

---

Key Features & Technology

1. Serverless Local WebSocket Communication

Without relying on cloud or relay servers, the M5Stack CoreS3 acts as a local WebSocket server (using the `esp_http_server` component). The Flutter app on the smartphone connects directly to transmit image binary data with ultra-low latency and send precise servo motor commands (in JSON format) bidirectionally.

2. Real-time Synchronization of Gyro Sensors & Servo Motors

Assuming the user wears a VR headset with the smartphone, the app detects head tilt via the smartphone's built-in gyro and accelerometer sensors. It sends steering command data to Stack-chan at a 100ms interval, minimizing lag and providing a natural sense of "stepping inside the robot."

3. VR Side-by-Side Rendering with Barrel Distortion Correction

The app decodes received JPEG frames and renders them Side-by-Side. To counteract the distortion caused by VR headset convex lenses, a real-time "barrel distortion correction" is applied using OpenGL/Flutter shaders, reducing eye strain and VR sickness during extended use.

4. Zero-Configuration Connection (SD card Wi-Fi & QR code scan)

We simplified the connection process to make it hassle-free:

* Auto Wi-Fi setup: Simply write your SSID and password to a `wifi.txt` file on a microSD card, insert it into the M5Stack CoreS3, and it will automatically connect to Wi-Fi at boot.

* QR Code Pairing: Tapping the IP address display on the M5Stack screen (LVGL UI) pops up a connection QR code. Scanning it with the smartphone app automatically pairs the devices.

---

Why This Project is Unique

The highlight of this project is the balance between ease of use and privacy/security.

Because it works entirely within a local network without an internet connection, you never have to worry about camera feeds or voice data being leaked to external servers. It is perfect for demonstrations at exhibitions, outdoor environments, or private homes. Additionally, it maximizes the hardware capabilities of both the smartphone (gyro, GPU shaders) and the ESP32-S3 processor to deliver a premium VR experience with a minimal setup.

#M5StackInnovationContest