Robo DeMo 1000 - Companion Robot for Distance Runners

What is your project about?

This project centers around a robot that follows or leads distance runners. It identifies other runners by the use of computer vision and then proceeds to either start motivating or demotivating the other identified runners through speech audio by leveraging "text to speech" AI services and computer vision ML models.

Why did you decide to make it?

I built this robot because I felt there was a need for a sports appliance that was also sociable and beefy enough to go the distance. Lots of people are sociable through speech and lots of people perform sports outdoors. Today, AI services are cool and accessible. Access to technologies, such as Viam and other open source software helps brings all those components together.

I have a passion for distance running and wanted to create a project that merged 2 or 3 of my interests.

For this application, the outdoor sport I'm focused on is distance running. Also, VIAM hosted an AI Challenge recently for the community. This project was developed on a hackathon timeline so there will be edits to be made.

Background:

I'm an avid distance runner who loves to encourage fellow runners during my training sessions. However, with the increasing number of people taking up running, it's become challenging to personally motivate everyone you encounter on your routes. To address this, I've developed Robo DeMo 1000, a runner robot sports appliance equipped with a webcam, computer vision capabilities, and text-to-speech services, enabling it to identify other runners and shout motivational or demotivational greetings as needed.

Use Case Scenario:

Training Day Routine:

• Before heading out for your daily run, you power up Robo DeMo 1000 and ensure it's calibrated and ready to go.
• You set the motivational tone for Robo DeMo based on your training goals for the day, whether it's a long-distance endurance run or a speed workout.

Encountering Fellow Runners:

• As you hit the trails or the streets, Robo DeMo's webcam scans the environment, identifying other runners based on predefined criteria such as distance, pace, or attire.
• When Robo DeMo spots another runner, it quickly analyzes their body language and pace to determine whether they could use a boost of motivation or if they seem to be in high spirits already.

Motivational Interactions:

• If the runner appears fatigued or struggling, Robo DeMo springs into action, delivering a personalized motivational message like, "You're doing great! Keep pushing through, you're stronger than you think!"
• Conversely, if the runner seems to be cruising along with confidence, Robo DeMo might offer a light-hearted challenge or playful banter to keep their momentum going, such as, "Nice pace! But can you catch me?"
• Robo DeMo's text-to-speech service ensures that the messages are conveyed clearly and audibly, cutting through the runner's fatigue and providing an unexpected source of encouragement.

Adaptive Feedback:

• Robo DeMo continuously learns and adapts based on the responses it receives from other runners. If a particular type of message consistently yields positive results, it integrates similar messages into its repertoire for future interactions.
• Additionally, Robo DeMo can adjust its volume and tone based on the ambient noise level to ensure that its messages are heard without being disruptive to other runners or bystanders.

Post-Run Analysis:

• After completing your run, you review the data collected by Robo DeMo, including the number of interactions, the effectiveness of different types of messages, and any feedback from other runners.
• This analysis helps you fine-tune Robo DeMo's behavior and messaging strategies, ensuring that it continues to provide meaningful and uplifting interactions during future runs.

Benefits:

• Enhances the sense of community among runners by providing personalized encouragement and motivation.
• Helps you spread positivity and support to a larger number of fellow runners during your training sessions.
• Offers a fun and innovative way to engage with other runners while improving your own performance through positive reinforcement.

Conclusion:

Robo DeMo 1000 not only enhances your own running experience but also enriches the experiences of those you encounter on your routes, creating a more supportive and uplifting environment for all. With its ability to deliver timely and personalized messages, Robo DeMo serves as a valuable companion in your quest for fitness and personal growth.

How does it work?

The system has both hardware and software components. We start with building a robot hardware base with components and then we integrate software elements.

The hardware is described below.

This 3D printed robot has 4 medium to large wheels(4.5in diameter), motor controllers, a Raspberry Pi 5, a PCA9685 PWM controller, an audio amplifier, a speaker, a XBOX controller, a USB soundcard, and USB webcam (Logitech C920).

Controlling a robot using a XBOX controller is pretty typical. It connects to the Raspberry Pi via Bluetooth.

The software was mostly developed using Python, open source libraries and the VIAM robotics platform.

It makes use of AI Services such as ElevenLabs 'text to speech' and uses a ML computer vision model (PeopleDetector) to identify runners. This robotic project was configured and integrated using the VIAM platform.

Software Scripts - see GitHub repo link

main.py

# This script started from the Viam (viam.com) python code sample generated after all the components were added to the configuration in the Viam web app.

The code supports a robot base. The robot base has a Raspberry Pi 5, PCA9685 (PWM), Motor controllers, 4 motors, a USB camera, USB soundcard, Audio Amplifier and Audio speaker, and wireless XBOX controller

# Remote Control

This feature is achieved by using Viam's built-in features.

# Text to Speech

# This feature is achieved by leveraging Eleven Labs API

This main.py script calls another script called motivator_demotivator_1000.py. It implements the AI Text to Speech.

This motivator_demotivator_1000.py script uses the ElevenLabs API for speech to text generation and PyGame libraries for audio output.

# Computer Vision - Runner Detection

Computer Vision is achieved by using Viam's built-in CV/ML features

A People Detector was developed to trigger on people. Vision -> MLmodel

headup_display.py

# This project leverages the PyGame, OpenCV, flask libraries.

Flask is used to serve the webpages for control

PyGame is used for playing audio locally through the RPI's speaker

OpenCV is called to interact with webcam.

This script creates a web app using Flask on http://127.0.0.1:5000

# Top Features

Centered Webcam view ()

Button for activating the 'main.py' - 'Robo DeMo 1000' - Uses ComputerVision to identify runners and shouts demotivational or motivational messages

Soundboard for taunting other runners in real life if Robo DeMo is just not enough.

Host System information (RPI5) is displayed on the bottom of the screen

# Design Considerations

No robot controls were added here because the robot is supposed to be semi autonomous.

VIAM Platform Website viam.com

I used this website to configure the robot.

Future work - There is going to be lots of future work

The more I worked on the project the more opportunities I saw for adding more features. Since this is part of a hackathon, I had to prioritize features that could be completed with the hackathon timeframe.

I will be adding field testing videos.

I hope to use Computer Vision facial recognition models to trigger the type of selected messages.

For example, if the person looks super stressed during a run then tell them something motivational or funny.

Future work: Save the output.mp3 files for later use to save on bandwidth and ElevenLabs API Usage.

Future work0: Custom head up display using a Flask Web App - Started on this 4/20/24. Could also add some lighting - see Future work7.

Future work1: Add wireless remote control via an ESP32 Microcontroller- Start and stop ESTOP function, instant jokes etc.

Future work2: Use ChatGPT to respond to speech shouted at the robot.

Future work3: I hope to use Computer Vision facial recognition models to trigger the type of selected messages. For example, if the person looks super stressed during a run then tell them something motivational or funny.

Future work4: Save the output.mp3 files for later use to save on bandwidth and ElevenLabs API Usage.

Future work5: Implement the Viam 'navigation service' using a GPS module and IMU (9 axis accelerometer).

Future work6: Use VIAM built-in computer vision to identify path markings for navigation and obstacle avoidance.

Future work7: Implement status RGB LEDs using the Neopixel library. The RPI5 currently does not support the Neopixel library due to a hardware change. # This feature could be implement using an Arduino or other microcontroller but I'm short on time.

# Things to indicate using Neopixel LEDs. Network Status / Direction travel / Processor Heartbeat / an attraction pattern to bring curious runners closer to the camera