See Spot Talk

Here at Hackster News, we often report on innovations in robotics and artificial intelligence (AI). But what happens when you combine these two technologies? A lot of digital ink has been spilled in recent years trying to convince us that this conglomeration will ultimately lead to a real-life Skynet and the end of humanity. But a more sober assessment of these technological advancements does not support such doomsday scenarios. Instead, it reveals the remarkable potential for improving various aspects of our daily lives.

This is not to say that no precautions should be taken. There are valid concerns about the ethical implications and potential misuse of AI-integrated robotics, but the current trajectory of this amalgamation suggests a future where these technologies serve as valuable tools in enhancing productivity, streamlining complex tasks, and fostering unprecedented advancements across diverse industries.

Or, these advancements may simply lead to the development of robot dogs that we can have a nice chat with. Boston Dynamics, the company behind the famous robot dog Spot, has leveraged some popular generative AI tools to turn their robot into an entertaining tour guide for visitors of their facility. Sure, talking with this robot might be a little bit unsettling, but it is hardly likely that a robot with something like a fancy autocomplete algorithm will be taking over the world any time soon. And if one can get past the initial strangeness of chatting with a robot, it looks like it could provide a great tour experience.

A stock Spot robot dog was fitted with a robotic arm, that when adorned with googly eyes, hats, and various other accessories looks vaguely like a head. Robust SDKs already exist to help Spot autonomously navigate, so getting around the facility was already a solved problem. As such, the team turned their attention to the chatting capabilities. A Seeed Studio ReSpeaker Mic Array v2.0 was attached to the back of the device to capture speech from nearby users. Visual information was acquired from a pair of cameras, one on the front of the robot, and another attached to the “head.”

After detecting the wake words “Hey, Spot” audio was captured and transmitted to OpenAI’s automatic speech recognition API called Whisper. This translated the speech into text. At the same time, images were regularly captured by the robot’s pairs of cameras before being fed into a BLIP-2 algorithm in either visual question answering mode or image captioning mode. This gave the robot real-time information about its immediate surroundings to help it better describe what tour guests were seeing, and also to provide better answers to their questions. The text result produced by this model was combined with the transcription of the user’s speech to form a prompt that was sent to OpenAI’s ChatGPT API, where the GPT-4 model was leveraged.

To keep things light and fun, these prompts are also given a personality, like a sarcastic, unhelpful robot, a 1920s archaeologist, or a Shakespearean actor. Finally, a cloud service from ElevenLabs was used to convert the textual response from ChatGPT into audio that can be played over a speaker on the robot.

The team noted that while their tour guide was not always 100% accurate in its answers to guests’ questions, it was not necessarily a problem for this application where the goal was primarily entertainment. Despite these issues, some interesting behavior was seen. For example, when asked who Marc Raibert is (the Executive Director of the Boston Dynamics AI Institute), the robot said that it did not know, but it made the connection that it could lead the questioner to the IT help desk for more information.

Perhaps the biggest issue with Spot is that the latency between question and answer can be as high as six seconds. But given how entertaining this robot is, I think visitors would still be thrilled to get a robotic tour.