Retro Video Game Translation Tablet with IoT Edge

Demonstration

Backstory

I arrived on the scene of Akihabara, a neon-lit mecca located in a prefecture of Tokyo that is considered “the center of the video game universe". Retro video games of the golden 80’s and 90’s eras decorate storefronts and second-hand store shelves in abundance, oftentimes in noteworthy condition with boxes and manuals re-wrapped in protective plastic. It is a sight to behold, and for import gaming enthusiasts it’s an amazing opportunity to pick up titles that never saw release outside of Japan.

I left with three suitcases full of import consoles and games. These included the successor to NEC’s PC-Engine known as the PC-FX, a computer-tower shaped system whose entire library was only ever released in Japan. Upon my return stateside, I discovered that some of the titles I’d come away with (particularly PC-FX games) were heavily laden with Japanese text. Unfortunately, I could not read these characters, rendering some of the games unplayable to me. Little did I know, this moment would later spark the inspiration to take on the issue of live-translating retro video game text using modern advances in technology.

Inspiration

In our modern technological age, it seems like everything is connected to the internet in some way. From smart light bulbs, to garage door openers, all the way to convenient virtual assistants that can perform a variety of tasks by calling on them by name. Many have referred to this paradigm as “the Internet of Things," a trend which has brought us enhanced versions of everyday devices, markedly enhanced by giving them accessibility to the internet and all that it has to offer.

Oftentimes, these “IoT” devices are augmented using advances in computer vision, artificial intelligence, and machine learning. In the software development world, adopting these types of advanced features into custom projects has become service oriented, allowing users to potentially take advantage of highly tuned AI models by calling a web endpoint which exposes the service.I was thrilled to learn that Microsoft offers an offline form of service-oriented AI in its Cognitive Services Containers offering. This can allow a user to run a local copy of services like Object Character Recognition (OCR), which traditionally requires significant compute resources in a local environment. Even more amazing, it can produce results without connection to the internet!

This seemed like it could be a great way to produce a system for making import video games accessible to almost any audience. We can use a locally running instance of the container to get the screen text, then all we need to do is translate the text and redraw it to a screen and voila! Instant access to a library of previously unplayable games!

Solving the Problem

To solve the problem of translating retro video games on the fly, one needs to detect text symbols in the video output, translate that text to a desired target language, and output the translated text over the original characters.

Capturing the video output can be done in a variety of ways, but I wanted something robust that could support the most game consoles with the least amount of effort. I stumbled across lakka.tv, a custom Linux distribution that prioritizes running the popular Retroarch emulator front-end which supports almost every video game console ever created, including the PC-FX. Using this, we can build a system around it to perform translation in response to screen capture events to effectively live-translate our games on demand.

To accomplish translation without interrupting gameplay, I perform the translation on a tablet device screen which also outputs the unaltered video output over HDMI. This way, we can translate games on the fly without having to deal with lag effects on the main screen.

To allow for deployment of the OCR module down to the device, I employed another Microsoft technology known as Azure IoT Edge. Getting this to run on Lakka required re compiling it’s Linux kernel with some additional features to support running containerized applications. Once I had the IoT Edge runtime running on my target device, I was able to use it to distribute containerized applications as modules using a configuration stored in the cloud.

I then created a custom module which monitors a particular area of the filesystem for screenshots. Once a screenshot is detected, it is sent over to the OCR module which extracts text from the image data and returns a bounding box of coordinates for each detected phrase. After obtaining the detected text and coordinates, I send the text over to the Azure Translator Text API service for translation and display the results over the original bounding box area by writing directly to the framebuffer of the tablet screen.

Thank goodness for the Libretro API, because the common features it provides allow this project to works with any compatible emulator core that supports Retroarch.

There are some additional features that IoT Edge allows which gives the project some interesting capabilities. For example, I can adjust a parameter in the cloud which propagates down to the device to allow the user to change the target language. In this way, users can translate to any of the 64 supported languages including Japanese and Klingon on demand. It also allows for use of custom fonts, so you can display characters in a wide variety of languages to match the target translation.

Steps to Reproduce

Hardware

The following items are required in order to successfully replicate this content:

• 1x Samsung Series 7 Slate Tablet OR1x X64 compatible PC tablet with at least 4GB RAM and HDMI video out

Software

• Visual Studio Code with IoT Edge extension installed
• 1x Azure Subscription (Free Trial available)

1.) Download the latest Lakka.img.gz release from this fork of Lakka-LibreELEC and install it to your tablet device by following these instructions.

2.) After installation, configure the device to output the RetroArch UI over HDMI by selecting the appropriate monitor index in Video Settings, you may also want to configure audio output to route over HDMI as well by configuring the appropriate Audio Device in Audio Settings.

3.) Configure the Wifi settings to connect to an approprite access point in the WiFi-Configuration Interface

4.) SSH into your Lakka Device by following these instructions and execute the following commands to install the Docker Add-On:

cd ~/.kodi/wget https://github.com/toolboc/Lakka-LibreELEC/releases/download/RetroArch-AI-with-IoTEdge_v1/service.system.docker-8.2.122.zipunzip service.system.docker-8.2.122.zipcd ~/.kodi/addons/service.system.docker/system.d/systemctl enable service.system.docker.servicereboot

Note: Docker will install to the storage partition and is accessible via the path:

/storage/.kodi/addons/service.system.docker/bin/docker

Verify that docker is running with:

export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/storage/.kodi/addons/service.system.docker/bindocker ps

5.) - Create an Azure IoT Hub by following these instructions then create a new IoT Edge Device in the Azure Portal and retrieve the connection string.

6.) - SSH into your Lakka Device by following these instructions and verify that docker is running with:

export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/storage/.kodi/addons/service.system.docker/bindocker ps

Create an instance of an azure-iot-edge-device-container with (be sure to replace with the key obtained in Step 5):

docker run --name edge-device-container --restart always -d --privileged -v /storage/screenshots:/storage/screenshots -v /dev/fb0:/dev/fb0 -e connectionString='<IoTHubDeviceConnectionString>' toolboc/azure-iot-edge-device-container

7.) - Install Visual Studio Code onto an available development machine and install the Azure IoT Edge extension for Visual Studio Code.

8.) - Clone or download a copy of this repo and open the RetroArch-AI-with-IoTEdge folder in Visual Studio Code. Next, press F1 and select Azure IoT Hub: Select IoT Hub and choose the IoT Hub you created in Step 5, follow the prompts to complete the process.

9.) - In VS Code, navigate to the deployment.template.json file and modify the following values:

• {YourPrivateRepoUsername} - The username provided for access to the Cogntive Services Computer Vision Containers Preview.
• {YourPrivateRepoPassword} - The password provided for access to the Cogntive Services Computer Vision Containers Preview.
• {YourCognitiveServicesApiKey} - Follow these instruction to create a Cognitive Services Resource in Azure, obtain and replace with the respective key.
• {YourBillingEndpointLocale} - The locale prefix used for the endpoint of your Cognitive Services Resource
• {YourTranslatorTextApiKey} - Follow these instruction to create a Cognitive Services Translator Text API Resource in Azure, obtain and replace with the respective key.

10.) - Create a deployment for the IoT Edge device by right-clicking deployment.template.json and select Generate IoT Edge Deployment Manifest. This will create a file under the config folder named deployment.amd64.json, right-click that file and select Create Deployment for Single Device and select the device you created in Step 5.

Wait a few minutes for the deployment to complete and you should see the logo for the RetroArch Screenshot Translator appear on the main screen of your Lakka machine.

Configure a Hotkey or manually select "Take Screenshot" while playing a game, the translated image is stored to /storage/screenshots/translated/ and displayed on the main screen of the Lakka Device.

Once you have the device confirmed working, you will want to check out the official documentation in the GitHub Repo to learn more about available features and how to use them.

Conclusion

At this time, the project can only translate from “English” due to feature availability in the OCR module, but is expected to support a variety of new languages in the future. Even with this limitation, the project is extremely fulfilling in that it enables players across the world the ability to access the English library of games in their preferred language. In the meantime, I will patiently await for Japanese OCR support to finally enable accessibilityto the PC-FX library in my native tongue!

The full project which goes by the name“RetroArch-AI-with-IoTEdge” and is open-source and available on GitHub @ https://github.com/toolboc/RetroArch-AI-with-IoTEdge

For those of you hoping for official support for live translation in Retroarch, I learned about an open bounty on bountysource which seeks to enable this functionality. The current solution takes a very different approach but is very promising and is slated for completion soon. It is a very exciting world that we live in where language barriers can now be alleviated with some help from a little localized Artificial Intelligence!

If you enjoyed this content and want to stay up to date on my latest projects, you can follow me @pjdecarlo on Twitter.

I am always interested to learn how others interpret this content for their own use. If you have any cool ideas to enhance this project or ideas of your own that were inspired by this content please share them in the comments.

Until next time, Happy hacking!