What'choo talkin' 'bout?

A picture with moving eyes has been a classic element in horror films almost as long as there have been horror films. Each time someone looks at the picture something seems a little off, but they cannot quite put their finger on it until the eyes finally move to meet theirs while they look on in terror. If that sort of shtick makes your skin crawl, you might not be a huge fan of the latest project by Redditor benbenson1. It may not have creepy moving eyes, but benbenson1’s AI-powered ePaper picture frame silently eavesdrops on your conversations and periodically displays images that represent exactly what you have been talking about. Spooky!

If you know how it works, it is not so spooky, however. You can rest easy knowing that the picture frame is not actually haunted, it is just running a whole lot of multiply-accumulate operations.

On the hardware front, the picture frame consists of a Raspberry Pi 4 Model B, a Pimoroni Inky Impression 7.3-inch seven-color ePaper HAT, and a Seeed Studio reSpeaker 2-Mics Pi HAT V2.0. Raw audio collected from the microphones is streamed to a nearby server with netcat. This server, which is equipped with an NVIDIA GeForce RTX 3060 12GB GPU, uses Faster-whisper to transcribe the audio. Transcriptions are then processed by the FLUX text-to-image generator, which is running locally on the same server. The images it produces are then sent back to the Raspberry Pi for display on the ePaper screen.

During normal operation, the picture frame will update every five minutes. There are also some buttons that can be used to change that behavior, however. One button makes it possible to generate an image from an intentionally spoken description. Another turns off the microphone if you don’t want to see everything you discuss displayed for all to see. You can also instruct the device to show the prompt used to generate an image, or save your favorite pictures for later viewing.

There is still some work yet to be done on this device before benbenson1 can call it a wrap. Some of the functions are not fully working, and the hardware doesn’t quite fit into the case. It also takes about 30 seconds for the screen to refresh, which is not too bad for the normal five minute updates, but is pretty bad for when new images are requested via a button press. benbenson1 is looking for some help from the community to get these issues resolved, so head on over to the GitHub repository and get your pull requests in!