This Amazing AI-Powered Machine Can Sort Every LEGO Brick Ever Made

You probably had at least a few different sets of LEGO bricks when you were a kid, and those bricks likely all eventually ended up mixed together in a single bin. Whenever you wanted to assemble a new creation, you had to sift through that bin to find the specific bricks you needed for the job. It would be great if a machine could automatically recognize and sort a bin full of LEGO bricks, and that’s exactly what Daniel West’s LEGO Sorting Machine does — and it can do it with every LEGO brick ever made.

LEGO bricks — at least the style we all know and love — have been manufactured since 1958. About 20 billion bricks are produced every year, and roughly 440 billion LEGO “elements” have been made since 1949 — though that number does include the earlier “Automatic Binding Bricks” that are different than the bricks that would ultimately become famous. It’s difficult to estimate how many unique types of bricks have been made over the years, and the number depends on whether or not you want to count differing colors, stickers, and other small variations as “unique.” But the number ranges from somewhere between 9,000 if you don’t even count colors as unique, to 60,000 if you judge any difference as unique.

Most artificial intelligence convolutional neural networks (CNNs) are trained on manually-compiled data. For example, if you were building a neural network to decide if a photo contains either a dog or a cat, you would train it with as many pictures of dogs as you could find for the “dog” output and then do the same for the cats. You want to have enough variety in those photos to account for different breeds, ages, lighting conditions, and so on. Even a modestly-sized training data set can contain hundreds of images of each type. So how did West assemble that training data for thousands of bricks — many of which are rare and difficult to acquire?

His solution was to use 3D models to generate the photos needed to train his convolutional neural network. Rebrickable has a database that includes 3D models for virtually every brick that has ever been produced. Using an automated workflow, West imported those models into Blender in order to render realistic images. Those were mixed with a handful of real images and randomized data in order to overcome the “Simulation to Reality” problem. A Raspberry Pi computer takes photos of the brick to be identified, and sends them to a separate, more powerful computer that runs the CNN. After identification, the Raspberry Pi is told how to sort that particular brick.

The machine that West built to do the sorting was, appropriately, made almost completely out of LEGO parts itself. That even includes the motors, which come from sets like LEGO Mindstorms. An entire bin of LEGO bricks can be dumped into the machine’s hopper, and a series of conveyor belts and a vibratory platform will separate them before they reach the camera. After identification, a bunch of gates are used to direct the brick into one of 28 different sorting bins. Even if you don’t care about LEGO, this is a very impressive use of artificial intelligence.