Put Your Back Into It

Reinforcement learning (RL) is a revolutionary paradigm in machine learning that enables agents to learn optimal actions through interactions with their environment. This learning approach is heavily inspired by behavioral psychology and revolves around the concept of learning by trial-and-error. By seeking to maximize cumulative rewards, RL algorithms develop strategies that adapt to different situations, making them particularly well-suited for tackling complex problems in various domains.

A key advantage of the technique is that it is able to learn from experience. This means that it can adapt to new situations and improve its performance over time. This is in contrast to other machine learning approaches, such as supervised learning, which require a large amount of labeled data to train. Another advantage of RL is that it is able to deal with uncertainty. This is because it does not rely on a model of the environment, but instead learns from its own experience. This makes it well-suited for problems where the environment is complex or changing.

However, as the complexity of the problems increases, so does the demand for computational resources. Deep reinforcement learning often involves training deep neural networks on a massive amount of data. This is required to effectively learn intricate strategies and policies for complex tasks. Coupled with the need for extensive exploration to uncover optimal actions, the result is lengthy training times and substantial energy consumption. This poses a challenge in applying RL to tasks that involve real-time decision-making or those with vast state spaces, requiring a delicate balance between the problem's complexity and the available resources.

This problem is acutely felt when working with robotic manipulators. Consider for a moment how a person might pick up a large box, perhaps grabbing it with both hands, using the forearms for support, and maybe using a knee to give it a lift into position. For a RL algorithm to test all of the possible ways every point on the body could be used to pick up an object it encounters might take millions of years to compute in a simulated environment.

Some of us have more patience than others, but a million years is a bit much for anyone, so we tend to opt for simpler, yet perhaps less optimal solutions, like restricting a robot to only use its fingertips. We may not have to settle for second best in the future, thanks to the work of a group of engineers at MIT. They have found some ways to simplify the problem of contact-rich manipulation planning, such that the processing can be done in a short amount of time, even on a laptop.

The key insight by the researchers that enabled this innovation was that RL algorithms perform a sort of smoothing of the problem space by trying lots of options and then computing a weighted average. But, the smoothing process is very inefficient, requiring vast amounts of data and hardware resources.

To speed up the process, the team instead developed an algorithm that does the smoothing on the front end. While there is an infinite number of possible contact points, not all of them are important. For example, two points very close together will generally yield a virtually identical result. So rather than try all of them, their algorithm averages away the multitude of contact points in the smoothing process to greatly reduce the problem search space.

That led to major improvements in processing speed, however, for complex problems there still may be a large number of variables to explore. To deal with this problem, another algorithm was designed to rapidly and efficiently search through all possible decisions that could be made. The combination of these two techniques brought computation times down to about a minute on a typical laptop in a series of experiments with robotic hands.

The early tests focused on tasks like grasping a pen, opening a door, or picking up a plate. But in the future, the team believes that their algorithms could allow factories to leverage smaller robots to complete the same tasks presently done by large ones, or help an exploratory robot to adapt to unknown conditions on a distant planet.