Shhh! Less Noise Makes for Better AI

When solving any complex problem, it is necessary to make some assumptions. These assumptions serve as the foundation for our analysis and help us simplify the problem into manageable components. This is fine and well so long as those assumptions actually hold true — but if not, they can silently sabotage the entire effort. For this reason, it is essential to test the validity of our assumptions. But some have become so foundational that they are generally just taken for granted.

One such assumption in reinforcement learning (RL) is that AI agents perform best when trained in an environment that closely matches the one they will be deployed in. This principle has guided the design of RL training for years, ensuring that an agent’s learned behaviors translate effectively from simulation to real-world application. However, a team of researchers from MIT, Harvard, and Yale has now discovered that this assumption does not always hold true. Their findings challenge conventional wisdom and introduce a novel concept they call the Indoor Training Effect.

The researchers found that, in some cases, AI agents trained in a low-noise, simplified environment performed better in a noisier, more unpredictable test environment than those trained directly in that noisy space. This is counterintuitive to traditional RL approaches, which attempt to match training conditions as closely as possible to deployment environments.

To explore this phenomenon, the researchers trained AI agents to play modified Atari games with varying levels of randomness. In one set of experiments, they introduced noise into the transition function, which governs how the game environment responds to an agent’s actions. For example, in the classic game Pac-Man, a transition function might define the probability that ghosts move up, down, left, or right.

According to conventional RL training methods, an agent trained in an environment with added randomness should be best prepared for deployment in that same environment. However, the results showed otherwise.

The team found that agents trained in a noise-free Pac-Man environment consistently outperformed those trained in a noisy version of the game — even when both were tested in the noisy environment. The same trend appeared across 60 different variations of Atari games, including Pong and Breakout, demonstrating the robustness of the Indoor Training Effect.

To understand why this effect occurs, the researchers examined how the AI agents explored their training spaces. When two agents — one trained in a noise-free environment and the other in a noisy environment — explored the same regions, the noise-free agent tended to perform better. The researchers hypothesize that this is because the noise-free training environment allows agents to learn the game’s fundamental mechanics without interference, building a stronger baseline of knowledge that translates well to uncertain settings.

However, if an agent’s exploration pattern in the noise-free environment was significantly different from the noisy environment, then the noisy-trained agent had an advantage. This suggests that while Indoor Training can be beneficial, its effectiveness depends on how much the training and testing environments influence the agent’s ability to explore critical areas.

The discovery of the Indoor Training Effect opens up new possibilities for designing more effective RL training strategies in the future. Instead of solely focusing on replicating real-world complexity, researchers may benefit from strategically simplifying training conditions to enhance learning efficiency and adaptability.