Researchers Develop Precocial Neural Networks That Demonstrate Inherent Skill Without Weight Tuning

Google Brain's Adam Gaier and David Ha's weight-agnostic neural networks are inspired by ducklings and snakes.

Researchers from Google Brain, the company's artificial intelligence and deep learning division, have published a paper suggesting that it's possible for an AI to become proficient at various tasks without lengthy weight-adjustment learning — previously considered a key step in the process.

"In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task," researchers Adam Gaier and David Ha explain. "We propose a search method for neural network architectures that can already perform a task without any explicit weight training. To evaluate these networks, we populate the connections with a single shared weight parameter sampled from a uniform random distribution, and measure the expected performance.

"We demonstrate that our method can find minimal neural network architectures that can perform several reinforcement learning tasks without weight training. On supervised learning domain, we find architectures that can achieve much higher than chance accuracy on MNIST using random weights."

Inspired by biology — in particular precocial species, those whose young already possess certain abilities from the moment of birth such as the duckling's innate ability to swim or the baby snake's ability to escape predators — the pair worked on creating "natural" neural networks which are already capable of performing a target task despite being given randomised weight parameters, the adjustment of which is usually a key step in "teaching" an AI.

The result is what the pair call weight-agnostic neural network (WANN) search, and for the three tested tasks — swinging and balancing a pole attached to the top of a cart, guiding a two-legged walker across randomly-generated terrain without it toppling over, and driving a racing car in a top-down environment — it proved impressively effective. "In contrast to the conventional fixed topology networks used as baselines," the pair report, "which only produce useful behaviours after extensive tuning, WANNs perform even with random shared weights."

The result is a simpler neural network, and one which can still be trained — but simply by adjustment of a single shared weight value, rather than having to tune multiple variables. "While WANNs are able to perform without training," the pair add, "this predisposition does not prevent them from reaching similar state-of-the-art performance when the weights are trained."

The team's paper is available on the project's GitHub page, along with slides from its presentation at the NeurIPS 2019 conference and interactive demos of the technique in use, while an interview with Gaier can be found on IEEE Spectrum.

Gareth Halfacree
Freelance journalist, technical author, hacker, tinkerer, erstwhile sysadmin. For hire: freelance@halfacree.co.uk.
Latest articles
Sponsored articles
Related articles
Latest articles
Read more
Related articles