The Path to Understanding

In recent years, there have been many notable successes in the field of artificial intelligence (AI), particularly with large language models and computer vision models. These models, such as OpenAI's ChatGPT and Meta AI's Llama, have demonstrated tremendous value in tasks like natural language understanding, text generation, and sentiment analysis. Similarly, computer vision models like convolutional neural networks have made significant strides in object detection, image classification, and facial recognition.

However, despite their achievements, these AI models operate within highly-constrained domains and focus on a small set of specific use cases. While they excel at tasks like language processing or image recognition, they lack a deep understanding of the physical world and struggle to solve broader, more complex problems. For reasons such as this, many researchers believe that achieving artificial general intelligence, which encompasses a comprehensive understanding of the world and the ability to solve a wide range of problems, remains a distant goal.

There is a growing belief that the next step toward more intelligent machines will be achieved through a paradigm known as physical AI. This new form of AI aims to enable machines to understand the world at a deeper level by incorporating data from a wide variety of data sources such as radars, cameras, accelerometers, temperature sensors, and much more. By integrating sensor data with large AI models, physical AI systems can learn complex patterns and dynamics that exist in the real world.

The team at Archetype AI is at the forefront of this trend. They are convinced that the previous decade’s Internet of Things boom that provided us with hundreds of billions of inexpensive sensors, in combination with the adoption of large-scale cloud computing systems, is the fuel that will help physical AI to finally arrive and live up to expectations. By leveraging these technologies, they have developed a new type of foundation model, called Newton, that they claim is the first to have an understanding of the physical world.

Newton has been dubbed a large behavior model because of its unusual ability to detect complex patterns in diverse sources of data. This was achieved by training the model with text, radar measurements, IMU data, chemical and environmental data, and more. All of these types of data are integrated into a single model, such that the relationships between them can be understood. The hope is that such a model will find patterns that are beyond humans’ ability to perceive, giving us a better window into the workings of physical systems.

By asking questions of Newton, what the team refers to as a “semantic lens” is created. This focuses the model’s attention on a particular aspect of its understanding of the world, given the sensor data it is receiving. In this way, the model can provide users with complex answers about what is going on in the world around them.

Newton is already having an impact on real-world problems by providing self-driving vehicles with situational awareness, improving safety and reducing waste at construction sites, and enhancing the energy efficiency of homes. Ultimately, Archetype AI hopes to encode the entire physical world into a single AI algorithm. That goal is likely quite a long way off, but you have to start somewhere.