Model Behavior

In the past several months, there has been an explosion in the popularity of large language models (LLMs). They work by using neural networks to analyze and learn from vast amounts of text data, such as books, articles, and other written content. By processing this data, these models can identify patterns and relationships between words, phrases, and sentences, allowing them to generate human-like responses to questions.

This gives LLMs the ability to perform a wide range of natural language processing tasks, such as language translation, text summarization, and sentiment analysis. This makes them highly versatile tools that can be used in a variety of industries, from healthcare and finance to education and entertainment.

But one significant drawback associated with large language models is their lack of interpretability. While the results they produce can be very impressive and convincing, it is generally challenging to understand how they arrive at their conclusions. This is because these models rely on complex neural networks that process and analyze vast amounts of data, making it difficult to trace the logic and reasoning behind their outputs. This lack of interpretability raises concerns about how much trust we should place in these models, and makes it challenging to identify and address potential errors or issues in the model's decision-making process.

OpenAI, the developers of the popular ChatGPT chatbot, powered by a LLM, have recently reported on some early results of their efforts to better understand the behavior of neurons in large language models. Their belief is that an understanding of what the individual components (neurons) of an LLM do should aid interpretability research. Towards that end, they have developed an approach that leverages LLMs to interpret other LLMs.

At a high level, the method operates by running a text prompt through a LLM that one wants to interpret, and watches each individual neuron, one at a time, to see when it is activated, and how strongly. These activation patterns are then shown to a second, more powerful LLM (in this case, OpenAI’s GPT-4 model), and it is asked to provide an explanation of the patterns that it is seeing.

To assess how accurate that explanation is, another set of steps are initiated in which the explainer model is supplied with another text prompt. It is then instructed to simulate how the neuron in the smaller model would behave to answer that prompt. Next, that initial model is also prompted with the same text. The simulated result from the explainer model is compared with the actual result from the model that we want to understand. The more similar the results are, the better we can take the explanation to be.

This is a relatively simple concept, and has the potential to serve as a powerful tool in enhancing interpretability, however there is still quite a lot of work to do. For starters, examining each neuron in isolation requires huge amounts of computational resources, and as the size of the models expands into the billions of neurons, and beyond, it quickly becomes impractical to make all the calculations. Further, the approach has been found to be very hit or miss thus far. Well, mostly miss, really. OpenAI set out to explain a GPT-2 model, with 307,200 neurons. Only about 1,000 of those neurons could be explained with a high level of accuracy.

Certainly, this approach does add to our knowledge of how LLMs function, and why they make the decisions that they do. Yet, there is a long way to go before we have a thorough explanation. But with the pace of advances we have been seeing in machine learning recently, we may not have to wait very long after all.