Machine Learning Confidence Crisis
This tool detects ML models that have learned to classify objects by using irrelevant details, like backgrounds or random patterns.
Is this an umbrella or an elephant? Shown a picture of one or the other, this question would be incredibly simple for a human to answer correctly. For a machine learning image classifier, it can be quite a lot more complicated. This is because these classifiers can zero in on elements of the background, the image border, or other irrelevant details — details a human would inherently understand to be irrelevant—and consider them to be of prime importance in identifying the object of interest in that image. Worse yet, when the models make bad predictions, they can do so with a very high degree of confidence that they are actually correct.
While we may not presently have better solutions to the classification problem that allow machines to ignore unimportant details as a person would, we do now have a method to at least help us recognize when they go astray in blissful ignorance. Researchers at MIT CSAIL and Amazon Web Services have collaborated to build a toolset that can identify models that are prone to this type of classification error, which they refer to as overinterpretation.
At a high level, this tool will methodically remove pieces of an image and then classify it. It will continue this process to find the smallest image subset that is still classified with a high degree of confidence. When small, irrelevant portions of images are found to produce high-confidence classifications, the model is exhibiting an overinterpretation error and will need further work to remediate this situation.
In their work, the team found that this is not just a problem that pops up with models trained on small, custom datasets. It has also been found in models trained on large, well-known, and commonly used datasets such as CIFAR-10 and ImageNet. This indicates that overinterpretation is likely to be a rampant problem in the world of image classification. This is a particularly troubling finding when considering that image classifiers are used in a wide range of applications, including medical diagnosis and autonomous vehicle navigation.
While the focus of this research was to detect the presence of overinterpretation in a model, it does raise the question that naturally follows — what can we do about it? Certainly further research will need to be conducted to provide an adequate answer, however, present findings that models can focus on image backgrounds or borders suggest that we may need new datasets, in which the objects we wish to classify are depicted with uninformative backgrounds. It may be the case, ironically, that if we want to recognize objects in the real world, we need to make use of images that were captured in controlled environments, rather than a diverse set of images from the real world.