Confusion Matrix: Friend or Foe?

This article describes the process of understanding confusion matrix in the context of data science model analysis.

If you’ve done any machine learning modeling, I’m certain you’re familiar with a confusion matrix. For those not familiar, a confusion matrix is a visual representation of how well (or poorly) a given model is performing. Let’s take a look at what a confusion matrix would look like for a simple classification problem which lends a 2x2 matrix.

Consider the following:

You’ve done it, you’ve completed your first simple model of the fruit classifier AI. Simply, with two classes of pictures (class 0 = no fruit, class 1 = fruit) the model tells us whether a photo contains fruit. So, given 20 images, 10 with fruit and 10 without fruit, it returns a prediction for the images. For this particular model, it classifies all images as containing fruit.

What does this mean for the future of your fruit classifier?

What could have caused this issue?

Is this an issue at all?

To help understand the results of this model, we can use a confusion matrix. The simplest confusion matrix is simply n rows and n columns of numbers, in this case, it would look something like this:

[0, 0]

[10, 10]

However, this does little to help us understand what the numbers are telling us. Adding some labels, borders, and usefully shaded colors, we can do much better. The image below shows what a more common (and useful) confusion matrix looks like:

Simple 2x2 confusion matrix

At first, this can be a lot to look at, so let’s break down what’s going on here.

Along the y axis, there’s information about the pictures, in this case, whether or not they contain fruit. This means anything in the first row does not have fruit in the picture, and those in the second row do.

For this confusion matrix, rows represent the actual images, NOT model predictions

On the x-axis, there’s information about what the model’s predictions. In other words, each column tells us what the model thinks in the picture.

For this confusion matrix, columns represent model predictions, NOT actual images

Bringing this together, this means the bottom right corresponds to pictures of fruit which the computer classified correctly. Here, this is true for 10 images.

In the bottom left, we see that the computer thinks there 10 images contain fruit, which are in fact fruitless.

False Positives in Green, True Positives in Yellow

One might ask:

Why bother with all these numbers, why not just say how many the computer got right and wrong?

It’s true that on the off diagonal both numbers are “wrong answers” given by the model, there’s a significant difference between a wrong classification in the bottom left and wrong answer in the top right. That is the difference between false positives (aka type 1 error) and false negatives (aka type 2 error).

Type 1 and Type 2 errors along off-diagonal

With the confusion matrix as a visual aid, it’s clear that all images are being classified as having fruit. There are several reasons a model would do this, one possibility is from a class imbalance, meaning the model was trained with more images of fruit than those without fruit. In either case, there’s some work to do before this model can be rolled out.

As you can see, there are in fact uses for the confusion matrix, and it isn’t simply there to confuse us all. Given the ease of use, aesthetics, and willingness to accommodate fruitless pictures, I classify the confusion matrix as Friend, not Foe.

With the different classifications and types of errors, other useful metrics can be extracted from the numbers within a confusion matrix. The most familiar metric is accuracy, there’s also precision, recall, and even something called the f1 score. These are all well-defined metrics that can be calculated with relative ease using various libraries like sklearn.