plotting confusion matrix for an image classification model

Question

I built an image classification CNN with keras. While the model itself works fine (it is predicting properly on new data), I am having problems plotting the confusion matrix and classification report for the model.

I trained the model using ImageDataGenerator

train_path = '../DATASET/TRAIN'
test_path = '../DATASET/TEST'
IMG_BREDTH = 30
IMG_HEIGHT = 60
num_classes = 2

train_batch = ImageDataGenerator(featurewise_center=False,
                                 samplewise_center=False, 
                                 featurewise_std_normalization=False, 
                                 samplewise_std_normalization=False, 
                                 zca_whitening=False, 
                                 rotation_range=45, 
                                 width_shift_range=0.2, 
                                 height_shift_range=0.2, 
                                 horizontal_flip=True, 
                                 vertical_flip=False).flow_from_directory(train_path, 
                                                                          target_size=(IMG_HEIGHT, IMG_BREDTH), 
                                                                          classes=['O', 'R'], 
                                                                          batch_size=100)

test_batch = ImageDataGenerator().flow_from_directory(test_path, 
                                                      target_size=(IMG_HEIGHT, IMG_BREDTH), 
                                                      classes=['O', 'R'], 
                                                      batch_size=100)

This is the code for the confusion matrix and classification report

batch_size = 100
target_names = ['O', 'R']
Y_pred = model.predict_generator(test_batch, 2513 // batch_size+1)
y_pred = np.argmax(Y_pred, axis=1)
print('Confusion Matrix')
cm = metrics.confusion_matrix(test_batch.classes, y_pred)
print(cm)
print('Classification Report')
print(metrics.classification_report(test_batch.classes, y_pred))

for the confusion matrix I get the rolling result (which seems to be wrong)

Confusion Matrix
[[1401    0]
 [1112    0]]

The False positives and true positives are 0. For the classification report I get this following output and warning

Classification Report
             precision    recall  f1-score   support

          0       0.56      1.00      0.72      1401
          1       0.00      0.00      0.00      1112

avg / total       0.31      0.56      0.40      2513

/Users/sashaanksekar/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)

I am trying to predict if an object is organic or recyclable. I have around 22000 train images and 2513 test images.

I am new to machine learning. what am I doing wrong?

Thanks in advance

Solution 1

If someone got here like me because of similar issue there may be several things that could help:

1. Make sure you set shuffle = False in your test set generator;
2. It's better to set the batch_size to a divisor of your image count. If not - make sure the generator doesn't skip any images;
3. Try training without augmentation first;

4. There seems to be an issue where output of the predict_generator is not consistent, try setting workers = 0 if possible, like this:

   predictions = model.predict_generator(testGenerator, steps = np.ceil(testGenerator.samples / testGenerator.batch_size), verbose=1, workers=0)

In my case the predictions changed each time I called predict_generator if I didn't do it.

6. When you have only two classes you have to use:

   predictedClasses = np.where(predictions>0.5, 1, 0) instead of np.argmax(Y_pred, axis=1) since in this case np.argmax will always output 0.

   np.where(predictions>0.5, 1, 0) returns 1 if prediction > 0.5 else returns 0.

Solution 2

To plot the confusion matrix do the following:

import matplotlib.pyplot as plt
import numpy as np

cm = metrics.confusion_matrix(test_batch.classes, y_pred)
# or
#cm = np.array([[1401,    0],[1112, 0]])

plt.imshow(cm, cmap=plt.cm.Blues)
plt.xlabel("Predicted labels")
plt.ylabel("True labels")
plt.xticks([], [])
plt.yticks([], [])
plt.title('Confusion matrix ')
plt.colorbar()
plt.show()

References:

https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/

https://machinelearningmastery.com/confusion-matrix-machine-learning/

Solution 3

I use sklearn plot_confusion_matrix

To use it I made a hack so when the sklearn estimator makes prediction dont complaints because is a Keras model. So, if model is a trained keras model:

X,y = test_generator.next()
y = np.argmax(y, axis=1)

from sklearn.metrics import plot_confusion_matrix
class newmodel(MLPClassifier):
    def __init__(self, model):
        self.model = model
    def predict(self, X):
        y = self.model.predict(X)
        return np.argmax(y,axis=1)

model1 = newmodel(model)
plot_confusion_matrix(model1, X, y , normalize='true', xticks_rotation = 'vertical', display_labels = list(train_generator.class_indices.keys()))

It works for me.

Comments

[Admin]

How should I interpret the confusion matrix?

[seralouk]

Hello. The confusion matrix shows a lot of thing at the same time. First of all, the performance is higher when the black blocks are on the diagonal. Here, you are very good at predicting the one class but very very bad at predicting the other class. Additionally, from the confusion matrix, you can calculate the sensitivity, specificity, false positive rate, false negative rate etc... I will add some references about the CM in my answer

[Admin]

I feel I have my y_pred initialisation wrong. How do I initialise y_pred?

[seralouk]

y_pred is the predicted labels of the testing data. Before model.predict_generatordo you fit the model using the training data ?

[Admin]

I used fit_generator to train the network. When I use the predict function to predict on new data, the model works fine most of the time. So the TP in the confusion matrix should not be 0.