Why does binary accuracy give high accuracy while categorical accuracy give low accuracy, in a multi-class classification problem?
So you need to understand what happens when you apply a binary_crossentropy
to a multiclass prediction.
- Let's assume that your output from
softmax
is(0.1, 0.2, 0.3, 0.4)
and one-hot encoded ground truth is(1, 0, 0, 0)
. binary_crossentropy
masks all outputs which are higher than0.5
so out of your network is turned to(0, 0, 0, 0)
vector.(0, 0, 0, 0)
matches ground truth(1, 0, 0, 0)
on 3 out of 4 indexes - this makes resulting accuracy to be at the level of 75% for a completely wrong answer!
To solve this you could use a single class accuracy, e.g. like this one:
def single_class_accuracy(interesting_class_id):
def fn(y_true, y_pred):
class_id_preds = K.argmax(y_pred, axis=-1)
# Replace class_id_preds with class_id_true for recall here
positive_mask = K.cast(K.equal(class_id_preds, interesting_class_id), 'int32')
true_mask = K.cast(K.equal(y_true, interesting_class_id), 'int32')
acc_mask = K.cast(K.equal(positive_mask, true_mask), 'float32')
class_acc = K.mean(acc_mask)
return class_acc
return fn
Ninja
Updated on July 28, 2022Comments
-
Ninja almost 2 years
I'm working on a multiclass classification problem using Keras and I'm using binary accuracy and categorical accuracy as metrics. When I evaluate my model I get a really high value for the binary accuracy and quite a low one in for the categorical accuracy. I tried to recreate the binary accuracy metric in my own code but I am not having much luck. My understanding is that this is the process I need to recreate:
def binary_accuracy(y_true, y_pred): return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1)
Here is my code:
from keras import backend as K preds = model.predict(X_test, batch_size = 128) print preds pos = 0.00 neg = 0.00 for i, val in enumerate(roundpreds): if val.tolist() == y_test[i]: pos += 1.0 else: neg += 1.0 print pos/(pos + neg)
But this gives a much lower value than the one given by binary accuracy. Is binary accuracy even an appropriate metric to be using in a multi-class problem? If so does anyone know where I am going wrong?