keras predict always output same value in multi-classification

Question

here is my code. I want to made a multi-classification using Keras. the vcl_acc is better when training, But the predict value always is a same value. I confused,please help me

train.py

# coding: UTF-8
# author: Sun Yongke ([email protected])
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import EarlyStopping
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.optimizers import SGD


# Dense(64) is a fully-connected layer with 64 hidden units.
# in the first layer, you must specify the expected input data shape:
# here, 20-dimensional vectors.

# dimensions of our images.
img_width, img_height = 300, 300
nb_epoch=20

train_data_dir = '../picture/samples_300_2/train'
validation_data_dir = '../picture/samples_300_2/validation'
# this is the augmentation configuration we will use for training
train_datagen = ImageDataGenerator(
    rescale=1./255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

# this is the augmentation configuration we will use for testing:
# only rescaling
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=40000,
    color_mode='grayscale',
    save_format="jpg",
    save_to_dir="after/train",
    class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=500,
    color_mode='grayscale',
    save_format="jpg",
    save_to_dir="after/test",
    class_mode='categorical')

model = Sequential()
# input: 100x100 images with 3 channels -> (3, 100, 100) tensors.
# this applies 32 convolution filters of size 3x3 each.
model.add(Convolution2D(32, 3, 3, border_mode='valid', input_shape=(img_width, img_height,1)))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Convolution2D(64, 3, 3, border_mode='valid'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
# Note: Keras does automatic shape inference.
model.add(Dense(256))
model.add(Activation('relu'))
model.add(Dropout(0.5))

model.add(Dense(14))
model.add(Activation('softmax'))

sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd)

nb_train_samples=len(train_generator.filenames)
nb_validation_samples=len(validation_generator.filenames)

early_stopping = EarlyStopping(monitor='val_loss', patience=2)

model.fit_generator(
    train_generator,
    samples_per_epoch=nb_train_samples,
    nb_epoch=nb_epoch,
    validation_data=validation_generator,
    nb_val_samples=nb_validation_samples, callbacks=[early_stopping])
#save model
model.save("sykm.2.h5")

training output as follow

Epoch 2/50
3005/3005 [==============================] - 270s - loss: 0.2227 - acc: 0.9294 - val_loss: 0.1985 - val_acc: 0.9316
Epoch 3/50
3005/3005 [==============================] - 269s - loss: 0.2105 - acc: 0.9310 - val_loss: 0.1858 - val_acc: 0.9338
Epoch 4/50
3005/3005 [==============================] - 271s - loss: 0.1964 - acc: 0.9333 - val_loss: 0.3572 - val_acc: 0.9160
Epoch 5/50
3005/3005 [==============================] - 268s - loss: 0.1881 - acc: 0.9349 - val_loss: 0.1513 - val_acc: 0.9413
Epoch 6/50
3005/3005 [==============================] - 268s - loss: 0.1935 - acc: 0.9342 - val_loss: 0.1581 - val_acc: 0.936

predict.py

# coding: UTF-8
# author: Sun Yongke ([email protected])
from keras.models import load_model
model = load_model('sykm.2.h5')
img_width, img_height = 300, 300


from keras.preprocessing.image import ImageDataGenerator
test_datagen = ImageDataGenerator(rescale=1./255)
validation_data_dir = 'samples_300/validation'
validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=32,
    class_mode='categorical')
nb_validation_samples=len(validation_generator.filenames)



out=model.predict_generator(validation_generator,nb_validation_samples)

print "out"
print out

the output always is same even I use different picture to test as follow

Using TensorFlow backend.
Found 60 images belonging to 2 classes.
out
[[ 0.06170857  0.06522226  0.06400252  0.08250671  0.07548683      0.07643672
0.07131153  0.07487586  0.07607967  0.04719007  0.07641899  0.08824327
0.05421595  0.08630092]
[ 0.06170857  0.06522226  0.06400252  0.08250671  0.07548683 0.07643672
0.07131153  0.07487586  0.07607967  0.04719007  0.07641899  0.08824327
0.05421595  0.08630092]
....]

Solution 1

Well, after 1 month looking for a solution, i tried everything: lowering the learning rate, changing the optimizer, using a bigger dataset, increasing and decreasing model complexity, changing the input shape to smaller and larger images, changin the imports from from keras import to from tensorflow.keras import and further to from tensorflow.python.keras import, changing the activation function of every layer, combining them, trying others datasets, etc... Nothing helped. Even if i used a net like VGG16/19 my results would be the same. But, yesterday, i was reading a book (Deep Learning with Keras - Antonio Gulli, Sujit Pal) and i realized that the autors use imports like this:

from keras.layers.core import Dense, Flatten, Dropout

and not like this:

from keras.layers import Dense, Flatten, Dropout

the same for Conv, i was using:

from keras.layers import Conv2D, MaxPooling2D, SeparableConv2D

and the autors use:

from keras.layers.convolutional import Conv2D, MaxPooling2D, SeparableConv2D

And when i changed the imports everything started to work finally! I don't know if this is a bug or something like this, cause now my models always works even in datasets that it was predicting the same class. So now i'm using the imports like this, for example:

from keras.layers.core import Dense, Dropout, Flatten
from keras.layers.convolutional import Conv2D, MaxPooling2D, SeparableConv2D

Try this, if doesn't work, look if your dataset is balanced, for example if you problem is to classify images of cats and dogs and you have 200 cat images and 800 dog images, try to use a number of images not so different cause it can cause problem: your model can 'think' well if i say that 10/10 images are dogs so i get 80% accuracy, but this is not what we want. You can use class_weight if you don't have more images to balance you dataset and everything will be ok, you can as well use data-augmentation. You can use callbacks too, like ReduceLROnPlateau which reduce your learning rate when your loss is not getting lower. You can increase your batch_size, don't forget to shuffle the data on your ImageDataGenerator and normalize your images, like this:

g2 = ImageDataGenerator(rescale=1./255)

All these things are very important, but nothing really helped me. The only thing that worked was importing keras.layers.core and keras.layers.convolutional, may this help you too!

Solution 2

It seems that your problem is caused by a huge class inbalance in your dataset. One can see that assigning a 0 class to each example gives you over 90% of accuracy. In order to deal with that you may use following strategies:

1. Rebalance your dataset: by either upsampling the less frequent class or downsampling the more frequent one.

2. Adjust class weights: by setting the higher class weight for less frequent class you'll be promoting your network training for putting more attention on the downsampled class.

3. Increase the time of training: in many cases - after longer period of training time network starts concentrating more on less frequent classes.

Comments

[dusa]

I have a perfectly balanced dataset, and still have the same problem. And it gets more awkward as I get 80 accuracy for validation (awkward considering it predicts only one class and the data is roughly %50 %50 but I get %80 accuracy somehow) Can't figure out why stackoverflow.com/questions/57960757/…

[Zain Ul Abidin]

I appriciate your struggle congratulations you just found a bug in the Keras module but it seems in later versions they have already solved it . .

[Locke]

@ZainUlAbidin I just ran into this issue with the latest version I installed earlier today. It appears to still be broken for from keras.layers import Dense, Activation.