Printing out the validation accuracy to the console for every batch or epoch (Keras)

Question

I'm using ImageDataGenerator and flow_from_directory to generate my data, and using model.fit_generator to fit the data.

This defaults to outputting the accuracy for training data set only. There doesn't seem to be an option to output validation accuracy to the terminal.

Here is the relevant portion of my code:

#train data generator


print('Starting Preprocessing')

train_datagen = ImageDataGenerator(preprocessing_function = preprocess)

train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size = (img_height, img_width),
batch_size = batch_size, 
class_mode = 'categorical')  #class_mode = 'categorical'


#same for validation
val_datagen = ImageDataGenerator(preprocessing_function = preprocess)

validation_generator = val_datagen.flow_from_directory(
        validation_data_dir,
        target_size = (img_height, img_width),
        batch_size=batch_size,
        class_mode='categorical')





########################Model Creation###################################

#create the base pre-trained model
print('Finished Preprocessing, starting model creating \n')
base_model = InceptionV3(weights='imagenet', include_top=False)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(12, activation='softmax')(x)
model = Model(input=base_model.input, output=predictions)




for layer in model.layers[:-34]:
   layer.trainable = False
for layer in model.layers[-34:]:
   layer.trainable = True


from keras.optimizers import SGD
model.compile(optimizer=SGD(lr=0.001, momentum=0.92),
              loss='categorical_crossentropy',
              metrics = ['accuracy'])



#############SAVE Model #######################################


file_name = str(datetime.datetime.now()).split(' ')[0] + '_{epoch:02d}.hdf5'
filepath = os.path.join(save_dir, file_name)



checkpoints =ModelCheckpoint(filepath, monitor='val_acc', verbose=1,
                                save_best_only=False, save_weights_only=False,
                                mode='auto', period=2)

###############Fit Model #############################

model.fit_generator(
train_generator,
steps_per_epoch =total_samples//batch_size,
epochs = epochs,
validation_data=validation_generator,
validation_steps=total_validation//batch_size,
callbacks = [checkpoints],
shuffle= True)

UPDATE OUTPUT:

Throughout training, I'm only getting the output of training accuracy, but at the end of training, I"m getting both training, validation accuracy.

Epoch 1/10

  1/363 [..............................] - ETA: 1:05:58 - loss: 2.4976 - acc: 0.0640
  2/363 [..............................] - ETA: 51:33 - loss: 2.4927 - acc: 0.0760  
  3/363 [..............................] - ETA: 48:55 - loss: 2.5067 - acc: 0.0787
  4/363 [..............................] - ETA: 47:26 - loss: 2.5110 - acc: 0.0770
  5/363 [..............................] - ETA: 46:30 - loss: 2.5021 - acc: 0.0824
  6/363 [..............................] - ETA: 45:56 - loss: 2.5063 - acc: 0.0820

Solution 1

The idea is that you go through you validation set after each epoch, not after each batch. If after every batch, you had to evaluate the performances of the model on the whole validation set, you would loose a lot of time.

After each epoch, you will have the corresponding losses and accuracies both for training and validation. But during one epoch, you will only have access to the training loss and accuracy.

Solution 2

Validation loss and validation accuracy gets printed for every epoch once you specify the validation_split.

model.fit(X, Y, epochs=1000, batch_size=10, validation_split=0.2)

I have used the above in my code, and val_loss and val_acc are getting printed for every epoch, but not after every batch.

Hope that answers your question.

Epoch 1/500
1267/1267 [==============================] - 0s 376us/step - loss: 0.6428        - acc: 0.6409 - val_loss: 0.5963 - val_acc: 0.6656

Solution 3

In fit_generator,

fit_generator(generator, steps_per_epoch=None, epochs=1, verbose=1, callbacks=None, **validation_data=None, validation_steps=None**, validation_freq=1, class_weight=None, max_queue_size=10, workers=1, use_multiprocessing=False, shuffle=True, initial_epoch=0)

since there is no validation_split parameter, you can create two different ImageDataGenerator flow, one for training and one for validating and then place that 'validation_generator' in validation_data. Then it will print the validation loss and accuracy.

Comments

[Moondra]

So there isn't a way to do this? In tensorflow (using their transfer learning script), I was able to do this and with saved bottlenecks it wasn't bad.

[mpariente]

From my knowledge no, there is no easy way to do this. I wonder, what is the point of knowing the validation loss at every batch?

[payne]

validation_split doesn't exist as a parameter with fit_generator.