How to solve "No Algorithm Worked" Keras Error?

Question

I tried to develop an FCN-16 model in Keras. I initialized the weights with similar FCN-16 model weights.

def FCN8 (nClasses, input_height=256, input_width=256):

    ## input_height and width must be devisible by 32 because maxpooling with filter size = (2,2) is operated 5 times,
    ## which makes the input_height and width 2^5 = 32 times smaller
    assert input_height % 32 == 0
    assert input_width % 32 == 0
    IMAGE_ORDERING = "channels_last"

    img_input = Input(shape=(input_height, input_width, 3))  ## Assume 224,224,3

    ## Block 1
    x = Conv2D(64, (3, 3), activation='relu', padding='same', name='conv1_1', data_format=IMAGE_ORDERING)(
        img_input)
    x = Conv2D(64, (3, 3), activation='relu', padding='same', name='conv1_2', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool', data_format=IMAGE_ORDERING)(x)
    f1 = x

    # Block 2
    x = Conv2D(128, (3, 3), activation='relu', padding='same', name='conv2_1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(128, (3, 3), activation='relu', padding='same', name='conv2_2', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool', data_format=IMAGE_ORDERING)(x)
    f2 = x

    # Block 3
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_3', data_format=IMAGE_ORDERING)(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block3_pool', data_format=IMAGE_ORDERING)(x)
    pool3 = x

    # Block 4
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_1', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_3', data_format=IMAGE_ORDERING)(x)
    pool4 = MaxPooling2D((2, 2), strides=(2, 2), name='block4_pool', data_format=IMAGE_ORDERING)(
        x)  ## (None, 14, 14, 512)

    # Block 5
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_1', data_format=IMAGE_ORDERING)(pool4)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_2', data_format=IMAGE_ORDERING)(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_3', data_format=IMAGE_ORDERING)(x)
    pool5 = MaxPooling2D((2, 2), strides=(2, 2), name='block5_pool', data_format=IMAGE_ORDERING)(
        x) 

    n = 4096
    o = (Conv2D(n, (7, 7), activation='relu', padding='same', name="fc6", data_format=IMAGE_ORDERING))(pool5)
    conv7 = (Conv2D(n, (1, 1), activation='relu', padding='same', name="fc7", data_format=IMAGE_ORDERING))(o)

    conv7 = (Conv2D(nClasses, (1, 1), activation='relu', padding='same', name="conv7_1", data_format=IMAGE_ORDERING))(conv7)

    conv7_4 = Conv2DTranspose(nClasses, kernel_size=(2, 2), strides=(2, 2),  data_format=IMAGE_ORDERING)(
        conv7)

    pool411 = (
        Conv2D(nClasses, (1, 1), activation='relu', padding='same', name="pool4_11",use_bias=False, data_format=IMAGE_ORDERING))(pool4)

    o = Add(name="add")([pool411, conv7_4])

    o = Conv2DTranspose(nClasses, kernel_size=(16, 16), strides=(16, 16), use_bias=False, data_format=IMAGE_ORDERING)(o)
    o = (Activation('softmax'))(o)

    GDI= Model(img_input, o)
    GDI.load_weights(Model_Weights_path)

    model = Model(img_input, o)

    return model

Then I did train, test split and trying to run the model as:

from keras import optimizers

sgd = optimizers.SGD(lr=1E-2, momentum=0.91,decay=5**(-4), nesterov=True)

model.compile(optimizer='sgd',loss='categorical_crossentropy',metrics=['accuracy'],)

hist1 = model.fit(X_train,y_train,validation_data=(X_test,y_test),batch_size=32,epochs=1000,verbose=2)

model.save("/content/drive/My Drive/HCI_prep/new.h5")

But this code is throwing error in the first epoch:

NotFoundError: 2 root error(s) found. (0) Not found: No algorithm worked! [[{{node pool4_11_3/Conv2D}}]] [[loss_4/mul/_629]] (1) Not found: No algorithm worked! [[{{node pool4_11_3/Conv2D}}]] 0 successful operations. 0 derived errors ignored.

Solution 1

add the following to your code:

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

And then restart the python kernel.

Solution 2

Had the same issue.

The padding='same' for MaxPooling didn't work for me.

I changed the color_mode parameter in the train and test generators from 'rgb' to 'grayscale' and then it worked for me.

Solution 3

This worked for me:

    import tensorflow as tf
    physical_devices = tf.config.list_physical_devices('GPU')
    tf.config.experimental.set_memory_growth(physical_devices[0], True)

Solution 4

My problem was that I called the model with an input_shape of (?,28,28,1) and later called it with (?,28,28,3).

Solution 5

In my case, this was solved by ending all processes, that still allocated memory on one of the GPUs. Apparently, one of them did not finish (correctly). I did not have to change any code.

View more solutions