K.gradients(loss, input_img)[0] return "None". (Keras CNN visualization with tensorflow backend)

Question

I have CNN models trained using Keras with Tensorflow backend. And I want to visualize my CNN filters with this tutorial: https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html

from keras import backend as K
from keras.models import load_model
import numpy as np

model = load_model('my_cnn_model.h5')
input_img = np.load('my_picture.npy')

# get the symbolic outputs of each "key" layer (we gave them unique names).
layer_dict = dict([(layer.name, layer) for layer in model.layers])

layer_name = 'block5_conv3'
filter_index = 0  # can be any integer from 0 to 511, as there are 512 filters in that layer

# build a loss function that maximizes the activation
# of the nth filter of the layer considered
layer_output = layer_dict[layer_name].output
loss = K.mean(layer_output[:, :, :, filter_index])

# compute the gradient of the input picture wrt this loss
grads = K.gradients(loss, input_img)[0]

# normalization trick: we normalize the gradient
grads /= (K.sqrt(K.mean(K.square(grads))) + 1e-5)

# this function returns the loss and grads given the input picture
iterate = K.function([input_img], [loss, grads])

However, when the code execute to this line:
grads = K.gradients(loss, input_img)[0]
I found it returns nothing but None object, so the program fail to progress after that.

I search for some solution. Some people say theinput_img should be tensorflow's Tensor type: https://github.com/keras-team/keras/issues/5455

But when I tried to convert the img to Tensor, the problem is still exist.
I tried the solution in the link above, but still fail.

There is also someone say that this problem exists because your CNN model is not differentiable. https://github.com/keras-team/keras/issues/8478

But my model use only the activate function of ReLU and Sigmoid(at output layer). Is this problem really caused by nondifferentiable problem?

Can anyone help me? Thank you very much!

Solution 1

If you have a Model instance, then to take the gradient of the loss with respect to the input, you should do:

grads = K.gradients(loss, model.input)[0]

model.input contains the symbolic tensor that represents the input to the model. Using a plain numpy array makes no sense because TensorFlow then has no idea how this connects to the computational graph, and returns None as the gradient.

Then you should also rewrite the iterate function as:

iterate = K.function([model.input], [loss, grads])

Solution 2

Below, it's my example. Hope to help someone.

gradient = keras.backend.gradients(model.output, model.input)[2]

iterate = keras.backend.function(model.input, [gradient])

grad = iterate([patches, depthes, poses])

[patches, depthes, poses] is my model.input

Solution 3

I too had faced the same error @Jexus. For me the problem was:

loss variable was None object. I had used

loss.assign_add(....)

instead of

loss = loss + .....

After changing that as mentioned, the loss was returning a tensor and hence

grads = K.gradients(loss, model.input)[0]

wasn't returning None.

Comments

[Jexus]

Thank you so much! I know where the problem is.

[Ryan Chase]

@Matias Valdenegro I'm facing a similar issue where I'm getting a NoneType, but in my case my model.input is the input to my fine tuned network, so I believe it's connected to the graph. Do you have any suggestions?: stackoverflow.com/questions/50310063/…

[Ryan Chase]

@Jexus - were you able to get your code to work? How did you alter your code to get it to work?

[Daniel Möller]

It would be very nice to complement this answer with how to use the function. We need sess = K.get_session() and results = sess.run(iterate, feed_dict={model.input: numpyBatchWithData}).

[Kareem Jeiroudi]

I think @DanielMöller shows how to use this function. I was calculating the gradient of the output of the network with respect to the input, and couldn't understand how to get this function to work k.gradients(_, _). Thnx @DanielMöller!

[Dr. Snoopy]

Actually you don't need a session, iterate is a python callable, so you can just call it as a function