Keras - Add attention mechanism to an LSTM model

Question

With the following code:

model = Sequential()

num_features = data.shape[2]
num_samples = data.shape[1]

model.add(
    LSTM(16, batch_input_shape=(None, num_samples, num_features), return_sequences=True, activation='tanh'))
model.add(PReLU())
model.add(Dropout(0.5))
model.add(LSTM(8, return_sequences=True, activation='tanh'))
model.add(Dropout(0.1))
model.add(PReLU())
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))

I'm trying to understand how can I add an attention mechanism before the first LSTM layer. I've found the following GitHub: keras-attention-mechanism by Philippe Rémy but couldn't figure out how exactly to use it with my code.

I would like to visualize the attention mechanism and see what are the features that the model focus on.

Any help would be appreciated, especially a code modification. Thanks :)

Solution

You may find an example of how to use a LSTM with an activation mechanism in Keras in this gist

https://gist.github.com/mbollmann/ccc735366221e4dba9f89d2aab86da1e

And in the following answer on SO:

How to add an attention mechanism in keras?

And to visualize your activations you can use the following repository https://github.com/philipperemy/keras-activations

Comments

[Shlomi Schwartz]

Thanks for your reply, how would you visualize the output as seen on the GitHub repository I've shared?

[Shlomi Schwartz]

thanks for your help, However, I'm looking for more points of view for my use-case :)