Merging layers on Keras (dot product)

Question

I've been following Towards Data Science's tutorial about word2vec and skip-gram models, but I stumbled upon a problem that I cannot solve, despite searching about it for hours and trying a lot of unsuccessful solutions.

https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa

The step that it shows you how to build the skip-gram model architecture seems deprecated because of the use of the Merge layer from keras.layers.

I've seem many discussions about it, and the majority of answers was the you need to use the Functional API of Keras to merge layers now. But the problem is, I'm a total beginner in Keras and have no idea how to translate my code from Sequential to Functional, here's the code that the author used (and I copied):

from keras.layers import Merge
from keras.layers.core import Dense, Reshape
from keras.layers.embeddings import Embedding
from keras.models import Sequential

# build skip-gram architecture
word_model = Sequential()
word_model.add(Embedding(vocab_size, embed_size,
                         embeddings_initializer="glorot_uniform",
                         input_length=1))
word_model.add(Reshape((embed_size, )))

context_model = Sequential()
context_model.add(Embedding(vocab_size, embed_size,
                  embeddings_initializer="glorot_uniform",
                  input_length=1))
context_model.add(Reshape((embed_size,)))

model = Sequential()
model.add(Merge([word_model, context_model], mode="dot"))
model.add(Dense(1, kernel_initializer="glorot_uniform", activation="sigmoid"))
model.compile(loss="mean_squared_error", optimizer="rmsprop")

# view model summary
print(model.summary())

# visualize model structure
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

SVG(model_to_dot(model, show_shapes=True, show_layer_names=False, 
rankdir='TB').create(prog='dot', format='svg'))

And when I run the block, the following error is shown:

ImportError                               Traceback (most recent call last)
<ipython-input-79-80d604373468> in <module>()
----> 1 from keras.layers import Merge
      2 from keras.layers.core import Dense, Reshape
      3 from keras.layers.embeddings import Embedding
      4 from keras.models import Sequential
      5 

ImportError: cannot import name 'Merge'

What I'm asking here is some guidance on how to transform this Sequential into a Functional API structure.

Solution 1

This did indeed change. For a dot product, you can now use the dot layer:

from keras.layers import dot
...
dot_product = dot([target, context], axes=1, normalize=False)
...

You have to set the axis parameter according to your data, of course. If you set normalize=True, this gives the cosine proximity. For more information, see the documentation.

To learn about the functional API to Keras, there is a good guide to the functional API in the documentation. It's not difficult to switch if you already understand the sequential API.

Solution 2

Merge seems deprecated so Instead of Merge use Dot directly on embedding (and not with models). Use below code.

from keras.layers import Input
from keras.models import Model
from keras.layers.embeddings import Embedding
from keras.layers.core import Dense, Reshape
from keras.layers import dot

input_target = Input((1,))
input_context = Input((1,))

embedding = Embedding(vocab_size, embed_size, input_length=1, name='embedding')

word_embedding = embedding(input_target)
word_embedding = Reshape((embed_size, 1))(word_embedding)
context_embedding = embedding(input_context)
context_embedding = Reshape((embed_size, 1))(context_embedding)

# now perform the dot product operation  
dot_product = dot([word_embedding, context_embedding], axes=1)
dot_product = Reshape((1,))(dot_product)

# add the sigmoid output layer
output = Dense(1, activation='sigmoid')(dot_product)

model = Model(input=[input_target, input_context], output=output)
model.compile(loss='mean_squared_error', optimizer='rmsprop')

# view model summary
print(model.summary())

# visualize model structure
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

SVG(model_to_dot(model, show_shapes=True, show_layer_names=False, 
                 rankdir='TB').create(prog='dot', format='svg'))

Comments

[IonicSolutions]

@Lucas If the answer was helpful, please consider upvoting/accepting it. Thanks!

[Lucas Figueiredo]

Sorry, it sure was helpful! But now I need to get to study a little more about sequential and functional models, and your answer gave me a kickstart, thanks!

[Tarun]

@IonicSolutions Hey I'm getting this error after using the dot Layer dot_3 was called with an input that isn't a symbolic tensor. Received type: <class 'keras.engine.sequential.Sequential'>. Full input: [<keras.engine.sequential.Sequential object at 0x0000016B8DC8FD68>, <keras.engine.sequential.Sequential object at 0x0000016B8DC8F668>]. All inputs to the layer should be tensors.

[IonicSolutions]

Please open a new question and show your code. I'm happy to take a look!

[Tarun]

Code is same as mention by @LucasFigueiredo. I have made changes as model.add(dot([word_model, context_model], axes=1, normalize=False))

[IonicSolutions]

Still, please open a new question, this is obviously a different problem, perhaps concerning a different version. It will be much quicker and easier if you post all information.

[Sam Dean]

Was there ever a solution to your problem @Tarun? I am getting the same thing for the exact same model.

[IonicSolutions]

Can you please open a new question with code for a minimal reproducible example and the exact Keras version you are using?