Merge 2 sequential models in Keras

Question

I a trying to merge 2 sequential models in keras. Here is the code:

model1 = Sequential(layers=[
    # input layers and convolutional layers
    Conv1D(128, kernel_size=12, strides=4, padding='valid', activation='relu', input_shape=input_shape),
    MaxPooling1D(pool_size=6),
    Conv1D(256, kernel_size=12, strides=4, padding='valid', activation='relu'),
    MaxPooling1D(pool_size=6),
    Dropout(.5),

])

model2 = Sequential(layers=[
    # input layers and convolutional layers
    Conv1D(128, kernel_size=20, strides=5, padding='valid', activation='relu', input_shape=input_shape),
    MaxPooling1D(pool_size=5),
    Conv1D(256, kernel_size=20, strides=5, padding='valid', activation='relu'),
    MaxPooling1D(pool_size=5),
    Dropout(.5),

])

model = merge([model1, model2], mode = 'sum')
Flatten(),
Dense(256, activation='relu'),
Dropout(.5),
Dense(128, activation='relu'),
Dropout(.35),
# output layer
Dense(5, activation='softmax')
return model

Here is the error log:

File "/nics/d/home/dsawant/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 392, in is_keras_tensor raise ValueError('Unexpectedly found an instance of type ' + str(type(x)) + '. ' ValueError: Unexpectedly found an instance of type <class 'keras.models.Sequential'>. Expected a symbolic tensor instance.

Some more log:

ValueError: Layer merge_1 was called with an input that isn't a symbolic tensor. Received type: class 'keras.models.Sequential'. Full input: [keras.models.Sequential object at 0x2b32d518a780, keras.models.Sequential object at 0x2b32d521ee80]. All inputs to the layer should be tensors.

How can I merge these 2 Sequential models that use different window sizes and apply functions like 'max', 'sum' etc to them?

Solution

Using the functional API brings you all possibilities.

When using the functional API, you need to keep track of inputs and outputs, instead of just defining layers.

You define a layer, then you call the layer with an input tensor to get the output tensor. Models and layers can be called exactly the same way.

For the merge layer, I prefer using other merge layers that are more intuitive, such as Add(), Multiply() and Concatenate() for instance.

from keras.layers import *

mergedOut = Add()([model1.output,model2.output])
    #Add() -> creates a merge layer that sums the inputs
    #The second parentheses "calls" the layer with the output tensors of the two models
    #it will demand that both model1 and model2 have the same output shape

This same idea apply to all the following layers. We keep updating the output tensor giving it to each layer and getting a new output (if we were interested in creating branches, we would use a different var for each output of interest to keep track of them):

mergedOut = Flatten()(mergedOut)    
mergedOut = Dense(256, activation='relu')(mergedOut)
mergedOut = Dropout(.5)(mergedOut)
mergedOut = Dense(128, activation='relu')(mergedOut)
mergedOut = Dropout(.35)(mergedOut)

# output layer
mergedOut = Dense(5, activation='softmax')(mergedOut)

Now that we created the "path", it's time to create the Model. Creating the model is just like telling at which input tensors it starts and where it ends:

from keras.models import Model

newModel = Model([model1.input,model2.input], mergedOut)
    #use lists if you want more than one input or output    

Notice that since this model has two inputs, you have to train it with two different X_training vars in a list:

newModel.fit([X_train_1, X_train_2], Y_train, ....)    

---

Now, suppose you wanted only one input, and both model1 and model2 would take the same input.

The functional API allows that quite easily by creating an input tensor and feeding it to the models (we call the models as if they were layers):

commonInput = Input(input_shape)

out1 = model1(commonInput)    
out2 = model2(commonInput)    

mergedOut = Add()([out1,out2])

In this case, the Model would consider this input:

oneInputModel = Model(commonInput,mergedOut)

Comments

[Digvijay Sawant]

So when we want to merge 2 models we cannot declare those 2 models as Sequential()? We have to use functional APIs. I keep getting the error that I stated above in my question when I call Concatenate(). Is that correct that in this case we cannot use Sequential()?

[Daniel Möller]

You can keep the sequential models, no problem, but the final model cannot be sequential, it's not feasible. The error message is about: "you are not passing tensors when calling the layer". You are very probably passing the models instead. Please notice the model1.output and model2.output tensors in my answer. ---- model1 is a model, while model1.output is a tensor.

[Daniel Möller]

The final model is a functional Model that contains two Sequential models and some additional layers in its path.

[Digvijay Sawant]

I am finding it difficult to understand the last part. I am training both my models on the same training data. Here is my fit() function - model.fit([train_data, train_data], train_labels, batch_size=256, epochs=5, validation_data=(test_data, test_labels), verbose=1, callbacks=callbacks) I still get an error saying : ValueError: The model expects 2input arrays, but only received one array. Found: array with shape (1807, 6000, 1)

[Daniel Möller]

It's probably because of the validation_data, it also needs two inputs as you did with train_data.