Is Keras thread safe?

Question

I'm using Python and Keras (currently using Theano backend, but I have no qualms with switching). I have a neural network that I load and process multiple sources of information with in parallel. Currently, I've been running each one in a separate process and it loads its own copy of the network from the file. This seems like a waste of RAM, so I was thinking it would be more efficient to have a single multi-threaded process with one instance of the network that is used by all threads. However, I'm wondering if Keras is thread safe with either backend. If I run .predict(x) on two different inputs at the same time in different threads, will I run into race conditions or other issues?

Thanks

Solution 1

Yes, Keras is thread safe, if you pay a little attention to it.

In fact, in reinforcement learning there is an algorithm called Asynchronous Advantage Actor Critics (A3C) where each agent relies on the same neural network to tell them what they should do in a given state. In other words, each thread calls model.predict concurrently as in your problem. An example implementation with Keras of it is here.

You should, however, pay extra attention to this line if you looked into the code: model._make_predict_function() # have to initialize before threading

This is never mentioned in the Keras docs, but its necessary to make it work concurrently. In short, _make_predict_function is a function that compiles the predict function. In multi thread setting, you have to manually call this function to compile predict in advance, otherwise the predict function will not be compiled until you run it the first time, which will be problematic when many threading calling it at once. You can see a detailed explanation here.

I have not met any other issues with multi threading in Keras till now.

Solution 2

to quote the kind fcholet:

_make_predict_function is a private API. We should not recommend calling it.

Here, the user should simply call predict first.

Note that Keras models can't be guaranteed to be thread-safe. Consider having independent copies of the model in each thread for CPU inference.

Comments

[asaf92]

I think that the answer by Daniel is more accurate and up to date.