How to solve grpc Deadline Exceeded error？

Question

I have grpc server which write by go and a python client, sometimes, the errors occurred as below:

eggs/grpcio-1.0.0-py2.7-linux-x86_64.egg/grpc/_channel.py\", line 432, in _end_unary_response_blocking\n    raise _Rendezvous(state, None, None, deadline)\nInternalServerError: Deadline Exceeded\n"}

grpc Deadlines concept：

gRPC allows clients to specify a deadline value when calling a remote method. This specifies how long the client wants to wait for a response from the server before the RPC finishes with the error DEADLINE_EXCEEDED. On the server side, the server can query the deadline to see if a particular method has timed out, or how much time is left to complete the method.

How the deadline is specified varies from language to language - for example, a deadline value is always required in Python, and not all languages have a default deadline.

is there way to solve this error?

Solution

As mentioned in the comment above, deadlines can be just about anything between the client and server, including the network and implementation of the server. As you are talking over a network, deadlines should be expected occasionally, for example during package losses. I general, there are a couple of options on what you can do here:

• Optimize the server implementation to handle your requests faster.
• In case the client sets a short deadline, increase this.
• Handle the deadline errors gracefully by:
  • Retrying. Just make sure to use an exponential back-off, so you don't make the problem worse in case of a server overload situation.
  • Report an error up-streams.
  • Sometimes it may also be possible to fall back to not use that grpc call and degrade the experience.