SocketTimeoutException: Read timed out httpclient

Question

I'm having a REST service which is serving almost ~20 M request per day. I'm getting the following exception. This issue is intermittent and I couldn't relate this issue with volume of the request. I got this exception during weekend as well when volume is very low. I do ensure that there is no stele connection by checking setStaleConnectionCheckEnabled. I'm using httpclient 4.3.4

Update: Some more details after further analysis -

From the log, I can see that service responded most of the request within 300 ms but it took time for client to connect to service itself and by the time response was received, it had crossed the threshold which is 6 sec and ran into read time out exception.

getCause(): java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:220)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
at com.sample.SampleClient.doGet(SampleClient.java:137)

Client :

public MyRespBean doGet(String host, String path, List<NameValuePair> nameValuePairs, Map<String, String> headers, AuthParams authParams)
throws URISyntaxException, IOException
{
  RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(6000).setConnectTimeout(6000).setStaleConnectionCheckEnabled(true).build();
  CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).build();
  URI uri = buildUri(host, path, nameValuePairs);
  HttpGet httpGet = new HttpGet(uri);
  httpGet.setHeader("Connection", "close");
  addHeaders(httpGet, headers, authParams);
 **//Error occurs at the following line**
 MyRespBean respBean = (MyRespBean)httpClient.execute(httpGet, new SimpleResponseHandler());
  httpClient.close();
  return respBean;
}

Solution

It's it right that you are the owner of the service?

If so, I would check why the service had a pause between packets more than 6 seconds while replying: judging by the exception, the connection was successful.

Possible reasons: periodically slow connection or a very long full GC on the service side. Or even a long periodic warm up phase (I don't know the details of the service implementation, but probably for some specific request the service had to spend more than 6 seconds to prepare response) - in other words, anything that prevents your service from answering during 6 seconds.

Comments

[Pankaj]

Yes, I own both service and client. From log, I can see that service responded within 300 ms but it took time to connect to service itself and by the time response was received, it had crossed the threshold which is 6 sec and ran into read time out exception.