Where should I start investigating SocketTimeoutException: Read timed out

Question

Every now and then I see following stacktrace in the log in which, HttpClient socket times out trying to access text/script content from another server. My question is what config settings should I check for my J2EE app running on Weblogic, on Linux? I am specifically looking for the following.

• JVM timeout params
• HttpClient params
• Weblogic timeout params or any other config like number of threads etc.
• J2EE Application settings like servlet config etc.
• Operating system resources like threads, file handlers and cpu
• Any other config settings that might influence the socket connection
• Would thread dumps help?

Here's my code

HTTPResponse httpClientResponse;
//do some stuff
httpClientResponse.getStatusCode(); // this is where it fails

and this is the stacktrace

java.net.SocketTimeoutException: Read timed out
at jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
at jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
at java.net.SocketInputStream.socketRead0(SocketInputStream.java)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at HTTPClient.BufferedInputStream.fillBuff(BufferedInputStream.java:206)
at HTTPClient.BufferedInputStream.read(BufferedInputStream.java:126)
at HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:356)
at HTTPClient.RespInputStream.read(RespInputStream.java:147)
at HTTPClient.RespInputStream.read(RespInputStream.java:108)
at HTTPClient.Response.readResponseHeaders(Response.java:1123)
at HTTPClient.Response.getHeaders(Response.java:846)
at HTTPClient.Response.getStatusCode(Response.java:331)
at HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:92)
at HTTPClient.HTTPResponse.handleResponseImpl(HTTPResponse.java:872)
at HTTPClient.HTTPResponse.access$000(HTTPResponse.java:62)
at HTTPClient.HTTPResponse$2.run(HTTPResponse.java:839)
at HTTPClient.HTTPResponse$2.run(HTTPResponse.java:837)
at
HTTPClient.HttpClientConfiguration.doAction(HttpClientConfiguration.java:666)
at HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:837)
at HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:242) 

Thanks

I will be updating my question with the FINDINGS below.

1. There is no explicit timeout set on HttpClient which means that http session timeout of the server might be taking an effect.
2. SO_TIMEOUT for HttpClient is 0 which means that it should wait indefinitely.

Solution 1

Track 1

As per the javadocs Httpclient does not seem to have have a default value of the Socket timeout. To answer the question in your update - the session timeout will not be in effect here. Weblogic's default is 30 minutes for session timeout.

The server session timeout represents the amount of time the HttpSession will be retained in memory if the user has not accessed the server.

The socket timeout is the amount of time to keep the server socket open while data is being transferred back to the caller. This could even be the server is still processing and writing back data but it's taking rather long and the client has just timed out waiting for it.

Some links suggest this default is 60 seconds but the javadocs dont say anything, in any case you can set this value to something like 120 seconds to see if it helps

http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/params/HttpConnectionParams.html#setSoTimeout(int)

What you need is to time the timeouts - if that's clear. Meaning - Do these errors appear after 30 sec, 60 sec or 5 minutes of the outgoing request?

I would change the SO_Timeout and try again

Track 2 - OS parameters

There are the recommended BEA parameters for NDD values which govern how long incoming connections are kept open and how many are queued and so on. On Solaris these are got by running

/usr/sbin/ndd -get /dev/tcp tcp_time_wait_interval 
/usr/sbin/ndd -get /dev/tcp tcp_conn_req_max_q 
/usr/sbin/ndd -get /dev/tcp tcp_conn_req_max_q0 
/usr/sbin/ndd -get /dev/tcp tcp_ip_abort_interval 
/usr/sbin/ndd -get /dev/tcp tcp_keepalive_interval 

Can you check the Oracle docs for the equivalent commands on Linux, and what values they should be set at. On Solaris my experience is the defaults are not enough and they need to be upped to BEA (Oracle) recommendations

Track 3: Weblogic / External Access Logs

Have you enabled HTTP Access Logs on the server? Do these failed requests show up with any response byte size or do they show 0 response size? What error code or HTTP status code is returned?

Or perhaps these timed out ones are not recorded in the access logs at all?

Here, I'm assuming the external server on which time outs occur is also Weblogic, if not - this question is directed to the external server team for their equivalent platform.

** Others **

Usually thread dumps help, but the thread dumps should be taken on the server which is having a timeout problem. You are the client and you have successfully obtained a connection, after that it times out when reading the response. So is the external server overloaded ? Lack of threads? CPU high? Too many concurrent requests?

Solution 2

One more aspect that has not been covered here is Firewall.

I have found that SocketTimeoutExceptions may often be related to a port not being open for communication or a firewall blocking communication from selected machines only.

In case you are debugging an issue make sure you also investigate if there is a firewall between the two machines trying to communicate and if there is one make sure the ports are available for communication between the two.

Interesting things about firewall related issues is that it does not let you know if the server is down or not responding. Typical behavior is to let the client wait forever. So you are always left in dark. A simple telnet on the server port should show if its available/open for communication.

Hope this helps.