What could cause RMI method calls to fail intermittently?

Question

Long story short, I have an RMI Server and Client. The Server and Client are capable of making RMI calls on each other. After the Client connects to the Server, the Server may make many hundreds of method calls in quick succession on the Client.

The problem is this - towards the end of a huge batch of Server-to-Client method calls, some will fail because RMI claims it cannot establish a connection from Server-to-Client, even though hundreds of calls before it will succeed. I can't post any real code because this project is rather large (about 50k lines), but here's the full stack trace of the exception that gets thrown:

java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketException: Connection reset
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(Unknown Source)
at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source)
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Source)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source)
at $Proxy0.findClassDefinition(Unknown Source)
at com.fabric.network.NetworkClassLoader.findClass(NetworkClassLoader.java:111)
at java.lang.ClassLoader.loadClass(Unknown Source)
at com.fabric.network.NetworkClassLoader.loadClass(NetworkClassLoader.java:131)
at java.lang.ClassLoader.loadClass(Unknown Source)
at com.fabric.network.MessageSocket$CustomObjectInputStream.resolveClass(MessageSocket.java:171)
at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
at java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at com.fabric.network.MessageSocket.receive(MessageSocket.java:118)
at com.fabric.application.driver.NodeRemoteDriver$IncomingMessageThread.run(NodeRemoteDriver.java:205)
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketException: Connection reset
at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)
at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Source)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source)
at $Proxy2.findClassDefinition(Unknown Source)
at com.fabric.network.ClassDefinitionCache.findClassDefinition(ClassDefinitionCache.java:78)
at com.fabric.management.host.NodeManagementServices.findClassDefinition(NodeManagementServices.java:231)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.DataInputStream.readByte(Unknown Source)
... 21 more

Again, sorry I can't provide much in the way of code, but I'm not necessarily asking for a code fix - I just want to wrap my head around why this might be happening.

EDIT

Added the full stack trace.

Solution

Ok, so after nearly pulling all my hair out, it turns out that RMI was trying to open up way too many ports. I am using a custom RMISocketFactory implementation under the hood. This custom implementation is a singleton, so I didn't think it was necessary to implement hashCode() and equals(). A very painful mistake indeed...

Turns out, RMI won't reuse sockets if RMI determines that the socket it needs to create is to be created by an RMISocketFactory that isn't equivalent to the factory that created the socket it wants to reuse. RMI relies on equals() and hashCode() to perform this check. Once I correctly implemented these two methods in my custom socket factory, these intermittent problems went away.

The description of this issue can be found here:

http://docs.oracle.com/javase/7/docs/technotes/guides/rmi/faq.html

At any rate, thanks to all for taking a look at this, I sure appreciate your time!

Additional Information

A secondary issue that I didn't notice before was that the ServerSocket I was using was running out of queue space for incoming connection requests, which also contributed to the connections being dropped. Using the constructor new ServerSocket(port, newConnectionQueueSize, bindAddress) with a larger newConnectionQueueSize contributed to solving this problem as well.