Load Balancing and Clustering using mod_proxy_ajp on Apache HTTP Server 2.2.21 and Tomcat 7.0.23

apache-2.2 tomcat load-balancing cluster sticky-sessions

14,669

Thanks to @Shane Madden's comments, I re-read the tomcat cluster docs in http://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html, especially in this part -> "If your Tomcat instances are running on the same machine, make sure the tcpListenPort attribute is unique for each instance, in most cases Tomcat is smart enough to resolve this on it's own by autodetecting available ports in the range 4000-4100".

The changes I've made is in each of the Tomcat instance's server.xml, I make sure each port is different (for example 4001, 4002, 4003) :

<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"  
  address="auto" port="4001" autoBind="100" 
  selectorTimeout="5000" maxThreads="6" />

And voila! load-balancing and clustering is working (with the most basic config of course). I hope this post can help others on initial settings.

14,669

Yusuf S

Just an ordinary software developer, trying to make my way in this highly competitive world.

Updated on September 18, 2022

Comments

Yusuf S over 1 year

I've been struggling to make load balancing and clustering working using these combination:

Apache HTTP Server 2.2.21 (httpd-2.2.21-win32-x86-openssl-0.9.8r) using mod_proxy_ajp with sticky session enabled.
Apache Tomcat 7.0.23 (apache-tomcat-7.0.23-windows-x64)
JDK 7 update 2 (jdk-7u2-windows-x64)
Windows 7 64 Bit
Spring 3.1

Some links that I've read:

This is my configuration:

httpd.conf

# Required Modules
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule status_module modules/mod_status.so

# Reverse Proxy
<Proxy balancer://mybalancer>
    BalancerMember ajp://localhost:8301 route=s1
    BalancerMember ajp://localhost:8302 route=s2
    BalancerMember ajp://localhost:8303 route=s3
</Proxy>
ProxyPass / balancer://mybalancer/ stickysession=JSESSIONID|jsessionid

# Forward Proxy
ProxyRequests Off

<Proxy *>
    Order deny,allow
    Deny from none
    Allow from localhost
</Proxy>

# Balancer-manager, for monitoring
<Location /balancer-manager>
    SetHandler balancer-manager

    Order deny,allow
    Deny from none
    Allow from localhost
</Location>

server.xml for each tomcat (difference only in port number)

<Server port="8001" shutdown="SHUTDOWN">
    <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
    <Listener className="org.apache.catalina.core.JasperListener" />
    <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
    <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
    <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

    <GlobalNamingResources>
        <Resource name="UserDatabase" auth="Container"
            type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved"
            factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" />
    </GlobalNamingResources>

    <Service name="Catalina">
        <Connector port="8101" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8201" />

        <Connector port="8301" protocol="AJP/1.3" redirectPort="8201" />

        <Engine name="Catalina" defaultHost="localhost" jvmRoute="s1">

            <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
                <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" 
                    notifyListenersOnReplication="true" />
                <Channel className="org.apache.catalina.tribes.group.GroupChannel">
                    <Membership className="org.apache.catalina.tribes.membership.McastService"
                        address="228.0.0.4" port="45564" frequency="500" dropTime="3000" />
                    <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                        <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />
                    </Sender>
                    <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                        address="auto" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6" />
                    <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" />
                    <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor" />
                </Channel>
                <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="" />
                <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve" />
                <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener" />
                <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener" />
            </Cluster> 

            <Realm className="org.apache.catalina.realm.LockOutRealm">
                <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase" />
            </Realm>
            <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">
                <Valve className="org.apache.catalina.valves.AccessLogValve"
                    directory="logs" prefix="localhost_access_log." suffix=".txt"
                    pattern="%h %l %u %t &quot;%r&quot; %s %b" />
            </Host>
        </Engine>
    </Service>

</Server>

port numbering (make sure no instances use the same port if deployed in the same server)

80 -> Apache HTTP Server Port
80xx -> Tomcat Server SHUTDOWN Port
81xx -> Tomcat Connector Port (HTTP)
82xx -> Tomcat SSL Redirect Port
83xx -> Tomcat AJP Port
40xx -> Tomcat tcp receive port for NioReceiver

Spring Controller

@Controller
@RequestMapping("/login")
public class LoginController {
    @RequestMapping(method=RequestMethod.GET)
    public String show(@ModelAttribute("user") User user, HttpServletRequest request) {
        user.setUsername("YUSUF");

        HttpSession session = request.getSession();
        Integer tambah = (Integer) session.getAttribute("tambah");
        if(tambah == null) tambah = new Integer(1);
        else tambah = new Integer(tambah.intValue() + 1);
        session.setAttribute("tambah", tambah);

        return "login";
    }
}

login.jsp

    <div class="mainFooter">
        Tambah = ${sessionScope.tambah}
        <br>
        ID = ${pageContext.session.id}
    </div>

So far the load balancing part is working, but the session replication is not.
Basically what I want is if I keep pressing refresh on the login page, the variable "tambah" will be incremented and kept in the session. And if the current tomcat node is down, the session will be replicated to the next tomcat node and the data is not gone. But this is what happening:

Login Screen:

Tambah = 39 
ID = C1D59C8CA5D10EB98C1DE08AC618204D.s1

I take the tomcat1 down and keep tomcat2 and tomcat3 running, here is the Login Screen:

Tambah = 1
ID = A83KJFO38FK30FJDL40FLREI39FKDKGD.s2

It seems the failover is not working, the session is not replicated, and the application created a new session. Can someone help point me to the right direction?

Thanks

EDIT: I've solved the question, thanks to @Shane Madden, this is the changes I've made:

<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                        address="auto" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6" />

For each Tomcat, the tcp receiver port must be different (if running on the same server), for example use 4001, 4002, 4003, etc.

ravi yarlagadda over 12 years

All instances are on the same node? I've never tried this, but I don't think that the other instances of Tomcat on the same node will get the multicast communication that's intended to be between cluster nodes (which are typically different systems).
Yusuf S over 12 years

@Shane Madden I'm sorry I don't quite understand what you mean? Basically I have 1 httpd server as a load balancer, with 3 tomcat instances behind which I named s1, s2, and s3. I wanted these 3 tomcat's session to be sticky and replication enabled for failover. All of these tomcat instances is in one balancer called "mybalancer". Do I answer your question? Thanks
ravi yarlagadda over 12 years

What I'm saying is that since all three Tomcat instances are on a single server, the cluster communication (which is built to talk between multiple instances, each on different servers) will not function. The communication is multicast IP traffic, which will send out to the network just fine, but won't come back - it won't work to facilitate communication between multiple instances on the same server.
Yusuf S over 12 years

@Shane Madden So what you mean is if I applied these exact same settings but on different server.. for example the tomcat instances is not on localhost, but on 3 different server like this: BalancerMember ajp://server1:8301 route=s1, BalancerMember ajp://server2:8302 route=s2, BalancerMember ajp://server3:8303 route=s3. It will work?
Yusuf S over 12 years

I've solved it! Your comment makes me re-read the tomcat cluster docs, especially in this part -> "If your Tomcat instances are running on the same machine, make sure the tcpListenPort attribute is unique for each instance, in most cases Tomcat is smart enough to resolve this on it's own by autodetecting available ports in the range 4000-4100". I just changed each port to 4001, 4002, 4003 (because the instance is in the same server). And it worked! Thanks very much! I would accept your answer if you put it in the answer section :)
ravi yarlagadda over 12 years

Nice, good find! Go ahead and post that as your own answer, since you found that solution yourself (and I was saying that it wouldn't work - and I was wrong!). I believe it won't let you post your own answer until another 4 hours from now, but leave another comment when you do so that it notifies me and I'll vote for it :)