How to limit number of sessions?

Question

I need a way to track and limit web sessions to a web app. A "session" is loosely defined as the single user browsing the pages of the said web app. I think it can be translated to:

• a session is defined as a tuple <clientIP,vHost> alternatively as <clientIP,serverIP,serverPort> or <cookie,vHost>, depending on the layer and the data available
• a session starts after the user has sent authentication data to a defined login URI
• a session ends after the user has hit the defined logout URI
• a session ends if a specified timeout has expired after the client has requested the last object

After the specified session limit has been reached, the next user should be directed to a custom error page. I also need a way to track the current number of sessions for monitoring purposes and the ability to whitelist the monitoring server (which is issuing queries to the webapp periodically) and exempt it from the limit.

What I can work with:

• RadWare AppDirector where the web application has an own farm defined and is running in reverse proxy mode
• Apache 2.2
• SLES 11 SP2

I would prefer not involving an additional proxy server, although would consider it if no other options remain.

The rationale behind all of this is that the aforementioned web app is easily overloaded and starts denying requests erratically, pissing off working users who (usually) lose form entry data in the process. By specifying a limit where an overload condition is less likely, we hope to create a well-defined failure condition where users would be told to return later if the load is likely to spike.

Edit: the web app is a 3-tier implementation with the first tier (presentation layer, implemented as CGI code in an Apache vHost) being rather simplistic and apparently limited to basic error handling and request load balancing among the application servers. It does not impose any significant load on the web servers it runs on - this is why we are running it in mere failover mode (no load balancing) in the AppDirector farm, which is supposed to somewhat simplify things.

Everything beyond this point is basically a black box to us - at the data tier we have an MSSQL database, but it is near impossible to get any meaningful information about the table structure from the vendor. The application servers are closed-source, the vendor has used a rather comprehensive framework for the implementation, but seems unable to answer even less complex operation-related questions.

Solution 1

The problem you are ultimately trying to solve is with the capacity of the application - and that's where you should be solving the problem. None of the components you mention has anything to do with session management for an HTTP application.

There are some tricks you can apply with the recent module in iptables or using fail2ban in the opposite way to the purpose it was designed for - but these both require a very detailled understanding of the tools and the problem domain. You could implement access control at the level of these components but driven by published state information from the application on the number of sessions.

I also need a way to track the current number of sessions for monitoring purposes

Assuming, for the time being, that the application is a black box with no scope for modification / instrumentation (which is highly improbable) you can get this information from your apache logs by including the session cookie - filter or tail the logs to maintain a list of active cookies - and remove entries from the list when they coincide with the logout URL or have not been seen for the TTL.

Solution 2

This is not exactly what you are asking for, but I've already done the following with F5 load balancers :

• Count the number of post request on the login page.
• Delay users if logins per second is above a first limit
• Send users to a maintenance page if logins per second is above a second limit

As the website was sometimes under heavy load (horse races), it helped.

Solution 3

This problem is solveable both by using the RadWare AppDirector, and (for completeness) likely also by using Apache mod_security as per your excellent finding in the comment below.

For an AppDirector solution I believe it is possible to create two farms mapping to the same backend server(s). These farms can have different criteria and operating conditions applied to them. One farm would be the "default" and the other would answer to URI:s which you define as being "a session". The latter would get a limit to the amount of sessions it accepts in the load balancer.

I am from now on going to substitute your "session" term for "logged in" for two reasons:

• It avoids ambiguity as it clearly defines the desired state in that the user is authenticated.
• The AppDirector User Guide and GUI redefines the term "connection" to have a meaning for all practical purposes identical to "session", see below. This adds confusion which we try to avoid.

It is also possible to show a sorry page if the "logged in" farm has reached the chosen connection limit.

Before getting to the how, I must clearly state I have no operating experience of the AppDirector product, but do administer a competing and slightly less advanced load balancer on a daily basis. The product I use can do this scenario right off the bat. I have found information through the AppDirector User Guide and what online documentation is available which suggests that the same is true for the AppDirector. However whilst concepts are similar, the terminology is different. I am simply doing a when-in-rome act with regards to wording, hoping to get it fairly right without being too obviously a clueless moron.

The greatest roadblock was getting access to a manual, which is not made available unless one is an active customer. Through some googling it was possible to find an old version which I hope is not too out of date, I also found a couple of knowledgebase articles and this link: Radware AppDirector – Configuration: Basic Application.

Here is a solution draft, as interpreted mainly through the User Guide:

Client entry to the load balancer is done through a VIP which is used to connect both the "default" sessions and the "logged in sessions". This is achieved through a L4 policy as per p.99 in the User Guide:

"When AppDirector receives the first packet of a session destined to a
Virtual IP address, it searches for a Layer 4 Policy that matches the
Layer 4 Protocol, Destination port, Source IP, etc. Then, based on this
information, AppDirector selects the farm allocated to this service and
the best server for the task from that farm, and forwards the packet to
that server.

The L4 policy can be tied to L7 policies which are used to select a suitable farm. The L7 policy process is described thus in the User Guide p.104:

"The Layer 7 content aware decision making mechanism allows you to have
a single point of entry to the site, and provides differentiated service
for different user groups.

A Layer 7 decision is made using a mechanism called Delayed Binding.
When Delayed Binding is used, AppDirector first performs a TCP handshake
with the client to receive the HTTP request. AppDirector parses the HTTP
request’s data, usually HTTP headers, and performs the load balancing
decision. Only after that, does AppDirector select a farm and a server.
Lastly, AppDirector initiates a TCP handshake with the server and
forwards the traffic to it
[...]
When Layer 7 Policies are used, farm selection is based on matching the
request data with a list of Layer 7 Policies defining the Layer 7
parameters differentiating the service. The process of server selection
within the farm can also be content-based, using a third Layer 7
parameter."

The methods available to define an L7 behaviour are decribed on p.106, of which you could pick a suitable method to choose routing to your "logged in" Farm rather than to the "default" Farm:

"Methods are the basic building blocks for Layer 7 service selection.
They define content by which traffic is differentiated. You can use
the same Method to select one or more services. The following Method
Types are available:

- URL: Looks for a specified host name and/or path in the HTTP request.
- File Type: Looks for a specified File Type in the HTTP request.
- Header Field: Looks for a specified Header Field in the HTTP request.
- Cookie: Looks for a specified Cookie in the HTTP request.
- Regular Expression: Looks for a regular expression anywhere in the
HTTP request. AppDirector supports Posix 1002.3 regular expressions;
the string can be up to 80 characters.
- Text: Looks for a text string anywhere in the HTTP request."

As seen in the Basic Application link, one could for instance create an L7 policy evaluating URI patterns for routing to different farms. The made up URI patterns '^/login?=true' and '^/loggedin' could be routed to your "logged in" farm. The made up pattern '^/logout' (and all other URI:s) could similarly be routed to a "default" farm.

A Farm is defined by the User Guide p.121 thus: "An AppDirector farm is a group of networked servers that provide the same service [...] A server that provides multiple services can be used in multiple farms."

A server is further differentiated through separating the definition of a backend server into two layers, the 'Physical Server' object layer which represents the ip address of a server and the 'Farm Server' object layer which represent services running on one or more Physical Servers.

Session limiting on a farm can according to the 'AppDirector User Guide' be done per each Farm Server object defined for a farm (as well as through other means) in addition to per Physical Server object. This is described amongst other places on p.137:

"The Connection Limit is the maximum number of users that can be directed
to a server for a service provided by the farm. The number of users allowed
depends on the Sessions mode selected because it determines the number of
active entries in the Client Table for sessions destined to the specific server.

When the Entry Per Session or Server Per Session modes are selected, the number
of active entries destined to the same server is higher than in the Regular
mode (see Regular, page 153).

When the Regular mode is selected, all requests from a single client IP destined
to the same server are reflected by a single entry in the Client Table (see
Client Table Views, page 164).

The default value for the Connection Limit parameter is 0. When it is configured
to 0, it is disabled for this server and there is no user number limit."

The Client Table and its 'Regular mode' is defined on p.153:

"The Layer 3 Client Table is always used when Entry Per Session is used.
AppDirector uses the Layer 3 Client Table to ensure Layer 3 persistency.

This table contains information about the server selected for each client
(Source IP address) in each farm, and it allows AppDirector to select a
server for a new session.
[...]
In the Regular mode, AppDirector maintains Layer 3 persistency. In this mode,
each entry is identified by the following parameters:
• Layer 4 Policy VIP Address
• Client IP Address
• Destination TCP/UDP Port Used from the Client to the Server"

In a screenshot of a server definition window on the Basic Application page, the server connection limit box is seen right beside the bandwith limit box.

So a bit depending on configuration but for the purposes of this answer, a 'connection' as defined through the Client Table and a 'session' as defined by you essentially ends up being the same thing. And a limit to that effect can be imposed per server object in a farm.

As the AppDirector differentiates between physical servers and farm servers, it would be possible to define two farm servers mapping to your Apache physical server object, one having a low connection limit.

However, Apache also needs to answer calls from both farm server objects, for instance through being called on two separate ports or ip addresses - one being used by each (farm/farm server) combo. The question then becomes, are you able to define two application server entry points? i.e. are you able to equip your Apache front end application(/vhost?) to answer on two ports or IP addresses (one per farm)? This is through a bit of guess work as I do not wish to spend too much time with the manual, but I'm sure you could solve this fairly elegantly when actually looking at the AppDirector GUI and the Apache.

Setting the connection limit has a little quirk. From Physical Servers, Connection Limit p.140:

"Connection Limit

Maximum number of Client Table entries that can run simultaneously on 
the physical server. This depends on the farm’s Sessions mode (see 
Sessions Modes, page 150). When the limit is reached, new requests are 
no longer directed to this server. All open sessions are continued.

When the Connection Limit parameter is configured to 0 (default), this 
mechanism is disabled for this physical server and there is no user 
number limit.

Note: When configuring the physical server, ensure that the Connection 
Limit in the farm servers with the same Server Name is lower than or 
equal to the Connection Limit in the physical server. Total number of 
active sessions that run simultaneously on the farm servers must not 
be higher than the Connection Limit value defined on the physical server."

You would therefore need to define a very high Connection Limit (with a wide margin to the max number possible through your user base) for the unrestricted, "default" farm server, and set the Connection Limit for the "logged in" farm server as low as you have to. The physical server definition would need to have the sum of the two as its Connection Limit, as a precondition to activating the desired session limit.

---

You also have this requirement in your question:

After the specified session limit has been reached, the next user should be
directed to a custom error page.

This is termed a 'No HTTP Service Page' in the User Guide, p.134:

When all servers belonging to a farm cannot be used for a specific
session, AppDirector can reply to a Web request (destined to port 80)
with a simple Web page, indicating that the service is currently not
available. Servers that cannot be used for a session include servers
in Not In Service or in No New Sessions mode. No HTTP Service Page is
configured for each farm. Each Web page is limited to 1K of HTML code.

---

For the monitoring part I have not done as thorough research but here is what I think:

track the current number of sessions for monitoring purposes

AppDirector seems to have MIBs. Probably a pain to find the right OID as it usually is, but you can probably snmp it to your tool of choice.

whitelist the monitoring server (which is issuing queries to the webapp
periodically) and exempt it from the limit.

This one could require some creative thinking. Assuming the AppDirector doesn't include a template for this right out of the box, how about:

• URIs outside of the "logged in" farm would not be affected by the session limit. So monitor away, it's the same backend server(s) anyway.
• Use the AppDirector health checks instead, they will likely not count toward the session limit you impose. Find a way to pass alerts to your monitoring server though :-)
• Set up a third farm, through which you pass health checks. Messy, but it would work.

Comments

[lsmooth]

Can you provide general details on the web app? Is it the same on each vHost? Or didn't you provide the details, because you maybe want a solution that is independent of the web app? Is it a proprietary web app?

[manjiki]

After the connection is closed, due to inactivity, can someone resume a session using the session cookie (if the application's timeout hasn't been reached)?

[the-wabbit]

@jijix this is a border case which is considered rare enough not to bother about it if it adds to complexity of implementation. Other than that, I think it could best be specified as "re-instate the session without checking against the session limits".

[Nils]

I usually do this by counting "typical" urls in the httpd-access-log. What is the rough concurrent number we are talking about here? Perhaps a thread-limit may help here on the httpd-side?

[hookenz]

I know HAProxy can limit number of connections. But you're asking something a little different...

[lsmooth]

Forget my questions above, this is exactly what I was getting at.

[Veniamin]

A you sure that RadWare AppDirector can not solve this problem at least in extended treatment? - session control is requested to prevent node overloading that can be achieved by other means.

[symcbean]

No I'm not sure - as I said above you can fudge session management elsewhere in the stack - but its much, MUCH more difficult to do it in the wrong place. Fixing the problem where the problem is occurring is much easier.

[Veniamin]

If you consider the right place for session management is a module integrated in the application or elsewhere on per-node basis that it is not clear how to make it working together with load balancing at AppDirector level.

[the-wabbit]

The idea of using the apache logs for session tracking has some merit. As for the rest - you know I have not written the app and have little (if any) influence on further development. It was not my decision to put it in place, my authority is limited to the infrastructure it runs in. The point that the application is suboptimal has been made clear to the management, but even if this were able to change anything, any change would take a significant amount of time. So my current job is to find an "as good as it gets" mode of operation.

[the-wabbit]

Thanks for the idea, I need to check with our AppDirector guys if we could implement something similar.

[the-wabbit]

@Veniamin a per-node approach even would work for us due to the implementation specifics (we do not use load balancing, just failover). I do not have enough expertise in Radware's appliance configuration to tell what they are capable of - it certainly would help to have anybody telling me "you might try doing this with feature foo" or "you can't, don't even try".

[Veniamin]

@syneticon-dj unfortunately I had not experience with Radware products and can not say something authoritative. My role here is student a rather then a teacher :). But some times ago we considered it for deployment, thus I ran through feature set and found it rich enough to hope it may fit your problem.

[symcbean]

@syneticon-dj: if you need to check if you could implement something like this (which doesn't actually solve the problem, BTW) and you can't fix the application, then really you are in a heap of trouble. On reflection I can think of at least 2 further ways to solve the problem - but they are both technically demanding - and implemented incorrectly will make the problem worse rather than better. You need more help than you'll get here.

[the-wabbit]

So far, I have found that the mod_security2 module is able of handling session in a way which looks very similar to what I've specified - secure.jwall.org/blog/2009/01/08/1231374852674.html. I am going to do some more research on your idea of 2 farms, if such an implementation were possible, it certainly would simplify things.

[the-wabbit]

Response from the Radware technical support: "In AD we can only limit bandwidth per farm. [...] [A] method to limit total number of sessions on a farm basis is currently not available." So it is mod_security then.

[ErikE]

Ok, that was unexpected. Unless I missed something I still think closing the farm through a failing health check would be a cleaner solution though. Your finding on mod_security is very interesting regardless.

[ErikE]

Possibly though the support was a bit narrow in interpreting the question, it appears a session limit on the server level in AppDirector is possible (to which there certainly is some logic). I googled several links, here is one: kb.radware.com/questions/2829/…

[the-wabbit]

The Radware KB article is concerning LinkProof - a WAN accelerator. I have no idea about what software it is running and if similar facilities would exist for the AppDirector. BTW: Session handling code surely is part of AppDirector's feature set - it has a session table which looks exactly what I would expect it to look. But apparently, there is no way to impose a limit on the number of sesssions - just on connections. The most I could get from it is limiting the number of hits on a given page (e.g. the login page) per time unit as "the other" Eric suggested.

[ErikE]

Oops and blush. I did not notice it was for LinkProof. My mistake entirely.

[ErikE]

I have found out how to set session limits, as specified in an AppDirector User Guide. It can impose session limits on the farm server level, but using the term connection limit (a bit of practical term redefinion going on there, but it is very explicit and clear). See the marked edit in my answer.