Good NFS multipath solution

Question

I have a couple of servers serving exactly the same content through NFS to several webfarms, all the servers involved are running CentOS 5

The content between both NFS servers is replicated using Unison (tried drbd in the past, killed the performance badly)

To distribute load I'm mounting a different NFS server from each of the farms.

• Webfarm1 mounts NFS1
• Webfarm2 mounts NFS2

The only problem I have is when one of the servers crashes I would like to automatically remount the content from the other NFS server until the server recovers and then returns to the first server after a grace time.

Before I start writing such application I would like to know if there's any Opensource solution already available that does something close to what I'm intending, all the intelligence of the application should be on the client side (server side solutions have all been tested and discarded (heartbeat, drbd, etc...) since I want to use both NFS servers in an active-active solution.

Also it should be important that all servers in a webfarm are hitting the same NFS server, otherwise I would incur in some nasty replication issues.

Thanks beforehand for your help!

Solution 1

Have a look at the RedHat Cluster Suite NFS Cookbook, subtitled "Setting up a Load­Balanced NFS Cluster with Failover Capabilities". I think this may have some good suggestions, if it's not exactly what you're looking for.

You can take the ideas presented in that document and implement them using Pacemaker/Corosync, which I find much easier to work with than the legacy RedHat cluster tools.

You're basically asking for an active/active cluster with automatic failover. Since you're serving out from two independent -- but synchronized -- copies of the data, you've actually sidestepped most of the complicating issues that crop up (involving locking and other state synchronization) if you're actually trying to create a load-balancing NFS front-end to some sort of shared storage (e.g., a cluster filesystem like GFS).

Solution 2

You could use CTDB to do that. It supports NFS with correct file locking behaviour and lock recovery. Also, it is an active-active solution i.e. all NFS servers are active at the same time and the clients access them via service IP addresses (DNS round-robin) which can move from server to server in case of a node failure.

However, it a) is a server-side solution and b) requires a cluster filesystem like e.g. GPFS or GFS (which OTOH could be an advantage in comparison to your unison solution).

Comments

[lynxman]

Thanks larsks, at the end I'll end up running some version of GFS over a cabinet but right now I have to expand the lifetime of these two servers, I'll have a look at this!

[user2751502]

If you do have plans to run NFS on top of GFS, there's a lot of good reading in RedHat's bugzilla. E.g., this search.