Is there a way to do something like LVM over NFS?

linux storage nfs lvm rhel4

16,113

Solution 1

While remarkably hackish, the route I ultimately ended-up following is this:

Using VMware ESXi, add datastores that are NFS mounts (from whereever).
Create vdisks on those datastores
Add the vdisks to a VM running RHEL (because I'm used to RHEL)
Put all of the added volumes into an LVM
Export the LVM partition via NFS

Pros:

simple
cheap
easy to replicate
with dynamic disk extension via VMware, the space can all be "allocated", but not "used" yet

Cons:

requires yet another layer (the hypervisor)
if any of the NFS mounts drops, the LVM could become corrupted (an issue faced under any of the potential solutions)

Solution 2

GlusterFS is very good for this job, you can also consider Lustre maybe (I have not used that one yet though) GlusterFS is NFS independent software but it would be very easy to move on that. You can also use it for Raid 10 networks which you might need in the future and it is very easy to scale.

Solution 3

You might also be interested in drbd+gfs.

Solution 4

I'm currently using IBM's GPFS on a HPC Linux cluster. Supports multiple direct attached nodes (we are using FibreChannel), others can have a network based block level access to the same volume.

Solution 5

Do you mean something like UnionFS (which isn't in any way analogous to LVM that I can think of), or just mounting several filesystems all next to each other (like /mnt/fs1, /mnt/fs2, /mnt/fs3)?

Also, re-exporting NFS mounts (and even NFS mounting filesystems that have filesystems mounted in them locally -- see the nohide option in exports(5)) is an exciting prospect, involving all sorts of corner cases and likely bug hideouts. "Here be monsters", indeed.

View more solutions

16,113

warren

I'm a hobbyist programmer, part-time sysadmin, and full-time analytics, big data, data center management, automation, and cloud computing architect and delivery engineer.

Updated on September 17, 2022

Comments

warren almost 2 years

I realize that since NFS is not block-level, LVM can't be used directly.

However: is there a way to combine multiple NFS exports (from, say, 3 servers) into one mount point on a different server?

Specifically, I'd like to be able to do this on RHEL 4 (or 5, and re-export the combined mount to my RHEL 4 server).

expansion
The reason I pegged lvm is that I want a bunch of exported mounts (servera:/mnt/export, serverb:/mnt/export, serverc:/mnt/export, etc) to all mount at /mnt/space so that my /mnt/space on this server (serverx) as one large filesystem.

Yes, I know that re-exporting is generally a Bad Thing™ but thought it might work, if there was a way to accomplish this on a newer release as opposed to an older one

From reading the unionfs docs, it appears that I can't use it over a remote connection - have I misread it? More accurately, since Union FS merges the contents of multiple branches, but makes them appear as one, it doesn't seem to go in reverse:
I'm trying to mount a bunch of NFS points in a merged fashion, then write to them - not caring where data goes, a la LVM .
warren over 14 years

the reason I pegged lvm is that I want a bunch of exported mounts (servera:/mnt/export, serverb:/mnt/export, serverc:/mnt/export) to all mount at /mnt/space so that my /mnt/space on this server (serverx) as one large filesystem
warren over 14 years

and yes, I know that re-exporting is generally a Bad Thing.. but thought it might work, if there was a way to accomplish this on a newer release as opposed to an older one
sybreon over 14 years

if NFS is required, cannot think of anything other than unionfs (and family).
warren over 14 years

from reading the unionfs docs, it appears that I can't use it over a remote connection - have I misread it?
warren over 14 years

more accurately, since Union FS merges the contents of multiple branches, but makes them appear as one, it doesn't seem to go in reverse: I'm trying to mount a bunch of NFS points in a merged fashion, then write to them - not caring where data goes, a la LVM
Rahim over 14 years

I don't think that's easily doable with NFS or UnionFS alone. GlusterFS lets you do something like this. It should even be possible to run the GlusterFS server and client on the same machine, and use each NFS volume as a GlusterFS "brick". Then use the distribute GlusterFS connector to, well, distribute your files among the bricks.
warren over 14 years

@Kamil - GlusterFS does look pretty close to what I'm looking for
warren over 14 years

this does look closest to what I'll need to implement
womble over 14 years

So, you don't want NFS, you want a cluster filesystem.
warren over 14 years

@womble - perhaps that is what I'm looking for, and if so, I worded my question poorly
YwH about 14 years

Reading about this kind of hack makes me feel diiirty. But not necessarily in a bad way. I'll have to keep this in mind someday when I try to do something just as crazy.
warren about 14 years

@Jed Danies: like I said, it's sketchy as all getout... but it works - and well :)
w00t about 14 years

Why not simply create files on the NFS mounts and add them as loopback devices? That way you can skip the VMware bit which adds a lot of overhead. See mail-archive.com/[email protected]/msg220815.htm‌l for an explanation on how to do it.
w00t about 14 years

BTW regarding the NFS mounts dropping, you could create a RAID5 set with dmraid, but you'd need to do some testing on how well that actually holds up.
warren about 14 years

@w00t - software raid over nfs. interesting idea :)