Secure Network Filesystems for Linux: What are people doing?

Question

NFSv3 is widespread, but the default security model is... quaint. CIFS can use Kerberos authentication, but without POSIX semantics it's a non-starter. AFS never did encrypt traffic on the wire and is krb4 — and basically a dead project. Fancy new experimental filesystems either never materialize or are focused on speed (and if you're lucky, data reliability) — for example, Lustre uses the same client-trust model as NFSv3. For home use, sshfs is nifty, but that sure doesn't scale.

And then of course there's NFSv4, with sec=krb5p. Great in theory, but after ten years, it seems to be troublingly unused in the real world. The Linux client has just now had the experimental tag removed. And if you look at EMC Celerra, Isilon, etc., it's all NFSv3. (Celerra supports NFSv4, but it's really buried in the documentation. Isilon apparently worked at adding the RPCGSS support to FreeBSD, so maybe it's coming, but it's not there now. ) I can't even tag this post as "nfsv4" because I'm new here and that'd be a new tag.

So, really. What are you all doing?

Solution 1

Since it's a specific question (What are you all doing), let's answer it: nothing. Most administrators and users just don't worry about NFS security, so everybody uses NFSv3. It's typically a controlled environment (in the sense that only well-known machines can attach to the network in the first place). If somebody gets caught abusing the infrastructure, they get fired or to jail.

For data that you really don't want anybody to be able to read, you encrypt them explicitly, e.g. Firefox password databases, ssh keys, or pgp keys. You do that because you know the admin could read them on the file server, so network file system security would not be of any help, anyway.

Solution 2

You seem to be asking two questions here:

What are we actually using? and What does this?

What I'm actually using is CIFS, in my use-cases POSIX is less important so I haven't had any problems. NFS3 is used in areas where security isn't important, such as my SLES install server. And finally, sshfs/gvfs for simple user-land sharing. Wireline encryption is not deemed needed, so that isn't a meaningful factor for us.

As for the other question, there seems to be six main requirements for what you're looking for:

1. Encrypts traffic on the wire.
2. Encrypts authentication.
3. Posix semantics.
4. Strong enforcement of server-based ACLs.
5. Is not userland.
6. Is actually used.

I suspect points 5 and 6 will be the killers here, but here goes (also, this is the point where a table would be really handy, but markdown/StackExchange doesn't support it).

NFSv3 + IPSec

1. Encrypted on the wire, pass
2. No encrypted authentication, fail
3. Posix semantics, pass
4. No strong enforcement of server-based ACLs, fail
5. Is not userland, pass
6. Is actually used, pass

NFSv4 + Krb + IPSec

1. Encrypted on the wire, pass
2. Encrypted authentication, pass
3. Posix semantics, pass
4. Strong enforcement of server-based ACLs, pass
5. Is not userland, pass
6. Is not actually used, fail

CIFS

1. Not encrypted on the wire, fail
2. Encrypted authentication
3. Posix semantics, pass (Samba & Kernel now, Windows has had a Posix layer since the NT days)
4. Strong enforcement of server-based ACLs, pass
5. Is not userland, pass
6. Is actually used, pass

CIFS + IPSec

1. Encrypted on the wire, pass
2. Encrypted authentication
3. Posix semantics, pass (Samba & Kernel now)
4. Strong enforcement of server-based ACLs, pass
5. Is not userland, pass
6. Is not actually used, fail

SSHFS

1. Encrypted on the wire, pass
2. Encrypted authentication, pass
3. Posix semantics, pass
4. Strong enforcement of server-based ACLs, pass
5. Is userland, fail
6. Is actually used, pass

AFP/NetATalk

1. Encrypted on the wire, fail
2. Encrypted authentication, pass
3. Posix semantics, pass
4. Strong enforcement of server-based ACLs, pass
5. Is not userland, pass
6. Is actually used, fail

And I'm not touching the distributed file-systems out there. There simply isn't one single thing that does it all. Some come close (CIFS) and some are already there but no one uses them (NFS4 + IPSec, CIFS+IPSec). For some reason a secure network filesystem is something that has been subjected to a lot of compromises over the years.

Solution 3

I've been using openafs in production for years, with both Linux and Windows clients. It works great, has an active development community, and has gotten much easier to install and administer over the last few years as the various linux distros have included packaging for it. It has its warts, but I've found that they are offset by more administrative flexibility, the ability to have clients and servers separated by slow links, the ease of offsite backups, and other positive AFSisms.

One thing I like in particular is running production internet-facing web servers on openafs, with the ACLs locked down. Without a kerberos ticket there is no process on the machine -- even one running as root -- that can write to the filesystem. I can't count how many times we've noticed attacks utterly fail because of that simple measure.

There are some pretty large openafs users -- the largest commercial user I know of is Morgan Stanley.

Solution 4

Well, to me it sounds like one of those Distributed Filesystems would be for you. I wouldn't quite want to recommend OpenAFS, as it is old, doesn't support IPv6 yet,..

I'm quite happy with GlusterFS myself. Gluster is pretty mature, performs okay and has a good feature set. However, as recently discussed in IRC, Gluster does not support IPv6 in a stable manner either. This feature will be scheduled for 3.6 or 3.7.

There is also a project called HekaFS, which builds on Gluster, which adds more advanced Authentification features and SSL. It is imo extremely well documented and very well designed.

What may alos be of interest to you is XtreemFS, which is designed for global grid computing, so it comes with SSL and stuff by default. My choice for usage fell on Gluster though, since the community seems more active and it way better documented.

Both are posix compliant of course.

Solution 5

I see that a lot of people in this thread are talking about data hiding, i.e. attacks not being able to snoop on your data. A equally important to think about data integrity and authenticity. Are those nfs packet really from your nfs server? Did a nfs packet get changed in transit?

View more solutions

Comments

[Deb]

I'd like to say, "NFS3 over IPSEC", but I can't.

[mattdm]

"NFS3 over IPSEC" helps with the on-the-wire issue, but doesn't address another fundamental NFS problem: if a client box gets rooted, or if you're in a environment where users get root on their own systems, they can trivially impersonate any remote user.

[Cry Havok]

There you are, new tag ;)

[Javier]

AFAIK, Kerberos was defined for NFS

[Hubert Kario]

I'm not so sure about the need for encrypting traffic on the wire in a LAN environment (authentication should be encrypted though). You should monitor for ARP poisoning anyway...

[mattdm]

Hubert, we're in an academic environment, and university policy mandates that all traffic on the wire be encrypted. On more controlled server-room-only networks, I'm not so concerned. Perhaps what you're saying reflects common real-world use, though.....

[Thiago Figueiro]

Celerra supports NFSv4.1

[Hubert Kario]

Then change the policy ;). Seriously though, only server machines should be allowed to bind to NFSv3, for clients there's sshfs.

[mattdm]

XCondE: is there documentation on that publicly at all? I'm not terribly surprised that they support it, but given that they don't seem to mention it ever at all, I'd be (pleasantly, really) surprised if it's much used.

[mattdm]

FTR: I've confirmed that Celerra supports krb5/krb5i/krb5p. However, it's very, very sparsely documented. (Giving support to the "no one is doing this stuff in the real world" theory.)

[user3789902]

Kerberos a dead project? i hardly think so.

[mattdm]

Kerberos is not a dead project -- I was talking about AFS. Ever since IBM cut it lose, OpenAFS is basically limping along. And even if it got developer support, it's hamstrung by the GPL-incompatible license (it can never be merged into the mainline kernel).

[Dan Pritts]

you are right that openafs is limping along, but it does do krb5 authentication. It even does have the ability to do encryption over the wire but, sadly, that encryption is ancient DES-based. IIRC there is an unfunded project to add modern encryption support. Not easy, apparently, the code isn't as modular as you'd hope.

[user2751502]

I think you're missing a key point. As the question says, the problem is with the NFS security model. While encryption is nice, it is, as you say, a solvable problem. The big problem with NFS is that once a filesystem is mounted on a system, anyone with root access on that system can access any file on that filesystem, regardless of ownership or permissions. A system like AFS or, in theory, NFSv4 with sec=krbp5, requires strong credentials to access files, and so represents a substantial increase in security. A rooted NFS client does not equate a massive data exposure.

[BillThor]

Unless you ask the user to enter credentials for each access, credentials will be stored. A root compromised client is likely to give up the stored key easily. Any networked file system will increase the file system exposure to compromise.

[al.]

You could have mentioned "NFSv4 + Krb" and added "7. Is it reasonably fast (i. e. compared to the same protocol stack without encryption)?" as a question. Which would probably be a fail for NFSv4 + krb5p, but pass for questions 1-6.

[user3789902]

might be time for a new secure network filesystem SNFS?

[Deb]

@user37899 The problem, as always, is convincing appliance vendors to support it, and organizations to deploy it.

[Rob Olmos]

@BillThor This is what I was thinking.. Is it still open to attack if the credentials are in kernel memory? I think a kernel module could be loaded up to read any piece of kernel memory.

[BillThor]

As long as the request is used in the context of a user with access to the shared storage, it likely doesn't matter where the credentials are. Credentials are often held by a background process, so anyone who can communicate with it can likely gain access to the shared storage. I would rank the risk to secured network storage about the same as local storage.

[user3789902]

dedicated network, until someone jacks into the rack switch with wireshark.

[Hubert Kario]

NFS over IPsec takes care of that (on wide area links) and monitoring for ARP poisoning does that on LAN

[giubacchio]

That's a physical security problem then. Once they are in the same room with the servers, it's game over anyway.

[mattdm]

We've had really bad luck trying to use CIFS in POSIX mode. Maybe it's time to revisit that.

[user3789902]

is anyone actually using ipsec with success?

[mattdm]

@BillThor: with kerberos, the risk is significantly mitigated, since the attacker would only have access to the filesystems of users who have forwarded their tickets, and only for the lifetime of those tickets. With system-based authentication (a la nfsv3), root can access and manipulate the files of any user, even if that user had nothing to do with the compromised system ever.

[mattdm]

I've used OpenAFS in production. Rumors of its aliveness are greatly exaggerated.

[Hubert Kario]

There have been new release this month and before that there have been fairly regular updates to support new versions of Windows and new versions of Linux kernel (newest release supports 3.0).

[Dan Pritts]

FWIW I am using CIFS + IPsec, but not with POSIX semantics. server is emc celerra, clients win7. ipsec tunnels done in lan-to-lan mode between cisco ASA (next to celerra) and the win7 builtin ipsec.