How to get decent NFS performance for workloads like git?
Solution 1
Try to use this Git setting, which makes a huge performance difference on NFS shares:
git config core.preloadindex true
Quote from the documentation available at: https://git-scm.com/docs/git-config#git-config-corepreloadIndex
This can speed up operations like git diff and git status especially on filesystems like NFS that have weak caching semantics and thus relatively high IO latencies. When enabled, Git will do the index comparison to the filesystem data in parallel, allowing overlapping IO’s. Defaults to true.
Solution 2
This article contains some useful tips to tune NFS performance.
Particularly the use of nfsstat -rc
to check how many 'retransmission retries' had been happening. If there are too many retries, that means the nfsd
daemon ran out of threads to service clients' requests, and you need to increase the number of available threads.
Also, ensure that your VirtualBox instance is not thinly-provisioned; thin provisioned storage on VirtualBox is a significant performance hit when writes happen.
Related videos on Youtube
Cera
Updated on September 18, 2022Comments
-
Cera over 1 year
I manage a
vagrant
setup for our developers running OSX to manage VirtualBox systems for development.In order to support
inotify
inside the linux machine, we eschew the usual method of sharing directories with VirtualBox: instead, the Virtualbox machine exposes an NFS share, which is mounted in OSX.Git (and related tools like sourcetree) are run in OSX, on the shared directory. The performance of this is extremely poor: it often takes up to 5 seconds just to run
git status
. Cloning a small repository into the mounted NFS drive can take a couple of minutes (5-10 seconds on a local drive).Obviously NFS performance will be worse than writing straight to a local SSD, but the NFS is just running over a virtual private network interface with Virtualbox.
I ran a couple of benchmarks. The first:
dd if=/dev/zero of=test bs=16 count=16384
Each result is based on 100 samples.
# local drive Min. 1st Qu. Median Mean 3rd Qu. Max. 0.990 1.000 1.020 1.021 1.030 1.130 # on the shared NFS drive Min. 1st Qu. Median Mean 3rd Qu. Max. 6.000 6.278 6.390 6.482 6.570 7.630 # inside the VirtualBox instance Min. 1st Qu. Median Mean 3rd Qu. Max. 0.3600 0.4800 0.5150 0.5552 0.5400 1.4500
The second benchmark:
ioping -c 500
local avg: 6 microseconds local IOPS: 65.5 k NFS Latency avg: 703 microseconds NFS OPS: 1.4 k
Clearly some latency is inevitable, but this is bad enough that it makes some simple tasks quite painful.
The owner of the volume is running Ubuntu 12.10 (Quantal), with default settings. The system mounting the volume is running OSX Mavericks.
At the moment the main nfsd connection is running over UDP, which seems ideal over a virtual connection. I'm not sure whether
statsd
andlockd
are running over TCP and UDP.I tried mounting with the
async
flag, and withrwsize
boosted, and it made little difference.What opportunities are there for seriously improving the performance of NFS in this environment?
-
Cera about 10 yearsAs it stands, the problem has not been solved but we've mitigated it substantially (improvements of up to 15x) by using a different Virtualbox network interface:
Am79C973
instead ofvirtio
. The lesson here is to have looked more closely at what the systems were doing: to a certain point, the bottleneck seems to be the virtual network performance. -
Ali about 8 yearsI have a similar issue. Running
git status
from the guest machine takes about 5 seconds. On the host machine which is OSX, it is quiet fast. My setup is latest vagrant, vb, shared NFS directory. How did you change the interface, I tried to do so but have not figured this out. Also, this became more noicable after some update. I did not have this issue in the beginning -
Adam Katz about 2 yearsHere's a post about using NFS with UnionFS so git uses local disks for most of its content. It gets a 70% speed improvement.
-
-
Cera about 10 yearsThanks for your response. The dynamic vs fixed volume thing is interesting - I did some reading that indicated that the performance difference was a myth, though if your experience is that it helped then it might be a try.
-
pepoluan about 10 yearsIt very much depends on the underlying physical storage... an empty partition with oodles of free space? The performance hit might be less noticeable. A partition with quite a lot of objects already? VBox will have to wait while the OS 'hunts around' for blocks of empty space to fulfill the growth request.
-
analytik almost 9 years@pepoluan - was that with a magnetic drive, or SSD?
-
pepoluan almost 9 years@analytik magnetic drive. With SSD, I think fragmentation will never be a problem, and you can safely go with thin provisioned storage. Heck, even if there's a performance hit, considering that the $/GB of SSD is still waaay higher than magnetic drive's, I will go thin provisioned.
-
Wildcard almost 7 yearsWhy does it make a performance difference?