Fastest way to chown a whole device (xfs)

Question

I need to chown 1.5 million files on a drive. I'm currently doing:

sudo chown -R www-data:www-data /root-of-device

but it takes an awfully long time to run. I was wondering if there was some sort of superfast low-level way to chown every file on the drive.

Solution 1

Use xargs -p or GNU parallel to speed things up considerably.

Solution 2

Unfortunately I do not think there is such a thing, but I would be pleasantly surprised if there was. You could write your own implementation in C and optimise it heavily. However the success of that depends on how well optimized chown is to begin with. And considering it's one of the core utilities I would say it's rather optimized. In addition you are bound most likely by i/o speed.

I have had some success avoiding limitations of ls and rm by piping the results of find to xargs, in the case a directory has a lot of files, i.e.:

find /path/* | xargs rm

So, a wild guess, maybe this can speed up chown, in case it is slower at recursively scanning a filesystem than find:

sudo find /path/* | xargs chown www-data:www-data

Comments

[Aidan Kane]

Ha! Don't fancy my chances of coming up with a better implementation to be honest. It's an Amazon EBS device which means that i/o performance is a bit unstable.

[aseq]

I updated my answer with a few suggestions that may help.

[geekosaur]

In theory you could abuse xfs_metadump and xfs_db to get something slightly faster; in practice it's not worth the effort or the potential failure modes.

[geekosaur]

On a single filesystem, the speedup will be less than you think; potentially it could be slower because of contention between the multiple threads of execution accessing the same filesystem.

[pfo]

That should not be the case in XFS, as it supports parallel meta data operation according to the number of allocation groups created at formatting time. A parallel chmod will be faster than the sequential version.

[aseq]

Yeah I think whichever way you look at it it will take a long time...

[Aidan Kane]

I tried xargs -P4 (number of allocation groups on my xfs) and it ran slightly faster than the regular chown. I ran it a few times though and I think the difference in timings was just AWS EBS i/o fluctuations.

[Aidan Kane]

Thanks for the hints. Playing around with various things it really looks like I'm not going to do much better than regular chown.

[vonbrand]

screen(1) is a solution, but perhaps better is to use nohup(1).

[Chris F]

So what's the complete pipe using chown ... and xargs -p ?