inodes, consumed space comparison for many small files (xfs, btrfs,ext4)

Question

I have an ext4 partition (LVM on a VM) with a large amount of small files, which I have to extend every 3-4 months.

Regarding the amount of space used by the inodes.

Does one of the xfs, btrfs or ext4 file systems use less space?

In other words, will switching to btrfs or xfs make a partition fill up with inodes slower then with ext4?

Solution 1

To start with your first question: yes, one of these filesystems uses less space. Even without further details it would be unlikely that they all use the exact same amount of space, given that they have different implementations. So one is bound to use less space than all of the others.

Btrfs has dynamic inode allocation, so there is no filling up like you have with the inode tables for ext4 (the size for which is set at ext4 filesystem creation time).

XFS is dynamic in a similar way, but has a limit (percentage of the filesystem that can be used for inodes), so there, whether you fill-up your inode allowance depends on the percentage set as well as the file count/file sizes

Solution 2

Yes, and be aware that everything depends on your needs:

Btrfs (pronounced as Butter FS, Better FS, or B-Tree FS)

Considering that the btrfs will be able for spanning over the multiple hard drives, it is a very good poit that it can support 16 times more drive space than the ext4. A maximum partition size of the btrfs file system is 16 exbibytes, as well as maximum file size is 16 exbibytes too.

Maximum Number of Files: 2**64

XFS

The XFS is a high-performance 64-bit journaling file system. XFS supports maximum file system size of 8 exbibytes for the 64-bit file system. Now, the RHEL 7.0 uses XFS as the default filesystem, including support for using XFS for the /boot partition.

Maximum Number of Files: 2**64

EXT4

The ext4 is well known because of bringing the speed improvements over ext3. The ext4 has some limits. The maximum file size is 16 tebibytes (which is roughly 17.6 terabytes). The largest volume/partition you can have with ext4 is 1 exbibyte. Like in the most modern file systems, it is a journaling file system that means that it will keep a journal of where the files are mainly located on the disk and of any other changes that happen to the disk. Regardless all of its features, it doesn’t support the transparent compression, the transparent encryption, or the data deduplication. The snapshots are supported technically, but such feature is experimental at best.

Maximum Number of Files: 4 billion

XFS vs Btrfs

XFS doesn't have any RAID, while Btrfs RAID is not yet completely stable and is in its early days. XFS is more and more mature than Btrfs, but we cannot deny that Btrfs is powerful and a good-growing FileSystem.

For now, XFS is my choice - specially since it's the default FS on RHEL 7 -, unless I really need Btrfs.

Solution 3

I suppose the problem you have is not the partition filling up with inodes per se, but running out of the number of inodes in the filesystem. ext4 reserves the inodes statically when the filesystem is created, but you can set the number with options to mkfs.ext4:

-i bytes-per-inode
Specify the bytes/inode ratio. mke2fs creates an inode for every bytes-per-inode bytes of space on the disk. The larger the bytes-per-inode ratio, the fewer inodes will be created.

-N number-of-inodes
Overrides the default calculation of the number of inodes that should be reserved for the filesystem (which is based on the number of blocks and the bytes-per-inode ratio). This allows the user to specify the number of desired inodes directly.

The manual states explicitly that the bytes per inode ratio cannot be changed after the FS is created, but the total number will scale to meet the ratio if the FS is resized.

You can also set the size of each inode. The default is 256 bytes on "most" filesystems, but can be reduced to 128 (the default for "small" filesystems). The extra space is used for storing extended attributes (e.g. SELinux labels), so if you don't need those, it should be safe to reduce the size to the minimum.

-I inode-size
Specify the size of each inode in bytes. The inode-size value must be a power of 2 larger or equal to 128.

df -i should show the number of inodes allocated and used. With default options, one 30 GB partition I looked at had one inode for every 16 kB, but if your files are very small, you could set, say, -i 4096 to have one inode for every data block on the system.

If your files are smaller than 4096, you may want to reduce the file system block size too, since all regular files will require one full data block anyway. (That is, on ext4. I don't know if other current filesystems do packing of small files.)

-b block-size
Specify the size of blocks in bytes. Valid block-size values are 1024, 2048 and 4096 bytes per block. If omitted, block-size is heuristically determined by the filesystem size and the expected usage of the filesystem (see the -T option).

mkfs.ext4 also has the -T <type> option that can be used as a shorthand for some or all of these. The settings are in /etc/mke2fs.conf, which on my Debian makes e.g. mkfs.ext4 -T small equivalent to

mkfs.ext4 -b 1024 -I 128 -i 4096

Which might not be a bad set of options for lots of small files (and no xattrs.)

If your files are even smaller than one kB, a file system may not be the best way to save the data, but something like a database or an application specific system should perhaps be considered.

Comments

[Matija Nalis]

if you use ext4 and expect to be storing mostly small types, you should create it with mkfs.ext4 -t news for best results. Also, I'd suggest testing - create (on lvm, or loopback device for example) each of the filesytems in turn, and start copying your real files to it until it fills up. When it fills, do df -i (or find | wc -l) to find which one managed to store most of your files - that way you'll know for sure.

[ilkkachu]

@MatijaNalis, -T with an uppercase T. The config file has some other useful-looking choices too

[drudru]

How small are the files?

[abadys]

Thanks for the quick reply! Looks like XFS is the safest choice for now. Can this inode usage percentage be changed dynamically? and if not, is there a way to predict how much will be needed?

[Anthon]

You should be able to do that with xfs_growfs -m XX

[Admin]

@ilkkachu "large amount of small files" what does it mean? It's all about inodes since files are all created via inodes and the inode is a data structure used to represent a filesystem object. So I think I have explained all the author needs, plus I've mentioned about maximum number of files.

[spuk]

did not address the question; on the topic, you just posted the theoretical maximum number of files, which can be tuned, while the question was more about how to tune that and which fs would be more optimized for the case stated