No more space on FreeNAS 9.3. But only half of space actually in use based on "du -sh" command
8,778
I try to sum up the given information and give some explanation (where I know, what it means):
About the zpool list output:
SIZE is the size of all discs, including the discs used for redundancy (so together with the output of zpool status that means, you have put together 8 3TB drives in a RAID-Z3 configuration)
ALLOC is the used space, also including the discs used for redundancy (so it is at least (5+3)/5=1,6 times higher than the real data size, because 3 of the 8 discs are used for redundancy)
Free is the free space, also including the discs used for redundancy (so the usable free space is at most 5/(5+3)=0,625 the shown free size)
The zfs list command says, there is 1,34T available, zpool list says 3,16T free, minus the redundancy space I would expect about 1,975T.
I know, that zfs needs some space for metadata, but on my linux system with 4 2TB drives in RAID-Z1 the difference is about 6%, your difference here is about 47%, I can't say where the difference come from.
About the output of df -Th
df doesn't find zfs snapshots, so it can't calculate them, so the space used is not the space used in the zfs filesystem. For allocated space in the zfs filesystem, use zfs list
Deleting snapshots can take time as you found out, seems to be by design, you can take a look at http://nex7.blogspot.de/2013/03/readme1st.html
After deleting the snapshots, if I count your df output, you have about 7,2T of data in your pool and about 4T free, making 11,2T all.
If I take your complete size (21,8T) minus the redundancy discs, I would expect about 5/(5+3)*21,8T=13,6T (a difference about 20%)
I think 20% is a little bit to much for the metadata needed by zfs, but I can't tell you, where it get lost. Maybe you have many very small files, which are using more metadata space than not so many bigger files.
I'm sorry, but I'm not so deep in the zfs internals, that I can tell you, where to get the metadata space usage.
You could try again a zfs list and compare the used space there with the used space you get from df. Maybe it gives a hint.
Edited 21.07.2015
About the size: 3T drives have actually 2,7 TiB (because the drive manufactures are using 1TB = 1.000.000.000.000 Bytes, where Computers using 1TiB = 1*1024*1024*1024*1024=1.099.511.627.776 Bytes.
2,7 * 8 drives = 21,8T
In normal RAID Systems, the smallest drive determines the usable size of the larger drives.
Related videos on Youtube
37 : 47
x22d Replacing a failed hard drive in my FreeNAS Server #FreeNAS #TrueNAS #ZFS
08 : 11
Configure raid in Freenas
07 : 24
How to Set Up Windows iSCSI Block Shares on TrueNAS & FreeNAS
06 : 22
How to swap a disk in FreeNAS and grow your pool.
25 : 16
How to fix a degraded pool state in TrueNAS ?
14 : 08
FreeNAS: Building a Dell R710 based FreeNAS storage server
05 : 48
Add/delete pool in TrueNAS/FreeNAS
04 : 44
How to Turn PC into Free NAS
Author by
陳敬翔
Updated on September 18, 2022Comments
-
陳敬翔 almost 2 years
This morning I got a warning on my FreeNAS 9.3 machine about "Not enough space"
warning: The capacity for the volume 'SeanVolume' is currently at 85%, while the recommended value is below 80%.It's of no use running the
scrubcommand after a period of time.I used SSH to login to FreeNAS host, and got some information:
[root@freenas] ~# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT SeanVolume 21.8T 18.6T 3.16T - 38% 85% 1.00x ONLINE /mnt freenas-boot 111G 680M 110G - - 0% 1.00x ONLINE -It shows it's ALLOC space only at 18.6T capacity, but it still had 6.7T by
du -shcommand:[root@freenas] ~# du -sh /mnt/SeanVolume/ 6.7T /mnt/SeanVolume/Additional information:
[root@freenas] ~# zpool status pool: SeanVolume state: ONLINE scan: scrub repaired 0 in 20h32m with 0 errors on Thu Jul 16 07:48:34 2015 config: NAME STATE READ WRITE CKSUM SeanVolume ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 gptid/f4986ea8-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 gptid/f51a05f7-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 gptid/f595d78e-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 gptid/f62490d4-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 gptid/f6a8e41e-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 gptid/f7266471-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 gptid/f7f3bf28-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 gptid/f8b8fa7b-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 cache gptid/f901ec00-f822-11e4-a7d4-d05099265144 ONLINE 0 0 0 gptid/dfa53351-1baa-11e5-ba0e-d05099265144 ONLINE 0 0 0 errors: No known data errorsResult of
df -THcommand:[root@freenas] ~# df -TH Filesystem Type Size Used Avail Capacity Mounted on freenas-boot/ROOT/default zfs 115G 698M 114G 1% / devfs devfs 1.0k 1.0k 0B 100% /dev tmpfs tmpfs 33M 5.5M 28M 16% /etc tmpfs tmpfs 4.2M 8.2k 4.2M 0% /mnt tmpfs tmpfs 11G 67M 11G 1% /var freenas-boot/grub zfs 114G 8.1M 114G 0% /boot/grub SeanVolume zfs 1.6T 822k 1.6T 0% /mnt/SeanVolume SeanVolume/CandyDataset zfs 1.6T 299k 1.6T 0% /mnt/SeanVolume/CandyDataset SeanVolume/CandyDataset/Applications zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/CandyDataset/Applications SeanVolume/CandyDataset/Documents zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/CandyDataset/Documents SeanVolume/CandyDataset/Downloads zfs 2.9T 1.3T 1.6T 46% /mnt/SeanVolume/CandyDataset/Downloads SeanVolume/CandyDataset/Music zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/CandyDataset/Music SeanVolume/CandyDataset/Pictures zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/CandyDataset/Pictures SeanVolume/CandyDataset/Temporary zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/CandyDataset/Temporary SeanVolume/CandyDataset/Videos zfs 1.8T 221G 1.6T 12% /mnt/SeanVolume/CandyDataset/Videos SeanVolume/PublicDataset zfs 1.6T 243k 1.6T 0% /mnt/SeanVolume/PublicDataset SeanVolume/PublicDataset/Applications zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/PublicDataset/Applications SeanVolume/PublicDataset/Documents zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/PublicDataset/Documents SeanVolume/PublicDataset/Downloads zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/PublicDataset/Downloads SeanVolume/PublicDataset/Music zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/PublicDataset/Music SeanVolume/PublicDataset/Pictures zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/PublicDataset/Pictures SeanVolume/PublicDataset/Temporary zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/PublicDataset/Temporary SeanVolume/PublicDataset/Videos zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/PublicDataset/Videos SeanVolume/SeanDataset zfs 3.9T 2.3T 1.6T 60% /mnt/SeanVolume/SeanDataset SeanVolume/SeanDataset/AppData zfs 1.6T 224k 1.6T 0% /mnt/SeanVolume/SeanDataset/AppData SeanVolume/SeanDataset/Applications zfs 1.6T 261k 1.6T 0% /mnt/SeanVolume/SeanDataset/Applications SeanVolume/SeanDataset/Documents zfs 3.0T 1.5T 1.6T 48% /mnt/SeanVolume/SeanDataset/Documents SeanVolume/SeanDataset/Downloads zfs 2.2T 647G 1.6T 29% /mnt/SeanVolume/SeanDataset/Downloads SeanVolume/SeanDataset/Music zfs 1.6T 411k 1.6T 0% /mnt/SeanVolume/SeanDataset/Music SeanVolume/SeanDataset/Pictures zfs 1.6T 2.2M 1.6T 0% /mnt/SeanVolume/SeanDataset/Pictures SeanVolume/SeanDataset/Saved Games zfs 1.6T 243k 1.6T 0% /mnt/SeanVolume/SeanDataset/Saved Games SeanVolume/SeanDataset/Temporary zfs 1.6T 533k 1.6T 0% /mnt/SeanVolume/SeanDataset/Temporary SeanVolume/SeanDataset/Videos zfs 2.4T 850G 1.6T 35% /mnt/SeanVolume/SeanDataset/Videos SeanVolume/jails zfs 1.6T 710M 1.6T 0% /mnt/SeanVolume/jails SeanVolume/jails/.warden-template-pluginjail zfs 1.6T 619M 1.6T 0% /mnt/SeanVolume/jails/.warden-template-pluginjail SeanVolume/jails/.warden-template-pluginjail-9.3 zfs 1.6T 619M 1.6T 0% /mnt/SeanVolume/jails/.warden-template-pluginjail-9.3 SeanVolume/jails/pluginjail zfs 1.6T 660M 1.6T 0% /mnt/SeanVolume/jails/pluginjail SeanVolume/ownCloudDataset zfs 1.8T 259G 1.6T 14% /mnt/SeanVolume/ownCloudDataset SeanVolume/.system zfs 1.6T 3.4M 1.6T 0% /var/db/system SeanVolume/.system/cores zfs 1.6T 1.4M 1.6T 0% /var/db/system/cores SeanVolume/.system/samba4 zfs 1.6T 6M 1.6T 0% /var/db/system/samba4 SeanVolume/.system/syslog-cd1fc29ce94d4a81a24df77359252261 zfs 1.6T 6M 1.6T 0% /var/db/system/syslog-cd1fc29ce94d4a81a24df77359252261 SeanVolume/.system/rrd-cd1fc29ce94d4a81a24df77359252261 zfs 1.6T 224k 1.6T 0% /var/db/system/rrd-cd1fc29ce94d4a81a24df77359252261 devfs devfs 1.0k 1.0k 0B 100% /mnt/SeanVolume/jails/pluginjail/dev procfs procfs 4.1k 4.1k 0B 100% /mnt/SeanVolume/jails/pluginjail/proc SeanVolume/jails/customplugin_2 zfs 1.6T 1.4G 1.6T 0% /mnt/SeanVolume/jails/customplugin_2 SeanVolume/SeanDataset/ISOImage zfs 1.6T 318k 1.6T 0% /mnt/SeanVolume/SeanDataset/ISOImage devfs devfs 1.0k 1.0k 0B 100% /mnt/SeanVolume/jails/customplugin_2/dev procfs procfs 4.1k 4.1k 0B 100% /mnt/SeanVolume/jails/customplugin_2/proc /mnt/SeanVolume/ownCloudDataset nullfs 1.8T 259G 1.6T 14% /mnt/SeanVolume/jails/customplugin_2/mediaRunning the scrub status (Web GUI):
Scrub status: Completed Errors: 0 Repaired: 0 Date: Thu Jul 16 07:48:34 2015What can I do to attempt to fix this (except reboot, because it is running ownCloud Plugin online)?
PS: 2015/07/17 Added:
[root@freenas] ~# zfs list NAME USED AVAIL REFER MOUNTPOINT SeanVolume 10.7T 1.34T 803K /mnt/SeanVolume SeanVolume/.system 304M 1.34T 3.21M legacy SeanVolume/.system/cores 5.53M 1.34T 1.32M legacy SeanVolume/.system/rrd-cd1fc29ce94d4a81a24df77359252261 219K 1.34T 219K legacy SeanVolume/.system/samba4 218M 1.34T 5.59M legacy SeanVolume/.system/syslog-cd1fc29ce94d4a81a24df77359252261 61.4M 1.34T 5.62M legacy SeanVolume/CandyDataset 1.42T 1.34T 292K /mnt/SeanVolume/CandyDataset SeanVolume/CandyDataset/Applications 365K 1.34T 219K /mnt/SeanVolume/CandyDataset/Applications SeanVolume/CandyDataset/Documents 365K 1.34T 219K /mnt/SeanVolume/CandyDataset/Documents SeanVolume/CandyDataset/Downloads 1.22T 1.34T 1.22T /mnt/SeanVolume/CandyDataset/Downloads SeanVolume/CandyDataset/Music 365K 1.34T 219K /mnt/SeanVolume/CandyDataset/Music SeanVolume/CandyDataset/Pictures 365K 1.34T 219K /mnt/SeanVolume/CandyDataset/Pictures SeanVolume/CandyDataset/Temporary 365K 1.34T 219K /mnt/SeanVolume/CandyDataset/Temporary SeanVolume/CandyDataset/Videos 206G 1.34T 206G /mnt/SeanVolume/CandyDataset/Videos SeanVolume/PublicDataset 3.03M 1.34T 237K /mnt/SeanVolume/PublicDataset SeanVolume/PublicDataset/Applications 365K 1.34T 219K /mnt/SeanVolume/PublicDataset/Applications SeanVolume/PublicDataset/Documents 365K 1.34T 219K /mnt/SeanVolume/PublicDataset/Documents SeanVolume/PublicDataset/Downloads 365K 1.34T 219K /mnt/SeanVolume/PublicDataset/Downloads SeanVolume/PublicDataset/Music 365K 1.34T 219K /mnt/SeanVolume/PublicDataset/Music SeanVolume/PublicDataset/Pictures 365K 1.34T 219K /mnt/SeanVolume/PublicDataset/Pictures SeanVolume/PublicDataset/Temporary 365K 1.34T 219K /mnt/SeanVolume/PublicDataset/Temporary SeanVolume/PublicDataset/Videos 365K 1.34T 219K /mnt/SeanVolume/PublicDataset/Videos SeanVolume/SeanDataset 8.04T 1.34T 2.14T /mnt/SeanVolume/SeanDataset SeanVolume/SeanDataset/AppData 2.64M 1.34T 219K /mnt/SeanVolume/SeanDataset/AppData SeanVolume/SeanDataset/Applications 178G 1.34T 256K /mnt/SeanVolume/SeanDataset/Applications SeanVolume/SeanDataset/Documents 2.58T 1.34T 1.33T /mnt/SeanVolume/SeanDataset/Documents SeanVolume/SeanDataset/Downloads 1.07T 1.34T 661G /mnt/SeanVolume/SeanDataset/Downloads SeanVolume/SeanDataset/ISOImage 103G 1.34T 310K /mnt/SeanVolume/SeanDataset/ISOImage SeanVolume/SeanDataset/Music 51.7G 1.34T 402K /mnt/SeanVolume/SeanDataset/Music SeanVolume/SeanDataset/Pictures 131G 1.34T 2.08M /mnt/SeanVolume/SeanDataset/Pictures SeanVolume/SeanDataset/Saved Games 75.7G 1.34T 237K /mnt/SeanVolume/SeanDataset/Saved Games SeanVolume/SeanDataset/Temporary 9.63G 1.34T 520K /mnt/SeanVolume/SeanDataset/Temporary SeanVolume/SeanDataset/Videos 846G 1.34T 792G /mnt/SeanVolume/SeanDataset/Videos SeanVolume/jails 3.80G 1.34T 678M /mnt/SeanVolume/jails SeanVolume/jails/.warden-template-pluginjail 6.33M 1.34T 591M /mnt/SeanVolume/jails/.warden-template-pluginjail SeanVolume/jails/.warden-template-pluginjail-9.3 597M 1.34T 591M /mnt/SeanVolume/jails/.warden-template-pluginjail-9.3 SeanVolume/jails/customplugin_2 1.17G 1.34T 1.27G /mnt/SeanVolume/jails/customplugin_2 SeanVolume/jails/pluginjail 1.28G 1.34T 630M /mnt/SeanVolume/jails/pluginjail SeanVolume/ownCloudDataset 1.21T 1.34T 255G /mnt/SeanVolume/ownCloudDataset freenas-boot 680M 107G 144K none freenas-boot/ROOT 670M 107G 144K none freenas-boot/ROOT/Initial-Install 8K 107G 659M legacy freenas-boot/ROOT/default 670M 107G 665M legacy freenas-boot/grub 7.76M 107G 7.76M legacy
2015/07/17 Added:
After I used
zfs list -t snapshotcommand, it show off many tiny snapshots (totally 2,534 files...).I am checking, and delete those snapshot files.
After delete, running
df -THagain!
2015/07/20 Add:
I got some space after I delete many snapshot files!
but...
The FreeNAS 9.3 Web GUI crashed...
[root@freenas] /etc/defaults# df -TH Filesystem Type Size Used Avail Capacity Mounted on freenas-boot/ROOT/default zfs 115G 698M 114G 1% / devfs devfs 1.0k 1.0k 0B 100% /dev tmpfs tmpfs 33M 5.5M 28M 16% /etc tmpfs tmpfs 4.2M 8.2k 4.2M 0% /mnt tmpfs tmpfs 11G 67M 11G 1% /var freenas-boot/grub zfs 114G 8.1M 114G 0% /boot/grub SeanVolume zfs 4T 822k 4T 0% /mnt/SeanVolume SeanVolume/CandyDataset zfs 4T 299k 4T 0% /mnt/SeanVolume/CandyDataset SeanVolume/CandyDataset/Downloads zfs 5.3T 1.3T 4T 25% /mnt/SeanVolume/CandyDataset/Downloads SeanVolume/CandyDataset/Videos zfs 4.2T 221G 4T 5% /mnt/SeanVolume/CandyDataset/Videos SeanVolume/PublicDataset zfs 4T 224k 4T 0% /mnt/SeanVolume/PublicDataset SeanVolume/SeanDataset zfs 9.6T 5.6T 4T 59% /mnt/SeanVolume/SeanDataset SeanVolume/jails zfs 4T 710M 4T 0% /mnt/SeanVolume/jails SeanVolume/jails/.warden-template-pluginjail zfs 4T 619M 4T 0% /mnt/SeanVolume/jails/.warden-template-pluginjail SeanVolume/jails/.warden-template-pluginjail-9.3 zfs 4T 619M 4T 0% /mnt/SeanVolume/jails/.warden-template-pluginjail-9.3 SeanVolume/jails/pluginjail zfs 4T 661M 4T 0% /mnt/SeanVolume/jails/pluginjail SeanVolume/ownCloudDataset zfs 4.3T 336G 4T 8% /mnt/SeanVolume/ownCloudDataset SeanVolume/.system zfs 4T 3.4M 4T 0% /var/db/system SeanVolume/.system/cores zfs 4T 1.4M 4T 0% /var/db/system/cores SeanVolume/.system/samba4 zfs 4T 6.1M 4T 0% /var/db/system/samba4 SeanVolume/.system/syslog-cd1fc29ce94d4a81a24df77359252261 zfs 4T 11M 4T 0% /var/db/system/syslog-cd1fc29ce94d4a81a24df77359252261 SeanVolume/.system/rrd-cd1fc29ce94d4a81a24df77359252261 zfs 4T 224k 4T 0% /var/db/system/rrd-cd1fc29ce94d4a81a24df77359252261 devfs devfs 1.0k 1.0k 0B 100% /mnt/SeanVolume/jails/pluginjail/dev procfs procfs 4.1k 4.1k 0B 100% /mnt/SeanVolume/jails/pluginjail/proc SeanVolume/jails/customplugin_2 zfs 4T 1.4G 4T 0% /mnt/SeanVolume/jails/customplugin_2 devfs devfs 1.0k 1.0k 0B 100% /mnt/SeanVolume/jails/customplugin_2/dev procfs procfs 4.1k 4.1k 0B 100% /mnt/SeanVolume/jails/customplugin_2/proc /mnt/SeanVolume/ownCloudDataset nullfs 4.3T 336G 4T 8% /mnt/SeanVolume/jails/customplugin_2/media-
Admin almost 9 yearszpool list includes in ALLOC the space needed for the parity Information. So ALLOC in zpool list is always more than a du shows. Try zfs list and post the output. -
Admin almost 9 yearsCould it be, that you use zfs snapshots? They will also allocate space, but will probably not be visible in a simple du command. You can find snapshots and the space they will use with a zfs list -t snapshot command.
-
-
陳敬翔 almost 9 yearsI really appreciate your detail-minded explanations, I learned a lot! In fact, I'm not sure why: I actually used the 6 3TB and 2 4T, which theoretically should have 24 ~ 26T, but last only 21.8 TB. Finally it's only 12.1TB after redundancy. And why a difference of 20%, it's possible to wait other ZFS experts to answer. Thank you very much!
-
aaaaa says reinstate Monica almost 9 yearsI have 24x1TB array in 4xRAIDZ2 (2 disk of redundancy per every 6 disks, theoretical volume of 16TB) and
zpool listshows 21.8TB.zfs listshows 13.3T. It seems thatzpool listshows space projected for compression. -
Sunzi almost 9 yearsNo. zpool list shows space including the redundancy drives. 1TB=0,909 TiB. 0,909Tib*24=21,8TiB so zpool list shows simply the complet space all drives have. zfs list shows the usable space (minus redundancy, minus metadata)
