ZFS didn't expand free space after add
Solution 1
Everything is operating normally. You're running into TB
vs TiB
confusion.
TL;DR
- 12TB raw == 10.8TiB raw (which is what is listed by
zpool list
) - 7.5TB usable == 6.8TiB usable (which is what is listed by
zfs list
)
You have:
- 3×3TB drives
- 2×1.5TB drives
yielding a total of 12TB of raw storage.
○ → units 12TB TiB
* 10.913936
/ 0.091625969
Which is roughly the space you get in zpool list:
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zstorage 10.8T 7.79T 3.05T 71% 1.00x ONLINE -
As for usable:
- 6TB (raidz1-0)
- 1.5TB (raidz1-1)
In TiB:
○ → units 7.5TB TiB
* 6.8212103
/ 0.14660155
You have:
NAME USED AVAIL REFER MOUNTPOINT
zstorage 5.19T 1.48T 5.19T /zstorage
The zstorage filesystem has 5.19TiB used and 1.48TiB available (6.67TiB total). Close enough to 6.82TiB for me (there will be some overhead).
Solution 2
You should check that the auto expand property is set for your zpool.
$ zpool get autoexpand zstorage
If the property is set to off, you should set it to on to have your pool automatically expand to fill the available space.
$ zpool set autoexpand=off zstorage
I'll include the relevant portion of the man page for clarity.
autoexpand=on | off
Controls automatic pool expansion when the underlying
LUN is grown. If set to on, the pool will be resized
according to the size of the expanded device. If the
device is part of a mirror or raidz then all devices
within that mirror/raidz group must be expanded before
the new space is made available to the pool. The default
behavior is off. This property can also be referred to
by its shortened column name, expand.
I believe you can also use
$ zpool online -e zstorage
to instruct the pool to manually expand, but I have not used that command myself. I'm not sure if the pool would need to be offlined first (I would expect it to be unnecessary).
zpool online [-e] pool device...
Brings the specified physical device online.
This command is not applicable to spares or cache dev-
ices.
-e
Expand the device to use all available space. If the
device is part of a mirror or raidz then all devices
must be expanded before the new space will become
available to the pool.
Related videos on Youtube
Kevin Wood
Updated on September 18, 2022Comments
-
Kevin Wood almost 2 years
I recently expanded my storage pool "zstorage" by adding two more drives. It had 3 3TB drives in RAID-Z, and I added two more 1.5TB drives also in RAID-Z (which I understand is effectively the same as mirroring). I know it's not the most efficient setup, but these are the drives I had. I expect that with parity, I should get to use 6TB from the first set and 1.5TB from the second for 7.5TB total.
After the add, everything looks fine, and from the commands below, you can see that there is now more space listed by zpool list, and the drives show up as I would expect with zpool status. However, the amount of disk space available did not increase, as you can see below.
Any idea why I did not get more disk space out of this? I seem to have gained nothing from adding the two 1.5TB drives.
kevin@atlas:~$ sudo zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT zstorage 10.8T 7.79T 3.05T 71% 1.00x ONLINE -
kevin@atlas:~$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT zstorage 5.19T 1.48T 5.19T /zstorage
kevin@atlas:~$ sudo zpool status -v
pool: zstorage state: ONLINE scan: scrub in progress since Sat Apr 5 10:49:13 2014 317G scanned out of 7.79T at 262M/s, 8h18m to go 1.50M repaired, 3.97% done config: NAME STATE READ WRITE CKSUM zstorage ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ata-WDC_WD30EZRX-00DC0B0_WD-WCC1T1735698 ONLINE 0 0 0 (repairing) ata-WDC_WD30EZRX-00DC0B0_WD-WMC1T0506289 ONLINE 0 0 0 ata-WDC_WD30EZRX-00MMMB0_WD-WCAWZ2711600 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 ata-WDC_WD15EADS-00P8B0_WD-WMAVU0454800 ONLINE 0 0 0 ata-WDC_WD15EADS-00P8B0_WD-WMAVU0524642 ONLINE 0 0 0 errors: No known data errors
kevin@atlas:~$ df -h /zstorage/
Filesystem Size Used Avail Use% Mounted on zstorage 6.7T 5.2T 1.5T 78% /zstorage
-
c4f4t0r about 10 yearsare you using zfs quota?
-
ewwhite about 10 yearsWow... Take a good read of nex7.blogspot.com/2013/03/readme1st.html
-
ewwhite about 10 yearsWhy is there a RAIDZ1 comprised of two disks?
-
ewwhite about 10 yearsRegardless of the reason, what you've done here was stupid and ill-advised. Use equal size disks! You can't just add random drives to the array like this. A two-disk raidz1 makes no sense. You've effectively reduced the performance and resiliency of your dataset by striping across a trio of disks and a pair of drives. This is completely a wipe-and-rebuild situation and it can't be undone.
-
Kevin Wood about 10 yearsI have 3 3TB disks and 2 1.5TB disks. This is for home use, and I don't intend to purchase any more disks. Now, that said, I absolutely would like to have the best configuration. :) Two-disk RAIDZ-1 is effectively the same as mirroring, is it not? I chose raidz simply because the other vdev was raidz, and I thought it didn't matter. Would 5-disk raidz-1 be the best option? I assume so, but this wasn't an option for me now since I started with 3-disk raidz. The problem seems out to be autoexpand, by the way. It was turned off, and I wasn't aware of the option.
-
Kevin Wood about 10 yearsLet me try and fix that last sentence... As I was writing I meant to say the problem "seems to be autoexpand" (having just noticed that on my pool), but now I'm not so sure that autoexpand is the fix either.
-
-
Steve Townsend about 10 yearsIncorrect - the autoexpand feature handles the case when the underlying LUN expands, not when you add new LUNs.
-
ewwhite about 10 yearsNope. It's not a TB versus TiB issue.
zpool list
will show the raw amount of pool space on raidz(2) volumes. -
Steve Townsend about 10 yearsPlease clarify.
TB
vsTiB
confusion explains the situation 100%. -
Kevin Wood about 10 yearsAs I understand, the commands show the raw free space, not counting for anything that will be used for parity. Writing 2MB would actually consume 3MB with parity. In any case, I can say with certainty that the "free" space did not increase when I added the new disks, but the "available" space did.
-
Kevin Wood about 10 yearsWell, autoexpand was off, which was the default configuration. I turned it on, ran zpool online -e zstorage <disk> for every disk, and ran zpool export and zpool import. After all of this, the problem still remains. It won't let me zpool online the vdevs "raidz1-0" and "raidz1-1", so I can't expand these. So in the end, I'm not sure that this is the solution.
-
Kevin Wood about 10 yearsOh man, I did have it all wrong. The space is there. I was just confused. Sorry all...
-
Steve Townsend about 10 yearsRight, since your LUN sizes didn't change. If you shut down your system, replaced your 3TB disks with 4TB disks,
dd
ed the data over to the 4TB disks, then turned everything back on this would be applicable. Or, more conventionally, expanded your LV or the LUN provided by your storage subsystem. :)