Trim with LVM and dm-crypt
Solution 1
I suggest using a different testing method. hdparm
is a bit weird as it gives device addresses rather than filesystem addresses, and it doesn't say which device those addresses relate to (e.g. it resolves partitions, but not devicemapper targets, etc.). Much easier to use something that sticks with filesystem addresses, that way it's consistent (maybe except for non-traditional filesystems like zfs/btrfs).
Create a test file: (not random on purpose)
# yes | dd iflag=fullblock bs=1M count=1 of=trim.test
Get the address, length and blocksize: (exact command depends on filefrag
version)
# filefrag -s -v trim.test
File size of trim.test is 1048576 (256 blocks, blocksize 4096)
ext logical physical expected length flags
0 0 34048 256 eof
trim.test: 1 extent found
Get the device and mountpoint:
# df trim.test
/dev/mapper/something 32896880 11722824 20838512 37% /mount/point
With this set up, you have a file trim.test
filled with yes
-pattern on /dev/mapper/something
at address 34048
with length of 256
blocks of 4096
bytes.
Reading that from the device directly should produce the yes
-pattern:
# dd bs=4096 skip=34048 count=256 if=/dev/mapper/something | hexdump -C
00000000 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a |y.y.y.y.y.y.y.y.|
*
00100000
If TRIM is enabled, this pattern should change when you delete the file. Note that caches need to be dropped also, otherwise dd
will not re-read the data from disk.
# rm trim.test
# sync
# fstrim -v /mount/point/ # when not using 'discard' mount option
# echo 1 > /proc/sys/vm/drop_caches
# dd bs=4096 skip=34048 count=256 if=/dev/mapper/something | hexdump -C
On most SSD that would result in a zero pattern:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00100000
If encryption is involved, you will see a random pattern instead:
00000000 1f c9 55 7d 07 15 00 d1 4a 1c 41 1a 43 84 15 c0 |..U}....J.A.C...|
00000010 24 35 37 fe 05 f7 43 93 1e f4 3c cc d8 83 44 ad |$57...C...<...D.|
00000020 46 80 c2 26 13 06 dc 20 7e 22 e4 94 21 7c 8b 2c |F..&... ~"..!|.,|
That's because physically trimmed, the crypto layer reads zeroes and decrypts those zeroes to "random" data.
If the yes
-pattern persists, most likely no trimming has been done.
Solution 2
This is just a script I would like to share if some lazy person come here. It was made out of the accepted answer from frostschutz.
#!/bin/bash # # This script is provided "as is" without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. # # License GPL2 # # by desgua 2014/04/29 function CLEAN { cd "$pasta" [ -f test-trim-by-desgua ] && rm test-trim-by-desgua && echo "Temp file removed" echo "Goodbye" exit 0 } trap 'echo ; echo "Aborted." ; CLEAN; echo ; exit 0' INT HUP if [[ "$(echo $USER)" != "root" ]]; then read -n 1 -p 'Become root? [Y/n]' a if [[ $a == "Y" || $a == "y" || $a == "" ]]; then sudo $0 $1 exit 0 else echo " This script needs root privilege. " exit 1 fi fi name=$(echo $0 | sed 's/.*\///') if [ $# -ne 1 ]; then echo " Usage: $name /folder/to/test/ " exit 1 fi pasta=$1 read -n 1 -p 'Use fstrim? [y/N]' a if [[ $a == "Y" || $a == "y" ]]; then fs=1 fi method= while [[ "$method" != "1" && "$method" != "2" ]]; do read -n 1 -s -p 'Choose a method: [1] hdparm (will fail in LUKS on LVM) [2] filefrag (warning: you may have to force quit - close the terminal - in some cases of success trim if you see an output that never ends) ' method done function SDATEST { disk=$(fdisk -l | grep /dev/sda) if [ "$disk" == "" ]; then echo " fdisk did not found /dev/sda " exit 1 fi } function TEST { echo "Entrying /" ; echo cd $pasta echo "Creating the file test-trim-by-desgua at $pasta" ; echo dd if=/dev/urandom of=test-trim-by-desgua count=10 bs=512k echo "Syncing and sleeping 2 seconds." ; echo sync sleep 2 hdparm --fibmap test-trim-by-desgua lbab=$(hdparm --fibmap test-trim-by-desgua | tail -n1 | awk '{ print $2 }') echo "As you can see, the file was created and its LBA begins at $lbab" ; echo echo "Syncing and sleeping 2 seconds." ; echo sync sleep 2 echo "Removing file test-trim-by-desgua" ; echo rm test-trim-by-desgua trap 'echo ; echo ; echo "Aborted." ; echo ; exit 0' INT echo "Syncing and sleeping 2 seconds." ; echo sync sleep 2 if [[ "$fs" == "1" ]]; then echo "fstrim $pasta && sleep 2" ; echo fstrim $pasta sleep 2 fi echo "This is readed from sector $lbab: " hdparm --read-sector $lbab /dev/sda pass=$(hdparm --read-sector $lbab /dev/sda | grep "0000 0000 0000 0000") if [[ $pass == "" ]]; then echo " Trim failed... You should see only 0000 0000 0000 0000 ... " else echo "Success!!!" fi exit 0 } function LUKSTEST { # Reference: https://unix.stackexchange.com/questions/85865/trim-with-lvm-and-dm-crypt# echo 1 > /proc/sys/vm/drop_caches cd $pasta echo "Creating a \"yes\" file." yes | dd iflag=fullblock bs=1M count=1 of=test-trim-by-desgua #position=`filefrag -s -v test-trim-by-desgua | grep "eof" | awk '{ print $3 }'` position=`filefrag -s -v test-trim-by-desgua | grep "eof" | sed 's| ||g ; s|.*255:|| ; s|\.\..*||'` [[ "$position" == "" ]] && echo "Could not find the position of the file. Are you on a LUKS on LVM?" && CLEAN; device=`df test-trim-by-desgua | grep "dev/" | awk '{ print $1 }'` yes=`dd bs=4096 skip=$position count=256 if=$device | hexdump -C` echo "In the next line you should see a pattern like: 00000000 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a |y.y.y.y.y.y.y.y.| $yes " if [[ "`echo "$yes" | grep "y.y.y"`" == "" ]]; then echo "The pattern could not be checked. Something went wrong. Exiting." CLEAN; else echo "Pattern confirmed." fi echo "Removing the temp file." rm test-trim-by-desgua echo "Syncing." sync sleep 1 if [[ "$fs" == "1" ]]; then echo "fstrim -v $pasta && sleep 2" ; echo fstrim -v $pasta sleep 2 fi # Drop cache echo 1 > /proc/sys/vm/drop_caches echo "In the next line you should **NOT** see a yes pattern like: 00000000 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a |y.y.y.y.y.y.y.y.| If you see, then trim is not working: `dd bs=4096 skip=$position count=256 if=$device | hexdump -C`" yes=`dd bs=4096 skip=$position count=256 if=$device | hexdump -C` if [[ "`echo "$yes" | grep "y.y.y"`" != "" ]]; then echo "TRIM not working." else echo "TRIM is working!" fi CLEAN; } if [[ "$method" == "1" ]]; then SDATEST; TEST; elif [[ "$method" == "2" ]]; then LUKSTEST; fi exit 0
Solution 3
Your test routine is wrong—you're getting sector numbers relative to the block device which the filesystem sits on—which in this case, is a logical volume. The logical volume, of course, does not start at the first sector of the physical volume (and may not even be contiguous).
Even if the logical volume started at sector 0 of the physical volume (which it doesn't), then the physical volume is actually another device-mapper target, this one for encryption. And probably there is a LUKS header in front, so the sector numbers don't match there, either.
If you want to work through mapping the sector number to the underlying disk, dmsetup tables
will give you the info you need. If you paste it here, make sure yours is a version that does not show the key in the output (it should show all 0's instead)! (There is no recovery from disclosing the key—it can not be changed—it is far worse than disclosing the password).
I suggest that to debug (once you get the sector mapping worked out) you start at the lowest level, and confirm it works there. TRIM a filesystem directly on /dev/sdaX and make sure that works (its quite possible that the device lies, and trim doesn't read back zeros). Then dm-crypt on top of that, and trim a filesystem on that, and make sure it works. Finally, put LVM on top, and check that works.
Related videos on Youtube
student
Updated on September 18, 2022Comments
-
student over 1 year
I tried to setup TRIM with LVM and dm-crypt on ubuntu 13.04 following this tutorial:
See the notes about my configuration and my testing procedure below.
Questions
Is there a reliable test if TRIM works properly?
Is my test routine wrong or is my TRIM not working?
If it's not working: what is wrong with my setup?
How can I debug TRIM for my setup and make TRIM work?
Configuration
Here ist my configuration:
cat /etc/crypttab
sda3_crypt UUID=[...] none luks,discard
and
cat /etc/lvm/lvm.conf
# [...] devices { # [ ... ] issue_discards = 1 # [ ... ] } # [...]
The SSD is a Samsung 840 Pro.
Here is my test-procedure
To test the setup I just did
sudo fstrim -v /
which resulted in/: [...] bytes were trimmed
Doing this again resulted in
/: 0 bytes were trimmed
which seems to make sense and indicated that TRIM seems to work.However then I did this test:
dd if=/dev/urandom of=tempfile count=100 bs=512k oflag=direct
sudo hdparm --fibmap tempfile tempfile: filesystem blocksize 4096, begins at LBA 0; assuming 512 byte sectors. byte_offset begin_LBA end_LBA sectors 0 5520384 5521407 1024 524288 5528576 5529599 1024 1048576 5523456 5525503 2048 2097152 5607424 5619711 12288 8388608 5570560 5603327 32768 25165824 5963776 5980159 16384 33554432 6012928 6029311 16384 41943040 6275072 6291455 16384 50331648 6635520 6639615 4096
sync
sudo hdparm --read-sector 5520384 /dev/sda /dev/sda: reading sector 5520384: succeeded 7746 4e11 bf42 0c93 25d3 2825 19fd 8eda bd93 8ec6 9942 bb98 ed55 87eb 53e1 01d5 c61a 3f52 19a1 0ae5 0798 c6e2 39d9 771a b89f 3fc5 e786 9b1d 3452 d5d7 9479 a80d 114a 7528 a79f f475 57dc aeaf 25f4 998c 3dd5 b44d 23bf 77f3 0ad9 8688 6518 28ee 81db 1473 08b5 befe 8f2e 5b86 c84e c7d2 1bdd 1065 6a23 fd0f 2951 d879 e823 021b fa84 b9c1 eadd 9154 c9f4 2ebe cd70 64ec 75a8 4d93 c8fa 3174 7277 1ffb e858 5eca 7586 8b2e 9dbc ab12 40ab eb17 8187 e67d 5e0d 0005 5867 b924 5cfd 6723 9e4a 6f5f 99a4 a3b0 eeac 454a 83b6 c528 1106 6682 ca77 4edf 2180 bf0c b175 fabb 3d4b 37e2 b834 9e3e 82f2 2fdd 2c6a c6ca 873f e71e f979 160f 5778 356f 2aea 6176 46b6 72b9 f76e ee51 979c 326b 1436 7cfe f677 bfcd 4c3c 9e11 4747 45c1 4bb2 4137 03a1 e4c8 e9dd 43b4 a3b4 ce1b d218 4161 bf64 727b 75d8 dcc2 e14c ebec 2126 25da 0300 12bd 6b1a 28b3 824f 3911 c960 527d 97cd de1b 9f08 9a8e dcdc e65f 1875 58ca be65 82bf e844 50b8 cc1b 7466 58b8 e708 bd3d c01f 64fb 9317 a77a e43b 671f e1fb e328 93a9 c9c7 291c 56e0 c6c1 f011 b94d 9dc7 71e6 c8b1 5720 b8c9 b1a6 14f1 7299 9122 912b 312a 0f2f a31a 8bf9 9f8c 54e6 96f3 60b8 04a7 7dc9 3caa db0a a837 e5d7 2752 b477 c22d 7598 44e1 84e9 25d4 5db5 9f19 f73b 85a0 c656 373a ec34 55fb e1fc 124e 4674 1ba8 1a84 6aa4 7cb5 455e f416 adc6 a125 c4d4 8323 4eee 2493 2920 4e38 524c 1981
sudo rm tempfile
sync
sudo fstrim /
sync
sudo hdparm --read-sector 5520384 /dev/sda /dev/sda: reading sector 5520384: succeeded 7746 4e11 bf42 0c93 25d3 2825 19fd 8eda bd93 8ec6 9942 bb98 ed55 87eb 53e1 01d5 c61a 3f52 19a1 0ae5 0798 c6e2 39d9 771a b89f 3fc5 e786 9b1d 3452 d5d7 9479 a80d 114a 7528 a79f f475 57dc aeaf 25f4 998c 3dd5 b44d 23bf 77f3 0ad9 8688 6518 28ee 81db 1473 08b5 befe 8f2e 5b86 c84e c7d2 1bdd 1065 6a23 fd0f 2951 d879 e823 021b fa84 b9c1 eadd 9154 c9f4 2ebe cd70 64ec 75a8 4d93 c8fa 3174 7277 1ffb e858 5eca 7586 8b2e 9dbc ab12 40ab eb17 8187 e67d 5e0d 0005 5867 b924 5cfd 6723 9e4a 6f5f 99a4 a3b0 eeac 454a 83b6 c528 1106 6682 ca77 4edf 2180 bf0c b175 fabb 3d4b 37e2 b834 9e3e 82f2 2fdd 2c6a c6ca 873f e71e f979 160f 5778 356f 2aea 6176 46b6 72b9 f76e ee51 979c 326b 1436 7cfe f677 bfcd 4c3c 9e11 4747 45c1 4bb2 4137 03a1 e4c8 e9dd 43b4 a3b4 ce1b d218 4161 bf64 727b 75d8 dcc2 e14c ebec 2126 25da 0300 12bd 6b1a 28b3 824f 3911 c960 527d 97cd de1b 9f08 9a8e dcdc e65f 1875 58ca be65 82bf e844 50b8 cc1b 7466 58b8 e708 bd3d c01f 64fb 9317 a77a e43b 671f e1fb e328 93a9 c9c7 291c 56e0 c6c1 f011 b94d 9dc7 71e6 c8b1 5720 b8c9 b1a6 14f1 7299 9122 912b 312a 0f2f a31a 8bf9 9f8c 54e6 96f3 60b8 04a7 7dc9 3caa db0a a837 e5d7 2752 b477 c22d 7598 44e1 84e9 25d4 5db5 9f19 f73b 85a0 c656 373a ec34 55fb e1fc 124e 4674 1ba8 1a84 6aa4 7cb5 455e f416 adc6 a125 c4d4 8323 4eee 2493 2920 4e38 524c 1981
This seems to indicate that TRIM doesn't work. Since
sudo hdparm -I /dev/sda | grep -i TRIM * Data Set Management TRIM supported (limit 8 blocks) * Deterministic read ZEROs after TRIM
Edit
Here is the output of
sudo dmsetup table
lubuntu--vg-root: 0 465903616 linear 252:0 2048 lubuntu--vg-swap_1: 0 33308672 linear 252:0 465905664 sda3_crypt: 0 499222528 crypt aes-xts-plain64 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0 8:3 4096 1 allow_discards
Here is my
/etc/fstab
:# <file system> <mount point> <type> <options> <dump> <pass> /dev/mapper/lubuntu--vg-root / ext4 errors=remount-ro 0 1 # /boot was on /dev/sda2 during installation UUID=f700d855-96d0-495e-a480-81f52b965bda /boot ext2 defaults 0 2 # /boot/efi was on /dev/sda1 during installation UUID=2296-2E49 /boot/efi vfat defaults 0 1 /dev/mapper/lubuntu--vg-swap_1 none swap sw 0 0 # tmp tmpfs /tmp tmpfs nodev,nosuid,noexec,mode=1777 0 0
Edit:
I finally reported it as a bug in https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1213631
Hope somebody will find a solution there or at least test the setup and verify the bug.
Update
Now it works, see accepted answer.
-
student almost 11 yearsSorry, this was a typo. I have
issue_discards = 1
in my config file. -
Razzlero almost 11 yearsIf I were you I would try to use an iSCSI target and test this via tcpdump/wireshark to see if the setup works, though I do not know if the Linux iSCSI target supports trim or not. I do believe that dm-crypt should not blank the blocks on the physical disk because that makes it easier to ignore the free space on the device when trying to brute-force it (I don't know if it does that or not, though). In addition, SSDs are not required to return zeros after blanking, since the wear-leveling can redirect the read to a different block than the one blanked.
-
frostschutz over 10 yearsAccording to bugzilla.redhat.com/show_bug.cgi?id=958096 I misunderstood the issue_discards = 1.
-
derobert almost 11 years@student OK, that's the wrong sector then (the first two paragraphs of my answer). I'll edit my answer to remove that sentence about sector 6575104, as it isn't relevant any more.
-
student almost 11 yearsThanks! Instead of
rm trim.test; sync
I did `rm trim.test; sync; sudo fstrim -v .; However the pattern y Pattern persists. So TRIM seems not to work. Do you think my config is correct? -
student almost 11 yearsI am not sure which device I should take for
dmsetup
. I just did:sudo dmsetup table /dev/mapper/lubuntu--vg-root
which gives0 465903616 linear 252:0 2048
-
derobert almost 11 years@student That means sector 0 is at sector 2048 on device 252:0. You'll have to figure out what 252:0 is, I'd guess its your dm crypto device (that's the major and minor number, will show up in /dev for example). And you'll need to look at the table for that device, to continue chasing it down to a block on an underlying device.
-
frostschutz almost 11 yearsYou've successfully tested that TRIM doesn't work, then... ;) as for your setup, you haven't explained it in great detail (filesystem, mount options, does dmsetup table show allow_discards for both LVM and crypt, anything else in the mix like raid...?).
-
student almost 11 yearsSee my edit. I do not use raid. Do you need more info?
-
frostschutz over 10 years@student: I feel bad for not noticing this earlier, edited the answer to drop caches before
hexdump
. -
student over 10 yearsThanks, that was the missing point. Now it seems to work!
-
desgua about 10 years@frostschutz Thank you for this great solution. I made a script to automate the process if some lazy person come here.
-
Marc.2377 almost 7 years
-
frostschutz almost 7 years@Marc.2377 Maybe true, but it worked for every SSD I tested so far, which makes it difficult for me to update or change the answer. Some people got old data because they didn't drop caches. If it doesn't work for you, maybe try a larger file (like 128MiB instead of 1MiB), but that's just guessing, I can't verify what would need to be done instead for SSD models that somehow insist on returning discarded data.
-
Marc.2377 almost 7 years@frostschutz Thank you for considering updating your answer, I wasn't expecting that really. Well, my ssd in particular (the "SV300S37A/120G") doesn't support
discard_zeroes_data
. I tried with a 250MiB file and even restarted the computer. I don't even have LVM set up, only dm-crypt. It's an old model, SATA rev.3.0, TRIM works but does not zero the blocks. -
frostschutz almost 7 years@Marc.2377 dm-crypt with
allow_discards
? Check withdmsetup table
should sayluks: ... crypt aes-xts-plain64 ... ... 1 allow_discards
otherwise it can't work. I don't have this model but another that doesn't support zeroes data either but yet this test still works, original data is gone after trim. -
frostschutz almost 7 years@Marc.2377 can't reproduce with any SSD I own - would be nice to get a similar gist for a model that doesn't work. Guess I got lucky, not sure if I want a model that returns original data after TRIM. I don't like losing data to TRIM either but that's a different problem (too much trim in linux in general).
-
Marc.2377 almost 7 years@frostschutz I don't like it a lot either, but... it's encrypted so no reason to bother much.
allow_discards
is definitely enabled (result fromdmsetup table
is like yours, sans the preceedingluks
part for some reason. I'll generate a gist soon and update you then.