Very poor performance on LUKS/LVM/RAID combination under Debian Squeeze
Solution 1
So now, I updateds my machine. The result is an increased performance. I get valies of about 110 MB/sec. So it seems that the encrytion was a problem for the cpu.
Solution 2
That looks like you have a CPU with 4 cores (or at least Linux treating it like 4) and dm-crypt takes one core completely without being able to use the others. If the CPU does not allow more than 70 MiB/s then increasing I/O speed does not make a difference, of course.
I am surprised, though. dm-crypt shall be multithreaded since kernel 2.6.38 (that was March 2011). Maybe you can increase the throughput by configuring a different cipher. Or you get a CPU with AES-NI (crypto in hardware, unlimited speed...). What is your cipher (cryptsetup luksDump /dev/... | grep Cipher
)?
Edit 1
I just found this on http://code.google.com/p/cryptsetup/wiki/FrequentlyAskedQuestions:
Since version 1.60 cryptsetup supports the "benchmark" command. Simply run as root:
cryptsetup benchmark
I have 1.4.2 (kernel 3.4.33) so I could not give that a try.
Related videos on Youtube
Christian Wolf
Updated on September 18, 2022Comments
-
Christian Wolf almost 2 years
I have the following issue: I have an encrypted partition on a lvm on a plain partition on the hd. Now I craeted a RAID array to increase the performance. That means, that I have the following stack: HD(s) -- partition(s) -- RAID -- LVM -- Cryptsetup/LUKS. Now the performance is more or less the same with and without RAID (some measurements below).
Can anybody give me a hint why the performance does not gain a boost?
The measurements: First the output of
hdparm -t ...
:/dev/server-multimedia/pics: Timing buffered disk reads: 208 MB in 3.00 seconds = 69.22 MB/sec /dev/mapper/pics: Timing buffered disk reads: 198 MB in 3.01 seconds = 65.77 MB/sec /dev/server_raid/pics: Timing buffered disk reads: 860 MB in 3.01 seconds = 286.09 MB/sec /dev/mapper/pics_test: Timing buffered disk reads: 204 MB in 3.00 seconds = 67.98 MB/sec
/dev/server-multimedia/pics
is the partition without raid and encryption./dev/mapper/pics
the partition opend by luks./dev/server_raid/pics
the raid based encrypted partition and/dev/mapper/pics_test
is the luks-opened, raid based partition.We can see that for the plain partition we do not loose much at all. Where the raid based partition seems to have a very bad performance.
I also checked the cpu state while the test ran. First with the plain partition:
top - 13:07:41 up 5 days, 2:41, 4 users, load average: 0.22, 0.27, 0.16 Tasks: 287 total, 2 running, 285 sleeping, 0 stopped, 0 zombie Cpu(s): 0.4%us, 1.2%sy, 0.3%ni, 95.1%id, 2.8%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 900980k total, 752636k used, 148344k free, 178452k buffers Swap: 26364332k total, 116044k used, 26248288k free, 95880k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25592 root 20 0 0 0 0 R 92 0.0 0:24.13 kcryptd 6159 root 20 0 3824 3820 1492 D 20 0.4 0:00.83 hdparm 25591 root 20 0 0 0 0 S 4 0.0 0:00.38 kcryptd_io 6168 christia 20 0 2612 1176 796 R 2 0.1 0:00.02 top
Now with raid:
top - 13:07:54 up 5 days, 2:41, 4 users, load average: 0.25, 0.27, 0.16 Tasks: 287 total, 3 running, 284 sleeping, 0 stopped, 0 zombie Cpu(s): 0.4%us, 1.2%sy, 0.3%ni, 95.1%id, 2.8%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 900980k total, 619508k used, 281472k free, 86760k buffers Swap: 26364332k total, 116044k used, 26248288k free, 55952k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5594 root 20 0 0 0 0 R 84 0.0 0:06.76 kcryptd 6159 root 20 0 3824 3820 1492 D 18 0.4 0:02.94 hdparm 404 root 20 0 0 0 0 R 4 0.0 102:20.06 md1_raid5 5593 root 20 0 0 0 0 S 4 0.0 0:00.39 kcryptd_io 6170 christia 20 0 2612 1180 796 R 2 0.1 0:00.01 top
I can see a quite high CPU load. But in the top lines most of the time the CPUs seems to do nothing (approx. 90% idle). So what? Is my CPU the bottle neck?
-
Spack about 11 yearsWhat's your RAID setup and what's your hardware configuration?
-
Christian Wolf about 11 years@ptman AMD Athlon 64 X2 5200+. Sector size of hd is 4KiB. RAID partitions start at (logical) sector 19531776. Should be aligned correctly for this layer. Then the RAID/LVM/partition layers I do not know how to check alignment.
-
-
tim about 11 yearsDebian Squeeze is 2.6.32 AFAIR.
-
Christian Wolf about 11 yearsIt's only 200% as ist a dual core but that's quite enough... PS: The benchmark is impossible as debian used version 1.1.3. Bad.
-
Hauke Laging about 11 years@ChristianWolf But probably a dual core with HT. If a task consumes about 100% on just two cores then the load would be about 0.5 instead of about 0.25. If you make a kernel update don't forget to update this thread.