[dm-crypt] Bad performance with software RAID5 and LUKS encryption

[Philipp Wendler <ml@philippwendler.de>]

Hi,

I have set up a Linux software RAID5 on three hard drives and want to
encrypt it with cryptsetup/LUKS. My tests showed that the encryption
leads to a massive performance decrease that I cannot explain.

The RAID5 is able to write 187 MB/s [1] without encryption. With
encryption on top of it, write speed is down to about 40 MB/s.

The RAID has a chunk size of 512K and a write intent bitmap. I used -c
aes-xts-plain -s 512 --align-payload=2048 as the parameters for
cryptsetup luksFormat, so the payload should be aligned to 2048 blocks
of 512 bytes (i.e., 1MB), if I understood this correctly. cryptsetup
luksDump shows a payload offset of 4096. So I think the alignment is
correct and fits to the RAID chunk size.

The CPU should not be the bottleneck here, as it has hardware support
for AES (aesni_intel). If I write on another drive (an SSD with LVM)
that is also encrypted, I do have a write speed of about 150 MB/s. top
shows that the CPU usage is indeed very low, only the RAID5 xor takes 14%.

I also tried putting a filesystem (ext4) directly on the unencrypted
RAID so see if the layering is problem. The filesystem decreases the
performance a little bit as expected, but by far not that much (write
speed varying, but > 100 MB/s).

Summary:
Disks + RAID5: good
Disks + RAID5 + ext4: good
Disks + RAID5 + encryption: bad
SSD + encryption + LVM + ext4: good

The read performance is not affected by the encryption, it is 207 MB/s
without and 205 MB/s with encryption (also showing that CPU power is not
the problem).

Is it possible to improve the write performance of the encrypted RAID?
What could be the reason of the bad performance?

Software versions:
Ubuntu 11.04
Linux 2.6.38
cryptsetup 1.1.3


Help / ideas / etc. would be very much appreciated.

Greetings,
Philipp


[1] All speed measurements were done with several runs of
dd if=/dev/zero of=DEV bs=100M count=100