dm_crypt: very low performance of random IO operations

dm_crypt: very low performance of random IO operations

[Yakov Hrebtov]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

I'm experirncing problems with performance of random IO operations of 
dm_crypt-encrypted devices.
Sequential IO is good (using AES-NI gives almost no difference comparing with 
unencrypted device), but random IO is poor comparing with unencrypted devices.

For benchmarking i've used simple utility, that reading small pieces of data 
from a raw disk device, in a random access pattern. The utility measures the 
number of seeks per second (reading operations per second), the device can do.

The source code of utility is attached. Also avalilable via link 
http://www.linuxinsight.com/files/seeker.c

I've tested 2 RAID arrays:
1. RAID 10 of 4 x Intel X25-E SSD drives (/dev/sdb)
2. RAID 10 of 8 x Seagate Savvio 15K 2.5" SAS drives (/dv/sdc)

Both arrays is handled by Adaptec 5805Z RAID card.

I've created encrypted mappings with:
cryptsetup --cipher=aes-cbc-essiv:sha256 --key-size=256 create cfs1 /dev/sdb
cryptsetup --cipher=aes-cbc-essiv:sha256 --key-size=256 create cfs2 /dev/sdc

Testing results:
1. SSD array:
    /dev/sdb: 3320 seeks/second
    /dev/mapper/cfs1: 99 seeks/second
2. SAS array:
    /dev/sdc: 257 seeks/second
    /dev/mapper/cfs2: 85 seeks/second

As you see the difference between clear and encrypted devices is huge, 
especially in SSD case.

The current behavior makes almost impossible to use encrypted devices to handle 
databases. You know, the typical database workload is random IO operations.

These tests done using kernel:
2.6.32-24-server #39-Ubuntu SMP x86_64 GNU/Linux

Some months before i've investigated also other kernels and distributions. The 
behavior is very similar.
If you need any additional info, a can provide it.

What do you think about this issue?

Thank you.



[-- Attachment #2: seeker.c --]
[-- Type: image/x-xbitmap, Size: 1580 bytes --]

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: dm_crypt: very low performance of random IO operations

[Milan Broz]

On 09/03/2010 10:28 AM, Yakov Hrebtov wrote:
> I'm experirncing problems with performance of random IO operations of 
> dm_crypt-encrypted devices.
> Sequential IO is good (using AES-NI gives almost no difference comparing with 
> unencrypted device), but random IO is poor comparing with unencrypted devices.

Just info for the list - this mail is duplicate of
https://bugzilla.kernel.org/show_bug.cgi?id=17892
so let's use this BZ to track it.

I have some ideas how to improve it, but the change must follow patches which
are currently waiting for upstream inclusion.
(Actually the per-cpu thread patches helps in this case too as side effect.)

Milan