Re: [dm-crypt] slow read performance, but fast writes?

[Nikolay Kichukov <hijacker@oldum.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
this query is best suited for md-raid mailing list as what you measure:

dd if=/dev/md3 of=/dev/null bs=1M

does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there.

Cheers,
- -Nik

On 09/14/2011 07:47 PM, Peter Merhaut wrote:
> that's what i suspected, so i ran badblocks on all 6 hdds
> simultaneously, getting 100-110MB/s each(!).
> so the io subsystem is capable of ~600MB/s read, with very low iowait.
> when i start badblocks on all 6 crypto devices. the read speed drops to
> 29MB/s with high iowait.
> 
> when i start it on just one crypto device, i got 100MB/s. start a second
> badblocks, the speed on both drops to 80MB/s.
> 
> so the problem must be with dm_crypt or the way dm_crypt interacts with
> the underlying storage. maybe a problem/limitation when 6 crypto threads
> are running simultaneously?
> 
> Am 09/14/2011 05:04 PM, schrieb Arno Wagner:
>> The RAID6 calculations should not matter much. For example
>> here is the RAID6 bootup-benchmark from my elderly
>> Athlon 64 X2 5600+:
>>  ...
>>  raid6: using algorithm sse2x2 (4867 MB/s)
>>
>> What I suspect for your problem on reading is an I/O
>> chip or attachment with asymmetrical performance.
>> This can happen, e.g. when the buffering is asymetrical
>> or one direction has a long switchover time when going 
>> to/from different targets while the other does not.
>> Reads have to read one stripe in turn from each disk, 
>> (with head movement, rotational latency, etc.) while
>> writes go to 1. OS buffer and 2. disk buffer.
>>
>> So writes are in principle faster on any Unix, as they 
>> get buffered by the kernel and the disks, while reads do not. 
>> Writes used to be slower because disk heads would need to
>> be positioned a lot better, but this difference has mostly 
>> vanished in modern drives.
>>
>> Still, I suspect a hardware problem in the contoller(s)
>> that the disks attach to or their attachment to the system.
>>
>> Arno
>>
>>
>> On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote:
>>> Hi,
>>>
>>> i've got a really strange problem with my setup, and i'm not quite sure
>>> why.
>>> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted
>>> with dm_crypt, on top of the encrypted block devices, there's a raid6.
>>> Normally you would assume, that writes are slower than reads, since
>>> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to
>>> calculate the 2 parity stripes).
>>> Let's take a look on the write performance first. Here's an average of
>>> 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M"
>>>
>>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>            0.12    0.05   94.15    2.17    0.00    3.50
>>>
>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
>>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>> sdb               9.30 15200.00    1.80 2972.70    44.40 73472.40   
>>> 49.43    15.53    5.21  201.72    5.09   0.26  76.68
>>> sdd               6.30 14495.00    1.60 3759.60    31.60 73756.00   
>>> 39.24     2.91    0.77  133.44    0.71   0.19  73.08
>>> sdf               3.10 14519.30    1.70 3650.00    19.20 73429.20   
>>> 40.23     2.86    0.78  137.06    0.72   0.20  72.52
>>> sde               6.10 14240.10    1.30 3964.20    29.60 73572.40   
>>> 37.12     2.80    0.70  173.38    0.65   0.18  71.70
>>> sdg               3.10 14859.40    1.10 3306.50    16.80 73410.00   
>>> 44.40     2.72    0.82    3.45    0.82   0.19  63.06
>>> sdc               0.20 14276.20    1.00 3909.40     4.80 73450.00   
>>> 37.57     2.65    0.67   41.80    0.66   0.17  65.29
>>> md3               0.00     0.00    0.00 4609.60     0.00 294870.40  
>>> 127.94     0.00    0.00    0.00    0.00   0.00   0.00
>>>
>>> Works as expected. Since this machine is powered by an quadcore, all
>>> this crypto and raid6 threads are nicely balanced over all 4 cores.
>>> Sequential Write averages out at about 300MB/s.
>>> So the CPU is capable of encrypting 6 times 72MB per second (while
>>> calculating parity for raid6).
>>>
>>> Now for the Reads.
>>>
>>> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
>>>
>>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>            0.15    0.05   37.61   19.90    0.00   42.29
>>>
>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
>>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>> sdb            5700.20     0.00 1141.90    0.00 27507.20     0.00   
>>> 48.18     6.37    5.52    5.52    0.00   0.42  48.27
>>> sdd            5420.60     0.00 1419.80    0.00 27462.40     0.00   
>>> 38.68     4.47    3.14    3.14    0.00   0.30  42.12
>>> sdf            5646.90     0.00 1198.10    0.00 27418.00     0.00   
>>> 45.77     5.50    4.57    4.57    0.00   0.36  42.91
>>> sde            5837.10     0.00 1006.40    0.00 27507.20     0.00   
>>> 54.66     6.69    6.61    6.61    0.00   0.49  49.45
>>> sdg            5671.10     0.00 1171.30    0.00 27401.20     0.00   
>>> 46.79     4.65    3.95    3.95    0.00   0.34  39.99
>>> sdc            5994.20     0.00  851.30    0.00 27485.60     0.00   
>>> 64.57     7.11    8.27    8.27    0.00   0.57  48.44
>>> md3               0.00     0.00 40927.60    0.00 163710.40     0.00    
>>> 8.00     0.00    0.00    0.00    0.00   0.00   0.00
>>>
>>> 42% idle and almost 20% iowait?
>>> I've already tried everything i thought of. changed kernel CONFIG_HZ,
>>> tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to
>>> ide mode, tuned read ahead for harddisks, crypto-devices and raid array
>>> (with different combinations).
>>> Same thing happens if i start badblocks on all 6 crypto devices.
>>> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD
>>> 910e) about 20%, the read rate increases at exactly 20%.
>>> Any advice would be greatly appreciated.
>>>
>>> thanks and best regards, Peter
>>> _______________________________________________
>>> dm-crypt mailing list
>>> dm-crypt@saout.de
>>> http://www.saout.de/mailman/listinfo/dm-crypt
>>
> 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOccpBAAoJEDFLYVOGGjgXOMAH/24/+1Q1hIbo0bPfIyDdGums
SREqEsEz+ty/eOpaoKyjnFM0Ec0RF6ktQ5eVCQndje0SZp0R7ViQueooa8Y5GPBc
4brlPx7hJJ2MD9qsv7u5QOhT5BzClFaD891efN5O11VYuz25aioVWw6aMJlXfu4Y
Mc4lVfgpyUwMXnLijtZYXxucGcJlrdGeEq7TcZ38DmM+hhhWzYfAWIlC5YNW7Jvs
TvNUbIRe6KjHvYwyaG82n+9OgqGkcv303iNLEzqTypyVu7oBtPU1C9B7+3QEwwXn
857kyuxXmq2f1BtXJzleXknng2x3YgyqT0wmN3CK1jBVMmz87ONrkryFLBUtgCU=
=jjZC
-----END PGP SIGNATURE-----