Re: [dm-crypt] slow read performance, but fast writes?

[Peter Merhaut <petermerhaut@gmail.com>]

that's what i suspected, so i ran badblocks on all 6 hdds
simultaneously, getting 100-110MB/s each(!).
so the io subsystem is capable of ~600MB/s read, with very low iowait.
when i start badblocks on all 6 crypto devices. the read speed drops to
29MB/s with high iowait.

when i start it on just one crypto device, i got 100MB/s. start a second
badblocks, the speed on both drops to 80MB/s.

so the problem must be with dm_crypt or the way dm_crypt interacts with
the underlying storage. maybe a problem/limitation when 6 crypto threads
are running simultaneously?

Am 09/14/2011 05:04 PM, schrieb Arno Wagner:
> The RAID6 calculations should not matter much. For example
> here is the RAID6 bootup-benchmark from my elderly
> Athlon 64 X2 5600+:
>  ...
>  raid6: using algorithm sse2x2 (4867 MB/s)
>
> What I suspect for your problem on reading is an I/O
> chip or attachment with asymmetrical performance.
> This can happen, e.g. when the buffering is asymetrical
> or one direction has a long switchover time when going 
> to/from different targets while the other does not.
> Reads have to read one stripe in turn from each disk, 
> (with head movement, rotational latency, etc.) while
> writes go to 1. OS buffer and 2. disk buffer.
>
> So writes are in principle faster on any Unix, as they 
> get buffered by the kernel and the disks, while reads do not. 
> Writes used to be slower because disk heads would need to
> be positioned a lot better, but this difference has mostly 
> vanished in modern drives.
>
> Still, I suspect a hardware problem in the contoller(s)
> that the disks attach to or their attachment to the system.
>
> Arno
>
>
> On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote:
>> Hi,
>>
>> i've got a really strange problem with my setup, and i'm not quite sure
>> why.
>> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted
>> with dm_crypt, on top of the encrypted block devices, there's a raid6.
>> Normally you would assume, that writes are slower than reads, since
>> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to
>> calculate the 2 parity stripes).
>> Let's take a look on the write performance first. Here's an average of
>> 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M"
>>
>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>            0.12    0.05   94.15    2.17    0.00    3.50
>>
>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>> sdb               9.30 15200.00    1.80 2972.70    44.40 73472.40   
>> 49.43    15.53    5.21  201.72    5.09   0.26  76.68
>> sdd               6.30 14495.00    1.60 3759.60    31.60 73756.00   
>> 39.24     2.91    0.77  133.44    0.71   0.19  73.08
>> sdf               3.10 14519.30    1.70 3650.00    19.20 73429.20   
>> 40.23     2.86    0.78  137.06    0.72   0.20  72.52
>> sde               6.10 14240.10    1.30 3964.20    29.60 73572.40   
>> 37.12     2.80    0.70  173.38    0.65   0.18  71.70
>> sdg               3.10 14859.40    1.10 3306.50    16.80 73410.00   
>> 44.40     2.72    0.82    3.45    0.82   0.19  63.06
>> sdc               0.20 14276.20    1.00 3909.40     4.80 73450.00   
>> 37.57     2.65    0.67   41.80    0.66   0.17  65.29
>> md3               0.00     0.00    0.00 4609.60     0.00 294870.40  
>> 127.94     0.00    0.00    0.00    0.00   0.00   0.00
>>
>> Works as expected. Since this machine is powered by an quadcore, all
>> this crypto and raid6 threads are nicely balanced over all 4 cores.
>> Sequential Write averages out at about 300MB/s.
>> So the CPU is capable of encrypting 6 times 72MB per second (while
>> calculating parity for raid6).
>>
>> Now for the Reads.
>>
>> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
>>
>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>            0.15    0.05   37.61   19.90    0.00   42.29
>>
>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>> sdb            5700.20     0.00 1141.90    0.00 27507.20     0.00   
>> 48.18     6.37    5.52    5.52    0.00   0.42  48.27
>> sdd            5420.60     0.00 1419.80    0.00 27462.40     0.00   
>> 38.68     4.47    3.14    3.14    0.00   0.30  42.12
>> sdf            5646.90     0.00 1198.10    0.00 27418.00     0.00   
>> 45.77     5.50    4.57    4.57    0.00   0.36  42.91
>> sde            5837.10     0.00 1006.40    0.00 27507.20     0.00   
>> 54.66     6.69    6.61    6.61    0.00   0.49  49.45
>> sdg            5671.10     0.00 1171.30    0.00 27401.20     0.00   
>> 46.79     4.65    3.95    3.95    0.00   0.34  39.99
>> sdc            5994.20     0.00  851.30    0.00 27485.60     0.00   
>> 64.57     7.11    8.27    8.27    0.00   0.57  48.44
>> md3               0.00     0.00 40927.60    0.00 163710.40     0.00    
>> 8.00     0.00    0.00    0.00    0.00   0.00   0.00
>>
>> 42% idle and almost 20% iowait?
>> I've already tried everything i thought of. changed kernel CONFIG_HZ,
>> tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to
>> ide mode, tuned read ahead for harddisks, crypto-devices and raid array
>> (with different combinations).
>> Same thing happens if i start badblocks on all 6 crypto devices.
>> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD
>> 910e) about 20%, the read rate increases at exactly 20%.
>> Any advice would be greatly appreciated.
>>
>> thanks and best regards, Peter
>> _______________________________________________
>> dm-crypt mailing list
>> dm-crypt@saout.de
>> http://www.saout.de/mailman/listinfo/dm-crypt
>