Re: [dm-crypt] slow read performance, but fast writes?

[Peter Merhaut <petermerhaut@gmail.com>]

Hi

how does that solve my issue with e.g. the extremely slow badblocks
performance on the crypto-devices? (absolutely NO md involved!)

best regards, Peter

Am 09/15/2011 11:49 AM, schrieb Nikolay Kichukov:
> Hi,
> this query is best suited for md-raid mailing list as what you measure:
>
> dd if=/dev/md3 of=/dev/null bs=1M
>
> does not involve any decryption. You are reading directly from the md
> device and there is no matter what is actually there.
>
> Cheers,
> -Nik
>
> On 09/14/2011 07:47 PM, Peter Merhaut wrote:
> > that's what i suspected, so i ran badblocks on all 6 hdds
> > simultaneously, getting 100-110MB/s each(!).
> > so the io subsystem is capable of ~600MB/s read, with very low iowait.
> > when i start badblocks on all 6 crypto devices. the read speed drops to
> > 29MB/s with high iowait.
>
> > when i start it on just one crypto device, i got 100MB/s. start a second
> > badblocks, the speed on both drops to 80MB/s.
>
> > so the problem must be with dm_crypt or the way dm_crypt interacts with
> > the underlying storage. maybe a problem/limitation when 6 crypto threads
> > are running simultaneously?
>
> > Am 09/14/2011 05:04 PM, schrieb Arno Wagner:
> >> The RAID6 calculations should not matter much. For example
> >> here is the RAID6 bootup-benchmark from my elderly
> >> Athlon 64 X2 5600+:
> >>  ...
> >>  raid6: using algorithm sse2x2 (4867 MB/s)
> >>
> >> What I suspect for your problem on reading is an I/O
> >> chip or attachment with asymmetrical performance.
> >> This can happen, e.g. when the buffering is asymetrical
> >> or one direction has a long switchover time when going
> >> to/from different targets while the other does not.
> >> Reads have to read one stripe in turn from each disk,
> >> (with head movement, rotational latency, etc.) while
> >> writes go to 1. OS buffer and 2. disk buffer.
> >>
> >> So writes are in principle faster on any Unix, as they
> >> get buffered by the kernel and the disks, while reads do not.
> >> Writes used to be slower because disk heads would need to
> >> be positioned a lot better, but this difference has mostly
> >> vanished in modern drives.
> >>
> >> Still, I suspect a hardware problem in the contoller(s)
> >> that the disks attach to or their attachment to the system.
> >>
> >> Arno
> >>
> >>
> >> On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote:
> >>> Hi,
> >>>
> >>> i've got a really strange problem with my setup, and i'm not quite
> sure
> >>> why.
> >>> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted
> >>> with dm_crypt, on top of the encrypted block devices, there's a raid6.
> >>> Normally you would assume, that writes are slower than reads, since
> >>> dm-crypt has to encrypt 2 additional parity stripes (and md raid6
> has to
> >>> calculate the 2 parity stripes).
> >>> Let's take a look on the write performance first. Here's an average of
> >>> 10 seconds iostat while running "dd if=/dev/zero
> of=/daten/testfile bs=1M"
> >>>
> >>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
> >>>            0.12    0.05   94.15    2.17    0.00    3.50
> >>>
> >>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> >>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>> sdb               9.30 15200.00    1.80 2972.70    44.40 73472.40   
> >>> 49.43    15.53    5.21  201.72    5.09   0.26  76.68
> >>> sdd               6.30 14495.00    1.60 3759.60    31.60 73756.00   
> >>> 39.24     2.91    0.77  133.44    0.71   0.19  73.08
> >>> sdf               3.10 14519.30    1.70 3650.00    19.20 73429.20   
> >>> 40.23     2.86    0.78  137.06    0.72   0.20  72.52
> >>> sde               6.10 14240.10    1.30 3964.20    29.60 73572.40   
> >>> 37.12     2.80    0.70  173.38    0.65   0.18  71.70
> >>> sdg               3.10 14859.40    1.10 3306.50    16.80 73410.00   
> >>> 44.40     2.72    0.82    3.45    0.82   0.19  63.06
> >>> sdc               0.20 14276.20    1.00 3909.40     4.80 73450.00   
> >>> 37.57     2.65    0.67   41.80    0.66   0.17  65.29
> >>> md3               0.00     0.00    0.00 4609.60     0.00 294870.40  
> >>> 127.94     0.00    0.00    0.00    0.00   0.00   0.00
> >>>
> >>> Works as expected. Since this machine is powered by an quadcore, all
> >>> this crypto and raid6 threads are nicely balanced over all 4 cores.
> >>> Sequential Write averages out at about 300MB/s.
> >>> So the CPU is capable of encrypting 6 times 72MB per second (while
> >>> calculating parity for raid6).
> >>>
> >>> Now for the Reads.
> >>>
> >>> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
> >>>
> >>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
> >>>            0.15    0.05   37.61   19.90    0.00   42.29
> >>>
> >>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> >>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>> sdb            5700.20     0.00 1141.90    0.00 27507.20     0.00   
> >>> 48.18     6.37    5.52    5.52    0.00   0.42  48.27
> >>> sdd            5420.60     0.00 1419.80    0.00 27462.40     0.00   
> >>> 38.68     4.47    3.14    3.14    0.00   0.30  42.12
> >>> sdf            5646.90     0.00 1198.10    0.00 27418.00     0.00   
> >>> 45.77     5.50    4.57    4.57    0.00   0.36  42.91
> >>> sde            5837.10     0.00 1006.40    0.00 27507.20     0.00   
> >>> 54.66     6.69    6.61    6.61    0.00   0.49  49.45
> >>> sdg            5671.10     0.00 1171.30    0.00 27401.20     0.00   
> >>> 46.79     4.65    3.95    3.95    0.00   0.34  39.99
> >>> sdc            5994.20     0.00  851.30    0.00 27485.60     0.00   
> >>> 64.57     7.11    8.27    8.27    0.00   0.57  48.44
> >>> md3               0.00     0.00 40927.60    0.00 163710.40    
> 0.00    
> >>> 8.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>
> >>> 42% idle and almost 20% iowait?
> >>> I've already tried everything i thought of. changed kernel CONFIG_HZ,
> >>> tried a tickless Kernel, enabled/disabled ncq, changed bios from
> ahci to
> >>> ide mode, tuned read ahead for harddisks, crypto-devices and raid
> array
> >>> (with different combinations).
> >>> Same thing happens if i start badblocks on all 6 crypto devices.
> >>> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD
> >>> 910e) about 20%, the read rate increases at exactly 20%.
> >>> Any advice would be greatly appreciated.
> >>>
> >>> thanks and best regards, Peter
> >>> _______________________________________________
> >>> dm-crypt mailing list
> >>> dm-crypt@saout.de
> >>> http://www.saout.de/mailman/listinfo/dm-crypt
> >>
>
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt@saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt