[dm-crypt] Poor write performance on software raid with 512k chunk size

[dm-crypt] Poor write performance on software raid with 512k chunk size

[Dennis Birkholz]

Hello together,

i experienced severe performance problems when using a crypt mapping on
a software raid 5 with 512k chunk size. Without a crypt mapping the
write speed was ~100M/sec, with the crypt mapping it had only ~20M/sec.
After is rebuild the raid on the same disks using 64k chunk size, the
write performance with the crypt mapping is now ~90M/sec. I am using a
self compiled 3.2.6 vanilla kernel on a debian squeeze with an AMD
Phenom(tm) II X4 955 processor.

For now reducing the chunk size solved the problem for me but 512k chunk
size was chosen as the default by mdadm so it would be nice to find the
reason for this performance bottleneck.

Greets,
Dennis

Re: [dm-crypt] Poor write performance on software raid with 512k chunk size

[Alasdair G Kergon]

As we've mentioned to others, try increasing 
   /sys/block/<devices>/queue/nr_requests
significantly (e.g. see if 50,000 has any impact).

Or experiment with these patches:
  http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/current/
(at your own risk, of course).

Alasdair

Re: [dm-crypt] Poor write performance on software raid with 512k chunk size

[Dennis Birkholz]

Thanks for the fast reply.

Am 26.02.2012 14:37, schrieb Alasdair G Kergon:
> As we've mentioned to others, try increasing 
>    /sys/block/<devices>/queue/nr_requests
> significantly (e.g. see if 50,000 has any impact).
> 
> Or experiment with these patches:
>   http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/current/
> (at your own risk, of course).

As I mentioned before, I chose a smaller chunk size for my RAID to solve
this problem. As I am not interested in rebuilding a 6TB RAID again, I
cannot verify if your suggestions increase write throughput with greater
chunk size.

I would suggest that the device mapper should set default values that
give maximum performance for the default values other kernel subsystems
use. As the RAID subsystem seems to use 512k chunks by default, the
device mapper should use settings by default that allow maximum
performance for this chunk size.

Greets,
Dennis