Re: [dm-crypt] dm-crypt + mdadm

[Arno Wagner <arno@wagner.name>]

On Fri, Nov 22, 2013 at 17:29:57 CET, Adam Boyhan wrote:
> Doing allot of testing with dm-crypt and mdadm. Running into an issues
> which is killing me.  We run a standard CentOS6 load with kernel
> "2.6.32-358.23.2.el6.x86_64"
> 
> Scenario #1: mdadm raid10 -> dm-crypt -> ext4 
> Scenario #2: dm-crypt -> mdadm raid10 -> ext4 
> 
> Scenario #1 has no issues and runs reliably but takes a significant hit in
> performance due to the single threaded nature of kryptd.  I read quite a
> bit about people instead encrypting each block device then putting
> software encryption on that.  This all goes well until I start to
> benchmark.  The writing process of the benchmark goes well, as soon as I
> hit the read portion of the test, the machine panics and locks up.  I
> initially tested this idea with raid0 and I don't have this panic, it
> seems that only raid10 causes the panic.  At this point I am stumped as to
> what I can do.  I am feel this is more an issue with mdadm than dm-crypt
> but I figured it was worth getting others opinions.  I appreciate any or
> all help, if I am missing any details let me know and I will provide them.

I have Scenario #1 running for years with a 3-way RAID1 (notebook drives
give me about 1 firmware crash/year, hence the 3-way...)

I do not have a performance hit. Writing runs at a sustained speed
of ~75MB/s with 3 "kworker"s at about 15% CPU each or 1 kworker at 45%
(it varies between the two states). Reading is similar. CPU is a
Phenom II 945 4-core. That speed is about what the disks can handle 
with ext3, non-encrypted gives about the same speed. (Not using ext4, 
still too new for me.) Kernel is a self-compiled stock kernel.org 
kernel 3.10.17.

I would suggest that 2.6.32 is at best called "historic" and entirely
unsuitable for any kind of performance-critical set-up. I also
strongly suspect (from some evaluation work on CentOS I did recently
for a customer) that Red-Hat has messed up updating their own
kernel and you may do better with a self-compiled 2.6.32.61.

The real benchmark would be with something like 3.10.20 or the like,
or maybe 3.12.1. (Careful: In my setup, 3.10.19 has a 
networking issue, but 3.10.17 has been running stable for some
time, so I would recommend that one.)

My bottom line is that
a) I do not trust the CentOS "stable" kernel one bit. 
b) 2.6.32 is so old, doing anything on it to improve
   performance is probably a waste of time. 2.6.32 is 4 years  
   old by now and the 2.6.x line is 10 years old...

Arno
-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
There are two ways of constructing a software design: One way is to make it
so simple that there are obviously no deficiencies, and the other way is to
make it so complicated that there are no obvious deficiencies. The first
method is far more difficult.  --Tony Hoare