md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

[Marc MERLIN]

I have a 5 drive md array with dmcrypt on top, and btrfs on top of that.
Kernel: 4.4

Honestly, the performance is not good, but it's hard to pin down what
exactly is to blame (there could be performance issues with btrfs too).
I have read that dmcrypt is supposed to be multithreaded, and that's
been true for multiple years.

I put dmcrypt on top so that doing a raid sync or rebuild does not
involve encryption/decryption.
But can someone confirm that indeed if I have a single md device, and 4
CPUs, my throughput is not limited by a single CPU decrypting the
device?

And if I'm going to recreate the array (using bcache this time), is
there a better way to recreate it to avoid performance issues?
It is an array with many small files (filesystem backups), so lots of
random non contiguous I/O

Currently, I have:
gargamel:~# mdadm --detail /dev/md8
/dev/md8:   
        Version : 1.2
  Creation Time : Sat Apr 19 23:03:59 2014
     Raid Level : raid5
     Array Size : 7813523456 (7451.56 GiB 8001.05 GB)
  Used Dev Size : 1953380864 (1862.89 GiB 2000.26 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Feb 11 08:26:45 2016
          State : active 
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 256K

gargamel:~# cryptsetup luksDump /dev/md8
LUKS header information for /dev/md8

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 3072
MK bits:        256

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

[Andreas Klauer]

> gargamel:~# cryptsetup luksDump /dev/md8
> LUKS header information for /dev/md8
> 
> Version:        1
> Cipher name:    aes
> Cipher mode:    xts-plain64
> Hash spec:      sha1
> Payload offset: 3072
> MK bits:        256

Does the box have AES-NI? What's your 'cryptsetup benchmark' look like?
Sometimes there can be a problem if the AES-NI module is loaded too late.
Without AES-NI your performance will suffer either way... 

You probably don't want encryption below the RAID; that would mean 
encrypting redundancy and parity so it's even more work to do, doubtful 
whether multicore CPU can offset that to make it worth it. Maybe if 
it's a NAS that has nothing else to do...

Regards
Andreas

Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

[Marc MERLIN]

On Thu, Feb 11, 2016 at 06:13:40PM +0100, Andreas Klauer wrote:
> > gargamel:~# cryptsetup luksDump /dev/md8
> > LUKS header information for /dev/md8
> > 
> > Version:        1
> > Cipher name:    aes
> > Cipher mode:    xts-plain64
> > Hash spec:      sha1
> > Payload offset: 3072
> > MK bits:        256
> 
> Does the box have AES-NI? What's your 'cryptsetup benchmark' look like?
> Sometimes there can be a problem if the AES-NI module is loaded too late.
> Without AES-NI your performance will suffer either way... 

It's a quad core HT CPU
model name      : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
 
As far as I can tell, AES-NI is working:

gargamel:~# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       420102 iterations per second
PBKDF2-sha256     250137 iterations per second
PBKDF2-sha512      87148 iterations per second
PBKDF2-ripemd160  394795 iterations per second
PBKDF2-whirlpool  125068 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b     1.2 MiB/s  1939.8 MiB/s
 serpent-cbc   128b    29.8 MiB/s   284.4 MiB/s
 twofish-cbc   128b    77.0 MiB/s   339.0 MiB/s
     aes-cbc   256b   451.2 MiB/s  1491.7 MiB/s
 serpent-cbc   256b    85.7 MiB/s   286.7 MiB/s
 twofish-cbc   256b   188.7 MiB/s   358.4 MiB/s
     aes-xts   256b  1572.1 MiB/s  1725.3 MiB/s
 serpent-xts   256b   272.8 MiB/s   291.2 MiB/s
 twofish-xts   256b   289.8 MiB/s   331.7 MiB/s
     aes-xts   512b  1355.1 MiB/s  1385.5 MiB/s
 serpent-xts   512b   318.4 MiB/s   299.8 MiB/s
 twofish-xts   512b   326.6 MiB/s   336.3 MiB/s

> You probably don't want encryption below the RAID; that would mean 
> encrypting redundancy and parity so it's even more work to do, doubtful 
> whether multicore CPU can offset that to make it worth it. Maybe if 
> it's a NAS that has nothing else to do...

It does other work, and I agree that encryption below the raid doesn't
sound like a great idea, which is why I haven't used it so far.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

[Mikael Abrahamsson]

On Thu, 11 Feb 2016, Marc MERLIN wrote:

> But can someone confirm that indeed if I have a single md device, and 4 
> CPUs, my throughput is not limited by a single CPU decrypting the 
> device?

There have been multiple discussions about this over time, and since you 
didn't qualify "not good" with a number, it's hard to say if this is your 
problem or not.

If you're getting around 100-300 megabyte/s or something, and "top" shows 
high CPU utilization for a few processes and some cores idling (use "1" 
and "H" option after starting top to see individual cores), and "iostat -x 
5" shows the drives as not topping out, then most likely you're indeed 
being hit by dmcrypt not being multithreaded and/or AES-NI not working.

I personally run the same kind of setup as you, I use md->dmcrypt->lvm->fs 
as stack. Performance isn't stellar (I just tried and I get 500 megabyte/s 
read on a machine I am pretty sure AES-NI is working from the start, but 
on another machine where I think AES-NI probably isn't loaded early 
enough, I only got 250 megabyte/s).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

[Jes Sorensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:
> On Thu, 11 Feb 2016, Marc MERLIN wrote:
>
>> But can someone confirm that indeed if I have a single md device,
>> and 4 CPUs, my throughput is not limited by a single CPU decrypting
>> the device?
>
> There have been multiple discussions about this over time, and since
> you didn't qualify "not good" with a number, it's hard to say if this
> is your problem or not.

About a year ago when I last received a new laptop, I put btrfs on top
of luks - it was a rather miserable experience. The system would freeze
up and be really slow.

I believe the issue was btrfs not acting well with crypto underneath it,
and there was hallway talk confirming this but I have no hard data. In
the end I gave up and put my /home on xfs on top of LUKS and used btrfs
for the uncrypted portions.

It may be the btrfs with crypto issues have been resolved since then.

Cheers,
Jes

Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

[Marc MERLIN]

On Fri, Feb 12, 2016 at 09:34:59AM -0500, Jes Sorensen wrote:
> Mikael Abrahamsson <swmike@swm.pp.se> writes:
> > On Thu, 11 Feb 2016, Marc MERLIN wrote:
> >
> >> But can someone confirm that indeed if I have a single md device,
> >> and 4 CPUs, my throughput is not limited by a single CPU decrypting
> >> the device?
> >
> > There have been multiple discussions about this over time, and since
> > you didn't qualify "not good" with a number, it's hard to say if this
> > is your problem or not.
> 
> About a year ago when I last received a new laptop, I put btrfs on top
> of luks - it was a rather miserable experience. The system would freeze
> up and be really slow.
> 
> I believe the issue was btrfs not acting well with crypto underneath it,
> and there was hallway talk confirming this but I have no hard data. In
> the end I gave up and put my /home on xfs on top of LUKS and used btrfs
> for the uncrypted portions.
> 
> It may be the btrfs with crypto issues have been resolved since then.

Thanks all for the feedback and experiences.

"not good" is indeed subjective, I can see the kernel hanging for a few
seconds at a time under heavy I/O and cron jobs taking too long.

I'm indeed going to just rebuild the FS with a new layout, and try
bcache in the process.

I still need to figure out what I should use as best chunk size for
mdadm, whether this is reasonable:
cryptsetup luksFormat --align-payload=8192 -s 256 -c aes-xts-plain                                     
and see if there are tuning parameters in bcache I should worry about.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

[John Stoffel]

Marc,

Did you look into lvcache instead of bcache?  The problem with bcache
is that you can't (as I understand it) remove it from a live volume,
while you can with lvache.

John

Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?

[Marc MERLIN]

On Fri, Feb 12, 2016 at 02:57:02PM -0500, John Stoffel wrote:
> 
> Marc,
> 
> Did you look into lvcache instead of bcache?  The problem with bcache
> is that you can't (as I understand it) remove it from a live volume,
> while you can with lvache.

I've used LVM2 in the past, and I'm not a big fan of the extra layers
(and back then clear slowdowns).
To be honest, I won't be needing to remove a cache from a volume while
it's running, so that's not too much a concern for me.

I did a bit of reading on one vs the other, and dmcache seemed to have
more users behind it and been around longer, so I went with that.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901