Re: [dm-crypt] aes-xts-plain with aes_x86_64 makes my SSD 5x slower than my encrypted HD

[Marc MERLIN <marc@merlins.org>]

On Mon, Jul 23, 2012 at 07:15:24PM +0200, Milan Broz wrote:
> > Mmmh, I have one possible thing. I have a preempt kernel. Could that be it?
> > http://marc.merlins.org/tmp/config-3.4.4-amd64-preempt.txt
> 
> Can you send me your kernel .config then? Preempt should not be problem.
> Which kernel version? which architecture? Any additional patches over
> mainline code?

I just sent you my config, it was in the URL above :)
No patches, kernel 3.4.4 from kernel.org, see above.

I tried without preempt and with volprempt and duplicated the same slow
speeds.

> > Disk /dev/sda: 512.1 GB, 512110190592 bytes
> > 255 heads, 63 sectors/track, 62260 cylinders, total 1000215216 sectors
> > Units = sectors of 1 * 512 = 512 bytes
> > Sector size (logical/physical): 512 bytes / 512 bytes
> > I/O size (minimum/optimal): 512 bytes / 512 bytes
> > Disk identifier: 0x09aaf50a
> > 
> >    Device Boot      Start         End      Blocks   Id  System
> > /dev/sda1   *        2048      502047      250000   83  Linux
> > /dev/sda2          502048    52930847    26214400   83  Linux
> > /dev/sda3        52930848    73902367    10485760   82  Linux swap / Solaris
> > /dev/sda4        73902368  1000215215   463156424   83  Linux
> 
> Hm. sda1 is apparently aligned. But I think
> other partitions are not aligned properly.
 
Really?
I used fdisk -H32 -S32 /dev/sda as recomended on
http://www.void.gr/kargig/blog/2012/01/11/linux-ssd-partition-alignment-tips/

> No idea which block size/page size your SSD internaly use, but to be safe,
> let's assume ideal is 256KiB (= 512 * 512 byte sectors). 
> 73902368 seems not to be multiple of 512...
> 
> Well, your sda2 configuration is created with +256MB, which is SI unit,
> so not multiple of 1024 but 1000! 
 
Oh my, I can't believe they changed that (been using linux since 1993 back
then +1MB in fdisk was a real MB).

> It should be created with +256M, so you end with:
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sda1            2048      526335      262144   83  Linux
> /dev/sda2          526336     ...
> 
> But I think this is not the core problem but you should try it to fix.
> (I hope I did not mixed up numbers above :)

It would be a problem for erase blocks if it is wrong, but not for reading,
especially when I can do filesystem reads at 260MB/s on the filesystem
created on the encrypted device.

But I'm confused, I thought 
fdisk -H32 -S32
would protect me from mis-alignment and fdisk/util-linux 2.20.1 was smart
enough to use proper boundaries on its own on SSDs anyway.
 
> ... Thinking about it, we can test it without partition change (for reads).
> 
> This test is perhaps useless but maybe someone can find it useful later.
> 
> Let's map properly aligned device to sda4 (read only). Result will be just
> garbage, but it is just for speed testing.
> 
> For mapping above, properly aligned (1MiB alignment = 2048 sectors)
> should start on sector 7390412. This is +1760 sector shift for sda4.
> 
> Map 1GiB device there:
> # dmsetup create test_plain -r --table "0 2097152 linear /dev/sda4 1760"
> 
> Map null cryptsetup over it
> # echo "password" | cryptsetup create -c null test_crypt /dev/mapper/test_plain
> 
> Now repeat read dd test for /dev/mapper/test_plain and /dev/mapper/test_crypt

Same thing:
gandalfthegreat:~# dmsetup create test_plain -r --table "0 2097152 linear /dev/sda4 1760"
gandalfthegreat:~# echo "password" | cryptsetup create -c null test_crypt /dev/mapper/test_plain  
gandalfthegreat:~# dmsetup status test_crypt
0 2097152 crypt 
gandalfthegreat:~# cryptsetup status test_crypt
/dev/mapper/test_crypt is active.
  type:    PLAIN
  cipher:  cipher_null-ecb
  keysize: 0 bits
  device:  /dev/mapper/test_plain
  offset:  0 sectors
  size:    2097152 sectors
  mode:    read/write

gandalfthegreat:~# hdparm -t -T /dev/mapper/test_plain 
/dev/mapper/test_plain:
 Timing cached reads:   13978 MB in  2.00 seconds = 6997.30 MB/sec
 Timing buffered disk reads: 958 MB in  3.00 seconds = 319.00 MB/sec

gandalfthegreat:~# hdparm -t -T /dev/mapper/test_crypt 
/dev/mapper/test_crypt:
 Timing cached reads:   15434 MB in  2.00 seconds = 7725.50 MB/sec
 Timing buffered disk reads:  76 MB in  3.03 seconds =  25.05 MB/sec
gandalfthegreat:~# 
 
> > I also used:
> > cryptsetup luksFormat --align-payload=8192
> 
> It will not help if underlying partition is misaligned.
 
That's correct of course :)  
Now I'm going to have to do some more reading to see whether I'm really
misaligned or not. I had done all the relevant reading on SSDs to make sure
I did use proper alignment.
I wonder why fdisk -H32 -S32 failed me.

> I think this one, try to increase it to 8192 or so...
> 
> > nr_requests:128

Didn't help.
gandalfthegreat:/sys/block/sda/queue# echo 8192 > nr_requests
gandalfthegreat:/sys/block/sda/queue# hdparm -t -T /dev/mapper/cryptroot 

/dev/mapper/cryptroot:
 Timing cached reads:   14970 MB in  2.00 seconds = 7492.34 MB/sec
 Timing buffered disk reads:  72 MB in  3.08 seconds =  23.35 MB/sec
gandalfthegreat:/sys/block/sda/queue# 


Thanks for the other suggestions. Hopefully we'll nail this somehow :)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/