[dm-crypt] Slow I/O with LUKS on amd64

[dm-crypt] Slow I/O with LUKS on amd64

[Thomas Damgaard]

Hi

I have recently switched from i386 to amd64. After switching to 64
bit, I/O on LUKS encrypted devices became horribly slow.
Do you have any idea what causes this? Or suggestions on how to debug
this further?

I have submitted a bug report on this on Launchpad. The report
includes details on performance, hardware, etc.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/731340


Regards,
Thomas

Re: [dm-crypt] Slow I/O with LUKS on amd64

[Milan Broz]

On 03/09/2011 09:11 AM, Thomas Damgaard wrote:
> I have recently switched from i386 to amd64. After switching to 64
> bit, I/O on LUKS encrypted devices became horribly slow.
> Do you have any idea what causes this? Or suggestions on how to debug
> this further?
> 
> I have submitted a bug report on this on Launchpad. The report
> includes details on performance, hardware, etc.

Well, seems you already proved the it is Ubuntu kernel only problem.

So generic suggestions - be sure that

- you are always using the same encryption parameters (cipher, key size)
(once device is unlocked, performance is kernel only thing, so this cannot
be cryptsetup/LUKS problem)

- if you are testing fs over dmcrypt, be sure the same fs with the *same*
mount parameters is used. If using barriers switch them off for test.
(barrier=0). Verify with dmesg log.

- check which kernel cryptoAPI module is used for the encryption
(kernel should load arch sepcific one, e.g. aes_x86_64 vs aes_i586, not
generic one). usually simple lsmod of loaded modules here gives you hint.
Try blacklist that module so generic one is used etc.

- try some new kernel, I think even Ubuntu provides upstream snapshots

There is no reason x86_64 should be slower - many people it using, even on the
same hw you have.

Milan

Re: [dm-crypt] Slow I/O with LUKS on amd64

[Carlo Wood]

On Wed, 09 Mar 2011 11:21:53 +0100
Milan Broz <mbroz@redhat.com> wrote:

Actually, I have been trying to use dmcrypt for my swap
(anything that is swapped HAS to be encrypted, of course,
because it can be anything).

However, while the disk benchmarked gives me the normal
50 MB/s for that disk/partition, reading/writing the swap
with encryption only gives me around 1 MB/s. This was
so unacceptable that I turned off swap all together:
I rather crash when running out of memory than having to
wait minutes every moment of the day (and for some reason
linux likes to use the swap a lot, even while normally
it is not running out of memory).

I have an Intel Core 2(tm) cpu (a Quad Core Extreme QX6700),
but also x86_64; stock debian kernel.

> On 03/09/2011 09:11 AM, Thomas Damgaard wrote:
> > I have recently switched from i386 to amd64. After switching to 64
> > bit, I/O on LUKS encrypted devices became horribly slow.
> > Do you have any idea what causes this? Or suggestions on how to
> > debug this further?
> > 
> > I have submitted a bug report on this on Launchpad. The report
> > includes details on performance, hardware, etc.
> 
> Well, seems you already proved the it is Ubuntu kernel only problem.
> 
> So generic suggestions - be sure that
> 
> - you are always using the same encryption parameters (cipher, key
> size) (once device is unlocked, performance is kernel only thing, so
> this cannot be cryptsetup/LUKS problem)
> 
> - if you are testing fs over dmcrypt, be sure the same fs with the
> *same* mount parameters is used. If using barriers switch them off
> for test. (barrier=0). Verify with dmesg log.
> 
> - check which kernel cryptoAPI module is used for the encryption
> (kernel should load arch sepcific one, e.g. aes_x86_64 vs aes_i586,
> not generic one). usually simple lsmod of loaded modules here gives
> you hint. Try blacklist that module so generic one is used etc.
> 
> - try some new kernel, I think even Ubuntu provides upstream snapshots
> 
> There is no reason x86_64 should be slower - many people it using,
> even on the same hw you have.
> 
> Milan
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt



-- 
Carlo Wood <carlo@alinoe.com>

Re: [dm-crypt] Slow I/O with LUKS on amd64

[Milan Broz]

On 03/09/2011 03:38 PM, Carlo Wood wrote:
> On Wed, 09 Mar 2011 11:21:53 +0100
> Milan Broz <mbroz@redhat.com> wrote:
> 
> Actually, I have been trying to use dmcrypt for my swap
> (anything that is swapped HAS to be encrypted, of course,
> because it can be anything).
> 
> However, while the disk benchmarked gives me the normal
> 50 MB/s for that disk/partition, reading/writing the swap
> with encryption only gives me around 1 MB/s. This was
> so unacceptable that I turned off swap all together:
> I rather crash when running out of memory than having to
> wait minutes every moment of the day (and for some reason
> linux likes to use the swap a lot, even while normally
> it is not running out of memory).
> 
> I have an Intel Core 2(tm) cpu (a Quad Core Extreme QX6700),
> but also x86_64; stock debian kernel.

It could be another problem with swap. I remeber such
reports some time ago, maybe it is back.

Anyway, I am using Debian on x86_64 as well (but with recompiled
upstream kernel) - all above ext3/ext4/LVM over dmcrypt
for my stable machine and see no such problems.

How did you test swap device access?
If you can reproduce it, please provide more info.

See for example this my system (Debian testing, except upstream kernel,
dmcrypt uses AES-NI), just plain dd from swap volume over LUKS:

# echo 3 > /proc/sys/vm/drop_caches

# dd if=/dev/notebook/swap of=/dev/null bs=4k
1024000+0 records in
1024000+0 records out
4194304000 bytes (4.2 GB) copied, 42.0784 s, 99.7 MB/s

# lsblk /dev/sda2
NAME                       MAJ:MIN RM   SIZE RO MOUNTPOINT TYPE
sda2                         8:2    0  87.8G  0            part
└─sda2_crypt (dm-0)        253:0    0  87.8G  0            CRYPT
  ├─notebook-data (dm-1)   253:1    0  68.3G  0 /home      LVM
  ├─notebook-swap (dm-2)   253:2    0   3.9G  0 [SWAP]     LVM
  └─notebook-root64 (dm-3) 253:3    0  15.6G  0 /          LVM

(lsblk is from new util-linux, I am just testing some new things here)

Milan

Re: [dm-crypt] Slow I/O with LUKS on amd64

[Thomas Damgaard]

2011/3/9 Milan Broz <mbroz@redhat.com>:
>> Could you try to see if you experience a significant performance
>> penalty on I/O if you boot the default Debian provided kernel?
>
> seems my Xorg does not work here, but I just tried it from console:
>
> # dd if=/dev/notebook/swap of=/dev/null bs=4k
> 1024000+0 records in
> 1024000+0 records out
> 4194304000 bytes (4.2 GB) copied, 40.7913 s, 103 MB/s

This test measures how fast one single file can be read. Maybe I
forgot to clarify that sequential read of one big file is *not* slow
on neither amd64 nor i386 kernels. However, random file access is very
slow. A quick way of measuring this is to do a find inside a folder
containing a large number of files and subfolders. For example: 'time
find /usr | wc -l'.
Will you try to see how long the following command takes on your
system with both kernels?
time find /usr |wc -l
Is there any noticable difference?

Just to be clear: sequential reads of large files is *not* horribly
slow on my 64 bit kernel neither. Here is a test:
~ # dd if=/home/tdn/Download/KNOPPIX_V6.4.4CD-2011-01-30-EN.iso
of=/dev/null
1430228+0 records in
1430228+0 records out
732276736 bytes (732 MB) copied, 9,06719 s, 80,8 MB/s

Here is my result of the above find-test:

~ $ time find /usr |wc -l
find /usr  0,26s user 1,56s system 0% cpu 3:42,68 total
wc -l  0,03s user 0,04s system 0% cpu 3:42,68 total


PS.
I am sorry that my last reply was made directly to you and not the
mailing list. This was a mistake. This mail is sent to the mailing
list.

Med venlig hilsen/Kind regards
Thomas Damgaard Nielsen
http://thomasdamgaard.dk

Re: [dm-crypt] Slow I/O with LUKS on amd64

[Hanno Foest]

On Wed Mar 9 09:11:21 CET 2011, Thomas Damgaard wrote:

> I have recently switched from i386 to amd64. After switching to 64
> bit, I/O on LUKS encrypted devices became horribly slow.

I had the same issue with Ubuntu 10.10. After installing a kernel from
the upcoming 11.04, doing a 'time /usr | wc -l' went from 3 minutes to 6
seconds. And yes, I dropped my caches. :)

I installed the following image, BTW:
//launchpadlibrarian.net/65900455/linux-image-2.6.38-6-generic_2.6.38-6.34_amd64.deb

Hanno