[dm-crypt] unusual writing speeds when benchmark speeds are high enough

[dm-crypt] unusual writing speeds when benchmark speeds are high enough

[.. ink ..]

[-- Attachment #1: Type: text/plain, Size: 2110 bytes --]

Sorry for my previous email without a title,resending the email with a
title this time.

A user of cryptsetup(through zuluCrypt) asked me a question i can not
answer and asking it here.

In his system,"cryptsetup benchmark" gives the following results:

[root <at> sheep 13] ~ > cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       341333 iterations per second
PBKDF2-sha256     193607 iterations per second
PBKDF2-sha512      64125 iterations per second
PBKDF2-ripemd160  303758 iterations per second
PBKDF2-whirlpool   64125 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b   183,6 MiB/s   215,4 MiB/s
 serpent-cbc   128b    68,0 MiB/s    68,9 MiB/s
 twofish-cbc   128b   167,7 MiB/s   177,5 MiB/s
     aes-cbc   256b   142,3 MiB/s   161,5 MiB/s
 serpent-cbc   256b    69,7 MiB/s    67,0 MiB/s
 twofish-cbc   256b   170,1 MiB/s   177,8 MiB/s
     aes-xts   256b   201,2 MiB/s   205,6 MiB/s
 serpent-xts   256b    69,7 MiB/s    67,6 MiB/s
 twofish-xts   256b   180,5 MiB/s   175,6 MiB/s
     aes-xts   512b   147,0 MiB/s   152,5 MiB/s
 serpent-xts   512b    70,5 MiB/s    67,8 MiB/s
 twofish-xts   512b   182,0 MiB/s   176,0 MiB/s

The system is decent enought.

The write speed he is getting though through the mapper is unusually
slow,the test is below:


[root <at> sheep 8] ~ > dd if=/dev/zero of=test.img bs=70M count=1
1+0 Datensätze ein
1+0 Datensätze aus
73400320 Bytes (73 MB) kopiert, 2,58497 s, 28,4 MB/s

[17.03.2013 21:50 Uhr]
[root <at> sheep 9] ~ > cryptsetup create test test.img
Passsatz eingeben:

[17.03.2013 21:51 Uhr]
[root <at> sheep 10] ~ > dd if=/dev/zero of=/dev/mapper/test bs=10M count=1
1+0 Datensätze ein
1+0 Datensätze aus
10485760 Bytes (10 MB) kopiert, 67,2014 s, 156 kB/s

[17.03.2013 21:52 Uhr]
[root <at> sheep 11] ~ > cryptsetup remove test

As the test show,He is getting 156kB/s when using dd over the encryption
mapper verses 28MB/s when using dd straight.

Where is the wrong in his system? version of cryptsetup used is 1.6.0

[-- Attachment #2: Type: text/html, Size: 2339 bytes --]

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[Matthias Schniedermeyer]

On 17.03.2013 17:36, .. ink .. wrote:
> Sorry for my previous email without a title,resending the email with a
> title this time.
> 
> A user of cryptsetup(through zuluCrypt) asked me a question i can not
> answer and asking it here.
> 
> In his system,"cryptsetup benchmark" gives the following results:
> 
> [root <at> sheep 13] ~ > cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1       341333 iterations per second
> PBKDF2-sha256     193607 iterations per second
> PBKDF2-sha512      64125 iterations per second
> PBKDF2-ripemd160  303758 iterations per second
> PBKDF2-whirlpool   64125 iterations per second
> #  Algorithm | Key |  Encryption |  Decryption
>      aes-cbc   128b   183,6 MiB/s   215,4 MiB/s
>  serpent-cbc   128b    68,0 MiB/s    68,9 MiB/s
>  twofish-cbc   128b   167,7 MiB/s   177,5 MiB/s
>      aes-cbc   256b   142,3 MiB/s   161,5 MiB/s
>  serpent-cbc   256b    69,7 MiB/s    67,0 MiB/s
>  twofish-cbc   256b   170,1 MiB/s   177,8 MiB/s
>      aes-xts   256b   201,2 MiB/s   205,6 MiB/s
>  serpent-xts   256b    69,7 MiB/s    67,6 MiB/s
>  twofish-xts   256b   180,5 MiB/s   175,6 MiB/s
>      aes-xts   512b   147,0 MiB/s   152,5 MiB/s
>  serpent-xts   512b    70,5 MiB/s    67,8 MiB/s
>  twofish-xts   512b   182,0 MiB/s   176,0 MiB/s
> 
> The system is decent enought.
> 
> The write speed he is getting though through the mapper is unusually
> slow,the test is below:
> 
> 
> [root <at> sheep 8] ~ > dd if=/dev/zero of=test.img bs=70M count=1
> 1+0 Datensätze ein
> 1+0 Datensätze aus
> 73400320 Bytes (73 MB) kopiert, 2,58497 s, 28,4 MB/s
> 
> [17.03.2013 21:50 Uhr]
> [root <at> sheep 9] ~ > cryptsetup create test test.img
> Passsatz eingeben:
> 
> [17.03.2013 21:51 Uhr]
> [root <at> sheep 10] ~ > dd if=/dev/zero of=/dev/mapper/test bs=10M count=1
> 1+0 Datensätze ein
> 1+0 Datensätze aus
> 10485760 Bytes (10 MB) kopiert, 67,2014 s, 156 kB/s
> 
> [17.03.2013 21:52 Uhr]
> [root <at> sheep 11] ~ > cryptsetup remove test
> 
> As the test show,He is getting 156kB/s when using dd over the encryption
> mapper verses 28MB/s when using dd straight.

dd was with conv=fsync?
Otherwise you are measuring buffer cache.

> Where is the wrong in his system? version of cryptsetup used is 1.6.0

I would say, there are quite a few details missing.
Hardware, Software?
28.4 MB/s also isn't what i would call fast, what is that?



-- 

Matthias

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[.. ink ..]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

I would say, there are quite a few details missing.
> Hardware, Software?
> 28.4 MB/s also isn't what i would call fast, what is that?
>
> The person responded and this is his computer spec:

intel core 2 duo e8500
2 gb ram
2x 80 GB HDD western digital 7,200 RPM
Kernel 3.2.18-pclos3.bfq

32bit os

No LVM volume or RAID stuff,the test was done from the user home directory.

What speed do you get from "dd" command?. People usually say the speeds i
post are not high enough but  i do not remember anybody saying what speeds
are normal,expected.

[-- Attachment #2: Type: text/html, Size: 902 bytes --]

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[Matthias Schniedermeyer]

On 18.03.2013 02:42, .. ink .. wrote:
> I would say, there are quite a few details missing.
> > Hardware, Software?
> > 28.4 MB/s also isn't what i would call fast, what is that?
> >
> > The person responded and this is his computer spec:
> 
> intel core 2 duo e8500
> 2 gb ram
> 2x 80 GB HDD western digital 7,200 RPM
> Kernel 3.2.18-pclos3.bfq
> 
> 32bit os
> 
> No LVM volume or RAID stuff,the test was done from the user home directory.
> 
> What speed do you get from "dd" command?. People usually say the speeds i
> post are not high enough but  i do not remember anybody saying what speeds
> are normal,expected.

My newest acquisition (got it friday), a 7200RPM HDD, does between 
190MB/s (outer rim) and 90MB/s (inner rim).
The performance i got out of it yesterday, when i tried the dd command, 
was 137MB/s (and 3.1 GB/s for the version without conv=fsync).

The HDD before that, a "green" model with around 5400 RPM, does between 
130MB/s and 60MB/s.
The best performance i usually see in reality is about 110MB/s, but it 
depends on where the files are written to and also where the source-file 
was placed on the source-HDD, so in the worst-case is about 60MB/s.

The SSD i have can reach (with encryption) about 400MB/s, but that is 
best-case performance when all moons are aligned. I haven't really 
benchmarked the real-world performance, the dd-example (with conv=fsync) 
only got 88MB/s.




-- 

Matthias

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[Heinz Diehl]

On 18.03.2013, Matthias Schniedermeyer wrote: 

> The SSD i have can reach (with encryption) about 400MB/s, but that is 
> best-case performance when all moons are aligned. I haven't really 
> benchmarked the real-world performance, the dd-example (with conv=fsync) 
> only got 88MB/s.

On all my SSD disks, cryptsetup performance is not even near that. I
have about 450 MB/s without encryption, and max. 120 MB/s with. 

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[Arno Wagner]

On Mon, Mar 18, 2013 at 11:01:48AM +0100, Heinz Diehl wrote:
> On 18.03.2013, Matthias Schniedermeyer wrote: 
> 
> > The SSD i have can reach (with encryption) about 400MB/s, but that is 
> > best-case performance when all moons are aligned. I haven't really 
> > benchmarked the real-world performance, the dd-example (with conv=fsync) 
> > only got 88MB/s.
> 
> On all my SSD disks, cryptsetup performance is not even near that. I
> have about 450 MB/s without encryption, and max. 120 MB/s with. 

Remember that you can get a crypto-only benchmark with
"cryptsetup benchmark" from cryptsetup v.1.60 onwards. 

Arno
-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[Matthias Schniedermeyer]

On 18.03.2013 11:01, Heinz Diehl wrote:
> On 18.03.2013, Matthias Schniedermeyer wrote: 
> 
> > The SSD i have can reach (with encryption) about 400MB/s, but that is 
> > best-case performance when all moons are aligned. I haven't really 
> > benchmarked the real-world performance, the dd-example (with conv=fsync) 
> > only got 88MB/s.
> 
> On all my SSD disks, cryptsetup performance is not even near that. I
> have about 450 MB/s without encryption, and max. 120 MB/s with. 

SSD performance has more 'it depends' than anything else.
It isn't funny. ;-)

But as i don't need my SSDs for write-heavy linear performance (much), 
the much better random-write rate than HDDs still make them worthwile. 
And the much better (random) read performance is the icing.


Overwriting appers to have better performance, but that is nothing i 
would do regularly.
dd if=/dev/zero of=zero bs=1M count=10k oflag=direct
10737418240 bytes (11 GB) copied, 116.629 s, 92.1 MB/s

dd if=/dev/zero of=zero bs=1M count=10k oflag=direct conv=notrunc
10737418240 bytes (11 GB) copied, 50.3905 s, 213 MB/s

dd if=/dev/zero of=zero bs=1M count=10k oflag=direct conv=notrunc
10737418240 bytes (11 GB) copied, 51.0608 s, 210 MB/s


My previous SSD with a Sandforce controller had much worse "worst case" 
performance as the Sandforce controller really doesn't like 
incompressible data and encrypted data looks like (or at least should 
look like) incompressible random-data.



-- 

Matthias

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[Arno Wagner]

On Sun, Mar 17, 2013 at 05:36:19PM -0400, .. ink .. wrote:
> Sorry for my previous email without a title,resending the email with a
> title this time.
[...]
> A user of cryptsetup(through zuluCrypt) asked me a question i can not
> answer and asking it here.
> 
> [root <at> sheep 8] ~ > dd if=/dev/zero of=test.img bs=70M count=1
> 1+0 Datensätze ein
> 1+0 Datensätze aus
> 73400320 Bytes (73 MB) kopiert, 2,58497 s, 28,4 MB/s
[...]

Is this over USB or on an USB memory-stick? If this is a
regular HDD, something seems to be slowing it down already.

Arno
-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[.. ink ..]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

I have asked the person for answers to the following questions,did i miss
any useful question? I will report the answers when i get them

what are your computer specifications?

what kernel version are you running?

are you on a 64 bit or 32 bit system?

did you try the test on your hard drive with the OS or on a USB stick?

did you run the test on an LVM volume or RAID device?

A few more tests would be useful too,what do these commands give you?

dd if=/dev/zero of=test1.img bs=10M count=1

dd if=/dev/urandom of=test2.img bs=10M count=1

[-- Attachment #2: Type: text/html, Size: 641 bytes --]

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[Milan Broz]

Please note that "benchmark" test is just appoximate.

If there is a problem with IO (storage - alignment etc)
always try to test with empty cipher - just use
-c null in luksFormat
(NOTE: this is plain copy, NO ENCRYPTION)
Available since http://code.google.com/p/cryptsetup/wiki/Cryptsetup143

This way you can check if the problem is in dmcrypt
overhead or some storage misconfiguration (data alignment etc).
(-c null will cause no kernel crypto is used at all, it is
plain data copy, just using the same dmcrypt IO path).

Also try direct io (add oflag=direct to dd) etc.
(This will eliminate page cache effect.)

In any case, try underlying device directly, then add another
layer, one per every step (LUKS, LVM, RAID, ...) so you can see where
is the problem. (See lsblk how the storage stack looks like.)

Milan

On 18.3.2013 0:19, .. ink .. wrote:
> I have asked the person for answers to the following questions,did i miss any useful question? I will report the answers when i get them
>
> what are your computer specifications?
>
> what kernel version are you running?
>
> are you on a 64 bit or 32 bit system?
>
> did you try the test on your hard drive with the OS or on a USB stick?
>
> did you run the test on an LVM volume or RAID device?
>
> A few more tests would be useful too,what do these commands give you?
>
> dd if=/dev/zero of=test1.img bs=10M count=1
>
> dd if=/dev/urandom of=test2.img bs=10M count=1
>
>
>
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
>

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[.. ink ..]

[-- Attachment #1: Type: text/plain, Size: 1839 bytes --]

On Mon, Mar 18, 2013 at 2:22 AM, Milan Broz <gmazyland@gmail.com> wrote:

> Please note that "benchmark" test is just appoximate.
>
> If there is a problem with IO (storage - alignment etc)
> always try to test with empty cipher - just use
> -c null in luksFormat
> (NOTE: this is plain copy, NO ENCRYPTION)
> Available since http://code.google.com/p/**cryptsetup/wiki/Cryptsetup143<http://code.google.com/p/cryptsetup/wiki/Cryptsetup143>
>
> This way you can check if the problem is in dmcrypt
> overhead or some storage misconfiguration (data alignment etc).
> (-c null will cause no kernel crypto is used at all, it is
> plain data copy, just using the same dmcrypt IO path).
>
> Also try direct io (add oflag=direct to dd) etc.
> (This will eliminate page cache effect.)
>
> In any case, try underlying device directly, then add another
> layer, one per every step (LUKS, LVM, RAID, ...) so you can see where
> is the problem. (See lsblk how the storage stack looks like.)
>
>
below is his new output with "null" cipher and "dd" using "oflag" option.It
looks like the problem in his system is in the crypto stuff since the speed
he is getting with "null" cipher is far greater than the speed he gets with
the default cipher.

[18.03.2013 08:49 Uhr]
[root@sheep 16] ~ > dd if=/dev/zero of=test.img  bs=80M count=1 oflag=direct
1+0 Datensätze ein
1+0 Datensätze aus
83886080 Bytes (84 MB) kopiert, 2,08489 s, 40,2 MB/s

[18.03.2013 08:50 Uhr]
[root@sheep 17] ~ > cryptsetup create -c null test test.img
Passsatz eingeben:

[18.03.2013 08:50 Uhr]
[root@sheep 18] ~ > dd if=/dev/zero of=/dev/mapper/test  bs=10M count=1
oflag=direct
1+0 Datensätze ein
1+0 Datensätze aus
10485760 Bytes (10 MB) kopiert, 0,108011 s, 97,1 MB/s

[18.03.2013 08:50 Uhr]
[root@sheep 19] ~ > cryptsetup remove test

[-- Attachment #2: Type: text/html, Size: 2392 bytes --]

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[.. ink ..]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

On Sun, Mar 17, 2013 at 7:07 PM, Arno Wagner <arno@wagner.name> wrote:

> On Sun, Mar 17, 2013 at 05:36:19PM -0400, .. ink .. wrote:
> > Sorry for my previous email without a title,resending the email with a
> > title this time.
> [...]
> > A user of cryptsetup(through zuluCrypt) asked me a question i can not
> > answer and asking it here.
> >
> > [root <at> sheep 8] ~ > dd if=/dev/zero of=test.img bs=70M count=1
> > 1+0 Datensätze ein
> > 1+0 Datensätze aus
> > 73400320 Bytes (73 MB) kopiert, 2,58497 s, 28,4 MB/s
> [...]
>
> Is this over USB or on an USB memory-stick? If this is a
> regular HDD, something seems to be slowing it down already.
>

In the interest of trying to figure out what speeds are "normal" i
conducted a survey in my distro(pclinuxos) forum and ask people to post
their "dd" commands and results can be seen from this thread:
http://www.pclinuxos.com/forum/index.php/topic,114335.0.html

If you go through the thread,you will see nobody went below 20MB/s.
Anything below this number is what i would consider "unusual",somebody went
above 1GB/s,that too i would consider "unusual".28MB/s is within expected
range and seem fast enough to me.One person had an unsually hight speed of
1GB/s,a few hit the 100MB/s mark and the majority are around 50MB/s +/-
30MB/s.

Maybe the disto is used by a bunch of people with old hardware(like me),or
maybe those who chose to participate use old hardware but these speeds are
not unsual to "real users" out there or atleast from users i am hearing
from.



<http://www.saout.de/mailman/listinfo/dm-crypt>

[-- Attachment #2: Type: text/html, Size: 2233 bytes --]

Re: [dm-crypt] unusual writing speeds when benchmark speeds are high enough

[Heinz Diehl]

On 17.03.2013, .. ink .. wrote: 

> 73400320 Bytes (73 MB) kopiert, 2,58497 s, 28,4 MB/s

Could you try a 

 blockdev --setra 8192 /dev/sda
 blockdev --setra 8192 /dev/sdb

and re-do the test?