[dm-crypt] Improving performance?

[dm-crypt] Improving performance?

[Lasse Jensen]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Hi. I have a RAID 5 array with 3 (soon upgrading to 4 + hotspare = 5)
encrypted drives connected to a system with a Core 2 Duo @ 2.5 ghz  running
Debian Squeeze.
Each drive has been formatted with

cryptsetup luksFormat /path/to/device

And put together in a array with

mdamd -C /dev/md0 --raid-level=5 /path/to/first-device /path/to/third-device
/path/to/third-device

It works great, and encrypting the devices separately allows me to run more
than one instance of kcryptd, thus using both cores in my server. It
compensates for the overhead of encrypting the checksumming data seperately,
compared to raw devices -> RAID -> encryption and still give me improved
speed.

At the moment, i get 70 mb/s sequential read speed locally. I would like to
boost it to at least 100 or even more, as 1) the raw drives support way more
and 2) i would like to fill my gigabit ethernet when copying files over the
network.

Now, what are my options?

A quadcore CPU like the Q6600 would double the number of cores and
theoretically double the throughput, but at cost of idle power. Note that
the server is idle most of the time.
A core i5. They have AES support in hardware, but it's an expensive solution
and i'm not even sure it has Linux support.
A PCI or PCIe based card, like the HiFN cards, but what card should i look
for and what speed should i expect?
Using the CUDA cores of my nVidia card, but no driver seems to exists for
that.

The first option is pretty straight forward, but what about the rest? Or are
there any other options i havent thought of?

-- 
Lasse Jensen (fafler at gmail dot com)

[-- Attachment #2: Type: text/html, Size: 1707 bytes --]

Re: [dm-crypt] Improving performance?

[Arno Wagner]

Hi Lasse,

have you done any benchmarks for kcryptd to determine the 
bottleneck is your CPU? My intuition would be that for this
setup and the stated speed it is likely, but better be sure
than to optimize nthe wrong parameter.

As to options, basically a faster CPU and/or more cores 
is one, the other is SSD, if the bottleneck is with the
disks. A third one is a different controller and/or
bus attachment of the controller. It is also possible
that a singel disk slows things down. HDDs can get donw
to 50% of the start-of-disk speed somehwere between the
50% coapacity mark and the end.

The second thing you need to ask you, is if you are
only interessded in linear read speed. If not, an
increase 70 -> 100 may well be insigificant in the
access mix you are using and not worth the investment.

Gr"usse,
Arno



On Thu, Nov 11, 2010 at 11:49:16AM +0100, Lasse Jensen wrote:
> Hi. I have a RAID 5 array with 3 (soon upgrading to 4 + hotspare = 5)
> encrypted drives connected to a system with a Core 2 Duo @ 2.5 ghz  running
> Debian Squeeze.
> Each drive has been formatted with
> 
> cryptsetup luksFormat /path/to/device
> 
> And put together in a array with
> 
> mdamd -C /dev/md0 --raid-level=5 /path/to/first-device /path/to/third-device
> /path/to/third-device
> 
> It works great, and encrypting the devices separately allows me to run more
> than one instance of kcryptd, thus using both cores in my server. It
> compensates for the overhead of encrypting the checksumming data seperately,
> compared to raw devices -> RAID -> encryption and still give me improved
> speed.
> 
> At the moment, i get 70 mb/s sequential read speed locally. I would like to
> boost it to at least 100 or even more, as 1) the raw drives support way more
> and 2) i would like to fill my gigabit ethernet when copying files over the
> network.
> 
> Now, what are my options?
> 
> A quadcore CPU like the Q6600 would double the number of cores and
> theoretically double the throughput, but at cost of idle power. Note that
> the server is idle most of the time.
> A core i5. They have AES support in hardware, but it's an expensive solution
> and i'm not even sure it has Linux support.
> A PCI or PCIe based card, like the HiFN cards, but what card should i look
> for and what speed should i expect?
> Using the CUDA cores of my nVidia card, but no driver seems to exists for
> that.
> 
> The first option is pretty straight forward, but what about the rest? Or are
> there any other options i havent thought of?
> 
> -- 
> Lasse Jensen (fafler at gmail dot com)

> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Improving performance?

[Heinz Diehl]

On 11.11.2010, Lasse Jensen wrote: 

> A core i5. They have AES support in hardware

I have an i5-450 which has not. There are different versions out there and
not all of them have hardware AES. and I for myself do not trust in such
hardware solutions either, but that's another story..

Re: [dm-crypt] Improving performance?

[Rick Moritz]

On Thu, 11 Nov 2010 18:03:39 +0100 Heinz Diehl <htd@fancy-poultry.org> wrote:

> On 11.11.2010, Lasse Jensen wrote: 
> 
> > A core i5. They have AES support in hardware
> 
> I have an i5-450 which has not. There are different versions out there and
> not all of them have hardware AES. and I for myself do not trust in such
> hardware solutions either, but that's another story..

I thought that the i5 series was comprised of the 6xx numbers, whereas a 450 sounds more like an i3, which has those features disabled.
As for trust, if we can't trust intel's hardware accelerated encryption, why should we trust any processor maker to not just poll for software AES and do mischief that way?
That's going way down the rabbit hole, and not really helpful.
Anyway, any i5 6xx (ie 32nm dual core with hyper threading and turbo) also has AES_NI exposed.

Re: [dm-crypt] Improving performance?

[Zdenek Kaspar]

Dne 11.11.2010 11:49, Lasse Jensen napsal(a):
> Now, what are my options?
> 
> A quadcore CPU like the Q6600 would double the number of cores and
> theoretically double the throughput, but at cost of idle power. Note
> that the server is idle most of the time.
> A core i5. They have AES support in hardware, but it's an expensive
> solution and i'm not even sure it has Linux support.
> A PCI or PCIe based card, like the HiFN cards, but what card should i
> look for and what speed should i expect?
> Using the CUDA cores of my nVidia card, but no driver seems to exists
> for that.

Hmm, maybe another option is change from ie: aes-cbc-essiv:sha256 to
something faster, ie: aes-xts-plain[64] ?

HTH, Z.

Re: [dm-crypt] Improving performance?

[Markus Krainz]

[-- Attachment #1: Type: text/plain, Size: 3601 bytes --]

Hi.

On 2010-11-11 11:49, Lasse Jensen wrote:
> Hi. I have a RAID 5 array with 3 (soon upgrading to 4 + hotspare = 5) 
> encrypted drives connected to a system with a Core 2 Duo @ 2.5 ghz  
> running Debian Squeeze.
> Each drive has been formatted with
>
> cryptsetup luksFormat /path/to/device
>
> And put together in a array with
>
> mdamd -C /dev/md0 --raid-level=5 /path/to/first-device 
> /path/to/third-device /path/to/third-device
>
> It works great, and encrypting the devices separately allows me to run 
> more than one instance of kcryptd, thus using both cores in my server. 
> It compensates for the overhead of encrypting the checksumming data 
> seperately, compared to raw devices -> RAID -> encryption and still 
> give me improved speed.

Have you ever done the test and unplugged a drive from your raid and 
assessed if the raid5 still works?
My experience with this setup is that cryptsetup and mdadm do not work 
well, if the underlying device suddenly disappears.

First unplug 1 of your 3 drives, look if it still works and mdadm 
recognises the missing drive using mdadm --detail /dev/raidname
Then reconnect the drive without restarting the computer simulating a 
new device to replace the old one.
Try if you can still open it with cryptsetup (using the same name).
Try if you can rebuild the array.

Could you please try it and poste the results here?


>
> At the moment, i get 70 mb/s sequential read speed locally. I would 
> like to boost it to at least 100 or even more, as 1) the raw drives 
> support way more and 2) i would like to fill my gigabit ethernet when 
> copying files over the network.
>
> Now, what are my options?
>
> A quadcore CPU like the Q6600 would double the number of cores and 
> theoretically double the throughput, but at cost of idle power. Note 
> that the server is idle most of the time.
> A core i5. They have AES support in hardware, but it's an expensive 
> solution and i'm not even sure it has Linux support.
> A PCI or PCIe based card, like the HiFN cards, but what card should i 
> look for and what speed should i expect?
> Using the CUDA cores of my nVidia card, but no driver seems to exists 
> for that.
>
> The first option is pretty straight forward, but what about the rest? 
> Or are there any other options i havent thought of?
>

I have a setup with an i5 and another one with q6600. Notice that the 
q6600 does not fit on the same motherboards as the i5.
dmcrypt/luks is used on top of the raid. The performance of the i5 is 
not great, despite hardware aes. Should not be this numbers a bit higher 
than 158 MB/sec?

~/httptunnel-3.3/ hdparm -t --direct /dev/md1

/dev/md1:
  Timing O_DIRECT disk reads:  936 MB in  3.00 seconds = 311.67 MB/sec

~/httptunnel-3.3/ hdparm -t --direct /dev/mapper/evol

/dev/mapper/evol:
  Timing O_DIRECT disk reads:  476 MB in  3.01 seconds = 158.30 MB/sec

cat /proc/cpuinfo | grep -E (model name|aes)
model name      : Intel(R) Core(TM) i5 CPU         660  @ 3.33GHz
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good 
xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx 
smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt aes lahf_lm ida 
arat dts tpr_shadow vnmi flexpriority ept vpid


Regards,
Markus Krainz

> -- 
> Lasse Jensen (fafler at gmail dot com)
>
>
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


[-- Attachment #2: Type: text/html, Size: 5272 bytes --]

Re: [dm-crypt] Improving performance?

[Lasse Jensen]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

On Thu, Nov 11, 2010 at 12:30 PM, Arno Wagner <arno@wagner.name> wrote:

> Hi Lasse,
>
> have you done any benchmarks for kcryptd to determine the
> bottleneck is your CPU? My intuition would be that for this
> setup and the stated speed it is likely, but better be sure
> than to optimize nthe wrong parameter.
>

I cant remember the numbers right now, but yes, i know for sure that my
WD15EARS drives can do a lot more than this. I think the number for a 3x 1.5
tb setup without any encryption was around 190 mb/s.

As to options, basically a faster CPU and/or more cores
> is one, the other is SSD, if the bottleneck is with the
> disks. A third one is a different controller and/or
> bus attachment of the controller. It is also possible
> that a singel disk slows things down. HDDs can get donw
> to 50% of the start-of-disk speed somehwere between the
> 50% coapacity mark and the end.
>

My need is capacity, redundancy, price, security, and speed in that order.
SSD would be way to expensive for at setup like mine. The drives are more
than able to fill up my current CPU.


> The second thing you need to ask you, is if you are
> only interessded in linear read speed. If not, an
> increase 70 -> 100 may well be insigificant in the
> access mix you are using and not worth the investment.
>

I'm mainly interested in having good speed for sequential reading speed, you
know, reading and transferring large files across the network. I rarely have
more than one or two concurrent file operations.

-- 
Lasse Jensen (fafler at gmail dot com)

[-- Attachment #2: Type: text/html, Size: 2210 bytes --]

Re: [dm-crypt] Improving performance?

[Lasse Jensen]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

On Thu, Nov 11, 2010 at 6:03 PM, Heinz Diehl <htd@fancy-poultry.org> wrote:

> On 11.11.2010, Lasse Jensen wrote:
>
> > A core i5. They have AES support in hardware
>
> I have an i5-450 which has not. There are different versions out there and
> not all of them have hardware AES. and I for myself do not trust in such
> hardware solutions either, but that's another story..
>
>
I am fully aware of that, but you have a really good point. As for trusting
the hardware implementation, i don't really know enough about this to make
sure there isn't a backdoor somewhere. I'm going to put my trust in Intel on
this one.


-- 
Lasse Jensen (fafler at gmail dot com)

[-- Attachment #2: Type: text/html, Size: 1052 bytes --]

Re: [dm-crypt] Improving performance?

[Markus Krainz]

[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]


On 2010-11-11 19:26, Lasse Jensen wrote:
> I havent tested my current setup this way, but my old setup, RAID 
> first, then encryption worked fine.

RAID first, den ecryption works for me, too.
But testing the ecrypting drives seperately, then RAID approach failed.
What good is fast performance if the RAID 5 does not work? :D

>     dmcrypt/luks is used on top of the raid. The performance of the i5
>     is not great, despite hardware aes. Should not be this numbers a
>     bit higher than 158 MB/sec?
>
>     ~/httptunnel-3.3/ hdparm -t --direct /dev/md1
>
>     /dev/md1:
>      Timing O_DIRECT disk reads:  936 MB in  3.00 seconds = 311.67 MB/sec
>
>     ~/httptunnel-3.3/ hdparm -t --direct /dev/mapper/evol
>
>     /dev/mapper/evol:
>      Timing O_DIRECT disk reads:  476 MB in  3.01 seconds = 158.30 MB/sec
>
>     cat /proc/cpuinfo | grep -E (model name|aes)
>     model name      : Intel(R) Core(TM) i5 CPU         660  @ 3.33GHz
>     flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
>     pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
>     pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts
>     rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64
>     monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2
>     popcnt aes lahf_lm ida arat dts tpr_shadow vnmi flexpriority ept vpid
>
>
> Well, it's still a lot better than my setup. Have you got any idea how 
> much power your system uses idle and under load?
>

I am afraid I cannot unplug the server right now and I do not have an 
ampere meter lying around.
But I went for a 32nm dual-core-CPU, a small motherboard with lots of 
sata plugs and an efficient power supply, so I figured you can not get 
much better power
consumption wise.

Forgot to mention, the above hdparam -t result is for cipher mode: 
xts-plain64 and AES-256.

Regards,

Markus Krainz


[-- Attachment #2: Type: text/html, Size: 3663 bytes --]

Re: [dm-crypt] Improving performance?

[epvdm]

On Thu, Nov 11, 2010 at 06:59:47PM +0100, Markus Krainz wrote:
> Hi.
> 
> On 2010-11-11 11:49, Lasse Jensen wrote:
> >Hi. I have a RAID 5 array with 3 (soon upgrading to 4 + hotspare =
> >5) encrypted drives connected to a system with a Core 2 Duo @ 2.5
> >ghz  running Debian Squeeze.

For what it's worth I use luks on top of mirrored pairs of drives and have 
frequently lost and replaced one drive of a pair without problems. 

eric

> >Each drive has been formatted with
> >
> >cryptsetup luksFormat /path/to/device
> >
> >And put together in a array with
> >
> >mdamd -C /dev/md0 --raid-level=5 /path/to/first-device
> >/path/to/third-device /path/to/third-device
> >
> >It works great, and encrypting the devices separately allows me to
> >run more than one instance of kcryptd, thus using both cores in my
> >server. It compensates for the overhead of encrypting the
> >checksumming data seperately, compared to raw devices -> RAID ->
> >encryption and still give me improved speed.
> 
> Have you ever done the test and unplugged a drive from your raid and
> assessed if the raid5 still works?
> My experience with this setup is that cryptsetup and mdadm do not
> work well, if the underlying device suddenly disappears.
> 
> First unplug 1 of your 3 drives, look if it still works and mdadm
> recognises the missing drive using mdadm --detail /dev/raidname
> Then reconnect the drive without restarting the computer simulating
> a new device to replace the old one.
> Try if you can still open it with cryptsetup (using the same name).
> Try if you can rebuild the array.
> 
> Could you please try it and poste the results here?
> 
> 
> >
> >At the moment, i get 70 mb/s sequential read speed locally. I
> >would like to boost it to at least 100 or even more, as 1) the raw
> >drives support way more and 2) i would like to fill my gigabit
> >ethernet when copying files over the network.
> >
> >Now, what are my options?
> >
> >A quadcore CPU like the Q6600 would double the number of cores and
> >theoretically double the throughput, but at cost of idle power.
> >Note that the server is idle most of the time.
> >A core i5. They have AES support in hardware, but it's an
> >expensive solution and i'm not even sure it has Linux support.
> >A PCI or PCIe based card, like the HiFN cards, but what card
> >should i look for and what speed should i expect?
> >Using the CUDA cores of my nVidia card, but no driver seems to
> >exists for that.
> >
> >The first option is pretty straight forward, but what about the
> >rest? Or are there any other options i havent thought of?
> >
> 
> I have a setup with an i5 and another one with q6600. Notice that
> the q6600 does not fit on the same motherboards as the i5.
> dmcrypt/luks is used on top of the raid. The performance of the i5
> is not great, despite hardware aes. Should not be this numbers a bit
> higher than 158 MB/sec?
> 
> ~/httptunnel-3.3/ hdparm -t --direct /dev/md1
> 
> /dev/md1:
>  Timing O_DIRECT disk reads:  936 MB in  3.00 seconds = 311.67 MB/sec
> 
> ~/httptunnel-3.3/ hdparm -t --direct /dev/mapper/evol
> 
> /dev/mapper/evol:
>  Timing O_DIRECT disk reads:  476 MB in  3.01 seconds = 158.30 MB/sec
> 
> cat /proc/cpuinfo | grep -E (model name|aes)
> model name      : Intel(R) Core(TM) i5 CPU         660  @ 3.33GHz
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
> pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
> xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl
> vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt aes
> lahf_lm ida arat dts tpr_shadow vnmi flexpriority ept vpid
> 
> 
> Regards,
> Markus Krainz
> 
> >-- 
> >Lasse Jensen (fafler at gmail dot com)
> >
> >
> >_______________________________________________
> >dm-crypt mailing list
> >dm-crypt@saout.de
> >http://www.saout.de/mailman/listinfo/dm-crypt
> 

> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] Improving performance?

[Heinz Diehl]

On 11.11.2010, Rick Moritz wrote: 

> I thought that the i5 series was comprised of the 6xx numbers, 
> whereas a 450 sounds more like an i3, which has those features disabled.

http://ark.intel.com/Product.aspx?id=49022

Re: [dm-crypt] Improving performance?

[Richard Zidlicky]

On Thu, Nov 11, 2010 at 11:49:16AM +0100, Lasse Jensen wrote:

> At the moment, i get 70 mb/s sequential read speed locally. I would like to
> boost it to at least 100 or even more, as 1) the raw drives support way more
> and 2) i would like to fill my gigabit ethernet when copying files over the
> network.
> 
> Now, what are my options?

wondering if loop-AES would be faster for you?

Richard

Re: [dm-crypt] Improving performance?

[Rick Moritz]

On Thu, 11 Nov 2010 21:59:00 +0100 Heinz Diehl <htd@fancy-poultry.org> wrote:

> On 11.11.2010, Rick Moritz wrote: 
> 
> > I thought that the i5 series was comprised of the 6xx numbers, 
> > whereas a 450 sounds more like an i3, which has those features disabled.
> 
> http://ark.intel.com/Product.aspx?id=49022

Ah, in intel mobile world, anything is possible ;)
But since the OP considered a server, I expected him to consider the 6xx series i5 desktop processors.

Re: [dm-crypt] Improving performance?

[Arno Wagner]

On Thu, Nov 11, 2010 at 07:56:30PM +0100, Markus Krainz wrote:
>
> On 2010-11-11 19:26, Lasse Jensen wrote:
>> I havent tested my current setup this way, but my old setup, RAID  
>> first, then encryption worked fine.
>
> RAID first, den ecryption works for me, too.
> But testing the ecrypting drives seperately, then RAID approach failed.
> What good is fast performance if the RAID 5 does not work? :D

I Have not tried encytion first, but I have several 3-way RAID1
with RAID first runnign without a problem for > 2 years now.
(It is partition RAID1, only 3 disks involved.)

I would imagine that encryption-first is a bit of a risk until
the kernel barriers have been cleaned up. BTW, I see significantly
better behavior with 2.6.36. Hope they manage to clean this mess 
up by 2.6.37. The kernel used to be tunable to not slow to a crawl
when writing large amounts of data onto slow devices. One of the
ugly secrets up to 2.6.35.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Improving performance?

[dave b]

FYI guys, there is a patch that hasn't yet been accepted into the
linux kernel --->
https://patchwork.kernel.org/patch/244031/

Re: [dm-crypt] Improving performance?

[Heinz Diehl]

On 12.11.2010, dave b wrote: 

> FYI guys, there is a patch that hasn't yet been accepted into the
> linux kernel --->
> https://patchwork.kernel.org/patch/244031/

Yes, we know :-)

AS far as I've read, it has one minor flaw which must be fixed before
it gets accepted and merged into the kernel. I have used it over some
months now, with good improvement and zero problems.