Re: [dm-crypt] Improving performance?

[Markus Krainz <ldm@gmx.at>]

[-- Attachment #1: Type: text/plain, Size: 3601 bytes --]

Hi.

On 2010-11-11 11:49, Lasse Jensen wrote:
> Hi. I have a RAID 5 array with 3 (soon upgrading to 4 + hotspare = 5) 
> encrypted drives connected to a system with a Core 2 Duo @ 2.5 ghz  
> running Debian Squeeze.
> Each drive has been formatted with
>
> cryptsetup luksFormat /path/to/device
>
> And put together in a array with
>
> mdamd -C /dev/md0 --raid-level=5 /path/to/first-device 
> /path/to/third-device /path/to/third-device
>
> It works great, and encrypting the devices separately allows me to run 
> more than one instance of kcryptd, thus using both cores in my server. 
> It compensates for the overhead of encrypting the checksumming data 
> seperately, compared to raw devices -> RAID -> encryption and still 
> give me improved speed.

Have you ever done the test and unplugged a drive from your raid and 
assessed if the raid5 still works?
My experience with this setup is that cryptsetup and mdadm do not work 
well, if the underlying device suddenly disappears.

First unplug 1 of your 3 drives, look if it still works and mdadm 
recognises the missing drive using mdadm --detail /dev/raidname
Then reconnect the drive without restarting the computer simulating a 
new device to replace the old one.
Try if you can still open it with cryptsetup (using the same name).
Try if you can rebuild the array.

Could you please try it and poste the results here?


>
> At the moment, i get 70 mb/s sequential read speed locally. I would 
> like to boost it to at least 100 or even more, as 1) the raw drives 
> support way more and 2) i would like to fill my gigabit ethernet when 
> copying files over the network.
>
> Now, what are my options?
>
> A quadcore CPU like the Q6600 would double the number of cores and 
> theoretically double the throughput, but at cost of idle power. Note 
> that the server is idle most of the time.
> A core i5. They have AES support in hardware, but it's an expensive 
> solution and i'm not even sure it has Linux support.
> A PCI or PCIe based card, like the HiFN cards, but what card should i 
> look for and what speed should i expect?
> Using the CUDA cores of my nVidia card, but no driver seems to exists 
> for that.
>
> The first option is pretty straight forward, but what about the rest? 
> Or are there any other options i havent thought of?
>

I have a setup with an i5 and another one with q6600. Notice that the 
q6600 does not fit on the same motherboards as the i5.
dmcrypt/luks is used on top of the raid. The performance of the i5 is 
not great, despite hardware aes. Should not be this numbers a bit higher 
than 158 MB/sec?

~/httptunnel-3.3/ hdparm -t --direct /dev/md1

/dev/md1:
  Timing O_DIRECT disk reads:  936 MB in  3.00 seconds = 311.67 MB/sec

~/httptunnel-3.3/ hdparm -t --direct /dev/mapper/evol

/dev/mapper/evol:
  Timing O_DIRECT disk reads:  476 MB in  3.01 seconds = 158.30 MB/sec

cat /proc/cpuinfo | grep -E (model name|aes)
model name      : Intel(R) Core(TM) i5 CPU         660  @ 3.33GHz
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good 
xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx 
smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt aes lahf_lm ida 
arat dts tpr_shadow vnmi flexpriority ept vpid


Regards,
Markus Krainz

> -- 
> Lasse Jensen (fafler at gmail dot com)
>
>
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


[-- Attachment #2: Type: text/html, Size: 5272 bytes --]