Re: [dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

["shmick@riseup.net" <shmick@riseup.net>]

"C. Dominik Bódi":
> (please CC me, I'm not subscribed to the list)
> 
> Hello,
> 
> I've discovered some strange behaviour on my Debian system, with
> dm-crypt on raid 5 showing slow performance and I am seeking ways to
> remedy that.
> 
> My setup:
> Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI)
> Intel X58 chipset
> Debian sid/unstable
> kernels tried: 3.12.9, 3.13.4 (Debian kernels)
> 4 x 3 TB SATA disks on a Intel X58 AHCI controller.
> That controller has 6 SATA ports and connects directly to the chipsets
> South Bridge, which connects to the North Bridge via a very fast DMI link.
> Second PCIe-SATA controller on motherboard (ACHI driver, probably
> JMicron chipset) - 2 SATA ports.
> 
> On that disk I have a RAID5 array (source media are the 4th partition on
> each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS
> volume. That LUKS volume is a PV for LVM.
> 
> 4 disks => RAID5 => LUKS Volume => LVM PV
> 
> Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read
> speed. Each single disk has about 170MB/s with hdparm, so that is ok.
> 
> However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s.
> Performing dd tests on the mounted LVM filesystems gets me about 160MB/s.

*only* 150 ?
there's lots of numbers here but maybe we should work backwards and KIS
ask yourself what do you want to achieve ?
when you know the results you want and/or need (2 different things) you
can work backwards & figure out out how to get there
do you *need* to have a certain read/write speed for your application or
desired conclusion in mind ?

do you have a recent version of cryptsetup with the benchmark option ?
that can give you an idea of in memory performance and depends on your
CPU & chosen ciphers/hashes

i didn't quite get if your RAID is a hardware or software array ?

> 
> Now I connected two of the four disks to the second SATA controller
> 
> hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the
> RAID5 array dropped to ca. 400MB/s. That was to be expected, as that
> second controller seems somewhat slower.
> 
> And now comes the funny part:
> Now I did a hdparm -t /dev/mapper/encrypted_raid5
> and that got me ca. 300MB/s
> 
> Running top whilst doing the hdparm tests showed me that, when all 4
> disks were on the Intel X58 AHCI controller, only 1 kworker thread was
> running on a single CPU core, utilizing 100% CPU time.
> 
> When 2 of those disks were connected to the second SATA controller, 2
> kworker threads were running on two different CPU cores.
> 
> It seems that when the disks sit on two different controllers, dm-crypt
> is using 2 kworker threads to handle the data, but uses only one kworker
> thread when all disks sit on a single controller.

i'll leave that to the experts ;-)

> 
> My guess is that one single cpu core can decrypt/encrypt the LUKS volume
> at about 150MB/s, which seems to be reasonable for that kind of CPU.
> Thus, dm-crypt using more than one kworker threads leads to doubling of
> the decryption speed, even if the two disks are slightly slower when
> connected to the second controller.
> 
> I'd like dm-crypt to use as many threads as possible (e.g. one kworker
> for each disk in the RAID5 array). That would give me reasonable speeds
> for the encrypted LUKS volume, close to that of the unencrypted RAID5
> volume.
> 
> I've experimented with the RAID5 arrays group_thread_cnt sysfs setting.
> That is set to zero by default, disabling multithreaded RAID5. However,
> enabling that and setting it to 2 or 4 did not change anything speed-wise.
> 
> I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity
> setting, yet. Would that help?
> 
> Is there a way to enforce dm-crypt using more than one kworker thread to
> encrypt/decrypt data for a crypt-on-raid5 setup?
> 
> Regards,
> Dominik
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
>