From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.riseup.net (mx1.riseup.net [198.252.153.129]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.saout.de (Postfix) with ESMTPS for ; Wed, 26 Feb 2014 00:58:17 +0100 (CET) Message-ID: <530D2E10.10707@riseup.net> Date: Wed, 26 Feb 2014 10:58:08 +1100 From: "shmick@riseup.net" MIME-Version: 1.0 References: <530CD101.40101@gmx.de> In-Reply-To: <530CD101.40101@gmx.de> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: Re: [dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: =?ISO-8859-1?Q?=22C=2E_Dominik_B=F3di=22?= , dm-crypt@saout.de "C. Dominik B=F3di": > (please CC me, I'm not subscribed to the list) >=20 > Hello, >=20 > I've discovered some strange behaviour on my Debian system, with > dm-crypt on raid 5 showing slow performance and I am seeking ways to > remedy that. >=20 > My setup: > Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI) > Intel X58 chipset > Debian sid/unstable > kernels tried: 3.12.9, 3.13.4 (Debian kernels) > 4 x 3 TB SATA disks on a Intel X58 AHCI controller. > That controller has 6 SATA ports and connects directly to the chipsets > South Bridge, which connects to the North Bridge via a very fast DMI link. > Second PCIe-SATA controller on motherboard (ACHI driver, probably > JMicron chipset) - 2 SATA ports. >=20 > On that disk I have a RAID5 array (source media are the 4th partition on > each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS > volume. That LUKS volume is a PV for LVM. >=20 > 4 disks =3D> RAID5 =3D> LUKS Volume =3D> LVM PV >=20 > Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read > speed. Each single disk has about 170MB/s with hdparm, so that is ok. >=20 > However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s. > Performing dd tests on the mounted LVM filesystems gets me about 160MB/s. *only* 150 ? there's lots of numbers here but maybe we should work backwards and KIS ask yourself what do you want to achieve ? when you know the results you want and/or need (2 different things) you can work backwards & figure out out how to get there do you *need* to have a certain read/write speed for your application or desired conclusion in mind ? do you have a recent version of cryptsetup with the benchmark option ? that can give you an idea of in memory performance and depends on your CPU & chosen ciphers/hashes i didn't quite get if your RAID is a hardware or software array ? >=20 > Now I connected two of the four disks to the second SATA controller >=20 > hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the > RAID5 array dropped to ca. 400MB/s. That was to be expected, as that > second controller seems somewhat slower. >=20 > And now comes the funny part: > Now I did a hdparm -t /dev/mapper/encrypted_raid5 > and that got me ca. 300MB/s >=20 > Running top whilst doing the hdparm tests showed me that, when all 4 > disks were on the Intel X58 AHCI controller, only 1 kworker thread was > running on a single CPU core, utilizing 100% CPU time. >=20 > When 2 of those disks were connected to the second SATA controller, 2 > kworker threads were running on two different CPU cores. >=20 > It seems that when the disks sit on two different controllers, dm-crypt > is using 2 kworker threads to handle the data, but uses only one kworker > thread when all disks sit on a single controller. i'll leave that to the experts ;-) >=20 > My guess is that one single cpu core can decrypt/encrypt the LUKS volume > at about 150MB/s, which seems to be reasonable for that kind of CPU. > Thus, dm-crypt using more than one kworker threads leads to doubling of > the decryption speed, even if the two disks are slightly slower when > connected to the second controller. >=20 > I'd like dm-crypt to use as many threads as possible (e.g. one kworker > for each disk in the RAID5 array). That would give me reasonable speeds > for the encrypted LUKS volume, close to that of the unencrypted RAID5 > volume. >=20 > I've experimented with the RAID5 arrays group_thread_cnt sysfs setting. > That is set to zero by default, disabling multithreaded RAID5. However, > enabling that and setting it to 2 or 4 did not change anything speed-wise. >=20 > I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity > setting, yet. Would that help? >=20 > Is there a way to enforce dm-crypt using more than one kworker thread to > encrypt/decrypt data for a crypt-on-raid5 setup? >=20 > Regards, > Dominik > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt >=20