* [dm-crypt] slow read performance, but fast writes?
@ 2011-09-14 14:04 Peter Merhaut
2011-09-14 15:04 ` Arno Wagner
2011-09-14 15:15 ` Milan Broz
0 siblings, 2 replies; 8+ messages in thread
From: Peter Merhaut @ 2011-09-14 14:04 UTC (permalink / raw)
To: dm-crypt
[-- Attachment #1: Type: text/plain, Size: 3643 bytes --]
Hi,
i've got a really strange problem with my setup, and i'm not quite sure
why.
First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted
with dm_crypt, on top of the encrypted block devices, there's a raid6.
Normally you would assume, that writes are slower than reads, since
dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to
calculate the 2 parity stripes).
Let's take a look on the write performance first. Here's an average of
10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M"
avg-cpu: %user %nice %system %iowait %steal %idle
0.12 0.05 94.15 2.17 0.00 3.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40
49.43 15.53 5.21 201.72 5.09 0.26 76.68
sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00
39.24 2.91 0.77 133.44 0.71 0.19 73.08
sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20
40.23 2.86 0.78 137.06 0.72 0.20 72.52
sde 6.10 14240.10 1.30 3964.20 29.60 73572.40
37.12 2.80 0.70 173.38 0.65 0.18 71.70
sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00
44.40 2.72 0.82 3.45 0.82 0.19 63.06
sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00
37.57 2.65 0.67 41.80 0.66 0.17 65.29
md3 0.00 0.00 0.00 4609.60 0.00 294870.40
127.94 0.00 0.00 0.00 0.00 0.00 0.00
Works as expected. Since this machine is powered by an quadcore, all
this crypto and raid6 threads are nicely balanced over all 4 cores.
Sequential Write averages out at about 300MB/s.
So the CPU is capable of encrypting 6 times 72MB per second (while
calculating parity for raid6).
Now for the Reads.
10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
avg-cpu: %user %nice %system %iowait %steal %idle
0.15 0.05 37.61 19.90 0.00 42.29
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00
48.18 6.37 5.52 5.52 0.00 0.42 48.27
sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00
38.68 4.47 3.14 3.14 0.00 0.30 42.12
sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00
45.77 5.50 4.57 4.57 0.00 0.36 42.91
sde 5837.10 0.00 1006.40 0.00 27507.20 0.00
54.66 6.69 6.61 6.61 0.00 0.49 49.45
sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00
46.79 4.65 3.95 3.95 0.00 0.34 39.99
sdc 5994.20 0.00 851.30 0.00 27485.60 0.00
64.57 7.11 8.27 8.27 0.00 0.57 48.44
md3 0.00 0.00 40927.60 0.00 163710.40 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
42% idle and almost 20% iowait?
I've already tried everything i thought of. changed kernel CONFIG_HZ,
tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to
ide mode, tuned read ahead for harddisks, crypto-devices and raid array
(with different combinations).
Same thing happens if i start badblocks on all 6 crypto devices.
Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD
910e) about 20%, the read rate increases at exactly 20%.
Any advice would be greatly appreciated.
thanks and best regards, Peter
[-- Attachment #2: Type: text/html, Size: 4828 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [dm-crypt] slow read performance, but fast writes? 2011-09-14 14:04 [dm-crypt] slow read performance, but fast writes? Peter Merhaut @ 2011-09-14 15:04 ` Arno Wagner 2011-09-14 16:47 ` Peter Merhaut 2011-09-14 15:15 ` Milan Broz 1 sibling, 1 reply; 8+ messages in thread From: Arno Wagner @ 2011-09-14 15:04 UTC (permalink / raw) To: dm-crypt The RAID6 calculations should not matter much. For example here is the RAID6 bootup-benchmark from my elderly Athlon 64 X2 5600+: ... raid6: using algorithm sse2x2 (4867 MB/s) What I suspect for your problem on reading is an I/O chip or attachment with asymmetrical performance. This can happen, e.g. when the buffering is asymetrical or one direction has a long switchover time when going to/from different targets while the other does not. Reads have to read one stripe in turn from each disk, (with head movement, rotational latency, etc.) while writes go to 1. OS buffer and 2. disk buffer. So writes are in principle faster on any Unix, as they get buffered by the kernel and the disks, while reads do not. Writes used to be slower because disk heads would need to be positioned a lot better, but this difference has mostly vanished in modern drives. Still, I suspect a hardware problem in the contoller(s) that the disks attach to or their attachment to the system. Arno On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote: > Hi, > > i've got a really strange problem with my setup, and i'm not quite sure > why. > First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted > with dm_crypt, on top of the encrypted block devices, there's a raid6. > Normally you would assume, that writes are slower than reads, since > dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to > calculate the 2 parity stripes). > Let's take a look on the write performance first. Here's an average of > 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M" > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.12 0.05 94.15 2.17 0.00 3.50 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 > 49.43 15.53 5.21 201.72 5.09 0.26 76.68 > sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 > 39.24 2.91 0.77 133.44 0.71 0.19 73.08 > sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 > 40.23 2.86 0.78 137.06 0.72 0.20 72.52 > sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 > 37.12 2.80 0.70 173.38 0.65 0.18 71.70 > sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 > 44.40 2.72 0.82 3.45 0.82 0.19 63.06 > sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 > 37.57 2.65 0.67 41.80 0.66 0.17 65.29 > md3 0.00 0.00 0.00 4609.60 0.00 294870.40 > 127.94 0.00 0.00 0.00 0.00 0.00 0.00 > > Works as expected. Since this machine is powered by an quadcore, all > this crypto and raid6 threads are nicely balanced over all 4 cores. > Sequential Write averages out at about 300MB/s. > So the CPU is capable of encrypting 6 times 72MB per second (while > calculating parity for raid6). > > Now for the Reads. > > 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M" > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.15 0.05 37.61 19.90 0.00 42.29 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00 > 48.18 6.37 5.52 5.52 0.00 0.42 48.27 > sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00 > 38.68 4.47 3.14 3.14 0.00 0.30 42.12 > sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00 > 45.77 5.50 4.57 4.57 0.00 0.36 42.91 > sde 5837.10 0.00 1006.40 0.00 27507.20 0.00 > 54.66 6.69 6.61 6.61 0.00 0.49 49.45 > sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00 > 46.79 4.65 3.95 3.95 0.00 0.34 39.99 > sdc 5994.20 0.00 851.30 0.00 27485.60 0.00 > 64.57 7.11 8.27 8.27 0.00 0.57 48.44 > md3 0.00 0.00 40927.60 0.00 163710.40 0.00 > 8.00 0.00 0.00 0.00 0.00 0.00 0.00 > > 42% idle and almost 20% iowait? > I've already tried everything i thought of. changed kernel CONFIG_HZ, > tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to > ide mode, tuned read ahead for harddisks, crypto-devices and raid array > (with different combinations). > Same thing happens if i start badblocks on all 6 crypto devices. > Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD > 910e) about 20%, the read rate increases at exactly 20%. > Any advice would be greatly appreciated. > > thanks and best regards, Peter > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dm-crypt] slow read performance, but fast writes? 2011-09-14 15:04 ` Arno Wagner @ 2011-09-14 16:47 ` Peter Merhaut 2011-09-15 9:49 ` Nikolay Kichukov 0 siblings, 1 reply; 8+ messages in thread From: Peter Merhaut @ 2011-09-14 16:47 UTC (permalink / raw) To: dm-crypt that's what i suspected, so i ran badblocks on all 6 hdds simultaneously, getting 100-110MB/s each(!). so the io subsystem is capable of ~600MB/s read, with very low iowait. when i start badblocks on all 6 crypto devices. the read speed drops to 29MB/s with high iowait. when i start it on just one crypto device, i got 100MB/s. start a second badblocks, the speed on both drops to 80MB/s. so the problem must be with dm_crypt or the way dm_crypt interacts with the underlying storage. maybe a problem/limitation when 6 crypto threads are running simultaneously? Am 09/14/2011 05:04 PM, schrieb Arno Wagner: > The RAID6 calculations should not matter much. For example > here is the RAID6 bootup-benchmark from my elderly > Athlon 64 X2 5600+: > ... > raid6: using algorithm sse2x2 (4867 MB/s) > > What I suspect for your problem on reading is an I/O > chip or attachment with asymmetrical performance. > This can happen, e.g. when the buffering is asymetrical > or one direction has a long switchover time when going > to/from different targets while the other does not. > Reads have to read one stripe in turn from each disk, > (with head movement, rotational latency, etc.) while > writes go to 1. OS buffer and 2. disk buffer. > > So writes are in principle faster on any Unix, as they > get buffered by the kernel and the disks, while reads do not. > Writes used to be slower because disk heads would need to > be positioned a lot better, but this difference has mostly > vanished in modern drives. > > Still, I suspect a hardware problem in the contoller(s) > that the disks attach to or their attachment to the system. > > Arno > > > On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote: >> Hi, >> >> i've got a really strange problem with my setup, and i'm not quite sure >> why. >> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted >> with dm_crypt, on top of the encrypted block devices, there's a raid6. >> Normally you would assume, that writes are slower than reads, since >> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to >> calculate the 2 parity stripes). >> Let's take a look on the write performance first. Here's an average of >> 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M" >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.12 0.05 94.15 2.17 0.00 3.50 >> >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >> avgrq-sz avgqu-sz await r_await w_await svctm %util >> sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 >> 49.43 15.53 5.21 201.72 5.09 0.26 76.68 >> sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 >> 39.24 2.91 0.77 133.44 0.71 0.19 73.08 >> sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 >> 40.23 2.86 0.78 137.06 0.72 0.20 72.52 >> sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 >> 37.12 2.80 0.70 173.38 0.65 0.18 71.70 >> sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 >> 44.40 2.72 0.82 3.45 0.82 0.19 63.06 >> sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 >> 37.57 2.65 0.67 41.80 0.66 0.17 65.29 >> md3 0.00 0.00 0.00 4609.60 0.00 294870.40 >> 127.94 0.00 0.00 0.00 0.00 0.00 0.00 >> >> Works as expected. Since this machine is powered by an quadcore, all >> this crypto and raid6 threads are nicely balanced over all 4 cores. >> Sequential Write averages out at about 300MB/s. >> So the CPU is capable of encrypting 6 times 72MB per second (while >> calculating parity for raid6). >> >> Now for the Reads. >> >> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M" >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.15 0.05 37.61 19.90 0.00 42.29 >> >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >> avgrq-sz avgqu-sz await r_await w_await svctm %util >> sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00 >> 48.18 6.37 5.52 5.52 0.00 0.42 48.27 >> sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00 >> 38.68 4.47 3.14 3.14 0.00 0.30 42.12 >> sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00 >> 45.77 5.50 4.57 4.57 0.00 0.36 42.91 >> sde 5837.10 0.00 1006.40 0.00 27507.20 0.00 >> 54.66 6.69 6.61 6.61 0.00 0.49 49.45 >> sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00 >> 46.79 4.65 3.95 3.95 0.00 0.34 39.99 >> sdc 5994.20 0.00 851.30 0.00 27485.60 0.00 >> 64.57 7.11 8.27 8.27 0.00 0.57 48.44 >> md3 0.00 0.00 40927.60 0.00 163710.40 0.00 >> 8.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >> 42% idle and almost 20% iowait? >> I've already tried everything i thought of. changed kernel CONFIG_HZ, >> tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to >> ide mode, tuned read ahead for harddisks, crypto-devices and raid array >> (with different combinations). >> Same thing happens if i start badblocks on all 6 crypto devices. >> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD >> 910e) about 20%, the read rate increases at exactly 20%. >> Any advice would be greatly appreciated. >> >> thanks and best regards, Peter >> _______________________________________________ >> dm-crypt mailing list >> dm-crypt@saout.de >> http://www.saout.de/mailman/listinfo/dm-crypt > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dm-crypt] slow read performance, but fast writes? 2011-09-14 16:47 ` Peter Merhaut @ 2011-09-15 9:49 ` Nikolay Kichukov 2011-09-15 9:53 ` Peter Merhaut 2011-09-15 10:45 ` Milan Broz 0 siblings, 2 replies; 8+ messages in thread From: Nikolay Kichukov @ 2011-09-15 9:49 UTC (permalink / raw) To: Peter Merhaut; +Cc: dm-crypt -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, this query is best suited for md-raid mailing list as what you measure: dd if=/dev/md3 of=/dev/null bs=1M does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there. Cheers, - -Nik On 09/14/2011 07:47 PM, Peter Merhaut wrote: > that's what i suspected, so i ran badblocks on all 6 hdds > simultaneously, getting 100-110MB/s each(!). > so the io subsystem is capable of ~600MB/s read, with very low iowait. > when i start badblocks on all 6 crypto devices. the read speed drops to > 29MB/s with high iowait. > > when i start it on just one crypto device, i got 100MB/s. start a second > badblocks, the speed on both drops to 80MB/s. > > so the problem must be with dm_crypt or the way dm_crypt interacts with > the underlying storage. maybe a problem/limitation when 6 crypto threads > are running simultaneously? > > Am 09/14/2011 05:04 PM, schrieb Arno Wagner: >> The RAID6 calculations should not matter much. For example >> here is the RAID6 bootup-benchmark from my elderly >> Athlon 64 X2 5600+: >> ... >> raid6: using algorithm sse2x2 (4867 MB/s) >> >> What I suspect for your problem on reading is an I/O >> chip or attachment with asymmetrical performance. >> This can happen, e.g. when the buffering is asymetrical >> or one direction has a long switchover time when going >> to/from different targets while the other does not. >> Reads have to read one stripe in turn from each disk, >> (with head movement, rotational latency, etc.) while >> writes go to 1. OS buffer and 2. disk buffer. >> >> So writes are in principle faster on any Unix, as they >> get buffered by the kernel and the disks, while reads do not. >> Writes used to be slower because disk heads would need to >> be positioned a lot better, but this difference has mostly >> vanished in modern drives. >> >> Still, I suspect a hardware problem in the contoller(s) >> that the disks attach to or their attachment to the system. >> >> Arno >> >> >> On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote: >>> Hi, >>> >>> i've got a really strange problem with my setup, and i'm not quite sure >>> why. >>> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted >>> with dm_crypt, on top of the encrypted block devices, there's a raid6. >>> Normally you would assume, that writes are slower than reads, since >>> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to >>> calculate the 2 parity stripes). >>> Let's take a look on the write performance first. Here's an average of >>> 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M" >>> >>> avg-cpu: %user %nice %system %iowait %steal %idle >>> 0.12 0.05 94.15 2.17 0.00 3.50 >>> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >>> avgrq-sz avgqu-sz await r_await w_await svctm %util >>> sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 >>> 49.43 15.53 5.21 201.72 5.09 0.26 76.68 >>> sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 >>> 39.24 2.91 0.77 133.44 0.71 0.19 73.08 >>> sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 >>> 40.23 2.86 0.78 137.06 0.72 0.20 72.52 >>> sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 >>> 37.12 2.80 0.70 173.38 0.65 0.18 71.70 >>> sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 >>> 44.40 2.72 0.82 3.45 0.82 0.19 63.06 >>> sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 >>> 37.57 2.65 0.67 41.80 0.66 0.17 65.29 >>> md3 0.00 0.00 0.00 4609.60 0.00 294870.40 >>> 127.94 0.00 0.00 0.00 0.00 0.00 0.00 >>> >>> Works as expected. Since this machine is powered by an quadcore, all >>> this crypto and raid6 threads are nicely balanced over all 4 cores. >>> Sequential Write averages out at about 300MB/s. >>> So the CPU is capable of encrypting 6 times 72MB per second (while >>> calculating parity for raid6). >>> >>> Now for the Reads. >>> >>> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M" >>> >>> avg-cpu: %user %nice %system %iowait %steal %idle >>> 0.15 0.05 37.61 19.90 0.00 42.29 >>> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >>> avgrq-sz avgqu-sz await r_await w_await svctm %util >>> sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00 >>> 48.18 6.37 5.52 5.52 0.00 0.42 48.27 >>> sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00 >>> 38.68 4.47 3.14 3.14 0.00 0.30 42.12 >>> sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00 >>> 45.77 5.50 4.57 4.57 0.00 0.36 42.91 >>> sde 5837.10 0.00 1006.40 0.00 27507.20 0.00 >>> 54.66 6.69 6.61 6.61 0.00 0.49 49.45 >>> sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00 >>> 46.79 4.65 3.95 3.95 0.00 0.34 39.99 >>> sdc 5994.20 0.00 851.30 0.00 27485.60 0.00 >>> 64.57 7.11 8.27 8.27 0.00 0.57 48.44 >>> md3 0.00 0.00 40927.60 0.00 163710.40 0.00 >>> 8.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> >>> 42% idle and almost 20% iowait? >>> I've already tried everything i thought of. changed kernel CONFIG_HZ, >>> tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to >>> ide mode, tuned read ahead for harddisks, crypto-devices and raid array >>> (with different combinations). >>> Same thing happens if i start badblocks on all 6 crypto devices. >>> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD >>> 910e) about 20%, the read rate increases at exactly 20%. >>> Any advice would be greatly appreciated. >>> >>> thanks and best regards, Peter >>> _______________________________________________ >>> dm-crypt mailing list >>> dm-crypt@saout.de >>> http://www.saout.de/mailman/listinfo/dm-crypt >> > > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJOccpBAAoJEDFLYVOGGjgXOMAH/24/+1Q1hIbo0bPfIyDdGums SREqEsEz+ty/eOpaoKyjnFM0Ec0RF6ktQ5eVCQndje0SZp0R7ViQueooa8Y5GPBc 4brlPx7hJJ2MD9qsv7u5QOhT5BzClFaD891efN5O11VYuz25aioVWw6aMJlXfu4Y Mc4lVfgpyUwMXnLijtZYXxucGcJlrdGeEq7TcZ38DmM+hhhWzYfAWIlC5YNW7Jvs TvNUbIRe6KjHvYwyaG82n+9OgqGkcv303iNLEzqTypyVu7oBtPU1C9B7+3QEwwXn 857kyuxXmq2f1BtXJzleXknng2x3YgyqT0wmN3CK1jBVMmz87ONrkryFLBUtgCU= =jjZC -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dm-crypt] slow read performance, but fast writes? 2011-09-15 9:49 ` Nikolay Kichukov @ 2011-09-15 9:53 ` Peter Merhaut 2011-09-15 10:45 ` Milan Broz 1 sibling, 0 replies; 8+ messages in thread From: Peter Merhaut @ 2011-09-15 9:53 UTC (permalink / raw) To: dm-crypt Hi how does that solve my issue with e.g. the extremely slow badblocks performance on the crypto-devices? (absolutely NO md involved!) best regards, Peter Am 09/15/2011 11:49 AM, schrieb Nikolay Kichukov: > Hi, > this query is best suited for md-raid mailing list as what you measure: > > dd if=/dev/md3 of=/dev/null bs=1M > > does not involve any decryption. You are reading directly from the md > device and there is no matter what is actually there. > > Cheers, > -Nik > > On 09/14/2011 07:47 PM, Peter Merhaut wrote: > > that's what i suspected, so i ran badblocks on all 6 hdds > > simultaneously, getting 100-110MB/s each(!). > > so the io subsystem is capable of ~600MB/s read, with very low iowait. > > when i start badblocks on all 6 crypto devices. the read speed drops to > > 29MB/s with high iowait. > > > when i start it on just one crypto device, i got 100MB/s. start a second > > badblocks, the speed on both drops to 80MB/s. > > > so the problem must be with dm_crypt or the way dm_crypt interacts with > > the underlying storage. maybe a problem/limitation when 6 crypto threads > > are running simultaneously? > > > Am 09/14/2011 05:04 PM, schrieb Arno Wagner: > >> The RAID6 calculations should not matter much. For example > >> here is the RAID6 bootup-benchmark from my elderly > >> Athlon 64 X2 5600+: > >> ... > >> raid6: using algorithm sse2x2 (4867 MB/s) > >> > >> What I suspect for your problem on reading is an I/O > >> chip or attachment with asymmetrical performance. > >> This can happen, e.g. when the buffering is asymetrical > >> or one direction has a long switchover time when going > >> to/from different targets while the other does not. > >> Reads have to read one stripe in turn from each disk, > >> (with head movement, rotational latency, etc.) while > >> writes go to 1. OS buffer and 2. disk buffer. > >> > >> So writes are in principle faster on any Unix, as they > >> get buffered by the kernel and the disks, while reads do not. > >> Writes used to be slower because disk heads would need to > >> be positioned a lot better, but this difference has mostly > >> vanished in modern drives. > >> > >> Still, I suspect a hardware problem in the contoller(s) > >> that the disks attach to or their attachment to the system. > >> > >> Arno > >> > >> > >> On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote: > >>> Hi, > >>> > >>> i've got a really strange problem with my setup, and i'm not quite > sure > >>> why. > >>> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted > >>> with dm_crypt, on top of the encrypted block devices, there's a raid6. > >>> Normally you would assume, that writes are slower than reads, since > >>> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 > has to > >>> calculate the 2 parity stripes). > >>> Let's take a look on the write performance first. Here's an average of > >>> 10 seconds iostat while running "dd if=/dev/zero > of=/daten/testfile bs=1M" > >>> > >>> avg-cpu: %user %nice %system %iowait %steal %idle > >>> 0.12 0.05 94.15 2.17 0.00 3.50 > >>> > >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > >>> avgrq-sz avgqu-sz await r_await w_await svctm %util > >>> sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 > >>> 49.43 15.53 5.21 201.72 5.09 0.26 76.68 > >>> sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 > >>> 39.24 2.91 0.77 133.44 0.71 0.19 73.08 > >>> sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 > >>> 40.23 2.86 0.78 137.06 0.72 0.20 72.52 > >>> sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 > >>> 37.12 2.80 0.70 173.38 0.65 0.18 71.70 > >>> sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 > >>> 44.40 2.72 0.82 3.45 0.82 0.19 63.06 > >>> sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 > >>> 37.57 2.65 0.67 41.80 0.66 0.17 65.29 > >>> md3 0.00 0.00 0.00 4609.60 0.00 294870.40 > >>> 127.94 0.00 0.00 0.00 0.00 0.00 0.00 > >>> > >>> Works as expected. Since this machine is powered by an quadcore, all > >>> this crypto and raid6 threads are nicely balanced over all 4 cores. > >>> Sequential Write averages out at about 300MB/s. > >>> So the CPU is capable of encrypting 6 times 72MB per second (while > >>> calculating parity for raid6). > >>> > >>> Now for the Reads. > >>> > >>> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M" > >>> > >>> avg-cpu: %user %nice %system %iowait %steal %idle > >>> 0.15 0.05 37.61 19.90 0.00 42.29 > >>> > >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > >>> avgrq-sz avgqu-sz await r_await w_await svctm %util > >>> sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00 > >>> 48.18 6.37 5.52 5.52 0.00 0.42 48.27 > >>> sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00 > >>> 38.68 4.47 3.14 3.14 0.00 0.30 42.12 > >>> sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00 > >>> 45.77 5.50 4.57 4.57 0.00 0.36 42.91 > >>> sde 5837.10 0.00 1006.40 0.00 27507.20 0.00 > >>> 54.66 6.69 6.61 6.61 0.00 0.49 49.45 > >>> sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00 > >>> 46.79 4.65 3.95 3.95 0.00 0.34 39.99 > >>> sdc 5994.20 0.00 851.30 0.00 27485.60 0.00 > >>> 64.57 7.11 8.27 8.27 0.00 0.57 48.44 > >>> md3 0.00 0.00 40927.60 0.00 163710.40 > 0.00 > >>> 8.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>> > >>> 42% idle and almost 20% iowait? > >>> I've already tried everything i thought of. changed kernel CONFIG_HZ, > >>> tried a tickless Kernel, enabled/disabled ncq, changed bios from > ahci to > >>> ide mode, tuned read ahead for harddisks, crypto-devices and raid > array > >>> (with different combinations). > >>> Same thing happens if i start badblocks on all 6 crypto devices. > >>> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD > >>> 910e) about 20%, the read rate increases at exactly 20%. > >>> Any advice would be greatly appreciated. > >>> > >>> thanks and best regards, Peter > >>> _______________________________________________ > >>> dm-crypt mailing list > >>> dm-crypt@saout.de > >>> http://www.saout.de/mailman/listinfo/dm-crypt > >> > > > _______________________________________________ > > dm-crypt mailing list > > dm-crypt@saout.de > > http://www.saout.de/mailman/listinfo/dm-crypt ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dm-crypt] slow read performance, but fast writes? 2011-09-15 9:49 ` Nikolay Kichukov 2011-09-15 9:53 ` Peter Merhaut @ 2011-09-15 10:45 ` Milan Broz 2011-09-15 11:07 ` Peter Merhaut 1 sibling, 1 reply; 8+ messages in thread From: Milan Broz @ 2011-09-15 10:45 UTC (permalink / raw) To: Nikolay Kichukov; +Cc: dm-crypt, Peter Merhaut On 09/15/2011 11:49 AM, Nikolay Kichukov wrote: > Hi, > this query is best suited for md-raid mailing list as what you measure: > > dd if=/dev/md3 of=/dev/null bs=1M > > does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there. heh. I read that as crypt device and it is MD directly, sorry for confusion. Seems my brain is still in different timezone :) yes, this test has nothing to do with crypt. Anyway, it is interesting that the problem is there even without crypt, perhaps you should ask on linux-raid list then. for badblock - I guess it just submits all IO from one cpu, so only one core is used for encryption in fact. There applies the same what I wrote before. Milan ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dm-crypt] slow read performance, but fast writes? 2011-09-15 10:45 ` Milan Broz @ 2011-09-15 11:07 ` Peter Merhaut 0 siblings, 0 replies; 8+ messages in thread From: Peter Merhaut @ 2011-09-15 11:07 UTC (permalink / raw) To: dm-crypt Yeah but why is it using one cpu for encryption? I got 6 encrypted block devices. When writing to them, all cpu cores are used for encryption, therefor it's fast. When reading, it seems to use 1 cpu for all 6 crypto threads? Am 09/15/2011 12:45 PM, schrieb Milan Broz: > On 09/15/2011 11:49 AM, Nikolay Kichukov wrote: >> Hi, >> this query is best suited for md-raid mailing list as what you measure: >> >> dd if=/dev/md3 of=/dev/null bs=1M >> >> does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there. > heh. I read that as crypt device and it is MD directly, sorry for confusion. > Seems my brain is still in different timezone :) > > yes, this test has nothing to do with crypt. > > Anyway, it is interesting that the problem is there even without crypt, > perhaps you should ask on linux-raid list then. > > for badblock - I guess it just submits all IO from one cpu, > so only one core is used for encryption in fact. There applies the same what I wrote before. > > Milan ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dm-crypt] slow read performance, but fast writes? 2011-09-14 14:04 [dm-crypt] slow read performance, but fast writes? Peter Merhaut 2011-09-14 15:04 ` Arno Wagner @ 2011-09-14 15:15 ` Milan Broz 1 sibling, 0 replies; 8+ messages in thread From: Milan Broz @ 2011-09-14 15:15 UTC (permalink / raw) To: Peter Merhaut; +Cc: dm-crypt On 09/14/2011 04:04 PM, Peter Merhaut wrote: > i've got a really strange problem with my setup, and i'm not quite sure why. It is known problem, current implementation depends on how are IO submitted and writes and reads differs this way (I think page cache interferes here). Reads are slower, because parallel operations are timed differently causing these waits. If you use some more realistic IO pattern than dd, you can get different output though. (dd is NOT good performance test. But everyone uses that:-) We have some ideas how to improve parallel processing inside dm-crypt but other things had higher priority still. (You can see this problem even measured against zero virtual devices, so read/write caching is not the real problem here.) Milan ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2011-09-15 11:07 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-09-14 14:04 [dm-crypt] slow read performance, but fast writes? Peter Merhaut 2011-09-14 15:04 ` Arno Wagner 2011-09-14 16:47 ` Peter Merhaut 2011-09-15 9:49 ` Nikolay Kichukov 2011-09-15 9:53 ` Peter Merhaut 2011-09-15 10:45 ` Milan Broz 2011-09-15 11:07 ` Peter Merhaut 2011-09-14 15:15 ` Milan Broz
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox