[dm-crypt] FSB saturation?

[dm-crypt] FSB saturation?

[Kyle Rose]

For the past week I've been trying to figure out why my
dm-crypt-on-RAID6 performance is so terrible. To wit:

root@yupa:/source# hdparm -t /dev/md1

/dev/md1: <---the RAID 6 block device
 Timing buffered disk reads: 1262 MB in  3.00 seconds = 420.19 MB/sec

root@yupa:/source# hdparm -t /dev/mapper/yupa

/dev/mapper/yupa: <---the dm-crypt device with /dev/md1 as the underlying store
 Timing buffered disk reads: 370 MB in  3.00 seconds = 123.26 MB/sec

The machine (a Core 2 Quad) is not anywhere near CPU bound in either
case (even with all the bulk encryption), and is obviously not disk
bound in the latter. Besides having used linux-2.6.38 and
cryptsetup-1.1.3 to build the system, post-hoc analysis of iostat
indicates that all layers are properly aligned. My working theory is
FSB contention caused by lots of memory copies during the process of
moving data through each layer. Has anyone else run into this?

Re: [dm-crypt] FSB saturation?

[Milan Broz]

On 05/14/2011 04:36 PM, Kyle Rose wrote:
> For the past week I've been trying to figure out why my
> dm-crypt-on-RAID6 performance is so terrible. To wit:
> 
> root@yupa:/source# hdparm -t /dev/md1
> 
> /dev/md1: <---the RAID 6 block device
>  Timing buffered disk reads: 1262 MB in  3.00 seconds = 420.19 MB/sec
> 
> root@yupa:/source# hdparm -t /dev/mapper/yupa
> 
> /dev/mapper/yupa: <---the dm-crypt device with /dev/md1 as the underlying store
>  Timing buffered disk reads: 370 MB in  3.00 seconds = 123.26 MB/sec

IIRC hdparm is doing direct io of size 2MB.

Can you try do the same but with dd (just dd from device without direct mode).
like "dd if=/dev/md1 of=/dev/null bs=64k count=4096"

Gives it better throughput now? (on 2.6.38 and above)

Milan

Re: [dm-crypt] FSB saturation?

[Kyle Rose]

Nope. :-)

root@yupa:~# dd if=/dev/md1 of=/dev/null bs=64k count=65536
65536+0 records in
65536+0 records out
4294967296 bytes (4.3 GB) copied, 10.1625 s, 423 MB/s
root@yupa:~# dd if=/dev/mapper/yupa of=/dev/null bs=64k count=65536
65536+0 records in
65536+0 records out
4294967296 bytes (4.3 GB) copied, 35.2504 s, 122 MB/s


On Sat, May 14, 2011 at 11:30 AM, Milan Broz <mbroz@redhat.com> wrote:
> On 05/14/2011 04:36 PM, Kyle Rose wrote:
>> For the past week I've been trying to figure out why my
>> dm-crypt-on-RAID6 performance is so terrible. To wit:
>>
>> root@yupa:/source# hdparm -t /dev/md1
>>
>> /dev/md1: <---the RAID 6 block device
>>  Timing buffered disk reads: 1262 MB in  3.00 seconds = 420.19 MB/sec
>>
>> root@yupa:/source# hdparm -t /dev/mapper/yupa
>>
>> /dev/mapper/yupa: <---the dm-crypt device with /dev/md1 as the underlying store
>>  Timing buffered disk reads: 370 MB in  3.00 seconds = 123.26 MB/sec
>
> IIRC hdparm is doing direct io of size 2MB.
>
> Can you try do the same but with dd (just dd from device without direct mode).
> like "dd if=/dev/md1 of=/dev/null bs=64k count=4096"
>
> Gives it better throughput now? (on 2.6.38 and above)
>
> Milan
>