dd slow while reading on multipathing, writing is fast

dd slow while reading on multipathing, writing is fast

[bart.coninckx]

Hi all,

something peculiar I cannot get my head round.
I have a setup where a backup server connects to iSCSI LUNs over multipathing.
While dd-ing to it, I get a spiffing 200 MB/sec, while reading it drops to 70 MB/sec. Local storage where I write the image onto is sufficiently fast.

Do you think this is multipathing related? I'm running the latest version from source.




Thx!


B.

Re: dd slow while reading on multipathing, writing is fast

[Christophe Varoqui]

On mar., 2011-02-08 at 15:44 +0100, bart.coninckx@telenet.be wrote:
> Hi all,
> 
> something peculiar I cannot get my head round.
> I have a setup where a backup server connects to iSCSI LUNs over multipathing.
> While dd-ing to it, I get a spiffing 200 MB/sec, while reading it drops to 70 MB/sec. Local storage where I write the image onto is sufficiently fast.
> 
> Do you think this is multipathing related? I'm running the latest version from source.
> 
please send at least
- kernel version
- multipath tools version
- device map string

Re: dd slow while reading on multipathing, writing is fast

[Bryn M. Reeves]

On 02/08/2011 02:44 PM, bart.coninckx@telenet.be wrote:
> Hi all,
> 
> something peculiar I cannot get my head round. I have a setup where a backup
> server connects to iSCSI LUNs over multipathing. While dd-ing to it, I get a
> spiffing 200 MB/sec, while reading it drops to 70 MB/sec. Local storage where
> I write the image onto is sufficiently fast.

As Christoph mentioned the versions of the kernel and tools you are using and
the configuration of the multipath device would be useful but there are a couple
of things that spring to mind; if you are seeing good performance on write but
poor on read then it's possible that your testing is not really measuring the
device performance.

Writes are buffered in memory (possibly both on the iSCSI initiator and target,
depending on configuration) whereas a read of an un-cached region of the device
will need to perform actual I/O before it can return. Although dd is a useful
tool for quick tests it's not a very rigorous way to benchmark performance
unless you take special steps to mitigate things like caching effects.

Another point is that you don't mention if you have tested the underlying iSCSI
device in isolation? I.e. remove multipath from the picture and verify that you
get the kind of performance you expect from a single path.

Regards,
Bryn.

Re: dd slow while reading on multipathing, writing is fast

[Bart Coninckx]

On 02/09/11 12:22, Bryn M. Reeves wrote:
> On 02/08/2011 02:44 PM, bart.coninckx@telenet.be wrote:
>> Hi all,
>>
>> something peculiar I cannot get my head round. I have a setup where a backup
>> server connects to iSCSI LUNs over multipathing. While dd-ing to it, I get a
>> spiffing 200 MB/sec, while reading it drops to 70 MB/sec. Local storage where
>> I write the image onto is sufficiently fast.
> 
> As Christoph mentioned the versions of the kernel and tools you are using and
> the configuration of the multipath device would be useful but there are a couple
> of things that spring to mind; if you are seeing good performance on write but
> poor on read then it's possible that your testing is not really measuring the
> device performance.
> 
> Writes are buffered in memory (possibly both on the iSCSI initiator and target,
> depending on configuration) whereas a read of an un-cached region of the device
> will need to perform actual I/O before it can return. Although dd is a useful
> tool for quick tests it's not a very rigorous way to benchmark performance
> unless you take special steps to mitigate things like caching effects.
> 
> Another point is that you don't mention if you have tested the underlying iSCSI
> device in isolation? I.e. remove multipath from the picture and verify that you
> get the kind of performance you expect from a single path.
> 
> Regards,
> Bryn.

All,


This is on Opensuse 11.3 with kernel-desktop-2.6.34.7-0.7.1.x86_64
(probably should change that to a default kernel). I upgraded the
multipahting-tools to the latest version because they are badly broken
on the 64-bit version of this distro.

I used bs and count combinations for the dd command that result into a
dump that surpasses available RAM, so there should be no caching happening.
I later on did indeed try on iscsi without multipathing on a bonded
interface and got about the same results, but I have to admit that I
forgot to change tc_reordering, which is necessary on Suse. probably
better do that again and report back. I did find however that if the
underlying iscsi device is an LVM snaphost, read speed goes down when
LVM snapshot size goed up (or so it seems).

b.