understanding of multipathing and speed

understanding of multipathing and speed

[Bart Coninckx]

Hi,

I would like to run my ideas by this list about multipathing and the results 
as far as storage speed is concerned.

I'm using multipathing to two iSCSI targets pointing to the same storage. It 
was my understanding that this provides for network path redundancy (and it 
does, I tested this) but also for added speed. 
I did some tests with Bonnie++ however while both paths were active and one 
path was down and the results are basically the same. 

Am I assuming wrong things? Or have I configured things wrong?

This is my config for now:

defaults {
        udev_dir                /dev
        polling_interval        10
        selector                "round-robin 0"
        path_grouping_policy    multibus
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio                    const
        path_checker            directio
        rr_min_io               100
        max_fds                 8192
        rr_weight               priorities
        failback                immediate
        no_path_retry           5
        user_friendly_names     no
}
blacklist {
       wwid 26353900f02796769
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z][[0-9]*]"
        devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
        device {
                vendor DEC.*
                product MSA[15]00
        }
}
multipaths {
 multipath {
                wwid                    
1494554000000000000000000010000000000000002000000
  alias                   lx03
  }
 multipath {
                wwid                    
1494554000000000000000000010000000100000002000000
  alias                   ws033
  }
 multipath {
                wwid                    
1494554000000000000000000010000000200000002000000
  alias                   ms01
  }
}


Thank you!!

B.

Re: understanding of multipathing and speed

[Christophe Varoqui]

On lun., 2010-07-05 at 20:37 +0200, Bart Coninckx wrote:
> Hi,
> 
> I would like to run my ideas by this list about multipathing and the results 
> as far as storage speed is concerned.
> 
> I'm using multipathing to two iSCSI targets pointing to the same storage. It 
> was my understanding that this provides for network path redundancy (and it 
> does, I tested this) but also for added speed. 
> I did some tests with Bonnie++ however while both paths were active and one 
> path was down and the results are basically the same. 
> 
> Am I assuming wrong things? Or have I configured things wrong?
> 
can you also include a 'multipath -l' output and sketch the
hba/switch/controller physical connections ?

thanks,
-- 
Christophe Varoqui <christophe.varoqui@opensvc.com>
OpenSVC

Re: understanding of multipathing and speed

[John A. Sullivan III]

On Mon, 2010-07-05 at 20:58 +0200, Christophe Varoqui wrote:
> On lun., 2010-07-05 at 20:37 +0200, Bart Coninckx wrote:
> > Hi,
> > 
> > I would like to run my ideas by this list about multipathing and the results 
> > as far as storage speed is concerned.
> > 
> > I'm using multipathing to two iSCSI targets pointing to the same storage. It 
> > was my understanding that this provides for network path redundancy (and it 
> > does, I tested this) but also for added speed. 
> > I did some tests with Bonnie++ however while both paths were active and one 
> > path was down and the results are basically the same. 
> > 
> > Am I assuming wrong things? Or have I configured things wrong?
> > 
> can you also include a 'multipath -l' output and sketch the
> hba/switch/controller physical connections ?
> 
> thanks,
I am by no means an expert but I did find a significant speed advantage
but it depended on how we tested.  We found if we sent a single thread
of reads or writes, there was no gain. However, as we started adding
multiple, simultaneous reads and writes, there was a dramatic gain.

We were actually testing two different scenarios:
1) dm-multipath for both fault tolerance and load balancing
2) dm-multipath for fault-tolerance and software RAID0 for load
balancing


This testing was mostly performed using Linux File I/O or by imitating
Linux File I/O by setting the maximum block size to 4KB. That meant our
iSCSI connection was limited more by latency than throughput.

We found the latter gave more well rounded performance, i.e., over a
wide range of I/O characteristics.  However, we subsequently learned
that it created a consistency problem for snapshooting on the SAN
back-end.  We could not guarantee that the snapshot would be in a
consistent RAID0 state.  Thus, we plan to convert our systems back to
multibus for load balancing.

Another item that surprised us was the setting for how many commands
should be issued before switching paths (I forget the parameter name).
I would have expected a lower number to give better throughput but to
drive up CPU utilization.  That did not turn out to be true even on
systems with plenty of CPU power to spare.  We found that a setting of
100 performed marginally but consistently better than a setting of 10.
I do not know why that is.  Perhaps this is again because we were
latency and not throughput bound.  Perhaps the switching adds latency.

I do not know the internals to know if multibus could possibly help for
a single iSCSI conversation.  The blocks must still be sent in series (I
assume) so I would think the packets would just round robin around the
NICS but not be sent in parallel.  When loading multiple iSCSI
conversation, I would think all the NICs could be firing at the same
time with different iSCSI series. Again, I am guessing in my ignorance.

Can anyone else confirm or refute these deductions? Thanks - John

Re: understanding of multipathing and speed

[Bart Coninckx]

On Monday 05 July 2010 20:58:30 Christophe Varoqui wrote:
> On lun., 2010-07-05 at 20:37 +0200, Bart Coninckx wrote:
> > Hi,
> >
> > I would like to run my ideas by this list about multipathing and the
> > results as far as storage speed is concerned.
> >
> > I'm using multipathing to two iSCSI targets pointing to the same storage.
> > It was my understanding that this provides for network path redundancy
> > (and it does, I tested this) but also for added speed.
> > I did some tests with Bonnie++ however while both paths were active and
> > one path was down and the results are basically the same.
> >
> > Am I assuming wrong things? Or have I configured things wrong?
> 
> can you also include a 'multipath -l' output and sketch the
> hba/switch/controller physical connections ?
> 
> thanks,
> 

Sure,

xen3:~ # multipath -l
lx03 (1494554000000000000000000010000000000000002000000) dm-3 IET,VIRTUAL-DISK
[size=10G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=-2][active]
 \_ 2:0:0:0 sdc 8:32  [active][undef]
 \_ 1:0:0:0 sdb 8:16  [active][undef]
ws033 (1494554000000000000000000010000000100000002000000) dm-2 IET,VIRTUAL-
DISK
[size=15G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=-2][active]
 \_ 2:0:0:1 sde 8:64  [active][undef]
 \_ 1:0:0:1 sdd 8:48  [active][undef]
ms01 (1494554000000000000000000010000000200000002000000) dm-1 IET,VIRTUAL-DISK
[size=40G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=-2][active]
 \_ 1:0:0:2 sdf 8:80  [active][undef]
 \_ 2:0:0:2 sdg 8:96  [active][undef]

I have two Gigabit NICs in this server each running over a separate switch to 
a separate gigabit NIC with a unique IP address on the storage IET iSCSI 
target.

Is this sufficient info?

Thx,

Bart

Re: understanding of multipathing and speed

[Christophe Varoqui]

[-- Attachment #1.1: Type: text/plain, Size: 612 bytes --]

(1494554000000000000000000010000000200000002000000) dm-1
> IET,VIRTUAL-DISK [size=40G][features=1 queue_if_no_path][hwhandler=0]
> \_ round-robin 0 [prio=-2][active]
>   \_ 1:0:0:2 sdf 8:80   [active][undef]
>   \_ 2:0:0:2 sdg 8:96   [active][undef]
> 
> I have two Gigabit NICs in this server each running over a separate
> switch to   a separate gigabit NIC with a unique IP address on the
> storage IET iSCSI   target.
> 
did you setup different ip networks for each independant lan ? You may want to verify the ip routing is not sending every other packet through the wrong interface ?

Bye

[-- Attachment #1.2: Type: text/html, Size: 1018 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: understanding of multipathing and speed

[Bart Coninckx]

On Tuesday 06 July 2010 06:16:55 Bart Coninckx wrote:
> On Monday 05 July 2010 20:58:30 Christophe Varoqui wrote:
> > On lun., 2010-07-05 at 20:37 +0200, Bart Coninckx wrote:
> > > Hi,
> > >
> > > I would like to run my ideas by this list about multipathing and the
> > > results as far as storage speed is concerned.
> > >
> > > I'm using multipathing to two iSCSI targets pointing to the same
> > > storage. It was my understanding that this provides for network path
> > > redundancy (and it does, I tested this) but also for added speed.
> > > I did some tests with Bonnie++ however while both paths were active and
> > > one path was down and the results are basically the same.
> > >
> > > Am I assuming wrong things? Or have I configured things wrong?
> >
> > can you also include a 'multipath -l' output and sketch the
> > hba/switch/controller physical connections ?
> >
> > thanks,
> 
> Sure,
> 
> xen3:~ # multipath -l
> lx03 (1494554000000000000000000010000000000000002000000) dm-3
>  IET,VIRTUAL-DISK [size=10G][features=1 queue_if_no_path][hwhandler=0]
> \_ round-robin 0 [prio=-2][active]
>  \_ 2:0:0:0 sdc 8:32  [active][undef]
>  \_ 1:0:0:0 sdb 8:16  [active][undef]
> ws033 (1494554000000000000000000010000000100000002000000) dm-2 IET,VIRTUAL-
> DISK
> [size=15G][features=1 queue_if_no_path][hwhandler=0]
> \_ round-robin 0 [prio=-2][active]
>  \_ 2:0:0:1 sde 8:64  [active][undef]
>  \_ 1:0:0:1 sdd 8:48  [active][undef]
> ms01 (1494554000000000000000000010000000200000002000000) dm-1
>  IET,VIRTUAL-DISK [size=40G][features=1 queue_if_no_path][hwhandler=0]
> \_ round-robin 0 [prio=-2][active]
>  \_ 1:0:0:2 sdf 8:80  [active][undef]
>  \_ 2:0:0:2 sdg 8:96  [active][undef]
> 
> I have two Gigabit NICs in this server each running over a separate switch
>  to a separate gigabit NIC with a unique IP address on the storage IET
>  iSCSI target.
> 
> Is this sufficient info?
> 
> Thx,
> 
> Bart
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 

Hi all,

to show my point, these are the results of running bonnie++ locally on the 
storage - the values I look at are Block values in K/sec in both sequential 
output (writing) and sequential input (reading):

Version 1.03e       ------Sequential Output------ --Sequential Input- --
Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
%CP
iscsi3           8G 69351  96 116112  32 41128  10 57874  82 107721  16 418.2   
0
                    ------Sequential Create------ --------Random 
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -
Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP
                 16  4533  99 +++++ +++ +++++ +++  4395  99 +++++ +++ 17122  
99
iscsi3,8G,69351,96,116112,32,41128,10,57874,82,107721,16,418.2,0,16,4533,99,
+++++,+++,+++++,+++,4395,99,+++++,+++,17122,99



So were are hitting roughly 110 MB/sec locally on the storage server.

Now these are the results do doing the same over multipath with two paths 
enabled:

Version 1.03e       ------Sequential Output------ --Sequential Input- --
Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
%CP
xen3             8G 63953  92 100525  26 26885   2 41957  55 68184   2 357.9   
0
                    ------Sequential Create------ --------Random 
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -
Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP
                 16  5326  98 +++++ +++ +++++ +++  5333  97 +++++ +++ 17179 
100
xen3,8G,63953,92,100525,26,26885,2,41957,55,68184,2,357.9,0,16,5326,98,+++++,
+++,+++++,+++,5333,97,+++++,+++,17179,100

You can see we hit somewhat less, probably due to TCP overhead (though this 
should cut things with 30%). Now the same with one path down:

Version 1.03e       ------Sequential Output------ --Sequential Input- --
Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
%CP
xen3             8G 33214  46 113811  29 27917   1 44474  58 68812   2 362.8   
0
                    ------Sequential Create------ --------Random 
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -
Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP
                 16  5294  98 +++++ +++ +++++ +++  5337  97 +++++ +++ 17183  
99
xen3,8G,33214,46,113811,29,27917,1,44474,58,68812,2,362.8,0,16,5294,98,+++++,
+++,+++++,+++,5337,97,+++++,+++,17183,99

As you can see, roughly the same K/sec for both output and input. Actually 
writing is even faster with one path down!
Can anyone make sense of these values?

thx!

B.
 

Re: understanding of multipathing and speed

[Bart Coninckx]

On Wednesday 07 July 2010 19:18:48 Bart Coninckx wrote:
> On Tuesday 06 July 2010 06:16:55 Bart Coninckx wrote:
> > On Monday 05 July 2010 20:58:30 Christophe Varoqui wrote:
> > > On lun., 2010-07-05 at 20:37 +0200, Bart Coninckx wrote:
> > > > Hi,
> > > >
> > > > I would like to run my ideas by this list about multipathing and the
> > > > results as far as storage speed is concerned.
> > > >
> > > > I'm using multipathing to two iSCSI targets pointing to the same
> > > > storage. It was my understanding that this provides for network path
> > > > redundancy (and it does, I tested this) but also for added speed.
> > > > I did some tests with Bonnie++ however while both paths were active
> > > > and one path was down and the results are basically the same.
> > > >
> > > > Am I assuming wrong things? Or have I configured things wrong?
> > >
> > > can you also include a 'multipath -l' output and sketch the
> > > hba/switch/controller physical connections ?
> > >
> > > thanks,
> >
> > Sure,
> >
> > xen3:~ # multipath -l
> > lx03 (1494554000000000000000000010000000000000002000000) dm-3
> >  IET,VIRTUAL-DISK [size=10G][features=1 queue_if_no_path][hwhandler=0]
> > \_ round-robin 0 [prio=-2][active]
> >  \_ 2:0:0:0 sdc 8:32  [active][undef]
> >  \_ 1:0:0:0 sdb 8:16  [active][undef]
> > ws033 (1494554000000000000000000010000000100000002000000) dm-2
> > IET,VIRTUAL- DISK
> > [size=15G][features=1 queue_if_no_path][hwhandler=0]
> > \_ round-robin 0 [prio=-2][active]
> >  \_ 2:0:0:1 sde 8:64  [active][undef]
> >  \_ 1:0:0:1 sdd 8:48  [active][undef]
> > ms01 (1494554000000000000000000010000000200000002000000) dm-1
> >  IET,VIRTUAL-DISK [size=40G][features=1 queue_if_no_path][hwhandler=0]
> > \_ round-robin 0 [prio=-2][active]
> >  \_ 1:0:0:2 sdf 8:80  [active][undef]
> >  \_ 2:0:0:2 sdg 8:96  [active][undef]
> >
> > I have two Gigabit NICs in this server each running over a separate
> > switch to a separate gigabit NIC with a unique IP address on the storage
> > IET iSCSI target.
> >
> > Is this sufficient info?
> >
> > Thx,
> >
> > Bart
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> 
> Hi all,
> 
> to show my point, these are the results of running bonnie++ locally on the
> storage - the values I look at are Block values in K/sec in both sequential
> output (writing) and sequential input (reading):
> 
> Version 1.03e       ------Sequential Output------ --Sequential Input- --
> Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
> Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> %CP
> iscsi3           8G 69351  96 116112  32 41128  10 57874  82 107721  16
>  418.2 0
>                     ------Sequential Create------ --------Random
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read--- -
> Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
> %CP
>                  16  4533  99 +++++ +++ +++++ +++  4395  99 +++++ +++ 17122
> 99
> iscsi3,8G,69351,96,116112,32,41128,10,57874,82,107721,16,418.2,0,16,4533,99
> , +++++,+++,+++++,+++,4395,99,+++++,+++,17122,99
> 
> 
> 
> So were are hitting roughly 110 MB/sec locally on the storage server.
> 
> Now these are the results do doing the same over multipath with two paths
> enabled:
> 
> Version 1.03e       ------Sequential Output------ --Sequential Input- --
> Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
> Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> %CP
> xen3             8G 63953  92 100525  26 26885   2 41957  55 68184   2
>  357.9 0
>                     ------Sequential Create------ --------Random
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read--- -
> Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
> %CP
>                  16  5326  98 +++++ +++ +++++ +++  5333  97 +++++ +++ 17179
> 100
> xen3,8G,63953,92,100525,26,26885,2,41957,55,68184,2,357.9,0,16,5326,98,++++
> +, +++,+++++,+++,5333,97,+++++,+++,17179,100
> 
> You can see we hit somewhat less, probably due to TCP overhead (though this
> should cut things with 30%). Now the same with one path down:
> 
> Version 1.03e       ------Sequential Output------ --Sequential Input- --
> Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
> Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> %CP
> xen3             8G 33214  46 113811  29 27917   1 44474  58 68812   2
>  362.8 0
>                     ------Sequential Create------ --------Random
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read--- -
> Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
> %CP
>                  16  5294  98 +++++ +++ +++++ +++  5337  97 +++++ +++ 17183
> 99
> xen3,8G,33214,46,113811,29,27917,1,44474,58,68812,2,362.8,0,16,5294,98,++++
> +, +++,+++++,+++,5337,97,+++++,+++,17183,99
> 
> As you can see, roughly the same K/sec for both output and input. Actually
> writing is even faster with one path down!
> Can anyone make sense of these values?
> 
> thx!
> 
> B.
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 

Maybe adding this one while doing a test with both paths active during "off 
hours", so no other intrusive factors: 

Version 1.03e       ------Sequential Output------ --Sequential Input- --
Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
%CP
xen3             8G 66510  93 80841  21 26821   1 45368  58 72095   2 361.2   
0
                    ------Sequential Create------ --------Random 
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -
Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP
                 16  5295  98 +++++ +++ +++++ +++  5318  98 +++++ +++ 17089 
100
xen3,8G,66510,93,80841,21,26821,1,45368,58,72095,2,361.2,0,16,5295,98,+++++,
+++,+++++,+++,5318,98,+++++,+++,17089,100


it show that the speed is exactly 70% of the speed when doing tests locally. 
So this might be the iSCSI TCP overhead. 

Should the speed of two round robin paths not compensate for this loss? Or is 
my local storage just to slow to have multipath having any benefit speed wise?


thx!

B.

Re: understanding of multipathing and speed

[Brem Belguebli]

Hi,
On Wed, 2010-07-07 at 23:01 +0200, Bart Coninckx wrote:
> On Wednesday 07 July 2010 19:18:48 Bart Coninckx wrote:
> > On Tuesday 06 July 2010 06:16:55 Bart Coninckx wrote:
> > > On Monday 05 July 2010 20:58:30 Christophe Varoqui wrote:
> > > > On lun., 2010-07-05 at 20:37 +0200, Bart Coninckx wrote:
> > > > > Hi,
> > > > >
> > > > > I would like to run my ideas by this list about multipathing and the
> > > > > results as far as storage speed is concerned.
> > > > >
> > > > > I'm using multipathing to two iSCSI targets pointing to the same
> > > > > storage. It was my understanding that this provides for network path
> > > > > redundancy (and it does, I tested this) but also for added speed.
> > > > > I did some tests with Bonnie++ however while both paths were active
> > > > > and one path was down and the results are basically the same.
> > > > >
> > > > > Am I assuming wrong things? Or have I configured things wrong?
> > > >
> > > > can you also include a 'multipath -l' output and sketch the
> > > > hba/switch/controller physical connections ?
> > > >
> > > > thanks,
> > >
> > > Sure,
> > >
> > > xen3:~ # multipath -l
> > > lx03 (1494554000000000000000000010000000000000002000000) dm-3
> > >  IET,VIRTUAL-DISK [size=10G][features=1 queue_if_no_path][hwhandler=0]
> > > \_ round-robin 0 [prio=-2][active]
> > >  \_ 2:0:0:0 sdc 8:32  [active][undef]
> > >  \_ 1:0:0:0 sdb 8:16  [active][undef]
> > > ws033 (1494554000000000000000000010000000100000002000000) dm-2
> > > IET,VIRTUAL- DISK
> > > [size=15G][features=1 queue_if_no_path][hwhandler=0]
> > > \_ round-robin 0 [prio=-2][active]
> > >  \_ 2:0:0:1 sde 8:64  [active][undef]
> > >  \_ 1:0:0:1 sdd 8:48  [active][undef]
> > > ms01 (1494554000000000000000000010000000200000002000000) dm-1
> > >  IET,VIRTUAL-DISK [size=40G][features=1 queue_if_no_path][hwhandler=0]
> > > \_ round-robin 0 [prio=-2][active]
> > >  \_ 1:0:0:2 sdf 8:80  [active][undef]
> > >  \_ 2:0:0:2 sdg 8:96  [active][undef]
> > >
> > > I have two Gigabit NICs in this server each running over a separate
> > > switch to a separate gigabit NIC with a unique IP address on the storage
> > > IET iSCSI target.
> > >
> > > Is this sufficient info?
> > >
> > > Thx,
> > >
> > > Bart
> > >
> > > --
> > > dm-devel mailing list
> > > dm-devel@redhat.com
> > > https://www.redhat.com/mailman/listinfo/dm-devel
> > 
> > Hi all,
> > 
> > to show my point, these are the results of running bonnie++ locally on the
> > storage - the values I look at are Block values in K/sec in both sequential
> > output (writing) and sequential input (reading):
> > 
> > Version 1.03e       ------Sequential Output------ --Sequential Input- --
> > Random-
> >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
> > Seeks--
> > Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> > %CP
> > iscsi3           8G 69351  96 116112  32 41128  10 57874  82 107721  16
> >  418.2 0
> >                     ------Sequential Create------ --------Random
> > Create--------
> >                     -Create-- --Read--- -Delete-- -Create-- --Read--- -
> > Delete--
> >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
> > %CP
> >                  16  4533  99 +++++ +++ +++++ +++  4395  99 +++++ +++ 17122
> > 99
> > iscsi3,8G,69351,96,116112,32,41128,10,57874,82,107721,16,418.2,0,16,4533,99
> > , +++++,+++,+++++,+++,4395,99,+++++,+++,17122,99
> > 
> > 
> > 
> > So were are hitting roughly 110 MB/sec locally on the storage server.
> > 
> > Now these are the results do doing the same over multipath with two paths
> > enabled:
> > 
> > Version 1.03e       ------Sequential Output------ --Sequential Input- --
> > Random-
> >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
> > Seeks--
> > Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> > %CP
> > xen3             8G 63953  92 100525  26 26885   2 41957  55 68184   2
> >  357.9 0
> >                     ------Sequential Create------ --------Random
> > Create--------
> >                     -Create-- --Read--- -Delete-- -Create-- --Read--- -
> > Delete--
> >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
> > %CP
> >                  16  5326  98 +++++ +++ +++++ +++  5333  97 +++++ +++ 17179
> > 100
> > xen3,8G,63953,92,100525,26,26885,2,41957,55,68184,2,357.9,0,16,5326,98,++++
> > +, +++,+++++,+++,5333,97,+++++,+++,17179,100
> > 
> > You can see we hit somewhat less, probably due to TCP overhead (though this
> > should cut things with 30%). Now the same with one path down:
> > 
> > Version 1.03e       ------Sequential Output------ --Sequential Input- --
> > Random-
> >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
> > Seeks--
> > Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> > %CP
> > xen3             8G 33214  46 113811  29 27917   1 44474  58 68812   2
> >  362.8 0
> >                     ------Sequential Create------ --------Random
> > Create--------
> >                     -Create-- --Read--- -Delete-- -Create-- --Read--- -
> > Delete--
> >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
> > %CP
> >                  16  5294  98 +++++ +++ +++++ +++  5337  97 +++++ +++ 17183
> > 99
> > xen3,8G,33214,46,113811,29,27917,1,44474,58,68812,2,362.8,0,16,5294,98,++++
> > +, +++,+++++,+++,5337,97,+++++,+++,17183,99
> > 
> > As you can see, roughly the same K/sec for both output and input. Actually
> > writing is even faster with one path down!
> > Can anyone make sense of these values?
> > 
> > thx!
> > 
> > B.
> > 
> > 
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> > 
> 
> Maybe adding this one while doing a test with both paths active during "off 
> hours", so no other intrusive factors: 
> 
> Version 1.03e       ------Sequential Output------ --Sequential Input- --
> Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
> Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
> %CP
> xen3             8G 66510  93 80841  21 26821   1 45368  58 72095   2 361.2   
> 0
>                     ------Sequential Create------ --------Random 
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read--- -
> Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
> %CP
>                  16  5295  98 +++++ +++ +++++ +++  5318  98 +++++ +++ 17089 
> 100
> xen3,8G,66510,93,80841,21,26821,1,45368,58,72095,2,361.2,0,16,5295,98,+++++,
> +++,+++++,+++,5318,98,+++++,+++,17089,100
> 
> 
> it show that the speed is exactly 70% of the speed when doing tests locally. 
> So this might be the iSCSI TCP overhead. 
> 
> Should the speed of two round robin paths not compensate for this loss? Or is 
> my local storage just to slow to have multipath having any benefit speed wise?
> 
To calculate your theoretical TCP throughput, a simple formula can be
applied : TP= TCP Window size / RTT 

In addition to this generally, you can take advantage of path
load-balancing IO's to capable to multiplex IO's devices , such as SAN
arrays with cache frontend.
Trying to load-balance on a single physical device, won't be, IMHO, of
any help except for pure failover purposes.
   
> 
> thx!
> 
> B.
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: understanding of multipathing and speed

[Bart Coninckx]

On Thursday 08 July 2010 00:06:39 Brem Belguebli wrote:

> > > --
> > > dm-devel mailing list
> > > dm-devel@redhat.com
> > > https://www.redhat.com/mailman/listinfo/dm-devel
> >
> > Maybe adding this one while doing a test with both paths active during
> > "off hours", so no other intrusive factors:
> >
> > Version 1.03e       ------Sequential Output------ --Sequential Input- --
> > Random-
> >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
> > Seeks--
> > Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
> > /sec %CP
> > xen3             8G 66510  93 80841  21 26821   1 45368  58 72095   2
> > 361.2 0
> >                     ------Sequential Create------ --------Random
> > Create--------
> >                     -Create-- --Read--- -Delete-- -Create-- --Read--- -
> > Delete--
> >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
> > /sec %CP
> >                  16  5295  98 +++++ +++ +++++ +++  5318  98 +++++ +++
> > 17089 100
> > xen3,8G,66510,93,80841,21,26821,1,45368,58,72095,2,361.2,0,16,5295,98,+++
> >++, +++,+++++,+++,5318,98,+++++,+++,17089,100
> >
> >
> > it show that the speed is exactly 70% of the speed when doing tests
> > locally. So this might be the iSCSI TCP overhead.
> >
> > Should the speed of two round robin paths not compensate for this loss?
> > Or is my local storage just to slow to have multipath having any benefit
> > speed wise?
> 
> To calculate your theoretical TCP throughput, a simple formula can be
> applied : TP= TCP Window size / RTT
> 
> In addition to this generally, you can take advantage of path
> load-balancing IO's to capable to multiplex IO's devices , such as SAN
> arrays with cache frontend.
> Trying to load-balance on a single physical device, won't be, IMHO, of
> any help except for pure failover purposes.
> 
> > thx!
> >
> > B.
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 

Hi Brem,

are you talking about storage with a sort of built-in active/active storage 
cluster? Do you think I don't have added speed because if the fact that both 
iSCSI targets point to the same storage device?


thx!!
Bart

Re: understanding of multipathing and speed

[bart.coninckx]

[-- Attachment #1.1: Type: text/plain, Size: 967 bytes --]



On Tuesday 06 July 2010 09:21:51 Christophe Varoqui wrote: 

> (1494554000000000000000000010000000200000002000000) dm-1 

> 

> > IET,VIRTUAL-DISK [size=40G][features=1 queue_if_no_path][hwhandler=0] 

> > \_ round-robin 0 [prio=-2][active] 

> > \_ 1:0:0:2 sdf 8:80 [active][undef] 

> > \_ 2:0:0:2 sdg 8:96 [active][undef] 

> > 

> > I have two Gigabit NICs in this server each running over a separate 

> > switch to a separate gigabit NIC with a unique IP address on the 

> > storage IET iSCSI target. 

> 

> did you setup different ip networks for each independant lan ? You may want 

> to verify the ip routing is not sending every other packet through the 

> wrong interface ? 

> 

> Bye 

> 







Hi Christophe, 



yes I did, the two initiators and targets have there own IP network. IP 

routing points to the correct interface per network. Maybe my test (bonnie++) 

is not an appropriate one to show the difference in speed? 



thx, 



Bart 


[-- Attachment #1.2: Type: text/html, Size: 2789 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: understanding of multipathing and speed

[Christophe Varoqui]

[-- Attachment #1.1: Type: text/plain, Size: 838 bytes --]

> routing points to the correct interface per network. Maybe my test
> (bonnie++) 
> 
> is not an appropriate one to show the difference in speed? 
> 
an interesting data point would be a perf run when iet target has only one target port on a bonded interface a multipath on client side.

And maybe the other combinations. Th eset combinations would be :
- client bonded, target not bonded
- client bonded, target bonded (i understand this is the scenario with good performance you reported earlier)
- client not bonded, target not bonded (perf expected from a single path in your testing)
- client not bonded, target bonded

the mixed setup would not be adequate for production as they require linked or single switch setups, but the information might help determine if the performance drop is due to multipath or iet multi-target.

Bye.

[-- Attachment #1.2: Type: text/html, Size: 1208 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: understanding of multipathing and speed

[brem belguebli]

Can't it  just be the max disk perf is reached during the second test,
adding extra paths won't be of any help or maybe could worsen the
expected perfs ....



2010/7/6 Christophe Varoqui <christophe.varoqui@gmail.com>:
>> routing points to the correct interface per network. Maybe my test
>> (bonnie++)
>>
>> is not an appropriate one to show the difference in speed?
>>
> an interesting data point would be a perf run when iet target has only one
> target port on a bonded interface a multipath on client side.
>
> And maybe the other combinations. Th eset combinations would be :
> - client bonded, target not bonded
> - client bonded, target bonded (i understand this is the scenario with good
> performance you reported earlier)
> - client not bonded, target not bonded (perf expected from a single path in
> your testing)
> - client not bonded, target bonded
>
> the mixed setup would not be adequate for production as they require linked
> or single switch setups, but the information might help determine if the
> performance drop is due to multipath or iet multi-target.
>
> Bye.
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

Re: understanding of multipathing and speed

[bart.coninckx]

I'm afraid not, the values are lower than while doing a bonnie++ test locally on the storage server. I have to dig deep into memory, but I believe they were about 120 MB/sec locally and 80 MB/sec across iSCSI and multipathing. Eventhough this is a Gigabit connections, I guess the TCP overhead "steals" 30%. But with multipathing I would expect 120MB/sec again. 




----- Originele e-mail  -----
Van: "brem belguebli" <brem.belguebli@gmail.com>
Aan: "Christophe Varoqui" <christophe.varoqui@gmail.com>, "device-mapper development" <dm-devel@redhat.com>
Verzonden: Dinsdag 6 juli 2010 10:37:17 GMT +01:00 Amsterdam / Berlijn / Bern / Rome / Stockholm / Wenen
Onderwerp: Re: [dm-devel] understanding of multipathing and speed

Can't it  just be the max disk perf is reached during the second test,
adding extra paths won't be of any help or maybe could worsen the
expected perfs ....



2010/7/6 Christophe Varoqui <christophe.varoqui@gmail.com>:
>> routing points to the correct interface per network. Maybe my test
>> (bonnie++)
>>
>> is not an appropriate one to show the difference in speed?
>>
> an interesting data point would be a perf run when iet target has only one
> target port on a bonded interface a multipath on client side.
>
> And maybe the other combinations. Th eset combinations would be :
> - client bonded, target not bonded
> - client bonded, target bonded (i understand this is the scenario with good
> performance you reported earlier)
> - client not bonded, target not bonded (perf expected from a single path in
> your testing)
> - client not bonded, target bonded
>
> the mixed setup would not be adequate for production as they require linked
> or single switch setups, but the information might help determine if the
> performance drop is due to multipath or iet multi-target.
>
> Bye.
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: understanding of multipathing and speed

[bart.coninckx]

[-- Attachment #1.1: Type: text/plain, Size: 1922 bytes --]


----- Originele e-mail ----- 
Van: "Christophe Varoqui" <christophe.varoqui@gmail.com> 
Aan: "bart coninckx" <bart.coninckx@telenet.be> 
Cc: dm-devel@redhat.com 
Verzonden: Dinsdag 6 juli 2010 10:25:00 GMT +01:00 Amsterdam / Berlijn / Bern / Rome / Stockholm / Wenen 
Onderwerp: Re: [dm-devel] understanding of multipathing and speed 



> routing points to the correct interface per network. Maybe my test 
> (bonnie++) 
> 
> is not an appropriate one to show the difference in speed? 
> 
an interesting data point would be a perf run when iet target has only one target port on a bonded interface a multipath on client side. 

And maybe the other combinations. Th eset combinations would be : 
- client bonded, target not bonded 
- client bonded, target bonded (i understand this is the scenario with good performance you reported earlier) 
- client not bonded, target not bonded (perf expected from a single path in your testing) 
- client not bonded, target bonded 

the mixed setup would not be adequate for production as they require linked or single switch setups, but the information might help determine if the performance drop is due to multipath or iet multi-target. 

Bye. 
-- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel 

Christophe, 

I have previously done no bonding in this setup, that was just to refer to how I would multipathing would behave speed wise when configured with round-robin, namely just like bonding does with round robin: the speed is almost doupled. 

This is now a production machine, so juggling with bonding and non-bonding is somewhat difficult. I did do a tcpdump on both interfaces while doing a copy and there seems to be traffic on both interfaces while I copy a file, but the speed does not prove to get higher with interfaces available. 

Would it matter if I experiment with rr-min-io? This is now set to 100. 

Cheers, 


Bart 

[-- Attachment #1.2.1: Type: text/html, Size: 2260 bytes --]

[-- Attachment #1.2.2: Type: text/plain, Size: 838 bytes --]

> routing points to the correct interface per network. Maybe my test
> (bonnie++) 
> 
> is not an appropriate one to show the difference in speed? 
> 
an interesting data point would be a perf run when iet target has only one target port on a bonded interface a multipath on client side.

And maybe the other combinations. Th eset combinations would be :
- client bonded, target not bonded
- client bonded, target bonded (i understand this is the scenario with good performance you reported earlier)
- client not bonded, target not bonded (perf expected from a single path in your testing)
- client not bonded, target bonded

the mixed setup would not be adequate for production as they require linked or single switch setups, but the information might help determine if the performance drop is due to multipath or iet multi-target.

Bye.

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]