multipath performance

multipath performance

[Andy]

I did some basic dd tests to get an idea of the speed of multipath vs
individual devices, some of the numbers seem not to make sense.

I ran several test with dd, 4 at a time reading different parts of the
device.

multibus - round robin among paths rr_min_io=1, 4 paths
individual - each dd going to a separate path

dd with iflag=direct
	multibus : 120 MB/s
	individual : 115 MB/s

dd without direct
	multibus : 44MB/s (why is this so bad)
	individual : 160MB/s (why is this so good)

Why is multibus without direct the worst, yet individual devices without
direct is the fastest.  And why doesn't multibus perform about the same as
going to the individual devices (25% performance hit using multibus)?

Multibus is round-robin, how about a mode that sends io out on all paths
simultaneously?

Another thing I noticed during some of my test is that performance for a
single dd will increase in the multibus mode as I up the blocksize parameter
of dd.  

Thanks,

Andy

Re: multipath performance

[malahal]

Andy [genanr@emsphone.com] wrote:
> I did some basic dd tests to get an idea of the speed of multipath vs
> individual devices, some of the numbers seem not to make sense.
> 
> I ran several test with dd, 4 at a time reading different parts of the
> device.
> 
> multibus - round robin among paths rr_min_io=1, 4 paths
> individual - each dd going to a separate path
> 
> dd with iflag=direct
> 	multibus : 120 MB/s
> 	individual : 115 MB/s
> 
> dd without direct
> 	multibus : 44MB/s (why is this so bad)
> 	individual : 160MB/s (why is this so good)
> 
> Why is multibus without direct the worst, yet individual devices without
> direct is the fastest.  And why doesn't multibus perform about the same as
> going to the individual devices (25% performance hit using multibus)?

I am assuming that you are only reading from the device based on your
iflag. When you do O_DIRECT, you inherently disable read-ahead. Also
note that the read-ahead benefit would be insignificant as you increase
the block size.

without direct I/O, you end up with very big I/O's in individual mode.
Same can't be said in multibus mode as it tries to send small I/O's
(actually bio's) on different paths and it makes the system not to merge
any adjacent requests. Use iostat to see the average I/O size in each
case.

You should be able to get the same behavior even in direct I/O mode
provided you use very block size with dd.

Re: multipath performance

[Andy]

On Thu, Jan 24, 2008 at 05:59:39PM -0800, malahal@us.ibm.com wrote:
> Andy [genanr@emsphone.com] wrote:
> > I did some basic dd tests to get an idea of the speed of multipath vs
> > individual devices, some of the numbers seem not to make sense.
> > 
> > I ran several test with dd, 4 at a time reading different parts of the
> > device.
> > 
> > multibus - round robin among paths rr_min_io=1, 4 paths
> > individual - each dd going to a separate path
> > 
> > dd with iflag=direct
> > 	multibus : 120 MB/s
> > 	individual : 115 MB/s
> > 
> > dd without direct
> > 	multibus : 44MB/s (why is this so bad)
> > 	individual : 160MB/s (why is this so good)
> > 
> > Why is multibus without direct the worst, yet individual devices without
> > direct is the fastest.  And why doesn't multibus perform about the same as
> > going to the individual devices (25% performance hit using multibus)?
> 
> I am assuming that you are only reading from the device based on your
> iflag. When you do O_DIRECT, you inherently disable read-ahead. Also
> note that the read-ahead benefit would be insignificant as you increase
> the block size.
> 
> without direct I/O, you end up with very big I/O's in individual mode.
> Same can't be said in multibus mode as it tries to send small I/O's
> (actually bio's) on different paths and it makes the system not to merge
> any adjacent requests. 

Is there any way (yet) to increase the bio size?  It would be nice if
individual paths could perform large I/O's when needed.

Andy