[linux-lvm] Paralell IO performance with LVM and XFS

[linux-lvm] Paralell IO performance with LVM and XFS

[Miguel Angel de Vega]

Hi!

I've a Dual P2 450 with a Disk Array with 4 scsi disks with XFS.
With one single disc I've 10 Mb/s reading and writing, so I think that if I 
stripe all disc of the array, it can give me 4x10Mb = 40Mb/s more or less, 
but I only gets 10 Mg/s!!  
These disks usually gives up to 30 Mg/s in SGI machines and other single scsi 
disks in the Dual P2 (with the same controller and XFS) gives 20Mg/s, so i 
think that the problem must be LVM (the options maybe??? bad stripped 
options??). 

I show how mount all the Logical Volumen:

partionated all discs like 8e
#pvcreate /dev/sda1 ...
#vgcreate test /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
#lvcreate -n stripped -i 4 -I 64 test -L 30000M
(I tryed with -I 128, 256 and 512 but not big differences)

#mkfs -t xfs /dev/test/stripped  (only supports 4kb of block size)

what can be the problem??
thanks

Miguel Angel de Vega fernandez
SGO
Madrid - Spain

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Russell Coker]

On Thu, 17 Jan 2002 13:29, Miguel Angel de Vega wrote:
> I've a Dual P2 450 with a Disk Array with 4 scsi disks with XFS.
> With one single disc I've 10 Mb/s reading and writing, so I think that if I
> stripe all disc of the array, it can give me 4x10Mb = 40Mb/s more or less,
> but I only gets 10 Mg/s!!

Striping across 4 disks will never give you 4* the throughput of 1 disk, but 
you would hope for double the throughput.

> These disks usually gives up to 30 Mg/s in SGI machines and other single
> scsi disks in the Dual P2 (with the same controller and XFS) gives 20Mg/s,
> so i think that the problem must be LVM (the options maybe??? bad stripped
> options??).

How are you testing the speed of the disks?

I suggest that you do the following:
Test an ext2 partition on a single unpartitioned disk.
Test an XFS partition on a single unpartitioned disk.
Test XFS on a simple LVM setup (single disk, single LV in the VG).
Test XFS on the 4 disk LVM setup.
Run 2 copies of the test program at the same time on XFS on the 4 disk LVM.
Run 4 copies of the test program at the same time on XFS on the 4 disk LVM.

Then send us all the raw data from the above.  NB Bonnie++ has good support 
for running 2 and 4 copies of the test at the same time.

Also for interest, run zcav (from the Bonnie++ package) on a raw disk to show 
the throughput of the disk (without a file system).

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Patrick Caulfield]

On Thu, Jan 17, 2002 at 01:56:39PM +0100, Russell Coker wrote:
> On Thu, 17 Jan 2002 13:29, Miguel Angel de Vega wrote:
> > I've a Dual P2 450 with a Disk Array with 4 scsi disks with XFS.
> > With one single disc I've 10 Mb/s reading and writing, so I think that if I
> > stripe all disc of the array, it can give me 4x10Mb = 40Mb/s more or less,
> > but I only gets 10 Mg/s!!
> 
> Striping across 4 disks will never give you 4* the throughput of 1 disk, but 
> you would hope for double the throughput.

A graph of some tests I did using bonnie++ and striping on fibre-channel disks
is online at http://www.cix.co.uk/~tykepenguin/stripetests.png

The most obvious bottleneck there was the actually FC interface which ran
out of capacity at 3 stripes.

patrick

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[mitch]

Patrick,

I can't seem to get this URL to work ... it tells me the file ain't
there!?

-- Mitch


On Thu, 17 Jan 2002, Patrick Caulfield wrote:

<snip>
> A graph of some tests I did using bonnie++ and striping on fibre-channel disks
> is online at http://www.cix.co.uk/~tykepenguin/stripetests.png
>
<snip>

> patrick

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Patrick Caulfield]

On Thu, Jan 17, 2002 at 10:43:45AM -0600, mitch@mdmiller.com wrote:
> Patrick,
> 
> I can't seem to get this URL to work ... it tells me the file ain't
> there!?

Sorry, superfluous s got in there after I tested it. try:

http://www.cix.co.uk/~tykepenguin/stripetest.png

...it's not that impressive, honest :)

patrick

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Austin Gonyou]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

Hmm..it seems like writing with LVM striping isn't very good. 
10MB/s or so. Is this IDE only, or SCSI too?

On Thu, 2002-01-17 at 10:43, Patrick Caulfield wrote:
> On Thu, Jan 17, 2002 at 10:43:45AM -0600, mitch@mdmiller.com wrote:
> > Patrick,
> > 
> > I can't seem to get this URL to work ... it tells me the file ain't
> > there!?
> 
> Sorry, superfluous s got in there after I tested it. try:
> 
> http://www.cix.co.uk/~tykepenguin/stripetest.png
> 
> ...it's not that impressive, honest :)
> 
> patrick
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
-- 
Austin Gonyou
Systems Architect, CCNA
Coremetrics, Inc.
Phone: 512-698-7250
email: austin@coremetrics.com

"It is the part of a good shepherd to shear his flock, not to skin it."
Latin Proverb

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Patrick Caulfield]

On Thu, Jan 17, 2002 at 11:06:31AM -0600, Austin Gonyou wrote:
> Hmm..it seems like writing with LVM striping isn't very good. 
> 10MB/s or so. Is this IDE only, or SCSI too?

It's the random stuff that's suffering. Sequential Input seems to
scale as far as the fibre-channel interface would allow at about 5M/s, and the
writes start to flatten out after two stripes.

The interface was an emulex Fibre-channel but the disks are quote slow ones.
Effectively it's SCSI.

patrick

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Russell Coker]

On Thu, 17 Jan 2002 16:21, Patrick Caulfield wrote:
> On Thu, Jan 17, 2002 at 01:56:39PM +0100, Russell Coker wrote:
> > Striping across 4 disks will never give you 4* the throughput of 1 disk,
> > but you would hope for double the throughput.
>
> A graph of some tests I did using bonnie++ and striping on fibre-channel
> disks is online at http://www.cix.co.uk/~tykepenguin/stripetest.png

Firstly, I suggest that you skip the per-char tests when graphing such 
things, it's generally not very interesting and just clutters the graph.

Next the seeks, block-io, and create/delete test results belong on different 
graphs because the Y axis measures different things, and because it needs 
different scales.

Finally what graphing program did you use?

> The most obvious bottleneck there was the actually FC interface which ran
> out of capacity at 3 stripes.

Why are you so certain that it's the FC interface at fault?  The FC interface 
should be able to do more than 60MB/s.

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Patrick Caulfield]

On Thu, Jan 17, 2002 at 06:39:58PM +0100, Russell Coker wrote:
> On Thu, 17 Jan 2002 16:21, Patrick Caulfield wrote:
> > On Thu, Jan 17, 2002 at 01:56:39PM +0100, Russell Coker wrote:
> > > Striping across 4 disks will never give you 4* the throughput of 1 disk,
> > > but you would hope for double the throughput.
> >
> > A graph of some tests I did using bonnie++ and striping on fibre-channel
> > disks is online at http://www.cix.co.uk/~tykepenguin/stripetest.png
> 
> Firstly, I suggest that you skip the per-char tests when graphing such 
> things, it's generally not very interesting and just clutters the graph.
> 
> Next the seeks, block-io, and create/delete test results belong on different 
> graphs because the Y axis measures different things, and because it needs 
> different scales.

Yeah, I just graphed the whole output - I only did it for my own interest
really, it's not meant to be an official test or especially meaningful. I just
thought it might be interesting. The only reason it's still on my web site is
that I forgot to remove it.

> Finally what graphing program did you use?

Err...I'd rather not say...
 
> > The most obvious bottleneck there was the actually FC interface which ran
> > out of capacity at 3 stripes.
> 
> Why are you so certain that it's the FC interface at fault?  The FC interface 
> should be able to do more than 60MB/s.

I don't suppose I am that certain to be honest but I had so much trouble with
the Linux drivers for this card that at the time I was prepared to blame
everything from Global Warming downwards on it. 

patrick

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Austin Gonyou]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

What card were you using? 

On Thu, 2002-01-17 at 11:58, Patrick Caulfield wrote:
> > Why are you so certain that it's the FC interface at fault?  The FC
> interface 
> > should be able to do more than 60MB/s.
> 
> I don't suppose I am that certain to be honest but I had so much trouble
> with
> the Linux drivers for this card that at the time I was prepared to blame
> everything from Global Warming downwards on it. 
> 
> patrick
> 
-- 
Austin Gonyou
Systems Architect, CCNA
Coremetrics, Inc.
Phone: 512-698-7250
email: austin@coremetrics.com

"It is the part of a good shepherd to shear his flock, not to skin it."
Latin Proverb

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Stephen Lord]

Miguel Angel de Vega wrote:

>Hi!
>
>I've a Dual P2 450 with a Disk Array with 4 scsi disks with XFS.
>With one single disc I've 10 Mb/s reading and writing, so I think that if I 
>stripe all disc of the array, it can give me 4x10Mb = 40Mb/s more or less, 
>but I only gets 10 Mg/s!!  
>These disks usually gives up to 30 Mg/s in SGI machines and other single scsi 
>disks in the Dual P2 (with the same controller and XFS) gives 20Mg/s, so i 
>think that the problem must be LVM (the options maybe??? bad stripped 
>options??). 
>

Because LVM has difficulties dealing with the differing request sizes issued
by XFS, we feed it requests in small chunks (which means lots of buffer 
heads).
XFS has some 512 byte chunks of metadata, it tends to want to do metadata
I/O in sizes of 512 bytes, 4K, 8K and various random sizes upto 32K for 
the log writes.

Since request mapping in LVM basically consists of taking an existing 
buffer head and
giving it a different address xfs takes a very conservative approach. 
Striping with the
raid code should give you considerably better performance.

Now if someone would go off and make lvm work with the bio interface in 
2.5 things
might go pretty fast.

Steve

>
>
>I show how mount all the Logical Volumen:
>
>partionated all discs like 8e
>#pvcreate /dev/sda1 ...
>#vgcreate test /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
>#lvcreate -n stripped -i 4 -I 64 test -L 30000M
>(I tryed with -I 128, 256 and 512 but not big differences)
>
>#mkfs -t xfs /dev/test/stripped  (only supports 4kb of block size)
>
>what can be the problem??
>thanks
>
>Miguel Angel de Vega fernandez
>SGO
>Madrid - Spain
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@sistina.com
>http://lists.sistina.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Heinz J . Mauelshagen]

On Thu, Jan 17, 2002 at 09:17:21PM -0600, Stephen Lord wrote:
> Miguel Angel de Vega wrote:
> 
> >Hi!
> >
> >I've a Dual P2 450 with a Disk Array with 4 scsi disks with XFS.
> >With one single disc I've 10 Mb/s reading and writing, so I think that if I 
> >stripe all disc of the array, it can give me 4x10Mb = 40Mb/s more or less, 
> >but I only gets 10 Mg/s!!  
> >These disks usually gives up to 30 Mg/s in SGI machines and other single scsi 
> >disks in the Dual P2 (with the same controller and XFS) gives 20Mg/s, so i 
> >think that the problem must be LVM (the options maybe??? bad stripped 
> >options??). 
> >
> 
> Because LVM has difficulties dealing with the differing request sizes issued
> by XFS, we feed it requests in small chunks (which means lots of buffer 
> heads).
> XFS has some 512 byte chunks of metadata, it tends to want to do metadata
> I/O in sizes of 512 bytes, 4K, 8K and various random sizes upto 32K for 
> the log writes.
> 
> Since request mapping in LVM basically consists of taking an existing 
> buffer head and
> giving it a different address xfs takes a very conservative approach. 
> Striping with the
> raid code should give you considerably better performance.
> 
> Now if someone would go off and make lvm work with the bio interface in 
> 2.5 things
> might go pretty fast.

Probably true but I am afraid that we will be able to spend much time
on the LVM1 driver in 2.5. The basic bio changes have already been done
AFAIS briefly, but the splitting of large(r) io request onto several extents
hasn't.
We would really appreciate further help ITR.

The new device-mapper driver soon to be released will support those splits
and hopefully will be ready for integration into 2.5 in a couple of weeks.

We'll release it soon plus the new version 2 tools for public testing.

Heinz

> 
> Steve
> 
> >
> >
> >I show how mount all the Logical Volumen:
> >
> >partionated all discs like 8e
> >#pvcreate /dev/sda1 ...
> >#vgcreate test /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
> >#lvcreate -n stripped -i 4 -I 64 test -L 30000M
> >(I tryed with -I 128, 256 and 512 but not big differences)
> >
> >#mkfs -t xfs /dev/test/stripped  (only supports 4kb of block size)
> >
> >what can be the problem??
> >thanks
> >
> >Miguel Angel de Vega fernandez
> >SGO
> >Madrid - Spain
> >
> >_______________________________________________
> >linux-lvm mailing list
> >linux-lvm@sistina.com
> >http://lists.sistina.com/mailman/listinfo/linux-lvm
> >read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> >
> 
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Paralell IO performance with LVM and XFS

[Patrick Caulfield]

On Thu, Jan 17, 2002 at 05:58:32PM +0000, Patrick Caulfield wrote:
> On Thu, Jan 17, 2002 at 06:39:58PM +0100, Russell Coker wrote:
> > 
> > Why are you so certain that it's the FC interface at fault?  The FC interface 
> > should be able to do more than 60MB/s.

Just for the record, I can't get more tham 60M/s out of it using MD striping
either.

patrick