Performance issue with 2.6 md raid0

Performance issue with 2.6 md raid0

[Mattias Wadenstein]

Greetings.

While testing a file server to store a couple of TB in resonably large
files (>1G), I noticed an odd performance behaviour with the md raid0 in a
pristine 2.6.2 kernel as compared to a 2.4.24 kernel.

When striping two md raid5:s, instead of going from about 160-200MB/s for
a single raid5 to 300M/s for the raid0 in 2.4.24, the 2.6.2 kernel gave
135M/s in single stream read performance.

The setup:
2 x 2.0 GHz Opteron 248, 4 gigs of ram (running 32-bit kernels)
2 x 8-port 3ware sata raid cards, acting as disk controllers (no hw raid)
16 x Maxtor 250-gig 7k2 rpm sata drives.
1 x system drive on onboard pata doing pretty much nothing.

The sata drives are configured in 2 8-disk md raid5s, not hw raid for
performance reasons, we get better numbers from the md driver in that case
than the hw raid on the card. Then I have created a raid0 of these two
raid5 devices.

I used jfs for these numbers, I have only seen minor differences in speed
in the single-stream case on this hardware though for different
filesystems I have tested (ext2, xfs, jfs, reiserfs). And the filesystem
numbers are reflected pretty close by doing a dd from /dev/md10. The same
goes for increasing the chunk-size to 4M instead of 32k, roughly the same
numbers. The system is not doing anything else.

The results (as meassured by bonnie++ -f -n0, all numbers in kB/s, all
numbers for a single stream[*]):
2.4.24, one of the raid5s: Write: 138273, Read: 212474
2.4.24, raid0 of two raid5s: Write: 215827, Read: 303388
2.6.2, one of the raid5s: Write: 159271, Read: 161327
2.6.2, raid0 of two raid5s: Write: 280691, Read: 134622

It is the last read value that really stands out.

Any ideas? Anything I should try? More info wanted?

Please Cc: me as I'm not a subscriber to this list.

[*]: For multiple streams, say a dozen or so readers, the aggregate
performance on the 2.6.2 raid0 went down to about 60MB/s, which is a bit
of a real performance problem for the intended use, I'd like to at least
saturate a single gigE interface and hopefully two with that many readers.

/Mattias Wadenstein

Re: Performance issue with 2.6 md raid0

[Nick Piggin]

Mattias Wadenstein wrote:

>Greetings.
>
>While testing a file server to store a couple of TB in resonably large
>files (>1G), I noticed an odd performance behaviour with the md raid0 in a
>pristine 2.6.2 kernel as compared to a 2.4.24 kernel.
>
>When striping two md raid5:s, instead of going from about 160-200MB/s for
>a single raid5 to 300M/s for the raid0 in 2.4.24, the 2.6.2 kernel gave
>135M/s in single stream read performance.
>
>

Can you try booting with elevator=deadline please?

Re: Performance issue with 2.6 md raid0

[Timothy Miller]

Nick Piggin wrote:
>
> 
> Can you try booting with elevator=deadline please?
> 

Is there a "what to do before you make a bug report" FAQ?  If not, we 
need one.



1. If I'm having performance issues, what sorts of tests should I run 
before reporting?

1.1. If it's an I/O performance problem, try entering 
"elevator=deadline" at the lilo boot prompt and see what difference it 
makes.

Re: Performance issue with 2.6 md raid0

[Mattias Wadenstein]

On Fri, 6 Feb 2004, Nick Piggin wrote:

> Mattias Wadenstein wrote:
>
> >Greetings.
> >
> >While testing a file server to store a couple of TB in resonably large
> >files (>1G), I noticed an odd performance behaviour with the md raid0 in a
> >pristine 2.6.2 kernel as compared to a 2.4.24 kernel.
> >
> >When striping two md raid5:s, instead of going from about 160-200MB/s for
> >a single raid5 to 300M/s for the raid0 in 2.4.24, the 2.6.2 kernel gave
> >135M/s in single stream read performance.
>
> Can you try booting with elevator=deadline please?

Ok, then I get 253267 kB/s write and 153187 kB/s read from the raid0. A
bit better, but still nowhere near the 2.4.24 numbers.

For a single raid5, 158028 kB/s write and 162944 kB/s read.

/Mattias Wadenstein

Re: Performance issue with 2.6 md raid0

[Nick Piggin]

Mattias Wadenstein wrote:

>On Fri, 6 Feb 2004, Nick Piggin wrote:
>
>
>>Mattias Wadenstein wrote:
>>
>>
>>>Greetings.
>>>
>>>While testing a file server to store a couple of TB in resonably large
>>>files (>1G), I noticed an odd performance behaviour with the md raid0 in a
>>>pristine 2.6.2 kernel as compared to a 2.4.24 kernel.
>>>
>>>When striping two md raid5:s, instead of going from about 160-200MB/s for
>>>a single raid5 to 300M/s for the raid0 in 2.4.24, the 2.6.2 kernel gave
>>>135M/s in single stream read performance.
>>>
>>Can you try booting with elevator=deadline please?
>>
>
>Ok, then I get 253267 kB/s write and 153187 kB/s read from the raid0. A
>bit better, but still nowhere near the 2.4.24 numbers.
>
>For a single raid5, 158028 kB/s write and 162944 kB/s read.
>
>

Any idea what is holding back performance? Is it IO or CPU bound?
Can you get a profile of each kernel while doing a read please?

Re: Performance issue with 2.6 md raid0

[Neil Brown]

On Friday February 6, piggin@cyberone.com.au wrote:
> 
> 
> Mattias Wadenstein wrote:
> 
> >On Fri, 6 Feb 2004, Nick Piggin wrote:
> >
> >
> >>Mattias Wadenstein wrote:
> >>
> >>
> >>>Greetings.
> >>>
> >>>While testing a file server to store a couple of TB in resonably large
> >>>files (>1G), I noticed an odd performance behaviour with the md raid0 in a
> >>>pristine 2.6.2 kernel as compared to a 2.4.24 kernel.
> >>>
> >>>When striping two md raid5:s, instead of going from about 160-200MB/s for
> >>>a single raid5 to 300M/s for the raid0 in 2.4.24, the 2.6.2 kernel gave
> >>>135M/s in single stream read performance.
> >>>
> >>Can you try booting with elevator=deadline please?
> >>
> >
> >Ok, then I get 253267 kB/s write and 153187 kB/s read from the raid0. A
> >bit better, but still nowhere near the 2.4.24 numbers.
> >
> >For a single raid5, 158028 kB/s write and 162944 kB/s read.
> >
> >
> 
> Any idea what is holding back performance? Is it IO or CPU bound?
> Can you get a profile of each kernel while doing a read please?
> 

Possibly the read-ahead size isn't getting set correctly.

What chunksize are you using on the raid0?
Are you free to rebuild the raid0 array?
If so, please rebuild it with a chunksize that is 2 or 4 times the
size of a raid5 stripe (i.e. raid5-chunksize * (raid5-drives - 1) ).

If not, can you change:

		if (mddev->queue->backing_dev_info.ra_pages < stripe)
			mddev->queue->backing_dev_info.ra_pages = stripe;

at about line 327 of drives/md/raid1.c to something like:

		mddev->queue->backing_dev_info.ra_pages = stripe * 32;

or whatever is needed to make ra_pages big enough to gold a couple of
raid5 stripes.

Thanks,
NeilBrown

Re: Performance issue with 2.6 md raid0

[Mattias Wadenstein]

On Fri, 6 Feb 2004, Neil Brown wrote:

> On Friday February 6, piggin@cyberone.com.au wrote:
> > Mattias Wadenstein wrote:
> > >On Fri, 6 Feb 2004, Nick Piggin wrote:
> > >>Mattias Wadenstein wrote:
> > >>>
> > >>>While testing a file server to store a couple of TB in resonably large
> > >>>files (>1G), I noticed an odd performance behaviour with the md raid0 in a
> > >>>pristine 2.6.2 kernel as compared to a 2.4.24 kernel.
> > >>>
> > >>>When striping two md raid5:s, instead of going from about 160-200MB/s for
> > >>>a single raid5 to 300M/s for the raid0 in 2.4.24, the 2.6.2 kernel gave
> > >>>135M/s in single stream read performance.
> > >>>
> > >>Can you try booting with elevator=deadline please?
> > >
> > >Ok, then I get 253267 kB/s write and 153187 kB/s read from the raid0. A
> > >bit better, but still nowhere near the 2.4.24 numbers.
> > >
> > >For a single raid5, 158028 kB/s write and 162944 kB/s read.
> >
> > Any idea what is holding back performance? Is it IO or CPU bound?

The CPU usage is not significant for the lower numbers (<30%) and seems
linear to the delivered bandwidth for the faster configurations (up to
80% or so when approaching 250M/s write and 360M/s read).

> > Can you get a profile of each kernel while doing a read please?
>
> Possibly the read-ahead size isn't getting set correctly.
>
> What chunksize are you using on the raid0?

I was only using 32k, but when I tried changing this to 4 megs I got no
improvement, leading me to guess that this wasn't it. Unfortunately I
changed it to "4M", something which mkraid happily accepted but
interpreted as "4k".

I didn't verify this against /proc/mdstat then, just noticed this now.

> Are you free to rebuild the raid0 array?

Yeah, the raid5s too if needed, but those take a while to resync before
benchmarking.

> If so, please rebuild it with a chunksize that is 2 or 4 times the
> size of a raid5 stripe (i.e. raid5-chunksize * (raid5-drives - 1) ).

Yes, this is much better. I have a 64k raid5 chunksize. A chunksize of
4096k gives 265313kB/s write and 368228kB/s for a single stream, 512k
chunksize also gives decent performance (slightly better read and rewrite
performance, somewhat lower read performance).

I'll continue tweaking and testing some, please respond (and Cc: me) if
you want to know anything more or if I should try something special.

/Mattias Wadenstein

Re: Performance issue with 2.6 md raid0

[Bill Davidsen]

Mattias Wadenstein wrote:
> Greetings.
> 
> While testing a file server to store a couple of TB in resonably large
> files (>1G), I noticed an odd performance behaviour with the md raid0 in a
> pristine 2.6.2 kernel as compared to a 2.4.24 kernel.
> 
> When striping two md raid5:s, instead of going from about 160-200MB/s for
> a single raid5 to 300M/s for the raid0 in 2.4.24, the 2.6.2 kernel gave
> 135M/s in single stream read performance.
> 
> The setup:
> 2 x 2.0 GHz Opteron 248, 4 gigs of ram (running 32-bit kernels)
> 2 x 8-port 3ware sata raid cards, acting as disk controllers (no hw raid)
> 16 x Maxtor 250-gig 7k2 rpm sata drives.
> 1 x system drive on onboard pata doing pretty much nothing.
> 
> The sata drives are configured in 2 8-disk md raid5s, not hw raid for
> performance reasons, we get better numbers from the md driver in that case
> than the hw raid on the card. Then I have created a raid0 of these two
> raid5 devices.
> 
> I used jfs for these numbers, I have only seen minor differences in speed
> in the single-stream case on this hardware though for different
> filesystems I have tested (ext2, xfs, jfs, reiserfs). And the filesystem
> numbers are reflected pretty close by doing a dd from /dev/md10. The same
> goes for increasing the chunk-size to 4M instead of 32k, roughly the same
> numbers. The system is not doing anything else.
> 
> The results (as meassured by bonnie++ -f -n0, all numbers in kB/s, all
> numbers for a single stream[*]):
> 2.4.24, one of the raid5s: Write: 138273, Read: 212474
> 2.4.24, raid0 of two raid5s: Write: 215827, Read: 303388
> 2.6.2, one of the raid5s: Write: 159271, Read: 161327
> 2.6.2, raid0 of two raid5s: Write: 280691, Read: 134622
> 
> It is the last read value that really stands out.
> 
> Any ideas? Anything I should try? More info wanted?
> 
> Please Cc: me as I'm not a subscriber to this list.
> 
> [*]: For multiple streams, say a dozen or so readers, the aggregate
> performance on the 2.6.2 raid0 went down to about 60MB/s, which is a bit
> of a real performance problem for the intended use, I'd like to at least
> saturate a single gigE interface and hopefully two with that many readers.

I believe what you see between 2.4 and 2.6 is just lack of readahead, 
and you should increase this for your read performance. With no other 
changes you should be able to pretty much match performance between 2.4 
and 2.6.

However, after some years of trying to tune stripe size on both Linux 
and AIX news servers, I suspect that the stripe size you have is too 
small. The optimal stripe size under heavy load seems to be at least 2x 
the largest typical io size, so that you don't have to seek on multiple 
drives all the time. Note "at least," in most cases you can go larger 
than that unless you create and delete a lot of files, in which case you 
can wind up with the inodes on one drive, which can be a real issue.

For this reason I find that most install programs are about worthless, 
they simply use too small a stripe size and generate a lot of overhead 
when used, because too many io are on multiple devices. That's good for 
huge io on separate busses where bus bandwidth is a factor, but very bad 
when most of the time is latency.

-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979