SCSI Disk layer performance

SCSI Disk layer performance

[Chris Worley]

My FC array seems to be faster than Linux can handle.

Even though I'm using FC-2 cards, Linux seems to put
an artificial limit at FC-1 speeds.  This is true in two cases:

        1) dd to the raw device (actually, the limit is at about
        120MB/s).
        2) dd over the file system, even reading/writing multiple files
        simultaneously (about 90MB/s). 
        
Using sgp_dd, I can easily get >190MB/s, but neither sg_dd or dd do as
well (so, it's not the qla2300 module... it's something above).

Is there any method that might assure more parallel I/O in the SCSI
layer?  Is 2.6 the answer?

I've also seen some differences between 2.4 kernels.  2.4.21 gets
120MB/s via "dd" to the raw device, until the sg module is loaded, then
it drops down to less than 100MB/s, and there are frequent memory
warnings.  2.4.23 and .24 don't get the warnings, but irrespective of
the sg module being loaded, get the slower speed on the raw device.  I'm
guessing that'll translate to even slower speeds on the file system, but
I've yet to test that.

Furthermore, my reads seem to be especially slow.  The SAN maker thinks
that this might be due to the I/O scheduling in the SCSI Disk layer. 
Specifically, it's easy to see from the SAN perspective (using a
statistics utility that shows read and write scsi command lengths) that
reads are broken into small chunks by the SCSI Disk layer.  In watching
read lengths from the SAN, using "sg_dd", you can exactly specify read
length, and no I/O's get broken up into chunks smaller than the bpt*512
(what is requested on the sg_dd command line is what the SAN sees).  In
"dd", you can specify a read block size on the command line, but the SAN
shows that Linux is breaking that up into much smaller chunks.

Is there any way to force Linux to use large chunks?

Thanks,

Chris 
-- 
Chris Worley <chrisw@lnxi.com>
Linux Networx

Re: SCSI Disk layer performance

[Christoph Hellwig]

On Wed, Jan 21, 2004 at 02:31:08PM -0700, Chris Worley wrote:
> My FC array seems to be faster than Linux can handle.
> 
> Even though I'm using FC-2 cards, Linux seems to put
> an artificial limit at FC-1 speeds.  This is true in two cases:

What kernel tree do you use?  If you're looking for decent scsi
performance use linux 2.6 or the SuSE/RH vendor trees.  Stock Linux 2.4
is basically unusable for higher scsi loads in SMP enviroments.

Re: SCSI Disk layer performance

[Xose Vazquez Perez]

Christoph Hellwig wrote:

> What kernel tree do you use?  If you're looking for decent scsi
> performance use linux 2.6 or the SuSE/RH vendor trees.  Stock Linux 2.4
                                           ^^^^^^
_Enterprise_ kernels, RHEL or SLES.
Fedora Core kernel is, more or less, a plain kernel.org.

I'm runnig RHEL 2.4.21-9.EL kernel and it's _much better_ than any 2.4.xx
kernel.org. I could call it 2.5 because it's a mix of 2.4 and 2.6 ;-)

-- 
Software is like sex, it's better when it's bug free.

Re: SCSI Disk layer performance

[Fabien Salvi]

Christoph Hellwig a e'crit:
> On Wed, Jan 21, 2004 at 02:31:08PM -0700, Chris Worley wrote:
> 
>>My FC array seems to be faster than Linux can handle.
>>
>>Even though I'm using FC-2 cards, Linux seems to put
>>an artificial limit at FC-1 speeds.  This is true in two cases:
> 
> 
> What kernel tree do you use?  If you're looking for decent scsi
> performance use linux 2.6 or the SuSE/RH vendor trees.  Stock Linux 2.4
> is basically unusable for higher scsi loads in SMP enviroments.

Can you give more details about this ?

Instead of using vendor trees, is it possible to use specific and 
validated patchs about this ?

vendors trees are awfully patched and difficult to maintain/upgrade for 
customers...

-- 
Fabien S

Re: SCSI Disk layer performance

[George Magklaras]

Xose Vazquez Perez wrote:
> Christoph Hellwig wrote:
> 
> 
>>What kernel tree do you use?  If you're looking for decent scsi
>>performance use linux 2.6 or the SuSE/RH vendor trees.  Stock Linux 2.4
> 
>                                            ^^^^^^
> _Enterprise_ kernels, RHEL or SLES.
> Fedora Core kernel is, more or less, a plain kernel.org.
> 
> I'm runnig RHEL 2.4.21-9.EL kernel and it's _much better_ than any 2.4.xx
> kernel.org. I could call it 2.5 because it's a mix of 2.4 and 2.6 ;-)
> 
The 2.4.21-9EL might have clever things backported into it, but it is 
still missing essential things that can make a difference in the block 
layer, like an anticipatory I/O schedule. My impression is that all 
2.4 kernels use the Linus Elevator. That works well with moderate SMP 
loads, but it does not perform well on my large-ish Hardware RAID0 
1.1Tbyte ext3 partition on a Power Vault 220S with a Dell PERC4 host 
adapter. The 2.6.1 does work a lot more better in that sense, giving 
me an average read of 90 Mbytes/sec as opposed to 
60+somethingMbytes/sec I used to get before with 2.4.21-9ELsmp.

The new I/O schedulers (let alone driver maturity issues) are the way 
to go for maximum performance! I won't place -for the moment- a 2.6 on 
a production grade system, but in a couple of releases, I will put 2.4 
in the museum :-) with respect to SCSI I/O performance.

Regards,
-- 
--
George B. Magklaras

Computer Systems Engineer/UNIX Systems Administrator
The Biotechnology Centre of Oslo

Re: SCSI Disk layer performance

[Patrick Mansfield]

On Thu, Jan 29, 2004 at 01:26:50PM +0100, George Magklaras wrote:

> The 2.4.21-9EL might have clever things backported into it, but it is 
> still missing essential things that can make a difference in the block 
> layer, like an anticipatory I/O schedule. My impression is that all 
> 2.4 kernels use the Linus Elevator. That works well with moderate SMP 
> loads, but it does not perform well on my large-ish Hardware RAID0 
> 1.1Tbyte ext3 partition on a Power Vault 220S with a Dell PERC4 host 
> adapter. The 2.6.1 does work a lot more better in that sense, giving 
> me an average read of 90 Mbytes/sec as opposed to 
> 60+somethingMbytes/sec I used to get before with 2.4.21-9ELsmp.
> 
> The new I/O schedulers (let alone driver maturity issues) are the way 
> to go for maximum performance! I won't place -for the moment- a 2.6 on 
> a production grade system, but in a couple of releases, I will put 2.4 
> in the museum :-) with respect to SCSI I/O performance.

How does the anticipatory scheduler compare to the deadline scheduler on
your system?

What types of IO are you doing?

-- Patrick Mansfield

Re: SCSI Disk layer performance

[Christoph Hellwig]

On Fri, Jan 23, 2004 at 12:02:46PM +0100, Fabien Salvi wrote:
> > What kernel tree do you use?  If you're looking for decent scsi
> > performance use linux 2.6 or the SuSE/RH vendor trees.  Stock Linux 2.4
> > is basically unusable for higher scsi loads in SMP enviroments.
> 
> Can you give more details about this ?
> 
> Instead of using vendor trees, is it possible to use specific and 
> validated patchs about this ?

It's rather hard as there's a lot of them and they depend on each other.
Doug Ledford sayd he's going to sort put it into the 2.4 BK repos
at linux-scsi.bkbits.net one day, you could monitor it.

Re: SCSI Disk layer performance

[Doug Ledford]

On Thu, 2004-01-29 at 13:59, Christoph Hellwig wrote:
> On Fri, Jan 23, 2004 at 12:02:46PM +0100, Fabien Salvi wrote:
> > > What kernel tree do you use?  If you're looking for decent scsi
> > > performance use linux 2.6 or the SuSE/RH vendor trees.  Stock Linux 2.4
> > > is basically unusable for higher scsi loads in SMP enviroments.
> > 
> > Can you give more details about this ?
> > 
> > Instead of using vendor trees, is it possible to use specific and 
> > validated patchs about this ?
> 
> It's rather hard as there's a lot of them and they depend on each other.
> Doug Ledford sayd he's going to sort put it into the 2.4 BK repos
> at linux-scsi.bkbits.net one day, you could monitor it.

Yes, I'm still working on it.  The sheer number of changes we make to
the SCSI stack internally makes it difficult to sort the bug fixes from
all the other stuff.  And, of course, if you call totally piss poor SCSI
performance a bug, then a lot of the patches are in fact bug fixes since
they drastically impact the 2.4 SCSI stack speed ;-)

In any case, I've sorted out the bare bugfixes (which is relatively
small actually), and am now putting together the 2.4-everything tree
that has all the scsi stuff in it.  Keep in mind that it isn't just a
"throw our patches on the current 2.4 bk tree from Marcelo".  I have to
merge the patches and check for areas where new code between 2.4.21 and
current makes the things like the iorl patch have bugs.

Anyway, it's coming.

-- 
  Doug Ledford <dledford@redhat.com>     919-754-3700 x44233
         Red Hat, Inc.
         1801 Varsity Dr.
         Raleigh, NC 27606