The good, the bad & the ugly (or VM, block devices, and SCSI :-)

The good, the bad & the ugly (or VM, block devices, and SCSI :-)

[Stephan von Krawczynski]

Hello all,

this is a message especially for:

Gerard Roudier <groudier@club-internet.fr>, symbios driver
Justin T. Gibbs, adaptec driver
Andrea and Rik, VM
Linus, the man with the big picture :-)

Everything started with a note from a friend who tried some small all-in-one
box with a sym53c1010 onboard. He (an all-time adaptec user like me) told me
the small box feels like flying, compared to his at-home box (with adaptec).
This made me curious for trying myself. I bought a Tekram DC390 U3W (which is
in fact a relabeled U3D) with symbios chipset. I simply replaced the controller
in my box (compiled a kernel with both drivers of course) and gave it a try
with bonnie. It did not look impressing. Effectively adaptec A29160 and DC390
had the same read and write speeds, only noticeable was that tekram performed
twice as many seeks as adaptec. This was reproducable in all bonnie-tests. Hm,
that should not make the big difference. Anyway I was too lazy to put the
adaptec back in and continued working (for several days). Today it hit me:
As Linus said something about testing pre6 I gave it a try and did the usual
nfs-copy, cd read test. I was pretty astonished to see the tekram perform very
well under heavy I/O load, here are the numbers:

A29160:                                     symbios:

cd read without nfs-load:                   cd read without nfs-load
2998,9 kB                                   3619,3 kB
3168,2 kB                                   3611,1 kB
2968,4 kB                                   3620,2 kB

cd read with nfs load:                      cd read with nfs load
1926,2 kB                                   3408,1 kB
2123,4 kB                                   3395,2 kB
2539,4 kB                                   3605,1 kB
2631,9 kB                                   3605,8 kB

The rest of the hardware involved is completely the same, only the controller
boards got exchanged. Another thing quite remarkable: during symbios tests the
network throughput derived from nfs load is _higher_ and looks more stable.
Whereas during adaptec the whole picture looks like having hiccup. More to say:
starting an application  during the tests results in waiting a bit (some 10-20
seconds) with tekram, but waiting pretty long (or even forever, "the ugly" ;-)
while using adaptec. This is particularly interesting for the vm guys since all
the scene is in high vm load with around 3-5 MB of free mem and a damn lot of
page cache. So if you try something around vm I can only urge you to perform
tests that do _no_ I/O at all, because you may be greatly bitten by your
controller (or its driver).
Another thing to mention: during the last cd-read tests with tekram setup I
already have been deeply impressed by the driver, so I decided to stress it
some more and start applications (like mozilla) in the background. And in the
end I was even more impressed, because it turned out (you can see in the last
two figures), that it got even _faster_. Obviously I cannot explain why.
If anybody wants me to test anything, feel free to ask.

My personal opinion: Justin has work to do.

Regards,
Stephan

Re: The good, the bad & the ugly (or VM, block devices, and SCSI :-)

[Gérard Roudier]

On Wed, 31 Oct 2001, Stephan von Krawczynski wrote:

> Hello all,
>
> this is a message especially for:
>
> Gerard Roudier <groudier@club-internet.fr>, symbios driver
> Justin T. Gibbs, adaptec driver
> Andrea and Rik, VM
> Linus, the man with the big picture :-)
>
> Everything started with a note from a friend who tried some small all-in-one
> box with a sym53c1010 onboard. He (an all-time adaptec user like me) told me
> the small box feels like flying, compared to his at-home box (with adaptec).
> This made me curious for trying myself. I bought a Tekram DC390 U3W (which is
> in fact a relabeled U3D) with symbios chipset. I simply replaced the controller
> in my box (compiled a kernel with both drivers of course) and gave it a try
> with bonnie. It did not look impressing. Effectively adaptec A29160 and DC390
> had the same read and write speeds, only noticeable was that tekram performed
> twice as many seeks as adaptec. This was reproducable in all bonnie-tests. Hm,
> that should not make the big difference. Anyway I was too lazy to put the
> adaptec back in and continued working (for several days). Today it hit me:
> As Linus said something about testing pre6 I gave it a try and did the usual
> nfs-copy, cd read test. I was pretty astonished to see the tekram perform very
> well under heavy I/O load, here are the numbers:
>
> A29160:                                     symbios:
>
> cd read without nfs-load:                   cd read without nfs-load
> 2998,9 kB                                   3619,3 kB
> 3168,2 kB                                   3611,1 kB
> 2968,4 kB                                   3620,2 kB
>
> cd read with nfs load:                      cd read with nfs load
> 1926,2 kB                                   3408,1 kB
> 2123,4 kB                                   3395,2 kB
> 2539,4 kB                                   3605,1 kB
> 2631,9 kB                                   3605,8 kB
>
> The rest of the hardware involved is completely the same, only the controller
> boards got exchanged. Another thing quite remarkable: during symbios tests the
> network throughput derived from nfs load is _higher_ and looks more stable.
> Whereas during adaptec the whole picture looks like having hiccup. More to say:
> starting an application  during the tests results in waiting a bit (some 10-20
> seconds) with tekram, but waiting pretty long (or even forever, "the ugly" ;-)
> while using adaptec. This is particularly interesting for the vm guys since all
> the scene is in high vm load with around 3-5 MB of free mem and a damn lot of
> page cache. So if you try something around vm I can only urge you to perform
> tests that do _no_ I/O at all, because you may be greatly bitten by your
> controller (or its driver).
> Another thing to mention: during the last cd-read tests with tekram setup I
> already have been deeply impressed by the driver, so I decided to stress it
> some more and start applications (like mozilla) in the background. And in the
> end I was even more impressed, because it turned out (you can see in the last
> two figures), that it got even _faster_. Obviously I cannot explain why.
> If anybody wants me to test anything, feel free to ask.
>
> My personal opinion: Justin has work to do.

Agreed here. Justin should write a clean SCSI access method for Linux for
free as he did for FreeBSD. :-)

Just considering the CD read thoughtput differences, we cannot get any
useful information that applies to software driver differences from your
report. Given the very low throughput it involves (about 3 MB/s) compared
to the capabilities of the controllers (160 MB/s), the results should be
explainable by something related to difference in configuration or to some
hardware or kernel weirdness. I cannot believe a single second that the
difference is due to the software drivers.

Thanks, anyway, for your report.

Regards,
  Gérard.

Re: The good, the bad & the ugly (or VM, block devices, and SCSI :-)

[Stephan von Krawczynski]

On Wed, 31 Oct 2001 18:29:25 +0100 (CET) Gérard Roudier <groudier@free.fr>
wrote:

> > A29160:                                     symbios:
> >
> > cd read without nfs-load:                   cd read without nfs-load
> > 2998,9 kB                                   3619,3 kB
> > 3168,2 kB                                   3611,1 kB
> > 2968,4 kB                                   3620,2 kB
> >
> > cd read with nfs load:                      cd read with nfs load
> > 1926,2 kB                                   3408,1 kB
> > 2123,4 kB                                   3395,2 kB
> > 2539,4 kB                                   3605,1 kB
> > 2631,9 kB                                   3605,8 kB
> >
> > [...]
> > My personal opinion: Justin has work to do.
> 
> Agreed here. Justin should write a clean SCSI access method for Linux for
> free as he did for FreeBSD. :-)

Just to make that clear: its not that I am in the position of _expecting_
anything. I only want to give a clear hint what (according to my limited
knowledge) the problem might be, and who could possibly solve it.

> Just considering the CD read thoughtput differences, we cannot get any
> useful information that applies to software driver differences from your
> report. Given the very low throughput it involves (about 3 MB/s) compared
> to the capabilities of the controllers (160 MB/s), the results should be
> explainable by something related to difference in configuration or to some
> hardware or kernel weirdness.

Well, what more can you expect from me, than the simple truth that the config
is the _same_ for both tests and the only thing I am doing is exchange the
scsi-controller (and therefore the used kernel-driver within the compiled
bzImage).
It is pretty clear that U160 cannot be reached by the CD-drive, because it is
located on the scsi-2 connector (50 pin internal). It is a TEAC CD-532S which
has (to my knowledge) not even wide-scsi but 8 bit data transfer). It is
specified as being 30x, so should have a max throughput of 4500 kB/s (150 kB/s
x 30). The values (at least symbios) are obviously not that far off, taking
into account that 30x means "somewhere on the disk we reach 30x" and not
"through the whole disk we have 30x".
The only difference I can confirm is in TCQ-depth being configured to 8 on
adaptec and 4 (!) on tekram. I reduced the tcq-depth on adaptec from 256 to 8,
because 
a) 256 doesn't work out anyway. I got switched back to 128 during workload
according to the driver
b) even 128 makes "feelable" latency during heavy I/O and concurrent
shell-typing stuff.
c) choose therefore 8, because the _old_ aic7xxx driver used 8, too, and was in
my opinion better in terms of latency _and_ throughput (but didn't compile any
more in some 2.4.x kernel, that's why I _had_ to switch over)

some additional infos:
Motherboard Asus CUV4X-D, 2 x P-III 1 GHz, 1 GB RAM

/proc/scsi/scsi:

Attached devices: 
Host: scsi0 Channel: 00 Id: 08 Lun: 00
  Vendor: IBM      Model: DDYS-T36950N     Rev: S96H
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi1 Channel: 00 Id: 02 Lun: 00
  Vendor: BNCHMARK Model: DLT1             Rev: 391B
  Type:   Sequential-Access                ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 03 Lun: 00
  Vendor: TEAC     Model: CD-ROM CD-532S   Rev: 1.0A
  Type:   CD-ROM                           ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 05 Lun: 00
  Vendor: TEAC     Model: CD-R58S          Rev: 1.0P
  Type:   CD-ROM                           ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 06 Lun: 00
  Vendor: HP       Model: C1537A           Rev: L005
  Type:   Sequential-Access                ANSI SCSI revision: 02

/proc/interrupts (tekram):
           CPU0       CPU1       
  0:      77797      76618    IO-APIC-edge  timer
  1:       3127       2991    IO-APIC-edge  keyboard
  2:          0          0          XT-PIC  cascade
  5:         74         83   IO-APIC-level  HiSax
  8:          1          1    IO-APIC-edge  rtc
  9:       4651       4171   IO-APIC-level  EMU10K1, eth0
 10:      16563      16459   IO-APIC-level  sym53c8xx, sym53c8xx, eth2
 11:      62677      62214   IO-APIC-level  eth1, nvidia
 12:       8842       8772    IO-APIC-edge  PS/2 Mouse
 14:        143         29    IO-APIC-edge  ide0
NMI:          0          0 
LOC:     154305     154182 
ERR:          0
MIS:          0

/proc/interrupts (adaptec):
           CPU0       CPU1       
  0:       5967       5266    IO-APIC-edge  timer
  1:        209        192    IO-APIC-edge  keyboard
  2:          0          0          XT-PIC  cascade
  5:          9          6   IO-APIC-level  HiSax
  8:          1          1    IO-APIC-edge  rtc
  9:        183        175   IO-APIC-level  EMU10K1, eth0
 10:       2309       2172   IO-APIC-level  aic7xxx, eth2
 11:       3532       3468   IO-APIC-level  eth1, nvidia
 12:        920       1103    IO-APIC-edge  PS/2 Mouse
 14:        124         48    IO-APIC-edge  ide0
NMI:          0          0 
LOC:      11124      10979 
ERR:          0
MIS:          0

And yes: eth2 is exactly the device where the nfs-load is coming from. This is
unintentional, it just worked out this way, but equal for both contestants.
And no: unfortunately I cannot manage to come to a config where the scsi-IRQ is
singular, I tried hard today, but the network is in fact a 4-port tulip card
which makes a pci-bridge and the irqs behind the bridge tend to do whatever
they like. In fact I moved the irq for the scsi-controllers via bios, but guess
what: eth2 followed wherever I went.
Keep in mind, even with no network traffic adaptec performs bad.
Ah and another thing, I tried _several_ adaptec controllers (even a 29160N),
all the same results.

> I cannot believe a single second that the
> difference is due to the software drivers.

I can. I did a whole lot of such tests during my former job for a company
producing scsi-controllers.

> Thanks, anyway, for your report.

Well, as already said, take it as a hint that your part of the story performs
pretty well.
;-)

Regards,
Stephan

Re: The good, the bad & the ugly (or VM, block devices, and SCSI :-)

[Gérard Roudier]

On Thu, 1 Nov 2001, Stephan von Krawczynski wrote:

> On Wed, 31 Oct 2001 18:29:25 +0100 (CET) Gérard Roudier <groudier@free.fr>
> wrote:
>
> > > A29160:                                     symbios:
> > >
> > > cd read without nfs-load:                   cd read without nfs-load
> > > 2998,9 kB                                   3619,3 kB
> > > 3168,2 kB                                   3611,1 kB
> > > 2968,4 kB                                   3620,2 kB
> > >
> > > cd read with nfs load:                      cd read with nfs load
> > > 1926,2 kB                                   3408,1 kB
> > > 2123,4 kB                                   3395,2 kB
> > > 2539,4 kB                                   3605,1 kB
> > > 2631,9 kB                                   3605,8 kB
> > >
> > > [...]
> > > My personal opinion: Justin has work to do.
> >
> > Agreed here. Justin should write a clean SCSI access method for Linux for
> > free as he did for FreeBSD. :-)
>
> Just to make that clear: its not that I am in the position of _expecting_
> anything. I only want to give a clear hint what (according to my limited
> knowledge) the problem might be, and who could possibly solve it.

That was clear. As clear it is that my reply here was kind of joke. :)

> > Just considering the CD read thoughtput differences, we cannot get any
> > useful information that applies to software driver differences from your
> > report. Given the very low throughput it involves (about 3 MB/s) compared
> > to the capabilities of the controllers (160 MB/s), the results should be
> > explainable by something related to difference in configuration or to some
> > hardware or kernel weirdness.
>
> Well, what more can you expect from me, than the simple truth that the config
> is the _same_ for both tests and the only thing I am doing is exchange the
> scsi-controller (and therefore the used kernel-driver within the compiled
> bzImage).

I can beleive that.

> It is pretty clear that U160 cannot be reached by the CD-drive, because it is
> located on the scsi-2 connector (50 pin internal). It is a TEAC CD-532S which
> has (to my knowledge) not even wide-scsi but 8 bit data transfer). It is
> specified as being 30x, so should have a max throughput of 4500 kB/s (150 kB/s
> x 30). The values (at least symbios) are obviously not that far off, taking
> into account that 30x means "somewhere on the disk we reach 30x" and not
> "through the whole disk we have 30x".
> The only difference I can confirm is in TCQ-depth being configured to 8 on
> adaptec and 4 (!) on tekram. I reduced the tcq-depth on adaptec from 256 to 8,
> because
> a) 256 doesn't work out anyway. I got switched back to 128 during workload
> according to the driver
> b) even 128 makes "feelable" latency during heavy I/O and concurrent
> shell-typing stuff.
> c) choose therefore 8, because the _old_ aic7xxx driver used 8, too, and was in
> my opinion better in terms of latency _and_ throughput (but didn't compile any
> more in some 2.4.x kernel, that's why I _had_ to switch over)

The TCQ-depth shouldn't matter as long as only the CD drive is accessed
given that such devices are unlikely to support tagged commands.
Nevertheless, you should check your boot log messages and also compare the
thoughput negotiation between controller and devices.

> some additional infos:
> Motherboard Asus CUV4X-D, 2 x P-III 1 GHz, 1 GB RAM
>
> /proc/scsi/scsi:
>
> Attached devices:
> Host: scsi0 Channel: 00 Id: 08 Lun: 00
>   Vendor: IBM      Model: DDYS-T36950N     Rev: S96H
>   Type:   Direct-Access                    ANSI SCSI revision: 03
> Host: scsi1 Channel: 00 Id: 02 Lun: 00
>   Vendor: BNCHMARK Model: DLT1             Rev: 391B
>   Type:   Sequential-Access                ANSI SCSI revision: 02
> Host: scsi1 Channel: 00 Id: 03 Lun: 00
>   Vendor: TEAC     Model: CD-ROM CD-532S   Rev: 1.0A
>   Type:   CD-ROM                           ANSI SCSI revision: 02
> Host: scsi1 Channel: 00 Id: 05 Lun: 00
>   Vendor: TEAC     Model: CD-R58S          Rev: 1.0P
>   Type:   CD-ROM                           ANSI SCSI revision: 02
> Host: scsi1 Channel: 00 Id: 06 Lun: 00
>   Vendor: HP       Model: C1537A           Rev: L005
>   Type:   Sequential-Access                ANSI SCSI revision: 02
>
> /proc/interrupts (tekram):
>            CPU0       CPU1
>   0:      77797      76618    IO-APIC-edge  timer
>   1:       3127       2991    IO-APIC-edge  keyboard
>   2:          0          0          XT-PIC  cascade
>   5:         74         83   IO-APIC-level  HiSax
>   8:          1          1    IO-APIC-edge  rtc
>   9:       4651       4171   IO-APIC-level  EMU10K1, eth0
>  10:      16563      16459   IO-APIC-level  sym53c8xx, sym53c8xx, eth2
>  11:      62677      62214   IO-APIC-level  eth1, nvidia
>  12:       8842       8772    IO-APIC-edge  PS/2 Mouse
>  14:        143         29    IO-APIC-edge  ide0
> NMI:          0          0
> LOC:     154305     154182
> ERR:          0
> MIS:          0
>
> /proc/interrupts (adaptec):
>            CPU0       CPU1
>   0:       5967       5266    IO-APIC-edge  timer
>   1:        209        192    IO-APIC-edge  keyboard
>   2:          0          0          XT-PIC  cascade
>   5:          9          6   IO-APIC-level  HiSax
>   8:          1          1    IO-APIC-edge  rtc
>   9:        183        175   IO-APIC-level  EMU10K1, eth0
>  10:       2309       2172   IO-APIC-level  aic7xxx, eth2
>  11:       3532       3468   IO-APIC-level  eth1, nvidia
>  12:        920       1103    IO-APIC-edge  PS/2 Mouse
>  14:        124         48    IO-APIC-edge  ide0
> NMI:          0          0
> LOC:      11124      10979
> ERR:          0
> MIS:          0

Indeed, the configs look very similar.

> And yes: eth2 is exactly the device where the nfs-load is coming from. This is
> unintentional, it just worked out this way, but equal for both contestants.
> And no: unfortunately I cannot manage to come to a config where the scsi-IRQ is
> singular, I tried hard today, but the network is in fact a 4-port tulip card
> which makes a pci-bridge and the irqs behind the bridge tend to do whatever
> they like. In fact I moved the irq for the scsi-controllers via bios, but guess
> what: eth2 followed wherever I went.

PCI defines 4 interrupt lines per slot and also defines some rules for
interrupt lines wiring behind bridges. OTOH, the routing glue to the
interrupt controller may well not allow all possible combinations. What
happens could just be that the routing glue just hardwires the both
interrupt lines here, and only moving a PCI board to another slot can
allow to use different IRQs for the 2 devices.

> Keep in mind, even with no network traffic adaptec performs bad.
> Ah and another thing, I tried _several_ adaptec controllers (even a 29160N),
> all the same results.
>
> > I cannot believe a single second that the
> > difference is due to the software drivers.
>
> I can. I did a whole lot of such tests during my former job for a company
> producing scsi-controllers.

Interesting, if you can elaborate...

> > Thanks, anyway, for your report.
>
> Well, as already said, take it as a hint that your part of the story performs
> pretty well.
> ;-)

The sym53c8xx driver beeing faster than the aic7xxx with CD devices using
Ultra160 controllers is an amuzing result. :)
I still cannot beleive that it is due to a aic7xxx driver fault. If I had
such a controller, I would for sure check that, but I haven't any.

Regards,
  Gérard.

Re: The good, the bad & the ugly (or VM, block devices, and SCSI :-)

[Stephan von Krawczynski]

> The TCQ-depth shouldn't matter as long as only the CD drive is      
accessed                                                              
> given that such devices are unlikely to support tagged commands.    
> Nevertheless, you should check your boot log messages and also      
compare the                                                           
> thoughput negotiation between controller and devices.               
                                                                      
Hm, I am not so sure about TCQ not having _any_ influence. Remember I 
read the CD _and_ write the image to disk (the U160 IBM). So it could 
well be that the results are an accummulation of two bad performing   
things on aic7xxx: the CD read and the disk write.                    
                                                                      
Hear my risky analysis of the problem:                                
The aic7xxx interrupt handler is the cause. It may be that it "wants" 
to much, meaning its hanging hard on its business eating up resources 
(and gaining nothing for it), and therefore playing bad with the      
network interrupt handling _and_ its own higher layer code.           
Or (another pure guess) its busted by its own memory handling         
strategy. I see during the cd read 2 or 3 times simple "freezes" (no  
bug deal, but remarkable) where the systems seems to "hang" for just  
1-2 seconds. symbios code does not show this, but runs smoothly       
through the whole test. This is _not_ related to VM, there is a lot   
free mem during these "small-freezes".                                
                                                                      
> [...]                                                               
> Indeed, the configs look very similar.                              
                                                                      
Believe me, it is.                                                    
                                                                      
> > I can. I did a whole lot of such tests during my former job for a 
company                                                               
> > producing scsi-controllers.                                       
>                                                                     
> Interesting, if you can elaborate...                                
                                                                      
These were the g'old Commodore times (Amiga). :-) Ask me off-list If  
you're interested ...                                                 
                                                                      
> > > Thanks, anyway, for your report.                                
> >                                                                   
> > Well, as already said, take it as a hint that your part of the    
story performs                                                        
> > pretty well.                                                      
> > ;-)                                                               
>                                                                     
> The sym53c8xx driver beeing faster than the aic7xxx with CD devices 
using                                                                 
> Ultra160 controllers is an amuzing result. :)                       
> I still cannot beleive that it is due to a aic7xxx driver fault. If 
I had                                                                 
> such a controller, I would for sure check that, but I haven't any.  
                                                                      
Hm, if I find some spare time, I try to proof it. I am almost sure it 
has nothing to do with hardware. Maybe I am going to earn another red 
herring from linus, which turns out to be red lobster afterwards ;-)  
Keep smiling, we are all having fun here :-)                          
                                                                      
Regards,                                                              
Stephan                                                               
                                                                      
                                                                      
                                                                      

Re: The good, the bad & the ugly (or VM, block devices, and SCSI :-)

[Sebastian Benoit]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

Stephan von Krawczynski(skraw@ithnet.com)@2001.10.31 16:45:39 +0000:
> A29160:                                     symbios:

Did you put them in the same PCI slot?
Whats the output of /proc/interrupts in both cases?
What motherboard?

/B.

-- 
Sebastian Benoit <ben-lists@andastra.de>
OpenPGP-Key ID 0x82AE75E4                             
fingerprint 0BDA 0CB7 9BCA AF77 28EE  D91A 396D 93BC 82AE 75E

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: The good, the bad & the ugly (or VM, block devices, and SCSI :-)

[Stephan von Krawczynski]

On Thu, 1 Nov 2001 11:40:08 +0100 Sebastian Benoit <ben-lists@andastra.de>
wrote:

> Stephan von Krawczynski(skraw@ithnet.com)@2001.10.31 16:45:39 +0000:
> > A29160:                                     symbios:
> 
> Did you put them in the same PCI slot?

yes.

> Whats the output of /proc/interrupts in both cases?
> What motherboard?

see other mail in list.

Regards,
Stephan