Serial Console and Slow SCSI Disk Access?

Serial Console and Slow SCSI Disk Access?

[Jeff Gold]

Does it make sense that enabling a serial console would reduce the 
bandwidth of a SCSI disk by more than a factor of ten?  This seems crazy 
to me but there's no arguing with empirical facts.  I have an IBM x345 
system on which I installed an unmodified 2.6.16.20 kernel built with 
gcc-4.0.2-8.fc4 (the kernel configuration file I used can be found at 
http://augart.com/jgold/kconfig).  With the control kernel command line 
I get about 70 MB/sec with hdparm -t but when I add "console=ttyS0,9600 
console=tty0" I get about 1.6 MB/sec instead.

Here are the complete kernel command lines from grub.conf (the second 
one doesn't actually include a backslash in grub.conf -- that's just to 
avoid email wrapping):

   ro root=/dev/sda3 pci=biosirq loglevel=7

   ro root=/dev/sda3 pci=biosirq console=ttyS0,9600n8 console=tty0 \
     loglevel=7

I have not been able to connect this behavior to any known kernel bugs 
using Google or mailing list archives.  What might I be doing wrong?

                                          Jeff

Re: Serial Console and Slow SCSI Disk Access?

[Mark Lord]

Jeff Gold wrote:
> Does it make sense that enabling a serial console would reduce the 
> bandwidth of a SCSI disk by more than a factor of ten?  This seems crazy 
> to me but there's no arguing with empirical facts.  I have an IBM x345 
> system on which I installed an unmodified 2.6.16.20 kernel built with 
> gcc-4.0.2-8.fc4 (the kernel configuration file I used can be found at 
> http://augart.com/jgold/kconfig).  With the control kernel command line 
> I get about 70 MB/sec with hdparm -t but when I add "console=ttyS0,9600 
> console=tty0" I get about 1.6 MB/sec instead.

This can happen if there are kernel messages being printed on the serial console.
If all is quiet, I would expect things to be as fast as normal elsewhere.

-ml

Re: Serial Console and Slow SCSI Disk Access?

[Jeff Gold]

Mark Lord wrote:
> This can happen if there are kernel messages being printed on the serial 
> console.
> If all is quiet, I would expect things to be as fast as normal elsewhere.

Thank you for the suggestion.  I don't see much in /var/log/messages 
(syslogd is running).  There are 3326 lines taking up about 256 kB 
there, and when I run hdparm runs no further messages are generated.

I don't have anything attached to the serial port at the moment.  Could 
that cause problems?  I'm going to attach something and see what 
happens.  Other advice is still welcome.

                                         Jeff

Re: Serial Console and Slow SCSI Disk Access?

[Helge Hafting]

Jeff Gold wrote:
> Mark Lord wrote:
>> This can happen if there are kernel messages being printed on the 
>> serial console.
>> If all is quiet, I would expect things to be as fast as normal 
>> elsewhere.
>
> Thank you for the suggestion.  I don't see much in /var/log/messages 
> (syslogd is running).  There are 3326 lines taking up about 256 kB 
> there, and when I run hdparm runs no further messages are generated.
>
> I don't have anything attached to the serial port at the moment.  
> Could that cause problems?  I'm going to attach something and see what 
> happens.  Other advice is still welcome. 
With nothing attached, any write to the serial device might go through
a lengthy timeout because of flow control.  I'd consider that a bug
in this case though, and there is usually no console printout
per scsi disk access either.


I would not be surprised if your serial console causes a longer boot time,
as all boot messages have to be transferred over the slow serial link
or in the worst case timed out one message at a time.

But I can't see why it'd make scsi disks slower. The scsi host adapter 
initialization
writes some messages of course, but there should be no more console accesses
during a hdparm test run.

Helge Hafting

Re: Serial Console and Slow SCSI Disk Access?

[Jeff Gold]

Helge Hafting wrote:
> With nothing attached, any write to the serial device might go
> through a lengthy timeout because of flow control.  [...] But I can't
> see why it'd make scsi disks slower. The scsi host adapter 
> initialization writes some messages of course, but there should be no
> more console accesses during a hdparm test run.

This makes sense to me.  When I attach a serial cable and use that to 
login (I've got agetty running), hdparm produces no console messages 
that I can see using minicom.  Still, the disk throughput is around 1.5 
MB/sec for some reason.  When I disable the serial console in grub.conf 
and reboot I get over 70 MB/sec again.

A combination of out-of-tree patches (mainly network related but also 
one to disable PM_TIMERS) seem to eliminate the issue even with the 
serial console enabled, at least for the moment.  That means I no longer 
have a problem, but the whole thing is mysterious to me.

                                        Jeff

Re: Serial Console and Slow SCSI Disk Access?

[Helge Hafting]

Jeff Gold wrote:
> Helge Hafting wrote:
>> With nothing attached, any write to the serial device might go
>> through a lengthy timeout because of flow control.  [...] But I can't
>> see why it'd make scsi disks slower. The scsi host adapter 
>> initialization writes some messages of course, but there should be no
>> more console accesses during a hdparm test run.
>
> This makes sense to me.  When I attach a serial cable and use that to 
> login (I've got agetty running), hdparm produces no console messages 
> that I can see using minicom.  Still, the disk throughput is around 
> 1.5 MB/sec for some reason.  When I disable the serial console in 
> grub.conf and reboot I get over 70 MB/sec again.
>
> A combination of out-of-tree patches (mainly network related but also 
> one to disable PM_TIMERS) seem to eliminate the issue even with the 
> serial console enabled, at least for the moment.  That means I no 
> longer have a problem, but the whole thing is mysterious to me.
I can see one possibility, that I didn't think of yesterday.
Do the scsi host adapter share its interrupt with the serial line?
(Boot a kernel that has the problem, and when scsi is slow, do a
cat /proc/interrupts
If the scsi and the serial driver share an interrupt, then that is the 
source
of the problem.

Linux can share interrupts without big performance problems - IF both of
the hardware drivers are written with interrupt sharing in mind.  Many
linux drivers are, but some are not.  So check if interrupt sharing is 
happening,
and if so, contact the maintainers of your scsi driver and your serial
port driver.  Ask them if such sharing is ok or not.  If you don't
understand the /proc/interrupt listing, just send it to the
maintainers so they can have a look.

Shared interrupts are to some extent set up by the bios, and to some
extent by linux.  So different kernel versions (or booting with a different
set of drivers loaded, or a different distribution) may make a difference.

You can often get a pci device to use a different interrupt by moving it
to another slot.  That tends to solve interrupt sharing problems. I assume
your scsi adapter is pci.

Helge Hafting

Re: Serial Console and Slow SCSI Disk Access?

[Jeff Gold]

Helge Hafting wrote:
> I can see one possibility, that I didn't think of yesterday.
> Do the scsi host adapter share its interrupt with the serial line?

That makes a lot of sense.  I'll check it out and follow your 
instructions as soon as I can.  Thanks.

                                      Jeff

Re: Serial Console and Slow SCSI Disk Access?

[David Lang]

On Tue, 20 Jun 2006, Helge Hafting wrote:

> Jeff Gold wrote:
>> Helge Hafting wrote:
>>> With nothing attached, any write to the serial device might go
>>> through a lengthy timeout because of flow control.  [...] But I can't
>>> see why it'd make scsi disks slower. The scsi host adapter initialization 
>>> writes some messages of course, but there should be no
>>> more console accesses during a hdparm test run.
>> 
>> This makes sense to me.  When I attach a serial cable and use that to login 
>> (I've got agetty running), hdparm produces no console messages that I can 
>> see using minicom.  Still, the disk throughput is around 1.5 MB/sec for 
>> some reason.  When I disable the serial console in grub.conf and reboot I 
>> get over 70 MB/sec again.
>> 
>> A combination of out-of-tree patches (mainly network related but also one 
>> to disable PM_TIMERS) seem to eliminate the issue even with the serial 
>> console enabled, at least for the moment.  That means I no longer have a 
>> problem, but the whole thing is mysterious to me.
> I can see one possibility, that I didn't think of yesterday.
> Do the scsi host adapter share its interrupt with the serial line?
> (Boot a kernel that has the problem, and when scsi is slow, do a
> cat /proc/interrupts
> If the scsi and the serial driver share an interrupt, then that is the source
> of the problem.
>
> Linux can share interrupts without big performance problems - IF both of
> the hardware drivers are written with interrupt sharing in mind.  Many
> linux drivers are, but some are not.  So check if interrupt sharing is 
> happening,
> and if so, contact the maintainers of your scsi driver and your serial
> port driver.  Ask them if such sharing is ok or not.  If you don't
> understand the /proc/interrupt listing, just send it to the
> maintainers so they can have a look.
>
> Shared interrupts are to some extent set up by the bios, and to some
> extent by linux.  So different kernel versions (or booting with a different
> set of drivers loaded, or a different distribution) may make a difference.
>
> You can often get a pci device to use a different interrupt by moving it
> to another slot.  That tends to solve interrupt sharing problems. I assume
> your scsi adapter is pci.
>

I ran into a similar problem (extremely slow scsi performance), but hadn't 
reported it as I made a lot of changes at the same time (moved from 2.6.7 to 
2.6.12.3 as well as enabling the serial console). but with this report I wanted 
to chime in with a weak 'me-too'. I haven't rebooted without the serial console 
yet to confirm this, but I will do so shortly, and try 2.6.17 to see if it has a 
similar problem or not.

David Lang

# cat /proc/interrupts
            CPU0       CPU1
   0:  458865186      29148    IO-APIC-edge  timer
   1:        216          7    IO-APIC-edge  i8042
   3:         78          1    IO-APIC-edge  serial
   4:         76          1    IO-APIC-edge  serial
   8:          8          1    IO-APIC-edge  rtc
   9:          0          0   IO-APIC-level  acpi
  12:         84         10    IO-APIC-edge  i8042
  14:        332         10    IO-APIC-edge  ide0
  24:    4426414          1   IO-APIC-level  eth2
  26:  426930991          1   IO-APIC-level  eth0
  27:      17795  365692411   IO-APIC-level  eth1
  29:     553120         87   IO-APIC-level  ioc0
NMI:          0          0
LOC:  458903065  458904855
ERR:          0
MIS:          0

# lspci
0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
0000:00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02)
0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 
12)
0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 
12)
0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:01:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
0000:02:02.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 02)
0000:02:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5702X 
Gigabit Ethernet (rev 02)
0000:02:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5702X 
Gigabit Ethernet (rev 02)
0000:03:04.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet 
Controller (rev 01)
0000:03:04.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet 
Controller (rev 01)
0000:03:06.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet 
Controller (rev 01)
0000:03:06.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet 
Controller (rev 01)
0000:04:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X 
Fusion-MPT Dual Ultra320 SCSI (rev 07)

Re: Serial Console and Slow SCSI Disk Access?

[Andrey Melnikoff]

David Lang <dlang@digitalinsight.com> wrote:
> On Tue, 20 Jun 2006, Helge Hafting wrote:

> > Jeff Gold wrote:
> >> Helge Hafting wrote:
> >>> With nothing attached, any write to the serial device might go
> >>> through a lengthy timeout because of flow control.  [...] But I can't
> >>> see why it'd make scsi disks slower. The scsi host adapter initialization 
> >>> writes some messages of course, but there should be no
> >>> more console accesses during a hdparm test run.
> >> 
> >> This makes sense to me.  When I attach a serial cable and use that to login 
> >> (I've got agetty running), hdparm produces no console messages that I can 
> >> see using minicom.  Still, the disk throughput is around 1.5 MB/sec for 
> >> some reason.  When I disable the serial console in grub.conf and reboot I 
> >> get over 70 MB/sec again.
> >> 
> >> A combination of out-of-tree patches (mainly network related but also one 
> >> to disable PM_TIMERS) seem to eliminate the issue even with the serial 
> >> console enabled, at least for the moment.  That means I no longer have a 
> >> problem, but the whole thing is mysterious to me.
> > I can see one possibility, that I didn't think of yesterday.
> > Do the scsi host adapter share its interrupt with the serial line?
> > (Boot a kernel that has the problem, and when scsi is slow, do a
> > cat /proc/interrupts
> > If the scsi and the serial driver share an interrupt, then that is the source
> > of the problem.

[skipp]

> I ran into a similar problem (extremely slow scsi performance), but hadn't 
> reported it as I made a lot of changes at the same time (moved from 2.6.7 to 
> 2.6.12.3 as well as enabling the serial console). but with this report I wanted 
> to chime in with a weak 'me-too'. I haven't rebooted without the serial console 
> yet to confirm this, but I will do so shortly, and try 2.6.17 to see if it has a 
> similar problem or not.

2.6.17.1 here, MB Intel SCB20, aic7899 onboard. aic7xxx compilled in, serial
core compilled in, serial console enabled.
With/without serial sonsole - speed 66.0 Mb/s.

Maybe problem related to MB chipset?