How to avoid serial port buffer overruns?

How to avoid serial port buffer overruns?

[Raphael Hertzog]

(Please CC me when replying)

Hello,

While using Linux on low-end (semi-embedded) hardware (386 SX 40Mhz, 8Mb
RAM), I discovered that Linux on that machine would suffer from serial
port buffer overruns quite easily if I use a baudrate high enough (I start
loosing bytes at >19200 bauds and I would like to make it reliable up to 
115 kbauds). I check if overruns are happening with
/proc/tty/driver/serial ("oe" field).

Back when I was using the 2.4 kernel, I reduced dramatically the frequency
of overruns by using the "low latency" and "preemptible kernel" patch [1]. But
it still happened sometimes at 115 kbauds if the system was a bit loaded
(with disk I/O for example).

Now I switched to stock 2.6 and while the stock kernel improved in
responsiveness, it still isn't enough by default (even with
CONFIG_PREEMPT=y and CONFIG_HZ=1000). So I wanted to try the "rt" patch of
Ingo Molnar and Thomas Gleixner, but the patched kernel doesn't boot (see
bug report in a separate mail on this list).

Other things that I tried which didn't help (enough) are:

- tuning with hdparm
  - make disk IRQ interruptible with hdparm -u 1 /dev/hda
  - activating DMA is suggested but my disk is a "disk on module"
    and doesn't support DMA
- using irqtune (http://cae.best.vwh.net/irqtune/) to reprioritize
  interrupts
  irqtune is old and it's very difficult to know if it still works
  reliably with recent kernel and as there's no way to "read" the
  interrupt priorities, I have no way to know if irqtune changed anything
  at all...

My questions are thus:
1/ Is there a way to patch the kernel to make it handle serial IRQ as
   the highest priority? If yes, how? Where should I look at to create
   such a patch?
2/ How can I identify why the serial interrupts are delayed? Or, in other
   words, how can I find the code blocking for too long the treatment of
   the serial IRQ?
   I suppose that network IRQ and disk IRQ are responsible for that but
   I'm not sure and furthermore disk interrupts are supposed to
   be interruptible given the hdparm config that I use.
3/ What other suggestions do you have to avoid those serial buffer
   overruns?

As usual, I'll gladly try out patches/ideas and will provide any
required additional information that I didn't include in this mail.

Some infos on the hardware:
http://www.icop.com.tw/products_detail.asp?ProductID=205
More detailed spec of the CPU are here:
http://www.dmp.com.tw/tech/m6117d/

bash-2.05a# cat /proc/interrupts
CPU0
0:   88503366          XT-PIC  timer
2:          0          XT-PIC  cascade
3:         11          XT-PIC  serial
4:         12          XT-PIC  serial
5:       4958          XT-PIC  NE2000
14:      10771          XT-PIC  ide0
NMI:          0
ERR:          0

Regards,

[1] I documented that in a blog post last year:
http://www.ouaza.com/wordpress/2005/10/19/serial-overrun-on-linux/
-- 
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/

Re: How to avoid serial port buffer overruns?

[Lennart Sorensen]

On Wed, Aug 16, 2006 at 12:45:59PM +0200, Raphael Hertzog wrote:
> While using Linux on low-end (semi-embedded) hardware (386 SX 40Mhz, 8Mb
> RAM), I discovered that Linux on that machine would suffer from serial
> port buffer overruns quite easily if I use a baudrate high enough (I start
> loosing bytes at >19200 bauds and I would like to make it reliable up to 
> 115 kbauds). I check if overruns are happening with
> /proc/tty/driver/serial ("oe" field).
> 
> Back when I was using the 2.4 kernel, I reduced dramatically the frequency
> of overruns by using the "low latency" and "preemptible kernel" patch [1]. But
> it still happened sometimes at 115 kbauds if the system was a bit loaded
> (with disk I/O for example).

In my experience, the way to avoid overruns on a serial port was to use
a buffered serial port UART (such as a 16550A for example).  I remember
my 486 wasn't reliable about 19200 or 38400 (depending on how busy the
cpu was) when using an 8250.  Using a 16550A based card and I could do
115200 without any issues since the UART had a 16 byte buffer to help
out the system.  Unless your 386 has an add in card for the serial port,
it almost certainly has a very crappy UART and it would be very hard to
make it work reliably at higher speeds.

--
Len Sorensen

Re: How to avoid serial port buffer overruns?

[Raphael Hertzog]

On Wed, 16 Aug 2006, Lennart Sorensen wrote:
> In my experience, the way to avoid overruns on a serial port was to use
> a buffered serial port UART (such as a 16550A for example).  I remember
> my 486 wasn't reliable about 19200 or 38400 (depending on how busy the
> cpu was) when using an 8250.  Using a 16550A based card and I could do
> 115200 without any issues since the UART had a 16 byte buffer to help
> out the system.  Unless your 386 has an add in card for the serial port,
> it almost certainly has a very crappy UART and it would be very hard to
> make it work reliably at higher speeds.

I forgot to mention the kind of UART in my mail but it is a 16650A (and
configured as such and detected as such) and I have overruns nevertheless.

bash-2.05a# cat /proc/tty/driver/serial
serinfo:1.0 driver revision:
0: uart:16550A port:000003F8 irq:4 tx:31 rx:7 RTS|CTS|DTR|DSR
1: uart:16550A port:000002F8 irq:3 tx:24 rx:7 RTS|CTS|DTR|DSR

(here there's no overrun but I almost didn't use the serial port since
last reboot)

Cheers,
-- 
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/

Re: How to avoid serial port buffer overruns?

[Lee Revell]

On Wed, 2006-08-16 at 12:45 +0200, Raphael Hertzog wrote:
> (Please CC me when replying)
> 
> Hello,
> 
> While using Linux on low-end (semi-embedded) hardware (386 SX 40Mhz, 8Mb
> RAM), I discovered that Linux on that machine would suffer from serial
> port buffer overruns quite easily if I use a baudrate high enough (I start
> loosing bytes at >19200 bauds and I would like to make it reliable up to 
> 115 kbauds). I check if overruns are happening with
> /proc/tty/driver/serial ("oe" field).
> 
> Back when I was using the 2.4 kernel, I reduced dramatically the frequency
> of overruns by using the "low latency" and "preemptible kernel" patch [1]. But
> it still happened sometimes at 115 kbauds if the system was a bit loaded
> (with disk I/O for example).
> 
> Now I switched to stock 2.6 and while the stock kernel improved in
> responsiveness, it still isn't enough by default (even with
> CONFIG_PREEMPT=y and CONFIG_HZ=1000). So I wanted to try the "rt" patch of
> Ingo Molnar and Thomas Gleixner, but the patched kernel doesn't boot (see
> bug report in a separate mail on this list).

Does the serial performance seem to have regressed from 2.4 to 2.6?  I
am chasing a similar issue with a serial MIDI card (supported by the bog
standard 8250 serial driver) that drops notes under 2.6 but works with
2.4.  I don't have details yet, but it sounds like a similar problem.

Lee

Re: How to avoid serial port buffer overruns?

[Paul Fulghum]

Lee Revell wrote:
> On Wed, 2006-08-16 at 12:45 +0200, Raphael Hertzog wrote:
>>Now I switched to stock 2.6 and while the stock kernel improved in
>>responsiveness, it still isn't enough by default (even with
>>CONFIG_PREEMPT=y and CONFIG_HZ=1000). So I wanted to try the "rt" patch of
>>Ingo Molnar and Thomas Gleixner, but the patched kernel doesn't boot (see
>>bug report in a separate mail on this list).
> 
> 
> Does the serial performance seem to have regressed from 2.4 to 2.6?  I
> am chasing a similar issue with a serial MIDI card (supported by the bog
> standard 8250 serial driver) that drops notes under 2.6 but works with
> 2.4.  I don't have details yet, but it sounds like a similar problem.

What specific 2.6 kernels are each of you using?

-- 
Paul Fulghum
Microgate Systems, Ltd.

Re: How to avoid serial port buffer overruns?

[Lee Revell]

On Wed, 2006-08-16 at 14:23 -0500, Paul Fulghum wrote:
> Lee Revell wrote:
> > On Wed, 2006-08-16 at 12:45 +0200, Raphael Hertzog wrote:
> >>Now I switched to stock 2.6 and while the stock kernel improved in
> >>responsiveness, it still isn't enough by default (even with
> >>CONFIG_PREEMPT=y and CONFIG_HZ=1000). So I wanted to try the "rt" patch of
> >>Ingo Molnar and Thomas Gleixner, but the patched kernel doesn't boot (see
> >>bug report in a separate mail on this list).
> > 
> > 
> > Does the serial performance seem to have regressed from 2.4 to 2.6?  I
> > am chasing a similar issue with a serial MIDI card (supported by the bog
> > standard 8250 serial driver) that drops notes under 2.6 but works with
> > 2.4.  I don't have details yet, but it sounds like a similar problem.
> 
> What specific 2.6 kernels are each of you using?
> 

2.6.15 and 2.6.16.  Here is the .config:

http://members.dca.net/rlrevell/serialbug-config

I don't know which 2.4 version they tested on that worked - just that
the problem started when moving to 2.6.

Lee

Re: How to avoid serial port buffer overruns?

[Paul Fulghum]

On Wed, 2006-08-16 at 17:12 -0400, Lee Revell wrote:
> 2.6.15 and 2.6.16.  Here is the .config:

Alan's rework of the receive tty buffering went
into 2.6.16 and cured some problems, but clearly not yours.
Some more adjustments are in 2.6.18-rc4, so that
would be interesting to try for diagnosing this.

I was wondering if the problem was interrupt latency,
the tty receive buffering, or something totally different.
I don't know if your problem and Raphael's are caused
by the same mechanism. I would still like to know which
kernel versions he has tried.

Does the MIDI device using the standard N_TTY line discipline?
Are you using the low_latency flag on the serial device?
What type of UART has been tested (16550? other?)
Are you seeing overruns or just lost data?

Thanks,
Paul

Re: How to avoid serial port buffer overruns?

[Russell King]

On Wed, Aug 16, 2006 at 05:24:26PM -0500, Paul Fulghum wrote:
> Does the MIDI device using the standard N_TTY line discipline?
> Are you using the low_latency flag on the serial device?
> What type of UART has been tested (16550? other?)
> Are you seeing overruns or just lost data?

MIDI uses its own driver - sound/drivers/serial-u16550.c.  My guess
is there's something in the system starving interrupt servicing.
Serial is very sensitive to that, and increases in other system
latencies tends to have an adverse impact on serial.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: How to avoid serial port buffer overruns?

[Lee Revell]

On Thu, 2006-08-17 at 00:10 +0100, Russell King wrote:
> On Wed, Aug 16, 2006 at 05:24:26PM -0500, Paul Fulghum wrote:
> > Does the MIDI device using the standard N_TTY line discipline?
> > Are you using the low_latency flag on the serial device?
> > What type of UART has been tested (16550? other?)
> > Are you seeing overruns or just lost data?
> 
> MIDI uses its own driver - sound/drivers/serial-u16550.c.  My guess
> is there's something in the system starving interrupt servicing.
> Serial is very sensitive to that, and increases in other system
> latencies tends to have an adverse impact on serial.
> 

Thanks.

Have you seen many other reports of serial working reliably in 2.4 but
not in 2.6?  Right now this is the only clue I have to go on...

Lee

Re: How to avoid serial port buffer overruns?

[Russell King]

On Wed, Aug 16, 2006 at 07:15:06PM -0400, Lee Revell wrote:
> On Thu, 2006-08-17 at 00:10 +0100, Russell King wrote:
> > MIDI uses its own driver - sound/drivers/serial-u16550.c.  My guess
> > is there's something in the system starving interrupt servicing.
> > Serial is very sensitive to that, and increases in other system
> > latencies tends to have an adverse impact on serial.
> 
> Have you seen many other reports of serial working reliably in 2.4 but
> not in 2.6?  Right now this is the only clue I have to go on...

There have been one or two, but the above is basically as far as I've
got.  Unfortunately, I don't have any machines slow enough (or maybe
with the right hardware) to exhibit the problem.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: How to avoid serial port buffer overruns?

[Lee Revell]

On Thu, 2006-08-17 at 00:19 +0100, Russell King wrote:
> On Wed, Aug 16, 2006 at 07:15:06PM -0400, Lee Revell wrote:
> > On Thu, 2006-08-17 at 00:10 +0100, Russell King wrote:
> > > MIDI uses its own driver - sound/drivers/serial-u16550.c.  My guess
> > > is there's something in the system starving interrupt servicing.
> > > Serial is very sensitive to that, and increases in other system
> > > latencies tends to have an adverse impact on serial.
> > 
> > Have you seen many other reports of serial working reliably in 2.4 but
> > not in 2.6?  Right now this is the only clue I have to go on...
> 
> There have been one or two, but the above is basically as far as I've
> got.  Unfortunately, I don't have any machines slow enough (or maybe
> with the right hardware) to exhibit the problem.
> 

OK, thanks.  FWIW here is the serial board we are using:

http://www.moschip.com/html/MCS9845.html

The hardware guy says "The mn9845cv, have in default 2 serial ports and
one ISA bus, where we have connected the tl16c554, quad serial port."

Hopefully Ingo's latency tracer can tell me what is holding off
interrupts.

Lee

R: How to avoid serial port buffer overruns?

[Giampaolo Tomassoni]

> OK, thanks.  FWIW here is the serial board we are using:
> 
> http://www.moschip.com/html/MCS9845.html
> 
> The hardware guy says "The mn9845cv, have in default 2 serial ports and
> one ISA bus, where we have connected the tl16c554, quad serial port."
> 
> Hopefully Ingo's latency tracer can tell me what is holding off
> interrupts.

That may be an interrupt-sharing issue: quad port often use at most two irq lines and, FWIK, ISA irqs are edge-triggered, not level-triggered. Are you tring to use two MIDI ports at the same time?

Giampaolo

> 
> Lee
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

R: How to avoid serial port buffer overruns?

[Giampaolo Tomassoni]

> On Thu, 2006-08-17 at 00:19 +0100, Russell King wrote:
> 
> 
> OK, thanks.  FWIW here is the serial board we are using:
> 
> http://www.moschip.com/html/MCS9845.html
> 
> The hardware guy says "The mn9845cv, have in default 2 serial ports and
> one ISA bus, where we have connected the tl16c554, quad serial port."
> 
> Hopefully Ingo's latency tracer can tell me what is holding off
> interrupts.

I beg your pardon: I'm not used that much to interrupts handling in Linux, but this piece of code from sound/drivers/serial-u16550.c in a linux-2.6.16:

	static irqreturn_t snd_uart16550_interrupt(int irq, void *dev_id, struct pt_regs *regs)
	{
	        snd_uart16550_t *uart;
	
	        uart = (snd_uart16550_t *) dev_id;
	        spin_lock(&uart->open_lock);
	        if (uart->filemode == SERIAL_MODE_NOT_OPENED) {
	                spin_unlock(&uart->open_lock);
	                return IRQ_NONE;
	        }
	        inb(uart->base + UART_IIR);             /* indicate to the UART that the interrupt has been serviced */
	        snd_uart16550_io_loop(uart);
	        spin_unlock(&uart->open_lock);
	        return IRQ_HANDLED;
	}

means to me that IRQ_HANDLED is returned even when the interrupt is not issued by the specific UART. This may lead to problems when two or more uarts share the same irq line and the irq line is edge-triggered instead of level-triggered, as is the case with ISA.

To my knowledge, IRQ_HANDLED should be returned when an interrupt had been served by that specific device handler. Returning a IRQ_HANDLED when the device didn't request for service, in the best case cuases interrupt latencies, in the worst (like in an ISA environment) impairs servicing requests from devices sharing the same IRQ line.

The byte returned from inb(uart->base + UART_IIR) can be used to detect if this is the requesting UART.

Am I wrong?

Regards,

	Giampaolo

Re: R: How to avoid serial port buffer overruns?

[Lee Revell]

On Fri, 2006-08-18 at 10:48 +0200, Giampaolo Tomassoni wrote:
> > On Thu, 2006-08-17 at 00:19 +0100, Russell King wrote:
> > 
> > 
> > OK, thanks.  FWIW here is the serial board we are using:
> > 
> > http://www.moschip.com/html/MCS9845.html
> > 
> > The hardware guy says "The mn9845cv, have in default 2 serial ports and
> > one ISA bus, where we have connected the tl16c554, quad serial port."
> > 
> > Hopefully Ingo's latency tracer can tell me what is holding off
> > interrupts.
> 
> I beg your pardon: I'm not used that much to interrupts handling in Linux, but this piece of code from sound/drivers/serial-u16550.c in a linux-2.6.16:

OK, they are not using serial-u16550 but 8250_fourport for some reason:

# cat /proc/tty/driver/serial 
serinfo:1.0 driver revision:
0: uart:16550A port:000003F8 irq:4 tx:0 rx:0
1: uart:unknown port:000002B8 irq:5
2: uart:unknown port:000003E8 irq:4
3: uart:unknown port:000002E8 irq:3
4: uart:16550A port:0000DD00 irq:185 tx:234335 rx:47502 RTS|DTR
5: uart:16550A port:0000E300 irq:185 tx:249926 rx:27732 RTS|DTR
6: uart:16550A port:0000E400 irq:185 tx:120958 rx:0 RTS|DTR
7: uart:16550A port:0000D000 irq:185 tx:0 rx:0
8: uart:16550A port:0000D100 irq:185 tx:0 rx:0 RTS|DTR
9: uart:16550A port:0000D200 irq:185 tx:0 rx:123406 RTS|DTR

It looks like no overruns are reported, but I have to find out whether
they have reproduced the bug since the last reboot.

Re: R: How to avoid serial port buffer overruns?

[Russell King]

On Fri, Aug 18, 2006 at 01:00:00PM -0400, Lee Revell wrote:
> On Fri, 2006-08-18 at 10:48 +0200, Giampaolo Tomassoni wrote:
> > > On Thu, 2006-08-17 at 00:19 +0100, Russell King wrote:
> > > 
> > > 
> > > OK, thanks.  FWIW here is the serial board we are using:
> > > 
> > > http://www.moschip.com/html/MCS9845.html
> > > 
> > > The hardware guy says "The mn9845cv, have in default 2 serial ports and
> > > one ISA bus, where we have connected the tl16c554, quad serial port."
> > > 
> > > Hopefully Ingo's latency tracer can tell me what is holding off
> > > interrupts.
> > 
> > I beg your pardon: I'm not used that much to interrupts handling in Linux, but this piece of code from sound/drivers/serial-u16550.c in a linux-2.6.16:
> 
> OK, they are not using serial-u16550 but 8250_fourport for some reason:

Doesn't look like it.  fourport cards have their ports at 0x1a0..0x1bf
and 0x2a0..0x2bf, and have some special and non-standard features.

> # cat /proc/tty/driver/serial 
> serinfo:1.0 driver revision:
> 0: uart:16550A port:000003F8 irq:4 tx:0 rx:0
> 1: uart:unknown port:000002B8 irq:5
> 2: uart:unknown port:000003E8 irq:4
> 3: uart:unknown port:000002E8 irq:3
> 4: uart:16550A port:0000DD00 irq:185 tx:234335 rx:47502 RTS|DTR
> 5: uart:16550A port:0000E300 irq:185 tx:249926 rx:27732 RTS|DTR
> 6: uart:16550A port:0000E400 irq:185 tx:120958 rx:0 RTS|DTR
> 7: uart:16550A port:0000D000 irq:185 tx:0 rx:0
> 8: uart:16550A port:0000D100 irq:185 tx:0 rx:0 RTS|DTR
> 9: uart:16550A port:0000D200 irq:185 tx:0 rx:123406 RTS|DTR
> 
> It looks like no overruns are reported, but I have to find out whether
> they have reproduced the bug since the last reboot.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: R: How to avoid serial port buffer overruns?

[Lee Revell]

On Fri, 2006-08-18 at 18:04 +0100, Russell King wrote:
> On Fri, Aug 18, 2006 at 01:00:00PM -0400, Lee Revell wrote:
> > On Fri, 2006-08-18 at 10:48 +0200, Giampaolo Tomassoni wrote:
> > > > On Thu, 2006-08-17 at 00:19 +0100, Russell King wrote:
> > > > 
> > > > 
> > > > OK, thanks.  FWIW here is the serial board we are using:
> > > > 
> > > > http://www.moschip.com/html/MCS9845.html
> > > > 
> > > > The hardware guy says "The mn9845cv, have in default 2 serial ports and
> > > > one ISA bus, where we have connected the tl16c554, quad serial port."
> > > > 
> > > > Hopefully Ingo's latency tracer can tell me what is holding off
> > > > interrupts.
> > > 
> > > I beg your pardon: I'm not used that much to interrupts handling in Linux, but this piece of code from sound/drivers/serial-u16550.c in a linux-2.6.16:
> > 
> > OK, they are not using serial-u16550 but 8250_fourport for some reason:
> 
> Doesn't look like it.  fourport cards have their ports at 0x1a0..0x1bf
> and 0x2a0..0x2bf, and have some special and non-standard features.
> 

Which driver is being used then?

# cat /proc/interrupts 
           CPU0       
  0:     801050    IO-APIC-edge  timer
  1:         42    IO-APIC-edge  i8042
  7:          0    IO-APIC-edge  parport0
  8:          4    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 12:      11022    IO-APIC-edge  i8042
 14:       2517    IO-APIC-edge  ide0
169:     536096   IO-APIC-level  ohci1394, ICE1712
185:       3908   IO-APIC-level  eth0, serial
193:       8610   IO-APIC-level  libata
201:        326   IO-APIC-level  uhci_hcd:usb1, uhci_hcd:usb2,
uhci_hcd:usb3, uhci_hcd:usb4, ehci_hcd:usb5
217:      84947   IO-APIC-level  nvidia
NMI:          0 
LOC:     800973 
ERR:          0
MIS:          0

# lsmod | egrep serial\|8250
8250_fourport           2048  0 [permanent]
8250_pnp                8704  0 
parport_serial          7680  0 
parport_pc             31984  1 parport_serial
8250_pci               19968  1 parport_serial
8250                   22704  12 8250_pnp,8250_pci
serial_core            19200  1 8250

Serial: 8250/16550 driver $Revision: 1.90 $ 10 ports, IRQ sharing
enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Linux agpgart interface v0.101 (c) Dave Jones
agpgart: Detected AGP bridge 0
agpgart: AGP aperture is 128M @ 0xe8000000
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
Real Time Clock Driver v1.12ac
input: PC Speaker as /class/input/input1
ACPI: PCI Interrupt 0000:00:0b.0[A] -> GSI 19 (level, low) -> IRQ 185
0000:00:0b.0: ttyS4 at I/O 0xdd00 (irq = 185) is a 16550A
0000:00:0b.0: ttyS5 at I/O 0xe300 (irq = 185) is a 16550A
0000:00:0b.0: ttyS6 at I/O 0xe400 (irq = 185) is a 16550A
0000:00:0b.0: ttyS7 at I/O 0xd000 (irq = 185) is a 16550A
0000:00:0b.0: ttyS8 at I/O 0xd100 (irq = 185) is a 16550A
0000:00:0b.0: ttyS9 at I/O 0xd200 (irq = 185) is a 16550A

Lee

Re: R: How to avoid serial port buffer overruns?

[Russell King]

On Fri, Aug 18, 2006 at 01:30:40PM -0400, Lee Revell wrote:
> On Fri, 2006-08-18 at 18:04 +0100, Russell King wrote:
> > On Fri, Aug 18, 2006 at 01:00:00PM -0400, Lee Revell wrote:
> > > On Fri, 2006-08-18 at 10:48 +0200, Giampaolo Tomassoni wrote:
> > > > > On Thu, 2006-08-17 at 00:19 +0100, Russell King wrote:
> > > > > 
> > > > > 
> > > > > OK, thanks.  FWIW here is the serial board we are using:
> > > > > 
> > > > > http://www.moschip.com/html/MCS9845.html
> > > > > 
> > > > > The hardware guy says "The mn9845cv, have in default 2 serial ports and
> > > > > one ISA bus, where we have connected the tl16c554, quad serial port."
> > > > > 
> > > > > Hopefully Ingo's latency tracer can tell me what is holding off
> > > > > interrupts.
> > > > 
> > > > I beg your pardon: I'm not used that much to interrupts handling in Linux, but this piece of code from sound/drivers/serial-u16550.c in a linux-2.6.16:
> > > 
> > > OK, they are not using serial-u16550 but 8250_fourport for some reason:
> > 
> > Doesn't look like it.  fourport cards have their ports at 0x1a0..0x1bf
> > and 0x2a0..0x2bf, and have some special and non-standard features.
> > 
> 
> Which driver is being used then?
> 
> # lsmod | egrep serial\|8250
> 8250_fourport           2048  0 [permanent]
> 8250_pnp                8704  0 
> parport_serial          7680  0 
> parport_pc             31984  1 parport_serial
> 8250_pci               19968  1 parport_serial
> 8250                   22704  12 8250_pnp,8250_pci
> serial_core            19200  1 8250
> 
> Serial: 8250/16550 driver $Revision: 1.90 $ 10 ports, IRQ sharing
> enabled
> serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> Linux agpgart interface v0.101 (c) Dave Jones
> agpgart: Detected AGP bridge 0
> agpgart: AGP aperture is 128M @ 0xe8000000
> parport: PnPBIOS parport detected.
> parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
> Real Time Clock Driver v1.12ac
> input: PC Speaker as /class/input/input1
> ACPI: PCI Interrupt 0000:00:0b.0[A] -> GSI 19 (level, low) -> IRQ 185
> 0000:00:0b.0: ttyS4 at I/O 0xdd00 (irq = 185) is a 16550A
> 0000:00:0b.0: ttyS5 at I/O 0xe300 (irq = 185) is a 16550A
> 0000:00:0b.0: ttyS6 at I/O 0xe400 (irq = 185) is a 16550A
> 0000:00:0b.0: ttyS7 at I/O 0xd000 (irq = 185) is a 16550A
> 0000:00:0b.0: ttyS8 at I/O 0xd100 (irq = 185) is a 16550A
> 0000:00:0b.0: ttyS9 at I/O 0xd200 (irq = 185) is a 16550A

That "0000:00:0b.0" looks like a PCI device ID.  If it were a fourport
board, it would be "serial8250.3" according to the current enumeration
in linux/serial_8250.h.

Also, another give away is that IRQ185 is being setup as a PCI interrupt
immediately prior to the devices being registered.

And I doubt that an ISA board (which is what fourport is) would ever get
such a high IRQ number.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: R: How to avoid serial port buffer overruns?

[Lee Revell]

On Fri, 2006-08-18 at 19:34 +0100, Russell King wrote:
> That "0000:00:0b.0" looks like a PCI device ID.  If it were a fourport
> board, it would be "serial8250.3" according to the current enumeration
> in linux/serial_8250.h.
> 
> Also, another give away is that IRQ185 is being setup as a PCI interrupt
> immediately prior to the devices being registered.
> 
> And I doubt that an ISA board (which is what fourport is) would ever get
> such a high IRQ number.
> 

So you're saying that the standard 8250 driver is being used?

This is the board they are using:

http://www.moschip.com/html/MCS9845.html

Lee

Re: R: How to avoid serial port buffer overruns?

[Russell King]

On Fri, Aug 18, 2006 at 02:52:53PM -0400, Lee Revell wrote:
> On Fri, 2006-08-18 at 19:34 +0100, Russell King wrote:
> > That "0000:00:0b.0" looks like a PCI device ID.  If it were a fourport
> > board, it would be "serial8250.3" according to the current enumeration
> > in linux/serial_8250.h.
> > 
> > Also, another give away is that IRQ185 is being setup as a PCI interrupt
> > immediately prior to the devices being registered.
> > 
> > And I doubt that an ISA board (which is what fourport is) would ever get
> > such a high IRQ number.
> > 
> 
> So you're saying that the standard 8250 driver is being used?

Yes, which is also the case with 8250_fourport.  8250_fourport is just
a probe module just like 8250_pnp or 8250_pci.

> http://www.moschip.com/html/MCS9845.html

That also clearly says its a PCI device. 8)

What problem are we talking about here again?  Sorry, I've completely lost
track and this particular thread of 26 messages is soo convoluted and too
much to re-read.

Since you only appear to be the messenger, wouldn't it be far better to get
the person with the problem to report and respond rather than sitting in
the middle?

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: R: How to avoid serial port buffer overruns?

[Russell King]

On Fri, Aug 18, 2006 at 08:01:06PM +0100, Russell King wrote:
> On Fri, Aug 18, 2006 at 02:52:53PM -0400, Lee Revell wrote:
> > On Fri, 2006-08-18 at 19:34 +0100, Russell King wrote:
> > > That "0000:00:0b.0" looks like a PCI device ID.  If it were a fourport
> > > board, it would be "serial8250.3" according to the current enumeration
> > > in linux/serial_8250.h.
> > > 
> > > Also, another give away is that IRQ185 is being setup as a PCI interrupt
> > > immediately prior to the devices being registered.
> > > 
> > > And I doubt that an ISA board (which is what fourport is) would ever get
> > > such a high IRQ number.
> > > 
> > 
> > So you're saying that the standard 8250 driver is being used?
> 
> Yes, which is also the case with 8250_fourport.  8250_fourport is just
> a probe module just like 8250_pnp or 8250_pci.
> 
> > http://www.moschip.com/html/MCS9845.html
> 
> That also clearly says its a PCI device. 8)
> 
> What problem are we talking about here again?  Sorry, I've completely lost
> track and this particular thread of 26 messages is soo convoluted and too
> much to re-read.
> 
> Since you only appear to be the messenger, wouldn't it be far better to get
> the person with the problem to report and respond rather than sitting in
> the middle?

BTW, let's have _one_ email which describes what problem it is, the
hardware which its being seen on, the kernel configuration, what
modules are loaded, the lspci output and so forth.  Let's not spread
all the information over 20 emails.

I just can't work with dribbled small bits of inforamtion sparsely
spread.  Neither can I work when it takes days to get that information -
I forget both the reported details and the thread of thought I was
following to maybe work towards a conclusion.

(Though at present I'm rather devoid of ideas in this area.)

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: R: How to avoid serial port buffer overruns?

[Lee Revell]

On Fri, 2006-08-18 at 20:01 +0100, Russell King wrote:
> What problem are we talking about here again?  Sorry, I've completely
> lost track and this particular thread of 26 messages is soo convoluted
> and too much to re-read.
> 

I thought I might have been seeing the same problem as the OP, but I've
found it's a separate issue.  I'll start a new thread.

The original poster in this thread was just wondering how to reduce the
number of serial overruns at baud rates over 19200.

> Since you only appear to be the messenger, wouldn't it be far better
> to get the person with the problem to report and respond rather than
> sitting in the middle?

Sorry, it took me a while but I have access to the machine with the
problem now.

Lee

Re: How to avoid serial port buffer overruns?

[Alan Cox]

Ar Iau, 2006-08-17 am 00:10 +0100, ysgrifennodd Russell King:
> MIDI uses its own driver - sound/drivers/serial-u16550.c.  My guess

How peculiar

> is there's something in the system starving interrupt servicing.
> Serial is very sensitive to that, and increases in other system
> latencies tends to have an adverse impact on serial.

I see no support for the 16650 specific bits in the driver, so that
alone may be a problem ?

Re: How to avoid serial port buffer overruns?

[Russell King]

On Thu, Aug 17, 2006 at 10:20:46AM +0100, Alan Cox wrote:
> Ar Iau, 2006-08-17 am 00:10 +0100, ysgrifennodd Russell King:
> > MIDI uses its own driver - sound/drivers/serial-u16550.c.  My guess
> 
> How peculiar
> 
> > is there's something in the system starving interrupt servicing.
> > Serial is very sensitive to that, and increases in other system
> > latencies tends to have an adverse impact on serial.
> 
> I see no support for the 16650 specific bits in the driver, so that
> alone may be a problem ?

Huh?  16550 not 16650.  Did you miss this?

        outb(UART_FCR_ENABLE_FIFO       /* Enable FIFO's (if available) */
             | UART_FCR_CLEAR_RCVR      /* Clear receiver FIFO */
             | UART_FCR_CLEAR_XMIT      /* Clear transmitter FIFO */
             | UART_FCR_TRIGGER_4       /* Set FIFO trigger at 4-bytes */
        /* NOTE: interrupt generated after T=(time)4-bytes
         * if less than UART_FCR_TRIGGER bytes received
         */
             ,uart->base + UART_FCR);   /* FIFO Control Register */

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: How to avoid serial port buffer overruns?

[Alan Cox]

Ar Iau, 2006-08-17 am 10:28 +0100, ysgrifennodd Russell King:
> On Thu, Aug 17, 2006 at 10:20:46AM +0100, Alan Cox wrote:
> > Ar Iau, 2006-08-17 am 00:10 +0100, ysgrifennodd Russell King:
> > > MIDI uses its own driver - sound/drivers/serial-u16550.c.  My guess
> > 
> > How peculiar
> > 
> > > is there's something in the system starving interrupt servicing.
> > > Serial is very sensitive to that, and increases in other system
> > > latencies tends to have an adverse impact on serial.
> > 
> > I see no support for the 16650 specific bits in the driver, so that
> > alone may be a problem ?
> 
> Huh?  16550 not 16650.  Did you miss this?

Raphael said "I forgot to mention the kind of UART in my mail but it is
a 16650A"

Re: How to avoid serial port buffer overruns?

[Paul Fulghum]

Alan Cox wrote:
> Raphael said "I forgot to mention the kind of UART in my mail but it is
> a 16650A"

There are two different applications reporting
problems in this one thread.

Lee's friend is the only one using MIDI,
Raphael is having problems with regular
serial communications.

Both are receive problems that started
with the 2.6 series, but may or may not be
the same problem. Lee is reporting lost data
and Raphael is reporting rx overruns.

I asked Lee to clarify which driver his friend is
using (the MIDI 16550 or standard serial 8250).
His original post stated "a serial MIDI card (supported
by the bog standard 8250 serial driver)".

-- 
Paul Fulghum
Microgate Systems, Ltd.

Re: How to avoid serial port buffer overruns?

[Raphael Hertzog]

On Thu, 17 Aug 2006, Alan Cox wrote:
> > > > is there's something in the system starving interrupt servicing.
> > > > Serial is very sensitive to that, and increases in other system
> > > > latencies tends to have an adverse impact on serial.
> > > 
> > > I see no support for the 16650 specific bits in the driver, so that
> > > alone may be a problem ?
> > 
> > Huh?  16550 not 16650.  Did you miss this?
> 
> Raphael said "I forgot to mention the kind of UART in my mail but it is
> a 16650A"

That's a typo. :-/ 
Two lines further in the same mail there was:

bash-2.05a# cat /proc/tty/driver/serial
serinfo:1.0 driver revision:
0: uart:16550A port:000003F8 irq:4 tx:31 rx:7 RTS|CTS|DTR|DSR
        ^^^^^^

So I meant 16550A. Sorry.

Regards,
-- 
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/

Re: How to avoid serial port buffer overruns?

[Raphael Hertzog]

On Wed, 16 Aug 2006, Paul Fulghum wrote:
> On Wed, 2006-08-16 at 17:12 -0400, Lee Revell wrote:
> > 2.6.15 and 2.6.16.  Here is the .config:
> 
> Alan's rework of the receive tty buffering went
> into 2.6.16 and cured some problems, but clearly not yours.
> Some more adjustments are in 2.6.18-rc4, so that
> would be interesting to try for diagnosing this.

I will try 2.6.18-rc4 and keep you informed.

> I was wondering if the problem was interrupt latency,
> the tty receive buffering, or something totally different.
> I don't know if your problem and Raphael's are caused
> by the same mechanism. I would still like to know which
> kernel versions he has tried.

I tried 2.6.17.7.

But I'm really not sure that the 2.6 is a regression from 2.4, in fact I
think it does better by default.

The stock 2.4.31 kernel I was using had serial overruns at 9600 bauds
already. Once patched with the low latency/preemptive kernel patchs, it
was way better and I had only overruns at 115200 bauds.

With the 2.6.17.7 kernel (configured with CONFIG_PREEMPT and
CONFIG_HZ=1000), I'm seeing overruns starting at 38400 bauds. So
compared to plain 2.4, it's better. However compared to the patched
2.4, it's worse.

(the figures are *very approximative* as the overruns haven't been
detected with the same test conditions on 2.4 and on 2.6)

I have no result with the 2.6.17.7 patched with the real time patch of
Ingo/Thomas since it currently doesn't work on my card (see my separate
bugreport).

(those questions may have been directed to Lee but I'll respond in my case
as well)
> Are you using the low_latency flag on the serial device?

I tried that option with the 2.4 kernel and it didn't improve the situation
at all, and I haven't retried it with the 2.6 yet. But I will do.

> What type of UART has been tested (16550? other?)

I'm using only 16550 and I have no choice here, it's an "off the shelf"
card.

> Are you seeing overruns or just lost data?

I'm seeing overruns.

Regards,
-- 
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/

Re: How to avoid serial port buffer overruns?

[Paul Fulghum]

Raphael Hertzog wrote:
> I tried 2.6.17.7.
> 
> But I'm really not sure that the 2.6 is a regression from 2.4, in fact I
> think it does better by default.
> 
> The stock 2.4.31 kernel I was using had serial overruns at 9600 bauds
> already. Once patched with the low latency/preemptive kernel patchs, it
> was way better and I had only overruns at 115200 bauds.
> 
> With the 2.6.17.7 kernel (configured with CONFIG_PREEMPT and
> CONFIG_HZ=1000), I'm seeing overruns starting at 38400 bauds. So
> compared to plain 2.4, it's better. However compared to the patched
> 2.4, it's worse.

This tells me your issue is not a problem with the
serial or tty code, but rather a matter of IRQ latency.
(Which you may have already known, but I was unclear on)
I do not expect 2.6.18-rc4 to make a difference.

For fun, have you tried playing with the rx FIFO trigger
level in the 16550A entry in drivers/serial/8250.c ?
You could try replacing UART_FCR_R_TRIG_10 (8 char trigger)
with UART_FCR_R_TRIG_01 (4 char trigger) or even
UART_FCR_R_TRIG_00 (1 char trigger).
That creates more interrupts, but allows
more time to activate the ISR before overrun.

Lee's issue may still merit investigation into the
serial/tty code.

-- 
Paul Fulghum
Microgate Systems, Ltd.

Re: How to avoid serial port buffer overruns?

[Lee Revell]

On Thu, 2006-08-17 at 18:10 +0200, Raphael Hertzog wrote:
> With the 2.6.17.7 kernel (configured with CONFIG_PREEMPT and
> CONFIG_HZ=1000), I'm seeing overruns starting at 38400 bauds. So
> compared to plain 2.4, it's better. However compared to the patched
> 2.4, it's worse.
> 

(Sorry for hijacking your thread)

Have you tried it with HZ=100?  HZ=1000 might just be too much for that
board.

Lee

Re: How to avoid serial port buffer overruns?

[Raphael Hertzog]

Hello,

I took the time to try some of your suggestions and I managed to get rid
of the overruns.

On Fri, 18 Aug 2006, Lee Revell wrote:
> Have you tried it with HZ=100?  HZ=1000 might just be too much for that
> board.

This indeed was a major problem and a bad choice of mine at the very
beginning. It goes way better with HZ=100.

On Fri, 18 Aug 2006, Paul Fulghul wrote:
> For fun, have you tried playing with the rx FIFO trigger
> level in the 16550A entry in drivers/serial/8250.c ?
> You could try replacing UART_FCR_R_TRIG_10 (8 char trigger)
> with UART_FCR_R_TRIG_01 (4 char trigger) or even
> UART_FCR_R_TRIG_00 (1 char trigger).
> That creates more interrupts, but allows
> more time to activate the ISR before overrun.

I changed the rx FIFO trigger level to 1 byte (UART_FCR_R_TRIG_00) and it
helped a lot as well. With this combination I completely resolved the
problem of overruns at full speed (115200 bauds).

For the sake of comparison, I made a similar change to the 2.4.31 kernel I
was using (ie a kernel with low latency/preemptible kernel patches).

It helped a lot as well: most of the time I wouldn't have overruns (before
they were very frequent, like at least one overrun in 10k chars received).
However from time to time I would suffer from a single big overrun (like 30
chars lost). And using heavily the disk on module will increase the
likelihood to have a buffer overrun.

With the 2.6.17.7 kernel (CONFIG_HZ=100 and patched to trigger IRQ at
1 byte received), I have been completely unable to reproduce the buffer
overruns whatever read/write operation I've been triggering during the data
exchange.

So all in all, the 2.6 kernel behaves better than the 2.4 in this
case.

Cheers,
-- 
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/

Re: How to avoid serial port buffer overruns?

[Robert Hancock]

Raphael Hertzog wrote:
> (Please CC me when replying)
> 
> Hello,
> 
> While using Linux on low-end (semi-embedded) hardware (386 SX 40Mhz, 8Mb
> RAM), I discovered that Linux on that machine would suffer from serial
> port buffer overruns quite easily if I use a baudrate high enough (I start
> loosing bytes at >19200 bauds and I would like to make it reliable up to 
> 115 kbauds). I check if overruns are happening with
> /proc/tty/driver/serial ("oe" field).

What kind of serial port are you using? If it's an unbuffered 8250-type 
port, it will NEVER be reliable at higher baud rates. You want a 16550 
(or better) port.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: How to avoid serial port buffer overruns?

[Erik Mouw]

On Wed, Aug 16, 2006 at 08:35:05AM -0600, Robert Hancock wrote:
> Raphael Hertzog wrote:
> >(Please CC me when replying)
> >
> >Hello,
> >
> >While using Linux on low-end (semi-embedded) hardware (386 SX 40Mhz, 8Mb
> >RAM), I discovered that Linux on that machine would suffer from serial
> >port buffer overruns quite easily if I use a baudrate high enough (I start
> >loosing bytes at >19200 bauds and I would like to make it reliable up to 
> >115 kbauds). I check if overruns are happening with
> >/proc/tty/driver/serial ("oe" field).
> 
> What kind of serial port are you using? If it's an unbuffered 8250-type 
> port, it will NEVER be reliable at higher baud rates. You want a 16550 
> (or better) port.

I've run a cable modem connection at 115k2 over a 16450 UART on a
386DX40 for years. The tricks were:

- hdparm -u 1 /dev/hda
- play with IRQ priorities in such a way that the serial port UART was
  handled first (there used to be a kernel module to accomplish that in
  old debian releases)


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands