[Xenomai] xeno_16550A : "unrecoverable" receive buffer overflow on read

[Xenomai] xeno_16550A : "unrecoverable" receive buffer overflow on read

[Zemskov, Evgeny]

Greetings !

I'm trying to use xeno_16550A driver to read a stream of high-frequency data
(~50 byte packet each 20 msec at 57600 baud) with low latencies.

After successful installation of xeno_16550A.ko driver module and 
initialization of rtser0 device, an attempt to read from the device fails :
rt_dev_read returns -5 (Input/output error),
and subsequent ioctl RTSER_RTIOC_GET_STATUS shows the value of 
LSR=0x62 and MSR=0x5.

The data from rtser0 device can be read successfully after following sequence :
 - rmmod xeno_16550A
 - init /dev/ttyS0 to use standard Linux 8250 driver with setserial
 - start and stop reading from /dev/ttyS0
 - insmod xeno_16550A with valid parameters

Naturally, /dev/ttyS0 and rtser0 point to the same built-in serial port 
at 0x3F8.

The problem doesn't appear when I send to the port ocassional bursts of data
(sized around 20-30 bytes), i.e. rtser0 can be read from consistently after 
each restart of the utility (i use modified cross-link with truncated write 
functionality).

It seems like that xeno_16550A is unable to handle overflow of UART Rx buffer,
which is flushed on initialization by 8250 driver.

Could anybody suggest where in sources (of both 16550A and 8250) to look for 
the solution ?

Thanks, Eugene


Evgeny Zemskov, Transas Ltd
evgeny.zemskov@transas.com
Office phone # +7 812 3253131 add. 274-03
Cell phone # +7 952 3943129

Re: [Xenomai] xeno_16550A : "unrecoverable" receive buffer overflow on read

[Jan Kiszka]

On 2013-02-12 13:33, Zemskov, Evgeny wrote:
> Greetings !
> 
> I'm trying to use xeno_16550A driver to read a stream of high-frequency data
> (~50 byte packet each 20 msec at 57600 baud) with low latencies.

The driver was once developed to handle high-rate data streams at 1
Mbps, so it should generally work.


> After successful installation of xeno_16550A.ko driver module and 
> initialization of rtser0 device, an attempt to read from the device fails :
> rt_dev_read returns -5 (Input/output error),
> and subsequent ioctl RTSER_RTIOC_GET_STATUS shows the value of 
> LSR=0x62 and MSR=0x5.

rt_16550_context::status is reset after each error reported by
rt_dev_read, rt_16550_context::saved_status is reset after
RTSER_RTIOC_GET_STATUS. So, unless the problem re-arises continuously,
you should be able to recover the port. without that steps below.

Is there a packet frequency that is still fine? Are you sure your reader
task is able to keep up with the data stream? What is your RX FIFO
depth? Your hardware is 16550A compatible?

> 
> The data from rtser0 device can be read successfully after following sequence :
>  - rmmod xeno_16550A
>  - init /dev/ttyS0 to use standard Linux 8250 driver with setserial
>  - start and stop reading from /dev/ttyS0
>  - insmod xeno_16550A with valid parameters
> 
> Naturally, /dev/ttyS0 and rtser0 point to the same built-in serial port 
> at 0x3F8.
> 
> The problem doesn't appear when I send to the port ocassional bursts of data
> (sized around 20-30 bytes), i.e. rtser0 can be read from consistently after 
> each restart of the utility (i use modified cross-link with truncated write 
> functionality).
> 
> It seems like that xeno_16550A is unable to handle overflow of UART Rx buffer,
> which is flushed on initialization by 8250 driver.
> 
> Could anybody suggest where in sources (of both 16550A and 8250) to look for 
> the solution ?

Check the rate of RX interrupts and what status reported on each
(rt_16550_rx_interrupt). Do they complete successfully at all, only fail
after a while? Do they come regularly? Maybe something (SMI?) is
blocking interrupts for a longer period, causing the FIFO overflow.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] xeno_16550A : "unrecoverable" receive buffer overflow on read

[Zemskov, Evgeny]

>Is there a packet frequency that is still fine? Are you sure your reader
>task is able to keep up with the data stream? What is your RX FIFO
>depth? Your hardware is 16550A compatible?

>Check the rate of RX interrupts and what status reported on each
>(rt_16550_rx_interrupt). Do they complete successfully at all, only fail
>after a while? Do they come regularly? Maybe something (SMI?) is
>blocking interrupts for a longer period, causing the FIFO overflow.

At the moment we're testing with low frequencies, the same problem 
occurs even at 1..2 second intervals between packets.
The reader is definitely quick enough (essentially, it only executes an rt_printf 
with read notification), and it handles in time packets with at least 50msec intervals.

Interesting thing is that problem occurs _only_ if packets are received by UART 
during the startup or shutdown of the reader application.
In latter case, rt_dev_read fails on first packet received after startup (probably, when
 next RX interrupt is triggered).
No problem occurs is data is received by UART when application is not running.

All RX interrupts are triggered and executed in xeno_16550A driver correctly 
(I've added printk's in rt_16550_rx_interrupt), even after rt_dev_read has failed 
(but the application is still running and keeps driver initialized)/

Hardware is a serial port built into ASUS P4 motherboard with i865 chipset. 
Couldn't find which UART are they using, but it works fine with everything.
I believe that its Rx FIFO depth is standard 16 bytes (at least Rx trigger 
level can be set to standard 1-4-8-14 bytes)

SMI seem to be disabled properly : in kernel configuration, CONFIG_XENO_HW_SMI_ALL 
and CONFIG_XENO_HW_SMI_WORKAROUND are enabled.
In /var/log/dmseg, only the message "Xenomai: SMI-enabled chipset found" is shown.

Thanks,
Eugene

Evgeny Zemskov, Transas Ltd
evgeny.zemskov@transas.com
Office phone # +7 812 3253131 add. 274-03
Cell phone # +7 952 3943129

________________________________________
From: Jan Kiszka [jan.kiszka@siemens.com]
Sent: Tuesday, February 12, 2013 5:15 PM
To: Zemskov, Evgeny
Cc: xenomai@xenomai.org
Subject: Re: xeno_16550A : "unrecoverable" receive buffer overflow on read

On 2013-02-12 13:33, Zemskov, Evgeny wrote:
> Greetings !
>
> I'm trying to use xeno_16550A driver to read a stream of high-frequency data
> (~50 byte packet each 20 msec at 57600 baud) with low latencies.

The driver was once developed to handle high-rate data streams at 1
Mbps, so it should generally work.


> After successful installation of xeno_16550A.ko driver module and
> initialization of rtser0 device, an attempt to read from the device fails :
> rt_dev_read returns -5 (Input/output error),
> and subsequent ioctl RTSER_RTIOC_GET_STATUS shows the value of
> LSR=0x62 and MSR=0x5.

rt_16550_context::status is reset after each error reported by
rt_dev_read, rt_16550_context::saved_status is reset after
RTSER_RTIOC_GET_STATUS. So, unless the problem re-arises continuously,
you should be able to recover the port. without that steps below.

Is there a packet frequency that is still fine? Are you sure your reader
task is able to keep up with the data stream? What is your RX FIFO
depth? Your hardware is 16550A compatible?

>
> The data from rtser0 device can be read successfully after following sequence :
>  - rmmod xeno_16550A
>  - init /dev/ttyS0 to use standard Linux 8250 driver with setserial
>  - start and stop reading from /dev/ttyS0
>  - insmod xeno_16550A with valid parameters
>
> Naturally, /dev/ttyS0 and rtser0 point to the same built-in serial port
> at 0x3F8.
>
> The problem doesn't appear when I send to the port ocassional bursts of data
> (sized around 20-30 bytes), i.e. rtser0 can be read from consistently after
> each restart of the utility (i use modified cross-link with truncated write
> functionality).
>
> It seems like that xeno_16550A is unable to handle overflow of UART Rx buffer,
> which is flushed on initialization by 8250 driver.
>
> Could anybody suggest where in sources (of both 16550A and 8250) to look for
> the solution ?

Check the rate of RX interrupts and what status reported on each
(rt_16550_rx_interrupt). Do they complete successfully at all, only fail
after a while? Do they come regularly? Maybe something (SMI?) is
blocking interrupts for a longer period, causing the FIFO overflow.

Jan

--
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] xeno_16550A : "unrecoverable" receive buffer overflow on read

[Jan Kiszka]

On 2013-02-13 13:22, Zemskov, Evgeny wrote:
>> Is there a packet frequency that is still fine? Are you sure your reader
>> task is able to keep up with the data stream? What is your RX FIFO
>> depth? Your hardware is 16550A compatible?
> 
>> Check the rate of RX interrupts and what status reported on each
>> (rt_16550_rx_interrupt). Do they complete successfully at all, only fail
>> after a while? Do they come regularly? Maybe something (SMI?) is
>> blocking interrupts for a longer period, causing the FIFO overflow.
> 
> At the moment we're testing with low frequencies, the same problem 
> occurs even at 1..2 second intervals between packets.
> The reader is definitely quick enough (essentially, it only executes an rt_printf 
> with read notification), and it handles in time packets with at least 50msec intervals.
> 
> Interesting thing is that problem occurs _only_ if packets are received by UART 
> during the startup or shutdown of the reader application.
> In latter case, rt_dev_read fails on first packet received after startup (probably, when
>  next RX interrupt is triggered).
> No problem occurs is data is received by UART when application is not running.
> 
> All RX interrupts are triggered and executed in xeno_16550A driver correctly 
> (I've added printk's in rt_16550_rx_interrupt), even after rt_dev_read has failed 
> (but the application is still running and keeps driver initialized)/

Sorry, I still don't fully get the error: rt_dev_read suddenly returns
-EIO and, after that, never works again until you do the hardware
re-init via the Linux driver?

After the first EIO, what does the interrupt handler read from the LSR
register from that on? Always RTSER_LSR_OVERRUN_ERR?

> 
> Hardware is a serial port built into ASUS P4 motherboard with i865 chipset. 
> Couldn't find which UART are they using, but it works fine with everything.
> I believe that its Rx FIFO depth is standard 16 bytes (at least Rx trigger 
> level can be set to standard 1-4-8-14 bytes)

And you leave the trigger depth at "1", right?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux