API to flush rx fifo?

API to flush rx fifo?

[Grant Edwards]

I see the uart_ops.flush_buffer method which is used to flush the
UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).

How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

-- 
Grant Edwards               grant.b.edwards        Yow! I'm shaving!!
                                  at               I'M SHAVING!!
                              gmail.com            

Re: API to flush rx fifo?

[Peter Hurley]

On 06/12/2013 04:03 PM, Grant Edwards wrote:
> I see the uart_ops.flush_buffer method which is used to flush the
> UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).
>
> How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

It doesn't.

If you're seeing stale i/o, it's more likely due to the flip buffers
not being flushed (which I think is wrong but I haven't had the
inclination to do all the historical research to fix that).

Regards,
Peter Hurley

Re: API to flush rx fifo?

[Grant Edwards]

On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:
> On 06/12/2013 04:03 PM, Grant Edwards wrote:
>> I see the uart_ops.flush_buffer method which is used to flush the
>> UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).
>>
>> How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?
>
> It doesn't.

Thanks.  I couldn't see any mechanism to do that, and I thought I must
be missing something.

> If you're seeing stale i/o, it's more likely due to the flip buffers
> not being flushed

Probably.  There is a scenario where you can get old data because the
rx fifo isn't flushed, but I suspect it's not what my customer is
complaining about.  FWIW, here's the scenario I'm worrying about:

   1) Enable either RTS/CTS or Xon/Xoff flow control for a UART driver
      that handles that flow control in hardware[1].

   2) Stop making read() calls on the tty device.

   3) The buffers in the tty layer fill up, so the uart driver stops
      transferring data from the rx fifo to the tty layer.

   4) The rx fifo fills up, and the flow control stops the other end
      from sending data.

   [all working OK up to this point, now you wait for an arbitrary
   amount of time]
      
   5) tcflush(TCIFLUSH) is called.

   [data in the tty layer gets flushed, but old data in the rx
    fifo remains]
   
   6) Now that there's room for more rx data in the tty layer, the
      uart driver resumes transferring (old) data from the rx fifo to
      the tty layer.

   7) read() is called and returns data that was received an arbitrary
      amount of time before tcflush(TCIFLUSH) was called.

My "old" drivers that interfaced directly with the tty layer handled
this scenario, but those drivers were becoming unmaintainable because
of instability of the tty API over the range of kernel versions I
support. So, I converted them over to be "uart drivers" for the common
serial layer which has a much more stable API (and generally requires
much less code). Now I've got a customer complaining about not being
able to flush data, so I'm looking closer at how the tcflush() calls
are handled.

[1] Because of a bug in the serial-driver layer's handling of the
    setting of Xon/Xoff characters by the settermios() call, it's not
    possible to correctly use Xon/Xoff support in a UART for the case
    where the user wants to use non-default Xon/Xoff characters.

-- 
Grant Edwards               grant.b.edwards        Yow! I feel ... JUGULAR ...
                                  at               
                              gmail.com            

Re: API to flush rx fifo?

[Peter Hurley]

On 06/14/2013 11:17 AM, Grant Edwards wrote:
> On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:
>> On 06/12/2013 04:03 PM, Grant Edwards wrote:
>>> I see the uart_ops.flush_buffer method which is used to flush the
>>> UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).
>>>
>>> How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?
>>
>> It doesn't.
>
> Thanks.  I couldn't see any mechanism to do that, and I thought I must
> be missing something.
>
>> If you're seeing stale i/o, it's more likely due to the flip buffers
>> not being flushed
>
> Probably.  There is a scenario where you can get old data because the
> rx fifo isn't flushed, but I suspect it's not what my customer is
> complaining about.  FWIW, here's the scenario I'm worrying about:
>
>     1) Enable either RTS/CTS or Xon/Xoff flow control for a UART driver
>        that handles that flow control in hardware[1].
>
>     2) Stop making read() calls on the tty device.
>
>     3) The buffers in the tty layer fill up, so the uart driver stops
>        transferring data from the rx fifo to the tty layer.
>
>     4) The rx fifo fills up, and the flow control stops the other end
>        from sending data.
>
>     [all working OK up to this point, now you wait for an arbitrary
>     amount of time]
>
>     5) tcflush(TCIFLUSH) is called.
>
>     [data in the tty layer gets flushed, but old data in the rx
>      fifo remains]

Yep. Your driver continues to push new data as it should, but that's
getting buffered up in the flip buffers. So that is what the app is
reading now that it has restarted read()s.

Your hardware rx fifo shouldn't have stale data in it because that
should generate an overrun; ie., if the flip buffers cannot accept
data because they're full then the next char pushed when space becomes
available should be a NUL flagged with TTY_OVERRUN.

Would a trial patch help (something that you might have to carry yourself
for a little while until I could do some historical research)?

>     6) Now that there's room for more rx data in the tty layer, the
>        uart driver resumes transferring (old) data from the rx fifo to
>        the tty layer.
>
>     7) read() is called and returns data that was received an arbitrary
>        amount of time before tcflush(TCIFLUSH) was called.
>
> My "old" drivers that interfaced directly with the tty layer handled
> this scenario, but those drivers were becoming unmaintainable because
> of instability of the tty API over the range of kernel versions I
> support. So, I converted them over to be "uart drivers" for the common
> serial layer which has a much more stable API (and generally requires
> much less code). Now I've got a customer complaining about not being
> able to flush data, so I'm looking closer at how the tcflush() calls
> are handled.
>
> [1] Because of a bug in the serial-driver layer's handling of the
>      setting of Xon/Xoff characters by the settermios() call, it's not
>      possible to correctly use Xon/Xoff support in a UART for the case
>      where the user wants to use non-default Xon/Xoff characters.

Can you be more specific about why setting non-default START_CHAR() and
STOP_CHAR() didn't work?  I recently overhauled this code so if there's
a problem here, we should probably fix it.

Regards,
Peter Hurley

Re: API to flush rx fifo?

[Grant Edwards]

On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:
> On 06/14/2013 11:17 AM, Grant Edwards wrote:
>> On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:
>>> On 06/12/2013 04:03 PM, Grant Edwards wrote:
>>>> I see the uart_ops.flush_buffer method which is used to flush the
>>>> UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).
>>>>
>>>> How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?
>>>
>>> It doesn't.
>>
>> Thanks.  I couldn't see any mechanism to do that, and I thought I must
>> be missing something.
>>
>>> If you're seeing stale i/o, it's more likely due to the flip buffers
>>> not being flushed
>>
>> Probably.  There is a scenario where you can get old data because the
>> rx fifo isn't flushed, but I suspect it's not what my customer is
>> complaining about.  FWIW, here's the scenario I'm worrying about:
>>
>>     1) Enable either RTS/CTS or Xon/Xoff flow control for a UART driver
>>        that handles that flow control in hardware[1].
>>
>>     2) Stop making read() calls on the tty device.
>>
>>     3) The buffers in the tty layer fill up, so the uart driver stops
>>        transferring data from the rx fifo to the tty layer.
>>
>>     4) The rx fifo fills up, and the flow control stops the other end
>>        from sending data.
>>
>>     [all working OK up to this point, now you wait for an arbitrary
>>     amount of time]
>>
>>     5) tcflush(TCIFLUSH) is called.
>>
>>     [data in the tty layer gets flushed, but old data in the rx
>>      fifo remains]
>
> Yep. Your driver continues to push new data as it should, but that's
> getting buffered up in the flip buffers. So that is what the app is
> reading now that it has restarted read()s.
>
> Your hardware rx fifo shouldn't have stale data in it because that
> should generate an overrun; ie., if the flip buffers cannot accept
> data because they're full then the next char pushed when space
> becomes available should be a NUL flagged with TTY_OVERRUN.

If flow control is enabled, there should be no rx overruns -- that's
what flow control is for.  In the scenario above, flow control is
enabled (and working).  In order to allow the UART to handle flow
control, the UART driver must stop reading data from the rx fifo when
the tty layer is "full".  The documentation for the serial core API
specifically states that UARTs are allowed to implement flow control
in hardware, and the only way that can be done is to alow the rx fifo
to fill up when the application stops makeing read() calls and the tty
layer fills up.

I think in newer kernels instead of explicitly checking for room in
the tty layer before unloading the rx fifo, the UART is supposed to
rely on the throttle/unthrottle callbacks, but the end result is the
same: when the tty layer gets "full", the UART driver stops reading
data from the rx fifo, and the rx fifo fills up.

> Would a trial patch help (something that you might have to carry
> yourself for a little while until I could do some historical
> research)?

Let's not do that yet.  The bug report from the customer is
sufficiently vague that I'm not sure the above scenario is what he's
run into.  I'm currently waiting on more detailed info, but probably
won't have an update for a few days.

>> [1] Because of a bug in the serial-driver layer's handling of the
>>      setting of Xon/Xoff characters by the settermios() call, it's not
>>      possible to correctly use Xon/Xoff support in a UART for the case
>>      where the user wants to use non-default Xon/Xoff characters.
>
> Can you be more specific about why setting non-default START_CHAR() and
> STOP_CHAR() didn't work?  I recently overhauled this code so if there's
> a problem here, we should probably fix it.

<pulling foot out of mouth>

It was fixed in 3.8!

I'll re-enable support for hardware supported xon/xoff flow control in
my drivers when they're being built for kernels >= 3.8.
  
-- 
Grant Edwards               grant.b.edwards        Yow! Hmmm ... an arrogant
                                  at               bouquet with a subtle
                              gmail.com            suggestion of POLYVINYL
                                                   CHLORIDE ...

Re: API to flush rx fifo?

[Peter Hurley]

On 06/14/2013 12:29 PM, Grant Edwards wrote:
> On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:
>> On 06/14/2013 11:17 AM, Grant Edwards wrote:
>>> On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:
>>>> On 06/12/2013 04:03 PM, Grant Edwards wrote:
>>>>> I see the uart_ops.flush_buffer method which is used to flush the
>>>>> UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).
>>>>>
>>>>> How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?
>>>>
>>>> It doesn't.
>>>
>>> Thanks.  I couldn't see any mechanism to do that, and I thought I must
>>> be missing something.
>>>
>>>> If you're seeing stale i/o, it's more likely due to the flip buffers
>>>> not being flushed
>>>
>>> Probably.  There is a scenario where you can get old data because the
>>> rx fifo isn't flushed, but I suspect it's not what my customer is
>>> complaining about.  FWIW, here's the scenario I'm worrying about:
>>>
>>>      1) Enable either RTS/CTS or Xon/Xoff flow control for a UART driver
>>>         that handles that flow control in hardware[1].
>>>
>>>      2) Stop making read() calls on the tty device.
>>>
>>>      3) The buffers in the tty layer fill up, so the uart driver stops
>>>         transferring data from the rx fifo to the tty layer.
>>>
>>>      4) The rx fifo fills up, and the flow control stops the other end
>>>         from sending data.
>>>
>>>      [all working OK up to this point, now you wait for an arbitrary
>>>      amount of time]
>>>
>>>      5) tcflush(TCIFLUSH) is called.
>>>
>>>      [data in the tty layer gets flushed, but old data in the rx
>>>       fifo remains]
>>
>> Yep. Your driver continues to push new data as it should, but that's
>> getting buffered up in the flip buffers. So that is what the app is
>> reading now that it has restarted read()s.
>>
>> Your hardware rx fifo shouldn't have stale data in it because that
>> should generate an overrun; ie., if the flip buffers cannot accept
>> data because they're full then the next char pushed when space
>> becomes available should be a NUL flagged with TTY_OVERRUN.
>
> If flow control is enabled, there should be no rx overruns -- that's
> what flow control is for.  In the scenario above, flow control is
> enabled (and working).  In order to allow the UART to handle flow
> control, the UART driver must stop reading data from the rx fifo when
> the tty layer is "full".  The documentation for the serial core API
> specifically states that UARTs are allowed to implement flow control
> in hardware, and the only way that can be done is to alow the rx fifo
> to fill up when the application stops makeing read() calls and the tty
> layer fills up.
>
> I think in newer kernels instead of explicitly checking for room in
> the tty layer before unloading the rx fifo, the UART is supposed to
> rely on the throttle/unthrottle callbacks, but the end result is the
> same: when the tty layer gets "full", the UART driver stops reading
> data from the rx fifo, and the rx fifo fills up.

AFAIK, only USB serial stops reading the rx fifo on throttle;
the serial core and other tty drivers continue to empty the rx fifo --
throttle only shuts off the transmitter on the other end.

Without handling throttle/unthrottle, how are you determining that the
tty layer is "full"?  Return code from tty_insert_flip_xxxx()?

Regards,
Peter Hurley

Re: API to flush rx fifo?

[Grant Edwards]

On Fri, Jun 14, 2013 at 01:09:56PM -0400, Peter Hurley wrote:
> On 06/14/2013 12:29 PM, Grant Edwards wrote:
>
> >> Your hardware rx fifo shouldn't have stale data in it because that
> >> should generate an overrun; ie., if the flip buffers cannot accept
> >> data because they're full then the next char pushed when space
> >> becomes available should be a NUL flagged with TTY_OVERRUN.
> >
> > If flow control is enabled, there should be no rx overruns -- that's
> > what flow control is for.  In the scenario above, flow control is
> > enabled (and working).  In order to allow the UART to handle flow
> > control, the UART driver must stop reading data from the rx fifo when
> > the tty layer is "full".  The documentation for the serial core API
> > specifically states that UARTs are allowed to implement flow control
> > in hardware, and the only way that can be done is to alow the rx fifo
> > to fill up when the application stops makeing read() calls and the tty
> > layer fills up.
> >
> > I think in newer kernels instead of explicitly checking for room in
> > the tty layer before unloading the rx fifo, the UART is supposed to
> > rely on the throttle/unthrottle callbacks, but the end result is the
> > same: when the tty layer gets "full", the UART driver stops reading
> > data from the rx fifo, and the rx fifo fills up.
> 
> AFAIK, only USB serial stops reading the rx fifo on throttle;

All the drivers I maintain do that.  It's the only way to get flow
control to work.  For UART with large FIFOs (e.g. 1KB) -- espcially
those attached via USB or Ethernet -- flow control driven by code in
serial_core just doesn't work right: you've got to let the UART handle
it.

> the serial core and other tty drivers continue to empty the rx fifo
> -- throttle only shuts off the transmitter on the other end.
> 
> Without handling throttle/unthrottle, how are you determining that the
> tty layer is "full"?  Return code from tty_insert_flip_xxxx()?

I check tty->receive_room.  What are you supposed to do for kernel
versions that don't have the throttle()/unthrottle() callbacks?

-- 
Grant

Re: API to flush rx fifo?

[Peter Hurley]

On 06/14/2013 01:39 PM, Grant Edwards wrote:
> On Fri, Jun 14, 2013 at 01:09:56PM -0400, Peter Hurley wrote:
>> On 06/14/2013 12:29 PM, Grant Edwards wrote:
>>
>>>> Your hardware rx fifo shouldn't have stale data in it because that
>>>> should generate an overrun; ie., if the flip buffers cannot accept
>>>> data because they're full then the next char pushed when space
>>>> becomes available should be a NUL flagged with TTY_OVERRUN.
>>>
>>> If flow control is enabled, there should be no rx overruns -- that's
>>> what flow control is for.  In the scenario above, flow control is
>>> enabled (and working).  In order to allow the UART to handle flow
>>> control, the UART driver must stop reading data from the rx fifo when
>>> the tty layer is "full".  The documentation for the serial core API
>>> specifically states that UARTs are allowed to implement flow control
>>> in hardware, and the only way that can be done is to alow the rx fifo
>>> to fill up when the application stops makeing read() calls and the tty
>>> layer fills up.
>>>
>>> I think in newer kernels instead of explicitly checking for room in
>>> the tty layer before unloading the rx fifo, the UART is supposed to
>>> rely on the throttle/unthrottle callbacks, but the end result is the
>>> same: when the tty layer gets "full", the UART driver stops reading
>>> data from the rx fifo, and the rx fifo fills up.
>>
>> AFAIK, only USB serial stops reading the rx fifo on throttle;
>
> All the drivers I maintain do that.  It's the only way to get flow
> control to work.  For UART with large FIFOs (e.g. 1KB) -- espcially
> those attached via USB or Ethernet -- flow control driven by code in
> serial_core just doesn't work right: you've got to let the UART handle
> it.

I had a similar situation with the firewire serial driver (which fakes
serial i/o over the firewire bus @ 250~400Mb/s). The existing throttle
mechanism is too ponderous to shut-off the transmitter before the
receiver overflows the flip buffers.

Currently, that driver buffers rx up on the receiver side but I plan to
change that; right now, the plan is to track the "high watermark" in the
flip buffers and self-throttle when that is too low. Also, I might expose
a way of upping the max flip buffer amount per-port (but probably not
until I add a no-flags-buffer for tty_insert_flip_string_fixed_flag()).

>> the serial core and other tty drivers continue to empty the rx fifo
>> -- throttle only shuts off the transmitter on the other end.
>>
>> Without handling throttle/unthrottle, how are you determining that the
>> tty layer is "full"?  Return code from tty_insert_flip_xxxx()?
>
> I check tty->receive_room.

That will be going away as a means of flow control because it's not
thread-safe (if you backscan this list, my 'lockless n_tty receive path'
patchset only keeps tty->receive_room for the non-flow controlled
line disciplines).

> What are you supposed to do for kernel
> versions that don't have the throttle()/unthrottle() callbacks?

Which versions specifically do you mean?

Regards,
Peter Hurley

Re: API to flush rx fifo?

[Grant Edwards]

On 2013-06-14, Grant Edwards <grant.b.edwards@gmail.com> wrote:

>> the serial core and other tty drivers continue to empty the rx fifo
>> -- throttle only shuts off the transmitter on the other end.

Let's say that the tty layer fills up, and calls the uart driver's
throttle() callback.  Now, instead of stopping the unloading of the rx
FIFO (like my drivers do), we explicitly send an XOFF and continue to
unload the rx FIFO.  Will the tty layer accept the 1KB (or 4KB or 8KB)
of data that's already in the rx FIFO without any loss/overrun? AFAIK,
it can't.  That's why my drivers (and the USB drivers) do what they do.

>From another point of view: not reading the rx fifo is how one "shuts
off the transmitter on the other end" (eventually) when one is using
the UART's flow control support -- at least that's how all the
flow-control-supporting UARTs I've seen work.  Stopping the other end
just because the tty layer is full is also wasteful: some of my
"UARTs" can buffer up to 9KB of receive data in the rx fifo[1] after
the tty layer fills up. They'll tell the other end to stop when it
needs to be stopped. Aside from that, having the flow state controlled
from two places (serial_core and the UART itself) tends to be
error-prone.

When you're dealing with high baud rates (e.g. 921K) and a bus like
USB or Ethernet, where there may be tens or hundreds of milliseconds
of latency between the serial_core and the physical UART chip, and
FIFO sizes are measured in KB.  In those situations there just isn't
any way for either the serial_core or tty layer to know when the
transmitter on the other end needs to be turned on or off.  You have
to let the UART handle it. All the tty/serial_core layer knows is
whether or not it has room for more receive data, and that's not the
same thing as knowing when the transmitter on other end of the serial
cable needs to be turned on/off.

So, when I add the UART-driver callbacks for throttle/unthrottle, what
they'll have to do is control whether we read data from the rx FIFO or
not.

So that implies that for tcflush(TCIFLUSH) to reliably do what a user
would expect, there needs to be an input_flush() callback so that the
uart driver can be told to flush the rx fifo.  [Yes, flushing the "rx
fifo" in some of these configurations isn't a simple task that takes a
negligible amount of time, but it can be done.]

>> Without handling throttle/unthrottle, how are you determining that the
>> tty layer is "full"?  Return code from tty_insert_flip_xxxx()?
>
> I check tty->receive_room.  What are you supposed to do for kernel
> versions that don't have the throttle()/unthrottle() callbacks?

Actually, in another driver I check the return value from 
tty_prepare_flip_string_flags().  For various (mostly historical)
reaons, one driver uses tty_prepare_flip_string_flags() and the other
uses uart_insert_char() [both followed later by a call to
tty_flip_buffer_push()].

And in yet another driver (that talks to the tty layer instead of
using the serial_core), I call the line discipline's receive_room()
method -- but that's moot for this thread.


[1] For ethernet/usb attached UARTs the "rx fifo" includes (from
    serial_core's POV) not only the hardware fifo in the UART (up to
    1KB), but also software buffers at either end of the network
    "connection" (up to 4KB at each end).

-- 
Grant Edwards               grant.b.edwards        Yow! Is this an out-take
                                  at               from the "BRADY BUNCH"?
                              gmail.com            

Re: API to flush rx fifo?

[Grant Edwards]

On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:

>> All the drivers I maintain do that.  It's the only way to get flow
>> control to work.  For UART with large FIFOs (e.g. 1KB) -- espcially
>> those attached via USB or Ethernet -- flow control driven by code in
>> serial_core just doesn't work right: you've got to let the UART handle
>> it.
>
> I had a similar situation with the firewire serial driver (which fakes
> serial i/o over the firewire bus @ 250~400Mb/s). The existing throttle
> mechanism is too ponderous to shut-off the transmitter before the
> receiver overflows the flip buffers.

Any time you combine high baud rates, large FIFOs, and latency, you
run into that problem.

>>> Without handling throttle/unthrottle, how are you determining that the
>>> tty layer is "full"?  Return code from tty_insert_flip_xxxx()?
>>
>> I check tty->receive_room.
>
> That will be going away as a means of flow control because it's not
> thread-safe (if you backscan this list, my 'lockless n_tty receive
> path' patchset only keeps tty->receive_room for the non-flow
> controlled line disciplines).

Good to know.

Will tty_prepare_flip_string_flags() continue to return a "room"
value?  If so, then I could also switch over to using that instead of
looking at tty->receive_room.  [The advantage being simpler backwards
compatibility.]

The tty_prepare_flip_string_flags() approach also uses a lot less CPU
time than the uart_insert_char() approach.

I'll probably add throttle()/unthrottle() callbacks that set/clear an
internal flag that tells the driver whether or not to read data from
the rx fifo. That should less overhead than polling the tty layer by
calling tty_prepare_flip_string_flags(). But, it doesn't help the
situation for kernels before 3.8.

>> What are you supposed to do for kernel versions that don't have the
>> throttle()/unthrottle() callbacks?
>
> Which versions specifically do you mean?

My serial_core UART drivers support 2.6.25 and newer. The tech support
guys would like support further back, but I've given up trying for
anything older than that. Before our recent change-over to serial_core
drivers (a few months back) I supported 2.6.18 and later. Tech support
would have liked to go back to 2.6.12, which is still in use by some
customers who've tested their systems with some ancient version of RH
and don't want to change it.  One customer just moved from 2.4 to 2.6
about a year ago.

I believe that the throttle/unthrottle callbacks didn't show up until
3.8.  I doubt we even have any customers running 3.8 yet. :)

-- 
Grant Edwards               grant.b.edwards        Yow! I like your SNOOPY
                                  at               POSTER!!
                              gmail.com            

Re: API to flush rx fifo?

[Peter Hurley]

On 06/14/2013 02:41 PM, Grant Edwards wrote:
> On 2013-06-14, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
>>> the serial core and other tty drivers continue to empty the rx fifo
>>> -- throttle only shuts off the transmitter on the other end.
>
> Let's say that the tty layer fills up, and calls the uart driver's
> throttle() callback.  Now, instead of stopping the unloading of the rx
> FIFO (like my drivers do), we explicitly send an XOFF and continue to
> unload the rx FIFO.  Will the tty layer accept the 1KB (or 4KB or 8KB)
> of data that's already in the rx FIFO without any loss/overrun? AFAIK,
> it can't.  That's why my drivers (and the USB drivers) do what they do.

The tty flip buffers are 64KB+ deep. That should be more than enough
for any rate <= 4Mbits/s or so.

Also, those lockless patchsets (which I'm about to refresh for inclusion
in 3.12) I think will dramatically improve the tty layer throughput,
which in turn should mean much less throttling.

I understand that the tty flow control design is anything but ideal.

>  From another point of view: not reading the rx fifo is how one "shuts
> off the transmitter on the other end" (eventually) when one is using
> the UART's flow control support -- at least that's how all the
> flow-control-supporting UARTs I've seen work.  Stopping the other end
> just because the tty layer is full is also wasteful: some of my
> "UARTs" can buffer up to 9KB of receive data in the rx fifo[1] after
> the tty layer fills up. They'll tell the other end to stop when it
> needs to be stopped. Aside from that, having the flow state controlled
> from two places (serial_core and the UART itself) tends to be
> error-prone.
>
> When you're dealing with high baud rates (e.g. 921K) and a bus like
> USB or Ethernet, where there may be tens or hundreds of milliseconds
> of latency between the serial_core and the physical UART chip, and
> FIFO sizes are measured in KB.  In those situations there just isn't
> any way for either the serial_core or tty layer to know when the
> transmitter on the other end needs to be turned on or off.  You have
> to let the UART handle it. All the tty/serial_core layer knows is
> whether or not it has room for more receive data, and that's not the
> same thing as knowing when the transmitter on other end of the serial
> cable needs to be turned on/off.
 >
> So, when I add the UART-driver callbacks for throttle/unthrottle, what
> they'll have to do is control whether we read data from the rx FIFO or
> not.

In this case, I would ignore the throttle and only use the unthrottle
to restart rx (you would stop rx whenever tty_insert_flip_xxx()
returns < requested and restart rx when you receive unthrottle()).

The main problem with the throttle/unthrottle flow control right now
is the wrong internal tty layer is initiating it. For drivers that use
the flip buffer interface, that should be the layer that implements
flow control with the driver, perhaps with settable high & low watermarks.

Unfortunately, currently it's the line discipline that's communicating
throttle and unthrottle to the driver (which doesn't make sense for high
bandwidth devices because the read buffer is only 4KB).

> So that implies that for tcflush(TCIFLUSH) to reliably do what a user
> would expect, there needs to be an input_flush() callback so that the
> uart driver can be told to flush the rx fifo.  [Yes, flushing the "rx
> fifo" in some of these configurations isn't a simple task that takes a
> negligible amount of time, but it can be done.]

Yep. Plus the flip buffers need to be dumped as well.

>>> Without handling throttle/unthrottle, how are you determining that the
>>> tty layer is "full"?  Return code from tty_insert_flip_xxxx()?
>>
>> I check tty->receive_room.  What are you supposed to do for kernel
>> versions that don't have the throttle()/unthrottle() callbacks?
>
> Actually, in another driver I check the return value from
> tty_prepare_flip_string_flags().  For various (mostly historical)
> reaons, one driver uses tty_prepare_flip_string_flags() and the other
> uses uart_insert_char() [both followed later by a call to
> tty_flip_buffer_push()].

The flip buffer interface is the preferred method of pushing rx data
to the line discipline, so this is ok.

Checking the return value from tty_prepare_xxx() or tty_insert_xxx()
is fine for self-throttling.

> And in yet another driver (that talks to the tty layer instead of
> using the serial_core), I call the line discipline's receive_room()
> method -- but that's moot for this thread.
>
>
> [1] For ethernet/usb attached UARTs the "rx fifo" includes (from
>      serial_core's POV) not only the hardware fifo in the UART (up to
>      1KB), but also software buffers at either end of the network
>      "connection" (up to 4KB at each end).
>

Re: API to flush rx fifo?

[Peter Hurley]

On 06/14/2013 03:12 PM, Grant Edwards wrote:
> On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:
>
>>> All the drivers I maintain do that.  It's the only way to get flow
>>> control to work.  For UART with large FIFOs (e.g. 1KB) -- espcially
>>> those attached via USB or Ethernet -- flow control driven by code in
>>> serial_core just doesn't work right: you've got to let the UART handle
>>> it.
>>
>> I had a similar situation with the firewire serial driver (which fakes
>> serial i/o over the firewire bus @ 250~400Mb/s). The existing throttle
>> mechanism is too ponderous to shut-off the transmitter before the
>> receiver overflows the flip buffers.
>
> Any time you combine high baud rates, large FIFOs, and latency, you
> run into that problem.
>
>>>> Without handling throttle/unthrottle, how are you determining that the
>>>> tty layer is "full"?  Return code from tty_insert_flip_xxxx()?
>>>
>>> I check tty->receive_room.
>>
>> That will be going away as a means of flow control because it's not
>> thread-safe (if you backscan this list, my 'lockless n_tty receive
>> path' patchset only keeps tty->receive_room for the non-flow
>> controlled line disciplines).
>
> Good to know.
>
> Will tty_prepare_flip_string_flags() continue to return a "room"
> value?  If so, then I could also switch over to using that instead of
> looking at tty->receive_room.  [The advantage being simpler backwards
> compatibility.]

Yes to both.

> The tty_prepare_flip_string_flags() approach also uses a lot less CPU
> time than the uart_insert_char() approach.
>
> I'll probably add throttle()/unthrottle() callbacks that set/clear an
> internal flag that tells the driver whether or not to read data from
> the rx fifo. That should less overhead than polling the tty layer by
> calling tty_prepare_flip_string_flags(). But, it doesn't help the
> situation for kernels before 3.8.
>
>>> What are you supposed to do for kernel versions that don't have the
>>> throttle()/unthrottle() callbacks?
>>
>> Which versions specifically do you mean?
>
> My serial_core UART drivers support 2.6.25 and newer. The tech support
> guys would like support further back, but I've given up trying for
> anything older than that. Before our recent change-over to serial_core
> drivers (a few months back) I supported 2.6.18 and later. Tech support
> would have liked to go back to 2.6.12, which is still in use by some
> customers who've tested their systems with some ancient version of RH
> and don't want to change it.  One customer just moved from 2.4 to 2.6
> about a year ago.
>
> I believe that the throttle/unthrottle callbacks didn't show up until
> 3.8.  I doubt we even have any customers running 3.8 yet. :)

Either backporting that commit or something similar specifically for
your driver is the only realistic solution. There was nothing suitable
before that.

[ FWIW, I think that commit went in for 3.7 not 3.8 ]

Re: API to flush rx fifo?

[Peter Hurley]

Hi Grant,

I know this is old but I had marked this for re-review in my mail,
and after re-looking at this problem, I realized I gave you some bad info.

Since you were referring to uart_ops, I'll assume this is for your
serial-core mini-driver(s).


On 06/14/2013 11:46 AM, Peter Hurley wrote:
> On 06/14/2013 11:17 AM, Grant Edwards wrote:
>> On 2013-06-14, Peter Hurley <peter@hurleysoftware.com> wrote:
>>> On 06/12/2013 04:03 PM, Grant Edwards wrote:
>>>> I see the uart_ops.flush_buffer method which is used to flush the
>>>> UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).

The N_TTY ldisc calls tty->ops->flush_buffer() for TCOFLUSH _and TCIOFLUSH_.

[ For a serial-core mini-driver, tty->ops->flush_buffer() is uart_flush_buffer(),
   which calls uart_ops.flush_buffer().
]

>>>>
>>>> How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?
>>>
>>> It doesn't.

The ioctl(TCIFLUSH) does in fact flush the tty buffers.

If you want to flush your hardware rx fifo, you can define the .ioctl()
method in your struct uart_ops and handle TCIFLUSH there.

Note: be sure to return -ENOIOCTLCMD so that TCIFLUSH processing still
gets seen by the ldisc!

Like this (apologies in advance for formatting; my mailer is dain-bramaged),

/* called with port->mutex held */
static int xxxxxx_ioctl(struct uart_port *port, unsigned int cmd,
			unsigned long arg)
{
	if (cmd == TCFLSH) {
		switch (arg) {
		case TCIFLUSH:
		case TCIOFLUSH:
			spin_lock_irq(&port->lock);
			xxxxxx_empty_rx_fifo(port);
			spin_unlock_irq(&port->lock);
			/* fall-through */
		}
	}
	return -ENOIOCTLCMD;
}


Sorry, hope this isn't too late :(

Regards,
Peter Hurley

Re: API to flush rx fifo?

[Grant Edwards]

On 2014-02-26, Peter Hurley <peter@hurleysoftware.com> wrote:
> Hi Grant,
>
> I know this is old but I had marked this for re-review in my mail,
> and after re-looking at this problem, I realized I gave you some bad info.
>
> Since you were referring to uart_ops, I'll assume this is for your
> serial-core mini-driver(s).

Yup.

Upon review of this thread, I see that you had warned me that
tty->receive_room was going away, and I completely forgot about that
warning and didn't do anything about it about that until my drivers
actually broke 8 months later.

Shame on me.

>>>>> How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?
>>>>
>>>> It doesn't.
>
> The ioctl(TCIFLUSH) does in fact flush the tty buffers.
>
> If you want to flush your hardware rx fifo, you can define the
> .ioctl() method in your struct uart_ops and handle TCIFLUSH there.
>
> Note: be sure to return -ENOIOCTLCMD so that TCIFLUSH processing
> still gets seen by the ldisc!
>
> Like this (apologies in advance for formatting; my mailer is
> dain-bramaged),
>
> /* called with port->mutex held */
> static int xxxxxx_ioctl(struct uart_port *port, unsigned int cmd,
> 			unsigned long arg)
> {
> 	if (cmd == TCFLSH) {
> 		switch (arg) {
> 		case TCIFLUSH:
> 		case TCIOFLUSH:
> 			spin_lock_irq(&port->lock);
> 			xxxxxx_empty_rx_fifo(port);
> 			spin_unlock_irq(&port->lock);
> 			/* fall-through */
> 		}
> 	}
> 	return -ENOIOCTLCMD;
> }
>
> Sorry, hope this isn't too late :(

Great!  That's certalinly simple enough.

It turns out the customer's problem had nothing to do with the UART's
rx fifos not getting flushed when TCIFLUSH was done, so I put that
issue on the list of things to be worried about later (and that's
where it still is).

I'll add the above handling to my drivers just to be complete, but I
don't think it's actually ever caused any problems.

-- 
Grant Edwards               grant.b.edwards        Yow! I was making donuts
                                  at               and now I'm on a bus!
                              gmail.com