Precise timing in ldisc? tty_put_char() in hrtimer context?

Precise timing in ldisc? tty_put_char() in hrtimer context?

[Tomáš Mudruňka]

Hello,
i am implementing niche rs485-based serial protocol as a tty line
discipline linux module. Requirement is to hit transmission window
with precision in low hundreds of microseconds (eg. +-200 uS).
Transmit window starts 500 uS after message is received.

It seems that hardware driver calls tty_receive_buf2() with sufficient
latency, so my idea was to start hrtimer using hrtimer_start() inside
of tty_receive_buf2(). And then in hrtimer handler call
for(...){tty_put_char(tty,...)} or tty->ops->write(), but it seems not
to be a good practice according to my online research. They say it's
recommended to call tty_put_char() from workqueue, but documentation
also says that calling a workqueue might introduce latency of couple
milliseconds, since it's scheduled thread context.

What would you suggest to handle such a strict timing requirements
between RX and TX data?

Thank you

Best regards
      Tomas Mudrunka

Re: Precise timing in ldisc? tty_put_char() in hrtimer context?

[Greg KH]

On Tue, Jan 07, 2025 at 07:07:42PM +0100, Tomáš Mudruňka wrote:
> Hello,
> i am implementing niche rs485-based serial protocol as a tty line
> discipline linux module. Requirement is to hit transmission window
> with precision in low hundreds of microseconds (eg. +-200 uS).
> Transmit window starts 500 uS after message is received.
> 
> It seems that hardware driver calls tty_receive_buf2() with sufficient
> latency, so my idea was to start hrtimer using hrtimer_start() inside
> of tty_receive_buf2(). And then in hrtimer handler call
> for(...){tty_put_char(tty,...)} or tty->ops->write(), but it seems not
> to be a good practice according to my online research. They say it's
> recommended to call tty_put_char() from workqueue, but documentation
> also says that calling a workqueue might introduce latency of couple
> milliseconds, since it's scheduled thread context.
> 
> What would you suggest to handle such a strict timing requirements
> between RX and TX data?

Use a different protocol or hardware implementation as having to be
forced to ensure that UART flows actually work that fast is going to be
very difficult for you to guarantee.  Just think, how do you know that
once the UART gets your byte, that it will not just sit around and wait
before it actually sends it out (hint, there is no such guarantee).

good luck!

greg k-h

Re: Precise timing in ldisc? tty_put_char() in hrtimer context?

[Tomáš Mudruňka]

> Use a different protocol or hardware implementation as having to be
> forced to ensure that UART flows actually work that fast is going to be
> very difficult for you to guarantee.  Just think, how do you know that
> once the UART gets your byte, that it will not just sit around and wait
> before it actually sends it out (hint, there is no such guarantee).

Thing is the timings are tunable and protocol is tolerant to message loss.
Tighter timings give me better latency, but higher loss. So it's about
tradeoffs and finding the sweetspot.

Actually i've already implemented this in userspace using N_TTY and it works.
But i am experimenting with the timings and i would like to see how tight
i can get by implementing this as custom ldsic instead of using N_TTY ldisc.
Just trying to squeeze couple extra microseconds out of this.

Originally this was running on microcontrollers and since those run realtime OS,
i was easily able to achieve 10x lower latency than with N_TTY.
I don't expect Linux with usb-serial dongle to perform same as realtime OS
with integrated UART peripheral, but i am pretty sure i can get to being
5x slower instead of 10x slower when compared to uC...

I am not very experienced with interrupt contexts of high precision timers
and similar stuff, so i would be happy to learn how to handle such
situations in general. Not just for sake of this experiment i am doing.

T.

Re: Precise timing in ldisc? tty_put_char() in hrtimer context?

[Theodore Ts'o]

On Tue, Jan 07, 2025 at 07:07:42PM +0100, Tomáš Mudruňka wrote:
> i am implementing niche rs485-based serial protocol as a tty line
> discipline linux module. Requirement is to hit transmission window
> with precision in low hundreds of microseconds (eg. +-200 uS).
> Transmit window starts 500 uS after message is received.
> 
> It seems that hardware driver calls tty_receive_buf2() with sufficient
> latency, so my idea was to start hrtimer using hrtimer_start() inside
> of tty_receive_buf2(). And then in hrtimer handler call
> for(...){tty_put_char(tty,...)} or tty->ops->write(), but it seems not
> to be a good practice according to my online research. They say it's
> recommended to call tty_put_char() from workqueue, but documentation
> also says that calling a workqueue might introduce latency of couple
> milliseconds, since it's scheduled thread context.

It's going to depend on the hardware driver, but note that the 8250
driver calls tty_receive_buf2() from a workqueue.  So you have one
scheduler thread context hop already.

One of the issues here is that serial port hardware is... interesting,
and the original design goal of the serial and tty layers included
supporting hardware with no FIFO's, so when receiving at 11,520
characters per second, there would be an interrupt every 8 uS.  The
goal was to make sure that we didn't lose any characters when running
at that rate, as well as being able to support multiple 115k baud
serial ports running at full speed on relatively 40 MHz 386
processors.  So the whole system is designed to optimize for
throughput at low oeverhead, and not latency.

If you only care about making it work on your specific serial port
hardware, you might be able to make things work, but some serial ports
have been known to drop transmit interrupts, so there are timeout
mechanisms which will wait for a little bit more time than needed to
empty the FIFO, and if we don't get an transmit interrupt and there
are characters waiting to be transmitted, we'll check to see if the
UART is ready to receive more characters and send the characters as a
backup mechanism if the transmit interrupt gets lost --- which some
hardware would be prone to do.  There is a lot of dodgy serial port
hardware out there.

So it's not just that Linux isn't a realtime OS.  It's that tty layer
wasn't architected for your use case.  Ultimately, if you want the
best latency, you might need to replace the driver which one that
polls and burns a full CPU instead of depending on interrupts, and
then bypass the tty layer altogether.  That would make your solution
specific to a particular hardware back-end, but full generality isn't
possible in any case, not if you are trying to hit tight latency
guarantees.

For example, there is one serial port hardware which would support up
to 128 serial ports, and doesn't use interrupts at all, but instead
has deep hardware buffers, and the driver runs interly off of timer
callbacks.  That's one case where the *hardware* has been optimized
for throughput and low CPU overhead for a very large number of active
serial ports (think: modem banks), and for which latency is terrible,
but that was explicitly designed by the hardware manufacturer for that
use case and a very different set of design constraints than what you
are trying to achieve.  So if you tried to run your rs485 protocol on
that hardware, it would very likely be hopeless.

Cheers,

					- Ted