Re: [PATCHv4] serial: imx: Add DMA buffer configuration via sysfs

Re: [PATCHv4] serial: imx: Add DMA buffer configuration via sysfs

[Uwe Kleine-König]

[-- Attachment #1: Type: text/plain, Size: 5072 bytes --]

Hello Greg,

[Cc += dt maintainers]

On Tue, Apr 06, 2021 at 09:13:04AM +0200, Greg Kroah-Hartman wrote:
> On Mon, Apr 05, 2021 at 11:44:46PM +0200, Sebastian Reichel wrote:
> > On Fri, Mar 05, 2021 at 01:42:52PM +0100, Sebastian Reichel wrote:
> > > On Fri, Mar 05, 2021 at 01:06:12PM +0100, Greg Kroah-Hartman wrote:
> > > > On Fri, Mar 05, 2021 at 12:50:58PM +0100, Sebastian Reichel wrote:
> > > > > From: Fabien Lahoudere <fabien.lahoudere@collabora.com>
> > > > > 
> > > > > In order to optimize serial communication (performance/throughput VS
> > > > > latency), we may need to tweak DMA period number and size. This adds
> > > > > sysfs attributes to configure those values before initialising DMA.
> > > > > The defaults will stay the same as before (16 buffers with a size of
> > > > > 1024 bytes). Afterwards the values can be read/write with the
> > > > > following sysfs files:
> > > > > 
> > > > > /sys/class/tty/ttymxc*/dma_buffer_size
> > > > > /sys/class/tty/ttymxc*/dma_buffer_count
> > > > 
> > > > Ick no.  Custom sysfs attributes for things like serial ports are crazy.
> > > > 
> > > > > This is mainly needed for GEHC CS ONE (arch/arm/boot/dts/imx53-ppd.dts),
> > > > > which has multiple microcontrollers connected via UART controlling. One
> > > > > of the UARTs is connected to an on-board microcontroller at 19200 baud,
> > > > > which constantly pushes critical data (so aging character detect
> > > > > interrupt will never trigger). This data must be processed at 50-200 Hz,
> > > > > so UART should return data in less than 5-20ms. With 1024 byte DMA
> > > > > buffer (and a constant data stream) the read operation instead needs
> > > > > 1024 byte / 19200 baud = 53.333ms, which is way too long (note: Worst
> > > > > Case would be remote processor sending data with short pauses <= 7
> > > > > characters, which would further increase this number). The current
> > > > > downstream kernel instead configures 24 bytes resulting in 1.25ms,
> > > > > but that is obviously not sensible for normal UART use cases and cannot
> > > > > be used as new default.
> > > > 
> > > > Why can't this be a device tree attribute? Why does this have to be a
> > > > sysfs thing that no one will know how to tune and set over time.  This
> > > > hardware should not force a user to manually tune it to get it to work
> > > > properly, this isn't the 1990's anymore :(
> > > > 
> > > > Please never force a user to choose stuff like this, they never will
> > > > know what to do.
> > > 
> > > This used to be a DT attribute in PATCHv1. It has been moved over to
> > > sysfs since PATCHv2, since it does not describe the hardware, but
> > > configuration. Unfortunately lore.kernel.org does not have the full
> > > thread, but this is the discussion:
> > > 
> > > https://lore.kernel.org/linux-serial/20170629182618.jpahpmuq364ldcv2@pengutronix.de/
> > > 
> > > From downstream POV this can be done either by adding a DT property
> > > to the UART node, or by adding a udev rule.
> > > 
> > > From my POV there is not a huge difference. In both cases we will
> > > be bound by an ABI afterwards, in both cases people will usually
> > > stick to the default value and in both cases people that do deviate
> > > from the default probably ran into problems and started to look
> > > for a solution.
> > 
> > ping? It's not very nice to get a rejected in cycles :(
> 
> I recommend working with the DT people here, as custom sysfs attributes
> for things like this that are really just describing the hardware is
> crazy.

I was one who expressed concerns in the earlier rounds that dt isn't the
right place for this. dt is about hardware description, but choosing
a good value for the dma buffer size is driver tuning and depends on the
individual requirements. (latency, throughput, memory consumption,
robustness under system load). I can even imagine use cases where the
settings should be changed dynamically, which cannot (easily) be done
using dt.

While I see your point that a driver specific sysfs property is
unusual/strange/whatever every downside you mentioned also applies to a
dt property (or a custom ioctl).

Among the solutions I can imagine, my preference order is:

 - automatic tuning
 - sysfs property
 - further discussion
 - dt property
 - custom ioctl

I wonder if there is a sensible way to implement a automatic tuning. In
the use case mentioned in the commit log, Sebastian's need is low
latency for a constantly sending microcontroller on the other side. Is
it sensible to make the used dma buffers smaller if we have a certain
throughput? Or is that too magic and doomed to fail covering most use
cases? If that doesn't work, I support Sebastian's approach to do that
in a sysfs property.

Sebastian, have you evaluated just not using dma for these UARTs?

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | https://www.pengutronix.de/ |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: EXT: Re: [PATCHv4] serial: imx: Add DMA buffer configuration via sysfs

[Ian Ray]

On Fri, Dec 08, 2023 at 10:02:05AM +0100, Uwe Kleine-K??nig wrote:
> be safe.
> 
> Hello Greg,
> 
> [Cc += dt maintainers]
> 
> On Tue, Apr 06, 2021 at 09:13:04AM +0200, Greg Kroah-Hartman wrote:
> > On Mon, Apr 05, 2021 at 11:44:46PM +0200, Sebastian Reichel wrote:
> > > On Fri, Mar 05, 2021 at 01:42:52PM +0100, Sebastian Reichel wrote:
> > > > On Fri, Mar 05, 2021 at 01:06:12PM +0100, Greg Kroah-Hartman wrote:
> > > > > On Fri, Mar 05, 2021 at 12:50:58PM +0100, Sebastian Reichel wrote:
> > > > > > From: Fabien Lahoudere <fabien.lahoudere@collabora.com>
> > > > > > 
> > > > > > In order to optimize serial communication (performance/throughput VS
> > > > > > latency), we may need to tweak DMA period number and size. This adds
> > > > > > sysfs attributes to configure those values before initialising DMA.
> > > > > > The defaults will stay the same as before (16 buffers with a size of
> > > > > > 1024 bytes). Afterwards the values can be read/write with the
> > > > > > following sysfs files:
> > > > > > 
> > > > > > /sys/class/tty/ttymxc*/dma_buffer_size
> > > > > > /sys/class/tty/ttymxc*/dma_buffer_count
> > > > > 
> > > > > Ick no.  Custom sysfs attributes for things like serial ports are crazy.
> > > > > 
> > > > > > This is mainly needed for GEHC CS ONE (arch/arm/boot/dts/imx53-ppd.dts),
> > > > > > which has multiple microcontrollers connected via UART controlling. One
> > > > > > of the UARTs is connected to an on-board microcontroller at 19200 baud,
> > > > > > which constantly pushes critical data (so aging character detect
> > > > > > interrupt will never trigger). This data must be processed at 50-200 Hz,
> > > > > > so UART should return data in less than 5-20ms. With 1024 byte DMA
> > > > > > buffer (and a constant data stream) the read operation instead needs
> > > > > > 1024 byte / 19200 baud = 53.333ms, which is way too long (note: Worst
> > > > > > Case would be remote processor sending data with short pauses <= 7
> > > > > > characters, which would further increase this number). The current
> > > > > > downstream kernel instead configures 24 bytes resulting in 1.25ms,
> > > > > > but that is obviously not sensible for normal UART use cases and cannot
> > > > > > be used as new default.
> > > > > 
> > > > > Why can't this be a device tree attribute? Why does this have to be a
> > > > > sysfs thing that no one will know how to tune and set over time.  This
> > > > > hardware should not force a user to manually tune it to get it to work
> > > > > properly, this isn't the 1990's anymore :(
> > > > > 
> > > > > Please never force a user to choose stuff like this, they never will
> > > > > know what to do.
> > > > 
> > > > This used to be a DT attribute in PATCHv1. It has been moved over to
> > > > sysfs since PATCHv2, since it does not describe the hardware, but
> > > > configuration. Unfortunately lore.kernel.org does not have the full
> > > > thread, but this is the discussion:
> > > > 
> > > > https://lore.kernel.org/linux-serial/20170629182618.jpahpmuq364ldcv2@pengutronix.de/
> > > > 
> > > > From downstream POV this can be done either by adding a DT property
> > > > to the UART node, or by adding a udev rule.
> > > > 
> > > > From my POV there is not a huge difference. In both cases we will
> > > > be bound by an ABI afterwards, in both cases people will usually
> > > > stick to the default value and in both cases people that do deviate
> > > > from the default probably ran into problems and started to look
> > > > for a solution.
> > > 
> > > ping? It's not very nice to get a rejected in cycles :(
> > 
> > I recommend working with the DT people here, as custom sysfs attributes
> > for things like this that are really just describing the hardware is
> > crazy.

Continuing here (see also [1]).

[1] https://lore.kernel.org/lkml/ZXr55QV4tnCz8GtI@4600ffe2ac4f/

> 
> I was one who expressed concerns in the earlier rounds that dt isn't the
> right place for this. dt is about hardware description, but choosing
> a good value for the dma buffer size is driver tuning and depends on the
> individual requirements. (latency, throughput, memory consumption,
> robustness under system load). I can even imagine use cases where the
> settings should be changed dynamically, which cannot (easily) be done
> using dt.
> 
> While I see your point that a driver specific sysfs property is
> unusual/strange/whatever every downside you mentioned also applies to a
> dt property (or a custom ioctl).
> 
> Among the solutions I can imagine, my preference order is:
> 
>  - automatic tuning

This might be too magical.  The algorithm would likely lose data before
it "warms up" and that would be unacceptable to our application.

>  - sysfs property
>  - further discussion
>  - dt property
>  - custom ioctl

- sysctl

Given the description [2] "configure kernel parameters at runtime" this
approach would appear to meet our needs.  I have not looked into this in
any detail yet -- but would like to get upstream feedback.

[2] https://man7.org/linux/man-pages/man8/sysctl.8.html

> 
> I wonder if there is a sensible way to implement a automatic tuning. In
> the use case mentioned in the commit log, Sebastian's need is low
> latency for a constantly sending microcontroller on the other side. Is
> it sensible to make the used dma buffers smaller if we have a certain
> throughput? Or is that too magic and doomed to fail covering most use
> cases? If that doesn't work, I support Sebastian's approach to do that
> in a sysfs property.
> 
> Sebastian, have you evaluated just not using dma for these UARTs?

We spent a significant amount of time analysing and profiling our
application.  DMA is absolutely required in order to avoid dropping
data.

This is especially true for the 4M baud UART.  This section quoted from
the original patch V4 [3].

[3] https://lore.kernel.org/lkml/20210305115058.92284-1-sebastian.reichel@collabora.com/

> The same device also has another microcontroller with a 4M baud UART
> connection exchanging lots of data. For this the same mechanism can
> be used to increase the buffer size (downstream uses 4K instead of
> the default 1K) with potentially slightly reduced buffer count. At
> this baud rate latency is not an issue (4096 byte / 4M baud = 0.977 ms).
> Before increasing the default buffer count from 4 to 16 in 76c38d30fee7,
> this was required to avoid data loss. With the changed default it's
> a performance optimization.

Note: Quite a lot has change since we submitted this patch -- GE
HealthCare has spun-off from GE, and Sebastian is no longer working on
the project.

Many thanks,
Ian

> 
> Best regards
> Uwe
> 
> -- 
> Pengutronix e.K.                           | Uwe Kleine-K?nig            |
> Industrial Linux Solutions                 | https://www.pengutronix.de/ |