From: "Théo Lebrun" <theo.lebrun@bootlin.com>
To: "Vineeth Karumanchi" <vineeth.karumanchi@amd.com>,
"Conor Dooley" <conor.dooley@microchip.com>,
"Andrew Lunn" <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
"Eric Dumazet" <edumazet@kernel.org>,
"Jakub Kicinski" <kuba@kernel.org>,
"Paolo Abeni" <pabeni@redhat.com>,
"Russell King" <linux@armlinux.org.uk>
Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
"Nicolai Buchwitz" <nb@tipi-net.de>,
"Paolo Valerio" <pvalerio@redhat.com>,
"Vladimir Kondratiev" <vladimir.kondratiev@mobileye.com>,
"Gregory CLEMENT" <gregory.clement@bootlin.com>,
"Benoît Monin" <benoit.monin@bootlin.com>,
"Tawfik Bayouk" <tawfik.bayouk@mobileye.com>,
"Thomas Petazzoni" <thomas.petazzoni@bootlin.com>
Subject: Re: [PATCH net-next v7 1/3] net: macb: cache ENST config across link down/up
Date: Thu, 19 Nov 2026 17:10:27 +0100 [thread overview]
Message-ID: <DMZBHDAP7B6U.SVPBZK6JHFEE@bootlin.com> (raw)
In-Reply-To: <DMZAUI64FXCL.13646UPPGKGTC@bootlin.com>
Hello Théo,
On Thu Nov 19, 2026 at 4:40 PM CET, Théo Lebrun wrote:
> Hello Vineeth,
>
> On Thu Nov 19, 2026 at 1:36 PM CET, Vineeth Karumanchi wrote:
>> Hi Théo,
>>
>> On 11/19/2026 2:23 AM, Théo Lebrun wrote:
>>> Currently, ENST is not explicitly restored across interface close/open.
>>> If HW keeps its register state then ENST is kept, otherwise it is lost;
>>> but we do not write the ENST schedule again on open.
>>>
>>> This causes trouble if the config is not compatible with the new open
>>> and link-up. The two issues that can occur are a link speed change (as
>>> all timings are configured in a unit relative to the link speed) or a
>>> change in the # of queues.
>>>
>>
>> Thanks for the patch.
>>
>> We are working on the same problem for CBS, covering the four scenarios
>> listed in:
>> https://lore.kernel.org/netdev/a04aa4d5-335b-47c1-b969-e89b18c9b142@amd.com/
>>
>> Since this patch reworks how ENST state is kept and restored, it would
>> be good to introduce a generic top-level tc structure here, which ENST,
>> CBS and any later TSN offloads can share, instead of an ENST-specific one.
>
> Sorry I forgot replying on that thread. It is still marked as unread in
> my mailbox!
>
> Let me quote it here and reply:
>
>
>
> On Thu Oct 8, 2026 at 5:27 PM CEST, Karumanchi, Vineeth wrote:
>> On 9/17/2026 3:02 PM, Théo Lebrun wrote:
>>> So we write cbs_control/0x04BC only on ndo_setup_tc(TC_SETUP_QDISC_CBS)
>>> callback. As Sashiko pointed how I think that causes an issue with HW
>>> resets & suspend:
>>> - what if configured then link down then link up?
>>> - what if configured before link up?
>>>
>>> Also when not using CBS we hope tx_sched_ctrl/0x0580 is at its reset
>>> value (0b00, fixed priority) for all queues. I would expect an
>>> unconditional writel tx_sched_ctrl/0x0580 in macb_init_hw(). This
>>> solves the down/up issue, the speed change and also ensures we don't
>>> inherit CBS or DWRR from the boot stages (super unlikely though).
>>>
>>> Also, it might be simpler to always use tx_sched_ctrl/0x0580 rather than
>>> sometimes tx_sched_ctrl/0x0580 and sometimes cbs_control/0x04BC, which
>>> are aliases.
>>>
>>> What do you think?
>>
>> Sorry for the delayed response.
>>
>> I tested the current implementation in the following four scenarios:
>>
>> 1. Configuring CBS while the link is down: CBS cannot be programmed
>> because the link speed is unknown.
>>
>> 2. Link down followed by link up: The hardware retains the CBS register
>> values, and CBS is effective again after the link comes up at the same
>> speed.
>
> Here you talk about link-down followed by link-up.
>
> But about a close then open: this behavior might be board specific.
> Runtime PM does suspend and this might shut off the IP on some
> platforms and not others. That would in turn mean the IP loses its
> state.
>
> There is no easy way of knowing if that occurs on any platform that uses
> MACB. It sounds probable though.
>
>> 3. Suspend/resume: The CBS register contents are lost after resume.
>> However, the CBS child qdisc remains installed in the kernel and is
>> still shown by tc as offloaded, leaving the software configuration
>> inconsistent with the hardware state.
>
> Again, this is platform specific. Some boards + suspend combo keep the
> SoC and its peripherals alive enough for them to keep their state.
>
> A board with little runtime PM doing suspend-to-idle probably won't lose
> CBS register state. A board doing proper suspend-to-RAM will surely
> lose its state.
>
>> 4. Link-speed change: The CBS register values are retained and shaping
>> remains enabled. However, those values were calculated for the previous
>> speed, so the resulting shaping bandwidth is incorrect.
>>
>> I see two possible approaches:
>>
>> - Retain the requested CBS parameters in the driver and restore the
>> hardware configuration, following the saved-state approach used by IGB.
>> For MACB, we would also recalculate the hardware idleslope on link-up
>> using the negotiated speed.
>> If the saved request cannot be supported at the new speed, retain the
>> request, leave CBS disabled for that queue, and warn the user that the
>> configuration is invalid for the new link speed.
>>
>> - Explicitly reset the scheduling configuration in macb_init_hw() and
>> require users to remove and recreate the CBS qdisc. This establishes a
>> known hardware state, but leaves the installed qdisc inconsistent until
>> users reconfigure it. It would also need handling for speed changes that
>> do not invoke macb_init_hw().
>>
>> Please let me know your thoughts on these approaches.
>
> Option 2 is not super doable. There is no API for the driver to
> say to the network stack and userspace "I'm dropping your qdisc, please
> re-enable one if you care".
>
> So ideally we would do option 1, but dealing with link speed changes
> sounds complex. I'm super afraid of introducing bugs with code that
> recomputes the timings on the fly. I'm sure I will create rounding bugs
> in the edge cases.
>
> That's why I made the cached state be in number of words and store the
> link speed alongside. This means we cannot introduce such bugs AND is
> less code overall AND is the common case.
>
> It feels like easier and smaller incremental changes.
To expand with some numbers on this.
- For link speed increases, our trouble is the max on/off time interval
that can be described. We might be asked for windows that we cannot
express with our hardware registers.
- For link speed decreases, our trouble is the on/off time interval
quantum. We risk introducing too much overlap when the desire was to
have none (pretty common with ENST).
link speed │ 10M │ 100M │ 1G │ 2.5G │ 5G │ 10G
───────────┼────────┼───────┼──────┼────────┼────────┼───────
max on/off │ 105 ms │ 10 ms │ 1 ms │ 419 µs │ 210 µs │ 105 µs
───────────┼────────┼───────┼──────┼────────┼────────┼───────
quantum │ 800 ns │ 80 ns │ 8 ns │ 3.2 ns │ 1.6 ns │ 0.8 ns
So only ENST configs that have all timings in that .8µs - 105µs can
safely-ish be described. But at 10M with the max on-time of 105µs, you
cannot even transmit a full 1500 byte frame.
That is talking about the extremes; it is true that it could make more
sense for less extreme link speed changes (eg from 100M to 1G).
Overall, this still conforts me in the current proposal: cache a
link-speed specific config. Userspace must have had looked at the link
speed to pick a sensible ENST schedule anyway.
Thanks,
--
Théo Lebrun, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
next prev parent reply other threads:[~2026-11-19 16:10 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 20:53 [PATCH net-next v7 0/3] net: macb: implement ethtool set channels count operation Théo Lebrun
2026-11-18 20:53 ` [PATCH net-next v7 1/3] net: macb: cache ENST config across link down/up Théo Lebrun
2026-11-19 11:49 ` Nicolai Buchwitz
2026-11-19 12:36 ` Vineeth Karumanchi
2026-11-19 15:40 ` Théo Lebrun
2026-11-19 16:10 ` Théo Lebrun [this message]
2026-11-20 6:44 ` Vineeth Karumanchi
2026-11-18 20:53 ` [PATCH net-next v7 2/3] net: macb: implement ethtool_ops.get|set_channels() Théo Lebrun
2026-11-19 11:54 ` Nicolai Buchwitz
2026-11-19 16:19 ` Théo Lebrun
2026-11-18 20:53 ` [PATCH net-next v7 3/3] net: macb: distribute evenly Tx SRAM segments Théo Lebrun
2026-11-19 11:45 ` Nicolai Buchwitz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMZBHDAP7B6U.SVPBZK6JHFEE@bootlin.com \
--to=theo.lebrun@bootlin.com \
--cc=andrew+netdev@lunn.ch \
--cc=benoit.monin@bootlin.com \
--cc=conor.dooley@microchip.com \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=gregory.clement@bootlin.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux@armlinux.org.uk \
--cc=nb@tipi-net.de \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pvalerio@redhat.com \
--cc=tawfik.bayouk@mobileye.com \
--cc=thomas.petazzoni@bootlin.com \
--cc=vineeth.karumanchi@amd.com \
--cc=vladimir.kondratiev@mobileye.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.