From: "Théo Lebrun" <theo.lebrun@bootlin.com>
To: "Vineeth Karumanchi" <vineeth.karumanchi@amd.com>,
"Conor Dooley" <conor.dooley@microchip.com>,
"Andrew Lunn" <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
"Eric Dumazet" <edumazet@kernel.org>,
"Jakub Kicinski" <kuba@kernel.org>,
"Paolo Abeni" <pabeni@redhat.com>,
"Russell King" <linux@armlinux.org.uk>
Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
"Nicolai Buchwitz" <nb@tipi-net.de>,
"Paolo Valerio" <pvalerio@redhat.com>,
"Vladimir Kondratiev" <vladimir.kondratiev@mobileye.com>,
"Gregory CLEMENT" <gregory.clement@bootlin.com>,
"Benoît Monin" <benoit.monin@bootlin.com>,
"Tawfik Bayouk" <tawfik.bayouk@mobileye.com>,
"Thomas Petazzoni" <thomas.petazzoni@bootlin.com>
Subject: Re: [PATCH net-next v7 1/3] net: macb: cache ENST config across link down/up
Date: Thu, 19 Nov 2026 17:10:27 +0100 [thread overview]
Message-ID: <DMZBHDAP7B6U.SVPBZK6JHFEE@bootlin.com> (raw)
In-Reply-To: <DMZAUI64FXCL.13646UPPGKGTC@bootlin.com>
Hello Théo,
On Thu Nov 19, 2026 at 4:40 PM CET, Théo Lebrun wrote:
> Hello Vineeth,
>
> On Thu Nov 19, 2026 at 1:36 PM CET, Vineeth Karumanchi wrote:
>> Hi Théo,
>>
>> On 11/19/2026 2:23 AM, Théo Lebrun wrote:
>>> Currently, ENST is not explicitly restored across interface close/open.
>>> If HW keeps its register state then ENST is kept, otherwise it is lost;
>>> but we do not write the ENST schedule again on open.
>>>
>>> This causes trouble if the config is not compatible with the new open
>>> and link-up. The two issues that can occur are a link speed change (as
>>> all timings are configured in a unit relative to the link speed) or a
>>> change in the # of queues.
>>>
>>
>> Thanks for the patch.
>>
>> We are working on the same problem for CBS, covering the four scenarios
>> listed in:
>> https://lore.kernel.org/netdev/a04aa4d5-335b-47c1-b969-e89b18c9b142@amd.com/
>>
>> Since this patch reworks how ENST state is kept and restored, it would
>> be good to introduce a generic top-level tc structure here, which ENST,
>> CBS and any later TSN offloads can share, instead of an ENST-specific one.
>
> Sorry I forgot replying on that thread. It is still marked as unread in
> my mailbox!
>
> Let me quote it here and reply:
>
>
>
> On Thu Oct 8, 2026 at 5:27 PM CEST, Karumanchi, Vineeth wrote:
>> On 9/17/2026 3:02 PM, Théo Lebrun wrote:
>>> So we write cbs_control/0x04BC only on ndo_setup_tc(TC_SETUP_QDISC_CBS)
>>> callback. As Sashiko pointed how I think that causes an issue with HW
>>> resets & suspend:
>>> - what if configured then link down then link up?
>>> - what if configured before link up?
>>>
>>> Also when not using CBS we hope tx_sched_ctrl/0x0580 is at its reset
>>> value (0b00, fixed priority) for all queues. I would expect an
>>> unconditional writel tx_sched_ctrl/0x0580 in macb_init_hw(). This
>>> solves the down/up issue, the speed change and also ensures we don't
>>> inherit CBS or DWRR from the boot stages (super unlikely though).
>>>
>>> Also, it might be simpler to always use tx_sched_ctrl/0x0580 rather than
>>> sometimes tx_sched_ctrl/0x0580 and sometimes cbs_control/0x04BC, which
>>> are aliases.
>>>
>>> What do you think?
>>
>> Sorry for the delayed response.
>>
>> I tested the current implementation in the following four scenarios:
>>
>> 1. Configuring CBS while the link is down: CBS cannot be programmed
>> because the link speed is unknown.
>>
>> 2. Link down followed by link up: The hardware retains the CBS register
>> values, and CBS is effective again after the link comes up at the same
>> speed.
>
> Here you talk about link-down followed by link-up.
>
> But about a close then open: this behavior might be board specific.
> Runtime PM does suspend and this might shut off the IP on some
> platforms and not others. That would in turn mean the IP loses its
> state.
>
> There is no easy way of knowing if that occurs on any platform that uses
> MACB. It sounds probable though.
>
>> 3. Suspend/resume: The CBS register contents are lost after resume.
>> However, the CBS child qdisc remains installed in the kernel and is
>> still shown by tc as offloaded, leaving the software configuration
>> inconsistent with the hardware state.
>
> Again, this is platform specific. Some boards + suspend combo keep the
> SoC and its peripherals alive enough for them to keep their state.
>
> A board with little runtime PM doing suspend-to-idle probably won't lose
> CBS register state. A board doing proper suspend-to-RAM will surely
> lose its state.
>
>> 4. Link-speed change: The CBS register values are retained and shaping
>> remains enabled. However, those values were calculated for the previous
>> speed, so the resulting shaping bandwidth is incorrect.
>>
>> I see two possible approaches:
>>
>> - Retain the requested CBS parameters in the driver and restore the
>> hardware configuration, following the saved-state approach used by IGB.
>> For MACB, we would also recalculate the hardware idleslope on link-up
>> using the negotiated speed.
>> If the saved request cannot be supported at the new speed, retain the
>> request, leave CBS disabled for that queue, and warn the user that the
>> configuration is invalid for the new link speed.
>>
>> - Explicitly reset the scheduling configuration in macb_init_hw() and
>> require users to remove and recreate the CBS qdisc. This establishes a
>> known hardware state, but leaves the installed qdisc inconsistent until
>> users reconfigure it. It would also need handling for speed changes that
>> do not invoke macb_init_hw().
>>
>> Please let me know your thoughts on these approaches.
>
> Option 2 is not super doable. There is no API for the driver to
> say to the network stack and userspace "I'm dropping your qdisc, please
> re-enable one if you care".
>
> So ideally we would do option 1, but dealing with link speed changes
> sounds complex. I'm super afraid of introducing bugs with code that
> recomputes the timings on the fly. I'm sure I will create rounding bugs
> in the edge cases.
>
> That's why I made the cached state be in number of words and store the
> link speed alongside. This means we cannot introduce such bugs AND is
> less code overall AND is the common case.
>
> It feels like easier and smaller incremental changes.
To expand with some numbers on this.
- For link speed increases, our trouble is the max on/off time interval
that can be described. We might be asked for windows that we cannot
express with our hardware registers.
- For link speed decreases, our trouble is the on/off time interval
quantum. We risk introducing too much overlap when the desire was to
have none (pretty common with ENST).
link speed │ 10M │ 100M │ 1G │ 2.5G │ 5G │ 10G
───────────┼────────┼───────┼──────┼────────┼────────┼───────
max on/off │ 105 ms │ 10 ms │ 1 ms │ 419 µs │ 210 µs │ 105 µs
───────────┼────────┼───────┼──────┼────────┼────────┼───────
quantum │ 800 ns │ 80 ns │ 8 ns │ 3.2 ns │ 1.6 ns │ 0.8 ns
So only ENST configs that have all timings in that .8µs - 105µs can
safely-ish be described. But at 10M with the max on-time of 105µs, you
cannot even transmit a full 1500 byte frame.
That is talking about the extremes; it is true that it could make more
sense for less extreme link speed changes (eg from 100M to 1G).
Overall, this still conforts me in the current proposal: cache a
link-speed specific config. Userspace must have had looked at the link
speed to pick a sensible ENST schedule anyway.
Thanks,
--
Théo Lebrun, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
next prev parent reply other threads:[~2026-11-19 16:10 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 20:53 [PATCH net-next v7 0/3] net: macb: implement ethtool set channels count operation Théo Lebrun
2026-11-18 20:53 ` [PATCH net-next v7 1/3] net: macb: cache ENST config across link down/up Théo Lebrun
2026-11-19 11:49 ` Nicolai Buchwitz
2026-11-19 12:36 ` Vineeth Karumanchi
2026-11-19 15:40 ` Théo Lebrun
2026-11-19 16:10 ` Théo Lebrun [this message]
2026-11-18 20:53 ` [PATCH net-next v7 2/3] net: macb: implement ethtool_ops.get|set_channels() Théo Lebrun
2026-11-19 11:54 ` Nicolai Buchwitz
2026-11-19 16:19 ` Théo Lebrun
2026-11-18 20:53 ` [PATCH net-next v7 3/3] net: macb: distribute evenly Tx SRAM segments Théo Lebrun
2026-11-19 11:45 ` Nicolai Buchwitz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMZBHDAP7B6U.SVPBZK6JHFEE@bootlin.com \
--to=theo.lebrun@bootlin.com \
--cc=andrew+netdev@lunn.ch \
--cc=benoit.monin@bootlin.com \
--cc=conor.dooley@microchip.com \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=gregory.clement@bootlin.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux@armlinux.org.uk \
--cc=nb@tipi-net.de \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pvalerio@redhat.com \
--cc=tawfik.bayouk@mobileye.com \
--cc=thomas.petazzoni@bootlin.com \
--cc=vineeth.karumanchi@amd.com \
--cc=vladimir.kondratiev@mobileye.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox