net: dsa: mv88e6xxx architecture

net: dsa: mv88e6xxx architecture

[Luke Howard]

I am working on TC support for the mv88e6xxx DSA driver. [1]

Two architectural questions.

mv88e6xxx switches support AVB admission control, where frames with AVB frame priorities are discarded unless their ATU entry has a flag indicating they belong to an AVB stream. This requires knowing the intent behind fdb_add() or mdb_add(). One, intrusive, option would be a NTF_EXT flag that could be set by the SRP [2] daemon. My current approach is a NET_DSA_MV88E6XXX_AVB Kconfig option which assumes all static FDB/MDB updates on MQPRIO-enabled ports should have the AVB flag set. MQPRIO and CBS are supported without this option, but will fail AVB/TSN switch certification (which floods the switch with “AVB” frames not setup by SRP).

The second question is whether Linux TCs should be mapped to AVB classes or to queues. The admission control described above requires dedicated, global queues for Class A and B AVB traffic. This effectively imposes a mapping between TCs and AVB classes (with a third TC for non-AVB traffic). On mv88e6xxx chips with more than 4 TX queues, a mapping between TCs and TX queues would provide more flexibility, particularly as it would also allow per-port MQPRIO policies (a feature not available on the 6352 family).

I think I have made the right set of tradeoffs but clearly there is compromise between supporting CBS generally, and supporting AVB/TSN properly. Any thoughts would be appreciated. More detail can be found in the avb.h comment in [1].

[1] https://github.com/PADL/linux/compare/158f238aa69d91ad74e535c73f552bd4b025109c...PADL:linux:marvell-fqtss?expand=1
[2] 802.1Q Clause 35, also our open source implementation at https://github.com/PADL/SwiftMRP

Re: net: dsa: mv88e6xxx architecture

[Vladimir Oltean]

Hi Luke,

On Thu, Dec 19, 2024 at 08:02:38AM +1100, Luke Howard wrote:
> I am working on TC support for the mv88e6xxx DSA driver. [1]
> 
> Two architectural questions.
> 
> mv88e6xxx switches support AVB admission control, where frames with
> AVB frame priorities are discarded unless their ATU entry has a flag
> indicating they belong to an AVB stream. This requires knowing the
> intent behind fdb_add() or mdb_add(). One, intrusive, option would be
> a NTF_EXT flag that could be set by the SRP [2] daemon. My current
> approach is a NET_DSA_MV88E6XXX_AVB Kconfig option which assumes all
> static FDB/MDB updates on MQPRIO-enabled ports should have the AVB
> flag set. MQPRIO and CBS are supported without this option, but will
> fail AVB/TSN switch certification (which floods the switch with “AVB”
> frames not setup by SRP).

I think we need an AVB primer. What identifies an AVB stream? Should the
kernel have a database of them? What actions need to be taken for AVB
streams different than for best effort traffic? Is it about scheduling
priority, or about resource reservations, or? Does the custom behavior
pertain only to AVB streams or is it a more widely useful mechanism?

The tradition in Linux networking is to first create abstractions for
the software data path, and then to offload those abstractions to
capable accelerators. This allows certain platforms to do the task
more efficiently, but still using the same user visible API as what a
pure software implementation would require.

> The second question is whether Linux TCs should be mapped to AVB
> classes or to queues. The admission control described above requires
> dedicated, global queues for Class A and B AVB traffic.

What is the lifetime of an AVB packet in the Marvell DSA switches (in
enhanced AVB mode)? Where do they go from these global queues? Still to
the individual port TX queues, I assume? I find it a bit difficult to
understand the concept of global queues.

> This effectively imposes a mapping between TCs and AVB classes (with a
> third TC for non-AVB traffic).

I don't understand this. A mapping between which port's traffic classes
and the global AVB classes? You're saying that mqprio TCs 2 and 1 of all
ports are backed by the same queues?

> On mv88e6xxx chips with more than 4 TX queues, a mapping between TCs
> and TX queues would provide more flexibility, particularly as it would
> also allow per-port MQPRIO policies (a feature not available on the
> 6352 family).
> 
> I think I have made the right set of tradeoffs but clearly there is
> compromise between supporting CBS generally, and supporting AVB/TSN
> properly. Any thoughts would be appreciated. More detail can be found
> in the avb.h comment in [1].
> 
> [1] https://github.com/PADL/linux/compare/158f238aa69d91ad74e535c73f552bd4b025109c...PADL:linux:marvell-fqtss?expand=1
> [2] 802.1Q Clause 35, also our open source implementation at https://github.com/PADL/SwiftMRP

I believe I don't understand enough of the Marvell AVB model to make any
comments yet.

Re: net: dsa: mv88e6xxx architecture

[Luke Howard]

Hi Vladimir,

> I think we need an AVB primer. What identifies an AVB stream? Should the
> kernel have a database of them? What actions need to be taken for AVB
> streams different than for best effort traffic? Is it about scheduling
> priority, or about resource reservations, or? Does the custom behavior
> pertain only to AVB streams or is it a more widely useful mechanism?

AVB is an umbrella term which encompasses 802.1AS (gPTP), 802.1Qav (CBS), 802.1Qat (SRP), and various application layer protocols. This patch series is concerned with CBS and, tangentially, SRP: the existing mv88e6xxx PTP support sufficies for 802.1AS, and the application layer protocols are of no concern to the kernel. Further, the existing TC abstractions for CBS also suffice, and the patch series simply implements those.

Your question: what is an AVB stream, and what (if any) state needs to be maintained in the kernel?

Stream reservations, between “talkers” (sources) and “listeners” (sinks) are coordinated by the Multiple Stream Reservation Protocol (MSRP). (Confusingly, _SRP_ itself is an umbrella term encompassing MVRP, MMRP and MSRP. The existing MVRP support in the kernel only implements part of the MRP state machine.)

MSRP coordinates talker advertisements (consisting of a 64-bit stream ID, a [typically multicast] DA, VID, bandwidth requirement, and PCP) with listener advertisements. It also coordinates “domain” advertisements which establish a mapping between AVB or SRP “classes”, and a (PCP, VID) tuple.

At a high level, the kernel does not need to know anything about SRP: it can, and IMO should (like the PTP state machine) be implemented completely in user space. Once SRP has established there is sufficient bandwidth and latency for a steam to flow, it adds the stream DA to the FDB or MDB, and configures the CBS policy appropriately on the egress port.

The catch is what to do with frames that share a priority with an AVB class but are not negotiated by SRP. These frames could crowd out frames from AVB streams. Marvell’s solution is a flag in the ATU which indicates that the DA was added by SRP. This is the one case where a new kernel interface (specifically, a flag passed down to {fdb,mdb}_add()), could be useful. In lieu of that, the patch currently uses a new Kconfig option to indicate that _all_ static FDB/MDB entries should have that flag set, noting that CBS is still usable without this option.

> What is the lifetime of an AVB packet in the Marvell DSA switches (in
> enhanced AVB mode)? Where do they go from these global queues? Still to
> the individual port TX queues, I assume? I find it a bit difficult to
> understand the concept of global queues.

A poor explanation on my part. Briefly: FPri is the switch’s internal frame priority derived from PCP or DSCP, and QPri is queue number.

* PCP/DSCP to FPri mappings are per port
* TX _queues_ are per-port
* The _mapping_ between FPri and QPri is per-switch (at least, pre-6390 family).
* Where the admission control described above is enabled, each SRP class must be assigned a designated FPri and QPri, and this configuration is per-switch

> I don't understand this. A mapping between which port's traffic classes
> and the global AVB classes? You're saying that mqprio TCs 2 and 1 of all
> ports are backed by the same queues?

The same queue _numbers_ but not the same queues. Configuring per-port FPri to QPri mappings is possible on the 6390 family, but not with SRP admission control. 

The current patch reference counts the MQPRIO policy so that it can be applied or removed from additional ports, but only replaced when one referent remains.

Cheers,
Luke

Re: net: dsa: mv88e6xxx architecture

[Andrew Lunn]

On Sat, Dec 21, 2024 at 08:43:50AM +1100, Luke Howard Bentata wrote:
> Hi Vladimir,
> 
> 
>     I think we need an AVB primer. What identifies an AVB stream? Should the
>     kernel have a database of them? What actions need to be taken for AVB
>     streams different than for best effort traffic? Is it about scheduling
>     priority, or about resource reservations, or? Does the custom behavior
>     pertain only to AVB streams or is it a more widely useful mechanism?
> 
> 
> AVB is an umbrella term which encompasses 802.1AS (gPTP), 802.1Qav (CBS),
> 802.1Qat (SRP), and various application layer protocols. This patch series is
> concerned with CBS and, tangentially, SRP: the existing mv88e6xxx PTP support
> sufficies for 802.1AS, and the application layer protocols are of no concern to
> the kernel. Further, the existing TC abstractions for CBS also suffice, and the
> patch series simply implements those.
> 
> Your question: what is an AVB stream, and what (if any) state needs to be
> maintained in the kernel?
> 
> Stream reservations, between “talkers” (sources) and “listeners” (sinks) are
> coordinated by the Multiple Stream Reservation Protocol (MSRP). (Confusingly,
> _SRP_ itself is an umbrella term encompassing MVRP, MMRP and MSRP. The existing
> MVRP support in the kernel only implements part of the MRP state machine.)
> 
> MSRP coordinates talker advertisements (consisting of a 64-bit stream ID, a
> [typically multicast] DA, VID, bandwidth requirement, and PCP) with listener
> advertisements. It also coordinates “domain” advertisements which establish a
> mapping between AVB or SRP “classes”, and a (PCP, VID) tuple.
> 
> At a high level, the kernel does not need to know anything about SRP: it can,
> and IMO should (like the PTP state machine) be implemented completely in user
> space. Once SRP has established there is sufficient bandwidth and latency for a
> steam to flow, it adds the stream DA to the FDB or MDB, and configures the CBS
> policy appropriately on the egress port.
> 
> The catch is what to do with frames that share a priority with an AVB class but
> are not negotiated by SRP. These frames could crowd out frames from AVB
> streams. Marvell’s solution is a flag in the ATU which indicates that the DA
> was added by SRP. This is the one case where a new kernel interface
> (specifically, a flag passed down to {fdb,mdb}_add()), could be useful.

For a moment, forget about Marvell. Think about a purely software
solution, maybe using the Linux bridge, and a collection of e1000e
cards. Does the same problem exist? How would you solve it?

First solve the generic software case first, and then offload it to
the hardware.

	Andrew

Re: net: dsa: mv88e6xxx architecture

[Vladimir Oltean]

On Sat, Dec 21, 2024 at 08:43:50AM +1100, Luke Howard Bentata wrote:
> Hi Vladimir,
> 
> > I think we need an AVB primer. What identifies an AVB stream? Should the
> > kernel have a database of them? What actions need to be taken for AVB
> > streams different than for best effort traffic? Is it about scheduling
> > priority, or about resource reservations, or? Does the custom behavior
> > pertain only to AVB streams or is it a more widely useful mechanism?
> 
> The catch is what to do with frames that share a priority with an AVB
> class but are not negotiated by SRP. These frames could crowd out
> frames from AVB streams. Marvell’s solution is a flag in the ATU which
> indicates that the DA was added by SRP.

Can you please state in a vendor-agnostic way what does the switch do
with that information, how does it treat those streams specially?

Re: net: dsa: mv88e6xxx architecture

[Luke Howard]

> For a moment, forget about Marvell. Think about a purely software
> solution, maybe using the Linux bridge, and a collection of e1000e
> cards. Does the same problem exist? How would you solve it?

One could:

* Add (e.g.) TCA_MQPRIO_TC_ENTRY_SRP to indicate the TC is associated with a SRP class
* Add (e.g.) NTF_EXT_SRP_MANAGED to indicate the FDB/MDB entry was inserted by the SRP daemon

Packets with TCs marked TCA_MQPRIO_TC_ENTRY_SRP to DAs not marked NTF_EXT_SRP_MANAGED would be dropped (or deprioritised).

For mv88e6xxx, TCA_MQPRIO_TC_ENTRY_SRP would be supported for “AVB” traffic classes, and NTF_EXT_SRP_MANAGED would map to MV88E6XXX_G1_ATU_DATA_STATE_{UC,MC}_STATIC_AVB_NRL.

Or, we do nothing. As far as I can tell the biggest issue with not supporting this is whether the bridge would pass the Avnu test suite. That’s not so important to me, but it might be to some other users.

Cheers,
Luke

Re: net: dsa: mv88e6xxx architecture

[Luke Howard]

>> The catch is what to do with frames that share a priority with an AVB
>> class but are not negotiated by SRP. These frames could crowd out
>> frames from AVB streams. Marvell’s solution is a flag in the ATU which
>> indicates that the DA was added by SRP.
> 
> Can you please state in a vendor-agnostic way what does the switch do
> with that information, how does it treat those streams specially?

Frames that are part of a stream (as identified by DA, VID and PCP) egress through designated priority queues.

Frames that are not part of a stream, but otherwise have a matching priority, can be discarded or have their priority remarked.

Re: net: dsa: mv88e6xxx architecture

[Andrew Lunn]

On Sat, Dec 21, 2024 at 09:52:27AM +1100, Luke Howard wrote:
>     For a moment, forget about Marvell. Think about a purely software
>     solution, maybe using the Linux bridge, and a collection of e1000e
>     cards. Does the same problem exist? How would you solve it?
> 
> 
> One could:
> 
> * Add (e.g.) TCA_MQPRIO_TC_ENTRY_SRP to indicate the TC is associated with a
> SRP class
> * Add (e.g.) NTF_EXT_SRP_MANAGED to indicate the FDB/MDB entry was inserted by
> the SRP daemon

Doesn't FDB/MDB imply you have a bridge? What about an isolated port
which is not a member of a bridge, there is only local traffic?

> Packets with TCs marked TCA_MQPRIO_TC_ENTRY_SRP to DAs not marked
> NTF_EXT_SRP_MANAGED would be dropped (or deprioritised).
> 
> For mv88e6xxx, TCA_MQPRIO_TC_ENTRY_SRP would be supported for “AVB” traffic
> classes, and NTF_EXT_SRP_MANAGED would map to MV88E6XXX_G1_ATU_DATA_STATE_
> {UC,MC}_STATIC_AVB_NRL.
> 
> Or, we do nothing. As far as I can tell the biggest issue with not supporting
> this is whether the bridge would pass the Avnu test suite. That’s not so
> important to me, but it might be to some other users.

It is back to, we use the hardware to accelerate what Linux can
already do in software. If you only use the switch ports in isolated
mode, no bridge, you could probably get away with not supporting AVB
on the linux bridge. But if you need the software bridge to setup the
acceleration via a hardware bridge, you will need the software bridge
to work with AVB without acceleration.

	Andrew

Re: net: dsa: mv88e6xxx architecture

[Luke Howard]

> Doesn't FDB/MDB imply you have a bridge? What about an isolated port
> which is not a member of a bridge, there is only local traffic?

I don’t believe local traffic is an issue (if it were, I imagine one could use cgroups to control which processes could send frames of a particular priority). (Kieran, feel free to correct me.)

What I’ve done in the current patch (behind a Kconfig option) is to assume that static FDB/MDB entries on MQPRIO ports are for AVB alone. But this may be incompatible with IGMP snooping, if that also results in the creation of static MDB entires.

Adding MQPRIO/CBS support without this admission control is definitely an option and perhaps a good first step, the proviso being that when bridging, SRP would not provide a guaranteed bandwidth reservation.