[PATCH net v5] net: stmmac: Prevent DSA tags from breaking COE

[PATCH net v5] net: stmmac: Prevent DSA tags from breaking COE

[Romain Gantois]

Some DSA tagging protocols change the EtherType field in the MAC header
e.g.  DSA_TAG_PROTO_(DSA/EDSA/BRCM/MTK/RTL4C_A/SJA1105). On TX these tagged
frames are ignored by the checksum offload engine and IP header checker of
some stmmac cores.

On RX, the stmmac driver wrongly assumes that checksums have been computed
for these tagged packets, and sets CHECKSUM_UNNECESSARY.

Add an additional check in the stmmac TX and RX hotpaths so that COE is
deactivated for packets with ethertypes that will not trigger the COE and
IP header checks.

Fixes: 6b2c6e4a938f ("net: stmmac: propagate feature flags to vlan")
Cc:  <stable@vger.kernel.org>
Reported-by: Richard Tresidder <rtresidd@electromag.com.au>
Link: https://lore.kernel.org/netdev/e5c6c75f-2dfa-4e50-a1fb-6bf4cdb617c2@electromag.com.au/
Reported-by: Romain Gantois <romain.gantois@bootlin.com>
Link: https://lore.kernel.org/netdev/c57283ed-6b9b-b0e6-ee12-5655c1c54495@bootlin.com/
Reviewed-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Romain Gantois <romain.gantois@bootlin.com>
---
Hello everyone,

This is the fifth version of my proposed fix for the stmmac checksum
offloading issue that has recently been reported.

significant changes in v4:
- Removed "inline" from declaration of stmmac_has_ip_ethertype

significant changes in v3:
- Use __vlan_get_protocol to make sure that 8021Q-encapsulated
  traffic is checked correctly.

significant changes in v2:
- Replaced the stmmac_link_up-based fix with an ethertype check in the TX
  and RX hotpaths.

The Checksum Offloading Engine of some stmmac cores (e.g. DWMAC1000)
computes an incorrect checksum when presented with DSA-tagged packets. This
causes all TCP/UDP transfers to break when the stmmac device is connected
to the CPU port of a DSA switch.

I ran some tests using different tagging protocols with DSA_LOOP, and all
of the protocols that set a custom ethertype field in the MAC header caused
the checksum offload engine to ignore the tagged packets. On TX, this
caused packets to egress with incorrect checksums. On RX, these packets
were similarly ignored by the COE, yet the stmmac driver set
CHECKSUM_UNNECESSARY, wrongly assuming that their checksums had been
verified in hardware.

Version 2 of this patch series fixes this issue by checking ethertype
fields in both the TX and RX hotpaths of the stmmac driver. On TX, if a
non-IP ethertype is detected, the packet is checksummed in software.  On
RX, the same condition causes stmmac to avoid setting CHECKSUM_UNNECESSARY.

To measure the performance degradation to the TX/RX hotpaths, I did some
iperf3 runs with 512-byte unfragmented UDP packets.

measured degradation on TX: -466 pps (-0.2%) on RX: -338 pps (-1.2%)
original performances on TX: 22kpps on RX: 27kpps

The performance hit on the RX path can be partly explained by the fact that
the stmmac driver doesn't set CHECKSUM_UNNECESSARY anymore.

The TX performance degradation observed in v2 seems to have improved.
It's not entirely clear to me why that is.

Best Regards,

Romain

Romain Gantois (1):
  net: stmmac: Prevent DSA tags from breaking COE

 .../net/ethernet/stmicro/stmmac/stmmac_main.c | 23 ++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

--
2.43.0
---
Changes in v5:
- Added missing "net" tag to subject of patch series
- Link to v4: https://lore.kernel.org/r/20240109-prevent_dsa_tags-v4-1-f888771fa2f6@bootlin.com
---
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 29 ++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index 37e64283f910..b30dba06dbd1 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -4371,6 +4371,25 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
 	return NETDEV_TX_OK;
 }
 
+/**
+ * stmmac_has_ip_ethertype() - Check if packet has IP ethertype
+ * @skb: socket buffer to check
+ *
+ * Check if a packet has an ethertype that will trigger the IP header checks
+ * and IP/TCP checksum engine of the stmmac core.
+ *
+ * Return: true if the ethertype can trigger the checksum engine, false otherwise
+ */
+static bool stmmac_has_ip_ethertype(struct sk_buff *skb)
+{
+	int depth = 0;
+	__be16 proto;
+
+	proto = __vlan_get_protocol(skb, eth_header_parse_protocol(skb), &depth);
+
+	return (depth <= ETH_HLEN) && (proto == htons(ETH_P_IP) || proto == htons(ETH_P_IPV6));
+}
+
 /**
  *  stmmac_xmit - Tx entry point of the driver
  *  @skb : the socket buffer
@@ -4435,9 +4454,13 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
 	/* DWMAC IPs can be synthesized to support tx coe only for a few tx
 	 * queues. In that case, checksum offloading for those queues that don't
 	 * support tx coe needs to fallback to software checksum calculation.
+	 *
+	 * Packets that won't trigger the COE e.g. most DSA-tagged packets will
+	 * also have to be checksummed in software.
 	 */
 	if (csum_insertion &&
-	    priv->plat->tx_queues_cfg[queue].coe_unsupported) {
+	    (priv->plat->tx_queues_cfg[queue].coe_unsupported ||
+	    !stmmac_has_ip_ethertype(skb))) {
 		if (unlikely(skb_checksum_help(skb)))
 			goto dma_map_err;
 		csum_insertion = !csum_insertion;
@@ -4997,7 +5020,7 @@ static void stmmac_dispatch_skb_zc(struct stmmac_priv *priv, u32 queue,
 	stmmac_rx_vlan(priv->dev, skb);
 	skb->protocol = eth_type_trans(skb, priv->dev);
 
-	if (unlikely(!coe))
+	if (unlikely(!coe) || !stmmac_has_ip_ethertype(skb))
 		skb_checksum_none_assert(skb);
 	else
 		skb->ip_summed = CHECKSUM_UNNECESSARY;
@@ -5513,7 +5536,7 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
 		stmmac_rx_vlan(priv->dev, skb);
 		skb->protocol = eth_type_trans(skb, priv->dev);
 
-		if (unlikely(!coe))
+		if (unlikely(!coe) || !stmmac_has_ip_ethertype(skb))
 			skb_checksum_none_assert(skb);
 		else
 			skb->ip_summed = CHECKSUM_UNNECESSARY;

---
base-commit: ac631873c9e7a50d2a8de457cfc4b9f86666403e
change-id: 20240108-prevent_dsa_tags-7bb0def0db81

Best regards,
-- 
Romain Gantois <romain.gantois@bootlin.com>


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH net v5] net: stmmac: Prevent DSA tags from breaking COE

[Jakub Kicinski]

On Thu, 11 Jan 2024 15:58:51 +0100 Romain Gantois wrote:
> Some DSA tagging protocols change the EtherType field in the MAC header
> e.g.  DSA_TAG_PROTO_(DSA/EDSA/BRCM/MTK/RTL4C_A/SJA1105). On TX these tagged
> frames are ignored by the checksum offload engine and IP header checker of
> some stmmac cores.
> 
> On RX, the stmmac driver wrongly assumes that checksums have been computed
> for these tagged packets, and sets CHECKSUM_UNNECESSARY.
> 
> Add an additional check in the stmmac TX and RX hotpaths so that COE is
> deactivated for packets with ethertypes that will not trigger the COE and
> IP header checks.
> 
> Fixes: 6b2c6e4a938f ("net: stmmac: propagate feature flags to vlan")
> Cc:  <stable@vger.kernel.org>

nit: double space

> +/**
> + * stmmac_has_ip_ethertype() - Check if packet has IP ethertype
> + * @skb: socket buffer to check
> + *
> + * Check if a packet has an ethertype that will trigger the IP header checks
> + * and IP/TCP checksum engine of the stmmac core.
> + *
> + * Return: true if the ethertype can trigger the checksum engine, false otherwise

nit: please don't go over 80 chars unless there's a good reason.
we are old school and stick to checkpatch --max-line-length=80 in netdev

>  	if (csum_insertion &&
> -	    priv->plat->tx_queues_cfg[queue].coe_unsupported) {
> +	    (priv->plat->tx_queues_cfg[queue].coe_unsupported ||
> +	    !stmmac_has_ip_ethertype(skb))) {

nit: minor misalignment here, the '!' should be under 'p'

>  		if (unlikely(skb_checksum_help(skb)))
>  			goto dma_map_err;
>  		csum_insertion = !csum_insertion;
> @@ -4997,7 +5020,7 @@ static void stmmac_dispatch_skb_zc(struct stmmac_priv *priv, u32 queue,
>  	stmmac_rx_vlan(priv->dev, skb);
>  	skb->protocol = eth_type_trans(skb, priv->dev);
>  
> -	if (unlikely(!coe))
> +	if (unlikely(!coe) || !stmmac_has_ip_ethertype(skb))

The lack of Rx side COE checking in this driver is kinda crazy.
Looking at enh_desc_coe_rdes0() it seems like RDES0_FRAME_TYPE
may be the indication we need here? 

We can dig into it as a follow up but I'm guessing that sending
an IPv6 packet with extension headers will also make the device
skip checksumming, or a UDP packet with csum of 0?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH net v5] net: stmmac: Prevent DSA tags from breaking COE

[Romain Gantois]

Hi Jakub,

On Fri, 12 Jan 2024, Jakub Kicinski wrote:

> > @@ -4997,7 +5020,7 @@ static void stmmac_dispatch_skb_zc(struct stmmac_priv *priv, u32 queue,
> >  	stmmac_rx_vlan(priv->dev, skb);
> >  	skb->protocol = eth_type_trans(skb, priv->dev);
> >  
> > -	if (unlikely(!coe))
> > +	if (unlikely(!coe) || !stmmac_has_ip_ethertype(skb))
> 
> The lack of Rx side COE checking in this driver is kinda crazy.
> Looking at enh_desc_coe_rdes0() it seems like RDES0_FRAME_TYPE
> may be the indication we need here? 

I don't think that RDES0_FRAME_TYPE would be enough, at least not on its own. 
That bit is set by checking the length/ethertype field to see if is an 
Ethernet II frame or an IEEE802.3 frame. But even Ethernet II frames with non-IP 
ethertypes will not be checksummed. Also protocols with a non-fixed ethertype 
field such as DSA_TAG_PROTO could trigger the bit, or not, depending on what 
they put in the DSA tag.

--
Romain Gantois, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH net v5] net: stmmac: Prevent DSA tags from breaking COE

[Jakub Kicinski]

On Tue, 16 Jan 2024 13:14:15 +0100 (CET) Romain Gantois wrote:
> > > @@ -4997,7 +5020,7 @@ static void stmmac_dispatch_skb_zc(struct stmmac_priv *priv, u32 queue,
> > >  	stmmac_rx_vlan(priv->dev, skb);
> > >  	skb->protocol = eth_type_trans(skb, priv->dev);
> > >  
> > > -	if (unlikely(!coe))
> > > +	if (unlikely(!coe) || !stmmac_has_ip_ethertype(skb))  
> > 
> > The lack of Rx side COE checking in this driver is kinda crazy.
> > Looking at enh_desc_coe_rdes0() it seems like RDES0_FRAME_TYPE
> > may be the indication we need here?   
> 
> I don't think that RDES0_FRAME_TYPE would be enough, at least not on its own. 
> That bit is set by checking the length/ethertype field to see if is an 
> Ethernet II frame or an IEEE802.3 frame. But even Ethernet II frames with non-IP 
> ethertypes will not be checksummed. Also protocols with a non-fixed ethertype 
> field such as DSA_TAG_PROTO could trigger the bit, or not, depending on what 
> they put in the DSA tag.

Hm, the comment in enh_desc_coe_rdes0() says:

	/* bits 5 7 0 | Frame status
	 * ----------------------------------------------------------
	 *      0 0 0 | IEEE 802.3 Type frame (length < 1536 octects)
	 *      1 0 0 | IPv4/6 No CSUM errorS.
	 *      1 0 1 | IPv4/6 CSUM PAYLOAD error
	 *      1 1 0 | IPv4/6 CSUM IP HR error
	 *      1 1 1 | IPv4/6 IP PAYLOAD AND HEADER errorS
	 *      0 0 1 | IPv4/6 unsupported IP PAYLOAD
	 *      0 1 1 | COE bypassed.. no IPv4/6 frame
	 *      0 1 0 | Reserved.
	 */

which makes it sound like bit 5 will not be set for a Ethernet II frame
with unsupported IP payload, or not an IP frame. Does the bit mean other
things in different descriptor formats?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH net v5] net: stmmac: Prevent DSA tags from breaking COE

[Romain Gantois]

On Tue, 16 Jan 2024, Jakub Kicinski wrote:

> Hm, the comment in enh_desc_coe_rdes0() says:
> 
> 	/* bits 5 7 0 | Frame status
> 	 * ----------------------------------------------------------
> 	 *      0 0 0 | IEEE 802.3 Type frame (length < 1536 octects)
> 	 *      1 0 0 | IPv4/6 No CSUM errorS.
> 	 *      1 0 1 | IPv4/6 CSUM PAYLOAD error
> 	 *      1 1 0 | IPv4/6 CSUM IP HR error
> 	 *      1 1 1 | IPv4/6 IP PAYLOAD AND HEADER errorS
> 	 *      0 0 1 | IPv4/6 unsupported IP PAYLOAD
> 	 *      0 1 1 | COE bypassed.. no IPv4/6 frame
> 	 *      0 1 0 | Reserved.
> 	 */
> 
> which makes it sound like bit 5 will not be set for a Ethernet II frame
> with unsupported IP payload, or not an IP frame. Does the bit mean other
> things in different descriptor formats?

The description of this bit in my datasheet is:

```
b5 FT Frame Type
When set, this bit indicates that the Receive Frame is an Ethernet-type frame 
(the Length/Type field is greater than or equal to 1,536). When this bit is 
reset, it indicates that the received frame is an IEEE 802.3 frame. This bit is 
not valid for Runt frames less than 14 bytes
```

There is no mention of a more subtle check to detect non-IP Ethernet II frames. 
I ran some tests on my hardware and EDSA-tagged packets consistently come in 
with status 0b100, so the MAC sets the frame type bit even for frames that don't 
have an IP ethertype.

Best Regards,

-- 
Romain Gantois, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH net v5] net: stmmac: Prevent DSA tags from breaking COE

[Jakub Kicinski]

On Tue, 16 Jan 2024 17:18:30 +0100 (CET) Romain Gantois wrote:
> > which makes it sound like bit 5 will not be set for a Ethernet II frame
> > with unsupported IP payload, or not an IP frame. Does the bit mean other
> > things in different descriptor formats?  
> 
> The description of this bit in my datasheet is:
> 
> ```
> b5 FT Frame Type
> When set, this bit indicates that the Receive Frame is an Ethernet-type frame 
> (the Length/Type field is greater than or equal to 1,536). When this bit is 
> reset, it indicates that the received frame is an IEEE 802.3 frame. This bit is 
> not valid for Runt frames less than 14 bytes
> ```
> 
> There is no mention of a more subtle check to detect non-IP Ethernet II frames. 
> I ran some tests on my hardware and EDSA-tagged packets consistently come in 
> with status 0b100, so the MAC sets the frame type bit even for frames that don't 
> have an IP ethertype.

Boo, who designed this thing :(

v6 is good to go in then, thank you for investigating and testing!

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel