[PATCH net 0/2] Fix a regression in the Gemini ethernet controller.

[PATCH net 0/2] Fix a regression in the Gemini ethernet controller.

[Linus Walleij]

These fixes were developed on top of the earlier fixes.

Finding the right solution is hard because the Gemini checksumming
engine is completely undocumented in the datasheets.

Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
---
Linus Walleij (2):
      net: ethernet: cortina: Drop software checksumming
      net: ethernet: cortina: Bypass checksumming engine of alien ethertypes

 drivers/net/ethernet/cortina/gemini.c | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)
---
base-commit: 33cc938e65a98f1d29d0a18403dbbee050dcad9a
change-id: 20231203-new-gemini-ethernet-regression-3c672de9cfd9

Best regards,
-- 
Linus Walleij <linus.walleij@linaro.org>

[PATCH net 1/2] net: ethernet: cortina: Drop software checksumming

[Linus Walleij]

The recent change to allow large frames without hardware checksumming
slotted in software checksumming in the driver if hardware could not
do it.

This will however upset TSO (TCP Segment Offloading). Typical
error dumps includes this:

skb len=2961 headroom=222 headlen=66 tailroom=0
(...)
WARNING: CPU: 0 PID: 956 at net/core/dev.c:3259 skb_warn_bad_offload+0x7c/0x108
gemini-ethernet-port: caps=(0x0000010000154813, 0x00002007ffdd7889)

And the packets do not go through.

After investigating I drilled it down to the introduction of the
software checksumming in the driver.

Since the segmenting of packets will be done by the hardware this
makes a bit of sense since in that case the hardware also needs to
be keeping track of the checksumming.

That begs the question why large TCP or UDP packets also have to
bypass the checksumming (like e.g. ICMP does). If the hardware is
splitting it into smaller packets per-MTU setting, and checksumming
them, why is this happening then? I don't know. I know it is needed,
from tests: the OpenWrt webserver uhttpd starts sending big skb:s (up
to 2047 bytes, the max MTU) and above 1514 bytes it starts to fail
and hang unless the bypass bit is set: the frames are not getting
through.

Keeping the size check but removing the software checksum makes things
work again. This was probably dubious to introduce in the first place.

Fixes: d4d0c5b4d279 ("net: ethernet: cortina: Handle large frames")
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
---
 drivers/net/ethernet/cortina/gemini.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/drivers/net/ethernet/cortina/gemini.c b/drivers/net/ethernet/cortina/gemini.c
index 78287cfcbf63..255fcffc1579 100644
--- a/drivers/net/ethernet/cortina/gemini.c
+++ b/drivers/net/ethernet/cortina/gemini.c
@@ -1145,7 +1145,6 @@ static int gmac_map_tx_bufs(struct net_device *netdev, struct sk_buff *skb,
 	dma_addr_t mapping;
 	unsigned short mtu;
 	void *buffer;
-	int ret;
 
 	mtu  = ETH_HLEN;
 	mtu += netdev->mtu;
@@ -1166,14 +1165,7 @@ static int gmac_map_tx_bufs(struct net_device *netdev, struct sk_buff *skb,
 		 * checksum buffer is only 1518 bytes, so when the frames get
 		 * bigger they get truncated, or the last few bytes get
 		 * overwritten by the FCS.
-		 *
-		 * Just use software checksumming and bypass on bigger frames.
 		 */
-		if (skb->ip_summed == CHECKSUM_PARTIAL) {
-			ret = skb_checksum_help(skb);
-			if (ret)
-				return ret;
-		}
 		word1 |= TSS_BYPASS_BIT;
 	} else if (skb->ip_summed == CHECKSUM_PARTIAL) {
 		int tcp = 0;

-- 
2.34.1

Re: [PATCH net 1/2] net: ethernet: cortina: Drop software checksumming

[Eric Dumazet]

On Fri, Dec 15, 2023 at 9:49 AM Linus Walleij <linus.walleij@linaro.org> wrote:
>
> The recent change to allow large frames without hardware checksumming
> slotted in software checksumming in the driver if hardware could not
> do it.
>
> This will however upset TSO (TCP Segment Offloading). Typical
> error dumps includes this:
>
> skb len=2961 headroom=222 headlen=66 tailroom=0
> (...)
> WARNING: CPU: 0 PID: 956 at net/core/dev.c:3259 skb_warn_bad_offload+0x7c/0x108
> gemini-ethernet-port: caps=(0x0000010000154813, 0x00002007ffdd7889)
>
> And the packets do not go through.
>
> After investigating I drilled it down to the introduction of the
> software checksumming in the driver.
>
> Since the segmenting of packets will be done by the hardware this
> makes a bit of sense since in that case the hardware also needs to
> be keeping track of the checksumming.
>
> That begs the question why large TCP or UDP packets also have to
> bypass the checksumming (like e.g. ICMP does). If the hardware is
> splitting it into smaller packets per-MTU setting, and checksumming
> them, why is this happening then? I don't know. I know it is needed,
> from tests: the OpenWrt webserver uhttpd starts sending big skb:s (up
> to 2047 bytes, the max MTU) and above 1514 bytes it starts to fail
> and hang unless the bypass bit is set: the frames are not getting
> through.
>
> Keeping the size check but removing the software checksum makes things
> work again. This was probably dubious to introduce in the first place.
>
> Fixes: d4d0c5b4d279 ("net: ethernet: cortina: Handle large frames")
> Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
> ---
>  drivers/net/ethernet/cortina/gemini.c | 8 --------
>  1 file changed, 8 deletions(-)
>
> diff --git a/drivers/net/ethernet/cortina/gemini.c b/drivers/net/ethernet/cortina/gemini.c
> index 78287cfcbf63..255fcffc1579 100644
> --- a/drivers/net/ethernet/cortina/gemini.c
> +++ b/drivers/net/ethernet/cortina/gemini.c
> @@ -1145,7 +1145,6 @@ static int gmac_map_tx_bufs(struct net_device *netdev, struct sk_buff *skb,
>         dma_addr_t mapping;
>         unsigned short mtu;
>         void *buffer;
> -       int ret;
>
>         mtu  = ETH_HLEN;
>         mtu += netdev->mtu;
> @@ -1166,14 +1165,7 @@ static int gmac_map_tx_bufs(struct net_device *netdev, struct sk_buff *skb,
>                  * checksum buffer is only 1518 bytes, so when the frames get
>                  * bigger they get truncated, or the last few bytes get
>                  * overwritten by the FCS.
> -                *
> -                * Just use software checksumming and bypass on bigger frames.
>                  */
> -               if (skb->ip_summed == CHECKSUM_PARTIAL) {
> -                       ret = skb_checksum_help(skb);
> -                       if (ret)
> -                               return ret;
> -               }

If the hardware does not support checksumming for frames bigger than
ETH_FRAME_LEN,
then an appropriate mitigation would be to have an ndo_features_check() ?

Depending on packet being gso or not, you would have to check skb->len
or shinfo->gso_size

The ndo_features_check could then take a more appropriate action
(forcing GSO, and/or forcing software checksumming)

This driver claims to support TSO, but I do not see it using
shinfo->gso_size, something must be very wrong...

I would simply remove this TSO part, before the driver really supports
TSO properly.

diff --git a/drivers/net/ethernet/cortina/gemini.c
b/drivers/net/ethernet/cortina/gemini.c
index 78287cfcbf6388f01bfab417c264f41f3a1a16f2..829cb69982fe1caf99b56363e9e0565fbaecc82e
100644
--- a/drivers/net/ethernet/cortina/gemini.c
+++ b/drivers/net/ethernet/cortina/gemini.c
@@ -79,8 +79,7 @@ MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all)");
 #define GMAC0_IRQ4_8 (GMAC0_MIB_INT_BIT | GMAC0_RX_OVERRUN_INT_BIT)

 #define GMAC_OFFLOAD_FEATURES (NETIF_F_SG | NETIF_F_IP_CSUM | \
-               NETIF_F_IPV6_CSUM | NETIF_F_RXCSUM | \
-               NETIF_F_TSO | NETIF_F_TSO_ECN | NETIF_F_TSO6)
+               NETIF_F_IPV6_CSUM | NETIF_F_RXCSUM )

 /**
  * struct gmac_queue_page - page buffer per-page info

Re: [PATCH net 1/2] net: ethernet: cortina: Drop software checksumming

[Eric Dumazet]

On Fri, Dec 15, 2023 at 10:32 AM Eric Dumazet <edumazet@google.com> wrote:
>
> On Fri, Dec 15, 2023 at 9:49 AM Linus Walleij <linus.walleij@linaro.org> wrote:
> >
> > The recent change to allow large frames without hardware checksumming
> > slotted in software checksumming in the driver if hardware could not
> > do it.
> >
> > This will however upset TSO (TCP Segment Offloading). Typical
> > error dumps includes this:
> >
> > skb len=2961 headroom=222 headlen=66 tailroom=0
> > (...)
> > WARNING: CPU: 0 PID: 956 at net/core/dev.c:3259 skb_warn_bad_offload+0x7c/0x108
> > gemini-ethernet-port: caps=(0x0000010000154813, 0x00002007ffdd7889)
> >
> > And the packets do not go through.
> >
> > After investigating I drilled it down to the introduction of the
> > software checksumming in the driver.
> >
> > Since the segmenting of packets will be done by the hardware this
> > makes a bit of sense since in that case the hardware also needs to
> > be keeping track of the checksumming.
> >
> > That begs the question why large TCP or UDP packets also have to
> > bypass the checksumming (like e.g. ICMP does). If the hardware is
> > splitting it into smaller packets per-MTU setting, and checksumming
> > them, why is this happening then? I don't know. I know it is needed,
> > from tests: the OpenWrt webserver uhttpd starts sending big skb:s (up
> > to 2047 bytes, the max MTU) and above 1514 bytes it starts to fail
> > and hang unless the bypass bit is set: the frames are not getting
> > through.
> >
> > Keeping the size check but removing the software checksum makes things
> > work again. This was probably dubious to introduce in the first place.
> >
> > Fixes: d4d0c5b4d279 ("net: ethernet: cortina: Handle large frames")
> > Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
> > ---
> >  drivers/net/ethernet/cortina/gemini.c | 8 --------
> >  1 file changed, 8 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/cortina/gemini.c b/drivers/net/ethernet/cortina/gemini.c
> > index 78287cfcbf63..255fcffc1579 100644
> > --- a/drivers/net/ethernet/cortina/gemini.c
> > +++ b/drivers/net/ethernet/cortina/gemini.c
> > @@ -1145,7 +1145,6 @@ static int gmac_map_tx_bufs(struct net_device *netdev, struct sk_buff *skb,
> >         dma_addr_t mapping;
> >         unsigned short mtu;
> >         void *buffer;
> > -       int ret;
> >
> >         mtu  = ETH_HLEN;
> >         mtu += netdev->mtu;
> > @@ -1166,14 +1165,7 @@ static int gmac_map_tx_bufs(struct net_device *netdev, struct sk_buff *skb,
> >                  * checksum buffer is only 1518 bytes, so when the frames get
> >                  * bigger they get truncated, or the last few bytes get
> >                  * overwritten by the FCS.
> > -                *
> > -                * Just use software checksumming and bypass on bigger frames.
> >                  */
> > -               if (skb->ip_summed == CHECKSUM_PARTIAL) {
> > -                       ret = skb_checksum_help(skb);
> > -                       if (ret)
> > -                               return ret;
> > -               }
>
> If the hardware does not support checksumming for frames bigger than
> ETH_FRAME_LEN,
> then an appropriate mitigation would be to have an ndo_features_check() ?
>
> Depending on packet being gso or not, you would have to check skb->len
> or shinfo->gso_size
>
> The ndo_features_check could then take a more appropriate action
> (forcing GSO, and/or forcing software checksumming)
>
> This driver claims to support TSO, but I do not see it using
> shinfo->gso_size, something must be very wrong...
>
> I would simply remove this TSO part, before the driver really supports
> TSO properly.
>
> diff --git a/drivers/net/ethernet/cortina/gemini.c
> b/drivers/net/ethernet/cortina/gemini.c
> index 78287cfcbf6388f01bfab417c264f41f3a1a16f2..829cb69982fe1caf99b56363e9e0565fbaecc82e
> 100644
> --- a/drivers/net/ethernet/cortina/gemini.c
> +++ b/drivers/net/ethernet/cortina/gemini.c
> @@ -79,8 +79,7 @@ MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all)");
>  #define GMAC0_IRQ4_8 (GMAC0_MIB_INT_BIT | GMAC0_RX_OVERRUN_INT_BIT)
>
>  #define GMAC_OFFLOAD_FEATURES (NETIF_F_SG | NETIF_F_IP_CSUM | \
> -               NETIF_F_IPV6_CSUM | NETIF_F_RXCSUM | \
> -               NETIF_F_TSO | NETIF_F_TSO_ECN | NETIF_F_TSO6)
> +               NETIF_F_IPV6_CSUM | NETIF_F_RXCSUM )
>
>  /**
>   * struct gmac_queue_page - page buffer per-page info


I do not have the datasheet for this NIC, but my naive attempt would
be something like:

diff --git a/drivers/net/ethernet/cortina/gemini.c
b/drivers/net/ethernet/cortina/gemini.c
index 78287cfcbf6388f01bfab417c264f41f3a1a16f2..3c902dac16f7d539349178ebc4a49c7e48edced5
100644
--- a/drivers/net/ethernet/cortina/gemini.c
+++ b/drivers/net/ethernet/cortina/gemini.c
@@ -1143,39 +1143,18 @@ static int gmac_map_tx_bufs(struct net_device
*netdev, struct sk_buff *skb,
        struct gmac_txdesc *txd;
        skb_frag_t *skb_frag;
        dma_addr_t mapping;
-       unsigned short mtu;
        void *buffer;
        int ret;

-       mtu  = ETH_HLEN;
-       mtu += netdev->mtu;
-       if (skb->protocol == htons(ETH_P_8021Q))
-               mtu += VLAN_HLEN;
-
        word1 = skb->len;
        word3 = SOF_BIT;

-       if (word1 > mtu) {
+       if (skb_is_gso(skb)) {
                word1 |= TSS_MTU_ENABLE_BIT;
-               word3 |= mtu;
+               word3 |= skb_shinfo(skb)->gso_size;
        }

-       if (skb->len >= ETH_FRAME_LEN) {
-               /* Hardware offloaded checksumming isn't working on frames
-                * bigger than 1514 bytes. A hypothesis about this is that the
-                * checksum buffer is only 1518 bytes, so when the frames get
-                * bigger they get truncated, or the last few bytes get
-                * overwritten by the FCS.
-                *
-                * Just use software checksumming and bypass on bigger frames.
-                */
-               if (skb->ip_summed == CHECKSUM_PARTIAL) {
-                       ret = skb_checksum_help(skb);
-                       if (ret)
-                               return ret;
-               }
-               word1 |= TSS_BYPASS_BIT;
-       } else if (skb->ip_summed == CHECKSUM_PARTIAL) {
+       if (skb->ip_summed == CHECKSUM_PARTIAL) {
                int tcp = 0;

                /* We do not switch off the checksumming on non TCP/UDP

Re: [PATCH net 1/2] net: ethernet: cortina: Drop software checksumming

[Linus Walleij]

On Fri, Dec 15, 2023 at 10:32 AM Eric Dumazet <edumazet@google.com> wrote:

> Depending on packet being gso or not, you would have to check skb->len
> or shinfo->gso_size
>
> The ndo_features_check could then take a more appropriate action
> (forcing GSO, and/or forcing software checksumming)

We had something like that before but it looked weird because
it was just looking at the MTU.

> This driver claims to support TSO, but I do not see it using
> shinfo->gso_size, something must be very wrong...
>
> I would simply remove this TSO part, before the driver really supports
> TSO properly.

I added a hunk dropping the TSO flags for v2. Sending out tomorrow!

Yours,
Linus Walleij

[PATCH net 2/2] net: ethernet: cortina: Bypass checksumming engine of alien ethertypes

[Linus Walleij]

We had workarounds were the ethernet checksumming engine would be bypassed
for larger frames, this fixed devices using DSA, but regressed devices
where the ethernet was connected directly to a PHY.

The devices with a PHY connected directly can't handle large frames
either way, with or without bypass. Looking at the size of the frame
is probably just wrong.

Rework the workaround such that we just bypass the checksumming engine if
the ethertype inside the actual frame is something else than 0x0800
(IPv4) or 0x86dd (IPv6). These are the only frames the checksumming engine
can actually handle. VLAN framing (0x8100) also works fine.

We can't inspect skb->protocol because DSA frames will sometimes have a
custom ethertype despite skb->protocol is e.g. 0x0800.

After this both devices with direct ethernet attached such as D-Link
DNS-313 and devices with a DSA switch with a custom ethertype such as
D-Link DIR-685 work fine.

Fixes: d4d0c5b4d279 ("net: ethernet: cortina: Handle large frames")
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
---
 drivers/net/ethernet/cortina/gemini.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/cortina/gemini.c b/drivers/net/ethernet/cortina/gemini.c
index 255fcffc1579..934016c8caa9 100644
--- a/drivers/net/ethernet/cortina/gemini.c
+++ b/drivers/net/ethernet/cortina/gemini.c
@@ -1144,7 +1144,9 @@ static int gmac_map_tx_bufs(struct net_device *netdev, struct sk_buff *skb,
 	skb_frag_t *skb_frag;
 	dma_addr_t mapping;
 	unsigned short mtu;
+	u16 ethertype;
 	void *buffer;
+	__be16 *p;
 
 	mtu  = ETH_HLEN;
 	mtu += netdev->mtu;
@@ -1159,12 +1161,21 @@ static int gmac_map_tx_bufs(struct net_device *netdev, struct sk_buff *skb,
 		word3 |= mtu;
 	}
 
-	if (skb->len >= ETH_FRAME_LEN) {
-		/* Hardware offloaded checksumming isn't working on frames
-		 * bigger than 1514 bytes. A hypothesis about this is that the
-		 * checksum buffer is only 1518 bytes, so when the frames get
-		 * bigger they get truncated, or the last few bytes get
-		 * overwritten by the FCS.
+	/* Dig out the the ethertype actually in the buffer and not what the
+	 * protocol claims to be. This is the raw data that the checksumming
+	 * offload engine will have to deal with.
+	 */
+	p = (__be16 *)(skb->data + 2 * ETH_ALEN);
+	ethertype = ntohs(*p);
+	if (ethertype == ETH_P_8021Q) {
+		p += 2; /* +2 sizeof(__be16) */
+		ethertype = ntohs(*p);
+	}
+
+	if (ethertype != ETH_P_IP && ethertype != ETH_P_IPV6) {
+		/* Hardware offloaded checksumming isn't working on non-IP frames.
+		 * This happens for example on some DSA switches using a custom
+		 * ethertype. Just bypass the engine for those.
 		 */
 		word1 |= TSS_BYPASS_BIT;
 	} else if (skb->ip_summed == CHECKSUM_PARTIAL) {

-- 
2.34.1