stmmac: zero udp checksum

stmmac: zero udp checksum

[Julian Labus]

Hi all,

in our community mesh network we recently discovered that a TP-Link 
Archer C2600 device is unable to receive IPv6 UDP packets with a zero 
checksum when RX checksum offloading is enabled. The device uses 
ipq806x-gmac-dwmac for its ethernet ports.

According to https://tools.ietf.org/html/rfc2460#section-8.1 this sounds 
like correct behavior as it says a UDP checksum must not be zero for 
IPv6 packets. But this definition was relaxed in 
https://tools.ietf.org/html/rfc6935#section-5 to allow zero checksums in 
tunneling protocols like VXLAN where we discovered the problem.

Can the behavior of the stmmac driver be changed to meet RFC6935 or 
would it be possible to make the (RX) Checksum Offloading Engine 
configurable via a device tree property to disable it in environments 
were it causes problems?

Best regards,
Julian Labus

Re: stmmac: zero udp checksum

[Andrew Lunn]

On Mon, Apr 05, 2021 at 02:53:15PM +0200, Julian Labus wrote:
> Hi all,
> 
> in our community mesh network we recently discovered that a TP-Link Archer
> C2600 device is unable to receive IPv6 UDP packets with a zero checksum when
> RX checksum offloading is enabled. The device uses ipq806x-gmac-dwmac for
> its ethernet ports.
> 
> According to https://tools.ietf.org/html/rfc2460#section-8.1 this sounds
> like correct behavior as it says a UDP checksum must not be zero for IPv6
> packets. But this definition was relaxed in
> https://tools.ietf.org/html/rfc6935#section-5 to allow zero checksums in
> tunneling protocols like VXLAN where we discovered the problem.
> 
> Can the behavior of the stmmac driver be changed to meet RFC6935 or would it
> be possible to make the (RX) Checksum Offloading Engine configurable via a
> device tree property to disable it in environments were it causes problems?

Hi Julian

I don't know the stmmac driver at all...

Have you played around with ethtool -k/-K? Can use this to turn off
hardware checksums?

I doubt a DT property would be accepted. What you probably want to do
is react on the NETDEV notifiers for when an upper interface is
changed. If a VXLAN interface is added, turn off hardware checksums.

	 Andrew

Re: stmmac: zero udp checksum

[Julian Labus]

On 05.04.21 15:27, Andrew Lunn wrote:
> On Mon, Apr 05, 2021 at 02:53:15PM +0200, Julian Labus wrote:
>> Hi all,
>>
>> in our community mesh network we recently discovered that a TP-Link Archer
>> C2600 device is unable to receive IPv6 UDP packets with a zero checksum when
>> RX checksum offloading is enabled. The device uses ipq806x-gmac-dwmac for
>> its ethernet ports.
>>
>> According to https://tools.ietf.org/html/rfc2460#section-8.1 this sounds
>> like correct behavior as it says a UDP checksum must not be zero for IPv6
>> packets. But this definition was relaxed in
>> https://tools.ietf.org/html/rfc6935#section-5 to allow zero checksums in
>> tunneling protocols like VXLAN where we discovered the problem.
>>
>> Can the behavior of the stmmac driver be changed to meet RFC6935 or would it
>> be possible to make the (RX) Checksum Offloading Engine configurable via a
>> device tree property to disable it in environments were it causes problems?
> 
> Hi Julian
> 
> I don't know the stmmac driver at all...
> 
> Have you played around with ethtool -k/-K? Can use this to turn off
> hardware checksums?
> 
> I doubt a DT property would be accepted. What you probably want to do
> is react on the NETDEV notifiers for when an upper interface is
> changed. If a VXLAN interface is added, turn off hardware checksums.
> 
> 	 Andrew

Hi Andrew,

yes, disabling the offloading via "ethtool -K <ifname> rx off" works and 
is used as a workaround in a startup script but the OpenWrt-based OS on 
the mentioned device does not provide a reliable way to trigger ethtool 
commands when network devices change.

Reacting to upper interface changes and disabling (rx) offloading per 
interface sounds right when looking at RFC6935 again.

"As an alternative, certain protocols that use UDP as a tunnel
encapsulation MAY enable zero-checksum mode for a specific port
(or set of ports) for sending and/or receiving."

But was is still a bit strange to me is that it seems like the stmmac 
driver behaves different than other ethernet drivers which do not drop 
UDP packets with zero checksums when rx checksumming is enabled.

Julian

Re: stmmac: zero udp checksum

[Andrew Lunn]

> But was is still a bit strange to me is that it seems like the stmmac driver
> behaves different than other ethernet drivers which do not drop UDP packets
> with zero checksums when rx checksumming is enabled.

To answer that, you need somebody with more knowledge of the stmmac
hardware. It is actually quite hard to do. It means you need to parse
more of the frame to determine if the frame contains a VXLAN
encapsulated frame. Probably the stmmac cannot do that. It sees the
checksum is wrong and drops the packet.

Have you looked at where it actually drops the packet?
Is it one of

https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/stmicro/stmmac/norm_desc.c#L95

or

https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/stmicro/stmmac/enh_desc.c#L87

It could be, you need to see if the checksum has fail, then check if
the checksum is actually zero, and then go deeper into the frame and
check if it is a vxlan frame. It could be the linux software checksum
code knows about this vxlan exception, so you can just run that before
deciding to drop the frame.

	Andrew

Re: stmmac: zero udp checksum

[Julian Labus]

On 05.04.21 18:42, Andrew Lunn wrote:
> Have you looked at where it actually drops the packet?
> Is it one of
> 
> https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/stmicro/stmmac/norm_desc.c#L95
> 
> or
> 
> https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/stmicro/stmmac/enh_desc.c#L87
> 
> It could be, you need to see if the checksum has fail, then check if
> the checksum is actually zero, and then go deeper into the frame and
> check if it is a vxlan frame. It could be the linux software checksum
> code knows about this vxlan exception, so you can just run that before
> deciding to drop the frame.
> 
> 	Andrew

No, I've not yet checked where the packet is actually dropped. This is 
my first time debugging a network problem in the kernel and I have 
little to no knowledge how this rx offloading works. But I don't think 
it happens in norm_desc.c#L95 as I would expect the ipc_csum_error 
counter to increase. But the only counters in "ethtool -S" which are 
increasing are mmc_rx_udp_err and mmc_rx_udp_err_octets. I'll try to get 
more information.

Julian

Re: stmmac: zero udp checksum

[Jakub Kicinski]

On Mon, 5 Apr 2021 18:42:53 +0200 Andrew Lunn wrote:
> > But was is still a bit strange to me is that it seems like the stmmac driver
> > behaves different than other ethernet drivers which do not drop UDP packets
> > with zero checksums when rx checksumming is enabled.  
> 
> To answer that, you need somebody with more knowledge of the stmmac
> hardware.

+1 stmmac maintainers could you advise?

> It is actually quite hard to do. It means you need to parse
> more of the frame to determine if the frame contains a VXLAN
> encapsulated frame. Probably the stmmac cannot do that. It sees the
> checksum is wrong and drops the packet.
>
> Have you looked at where it actually drops the packet?
> Is it one of
> 
> https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/stmicro/stmmac/norm_desc.c#L95
> 
> or
> 
> https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/stmicro/stmmac/enh_desc.c#L87
> 
> It could be, you need to see if the checksum has fail, then check if
> the checksum is actually zero, and then go deeper into the frame and
> check if it is a vxlan frame. It could be the linux software checksum
> code knows about this vxlan exception, so you can just run that before
> deciding to drop the frame.

To be clear the expectation is that devices / drivers will only drop
packets on L2 / FCS errors. If L3 or L4 csum is incorrect the packet
should be passed up the stack and kernel will handle it how it sees fit.

Re: stmmac: zero udp checksum

[Julian Labus]

Hi stmmac maintainers,

it's around 1 1/2 months without any response on this topic.
Could you please advice how to proceed with this problem?

Kind regards,
Julian

On 06.04.21 01:03, Jakub Kicinski wrote:
> On Mon, 5 Apr 2021 18:42:53 +0200 Andrew Lunn wrote:
>>> But was is still a bit strange to me is that it seems like the stmmac driver
>>> behaves different than other ethernet drivers which do not drop UDP packets
>>> with zero checksums when rx checksumming is enabled.
>>
>> To answer that, you need somebody with more knowledge of the stmmac
>> hardware.
> 
> +1 stmmac maintainers could you advise?
> 
>> It is actually quite hard to do. It means you need to parse
>> more of the frame to determine if the frame contains a VXLAN
>> encapsulated frame. Probably the stmmac cannot do that. It sees the
>> checksum is wrong and drops the packet.
>>
>> Have you looked at where it actually drops the packet?
>> Is it one of
>>
>> https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/stmicro/stmmac/norm_desc.c#L95
>>
>> or
>>
>> https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/stmicro/stmmac/enh_desc.c#L87
>>
>> It could be, you need to see if the checksum has fail, then check if
>> the checksum is actually zero, and then go deeper into the frame and
>> check if it is a vxlan frame. It could be the linux software checksum
>> code knows about this vxlan exception, so you can just run that before
>> deciding to drop the frame.
> 
> To be clear the expectation is that devices / drivers will only drop
> packets on L2 / FCS errors. If L3 or L4 csum is incorrect the packet
> should be passed up the stack and kernel will handle it how it sees fit.
>