Issue with r8169 (using r8168 for now) driving 8111E

Issue with r8169 (using r8168 for now) driving 8111E

[Mason Loring Bliss]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

Hi all. I was happily running Debian Buster with Linux 4.19.194 driving a
pair of 8111E NICs as included on this board:

    https://www.aaeon.com/en/p/mini-itx-emb-cv1

I upgraded to Debian Bullseye running 5.10.127 and started seeing this
popping up regularly in dmesg, with the status varying:

    r8169 0000:03:00.0 eth1: Rx ERROR. status = 3529c123

As this box is being used as a firewall, I didn't want to leave it with an
obvious issue, so I installed r8168-dkms and it appears to function with no
issues.

If it matters, I was not seeing the error against eth0, just eth1, and in
this case eth1 is used for PPPoE to the world, while eth0 talks to my
internal network.

I've not yet tried the Debian-backports kernel to see if the r8169 there
works, but I can do so given some scheduled downtime. I'm writing in
advance of that in case the nature of the issue jogs a memory of something
already seen and addressed.

If you tell me what debugging data might be useful, I can supply it.

-- 
Mason Loring Bliss  ((   If I have not seen as far as others, it is because
 mason@blisses.org   ))   giants were standing on my shoulders. - Hal Abelson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Issue with r8169 (using r8168 for now) driving 8111E

[Heiner Kallweit]

On 23.07.2022 21:15, Mason Loring Bliss wrote:
> Hi all. I was happily running Debian Buster with Linux 4.19.194 driving a
> pair of 8111E NICs as included on this board:
> 
>     https://www.aaeon.com/en/p/mini-itx-emb-cv1
> 
> I upgraded to Debian Bullseye running 5.10.127 and started seeing this
> popping up regularly in dmesg, with the status varying:
> 
>     r8169 0000:03:00.0 eth1: Rx ERROR. status = 3529c123
> 
> As this box is being used as a firewall, I didn't want to leave it with an
> obvious issue, so I installed r8168-dkms and it appears to function with no
> issues.
> 
> If it matters, I was not seeing the error against eth0, just eth1, and in
> this case eth1 is used for PPPoE to the world, while eth0 talks to my
> internal network.
> 
> I've not yet tried the Debian-backports kernel to see if the r8169 there
> works, but I can do so given some scheduled downtime. I'm writing in
> advance of that in case the nature of the issue jogs a memory of something
> already seen and addressed.
> 
> If you tell me what debugging data might be useful, I can supply it.
> 

The error message indicates an incoming packet CRC error.
4.19 doesn't report rx errors per default whilst more recent
kernel versions do.

Re: Issue with r8169 (using r8168 for now) driving 8111E

[Mason Loring Bliss]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

On Sat, Jul 23, 2022 at 10:48:27PM +0200, Heiner Kallweit wrote:

> The error message indicates an incoming packet CRC error. 4.19 doesn't
> report rx errors per default whilst more recent kernel versions do.

Ah, thank you. That sounds like a complete answer, and not an issue the
driver would be causing. I will go back to the in-kernel driver and focus
on identifying the issue with my upstream connection now.

-- 
Mason Loring Bliss   ((  "In the drowsy dark cave of the mind dreams
mason@blisses.org     ))  build  their nest  with fragments  dropped
http://blisses.org/  ((   from day's caravan." - Rabindranath Tagore

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Issue with r8169 (using r8168 for now) driving 8111E

[Francois Romieu]

(more than current r8169 maintainer Cced -> Heiner Kallweit)

Mason Loring Bliss <mason@blisses.org> :
> Hi all. I was happily running Debian Buster with Linux 4.19.194 driving a
> pair of 8111E NICs as included on this board:
> 
>     https://www.aaeon.com/en/p/mini-itx-emb-cv1
> 
> I upgraded to Debian Bullseye running 5.10.127 and started seeing this
> popping up regularly in dmesg, with the status varying:
> 
>     r8169 0000:03:00.0 eth1: Rx ERROR. status = 3529c123

This message was emitted as INFO in 4.19 whereas 5.10 considers it WARN.

You may check/enable the relevant message level with ethtool. I would
naïvely expect v4.19 to silently detect the error.

For your information, the content of the faulty descriptor word is
summarized below:

OWN  0
EOR  0
FS   1 (first segment)
LS   1 (last segment)

MAR  0 (multicast)
PAM  1 (physical address match)
BAR  0
RESV 1 (reserved)

RESV 1 (reserved)
RWT  0
RES  1 (receive error summay)
RUNT 0

CRC  1 (CRC error)
UDP  0
TCP  0
IP   1 (IP checksum failure)

UDPF 1 (udp checksum failure)
TCPF 1 (tcp checksum failure)

Frame length 0x123 -> 291

The driver should receive and process this kind of CRC errored packet since
79d0c1d26e1eac0dc5b201e66b65cc5e4e706743 ("r8169: Support RX-FCS flag.")
provided ethtool was not used to disable it (see -k/-K and rx-fcs/rx-all).
tcpdump may thus help to identify some pattern in the packet.

HTH.

-- 
Ueimor

Re: Issue with r8169 (using r8168 for now) driving 8111E

[Mason Loring Bliss]

[-- Attachment #1: Type: text/plain, Size: 1067 bytes --]

On Sun, Jul 24, 2022 at 12:52:13AM +0200, Francois Romieu wrote:

> The driver should receive and process this kind of CRC errored packet since
> 79d0c1d26e1eac0dc5b201e66b65cc5e4e706743 ("r8169: Support RX-FCS flag.")
> provided ethtool was not used to disable it (see -k/-K and rx-fcs/rx-all).
> tcpdump may thus help to identify some pattern in the packet.

Thank you for the additional detail!

It appears to be the case that having an APC surge supressor inline,
despite it being labelled as gigabit-capable, was the cause of the CRC
errors. I changed cables, and that not helping, I swapped in different APC
units, and the errors persisted until I went straight from NIC to ONT. Now
I've gone a couple hours without seeing an error, and haven't in fact seen
one since the link came back up after the most recent change.

I'm glad the driver stopped suppressing these errors, or I'd not have known
the inline surge supression was a bit flaky. Thank you.

-- 
Mason Loring Bliss    mason@blisses.org
They also surf, who only stand on waves.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]