i40e XDP program stops transmitting after link down/up

Seth Forshee
2024-01-30T14:17:17Z

I got a inquiry from a colleague about a behavior he's seeing with i40e
but not with other NICs. The interfaces are bonded with a XDP
load-balancer program attached to them. After 'ip link set ethX down; ip
link set ethX up' on one of the interfaces the XDP program on that
interface is no longer transmitting packets. He found that tx starts
again after running 'sudo ethtool -t ethX'.

There's a 'i40e 0000:d8:00.1: VSI seid 391 XDP Tx ring 0 disable
timeout' message in dmesg when disabling the interface. I've included
the relevant portions from dmesg below.

This was first observed with a 6.1 kernel, but we've confirmed that the
behavior is the same in 6.7. I realize the firmware is pretty old, so
far our attempts to update the NVM have failed.

Thanks,
Seth

[ 0.000000] Linux version 6.7.0 (root@616a530b3729) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #2 SMP Thu Jan 25 10:37:21 EST 2024
...
[ 9.038171] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[ 9.044447] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
...
[ 9.064833] i40e 0000:d8:00.0: fw 7.0.50775 api 1.8 nvm 7.00 0x80004c97 1.2154.0 [8086:1583] [8086:0002]
...
[ 9.320886] i40e 0000:d8:00.0: MAC address: xx:xx:xx:xx:xx:xx
[ 9.327030] i40e 0000:d8:00.0: FW LLDP is enabled
[ 9.344331] i40e 0000:d8:00.0 eth0: NIC Link is Up, 40 Gbps Full Duplex, Flow Control: None
[ 9.355552] i40e 0000:d8:00.0: PCI-Express: Speed 8.0GT/s Width x8
[ 9.374074] i40e 0000:d8:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 32 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
...
[ 9.401522] i40e 0000:d8:00.1: fw 7.0.50775 api 1.8 nvm 7.00 0x80004c97 1.2154.0 [8086:1583] [8086:0002]
...
[ 9.652066] i40e 0000:d8:00.1: MAC address: xx:xx:xx:xx:xx:xx
[ 9.658040] i40e 0000:d8:00.1: FW LLDP is enabled
[ 9.688622] i40e 0000:d8:00.1 eth1: NIC Link is Up, 40 Gbps Full Duplex, Flow Control: None
[ 9.699822] i40e 0000:d8:00.1: PCI-Express: Speed 8.0GT/s Width x8
[ 9.719259] i40e 0000:d8:00.1: Features: PF-id[1] VFs: 64 VSIs: 66 QP: 32 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 9.401522] i40e 0000:d8:00.1: fw 7.0.50775 api 1.8 nvm 7.00 0x80004c97 1.2154.0 [8086:1583] [8086:0002]
...
[ 13.948968] i40e 0000:d8:00.0: FW LLDP is disabled, attempting SW DCB
[ 13.956753] i40e 0000:d8:00.0: SW DCB initialization succeeded.
[ 13.970005] i40e 0000:d8:00.0: FW LLDP is disabled
...
[ 14.113004] i40e 0000:d8:00.1: FW LLDP is disabled, attempting SW DCB
[ 14.120800] i40e 0000:d8:00.1: SW DCB initialization succeeded.
[ 14.133940] i40e 0000:d8:00.1: FW LLDP is disabled
...
[ 14.267874] bonding: bond0 is being created...
[ 14.979070] i40e 0000:d8:00.1 eth1: set new mac address xx:xx:xx:xx:xx:xx
[ 14.994445] bond0: (slave eth1): Enslaving as a backup interface with an up link
[ 14.994681] i40e 0000:d8:00.0 eth0: set new mac address xx:xx:xx:xx:xx:xx
[ 15.010359] bond0: (slave eth0): Enslaving as a backup interface with an up link
[ 15.912874] i40e 0000:d8:00.0: Stop LLDP AQ command failed =0x1
[ 15.944842] i40e 0000:d8:00.1: Stop LLDP AQ command failed =0x1
[ 41.262871] 8021q: 802.1Q VLAN Support v1.8
[ 41.262890] 8021q: adding VLAN 0 to HW filter on device eth0
[ 41.262902] 8021q: adding VLAN 0 to HW filter on device eth1
[ 41.262914] 8021q: adding VLAN 0 to HW filter on device bond0
[ 48.272456] i40e 0000:d8:00.0: FW LLDP is disabled, attempting SW DCB
[ 48.280233] i40e 0000:d8:00.0: SW DCB initialization succeeded.
[ 48.307415] i40e 0000:d8:00.0: User requested queue count/HW max RSS count: 12/32
[ 48.440266] i40e 0000:d8:00.1: FW LLDP is disabled, attempting SW DCB
[ 48.448025] i40e 0000:d8:00.1: SW DCB initialization succeeded.
[ 48.475051] i40e 0000:d8:00.1: User requested queue count/HW max RSS count: 12/32
[ 58.935900] i40e 0000:d8:00.0: FW LLDP is disabled, attempting SW DCB
[ 58.945123] i40e 0000:d8:00.0: SW DCB initialization succeeded.
[ 59.131772] i40e 0000:d8:00.1: FW LLDP is disabled, attempting SW DCB
[ 59.139560] i40e 0000:d8:00.1: SW DCB initialization succeeded.
[ 336.363825] i40e 0000:d8:00.1: VSI seid 391 XDP Tx ring 0 disable timeout
[ 336.603619] bond0: (slave eth1): link status definitely down, disabling slave
[ 345.464976] 8021q: adding VLAN 0 to HW filter on device eth1
[ 345.547358] bond0: (slave eth1): link status definitely up, 40000 Mbps full duplex
[ 345.547379] bond0: active interface up!