ICE + XSK ZC - page faults on 6.1 LTS when process exits?

ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Alasdair McWilliam]

Hi,

I've been testing apps that use XSK+ZC on ICE with newer builds of the 6.1 LTS kernel in preparation for some production upgrades, and I've started to notice some instability on newer versions. I can reproduce the issue easily in the lab.

Config:
- Known good multi-threaded application (i.e. production grade)
- Uses eBPF and AF_XDP with zero copy to act as 'bump in wire' in network
- Xeon's with Intel E810-CQDA2 (firmware: 3.20 0x8000d83e 1.3146.0)
- Effectively a vanilla rebuild of 6.1 using configs from el-repo project

Scenario:
- Noticing hard kernel faults when shutting down application
- Can happen if the process is shut down via systemctl stop
- Can even happen with a simple kill -9 command to the PID
- Appears in builds after 6.1.87

Tested kernels:
- 6.1.84: process exits smoothly
- 6.1.87: process exits smoothly
- 6.1.97: BUG: unable to handle page fault for address
- 6.1.106: BUG: unable to handle page fault for address

Kdump log is below [1] from 6.1.106 but does seem to be the same in the earlier version.

Can anyone advise if this is a known issue?

I don't have any builds between 6.1.87 and 6.1.97 but I can spend some time trying to pinpoint the exact version things start to go wrong in, if it would help anyone better equipped than me to debug!

Kind regards
Alasdair

[1] kdump log

[  158.666867] BUG: unable to handle page fault for address: ffffa6510e5580c0
[  158.666887] #PF: supervisor read access in kernel mode
[  158.666896] #PF: error_code(0x0000) - not-present page
[  158.666903] PGD 100000067 P4D 100000067 PUD 106dc4067 PMD 0
[  158.666914] Oops: 0000 [#1] PREEMPT SMP PTI
[  158.666922] CPU: 7 PID: 1808 Comm: tlndd.bin Kdump: loaded Tainted: G            E      6.1.106-1.X.el9.x86_64 #1
[  158.666940] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS 3.2 12/16/2019
[  158.666950] RIP: 0010:xp_free+0x11/0x80
[  158.666962] Code: 8b 04 d0 48 83 e0 fe 48 01 f0 c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 56 41 55 41 54 55 48 8d 6f 58 53 <48> 8b 47 58 48 39 c5 74 0d 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc
[  158.666985] RSP: 0018:ffffa65089e8b760 EFLAGS: 00010202
[  158.666993] RAX: ffff8fcf077c0000 RBX: 0000000000000001 RCX: 0000000000000000
[  158.667003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa6510e558068
[  158.667012] RBP: ffffa6510e5580c0 R08: fffff8c50415a108 R09: ffff8fc7cac60000
[  158.667022] R10: 0000000000000219 R11: ffffffffffffffff R12: 0000000000000fff
[  158.667031] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8fc7c139d340
[  158.667040] FS:  00007f8504996880(0000) GS:ffff8fcedfdc0000(0000) knlGS:0000000000000000
[  158.667050] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  158.667058] CR2: ffffa6510e5580c0 CR3: 00000001448e2002 CR4: 00000000001706e0
[  158.667068] Call Trace:
[  158.667075]  <TASK>
[  158.667082]  ? show_trace_log_lvl+0x1c4/0x2df
[  158.667094]  ? show_trace_log_lvl+0x1c4/0x2df
[  158.667103]  ? ice_xsk_clean_rx_ring+0x39/0x60 [ice]
[  158.667157]  ? __die_body.cold+0x8/0xd
[  158.667166]  ? page_fault_oops+0xac/0x150
[  158.667176]  ? fixup_exception+0x22/0x340
[  158.667185]  ? exc_page_fault+0xb2/0x150
[  158.667195]  ? asm_exc_page_fault+0x22/0x30
[  158.667206]  ? xp_free+0x11/0x80
[  158.667215]  ice_xsk_clean_rx_ring+0x39/0x60 [ice]
[  158.667250]  ice_clean_rx_ring+0x157/0x180 [ice]
[  158.667284]  ice_down+0x172/0x2b0 [ice]
[  158.667311]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
[  158.667337]  ice_xdp_setup_prog+0xe3/0x3b0 [ice]
[  158.667364]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
[  158.667391]  dev_xdp_install+0xc7/0x100
[  158.667402]  dev_xdp_attach+0x1e0/0x560
[  158.667412]  do_setlink+0x7a8/0xc10
[  158.667422]  ? __nla_validate_parse+0x12b/0x1b0
[  158.667436]  __rtnl_newlink+0x540/0x650
[  158.667446]  rtnl_newlink+0x44/0x70
[  158.667454]  rtnetlink_rcv_msg+0x15c/0x3d0
[  158.667477]  ? rtnl_calcit.isra.0+0x140/0x140
[  158.667485]  netlink_rcv_skb+0x51/0x100
[  158.667727]  netlink_unicast+0x246/0x360
[  158.667953]  netlink_sendmsg+0x24e/0x4b0
[  158.668173]  __sock_sendmsg+0x62/0x70
[  158.668389]  ____sys_sendmsg+0x247/0x2d0
[  158.668602]  ? copy_msghdr_from_user+0x6d/0xa0
[  158.668815]  ___sys_sendmsg+0x88/0xd0
[  158.669028]  ? __sk_destruct+0x156/0x230
[  158.669234]  ? kmem_cache_free+0x134/0x300
[  158.669437]  ? rcu_nocb_try_bypass+0x4a/0x440
[  158.669634]  ? __sk_destruct+0x156/0x230
[  158.669825]  ? _raw_spin_unlock_irqrestore+0x23/0x40
[  158.670010]  ? mod_objcg_state+0xc9/0x2f0
[  158.670186]  ? refill_obj_stock+0xae/0x160
[  158.670359]  ? rseq_get_rseq_cs.isra.0+0x16/0x220
[  158.670529]  ? rcu_nocb_try_bypass+0x4a/0x440
[  158.670696]  ? rseq_ip_fixup+0x72/0x1e0
[  158.670860]  __sys_sendmsg+0x59/0xa0
[  158.671021]  ? syscall_trace_enter.constprop.0+0x11e/0x190
[  158.671185]  do_syscall_64+0x35/0x80
[  158.671345]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  158.671503] RIP: 0033:0x7f850510f917
[  158.671658] Code: 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
[  158.671993] RSP: 002b:00007ffcc805f238 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[  158.672171] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f850510f917
[  158.672352] RDX: 0000000000000000 RSI: 000000000198e9e8 RDI: 0000000000000009
[  158.672534] RBP: 0000000001933c00 R08: 0000000001935980 R09: 0000000000460e48
[  158.672716] R10: 0000000000000011 R11: 0000000000000246 R12: 0000000001933c30
[  158.672899] R13: 0000000000515fd8 R14: 000000000198e9d0 R15: 0000000000513690
[  158.673086]  </TASK>
[  158.673269] Modules linked in: bonding(E) tls(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) intel_rapl_msr(E) intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) kvm(E) irqbypass(E) rapl(E) intel_cstate(E) ast(E) intel_uncore(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E) pcspkr(E) mei_me(E) drm_kms_helper(E) i2c_i801(E) lpc_ich(E) mei(E) i2c_smbus(E) mxm_wmi(E) ioatdma(E) acpi_ipmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) acpi_power_meter(E) joydev(E) drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) sg(E) ahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libahci(E) crc32c_intel(E) ice(E)
[  158.673314]  polyval_clmulni(E) polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E) i2c_algo_bit(E) dca(E) wmi(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[  158.675578] CR2: ffffa6510e5580c0

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Magnus Karlsson]

On Thu, 22 Aug 2024 at 18:25, Alasdair McWilliam
<alasdair.mcwilliam@outlook.com> wrote:
>
> Hi,
>
> I've been testing apps that use XSK+ZC on ICE with newer builds of the 6.1 LTS kernel in preparation for some production upgrades, and I've started to notice some instability on newer versions. I can reproduce the issue easily in the lab.
>
> Config:
> - Known good multi-threaded application (i.e. production grade)
> - Uses eBPF and AF_XDP with zero copy to act as 'bump in wire' in network
> - Xeon's with Intel E810-CQDA2 (firmware: 3.20 0x8000d83e 1.3146.0)
> - Effectively a vanilla rebuild of 6.1 using configs from el-repo project
>
> Scenario:
> - Noticing hard kernel faults when shutting down application
> - Can happen if the process is shut down via systemctl stop
> - Can even happen with a simple kill -9 command to the PID
> - Appears in builds after 6.1.87
>
> Tested kernels:
> - 6.1.84: process exits smoothly
> - 6.1.87: process exits smoothly
> - 6.1.97: BUG: unable to handle page fault for address
> - 6.1.106: BUG: unable to handle page fault for address
>
> Kdump log is below [1] from 6.1.106 but does seem to be the same in the earlier version.
>
> Can anyone advise if this is a known issue?
>
> I don't have any builds between 6.1.87 and 6.1.97 but I can spend some time trying to pinpoint the exact version things start to go wrong in, if it would help anyone better equipped than me to debug!

Hi Alasdair,

It would be of great help if you could pinpoint the exact version for
this breakage. Hopefully we could then find the commit in the ice
driver that breaks your app, since there should be just a handful of
commits in the ice driver for any stable release.

> Kind regards
> Alasdair
>
> [1] kdump log
>
> [  158.666867] BUG: unable to handle page fault for address: ffffa6510e5580c0
> [  158.666887] #PF: supervisor read access in kernel mode
> [  158.666896] #PF: error_code(0x0000) - not-present page
> [  158.666903] PGD 100000067 P4D 100000067 PUD 106dc4067 PMD 0
> [  158.666914] Oops: 0000 [#1] PREEMPT SMP PTI
> [  158.666922] CPU: 7 PID: 1808 Comm: tlndd.bin Kdump: loaded Tainted: G            E      6.1.106-1.X.el9.x86_64 #1
> [  158.666940] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS 3.2 12/16/2019
> [  158.666950] RIP: 0010:xp_free+0x11/0x80
> [  158.666962] Code: 8b 04 d0 48 83 e0 fe 48 01 f0 c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 56 41 55 41 54 55 48 8d 6f 58 53 <48> 8b 47 58 48 39 c5 74 0d 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc
> [  158.666985] RSP: 0018:ffffa65089e8b760 EFLAGS: 00010202
> [  158.666993] RAX: ffff8fcf077c0000 RBX: 0000000000000001 RCX: 0000000000000000
> [  158.667003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa6510e558068
> [  158.667012] RBP: ffffa6510e5580c0 R08: fffff8c50415a108 R09: ffff8fc7cac60000
> [  158.667022] R10: 0000000000000219 R11: ffffffffffffffff R12: 0000000000000fff
> [  158.667031] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8fc7c139d340
> [  158.667040] FS:  00007f8504996880(0000) GS:ffff8fcedfdc0000(0000) knlGS:0000000000000000
> [  158.667050] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  158.667058] CR2: ffffa6510e5580c0 CR3: 00000001448e2002 CR4: 00000000001706e0
> [  158.667068] Call Trace:
> [  158.667075]  <TASK>
> [  158.667082]  ? show_trace_log_lvl+0x1c4/0x2df
> [  158.667094]  ? show_trace_log_lvl+0x1c4/0x2df
> [  158.667103]  ? ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> [  158.667157]  ? __die_body.cold+0x8/0xd
> [  158.667166]  ? page_fault_oops+0xac/0x150
> [  158.667176]  ? fixup_exception+0x22/0x340
> [  158.667185]  ? exc_page_fault+0xb2/0x150
> [  158.667195]  ? asm_exc_page_fault+0x22/0x30
> [  158.667206]  ? xp_free+0x11/0x80
> [  158.667215]  ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> [  158.667250]  ice_clean_rx_ring+0x157/0x180 [ice]
> [  158.667284]  ice_down+0x172/0x2b0 [ice]
> [  158.667311]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> [  158.667337]  ice_xdp_setup_prog+0xe3/0x3b0 [ice]
> [  158.667364]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> [  158.667391]  dev_xdp_install+0xc7/0x100
> [  158.667402]  dev_xdp_attach+0x1e0/0x560
> [  158.667412]  do_setlink+0x7a8/0xc10
> [  158.667422]  ? __nla_validate_parse+0x12b/0x1b0
> [  158.667436]  __rtnl_newlink+0x540/0x650
> [  158.667446]  rtnl_newlink+0x44/0x70
> [  158.667454]  rtnetlink_rcv_msg+0x15c/0x3d0
> [  158.667477]  ? rtnl_calcit.isra.0+0x140/0x140
> [  158.667485]  netlink_rcv_skb+0x51/0x100
> [  158.667727]  netlink_unicast+0x246/0x360
> [  158.667953]  netlink_sendmsg+0x24e/0x4b0
> [  158.668173]  __sock_sendmsg+0x62/0x70
> [  158.668389]  ____sys_sendmsg+0x247/0x2d0
> [  158.668602]  ? copy_msghdr_from_user+0x6d/0xa0
> [  158.668815]  ___sys_sendmsg+0x88/0xd0
> [  158.669028]  ? __sk_destruct+0x156/0x230
> [  158.669234]  ? kmem_cache_free+0x134/0x300
> [  158.669437]  ? rcu_nocb_try_bypass+0x4a/0x440
> [  158.669634]  ? __sk_destruct+0x156/0x230
> [  158.669825]  ? _raw_spin_unlock_irqrestore+0x23/0x40
> [  158.670010]  ? mod_objcg_state+0xc9/0x2f0
> [  158.670186]  ? refill_obj_stock+0xae/0x160
> [  158.670359]  ? rseq_get_rseq_cs.isra.0+0x16/0x220
> [  158.670529]  ? rcu_nocb_try_bypass+0x4a/0x440
> [  158.670696]  ? rseq_ip_fixup+0x72/0x1e0
> [  158.670860]  __sys_sendmsg+0x59/0xa0
> [  158.671021]  ? syscall_trace_enter.constprop.0+0x11e/0x190
> [  158.671185]  do_syscall_64+0x35/0x80
> [  158.671345]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> [  158.671503] RIP: 0033:0x7f850510f917
> [  158.671658] Code: 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
> [  158.671993] RSP: 002b:00007ffcc805f238 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
> [  158.672171] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f850510f917
> [  158.672352] RDX: 0000000000000000 RSI: 000000000198e9e8 RDI: 0000000000000009
> [  158.672534] RBP: 0000000001933c00 R08: 0000000001935980 R09: 0000000000460e48
> [  158.672716] R10: 0000000000000011 R11: 0000000000000246 R12: 0000000001933c30
> [  158.672899] R13: 0000000000515fd8 R14: 000000000198e9d0 R15: 0000000000513690
> [  158.673086]  </TASK>
> [  158.673269] Modules linked in: bonding(E) tls(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) intel_rapl_msr(E) intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) kvm(E) irqbypass(E) rapl(E) intel_cstate(E) ast(E) intel_uncore(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E) pcspkr(E) mei_me(E) drm_kms_helper(E) i2c_i801(E) lpc_ich(E) mei(E) i2c_smbus(E) mxm_wmi(E) ioatdma(E) acpi_ipmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) acpi_power_meter(E) joydev(E) drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) sg(E) ahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libahci(E) crc32c_intel(E) ice(E)
> [  158.673314]  polyval_clmulni(E) polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E) i2c_algo_bit(E) dca(E) wmi(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
> [  158.675578] CR2: ffffa6510e5580c0

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Maciej Fijalkowski]

On Fri, Aug 23, 2024 at 10:17:35AM +0200, Magnus Karlsson wrote:
> On Thu, 22 Aug 2024 at 18:25, Alasdair McWilliam
> <alasdair.mcwilliam@outlook.com> wrote:
> >
> > Hi,
> >
> > I've been testing apps that use XSK+ZC on ICE with newer builds of the 6.1 LTS kernel in preparation for some production upgrades, and I've started to notice some instability on newer versions. I can reproduce the issue easily in the lab.
> >
> > Config:
> > - Known good multi-threaded application (i.e. production grade)
> > - Uses eBPF and AF_XDP with zero copy to act as 'bump in wire' in network
> > - Xeon's with Intel E810-CQDA2 (firmware: 3.20 0x8000d83e 1.3146.0)
> > - Effectively a vanilla rebuild of 6.1 using configs from el-repo project
> >
> > Scenario:
> > - Noticing hard kernel faults when shutting down application
> > - Can happen if the process is shut down via systemctl stop
> > - Can even happen with a simple kill -9 command to the PID
> > - Appears in builds after 6.1.87
> >
> > Tested kernels:
> > - 6.1.84: process exits smoothly
> > - 6.1.87: process exits smoothly
> > - 6.1.97: BUG: unable to handle page fault for address
> > - 6.1.106: BUG: unable to handle page fault for address
> >
> > Kdump log is below [1] from 6.1.106 but does seem to be the same in the earlier version.
> >
> > Can anyone advise if this is a known issue?
> >
> > I don't have any builds between 6.1.87 and 6.1.97 but I can spend some time trying to pinpoint the exact version things start to go wrong in, if it would help anyone better equipped than me to debug!
> 
> Hi Alasdair,
> 
> It would be of great help if you could pinpoint the exact version for
> this breakage. Hopefully we could then find the commit in the ice
> driver that breaks your app, since there should be just a handful of
> commits in the ice driver for any stable release.

$ git log --oneline v6.1.87..v6.1.97 drivers/net/ethernet/intel/ice/
dd37b86999fd ice: Fix VSI list rule with ICE_SW_LKUP_LAST type
224b69e8751c ice: avoid IRQ collision to fix init failure on ACPI S3 resume
531d85b4fb66 ice: move RDMA init to ice_idc.c
a62c50545b4d ice: remove af_xdp_zc_qps bitmap
447a5433bd1e ice: remove null checks before devm_kfree() calls
a388961be5ed ice: Introduce new parameters in ice_sched_node
17ccdebe5ac7 ice: fix iteration of TLVs in Preserved Fields Area
07cbc5512023 ice: fix accounting if a VLAN already exists
5ef3a27c6142 ice: Interpret .set_channels() input differently
90cbd4c081bb ice: remove unnecessary duplicate checks for VF VSI ID
59161a21cae0 ice: pass VSI pointer into ice_vc_isvalid_q_id
6a6ebec40820 ice: tc: allow zero flags in parsing tc flower

can you revert a62c50545b4d and see if the issue persists?

> 
> > Kind regards
> > Alasdair
> >
> > [1] kdump log
> >
> > [  158.666867] BUG: unable to handle page fault for address: ffffa6510e5580c0
> > [  158.666887] #PF: supervisor read access in kernel mode
> > [  158.666896] #PF: error_code(0x0000) - not-present page
> > [  158.666903] PGD 100000067 P4D 100000067 PUD 106dc4067 PMD 0
> > [  158.666914] Oops: 0000 [#1] PREEMPT SMP PTI
> > [  158.666922] CPU: 7 PID: 1808 Comm: tlndd.bin Kdump: loaded Tainted: G            E      6.1.106-1.X.el9.x86_64 #1
> > [  158.666940] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS 3.2 12/16/2019
> > [  158.666950] RIP: 0010:xp_free+0x11/0x80
> > [  158.666962] Code: 8b 04 d0 48 83 e0 fe 48 01 f0 c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 56 41 55 41 54 55 48 8d 6f 58 53 <48> 8b 47 58 48 39 c5 74 0d 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc
> > [  158.666985] RSP: 0018:ffffa65089e8b760 EFLAGS: 00010202
> > [  158.666993] RAX: ffff8fcf077c0000 RBX: 0000000000000001 RCX: 0000000000000000
> > [  158.667003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa6510e558068
> > [  158.667012] RBP: ffffa6510e5580c0 R08: fffff8c50415a108 R09: ffff8fc7cac60000
> > [  158.667022] R10: 0000000000000219 R11: ffffffffffffffff R12: 0000000000000fff
> > [  158.667031] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8fc7c139d340
> > [  158.667040] FS:  00007f8504996880(0000) GS:ffff8fcedfdc0000(0000) knlGS:0000000000000000
> > [  158.667050] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  158.667058] CR2: ffffa6510e5580c0 CR3: 00000001448e2002 CR4: 00000000001706e0
> > [  158.667068] Call Trace:
> > [  158.667075]  <TASK>
> > [  158.667082]  ? show_trace_log_lvl+0x1c4/0x2df
> > [  158.667094]  ? show_trace_log_lvl+0x1c4/0x2df
> > [  158.667103]  ? ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> > [  158.667157]  ? __die_body.cold+0x8/0xd
> > [  158.667166]  ? page_fault_oops+0xac/0x150
> > [  158.667176]  ? fixup_exception+0x22/0x340
> > [  158.667185]  ? exc_page_fault+0xb2/0x150
> > [  158.667195]  ? asm_exc_page_fault+0x22/0x30
> > [  158.667206]  ? xp_free+0x11/0x80
> > [  158.667215]  ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> > [  158.667250]  ice_clean_rx_ring+0x157/0x180 [ice]
> > [  158.667284]  ice_down+0x172/0x2b0 [ice]
> > [  158.667311]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> > [  158.667337]  ice_xdp_setup_prog+0xe3/0x3b0 [ice]
> > [  158.667364]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> > [  158.667391]  dev_xdp_install+0xc7/0x100
> > [  158.667402]  dev_xdp_attach+0x1e0/0x560
> > [  158.667412]  do_setlink+0x7a8/0xc10
> > [  158.667422]  ? __nla_validate_parse+0x12b/0x1b0
> > [  158.667436]  __rtnl_newlink+0x540/0x650
> > [  158.667446]  rtnl_newlink+0x44/0x70
> > [  158.667454]  rtnetlink_rcv_msg+0x15c/0x3d0
> > [  158.667477]  ? rtnl_calcit.isra.0+0x140/0x140
> > [  158.667485]  netlink_rcv_skb+0x51/0x100
> > [  158.667727]  netlink_unicast+0x246/0x360
> > [  158.667953]  netlink_sendmsg+0x24e/0x4b0
> > [  158.668173]  __sock_sendmsg+0x62/0x70
> > [  158.668389]  ____sys_sendmsg+0x247/0x2d0
> > [  158.668602]  ? copy_msghdr_from_user+0x6d/0xa0
> > [  158.668815]  ___sys_sendmsg+0x88/0xd0
> > [  158.669028]  ? __sk_destruct+0x156/0x230
> > [  158.669234]  ? kmem_cache_free+0x134/0x300
> > [  158.669437]  ? rcu_nocb_try_bypass+0x4a/0x440
> > [  158.669634]  ? __sk_destruct+0x156/0x230
> > [  158.669825]  ? _raw_spin_unlock_irqrestore+0x23/0x40
> > [  158.670010]  ? mod_objcg_state+0xc9/0x2f0
> > [  158.670186]  ? refill_obj_stock+0xae/0x160
> > [  158.670359]  ? rseq_get_rseq_cs.isra.0+0x16/0x220
> > [  158.670529]  ? rcu_nocb_try_bypass+0x4a/0x440
> > [  158.670696]  ? rseq_ip_fixup+0x72/0x1e0
> > [  158.670860]  __sys_sendmsg+0x59/0xa0
> > [  158.671021]  ? syscall_trace_enter.constprop.0+0x11e/0x190
> > [  158.671185]  do_syscall_64+0x35/0x80
> > [  158.671345]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> > [  158.671503] RIP: 0033:0x7f850510f917
> > [  158.671658] Code: 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
> > [  158.671993] RSP: 002b:00007ffcc805f238 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
> > [  158.672171] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f850510f917
> > [  158.672352] RDX: 0000000000000000 RSI: 000000000198e9e8 RDI: 0000000000000009
> > [  158.672534] RBP: 0000000001933c00 R08: 0000000001935980 R09: 0000000000460e48
> > [  158.672716] R10: 0000000000000011 R11: 0000000000000246 R12: 0000000001933c30
> > [  158.672899] R13: 0000000000515fd8 R14: 000000000198e9d0 R15: 0000000000513690
> > [  158.673086]  </TASK>
> > [  158.673269] Modules linked in: bonding(E) tls(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) intel_rapl_msr(E) intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) kvm(E) irqbypass(E) rapl(E) intel_cstate(E) ast(E) intel_uncore(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E) pcspkr(E) mei_me(E) drm_kms_helper(E) i2c_i801(E) lpc_ich(E) mei(E) i2c_smbus(E) mxm_wmi(E) ioatdma(E) acpi_ipmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) acpi_power_meter(E) joydev(E) drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) sg(E) ahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libahci(E) crc32c_intel(E) ice(E)
> > [  158.673314]  polyval_clmulni(E) polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E) i2c_algo_bit(E) dca(E) wmi(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
> > [  158.675578] CR2: ffffa6510e5580c0

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Alasdair McWilliam]

Hi Maciej, Magnus,
 
Apologies for slow reply – bank holiday in the UK yesterday.
 
Just a quick update.

It’s quicker and easier for me to build a released version of code than it is to build a production kernel from a git tree due to build apparatus.
 
So, based on the suggestion to back out commit a62c50545b4d, I have taken the first step of identifying that said commit was included in 6.1.95. I’ve run both 6.1.95 and 6.1.94 through a build to test both. Some quick and dirty testing shows:

I can reproduce the issue on 6.1.95
I cannot (so far) reproduce the issue on 6.1.94
 
I’ve only tested the latter version 3-4 times so I’m going to keep throwing dead processes at it in different ways to just to be sure 6.1.94 is not affected. Then, to validate, I will grab the actual git tree at 6.1.95 and manually back out a62c50545b4d and re-test. But, this will take me a little longer.

Thanks
Alasdair

________________________________________
From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Sent: 23 August 2024 15:09
To: Magnus Karlsson
Cc: Alasdair McWilliam; xdp-newbies@vger.kernel.org
Subject: Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

On Fri, Aug 23, 2024 at 10:17:35AM +0200, Magnus Karlsson wrote:
> On Thu, 22 Aug 2024 at 18:25, Alasdair McWilliam
> <alasdair.mcwilliam@outlook.com> wrote:
> >
> > Hi,
> >
> > I've been testing apps that use XSK+ZC on ICE with newer builds of the 6.1 LTS kernel in preparation for some production upgrades, and I've started to notice some instability on newer versions. I can reproduce the issue easily in the lab.
> >
> > Config:
> > - Known good multi-threaded application (i.e. production grade)
> > - Uses eBPF and AF_XDP with zero copy to act as 'bump in wire' in network
> > - Xeon's with Intel E810-CQDA2 (firmware: 3.20 0x8000d83e 1.3146.0)
> > - Effectively a vanilla rebuild of 6.1 using configs from el-repo project
> >
> > Scenario:
> > - Noticing hard kernel faults when shutting down application
> > - Can happen if the process is shut down via systemctl stop
> > - Can even happen with a simple kill -9 command to the PID
> > - Appears in builds after 6.1.87
> >
> > Tested kernels:
> > - 6.1.84: process exits smoothly
> > - 6.1.87: process exits smoothly
> > - 6.1.97: BUG: unable to handle page fault for address
> > - 6.1.106: BUG: unable to handle page fault for address
> >
> > Kdump log is below [1] from 6.1.106 but does seem to be the same in the earlier version.
> >
> > Can anyone advise if this is a known issue?
> >
> > I don't have any builds between 6.1.87 and 6.1.97 but I can spend some time trying to pinpoint the exact version things start to go wrong in, if it would help anyone better equipped than me to debug!
>
> Hi Alasdair,
>
> It would be of great help if you could pinpoint the exact version for
> this breakage. Hopefully we could then find the commit in the ice
> driver that breaks your app, since there should be just a handful of
> commits in the ice driver for any stable release.

$ git log --oneline v6.1.87..v6.1.97 drivers/net/ethernet/intel/ice/
dd37b86999fd ice: Fix VSI list rule with ICE_SW_LKUP_LAST type
224b69e8751c ice: avoid IRQ collision to fix init failure on ACPI S3 resume
531d85b4fb66 ice: move RDMA init to ice_idc.c
a62c50545b4d ice: remove af_xdp_zc_qps bitmap
447a5433bd1e ice: remove null checks before devm_kfree() calls
a388961be5ed ice: Introduce new parameters in ice_sched_node
17ccdebe5ac7 ice: fix iteration of TLVs in Preserved Fields Area
07cbc5512023 ice: fix accounting if a VLAN already exists
5ef3a27c6142 ice: Interpret .set_channels() input differently
90cbd4c081bb ice: remove unnecessary duplicate checks for VF VSI ID
59161a21cae0 ice: pass VSI pointer into ice_vc_isvalid_q_id
6a6ebec40820 ice: tc: allow zero flags in parsing tc flower

can you revert a62c50545b4d and see if the issue persists?

>
> > Kind regards
> > Alasdair
> >
> > [1] kdump log
> >
> > [  158.666867] BUG: unable to handle page fault for address: ffffa6510e5580c0
> > [  158.666887] #PF: supervisor read access in kernel mode
> > [  158.666896] #PF: error_code(0x0000) - not-present page
> > [  158.666903] PGD 100000067 P4D 100000067 PUD 106dc4067 PMD 0
> > [  158.666914] Oops: 0000 [#1] PREEMPT SMP PTI
> > [  158.666922] CPU: 7 PID: 1808 Comm: tlndd.bin Kdump: loaded Tainted: G            E      6.1.106-1.X.el9.x86_64 #1
> > [  158.666940] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS 3.2 12/16/2019
> > [  158.666950] RIP: 0010:xp_free+0x11/0x80
> > [  158.666962] Code: 8b 04 d0 48 83 e0 fe 48 01 f0 c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 56 41 55 41 54 55 48 8d 6f 58 53 <48> 8b 47 58 48 39 c5 74 0d 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc
> > [  158.666985] RSP: 0018:ffffa65089e8b760 EFLAGS: 00010202
> > [  158.666993] RAX: ffff8fcf077c0000 RBX: 0000000000000001 RCX: 0000000000000000
> > [  158.667003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa6510e558068
> > [  158.667012] RBP: ffffa6510e5580c0 R08: fffff8c50415a108 R09: ffff8fc7cac60000
> > [  158.667022] R10: 0000000000000219 R11: ffffffffffffffff R12: 0000000000000fff
> > [  158.667031] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8fc7c139d340
> > [  158.667040] FS:  00007f8504996880(0000) GS:ffff8fcedfdc0000(0000) knlGS:0000000000000000
> > [  158.667050] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  158.667058] CR2: ffffa6510e5580c0 CR3: 00000001448e2002 CR4: 00000000001706e0
> > [  158.667068] Call Trace:
> > [  158.667075]  <TASK>
> > [  158.667082]  ? show_trace_log_lvl+0x1c4/0x2df
> > [  158.667094]  ? show_trace_log_lvl+0x1c4/0x2df
> > [  158.667103]  ? ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> > [  158.667157]  ? __die_body.cold+0x8/0xd
> > [  158.667166]  ? page_fault_oops+0xac/0x150
> > [  158.667176]  ? fixup_exception+0x22/0x340
> > [  158.667185]  ? exc_page_fault+0xb2/0x150
> > [  158.667195]  ? asm_exc_page_fault+0x22/0x30
> > [  158.667206]  ? xp_free+0x11/0x80
> > [  158.667215]  ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> > [  158.667250]  ice_clean_rx_ring+0x157/0x180 [ice]
> > [  158.667284]  ice_down+0x172/0x2b0 [ice]
> > [  158.667311]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> > [  158.667337]  ice_xdp_setup_prog+0xe3/0x3b0 [ice]
> > [  158.667364]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> > [  158.667391]  dev_xdp_install+0xc7/0x100
> > [  158.667402]  dev_xdp_attach+0x1e0/0x560
> > [  158.667412]  do_setlink+0x7a8/0xc10
> > [  158.667422]  ? __nla_validate_parse+0x12b/0x1b0
> > [  158.667436]  __rtnl_newlink+0x540/0x650
> > [  158.667446]  rtnl_newlink+0x44/0x70
> > [  158.667454]  rtnetlink_rcv_msg+0x15c/0x3d0
> > [  158.667477]  ? rtnl_calcit.isra.0+0x140/0x140
> > [  158.667485]  netlink_rcv_skb+0x51/0x100
> > [  158.667727]  netlink_unicast+0x246/0x360
> > [  158.667953]  netlink_sendmsg+0x24e/0x4b0
> > [  158.668173]  __sock_sendmsg+0x62/0x70
> > [  158.668389]  ____sys_sendmsg+0x247/0x2d0
> > [  158.668602]  ? copy_msghdr_from_user+0x6d/0xa0
> > [  158.668815]  ___sys_sendmsg+0x88/0xd0
> > [  158.669028]  ? __sk_destruct+0x156/0x230
> > [  158.669234]  ? kmem_cache_free+0x134/0x300
> > [  158.669437]  ? rcu_nocb_try_bypass+0x4a/0x440
> > [  158.669634]  ? __sk_destruct+0x156/0x230
> > [  158.669825]  ? _raw_spin_unlock_irqrestore+0x23/0x40
> > [  158.670010]  ? mod_objcg_state+0xc9/0x2f0
> > [  158.670186]  ? refill_obj_stock+0xae/0x160
> > [  158.670359]  ? rseq_get_rseq_cs.isra.0+0x16/0x220
> > [  158.670529]  ? rcu_nocb_try_bypass+0x4a/0x440
> > [  158.670696]  ? rseq_ip_fixup+0x72/0x1e0
> > [  158.670860]  __sys_sendmsg+0x59/0xa0
> > [  158.671021]  ? syscall_trace_enter.constprop.0+0x11e/0x190
> > [  158.671185]  do_syscall_64+0x35/0x80
> > [  158.671345]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> > [  158.671503] RIP: 0033:0x7f850510f917
> > [  158.671658] Code: 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
> > [  158.671993] RSP: 002b:00007ffcc805f238 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
> > [  158.672171] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f850510f917
> > [  158.672352] RDX: 0000000000000000 RSI: 000000000198e9e8 RDI: 0000000000000009
> > [  158.672534] RBP: 0000000001933c00 R08: 0000000001935980 R09: 0000000000460e48
> > [  158.672716] R10: 0000000000000011 R11: 0000000000000246 R12: 0000000001933c30
> > [  158.672899] R13: 0000000000515fd8 R14: 000000000198e9d0 R15: 0000000000513690
> > [  158.673086]  </TASK>
> > [  158.673269] Modules linked in: bonding(E) tls(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) intel_rapl_msr(E) intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) kvm(E) irqbypass(E) rapl(E) intel_cstate(E) ast(E) intel_uncore(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E) pcspkr(E) mei_me(E) drm_kms_helper(E) i2c_i801(E) lpc_ich(E) mei(E) i2c_smbus(E) mxm_wmi(E) ioatdma(E) acpi_ipmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) acpi_power_meter(E) joydev(E) drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) sg(E) ahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libahci(E) crc32c_intel(E) ice(E)
> > [  158.673314]  polyval_clmulni(E) polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E) i2c_algo_bit(E) dca(E) wmi(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
> > [  158.675578] CR2: ffffa6510e5580c0

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Alasdair McWilliam]

Good evening,

Looks like commit a62c50545b4d is the culprit.

I've produced a production-grade build of kernel 6.1.95 with commit a62c50545b4d backed out. Seems I can no longer trigger the fault. I can kill -9 the process while pushing 50Gbps / 14Mpps and the process is just restarted and resumes like it should.

I'm going to back out the same commit from 6.1.106 for our production builds and verify that fixes the issue there too.

Can you advise if this will be reversed in future commits, or if you have an alternate fix in the wings?

Thank you ! :-)
Alasdair


________________________________________
From: Alasdair McWilliam <alasdair.mcwilliam@outlook.com>
Sent: 27 August 2024 14:33
To: Maciej Fijalkowski; Magnus Karlsson
Cc: xdp-newbies@vger.kernel.org
Subject: Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

Hi Maciej, Magnus,

Apologies for slow reply – bank holiday in the UK yesterday.

Just a quick update – it’s quicker and easier for me to build a released version of code than it is to build a production kernel from a git tree due to build apparatus.

Based on the suggestion to back out commit a62c50545b4d, I have taken the first step of identifying that said commit was included in 6.1.95. So, I’ve run both 6.1.95 and 6.1.94 through a build to test both. Some quick and dirty testing shows:

  *   I can reproduce the issue on 6.1.95
  *   I cannot so far reproduce the issue on 6.1.94

I’ve only tested the latter version 3-4 times so I’m going to keep throwing dead processes at it in different ways to just to be sure 6.1.94 is not affected. Then, to validate, I will grab the actual git tree at 6.1.95 and manually back out a62c50545b4d and re-test. But, this will take me a little longer.

Thanks
Alasdair


From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Date: Friday, 23 August 2024 at 15:09
To: Magnus Karlsson <magnus.karlsson@gmail.com>
Cc: Alasdair McWilliam <alasdair.mcwilliam@outlook.com>, xdp-newbies@vger.kernel.org <xdp-newbies@vger.kernel.org>
Subject: Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?
On Fri, Aug 23, 2024 at 10:17:35AM +0200, Magnus Karlsson wrote:
> On Thu, 22 Aug 2024 at 18:25, Alasdair McWilliam
> <alasdair.mcwilliam@outlook.com> wrote:
> >
> > Hi,
> >
> > I've been testing apps that use XSK+ZC on ICE with newer builds of the 6.1 LTS kernel in preparation for some production upgrades, and I've started to notice some instability on newer versions. I can reproduce the issue easily in the lab.
> >
> > Config:
> > - Known good multi-threaded application (i.e. production grade)
> > - Uses eBPF and AF_XDP with zero copy to act as 'bump in wire' in network
> > - Xeon's with Intel E810-CQDA2 (firmware: 3.20 0x8000d83e 1.3146.0)
> > - Effectively a vanilla rebuild of 6.1 using configs from el-repo project
> >
> > Scenario:
> > - Noticing hard kernel faults when shutting down application
> > - Can happen if the process is shut down via systemctl stop
> > - Can even happen with a simple kill -9 command to the PID
> > - Appears in builds after 6.1.87
> >
> > Tested kernels:
> > - 6.1.84: process exits smoothly
> > - 6.1.87: process exits smoothly
> > - 6.1.97: BUG: unable to handle page fault for address
> > - 6.1.106: BUG: unable to handle page fault for address
> >
> > Kdump log is below [1] from 6.1.106 but does seem to be the same in the earlier version.
> >
> > Can anyone advise if this is a known issue?
> >
> > I don't have any builds between 6.1.87 and 6.1.97 but I can spend some time trying to pinpoint the exact version things start to go wrong in, if it would help anyone better equipped than me to debug!
>
> Hi Alasdair,
>
> It would be of great help if you could pinpoint the exact version for
> this breakage. Hopefully we could then find the commit in the ice
> driver that breaks your app, since there should be just a handful of
> commits in the ice driver for any stable release.

$ git log --oneline v6.1.87..v6.1.97 drivers/net/ethernet/intel/ice/
dd37b86999fd ice: Fix VSI list rule with ICE_SW_LKUP_LAST type
224b69e8751c ice: avoid IRQ collision to fix init failure on ACPI S3 resume
531d85b4fb66 ice: move RDMA init to ice_idc.c
a62c50545b4d ice: remove af_xdp_zc_qps bitmap
447a5433bd1e ice: remove null checks before devm_kfree() calls
a388961be5ed ice: Introduce new parameters in ice_sched_node
17ccdebe5ac7 ice: fix iteration of TLVs in Preserved Fields Area
07cbc5512023 ice: fix accounting if a VLAN already exists
5ef3a27c6142 ice: Interpret .set_channels() input differently
90cbd4c081bb ice: remove unnecessary duplicate checks for VF VSI ID
59161a21cae0 ice: pass VSI pointer into ice_vc_isvalid_q_id
6a6ebec40820 ice: tc: allow zero flags in parsing tc flower

can you revert a62c50545b4d and see if the issue persists?

>
> > Kind regards
> > Alasdair
> >
> > [1] kdump log
> >
> > [  158.666867] BUG: unable to handle page fault for address: ffffa6510e5580c0
> > [  158.666887] #PF: supervisor read access in kernel mode
> > [  158.666896] #PF: error_code(0x0000) - not-present page
> > [  158.666903] PGD 100000067 P4D 100000067 PUD 106dc4067 PMD 0
> > [  158.666914] Oops: 0000 [#1] PREEMPT SMP PTI
> > [  158.666922] CPU: 7 PID: 1808 Comm: tlndd.bin Kdump: loaded Tainted: G            E      6.1.106-1.X.el9.x86_64 #1
> > [  158.666940] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS 3.2 12/16/2019
> > [  158.666950] RIP: 0010:xp_free+0x11/0x80
> > [  158.666962] Code: 8b 04 d0 48 83 e0 fe 48 01 f0 c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 56 41 55 41 54 55 48 8d 6f 58 53 <48> 8b 47 58 48 39 c5 74 0d 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc
> > [  158.666985] RSP: 0018:ffffa65089e8b760 EFLAGS: 00010202
> > [  158.666993] RAX: ffff8fcf077c0000 RBX: 0000000000000001 RCX: 0000000000000000
> > [  158.667003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa6510e558068
> > [  158.667012] RBP: ffffa6510e5580c0 R08: fffff8c50415a108 R09: ffff8fc7cac60000
> > [  158.667022] R10: 0000000000000219 R11: ffffffffffffffff R12: 0000000000000fff
> > [  158.667031] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8fc7c139d340
> > [  158.667040] FS:  00007f8504996880(0000) GS:ffff8fcedfdc0000(0000) knlGS:0000000000000000
> > [  158.667050] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  158.667058] CR2: ffffa6510e5580c0 CR3: 00000001448e2002 CR4: 00000000001706e0
> > [  158.667068] Call Trace:
> > [  158.667075]  <TASK>
> > [  158.667082]  ? show_trace_log_lvl+0x1c4/0x2df
> > [  158.667094]  ? show_trace_log_lvl+0x1c4/0x2df
> > [  158.667103]  ? ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> > [  158.667157]  ? __die_body.cold+0x8/0xd
> > [  158.667166]  ? page_fault_oops+0xac/0x150
> > [  158.667176]  ? fixup_exception+0x22/0x340
> > [  158.667185]  ? exc_page_fault+0xb2/0x150
> > [  158.667195]  ? asm_exc_page_fault+0x22/0x30
> > [  158.667206]  ? xp_free+0x11/0x80
> > [  158.667215]  ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> > [  158.667250]  ice_clean_rx_ring+0x157/0x180 [ice]
> > [  158.667284]  ice_down+0x172/0x2b0 [ice]
> > [  158.667311]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> > [  158.667337]  ice_xdp_setup_prog+0xe3/0x3b0 [ice]
> > [  158.667364]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> > [  158.667391]  dev_xdp_install+0xc7/0x100
> > [  158.667402]  dev_xdp_attach+0x1e0/0x560
> > [  158.667412]  do_setlink+0x7a8/0xc10
> > [  158.667422]  ? __nla_validate_parse+0x12b/0x1b0
> > [  158.667436]  __rtnl_newlink+0x540/0x650
> > [  158.667446]  rtnl_newlink+0x44/0x70
> > [  158.667454]  rtnetlink_rcv_msg+0x15c/0x3d0
> > [  158.667477]  ? rtnl_calcit.isra.0+0x140/0x140
> > [  158.667485]  netlink_rcv_skb+0x51/0x100
> > [  158.667727]  netlink_unicast+0x246/0x360
> > [  158.667953]  netlink_sendmsg+0x24e/0x4b0
> > [  158.668173]  __sock_sendmsg+0x62/0x70
> > [  158.668389]  ____sys_sendmsg+0x247/0x2d0
> > [  158.668602]  ? copy_msghdr_from_user+0x6d/0xa0
> > [  158.668815]  ___sys_sendmsg+0x88/0xd0
> > [  158.669028]  ? __sk_destruct+0x156/0x230
> > [  158.669234]  ? kmem_cache_free+0x134/0x300
> > [  158.669437]  ? rcu_nocb_try_bypass+0x4a/0x440
> > [  158.669634]  ? __sk_destruct+0x156/0x230
> > [  158.669825]  ? _raw_spin_unlock_irqrestore+0x23/0x40
> > [  158.670010]  ? mod_objcg_state+0xc9/0x2f0
> > [  158.670186]  ? refill_obj_stock+0xae/0x160
> > [  158.670359]  ? rseq_get_rseq_cs.isra.0+0x16/0x220
> > [  158.670529]  ? rcu_nocb_try_bypass+0x4a/0x440
> > [  158.670696]  ? rseq_ip_fixup+0x72/0x1e0
> > [  158.670860]  __sys_sendmsg+0x59/0xa0
> > [  158.671021]  ? syscall_trace_enter.constprop.0+0x11e/0x190
> > [  158.671185]  do_syscall_64+0x35/0x80
> > [  158.671345]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> > [  158.671503] RIP: 0033:0x7f850510f917
> > [  158.671658] Code: 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
> > [  158.671993] RSP: 002b:00007ffcc805f238 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
> > [  158.672171] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f850510f917
> > [  158.672352] RDX: 0000000000000000 RSI: 000000000198e9e8 RDI: 0000000000000009
> > [  158.672534] RBP: 0000000001933c00 R08: 0000000001935980 R09: 0000000000460e48
> > [  158.672716] R10: 0000000000000011 R11: 0000000000000246 R12: 0000000001933c30
> > [  158.672899] R13: 0000000000515fd8 R14: 000000000198e9d0 R15: 0000000000513690
> > [  158.673086]  </TASK>
> > [  158.673269] Modules linked in: bonding(E) tls(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) intel_rapl_msr(E) intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) kvm(E) irqbypass(E) rapl(E) intel_cstate(E) ast(E) intel_uncore(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E) pcspkr(E) mei_me(E) drm_kms_helper(E) i2c_i801(E) lpc_ich(E) mei(E) i2c_smbus(E) mxm_wmi(E) ioatdma(E) acpi_ipmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) acpi_power_meter(E) joydev(E) drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) sg(E) ahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libahci(E) crc32c_intel(E) ice(E)
> > [  158.673314]  polyval_clmulni(E) polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E) i2c_algo_bit(E) dca(E) wmi(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
> > [  158.675578] CR2: ffffa6510e5580c0

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Maciej Fijalkowski]

On Mon, Sep 02, 2024 at 04:09:33PM +0000, Alasdair McWilliam wrote:
> Good evening,
> 
> Looks like commit a62c50545b4d is the culprit.
> 
> I've produced a production-grade build of kernel 6.1.95 with commit
> a62c50545b4d backed out. Seems I can no longer trigger the fault. I can
> kill -9 the process while pushing 50Gbps / 14Mpps and the process is
> just restarted and resumes like it should.
> 
> I'm going to back out the same commit from 6.1.106 for our production
> builds and verify that fixes the issue there too.
> 
> Can you advise if this will be reversed in future commits, or if you
> have an alternate fix in the wings?

We've been working recently on somewhat related issues and it looks like
not every commit from [0] has been backported.

$ git log --oneline v6.1.103..v6.1.104 drivers/net/ethernet/intel/ice/
5a80b682e3e1 ice: add missing WRITE_ONCE when clearing ice_rx_ring::xdp_prog
8782f0fcb19d ice: replace synchronize_rcu with synchronize_net
15115033f056 ice: don't busy wait for Rx queue disable in ice_qp_dis()
3dbc58774e58 ice: respect netif readiness in AF_XDP ZC related ndo's

can you apply the rest of it on top of 6.1.107 and see the result?

[0]: https://lore.kernel.org/all/20240729200716.681496-1-anthony.l.nguyen@intel.com/

> 
> Thank you ! :-)
> Alasdair
> 
> 
> ________________________________________
> From: Alasdair McWilliam <alasdair.mcwilliam@outlook.com>
> Sent: 27 August 2024 14:33
> To: Maciej Fijalkowski; Magnus Karlsson
> Cc: xdp-newbies@vger.kernel.org
> Subject: Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?
> 
> Hi Maciej, Magnus,
> 
> Apologies for slow reply – bank holiday in the UK yesterday.
> 
> Just a quick update – it’s quicker and easier for me to build a released version of code than it is to build a production kernel from a git tree due to build apparatus.
> 
> Based on the suggestion to back out commit a62c50545b4d, I have taken the first step of identifying that said commit was included in 6.1.95. So, I’ve run both 6.1.95 and 6.1.94 through a build to test both. Some quick and dirty testing shows:
> 
>   *   I can reproduce the issue on 6.1.95
>   *   I cannot so far reproduce the issue on 6.1.94
> 
> I’ve only tested the latter version 3-4 times so I’m going to keep throwing dead processes at it in different ways to just to be sure 6.1.94 is not affected. Then, to validate, I will grab the actual git tree at 6.1.95 and manually back out a62c50545b4d and re-test. But, this will take me a little longer.
> 
> Thanks
> Alasdair
> 
> 
> From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
> Date: Friday, 23 August 2024 at 15:09
> To: Magnus Karlsson <magnus.karlsson@gmail.com>
> Cc: Alasdair McWilliam <alasdair.mcwilliam@outlook.com>, xdp-newbies@vger.kernel.org <xdp-newbies@vger.kernel.org>
> Subject: Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?
> On Fri, Aug 23, 2024 at 10:17:35AM +0200, Magnus Karlsson wrote:
> > On Thu, 22 Aug 2024 at 18:25, Alasdair McWilliam
> > <alasdair.mcwilliam@outlook.com> wrote:
> > >
> > > Hi,
> > >
> > > I've been testing apps that use XSK+ZC on ICE with newer builds of the 6.1 LTS kernel in preparation for some production upgrades, and I've started to notice some instability on newer versions. I can reproduce the issue easily in the lab.
> > >
> > > Config:
> > > - Known good multi-threaded application (i.e. production grade)
> > > - Uses eBPF and AF_XDP with zero copy to act as 'bump in wire' in network
> > > - Xeon's with Intel E810-CQDA2 (firmware: 3.20 0x8000d83e 1.3146.0)
> > > - Effectively a vanilla rebuild of 6.1 using configs from el-repo project
> > >
> > > Scenario:
> > > - Noticing hard kernel faults when shutting down application
> > > - Can happen if the process is shut down via systemctl stop
> > > - Can even happen with a simple kill -9 command to the PID
> > > - Appears in builds after 6.1.87
> > >
> > > Tested kernels:
> > > - 6.1.84: process exits smoothly
> > > - 6.1.87: process exits smoothly
> > > - 6.1.97: BUG: unable to handle page fault for address
> > > - 6.1.106: BUG: unable to handle page fault for address
> > >
> > > Kdump log is below [1] from 6.1.106 but does seem to be the same in the earlier version.
> > >
> > > Can anyone advise if this is a known issue?
> > >
> > > I don't have any builds between 6.1.87 and 6.1.97 but I can spend some time trying to pinpoint the exact version things start to go wrong in, if it would help anyone better equipped than me to debug!
> >
> > Hi Alasdair,
> >
> > It would be of great help if you could pinpoint the exact version for
> > this breakage. Hopefully we could then find the commit in the ice
> > driver that breaks your app, since there should be just a handful of
> > commits in the ice driver for any stable release.
> 
> $ git log --oneline v6.1.87..v6.1.97 drivers/net/ethernet/intel/ice/
> dd37b86999fd ice: Fix VSI list rule with ICE_SW_LKUP_LAST type
> 224b69e8751c ice: avoid IRQ collision to fix init failure on ACPI S3 resume
> 531d85b4fb66 ice: move RDMA init to ice_idc.c
> a62c50545b4d ice: remove af_xdp_zc_qps bitmap
> 447a5433bd1e ice: remove null checks before devm_kfree() calls
> a388961be5ed ice: Introduce new parameters in ice_sched_node
> 17ccdebe5ac7 ice: fix iteration of TLVs in Preserved Fields Area
> 07cbc5512023 ice: fix accounting if a VLAN already exists
> 5ef3a27c6142 ice: Interpret .set_channels() input differently
> 90cbd4c081bb ice: remove unnecessary duplicate checks for VF VSI ID
> 59161a21cae0 ice: pass VSI pointer into ice_vc_isvalid_q_id
> 6a6ebec40820 ice: tc: allow zero flags in parsing tc flower
> 
> can you revert a62c50545b4d and see if the issue persists?
> 
> >
> > > Kind regards
> > > Alasdair
> > >
> > > [1] kdump log
> > >
> > > [  158.666867] BUG: unable to handle page fault for address: ffffa6510e5580c0
> > > [  158.666887] #PF: supervisor read access in kernel mode
> > > [  158.666896] #PF: error_code(0x0000) - not-present page
> > > [  158.666903] PGD 100000067 P4D 100000067 PUD 106dc4067 PMD 0
> > > [  158.666914] Oops: 0000 [#1] PREEMPT SMP PTI
> > > [  158.666922] CPU: 7 PID: 1808 Comm: tlndd.bin Kdump: loaded Tainted: G            E      6.1.106-1.X.el9.x86_64 #1
> > > [  158.666940] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS 3.2 12/16/2019
> > > [  158.666950] RIP: 0010:xp_free+0x11/0x80
> > > [  158.666962] Code: 8b 04 d0 48 83 e0 fe 48 01 f0 c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 56 41 55 41 54 55 48 8d 6f 58 53 <48> 8b 47 58 48 39 c5 74 0d 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc
> > > [  158.666985] RSP: 0018:ffffa65089e8b760 EFLAGS: 00010202
> > > [  158.666993] RAX: ffff8fcf077c0000 RBX: 0000000000000001 RCX: 0000000000000000
> > > [  158.667003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa6510e558068
> > > [  158.667012] RBP: ffffa6510e5580c0 R08: fffff8c50415a108 R09: ffff8fc7cac60000
> > > [  158.667022] R10: 0000000000000219 R11: ffffffffffffffff R12: 0000000000000fff
> > > [  158.667031] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8fc7c139d340
> > > [  158.667040] FS:  00007f8504996880(0000) GS:ffff8fcedfdc0000(0000) knlGS:0000000000000000
> > > [  158.667050] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [  158.667058] CR2: ffffa6510e5580c0 CR3: 00000001448e2002 CR4: 00000000001706e0
> > > [  158.667068] Call Trace:
> > > [  158.667075]  <TASK>
> > > [  158.667082]  ? show_trace_log_lvl+0x1c4/0x2df
> > > [  158.667094]  ? show_trace_log_lvl+0x1c4/0x2df
> > > [  158.667103]  ? ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> > > [  158.667157]  ? __die_body.cold+0x8/0xd
> > > [  158.667166]  ? page_fault_oops+0xac/0x150
> > > [  158.667176]  ? fixup_exception+0x22/0x340
> > > [  158.667185]  ? exc_page_fault+0xb2/0x150
> > > [  158.667195]  ? asm_exc_page_fault+0x22/0x30
> > > [  158.667206]  ? xp_free+0x11/0x80
> > > [  158.667215]  ice_xsk_clean_rx_ring+0x39/0x60 [ice]
> > > [  158.667250]  ice_clean_rx_ring+0x157/0x180 [ice]
> > > [  158.667284]  ice_down+0x172/0x2b0 [ice]
> > > [  158.667311]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> > > [  158.667337]  ice_xdp_setup_prog+0xe3/0x3b0 [ice]
> > > [  158.667364]  ? ice_xdp_setup_prog+0x3b0/0x3b0 [ice]
> > > [  158.667391]  dev_xdp_install+0xc7/0x100
> > > [  158.667402]  dev_xdp_attach+0x1e0/0x560
> > > [  158.667412]  do_setlink+0x7a8/0xc10
> > > [  158.667422]  ? __nla_validate_parse+0x12b/0x1b0
> > > [  158.667436]  __rtnl_newlink+0x540/0x650
> > > [  158.667446]  rtnl_newlink+0x44/0x70
> > > [  158.667454]  rtnetlink_rcv_msg+0x15c/0x3d0
> > > [  158.667477]  ? rtnl_calcit.isra.0+0x140/0x140
> > > [  158.667485]  netlink_rcv_skb+0x51/0x100
> > > [  158.667727]  netlink_unicast+0x246/0x360
> > > [  158.667953]  netlink_sendmsg+0x24e/0x4b0
> > > [  158.668173]  __sock_sendmsg+0x62/0x70
> > > [  158.668389]  ____sys_sendmsg+0x247/0x2d0
> > > [  158.668602]  ? copy_msghdr_from_user+0x6d/0xa0
> > > [  158.668815]  ___sys_sendmsg+0x88/0xd0
> > > [  158.669028]  ? __sk_destruct+0x156/0x230
> > > [  158.669234]  ? kmem_cache_free+0x134/0x300
> > > [  158.669437]  ? rcu_nocb_try_bypass+0x4a/0x440
> > > [  158.669634]  ? __sk_destruct+0x156/0x230
> > > [  158.669825]  ? _raw_spin_unlock_irqrestore+0x23/0x40
> > > [  158.670010]  ? mod_objcg_state+0xc9/0x2f0
> > > [  158.670186]  ? refill_obj_stock+0xae/0x160
> > > [  158.670359]  ? rseq_get_rseq_cs.isra.0+0x16/0x220
> > > [  158.670529]  ? rcu_nocb_try_bypass+0x4a/0x440
> > > [  158.670696]  ? rseq_ip_fixup+0x72/0x1e0
> > > [  158.670860]  __sys_sendmsg+0x59/0xa0
> > > [  158.671021]  ? syscall_trace_enter.constprop.0+0x11e/0x190
> > > [  158.671185]  do_syscall_64+0x35/0x80
> > > [  158.671345]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> > > [  158.671503] RIP: 0033:0x7f850510f917
> > > [  158.671658] Code: 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
> > > [  158.671993] RSP: 002b:00007ffcc805f238 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
> > > [  158.672171] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f850510f917
> > > [  158.672352] RDX: 0000000000000000 RSI: 000000000198e9e8 RDI: 0000000000000009
> > > [  158.672534] RBP: 0000000001933c00 R08: 0000000001935980 R09: 0000000000460e48
> > > [  158.672716] R10: 0000000000000011 R11: 0000000000000246 R12: 0000000001933c30
> > > [  158.672899] R13: 0000000000515fd8 R14: 000000000198e9d0 R15: 0000000000513690
> > > [  158.673086]  </TASK>
> > > [  158.673269] Modules linked in: bonding(E) tls(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) intel_rapl_msr(E) intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) kvm(E) irqbypass(E) rapl(E) intel_cstate(E) ast(E) intel_uncore(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E) pcspkr(E) mei_me(E) drm_kms_helper(E) i2c_i801(E) lpc_ich(E) mei(E) i2c_smbus(E) mxm_wmi(E) ioatdma(E) acpi_ipmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) acpi_power_meter(E) joydev(E) drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) sg(E) ahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libahci(E) crc32c_intel(E) ice(E)
> > > [  158.673314]  polyval_clmulni(E) polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E) i2c_algo_bit(E) dca(E) wmi(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
> > > [  158.675578] CR2: ffffa6510e5580c0

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Alasdair McWilliam]

On 04/09/2024 11:30, Maciej Fijalkowski wrote:
> On Mon, Sep 02, 2024 at 04:09:33PM +0000, Alasdair McWilliam wrote:
>> Good evening,
>>
>> Looks like commit a62c50545b4d is the culprit.
>>
>> I've produced a production-grade build of kernel 6.1.95 with commit
>> a62c50545b4d backed out. Seems I can no longer trigger the fault. I can
>> kill -9 the process while pushing 50Gbps / 14Mpps and the process is
>> just restarted and resumes like it should.
>>
>> I'm going to back out the same commit from 6.1.106 for our production
>> builds and verify that fixes the issue there too.
>>
>> Can you advise if this will be reversed in future commits, or if you
>> have an alternate fix in the wings?
> 
> We've been working recently on somewhat related issues and it looks like
> not every commit from [0] has been backported.
> 
> $ git log --oneline v6.1.103..v6.1.104 drivers/net/ethernet/intel/ice/
> 5a80b682e3e1 ice: add missing WRITE_ONCE when clearing ice_rx_ring::xdp_prog
> 8782f0fcb19d ice: replace synchronize_rcu with synchronize_net
> 15115033f056 ice: don't busy wait for Rx queue disable in ice_qp_dis()
> 3dbc58774e58 ice: respect netif readiness in AF_XDP ZC related ndo's
> 
> can you apply the rest of it on top of 6.1.107 and see the result?

The first one I've attempted doesn't apply cleanly to 6.1.107.

Eg: d59227179949 ("ice: modify error handling when setting XSK pool in
ndo_bpf"). The above looks to have been based on code from around 6.8 or
6.9 where the makeup of routines like ice_qp_ena() has changed. Looks
like this happened around a292ba981324 ("ice: make ice_vsi_cfg_txq()
static").

Should I try and apply a292ba981324 as well?

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Alasdair McWilliam]

On 05/09/2024 13:50, Alasdair McWilliam wrote:


>> We've been working recently on somewhat related issues and it looks like
>> not every commit from [0] has been backported.
>>
>> $ git log --oneline v6.1.103..v6.1.104 drivers/net/ethernet/intel/ice/
>> 5a80b682e3e1 ice: add missing WRITE_ONCE when clearing ice_rx_ring::xdp_prog
>> 8782f0fcb19d ice: replace synchronize_rcu with synchronize_net
>> 15115033f056 ice: don't busy wait for Rx queue disable in ice_qp_dis()
>> 3dbc58774e58 ice: respect netif readiness in AF_XDP ZC related ndo's
>>
>> can you apply the rest of it on top of 6.1.107 and see the result?


> 
> The first one I've attempted doesn't apply cleanly to 6.1.107.
> 
> Eg: d59227179949 ("ice: modify error handling when setting XSK pool in
> ndo_bpf"). The above looks to have been based on code from around 6.8 or
> 6.9 where the makeup of routines like ice_qp_ena() has changed. Looks
> like this happened around a292ba981324 ("ice: make ice_vsi_cfg_txq()
> static").
> 
> Should I try and apply a292ba981324 as well?
> 

Good evening,

I just wondered if there was perhaps any further feedback on the above.

Thanks in advance!
Alasdair.

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Thorsten Leemhuis]

[CCing a few people that were involved in mainlining the culprit
(8adbf5a42341f6e ("ice: remove af_xdp_zc_qps bitmap") in case they want
to provide advice]

On 13.09.24 17:54, Alasdair McWilliam wrote:
> On 05/09/2024 13:50, Alasdair McWilliam wrote:
> 
>>> We've been working recently on somewhat related issues and it looks like
>>> not every commit from [0] has been backported.
>>>
>>> $ git log --oneline v6.1.103..v6.1.104 drivers/net/ethernet/intel/ice/
>>> 5a80b682e3e1 ice: add missing WRITE_ONCE when clearing ice_rx_ring::xdp_prog
>>> 8782f0fcb19d ice: replace synchronize_rcu with synchronize_net
>>> 15115033f056 ice: don't busy wait for Rx queue disable in ice_qp_dis()
>>> 3dbc58774e58 ice: respect netif readiness in AF_XDP ZC related ndo's
>>>
>>> can you apply the rest of it on top of 6.1.107 and see the result?
>
>> The first one I've attempted doesn't apply cleanly to 6.1.107.
>>
>> Eg: d59227179949 ("ice: modify error handling when setting XSK pool in
>> ndo_bpf"). The above looks to have been based on code from around 6.8 or
>> 6.9 where the makeup of routines like ice_qp_ena() has changed. Looks
>> like this happened around a292ba981324 ("ice: make ice_vsi_cfg_txq()
>> static").
>>
>> Should I try and apply a292ba981324 as well?
> 
> I just wondered if there was perhaps any further feedback on the above.

Hmmm. No reply afaics -- but that's how it is sometimes with
stable/longterm kernels series, as mainline developers are not required
to participate in their development.

Still it would be good to fix the problem. So unless the developers come
up with plan, it might be best to just revert a62c50545b4d in 6.1.y;
guess asking Greg to do so might be best way ahead unless some solutions
comes into sight within a few days.

Ciao, Thorsten

#regzbot poke

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Alasdair McWilliam]

Good day,

On 27/09/2024 12:32, Thorsten Leemhuis wrote:

> [CCing a few people that were involved in mainlining the culprit
> (8adbf5a42341f6e ("ice: remove af_xdp_zc_qps bitmap") in case they want
> to provide advice]
> 
> On 13.09.24 17:54, Alasdair McWilliam wrote:
>> On 05/09/2024 13:50, Alasdair McWilliam wrote:
>>
>>>> We've been working recently on somewhat related issues and it looks like
>>>> not every commit from [0] has been backported.
>>>>
>>>> $ git log --oneline v6.1.103..v6.1.104 drivers/net/ethernet/intel/ice/
>>>> 5a80b682e3e1 ice: add missing WRITE_ONCE when clearing ice_rx_ring::xdp_prog
>>>> 8782f0fcb19d ice: replace synchronize_rcu with synchronize_net
>>>> 15115033f056 ice: don't busy wait for Rx queue disable in ice_qp_dis()
>>>> 3dbc58774e58 ice: respect netif readiness in AF_XDP ZC related ndo's
>>>>
>>>> can you apply the rest of it on top of 6.1.107 and see the result?
>>
>>> The first one I've attempted doesn't apply cleanly to 6.1.107.
>>>
>>> Eg: d59227179949 ("ice: modify error handling when setting XSK pool in
>>> ndo_bpf"). The above looks to have been based on code from around 6.8 or
>>> 6.9 where the makeup of routines like ice_qp_ena() has changed. Looks
>>> like this happened around a292ba981324 ("ice: make ice_vsi_cfg_txq()
>>> static").
>>>
>>> Should I try and apply a292ba981324 as well?
>>
>> I just wondered if there was perhaps any further feedback on the above.
> 
> Hmmm. No reply afaics -- but that's how it is sometimes with
> stable/longterm kernels series, as mainline developers are not required
> to participate in their development.
> 
> Still it would be good to fix the problem. So unless the developers come
> up with plan, it might be best to just revert a62c50545b4d in 6.1.y;
> guess asking Greg to do so might be best way ahead unless some solutions
> comes into sight within a few days.
>

It's been a minute since I've looked at this due to other commitments
but accidentally bumped into the fault again when testing the latest 6.6
LTS for a new feature of our software. (I forgot to revert the commit
for "ice: remove af_xdp_zc_qps bitmap" in our build system.)

This led me to wonder about the current version, and can trigger the
same crash on 6.11.5 [3].

Reverting "ice: remove af_xdp_zc_qps bitmap" [1] in the current mainline
is a little more complicated as commit ebc33a3f8d0a ("ice: improve
updating ice_{t,r}x_ring::xsk_pool") also changes things a little so the
reversion doesn't work cleanly.

I have tweaked everything a little the below patch [2] applies cleanly
to 6.11.5 and 6.12-rc5 and seems to fix the fault.

Thought I'd bubble this up as it's definitely still an issue in the
mainline kernel as of now.

Thanks
Alasdair

[1] Commit adbf5a42341f6ea038d3626cd4437d9f0ad0b2dd

[2]
https://github.com/OpenSource-THG/kernel-patches/tree/main/2024-11-ice-xskzc-page-fault

[3] 6.11.5 ooops

[  565.069120] BUG: unable to handle page fault for address:
ffffa566707380c4
[  565.069144] #PF: supervisor read access in kernel mode
[  565.069155] #PF: error_code(0x0000) - not-present page
[  565.069167] PGD 100000067 P4D 100000067 PUD 20ef17067 PMD 0
[  565.069183] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
[  565.069195] CPU: 7 UID: 0 PID: 6967 Comm: tlndd.bin Kdump: loaded
Tainted: G            E
6.11.5-1.thg.836e8867d7.241031.135507.el9.x86_64 #1
[  565.069220] Tainted: [E]=UNSIGNED_MODULE
[  565.069228] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS
3.2 12/16/2019
[  565.069241] RIP: 0010:ice_xsk_clean_rx_ring+0x37/0x110 [ice]
[  565.069338] Code: 55 53 48 83 ec 08 44 0f b7 af a4 00 00 00 0f b7 af
a2 00 00 00 66 41 39 ed 74 33 48 89 fb 48 8b 4b 38 41 0f b7 c5 4c 8b 34
c1 <41> f6 46 34 01 75 30 4c 89 f7 41 83 c5 01 e8 f6 0c 7e ce 31 c0 66
[  565.069365] RSP: 0018:ffffa5660f8f36d8 EFLAGS: 00010293
[  565.069375] RAX: 0000000000000000 RBX: ffff8bb105d38600 RCX:
ffff8bb184930000
[  565.069387] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff8bb105d38600
[  565.069400] RBP: 00000000000007ff R08: 000000000000050b R09:
0000000000000000
[  565.069411] R10: ffff8bb10f910000 R11: 0000000000000020 R12:
0000000000000004
[  565.069422] R13: 0000000000000000 R14: ffffa56670738090 R15:
ffff8bb1116b5740
[  565.069434] FS:  00007f677a5d1dc0(0000) GS:ffff8bb85fd80000(0000)
knlGS:0000000000000000
[  565.069447] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  565.069457] CR2: ffffa566707380c4 CR3: 0000000120164005 CR4:
00000000001706f0
[  565.069470] Call Trace:
[  565.069480]  <TASK>
[  565.069489]  ? __die+0x20/0x70
[  565.069504]  ? page_fault_oops+0x80/0x150
[  565.069517]  ? exc_page_fault+0xcd/0x170
[  565.069531]  ? asm_exc_page_fault+0x22/0x30
[  565.069546]  ? ice_xsk_clean_rx_ring+0x37/0x110 [ice]
[  565.069598]  ice_clean_rx_ring+0x16e/0x190 [ice]
[  565.069650]  ice_down+0x2f8/0x3c0 [ice]
[  565.069692]  ice_xdp_setup_prog+0x193/0x460 [ice]
[  565.069734]  ice_xdp+0x7a/0xb0 [ice]
[  565.069774]  ? __pfx_ice_xdp+0x10/0x10 [ice]
[  565.069813]  dev_xdp_install+0xc7/0x100
[  565.069829]  dev_xdp_attach+0x205/0x5d0
[  565.069841]  do_setlink+0x7d3/0xc20
[  565.069853]  ? dequeue_skb+0x80/0x4f0
[  565.069866]  ? __nla_validate_parse+0x125/0x1d0
[  565.069880]  __rtnl_newlink+0x4f7/0x630
[  565.069892]  ? __kmalloc_cache_noprof+0x225/0x2b0
[  565.069905]  rtnl_newlink+0x44/0x70
[  565.069915]  rtnetlink_rcv_msg+0x15c/0x410
[  565.069928]  ? avc_has_perm_noaudit+0x67/0xf0
[  565.069943]  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
[  565.069956]  netlink_rcv_skb+0x57/0x100
[  565.069969]  netlink_unicast+0x246/0x370
[  565.069980]  netlink_sendmsg+0x1f6/0x430
[  565.069991]  ____sys_sendmsg+0x3be/0x3f0
[  565.070003]  ? import_iovec+0x16/0x20
[  565.070015]  ? copy_msghdr_from_user+0x6d/0xa0
[  565.070028]  ___sys_sendmsg+0x88/0xd0
[  565.070038]  ? __memcg_slab_free_hook+0xd5/0x120
[  565.070050]  ? __inode_wait_for_writeback+0x7d/0xf0
[  565.070065]  ? mod_objcg_state+0xc9/0x2f0
[  565.070076]  __sys_sendmsg+0x59/0xa0
[  565.070086]  ? syscall_trace_enter+0xfb/0x190
[  565.070098]  do_syscall_64+0x60/0x180
[  565.070111]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  565.070126] RIP: 0033:0x7f677ab0f94d
[  565.070136] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 0a 67
f7 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 5e 67 f7 ff 48
[  565.070164] RSP: 002b:00007ffd1e4f7a60 EFLAGS: 00000293 ORIG_RAX:
000000000000002e
[  565.070178] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
00007f677ab0f94d
[  565.070191] RDX: 0000000000000000 RSI: 000000001d698848 RDI:
000000000000000a
[  565.070203] RBP: 000000001d5350e0 R08: 0000000000000000 R09:
0000000000465f98
[  565.070215] R10: 0000000000200000 R11: 0000000000000293 R12:
000000001d535110
[  565.070227] R13: 000000000051d798 R14: 000000001d698830 R15:
000000001d5384b0
[  565.070240]  </TASK>
[  565.070248] Modules linked in: bonding(E) tls(E) nft_fib_inet(E)
nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E)
nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E)
nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_
defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E)
libcrc32c(E) nfnetlink(E) vfat(E) fat(E) intel_rapl_msr(E)
intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E)
intel_powerclamp(E) coretemp(E) kvm_intel(E) ipmi_ssif(
E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) rapl(E)
intel_cstate(E) intel_uncore(E) ast(E) i2c_i801(E) pcspkr(E) mei_me(E)
drm_shmem_helper(E) mxm_wmi(E) drm_kms_helper(E) i2c_mux(E) mei(E)
i2c_smbus(E) lpc_ich(E) ioat
dma(E) acpi_power_meter(E) ipmi_si(E) acpi_ipmi(E) joydev(E)
ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) drm(E) fuse(E) ext4(E)
mbcache(E) jbd2(E) sd_mod(E) sg(E) ice(E) ahci(E) crct10dif_pclmul(E)
crc32_pclmul(E) crc32c_intel(E) lib
ahci(E) polyval_clmulni(E) igb(E) polyval_generic(E) libata(E)
ghash_clmulni_intel(E)
[  565.070304]  i2c_algo_bit(E) dca(E) libie(E) wmi(E) dm_mirror(E)
dm_region_hash(E) dm_log(E) dm_mod(E)
[  565.071430] CR2: ffffa566707380c4

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Larysa Zaremba]

On Fri, Nov 01, 2024 at 12:37:41PM +0000, Alasdair McWilliam wrote:
> Good day,
> 
> On 27/09/2024 12:32, Thorsten Leemhuis wrote:
> 
> > [CCing a few people that were involved in mainlining the culprit
> > (8adbf5a42341f6e ("ice: remove af_xdp_zc_qps bitmap") in case they want
> > to provide advice]
> > 
> > On 13.09.24 17:54, Alasdair McWilliam wrote:
> >> On 05/09/2024 13:50, Alasdair McWilliam wrote:
> >>
> >>>> We've been working recently on somewhat related issues and it looks like
> >>>> not every commit from [0] has been backported.
> >>>>
> >>>> $ git log --oneline v6.1.103..v6.1.104 drivers/net/ethernet/intel/ice/
> >>>> 5a80b682e3e1 ice: add missing WRITE_ONCE when clearing ice_rx_ring::xdp_prog
> >>>> 8782f0fcb19d ice: replace synchronize_rcu with synchronize_net
> >>>> 15115033f056 ice: don't busy wait for Rx queue disable in ice_qp_dis()
> >>>> 3dbc58774e58 ice: respect netif readiness in AF_XDP ZC related ndo's
> >>>>
> >>>> can you apply the rest of it on top of 6.1.107 and see the result?
> >>
> >>> The first one I've attempted doesn't apply cleanly to 6.1.107.
> >>>
> >>> Eg: d59227179949 ("ice: modify error handling when setting XSK pool in
> >>> ndo_bpf"). The above looks to have been based on code from around 6.8 or
> >>> 6.9 where the makeup of routines like ice_qp_ena() has changed. Looks
> >>> like this happened around a292ba981324 ("ice: make ice_vsi_cfg_txq()
> >>> static").
> >>>
> >>> Should I try and apply a292ba981324 as well?
> >>
> >> I just wondered if there was perhaps any further feedback on the above.
> > 
> > Hmmm. No reply afaics -- but that's how it is sometimes with
> > stable/longterm kernels series, as mainline developers are not required
> > to participate in their development.
> > 
> > Still it would be good to fix the problem. So unless the developers come
> > up with plan, it might be best to just revert a62c50545b4d in 6.1.y;
> > guess asking Greg to do so might be best way ahead unless some solutions
> > comes into sight within a few days.
> >
> 
> It's been a minute since I've looked at this due to other commitments
> but accidentally bumped into the fault again when testing the latest 6.6
> LTS for a new feature of our software. (I forgot to revert the commit
> for "ice: remove af_xdp_zc_qps bitmap" in our build system.)
> 
> This led me to wonder about the current version, and can trigger the
> same crash on 6.11.5 [3].
> 
> Reverting "ice: remove af_xdp_zc_qps bitmap" [1] in the current mainline
> is a little more complicated as commit ebc33a3f8d0a ("ice: improve
> updating ice_{t,r}x_ring::xsk_pool") also changes things a little so the
> reversion doesn't work cleanly.
> 
> I have tweaked everything a little the below patch [2] applies cleanly
> to 6.11.5 and 6.12-rc5 and seems to fix the fault.
> 
> Thought I'd bubble this up as it's definitely still an issue in the
> mainline kernel as of now.
> 
> Thanks
> Alasdair
> 

Hello,
Could you please share the reproduction steps? I will look into this.

Larysa

> [1] Commit adbf5a42341f6ea038d3626cd4437d9f0ad0b2dd
> 
> [2]
> https://github.com/OpenSource-THG/kernel-patches/tree/main/2024-11-ice-xskzc-page-fault
> 
> [3] 6.11.5 ooops
> 
> [  565.069120] BUG: unable to handle page fault for address:
> ffffa566707380c4
> [  565.069144] #PF: supervisor read access in kernel mode
> [  565.069155] #PF: error_code(0x0000) - not-present page
> [  565.069167] PGD 100000067 P4D 100000067 PUD 20ef17067 PMD 0
> [  565.069183] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
> [  565.069195] CPU: 7 UID: 0 PID: 6967 Comm: tlndd.bin Kdump: loaded
> Tainted: G            E
> 6.11.5-1.thg.836e8867d7.241031.135507.el9.x86_64 #1
> [  565.069220] Tainted: [E]=UNSIGNED_MODULE
> [  565.069228] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS
> 3.2 12/16/2019
> [  565.069241] RIP: 0010:ice_xsk_clean_rx_ring+0x37/0x110 [ice]
> [  565.069338] Code: 55 53 48 83 ec 08 44 0f b7 af a4 00 00 00 0f b7 af
> a2 00 00 00 66 41 39 ed 74 33 48 89 fb 48 8b 4b 38 41 0f b7 c5 4c 8b 34
> c1 <41> f6 46 34 01 75 30 4c 89 f7 41 83 c5 01 e8 f6 0c 7e ce 31 c0 66
> [  565.069365] RSP: 0018:ffffa5660f8f36d8 EFLAGS: 00010293
> [  565.069375] RAX: 0000000000000000 RBX: ffff8bb105d38600 RCX:
> ffff8bb184930000
> [  565.069387] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffff8bb105d38600
> [  565.069400] RBP: 00000000000007ff R08: 000000000000050b R09:
> 0000000000000000
> [  565.069411] R10: ffff8bb10f910000 R11: 0000000000000020 R12:
> 0000000000000004
> [  565.069422] R13: 0000000000000000 R14: ffffa56670738090 R15:
> ffff8bb1116b5740
> [  565.069434] FS:  00007f677a5d1dc0(0000) GS:ffff8bb85fd80000(0000)
> knlGS:0000000000000000
> [  565.069447] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  565.069457] CR2: ffffa566707380c4 CR3: 0000000120164005 CR4:
> 00000000001706f0
> [  565.069470] Call Trace:
> [  565.069480]  <TASK>
> [  565.069489]  ? __die+0x20/0x70
> [  565.069504]  ? page_fault_oops+0x80/0x150
> [  565.069517]  ? exc_page_fault+0xcd/0x170
> [  565.069531]  ? asm_exc_page_fault+0x22/0x30
> [  565.069546]  ? ice_xsk_clean_rx_ring+0x37/0x110 [ice]
> [  565.069598]  ice_clean_rx_ring+0x16e/0x190 [ice]
> [  565.069650]  ice_down+0x2f8/0x3c0 [ice]
> [  565.069692]  ice_xdp_setup_prog+0x193/0x460 [ice]
> [  565.069734]  ice_xdp+0x7a/0xb0 [ice]
> [  565.069774]  ? __pfx_ice_xdp+0x10/0x10 [ice]
> [  565.069813]  dev_xdp_install+0xc7/0x100
> [  565.069829]  dev_xdp_attach+0x205/0x5d0
> [  565.069841]  do_setlink+0x7d3/0xc20
> [  565.069853]  ? dequeue_skb+0x80/0x4f0
> [  565.069866]  ? __nla_validate_parse+0x125/0x1d0
> [  565.069880]  __rtnl_newlink+0x4f7/0x630
> [  565.069892]  ? __kmalloc_cache_noprof+0x225/0x2b0
> [  565.069905]  rtnl_newlink+0x44/0x70
> [  565.069915]  rtnetlink_rcv_msg+0x15c/0x410
> [  565.069928]  ? avc_has_perm_noaudit+0x67/0xf0
> [  565.069943]  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> [  565.069956]  netlink_rcv_skb+0x57/0x100
> [  565.069969]  netlink_unicast+0x246/0x370
> [  565.069980]  netlink_sendmsg+0x1f6/0x430
> [  565.069991]  ____sys_sendmsg+0x3be/0x3f0
> [  565.070003]  ? import_iovec+0x16/0x20
> [  565.070015]  ? copy_msghdr_from_user+0x6d/0xa0
> [  565.070028]  ___sys_sendmsg+0x88/0xd0
> [  565.070038]  ? __memcg_slab_free_hook+0xd5/0x120
> [  565.070050]  ? __inode_wait_for_writeback+0x7d/0xf0
> [  565.070065]  ? mod_objcg_state+0xc9/0x2f0
> [  565.070076]  __sys_sendmsg+0x59/0xa0
> [  565.070086]  ? syscall_trace_enter+0xfb/0x190
> [  565.070098]  do_syscall_64+0x60/0x180
> [  565.070111]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [  565.070126] RIP: 0033:0x7f677ab0f94d
> [  565.070136] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 0a 67
> f7 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f
> 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 5e 67 f7 ff 48
> [  565.070164] RSP: 002b:00007ffd1e4f7a60 EFLAGS: 00000293 ORIG_RAX:
> 000000000000002e
> [  565.070178] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> 00007f677ab0f94d
> [  565.070191] RDX: 0000000000000000 RSI: 000000001d698848 RDI:
> 000000000000000a
> [  565.070203] RBP: 000000001d5350e0 R08: 0000000000000000 R09:
> 0000000000465f98
> [  565.070215] R10: 0000000000200000 R11: 0000000000000293 R12:
> 000000001d535110
> [  565.070227] R13: 000000000051d798 R14: 000000001d698830 R15:
> 000000001d5384b0
> [  565.070240]  </TASK>
> [  565.070248] Modules linked in: bonding(E) tls(E) nft_fib_inet(E)
> nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E)
> nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E)
> nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_
> defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E)
> libcrc32c(E) nfnetlink(E) vfat(E) fat(E) intel_rapl_msr(E)
> intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E)
> intel_powerclamp(E) coretemp(E) kvm_intel(E) ipmi_ssif(
> E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) rapl(E)
> intel_cstate(E) intel_uncore(E) ast(E) i2c_i801(E) pcspkr(E) mei_me(E)
> drm_shmem_helper(E) mxm_wmi(E) drm_kms_helper(E) i2c_mux(E) mei(E)
> i2c_smbus(E) lpc_ich(E) ioat
> dma(E) acpi_power_meter(E) ipmi_si(E) acpi_ipmi(E) joydev(E)
> ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) drm(E) fuse(E) ext4(E)
> mbcache(E) jbd2(E) sd_mod(E) sg(E) ice(E) ahci(E) crct10dif_pclmul(E)
> crc32_pclmul(E) crc32c_intel(E) lib
> ahci(E) polyval_clmulni(E) igb(E) polyval_generic(E) libata(E)
> ghash_clmulni_intel(E)
> [  565.070304]  i2c_algo_bit(E) dca(E) libie(E) wmi(E) dm_mirror(E)
> dm_region_hash(E) dm_log(E) dm_mod(E)
> [  565.071430] CR2: ffffa566707380c4
> 

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Alasdair McWilliam]

On 04/11/2024 07:11, Larysa Zaremba wrote:

>> It's been a minute since I've looked at this due to other commitments
>> but accidentally bumped into the fault again when testing the latest 6.6
>> LTS for a new feature of our software. (I forgot to revert the commit
>> for "ice: remove af_xdp_zc_qps bitmap" in our build system.)
>>
>> This led me to wonder about the current version, and can trigger the
>> same crash on 6.11.5 [3].
>>
>> Reverting "ice: remove af_xdp_zc_qps bitmap" [1] in the current mainline
>> is a little more complicated as commit ebc33a3f8d0a ("ice: improve
>> updating ice_{t,r}x_ring::xsk_pool") also changes things a little so the
>> reversion doesn't work cleanly.
>>
>> I have tweaked everything a little the below patch [2] applies cleanly
>> to 6.11.5 and 6.12-rc5 and seems to fix the fault.
>>
>> Thought I'd bubble this up as it's definitely still an issue in the
>> mainline kernel as of now.
>>
>> Thanks
>> Alasdair
>>
> 
> Hello,
> Could you please share the reproduction steps? I will look into this.

Hello,

I should probably have provided better steps to reproduce - apologies.

Our stack uses AF_XDP in zero copy mode with shared UMEM between XSK
sockets.

To isolate other bugs in the past we've used a modified xdpsock app
based on code previously in kernel samples. The original sample has
since been taken out the kernel repo, but we maintained the modified
version in our public repos here [1].

There's lots in the readme but suffice to say if you run the build.sh
with bash, it will compile xdpsock_multi user-space app and accompanying
xdpsock_multi.bpf eBPF app. You'll also need to necessary dependencies
libxdp/libbpf et al.

I can reproduce the issue with this app using 8 channels. It can fault
in two ways (step C or D) below.

Terminal 1:

A# ethtool -L <nic> combined 8
B# ./xdpsock_multi --l2fwd --interface ice1_1 --zero-copy --channels 8

Terminal 2:

C# kill -9 $(pidof xdpsock_multi)
D# ip link set dev <nic> xdp off

Sometimes the act of killing the process (step C) causes a kernel crash [2].

Other times, it may survive, leaving an orphaned XDP program attached to
the NIC. Unloading this manually (step D) causes a kernel crash [3].

Stack traces are actually different so hence I've provided both.

Affects:
6.1.x
6.6.x
6.11.x

Hardware is E810-CQDA2
Firmware is 3.20 0x8000d83e 1.3146.0

Let me know if you need anything further.

Thanks!
Alasdair


[1] https://github.com/OpenSource-THG/xdpsock-sample

[2] Kernel crash triggered by step C

[  220.921136] BUG: unable to handle page fault for address:
ffffa3eee1637f14
[  220.921175] #PF: supervisor write access in kernel mode
[  220.921196] #PF: error_code(0x0002) - not-present page
[  220.921217] PGD 100000067 P4D 100000067 PUD 100238067 PMD 0
[  220.921244] Oops: Oops: 0002 [#1] PREEMPT SMP PTI
[  220.921267] CPU: 5 UID: 0 PID: 0 Comm: swapper/5 Kdump: loaded
Tainted: G            E
6.11.5-1.thg.836e8867d7.241031.135507.el9.x86_64 #1
[  220.921315] Tainted: [E]=UNSIGNED_MODULE
[  220.921331] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS
3.2 12/16/2019
[  220.921357] RIP: 0010:ice_clean_rx_irq_zc+0xde/0x7d0 [ice]
[  220.921489] Code: 0f 84 d0 01 00 00 44 3b 7c 24 08 0f 84 a1 02 00 00
48 8b 53 38 41 0f b7 4d 04 4c 8b 24 c2 89 c8 81 e1 ff 3f 00 00 66 25 ff
3f <41> c7 44 24 34 00 00 00 00 49 8b 74 24 18 48 8d 96 00 01 00 00 49
[  220.921518] RSP: 0018:ffffa3eec64d0d88 EFLAGS: 00010206
[  220.921529] RAX: 000000000000014d RBX: ffff89bbc2aa2a00 RCX:
000000000000014d
[  220.921542] RDX: ffff89b408830000 RSI: 0000000000000040 RDI:
ffff89bbc2aa2a00
[  220.921554] RBP: 0000000000000000 R08: 0000000000000000 R09:
ffff89b407655000
[  220.921566] R10: 0000ffffffffffff R11: ffffa3eec64d0ff8 R12:
ffffa3eee1637ee0
[  220.921578] R13: ffff89b414710000 R14: ffff89bbc7919500 R15:
0000000000000000
[  220.921591] FS:  0000000000000000(0000) GS:ffff89bb5fc80000(0000)
knlGS:0000000000000000
[  220.921605] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  220.921616] CR2: ffffa3eee1637f14 CR3: 00000001d9820006 CR4:
00000000001706f0
[  220.921628] Call Trace:
[  220.921639]  <IRQ>
[  220.921647]  ? __die+0x20/0x70
[  220.921663]  ? page_fault_oops+0x80/0x150
[  220.921676]  ? exc_page_fault+0xcd/0x170
[  220.921690]  ? asm_exc_page_fault+0x22/0x30
[  220.921707]  ? ice_clean_rx_irq_zc+0xde/0x7d0 [ice]
[  220.921759]  ? ice_clean_tx_irq+0x166/0x3c0 [ice]
[  220.921808]  ice_napi_poll+0xb2/0x2a0 [ice]
[  220.921858]  __napi_poll+0x2c/0x1b0
[  220.921870]  net_rx_action+0x30d/0x3e0
[  220.921881]  ? __raise_softirq_irqoff+0x18/0x80
[  220.921896]  ? __napi_schedule+0xa6/0xc0
[  220.921907]  ? ice_msix_clean_rings+0x4f/0x60 [ice]
[  220.921959]  handle_softirqs+0xf0/0x2e0
[  220.921972]  __irq_exit_rcu+0x80/0xe0
[  220.921983]  common_interrupt+0xb7/0xd0
[  220.921995]  </IRQ>
[  220.922001]  <TASK>
[  220.922008]  asm_common_interrupt+0x22/0x40
[  220.922022] RIP: 0010:cpuidle_enter_state+0xc8/0x420
[  220.922034] Code: 0e b6 3e ff e8 09 ee ff ff 8b 55 04 49 89 c5 0f 1f
44 00 00 31 ff e8 97 69 3d ff 45 84 ff 0f 85 38 02 00 00 fb 0f 1f 44 00
00 <45> 85 f6 0f 88 6a 01 00 00 49 63 d6 4c 2b 2c 24 48 8d 04 52 48 8d
[  220.922061] RSP: 0018:ffffa3eec4377e78 EFLAGS: 00000246
[  220.922072] RAX: ffff89bb5fc80000 RBX: 0000000000000004 RCX:
000000000000001f
[  220.922085] RDX: 0000000000000005 RSI: ffffffffb255a8a3 RDI:
ffffffffb2533173
[  220.922098] RBP: ffff89bb5fcc0cc8 R08: 000000336fecb8ce R09:
0000000000000018
[  220.922109] R10: 000000000000453f R11: ffff89bb5fcb47e4 R12:
ffffffffb32bdce0
[  220.922121] R13: 000000336fecb8ce R14: 0000000000000004 R15:
0000000000000000
[  220.922135]  ? cpuidle_enter_state+0xb9/0x420
[  220.922147]  cpuidle_enter+0x29/0x40
[  220.922161]  cpuidle_idle_call+0x100/0x170
[  220.922175]  do_idle+0x7d/0xd0
[  220.922185]  cpu_startup_entry+0x25/0x30
[  220.922195]  start_secondary+0x116/0x140
[  220.922206]  common_startup_64+0x13e/0x141
[  220.922222]  </TASK>
[  220.922229] Modules linked in: bonding(E) tls(E) nft_fib_inet(E)
nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E)
nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E)
nft_chain_nat(E) nf_nat(E) nf_conntr
ack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E)
nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) intel_rapl_msr(E)
intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E)
intel_powerclamp(E) coretemp(E) kvm
_intel(E) ipmi_ssif(E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E)
iTCO_vendor_support(E) rapl(E) intel_cstate(E) ast(E) intel_uncore(E)
drm_shmem_helper(E) pcspkr(E) drm_kms_helper(E) i2c_i801(E) mei_me(E)
i2c_mux(E) mxm_wmi(E) mei(E
) i2c_smbus(E) lpc_ich(E) ioatdma(E) acpi_power_meter(E) ipmi_si(E)
acpi_ipmi(E) ipmi_devintf(E) ipmi_msghandler(E) joydev(E) acpi_pad(E)
drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) sg(E) ice(E) ahci(E)
crct10dif_pclmu
l(E) crc32_pclmul(E) libahci(E) crc32c_intel(E) polyval_clmulni(E)
polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E)
[  220.922280]  i2c_algo_bit(E) dca(E) libie(E) wmi(E) dm_mirror(E)
dm_region_hash(E) dm_log(E) dm_mod(E)
[  220.922416] CR2: ffffa3eee1637f14

[3] Kernel crash triggered by step D

[  894.619896] BUG: unable to handle page fault for address:
ffffb5818c2d7f14
[  894.619921] #PF: supervisor read access in kernel mode
[  894.619932] #PF: error_code(0x0000) - not-present page
[  894.619942] PGD 100000067 P4D 100000067 PUD 100237067 PMD 0
[  894.619957] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
[  894.619970] CPU: 5 UID: 0 PID: 2540 Comm: ip Kdump: loaded Tainted: G
           E      6.11.5-1.thg.836e8867d7.241031.135507.el9.x86_64 #1
[  894.619994] Tainted: [E]=UNSIGNED_MODULE
[  894.620002] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS
3.2 12/16/2019
[  894.620014] RIP: 0010:ice_xsk_clean_rx_ring+0x37/0x110 [ice]
[  894.620086] Code: 55 53 48 83 ec 08 44 0f b7 af a4 00 00 00 0f b7 af
a2 00 00 00 66 41 39 ed 74 33 48 89 fb 48 8b 4b 38 41 0f b7 c5 4c 8b 34
c1 <41> f6 46 34 01 75 30 4c 89 f7 41 83 c5 01 e8 f6 5c c6 da 31 c0 66
[  894.620113] RSP: 0018:ffffb58189c376d8 EFLAGS: 00010293
[  894.620124] RAX: 0000000000000000 RBX: ffff92f681f6b800 RCX:
ffff9302f2860000
[  894.620136] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff92f681f6b800
[  894.620148] RBP: 00000000000007ff R08: 000000000000081f R09:
0000000000000000
[  894.620159] R10: ffff92f684dc0000 R11: 0000000000000020 R12:
0000000000000010
[  894.620171] R13: 0000000000000000 R14: ffffb5818c2d7ee0 R15:
ffff92f681fcd740
[  894.620183] FS:  00007f7ee9e27740(0000) GS:ffff92fd9fc80000(0000)
knlGS:0000000000000000
[  894.620196] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  894.620206] CR2: ffffb5818c2d7f14 CR3: 000000010e25e003 CR4:
00000000001706f0
[  894.620218] Call Trace:
[  894.620228]  <TASK>
[  894.620236]  ? __die+0x20/0x70
[  894.620254]  ? page_fault_oops+0x80/0x150
[  894.620268]  ? exc_page_fault+0xcd/0x170
[  894.620283]  ? asm_exc_page_fault+0x22/0x30
[  894.620298]  ? ice_xsk_clean_rx_ring+0x37/0x110 [ice]
[  894.620350]  ice_clean_rx_ring+0x16e/0x190 [ice]
[  894.620401]  ice_down+0x2f8/0x3c0 [ice]
[  894.620443]  ice_xdp_setup_prog+0x193/0x460 [ice]
[  894.620485]  ice_xdp+0x7a/0xb0 [ice]
[  894.620527]  ? __pfx_ice_xdp+0x10/0x10 [ice]
[  894.620567]  dev_xdp_install+0xc7/0x100
[  894.620584]  dev_xdp_attach+0x205/0x5d0
[  894.620597]  do_setlink+0x7d3/0xc20
[  894.620611]  ? __nla_validate_parse+0x125/0x1d0
[  894.620626]  __rtnl_newlink+0x4f7/0x630
[  894.620639]  ? __kmalloc_cache_noprof+0x225/0x2b0
[  894.620652]  rtnl_newlink+0x44/0x70
[  894.620662]  rtnetlink_rcv_msg+0x15c/0x410
[  894.620676]  ? __rmqueue_pcplist+0x5f/0x2c0
[  894.620686]  ? __rmqueue_pcplist+0x5f/0x2c0
[  894.620695]  ? avc_has_perm_noaudit+0x67/0xf0
[  894.620708]  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
[  894.620721]  netlink_rcv_skb+0x57/0x100
[  894.620735]  netlink_unicast+0x246/0x370
[  894.620747]  netlink_sendmsg+0x1f6/0x430
[  894.620758]  ____sys_sendmsg+0x3be/0x3f0
[  894.620771]  ? import_iovec+0x16/0x20
[  894.620783]  ? copy_msghdr_from_user+0x6d/0xa0
[  894.620795]  ___sys_sendmsg+0x88/0xd0
[  894.620807]  ? __mod_memcg_lruvec_state+0xce/0x1c0
[  894.620822]  ? mod_objcg_state+0xc9/0x2f0
[  894.620833]  __sys_sendmsg+0x59/0xa0
[  894.620844]  ? syscall_trace_enter+0xfb/0x190
[  894.620856]  do_syscall_64+0x60/0x180
[  894.620867]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  894.620881] RIP: 0033:0x7f7ee9d0f917
[  894.620891] Code: 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f
1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
[  894.620920] RSP: 002b:00007ffd0b9a9e58 EFLAGS: 00000246 ORIG_RAX:
000000000000002e
[  894.620935] RAX: ffffffffffffffda RBX: 000000006728b03b RCX:
00007f7ee9d0f917
[  894.620948] RDX: 0000000000000000 RSI: 00007ffd0b9a9ec0 RDI:
0000000000000003
[  894.620959] RBP: 0000000000000000 R08: 0000000000000001 R09:
0000000000000078
[  894.620971] R10: 000000000000009b R11: 0000000000000246 R12:
0000000000000001
[  894.620983] R13: 00007ffd0b9a9f70 R14: 0000000000000000 R15:
000055784e873040
[  894.620997]  </TASK>
[  894.621004] Modules linked in: bonding(E) tls(E) nft_fib_inet(E)
nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E)
nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E)
nft_chain_nat(E) nf_nat(E) nf_conntr
ack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E)
nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) intel_rapl_msr(E)
intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E)
intel_powerclamp(E) coretemp(E) kvm
_intel(E) ipmi_ssif(E) iTCO_wdt(E) intel_pmc_bxt(E) kvm(E)
iTCO_vendor_support(E) rapl(E) ast(E) mei_me(E) intel_cstate(E)
intel_uncore(E) drm_shmem_helper(E) pcspkr(E) i2c_i801(E) i2c_mux(E)
drm_kms_helper(E) mei(E) mxm_wmi(E
) lpc_ich(E) i2c_smbus(E) ioatdma(E) acpi_power_meter(E) ipmi_si(E)
acpi_ipmi(E) ipmi_devintf(E) ipmi_msghandler(E) joydev(E) acpi_pad(E)
drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) sg(E) ice(E) ahci(E)
crct10dif_pclmu
l(E) crc32_pclmul(E) crc32c_intel(E) libahci(E) polyval_clmulni(E)
polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E)
[  894.621056]  i2c_algo_bit(E) dca(E) libie(E) wmi(E) dm_mirror(E)
dm_region_hash(E) dm_log(E) dm_mod(E)
[  894.621193] CR2: ffffb5818c2d7f14

Re: ICE + XSK ZC - page faults on 6.1 LTS when process exits?

[Larysa Zaremba]

On Mon, Nov 04, 2024 at 12:18:07PM +0000, Alasdair McWilliam wrote:
> On 04/11/2024 07:11, Larysa Zaremba wrote:
> 
> >> It's been a minute since I've looked at this due to other commitments
> >> but accidentally bumped into the fault again when testing the latest 6.6
> >> LTS for a new feature of our software. (I forgot to revert the commit
> >> for "ice: remove af_xdp_zc_qps bitmap" in our build system.)
> >>
> >> This led me to wonder about the current version, and can trigger the
> >> same crash on 6.11.5 [3].
> >>
> >> Reverting "ice: remove af_xdp_zc_qps bitmap" [1] in the current mainline
> >> is a little more complicated as commit ebc33a3f8d0a ("ice: improve
> >> updating ice_{t,r}x_ring::xsk_pool") also changes things a little so the
> >> reversion doesn't work cleanly.
> >>
> >> I have tweaked everything a little the below patch [2] applies cleanly
> >> to 6.11.5 and 6.12-rc5 and seems to fix the fault.
> >>
> >> Thought I'd bubble this up as it's definitely still an issue in the
> >> mainline kernel as of now.
> >>
> >> Thanks
> >> Alasdair
> >>
> > 
> > Hello,
> > Could you please share the reproduction steps? I will look into this.
> 
> Hello,
> 
> I should probably have provided better steps to reproduce - apologies.
> 
> Our stack uses AF_XDP in zero copy mode with shared UMEM between XSK
> sockets.

Thanks! Just letting you and anyone interested know that I was able to reliably 
reproduce the issue and have found the root cause. Hopefully, will be able to 
send the exact fix soon.

> 
> To isolate other bugs in the past we've used a modified xdpsock app
> based on code previously in kernel samples. The original sample has
> since been taken out the kernel repo, but we maintained the modified
> version in our public repos here [1].
> 
> There's lots in the readme but suffice to say if you run the build.sh
> with bash, it will compile xdpsock_multi user-space app and accompanying
> xdpsock_multi.bpf eBPF app. You'll also need to necessary dependencies
> libxdp/libbpf et al.
> 
> I can reproduce the issue with this app using 8 channels. It can fault
> in two ways (step C or D) below.
> 
> Terminal 1:
> 
> A# ethtool -L <nic> combined 8
> B# ./xdpsock_multi --l2fwd --interface ice1_1 --zero-copy --channels 8
> 
> Terminal 2:
> 
> C# kill -9 $(pidof xdpsock_multi)
> D# ip link set dev <nic> xdp off
> 
> Sometimes the act of killing the process (step C) causes a kernel crash [2].
> 
> Other times, it may survive, leaving an orphaned XDP program attached to
> the NIC. Unloading this manually (step D) causes a kernel crash [3].
> 
> Stack traces are actually different so hence I've provided both.
> 
> Affects:
> 6.1.x
> 6.6.x
> 6.11.x
> 
> Hardware is E810-CQDA2
> Firmware is 3.20 0x8000d83e 1.3146.0
> 
> Let me know if you need anything further.
> 
> Thanks!
> Alasdair
> 
> 
> [1] https://github.com/OpenSource-THG/xdpsock-sample
> 
> [2] Kernel crash triggered by step C
> 
> [  220.921136] BUG: unable to handle page fault for address:
> ffffa3eee1637f14
> [  220.921175] #PF: supervisor write access in kernel mode
> [  220.921196] #PF: error_code(0x0002) - not-present page
> [  220.921217] PGD 100000067 P4D 100000067 PUD 100238067 PMD 0
> [  220.921244] Oops: Oops: 0002 [#1] PREEMPT SMP PTI
> [  220.921267] CPU: 5 UID: 0 PID: 0 Comm: swapper/5 Kdump: loaded
> Tainted: G            E
> 6.11.5-1.thg.836e8867d7.241031.135507.el9.x86_64 #1
> [  220.921315] Tainted: [E]=UNSIGNED_MODULE
> [  220.921331] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS
> 3.2 12/16/2019
> [  220.921357] RIP: 0010:ice_clean_rx_irq_zc+0xde/0x7d0 [ice]
> [  220.921489] Code: 0f 84 d0 01 00 00 44 3b 7c 24 08 0f 84 a1 02 00 00
> 48 8b 53 38 41 0f b7 4d 04 4c 8b 24 c2 89 c8 81 e1 ff 3f 00 00 66 25 ff
> 3f <41> c7 44 24 34 00 00 00 00 49 8b 74 24 18 48 8d 96 00 01 00 00 49
> [  220.921518] RSP: 0018:ffffa3eec64d0d88 EFLAGS: 00010206
> [  220.921529] RAX: 000000000000014d RBX: ffff89bbc2aa2a00 RCX:
> 000000000000014d
> [  220.921542] RDX: ffff89b408830000 RSI: 0000000000000040 RDI:
> ffff89bbc2aa2a00
> [  220.921554] RBP: 0000000000000000 R08: 0000000000000000 R09:
> ffff89b407655000
> [  220.921566] R10: 0000ffffffffffff R11: ffffa3eec64d0ff8 R12:
> ffffa3eee1637ee0
> [  220.921578] R13: ffff89b414710000 R14: ffff89bbc7919500 R15:
> 0000000000000000
> [  220.921591] FS:  0000000000000000(0000) GS:ffff89bb5fc80000(0000)
> knlGS:0000000000000000
> [  220.921605] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  220.921616] CR2: ffffa3eee1637f14 CR3: 00000001d9820006 CR4:
> 00000000001706f0
> [  220.921628] Call Trace:
> [  220.921639]  <IRQ>
> [  220.921647]  ? __die+0x20/0x70
> [  220.921663]  ? page_fault_oops+0x80/0x150
> [  220.921676]  ? exc_page_fault+0xcd/0x170
> [  220.921690]  ? asm_exc_page_fault+0x22/0x30
> [  220.921707]  ? ice_clean_rx_irq_zc+0xde/0x7d0 [ice]
> [  220.921759]  ? ice_clean_tx_irq+0x166/0x3c0 [ice]
> [  220.921808]  ice_napi_poll+0xb2/0x2a0 [ice]
> [  220.921858]  __napi_poll+0x2c/0x1b0
> [  220.921870]  net_rx_action+0x30d/0x3e0
> [  220.921881]  ? __raise_softirq_irqoff+0x18/0x80
> [  220.921896]  ? __napi_schedule+0xa6/0xc0
> [  220.921907]  ? ice_msix_clean_rings+0x4f/0x60 [ice]
> [  220.921959]  handle_softirqs+0xf0/0x2e0
> [  220.921972]  __irq_exit_rcu+0x80/0xe0
> [  220.921983]  common_interrupt+0xb7/0xd0
> [  220.921995]  </IRQ>
> [  220.922001]  <TASK>
> [  220.922008]  asm_common_interrupt+0x22/0x40
> [  220.922022] RIP: 0010:cpuidle_enter_state+0xc8/0x420
> [  220.922034] Code: 0e b6 3e ff e8 09 ee ff ff 8b 55 04 49 89 c5 0f 1f
> 44 00 00 31 ff e8 97 69 3d ff 45 84 ff 0f 85 38 02 00 00 fb 0f 1f 44 00
> 00 <45> 85 f6 0f 88 6a 01 00 00 49 63 d6 4c 2b 2c 24 48 8d 04 52 48 8d
> [  220.922061] RSP: 0018:ffffa3eec4377e78 EFLAGS: 00000246
> [  220.922072] RAX: ffff89bb5fc80000 RBX: 0000000000000004 RCX:
> 000000000000001f
> [  220.922085] RDX: 0000000000000005 RSI: ffffffffb255a8a3 RDI:
> ffffffffb2533173
> [  220.922098] RBP: ffff89bb5fcc0cc8 R08: 000000336fecb8ce R09:
> 0000000000000018
> [  220.922109] R10: 000000000000453f R11: ffff89bb5fcb47e4 R12:
> ffffffffb32bdce0
> [  220.922121] R13: 000000336fecb8ce R14: 0000000000000004 R15:
> 0000000000000000
> [  220.922135]  ? cpuidle_enter_state+0xb9/0x420
> [  220.922147]  cpuidle_enter+0x29/0x40
> [  220.922161]  cpuidle_idle_call+0x100/0x170
> [  220.922175]  do_idle+0x7d/0xd0
> [  220.922185]  cpu_startup_entry+0x25/0x30
> [  220.922195]  start_secondary+0x116/0x140
> [  220.922206]  common_startup_64+0x13e/0x141
> [  220.922222]  </TASK>
> [  220.922229] Modules linked in: bonding(E) tls(E) nft_fib_inet(E)
> nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E)
> nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E)
> nft_chain_nat(E) nf_nat(E) nf_conntr
> ack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E)
> nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) intel_rapl_msr(E)
> intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E)
> intel_powerclamp(E) coretemp(E) kvm
> _intel(E) ipmi_ssif(E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E)
> iTCO_vendor_support(E) rapl(E) intel_cstate(E) ast(E) intel_uncore(E)
> drm_shmem_helper(E) pcspkr(E) drm_kms_helper(E) i2c_i801(E) mei_me(E)
> i2c_mux(E) mxm_wmi(E) mei(E
> ) i2c_smbus(E) lpc_ich(E) ioatdma(E) acpi_power_meter(E) ipmi_si(E)
> acpi_ipmi(E) ipmi_devintf(E) ipmi_msghandler(E) joydev(E) acpi_pad(E)
> drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) sg(E) ice(E) ahci(E)
> crct10dif_pclmu
> l(E) crc32_pclmul(E) libahci(E) crc32c_intel(E) polyval_clmulni(E)
> polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E)
> [  220.922280]  i2c_algo_bit(E) dca(E) libie(E) wmi(E) dm_mirror(E)
> dm_region_hash(E) dm_log(E) dm_mod(E)
> [  220.922416] CR2: ffffa3eee1637f14
> 
> [3] Kernel crash triggered by step D
> 
> [  894.619896] BUG: unable to handle page fault for address:
> ffffb5818c2d7f14
> [  894.619921] #PF: supervisor read access in kernel mode
> [  894.619932] #PF: error_code(0x0000) - not-present page
> [  894.619942] PGD 100000067 P4D 100000067 PUD 100237067 PMD 0
> [  894.619957] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
> [  894.619970] CPU: 5 UID: 0 PID: 2540 Comm: ip Kdump: loaded Tainted: G
>            E      6.11.5-1.thg.836e8867d7.241031.135507.el9.x86_64 #1
> [  894.619994] Tainted: [E]=UNSIGNED_MODULE
> [  894.620002] Hardware name: Supermicro SYS-1028R-TDW/X10DDW-i, BIOS
> 3.2 12/16/2019
> [  894.620014] RIP: 0010:ice_xsk_clean_rx_ring+0x37/0x110 [ice]
> [  894.620086] Code: 55 53 48 83 ec 08 44 0f b7 af a4 00 00 00 0f b7 af
> a2 00 00 00 66 41 39 ed 74 33 48 89 fb 48 8b 4b 38 41 0f b7 c5 4c 8b 34
> c1 <41> f6 46 34 01 75 30 4c 89 f7 41 83 c5 01 e8 f6 5c c6 da 31 c0 66
> [  894.620113] RSP: 0018:ffffb58189c376d8 EFLAGS: 00010293
> [  894.620124] RAX: 0000000000000000 RBX: ffff92f681f6b800 RCX:
> ffff9302f2860000
> [  894.620136] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffff92f681f6b800
> [  894.620148] RBP: 00000000000007ff R08: 000000000000081f R09:
> 0000000000000000
> [  894.620159] R10: ffff92f684dc0000 R11: 0000000000000020 R12:
> 0000000000000010
> [  894.620171] R13: 0000000000000000 R14: ffffb5818c2d7ee0 R15:
> ffff92f681fcd740
> [  894.620183] FS:  00007f7ee9e27740(0000) GS:ffff92fd9fc80000(0000)
> knlGS:0000000000000000
> [  894.620196] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  894.620206] CR2: ffffb5818c2d7f14 CR3: 000000010e25e003 CR4:
> 00000000001706f0
> [  894.620218] Call Trace:
> [  894.620228]  <TASK>
> [  894.620236]  ? __die+0x20/0x70
> [  894.620254]  ? page_fault_oops+0x80/0x150
> [  894.620268]  ? exc_page_fault+0xcd/0x170
> [  894.620283]  ? asm_exc_page_fault+0x22/0x30
> [  894.620298]  ? ice_xsk_clean_rx_ring+0x37/0x110 [ice]
> [  894.620350]  ice_clean_rx_ring+0x16e/0x190 [ice]
> [  894.620401]  ice_down+0x2f8/0x3c0 [ice]
> [  894.620443]  ice_xdp_setup_prog+0x193/0x460 [ice]
> [  894.620485]  ice_xdp+0x7a/0xb0 [ice]
> [  894.620527]  ? __pfx_ice_xdp+0x10/0x10 [ice]
> [  894.620567]  dev_xdp_install+0xc7/0x100
> [  894.620584]  dev_xdp_attach+0x205/0x5d0
> [  894.620597]  do_setlink+0x7d3/0xc20
> [  894.620611]  ? __nla_validate_parse+0x125/0x1d0
> [  894.620626]  __rtnl_newlink+0x4f7/0x630
> [  894.620639]  ? __kmalloc_cache_noprof+0x225/0x2b0
> [  894.620652]  rtnl_newlink+0x44/0x70
> [  894.620662]  rtnetlink_rcv_msg+0x15c/0x410
> [  894.620676]  ? __rmqueue_pcplist+0x5f/0x2c0
> [  894.620686]  ? __rmqueue_pcplist+0x5f/0x2c0
> [  894.620695]  ? avc_has_perm_noaudit+0x67/0xf0
> [  894.620708]  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> [  894.620721]  netlink_rcv_skb+0x57/0x100
> [  894.620735]  netlink_unicast+0x246/0x370
> [  894.620747]  netlink_sendmsg+0x1f6/0x430
> [  894.620758]  ____sys_sendmsg+0x3be/0x3f0
> [  894.620771]  ? import_iovec+0x16/0x20
> [  894.620783]  ? copy_msghdr_from_user+0x6d/0xa0
> [  894.620795]  ___sys_sendmsg+0x88/0xd0
> [  894.620807]  ? __mod_memcg_lruvec_state+0xce/0x1c0
> [  894.620822]  ? mod_objcg_state+0xc9/0x2f0
> [  894.620833]  __sys_sendmsg+0x59/0xa0
> [  894.620844]  ? syscall_trace_enter+0xfb/0x190
> [  894.620856]  do_syscall_64+0x60/0x180
> [  894.620867]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [  894.620881] RIP: 0033:0x7f7ee9d0f917
> [  894.620891] Code: 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f
> 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f
> 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
> [  894.620920] RSP: 002b:00007ffd0b9a9e58 EFLAGS: 00000246 ORIG_RAX:
> 000000000000002e
> [  894.620935] RAX: ffffffffffffffda RBX: 000000006728b03b RCX:
> 00007f7ee9d0f917
> [  894.620948] RDX: 0000000000000000 RSI: 00007ffd0b9a9ec0 RDI:
> 0000000000000003
> [  894.620959] RBP: 0000000000000000 R08: 0000000000000001 R09:
> 0000000000000078
> [  894.620971] R10: 000000000000009b R11: 0000000000000246 R12:
> 0000000000000001
> [  894.620983] R13: 00007ffd0b9a9f70 R14: 0000000000000000 R15:
> 000055784e873040
> [  894.620997]  </TASK>
> [  894.621004] Modules linked in: bonding(E) tls(E) nft_fib_inet(E)
> nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E)
> nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E)
> nft_chain_nat(E) nf_nat(E) nf_conntr
> ack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E)
> nf_tables(E) libcrc32c(E) nfnetlink(E) vfat(E) fat(E) intel_rapl_msr(E)
> intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E)
> intel_powerclamp(E) coretemp(E) kvm
> _intel(E) ipmi_ssif(E) iTCO_wdt(E) intel_pmc_bxt(E) kvm(E)
> iTCO_vendor_support(E) rapl(E) ast(E) mei_me(E) intel_cstate(E)
> intel_uncore(E) drm_shmem_helper(E) pcspkr(E) i2c_i801(E) i2c_mux(E)
> drm_kms_helper(E) mei(E) mxm_wmi(E
> ) lpc_ich(E) i2c_smbus(E) ioatdma(E) acpi_power_meter(E) ipmi_si(E)
> acpi_ipmi(E) ipmi_devintf(E) ipmi_msghandler(E) joydev(E) acpi_pad(E)
> drm(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sd_mod(E) sg(E) ice(E) ahci(E)
> crct10dif_pclmu
> l(E) crc32_pclmul(E) crc32c_intel(E) libahci(E) polyval_clmulni(E)
> polyval_generic(E) igb(E) libata(E) ghash_clmulni_intel(E)
> [  894.621056]  i2c_algo_bit(E) dca(E) libie(E) wmi(E) dm_mirror(E)
> dm_region_hash(E) dm_log(E) dm_mod(E)
> [  894.621193] CR2: ffffb5818c2d7f14