* [FYI] Input route ref count underflow since probably 6.6.22
@ 2024-06-28 11:10 Jakub Sitnicki
2024-07-01 23:38 ` Jakub Kicinski
0 siblings, 1 reply; 4+ messages in thread
From: Jakub Sitnicki @ 2024-06-28 11:10 UTC (permalink / raw)
To: netdev; +Cc: kernel-team
Hi,
We've observed an unbalanced dst_release() on an input route in v6.6.y.
First noticed in 6.6.22. Or at least that is how far back our logs go.
We have just started looking into it and don't have much context yet,
except that:
1. the issue is architecture agnostic, seen both on x86_64 and arm64;
2. the backtrace, we realize, doesn't point to the source of problem,
it's just where the ref count underflow manifests itself;
3. while have out-of-tree modules, they are for the crypto subsystem.
We will follow up as we collect more info on this, but we would
appreciate any hints or pointers to potential suspects, if anything
comes to mind.
Decoded warning reports follow.
Thanks,
-jkbs
* arm64
------------[ cut here ]------------
rcuref - imbalanced put()
WARNING: CPU: 20 PID: 180350 at lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
Modules linked in: overlay mptcp_diag xsk_diag raw_diag unix_diag af_packet_diag netlink_diag nft_compat esp4 xt_hashlimit ip_set_hash_netport xt_length nf_conntrack_netlink nft_fwd_netdev nf_dup_netdev xfrm_interface xfrm6_tunnel nft_numgen nft_log nft_limit dummy ip_gre gre cls_bpf xfrm_user xfrm_algo fou6 ip6_tunnel tunnel6 ipip mpls_gso mpls_iptunnel mpls_router sit tunnel4 fou ip_tunnel ip6_udp_tunnel udp_tunnel nft_ct nf_tables zstd zram zsmalloc xgene_edac sch_ingress tcp_diag udp_diag inet_diag veth tun tcp_bbr sch_fq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6table_security ip6table_nat ip6_tables xt_LOG nf_log_syslog ipt_REJECT nf_reject_ipv4 xt_tcpmss iptable_filter xt_TCPMSS xt_bpf xt_limit xt_multiport xt_NFLOG nfnetlink_log xt_connbytes xt_connlabel xt_statistic xt_mark xt_connmark xt_conntrack iptable_mangle xt_nat iptable_nat nf_nat xt_owner xt_set xt_comment xt_tcpudp xt_CT nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 iptable_raw ip_set_hash_ip ip_set_hash_net ip_set raid0 md_mod dm_crypt trusted asn1_encoder tee algif_skcipher af_alg 8021q garp mrp stp llc nvme_fabrics acpi_ipmi mlx5_core crct10dif_ce ghash_ce ipmi_ssif mlxfw sha2_ce sha256_arm64 ipmi_devintf nvme sha1_ce xhci_pci tls tiny_power_button arm_spe_pmu ipmi_msghandler xhci_hcd nvme_core psample button i2c_designware_platform i2c_designware_core cppc_cpufreq arm_dsu_pmu tpm_tis tpm_tis_core fuse dm_mod dax nfnetlink efivarfs ip_tables x_tables bcmcrypt(O) aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher [last unloaded: kheaders]
CPU: 20 PID: 180350 Comm: napi/iconduit-g Tainted: G O 6.6.32-cloudflare-2024.5.16 #1
Hardware name: GIGABYTE R152-P30-CD/MP32-AR1-00, BIOS F33e (SCP: 2.10.20230517) 02/21/2024
pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
lr : rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
sp : ffff8000cb6eb760
x29: ffff8000cb6eb760 x28: 00000000f0d32f50 x27: ffff080223064cf0
x26: 0000000000000001 x25: ffffbea2e405703c x24: 00000000ce087c9f
x23: 0000000000000000 x22: ffff080223064cf0 x21: ffff0831525c1e00
x20: ffff07ff8a3eef00 x19: ffff0831525c1e40 x18: 0000000000000004
x17: 0000000000000002 x16: 0000000000000001 x15: 0000000000000000
x14: 0000000000000000 x13: 2928747570206465 x12: ffff087d3edbffa8
x11: ffff087d3eb00000 x10: ffff087d3edc0000 x9 : ffffbea2e35e7c78
x8 : 0000000000000001 x7 : 00000000000bffe8 x6 : c0000000ffff7fff
x5 : ffff087d3f0eee88 x4 : 0000000000000000 x3 : ffff49da5a45f000
x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff081eaa143f00
Call trace:
rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
dst_release (include/linux/rcuref.h:94 include/linux/rcuref.h:150 net/core/dst.c:166)
rt_cache_route (net/ipv4/route.c:1497)
rt_set_nexthop.isra.0 (net/ipv4/route.c:1604 (discriminator 1))
ip_route_input_slow (include/net/lwtunnel.h:140 net/ipv4/route.c:1873 net/ipv4/route.c:2152 net/ipv4/route.c:2338)
ip_route_input_noref (net/ipv4/route.c:2485 net/ipv4/route.c:2496)
ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 (discriminator 1))
ip_sublist_rcv (net/ipv4/ip_input.c:613 (discriminator 1) net/ipv4/ip_input.c:639 (discriminator 1))
ip_list_rcv (net/ipv4/ip_input.c:675)
__netif_receive_skb_list_core (net/core/dev.c:5598 net/core/dev.c:5646)
netif_receive_skb_list_internal (net/core/dev.c:5700 net/core/dev.c:5789)
napi_complete_done (include/linux/list.h:37 (discriminator 2) include/net/gro.h:449 (discriminator 2) include/net/gro.h:444 (discriminator 2) net/core/dev.c:6129 (discriminator 2))
veth_poll (drivers/net/veth.c:1008 (discriminator 1)) veth
__napi_poll (net/core/dev.c:6559)
bpf_trampoline_6442466812+0xbc/0x1000
__napi_poll (net/core/dev.c:6546)
napi_threaded_poll (include/linux/netpoll.h:89 net/core/dev.c:6703)
kthread (kernel/kthread.c:388)
ret_from_fork (arch/arm64/kernel/entry.S:862)
---[ end trace 0000000000000000 ]---
* x86_64
------------[ cut here ]------------
rcuref - imbalanced put()
WARNING: CPU: 18 PID: 164489 at lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
Modules linked in: macvlan overlay nft_compat esp4 xt_hashlimit ip_set_hash_netport xt_length nf_conntrack_netlink nft_fwd_netdev nf_dup_netdev xfrm_interface xfrm6_tunnel nft_numgen nft_log nft_limit dummy ip_gre gre cls_bpf xfrm_user xfrm_algo fou6 ip6_tunnel tunnel6 ipip nft_ct nf_tables mpls_gso mpls_iptunnel mpls_router sit tunnel4 fou ip_tunnel ip6_udp_tunnel udp_tunnel zstd zram zsmalloc sch_ingress tcp_diag udp_diag inet_diag veth tun tcp_bbr sch_fq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6table_security ip6table_nat ip6_tables xt_LOG nf_log_syslog ipt_REJECT nf_reject_ipv4 xt_tcpmss iptable_filter xt_TCPMSS xt_bpf xt_limit xt_multiport xt_NFLOG nfnetlink_log xt_connbytes xt_connlabel xt_statistic xt_mark xt_connmark xt_conntrack iptable_mangle xt_nat iptable_nat nf_nat xt_owner xt_set xt_comment xt_tcpudp xt_CT nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw ip_set_hash_ip ip_set_hash_net ip_set raid0
md_mod essiv dm_crypt trusted asn1_encoder tee 8021q garp mrp stp llc nvme_fabrics amd64_edac ipmi_ssif kvm_amd kvm irqbypass crc32_pclmul crc32c_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel xhci_pci acpi_ipmi mlx5_core rapl mlxfw nvme ipmi_si tls ipmi_devintf tiny_power_button xhci_hcd nvme_core ccp psample i2c_piix4 ipmi_msghandler button fuse dm_mod dax nfnetlink efivarfs ip_tables x_tables bcmcrypt(O) crypto_simd cryptd [last unloaded: kheaders]
CPU: 18 PID: 164489 Comm: napi/iconduit-g Kdump: loaded Tainted: G O 6.6.32-cloudflare-2024.5.16 #1
Hardware name: GIGABYTE R162-Z12-CD1/MZ12-HD4-CD, BIOS M06-sig 12/28/2022
RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
Code: 31 c0 eb da 80 3d 23 a5 38 02 00 74 0a c7 03 00 00 00 e0 31 c0 eb c7 48 c7 c7 ef cd 0a 90 c6 05 09 a5 38 02 01 e8 69 cc 9c ff <0f> 0b eb df cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90
All code
========
0: 31 c0 xor %eax,%eax
2: eb da jmp 0xffffffffffffffde
4: 80 3d 23 a5 38 02 00 cmpb $0x0,0x238a523(%rip) # 0x238a52e
b: 74 0a je 0x17
d: c7 03 00 00 00 e0 movl $0xe0000000,(%rbx)
13: 31 c0 xor %eax,%eax
15: eb c7 jmp 0xffffffffffffffde
17: 48 c7 c7 ef cd 0a 90 mov $0xffffffff900acdef,%rdi
1e: c6 05 09 a5 38 02 01 movb $0x1,0x238a509(%rip) # 0x238a52e
25: e8 69 cc 9c ff call 0xffffffffff9ccc93
2a:* 0f 0b ud2 <-- trapping instruction
2c: eb df jmp 0xd
2e: cc int3
2f: cc int3
30: cc int3
31: cc int3
32: cc int3
33: 90 nop
34: 90 nop
35: 90 nop
36: 90 nop
37: 90 nop
38: 90 nop
39: 90 nop
3a: 90 nop
3b: 90 nop
3c: 90 nop
3d: 90 nop
3e: 90 nop
3f: 90 nop
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: eb df jmp 0xffffffffffffffe3
4: cc int3
5: cc int3
6: cc int3
7: cc int3
8: cc int3
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 90 nop
RSP: 0018:ffffc90047a23908 EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff888cedda4e80 RCX: 0000000000000027
RDX: ffff88a43f720748 RSI: 0000000000000001 RDI: ffff88a43f720740
RBP: ffffc90047a23988 R08: 0000000000000000 R09: ffffc90047a23798
R10: ffff88e06f2cc1a8 R11: 0000000000000003 R12: ffff888cedda4e40
R13: ffffc90047a23a98 R14: 0000000000000000 R15: 000000000a8cf4d5
FS: 0000000000000000(0000) GS:ffff88a43f700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6e59776000 CR3: 000000417ef0c005 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
<TASK>
? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
? __warn (kernel/panic.c:681)
? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
? report_bug (lib/bug.c:180 lib/bug.c:219)
? srso_alias_return_thunk (arch/x86/lib/retpoline.S:175)
? prb_read_valid (kernel/printk/printk_ringbuffer.c:1941)
? handle_bug (arch/x86/kernel/traps.c:237)
? exc_invalid_op (arch/x86/kernel/traps.c:258 (discriminator 1))
? asm_exc_invalid_op (arch/x86/include/asm/idtentry.h:568)
? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
dst_release (arch/x86/include/asm/preempt.h:95 include/linux/rcuref.h:151 net/core/dst.c:166)
rt_cache_route (net/ipv4/route.c:1497)
rt_set_nexthop.isra.0 (net/ipv4/route.c:1604 (discriminator 1))
ip_route_input_slow (include/net/lwtunnel.h:140 net/ipv4/route.c:1873 net/ipv4/route.c:2152 net/ipv4/route.c:2338)
? srso_alias_return_thunk (arch/x86/lib/retpoline.S:175)
ip_route_input_noref (net/ipv4/route.c:2485 net/ipv4/route.c:2496)
ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 (discriminator 1))
ip_sublist_rcv (net/ipv4/ip_input.c:613 (discriminator 1) net/ipv4/ip_input.c:639 (discriminator 1))
? __pfx_ip_rcv_finish (net/ipv4/ip_input.c:436)
ip_list_rcv (net/ipv4/ip_input.c:675)
__netif_receive_skb_list_core (net/core/dev.c:5598 (discriminator 3) net/core/dev.c:5646 (discriminator 3))
? srso_alias_return_thunk (arch/x86/lib/retpoline.S:175)
netif_receive_skb_list_internal (net/core/dev.c:5700 net/core/dev.c:5789)
napi_complete_done (include/linux/list.h:37 (discriminator 2) include/net/gro.h:449 (discriminator 2) include/net/gro.h:444 (discriminator 2) net/core/dev.c:6129 (discriminator 2))
veth_poll (drivers/net/veth.c:1008 (discriminator 1)) veth
? srso_alias_return_thunk (arch/x86/lib/retpoline.S:175)
? psi_group_change (kernel/sched/psi.c:873)
? srso_alias_return_thunk (arch/x86/lib/retpoline.S:175)
? __perf_event_task_sched_in (arch/x86/include/asm/atomic.h:23 include/linux/atomic/atomic-arch-fallback.h:444 include/linux/atomic/atomic-instrumented.h:33 kernel/events/core.c:4014)
? srso_alias_return_thunk (arch/x86/lib/retpoline.S:175)
? finish_task_switch.isra.0 (arch/x86/include/asm/irqflags.h:42 arch/x86/include/asm/irqflags.h:77 kernel/sched/sched.h:1386 kernel/sched/core.c:5138 kernel/sched/core.c:5256)
__napi_poll (net/core/dev.c:6559)
bpf_trampoline_6442482065+0x79/0x1000
? schedule (arch/x86/include/asm/preempt.h:85 (discriminator 13) kernel/sched/core.c:6773 (discriminator 13))
__napi_poll (net/core/dev.c:6546)
napi_threaded_poll (include/linux/netpoll.h:89 net/core/dev.c:6703)
? __pfx_napi_threaded_poll (net/core/dev.c:6686)
kthread (kernel/kthread.c:388)
? __pfx_kthread (kernel/kthread.c:341)
ret_from_fork (arch/x86/kernel/process.c:153)
? __pfx_kthread (kernel/kthread.c:341)
ret_from_fork_asm (arch/x86/entry/entry_64.S:314)
</TASK>
---[ end trace 0000000000000000 ]---
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [FYI] Input route ref count underflow since probably 6.6.22
2024-06-28 11:10 [FYI] Input route ref count underflow since probably 6.6.22 Jakub Sitnicki
@ 2024-07-01 23:38 ` Jakub Kicinski
2024-07-02 14:21 ` Jakub Sitnicki
0 siblings, 1 reply; 4+ messages in thread
From: Jakub Kicinski @ 2024-07-01 23:38 UTC (permalink / raw)
To: Jakub Sitnicki; +Cc: netdev, kernel-team
On Fri, 28 Jun 2024 13:10:53 +0200 Jakub Sitnicki wrote:
> We've observed an unbalanced dst_release() on an input route in v6.6.y.
> First noticed in 6.6.22. Or at least that is how far back our logs go.
>
> We have just started looking into it and don't have much context yet,
> except that:
>
> 1. the issue is architecture agnostic, seen both on x86_64 and arm64;
> 2. the backtrace, we realize, doesn't point to the source of problem,
> it's just where the ref count underflow manifests itself;
> 3. while have out-of-tree modules, they are for the crypto subsystem.
>
> We will follow up as we collect more info on this, but we would
> appreciate any hints or pointers to potential suspects, if anything
> comes to mind.
Hi! Luck would have it the same crash landed on my desk.
Did you manage to narrow it down?
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [FYI] Input route ref count underflow since probably 6.6.22
2024-07-01 23:38 ` Jakub Kicinski
@ 2024-07-02 14:21 ` Jakub Sitnicki
2025-01-13 12:02 ` Jakub Sitnicki
0 siblings, 1 reply; 4+ messages in thread
From: Jakub Sitnicki @ 2024-07-02 14:21 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: netdev, kernel-team
On Mon, Jul 01, 2024 at 04:38 PM -07, Jakub Kicinski wrote:
> On Fri, 28 Jun 2024 13:10:53 +0200 Jakub Sitnicki wrote:
>> We've observed an unbalanced dst_release() on an input route in v6.6.y.
>> First noticed in 6.6.22. Or at least that is how far back our logs go.
>>
>> We have just started looking into it and don't have much context yet,
>> except that:
>>
>> 1. the issue is architecture agnostic, seen both on x86_64 and arm64;
>> 2. the backtrace, we realize, doesn't point to the source of problem,
>> it's just where the ref count underflow manifests itself;
>> 3. while have out-of-tree modules, they are for the crypto subsystem.
>>
>> We will follow up as we collect more info on this, but we would
>> appreciate any hints or pointers to potential suspects, if anything
>> comes to mind.
>
> Hi! Luck would have it the same crash landed on my desk.
> Did you manage to narrow it down?
Nothing yet. Will keep you posted.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [FYI] Input route ref count underflow since probably 6.6.22
2024-07-02 14:21 ` Jakub Sitnicki
@ 2025-01-13 12:02 ` Jakub Sitnicki
0 siblings, 0 replies; 4+ messages in thread
From: Jakub Sitnicki @ 2025-01-13 12:02 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: netdev, kernel-team
On Tue, Jul 02, 2024 at 04:21 PM +02, Jakub Sitnicki wrote:
> On Mon, Jul 01, 2024 at 04:38 PM -07, Jakub Kicinski wrote:
>> On Fri, 28 Jun 2024 13:10:53 +0200 Jakub Sitnicki wrote:
>>> We've observed an unbalanced dst_release() on an input route in v6.6.y.
>>> First noticed in 6.6.22. Or at least that is how far back our logs go.
>>>
>>> We have just started looking into it and don't have much context yet,
>>> except that:
>>>
>>> 1. the issue is architecture agnostic, seen both on x86_64 and arm64;
>>> 2. the backtrace, we realize, doesn't point to the source of problem,
>>> it's just where the ref count underflow manifests itself;
>>> 3. while have out-of-tree modules, they are for the crypto subsystem.
>>>
>>> We will follow up as we collect more info on this, but we would
>>> appreciate any hints or pointers to potential suspects, if anything
>>> comes to mind.
>>
>> Hi! Luck would have it the same crash landed on my desk.
>> Did you manage to narrow it down?
>
> Nothing yet. Will keep you posted.
Finally circling back to this to hunt it down...
We've tweaked "imbalanced put()" warning to log the ref count, that is
the result atomic_read(&ref->refcnt) load in rcuref_put_slowpath():
---8<---
diff --git a/lib/rcuref.c b/lib/rcuref.c
index 5ec00a4a64d1..53006542e785 100644
--- a/lib/rcuref.c
+++ b/lib/rcuref.c
@@ -264,7 +264,8 @@ bool rcuref_put_slowpath(rcuref_t *ref)
* put() operation is imbalanced. Warn, put the reference count back to
* DEAD and tell the caller to not deconstruct the object.
*/
- if (WARN_ONCE(cnt >= RCUREF_RELEASED, "rcuref - imbalanced put()")) {
+ if (WARN_ONCE(cnt >= RCUREF_RELEASED,
+ "rcuref - imbalanced put(): refcnt=%#x", cnt)) {
atomic_set(&ref->refcnt, RCUREF_DEAD);
return false;
}
--->8---
So far we're only seeing reports with cnt == RCUREF_DEAD, for example:
------------[ cut here ]------------
rcuref - imbalanced put(): refcnt=0xe0000000
WARNING: CPU: 105 PID: 173613 at lib/rcuref.c:267 rcuref_put_slowpath+0x6d/0x80
Modules linked in: overlay mptcp_diag xsk_diag raw_diag unix_diag af_packet_diag netlink_diag esp4 xt_hashlimit ip_set_hash_netport nft_compat nf_conntrack_netlink xfrm_interface xfrm6_tunnel sit nft_numgen nft_log nft_limit dummy ip_gre gre xfrm_user xfrm_algo mpls_gso mpls_iptunnel mpls_router fou6 ip6_tunnel tunnel6 ipip tunnel4 fou ip_tunnel ip6_udp_tunnel udp_tunnel cls_bpf nft_fwd_netdev nf_dup_netdev nft_ct nf_tables zstd zram zsmalloc sch_ingress tcp_diag udp_diag inet_diag veth tun tcp_bbr sch_fq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6table_security ip6table_nat ip6_tables xt_limit xt_LOG nf_log_syslog ipt_REJECT nf_reject_ipv4 xt_multiport xt_tcpmss iptable_filter xt_length xt_TCPMSS xt_DSCP xt_bpf xt_NFLOG nfnetlink_log xt_connbytes xt_connlabel xt_statistic xt_connmark xt_conntrack iptable_mangle xt_mark xt_nat iptable_nat nf_nat xt_owner xt_set xt_comment xt_tcpudp xt_CT nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 iptable_raw ip_set_hash_ip ip_set_hash_net ip_set raid0 md_mod essiv dm_crypt trusted asn1_encoder tee dm_mod dax nvme_fabrics 8021q garp mrp stp llc ipmi_ssif amd64_edac kvm_amd kvm irqbypass crc32_pclmul crc32c_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel mlx5_core rapl acpi_ipmi xhci_pci mlxfw nvme ipmi_si ipmi_devintf tls tiny_power_button xhci_hcd ccp i2c_piix4 nvme_core psample ipmi_msghandler button fuse nfnetlink efivarfs ip_tables x_tables bcmcrypt(O) crypto_simd cryptd [last unloaded: kheaders]
CPU: 105 PID: 173613 Comm: napi/iconduit-g Kdump: loaded Tainted: G O 6.6.69-cloudflare-2025.1.1 #1
Hardware name: HYVE EDGE-METAL-GEN11/HS1811D_Lite, BIOS V0.11-sig 12/23/2022
RIP: 0010:rcuref_put_slowpath+0x6d/0x80
Code: eb da 80 3d 85 0c 37 02 00 74 0a c7 03 00 00 00 e0 31 c0 eb c7 89 c6 48 c7 c7 80 41 af b9 c6 05 69 0c 37 02 01 e8 83 46 9b ff <0f> 0b eb dd 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 90 90 90
RSP: 0018:ffffc90042bb7908 EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff88c88d886f40 RCX: 0000000000000027
RDX: ffff88c81faa0788 RSI: 0000000000000001 RDI: ffff88c81faa0780
RBP: ffffc90042bb7988 R08: 0000000000000000 R09: ffffc90042bb7798
R10: ffff88e04f2cc1a8 R11: 0000000000000003 R12: ffff88c88d886f00
R13: ffffc90042bb7a98 R14: 0000000000000000 R15: 0000000039277c9f
FS: 0000000000000000(0000) GS:ffff88c81fa80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc4a9f48000 CR3: 000000306fd7a004 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
<TASK>
? rcuref_put_slowpath+0x6d/0x80
? __warn+0x81/0x130
? rcuref_put_slowpath+0x6d/0x80
? report_bug+0x16f/0x1a0
? srso_alias_return_thunk+0x5/0xfbef5
? prb_read_valid+0x1b/0x30
? handle_bug+0x53/0x90
? exc_invalid_op+0x17/0x70
? asm_exc_invalid_op+0x1a/0x20
? rcuref_put_slowpath+0x6d/0x80
? rcuref_put_slowpath+0x6d/0x80
dst_release+0x3d/0xa0
rt_cache_route+0x6b/0xa0
rt_set_nexthop.isra.0+0x16c/0x400
ip_route_input_slow+0x886/0xbe0
? srso_alias_return_thunk+0x5/0xfbef5
ip_route_input_noref+0x95/0xa0
ip_rcv_finish_core.isra.0+0xc6/0x450
ip_sublist_rcv+0xe5/0x380
? __pfx_ip_rcv_finish+0x10/0x10
ip_list_rcv+0x165/0x190
__netif_receive_skb_list_core+0x30f/0x340
? srso_alias_return_thunk+0x5/0xfbef5
netif_receive_skb_list_internal+0x1bd/0x330
napi_complete_done+0x72/0x200
veth_poll+0xe4/0x1d0 [veth]
? srso_alias_return_thunk+0x5/0xfbef5
? psi_group_change+0x177/0x3c0
? srso_alias_return_thunk+0x5/0xfbef5
? __perf_event_task_sched_in+0x86/0x210
? srso_alias_return_thunk+0x5/0xfbef5
? finish_task_switch.isra.0+0x85/0x280
__napi_poll+0x2b/0x1b0
bpf_trampoline_6442534135+0x7d/0x1000
? schedule+0x5e/0xd0
__napi_poll+0x5/0x1b0
napi_threaded_poll+0x22c/0x270
? __pfx_napi_threaded_poll+0x10/0x10
kthread+0xe8/0x120
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1b/0x30
</TASK>
---[ end trace 0000000000000000 ]---
... which would suggest that we're dealing with a race to release the
last reference! At least I don't see any other explantion for this
state. On paper it could look like (wide text ahead):
refcnt CPU 0 CPU 1
0x00000000 (RCUREF_ONEREF) dst_release(dst) { dst_release(dst) {
" rcuref_put(&dst->__rcuref) { rcuref_put(&dst->__rcuref) {
" rcuref_put { rcuref_put {
" preempt_disable preempt_disable
" __rcuref_put { __rcuref_put {
" atomic_add_negative_release(-1, &ref->refcnt) ...
0xFFFFFFFF (RCUREF_NOREF) rcuref_put_slowpath { ...
" cnt = atomic_read(&ref->refcnt) ...
" ... atomic_add_negative_release(-1, &ref->refcnt)
0xFFFFFFFE (RCUREF_NOREF-1) atomic_try_cmpxchg_release(&ref->refcnt, &cnt, RCUREF_DEAD) ...
0xE0000000 (RCUREF_DEAD) } → false rcuref_put_slowpath {
" } → false cnt = atomic_read(&ref->refcnt)
" preempt_enable WARN_ONCE(cnt >= RCUREF_RELEASED, "rcuref - imbalanced put()")
" } → false atomic_set(&ref->refcnt, RCUREF_DEAD)
" } → false } → false
" } } → false
" preempt_enable
" } → false
" } → false
" }
That's the only new clue so far.
Plan is to add more logging to dump the skb that triggers it or at least
dig out the dst dev name. We need more hints.
Side note - annoyingly, rcuref_put_slowpath is notrace, same as every
*.o under lib/ by default. That makes tracing the "imbalanced put()"
branch impossible when running w/o CONFIG_KPROBE_EVENTS_ON_NOTRACE.
I will see if I can lift that restriction.
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-01-13 12:02 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-06-28 11:10 [FYI] Input route ref count underflow since probably 6.6.22 Jakub Sitnicki
2024-07-01 23:38 ` Jakub Kicinski
2024-07-02 14:21 ` Jakub Sitnicki
2025-01-13 12:02 ` Jakub Sitnicki
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).