[syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    5ce74bc1b7cb Add linux-next specific files for 20251211
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=11a241b4580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=a94030c847137a18
dashboard link: https://syzkaller.appspot.com/bug?extid=4393c47753b7808dac7d
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/e1660176a50f/disk-5ce74bc1.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/616ede012bbd/vmlinux-5ce74bc1.xz
kernel image: https://storage.googleapis.com/syzbot-assets/7c0c5bc40fd8/bzImage-5ce74bc1.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+4393c47753b7808dac7d@syzkaller.appspotmail.com

------------[ cut here ]------------
conntrack cleanup blocked for 60s
WARNING: net/netfilter/nf_conntrack_core.c:2512 at nf_conntrack_cleanup_net_list+0x234/0x340 net/netfilter/nf_conntrack_core.c:2511, CPU#1: kworker/u8:0/12
Modules linked in:
CPU: 1 UID: 0 PID: 12 Comm: kworker/u8:0 Tainted: G             L      syzkaller #0 PREEMPT(full) 
Tainted: [L]=SOFTLOCKUP
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/25/2025
Workqueue: netns cleanup_net
RIP: 0010:nf_conntrack_cleanup_net_list+0x234/0x340 net/netfilter/nf_conntrack_core.c:2511
Code: 08 48 89 df e8 fd 78 a3 f8 4c 8b 3b 49 39 df 74 69 e8 20 16 3d f8 45 31 e4 e9 8e fe ff ff e8 13 16 3d f8 48 8d 3d ec 2f 0c 06 <67> 48 0f b9 3a eb c0 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c cd fe ff
RSP: 0018:ffffc90000117870 EFLAGS: 00010293
RAX: ffffffff8984a13d RBX: ffffc90000117a00 RCX: ffff88801ce8db80
RDX: 0000000000000000 RSI: ffffffffffffffcb RDI: ffffffff8f90d130
RBP: 0000000000000001 R08: ffff88807ce6b003 R09: 1ffff1100f9cd600
R10: dffffc0000000000 R11: ffffed100f9cd601 R12: 0000000000000001
R13: dffffc0000000000 R14: 00000001000068e3 R15: 0000000100006918
FS:  0000000000000000(0000) GS:ffff888125f32000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000020000 CR3: 00000000306ac000 CR4: 00000000003526f0
Call Trace:
 <TASK>
 ops_exit_list net/core/net_namespace.c:205 [inline]
 ops_undo_list+0x525/0x990 net/core/net_namespace.c:252
 cleanup_net+0x4d8/0x7a0 net/core/net_namespace.c:696
 process_one_work+0x93a/0x15a0 kernel/workqueue.c:3279
 process_scheduled_works kernel/workqueue.c:3362 [inline]
 worker_thread+0x9b0/0xee0 kernel/workqueue.c:3443
 kthread+0x711/0x8a0 kernel/kthread.c:463
 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
 </TASK>
----------------
Code disassembly (best guess):
   0:	08 48 89             	or     %cl,-0x77(%rax)
   3:	df e8                	fucomip %st(0),%st
   5:	fd                   	std
   6:	78 a3                	js     0xffffffab
   8:	f8                   	clc
   9:	4c 8b 3b             	mov    (%rbx),%r15
   c:	49 39 df             	cmp    %rbx,%r15
   f:	74 69                	je     0x7a
  11:	e8 20 16 3d f8       	call   0xf83d1636
  16:	45 31 e4             	xor    %r12d,%r12d
  19:	e9 8e fe ff ff       	jmp    0xfffffeac
  1e:	e8 13 16 3d f8       	call   0xf83d1636
  23:	48 8d 3d ec 2f 0c 06 	lea    0x60c2fec(%rip),%rdi        # 0x60c3016
* 2a:	67 48 0f b9 3a       	ud1    (%edx),%rdi <-- trapping instruction
  2f:	eb c0                	jmp    0xfffffff1
  31:	89 e9                	mov    %ebp,%ecx
  33:	80 e1 07             	and    $0x7,%cl
  36:	80 c1 03             	add    $0x3,%cl
  39:	38 c1                	cmp    %al,%cl
  3b:	0f                   	.byte 0xf
  3c:	8c cd                	mov    %cs,%ebp
  3e:	fe                   	(bad)
  3f:	ff                   	.byte 0xff


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[Jakub Kicinski]

On Thu, 11 Dec 2025 10:38:31 -0800 syzbot wrote:
> ------------[ cut here ]------------
> conntrack cleanup blocked for 60s
> WARNING: net/netfilter/nf_conntrack_core.c:2512 at

Yes, I was about to comment on the patch which added the warning..

There is still a leak somewhere. Running ip_defrag.sh and then load /
unload ipvlan repros this (modprobe ipvlan is a quick check if the
cleanup thread is wedged, if it is modprobe will hang, if it isn't
run ip_defrag.sh, again etc).

I looked around last night but couldn't find an skb stuck anywhere.
The nf_conntrack_net->count was == 1

Re: [syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[Florian Westphal]

Jakub Kicinski <kuba@kernel.org> wrote:
> On Thu, 11 Dec 2025 10:38:31 -0800 syzbot wrote:
> > ------------[ cut here ]------------
> > conntrack cleanup blocked for 60s
> > WARNING: net/netfilter/nf_conntrack_core.c:2512 at
> 
> Yes, I was about to comment on the patch which added the warning..
> 
> There is still a leak somewhere. Running ip_defrag.sh and then load /
> unload ipvlan repros this (modprobe ipvlan is a quick check if the
> cleanup thread is wedged, if it is modprobe will hang, if it isn't
> run ip_defrag.sh, again etc).
> 
> I looked around last night but couldn't find an skb stuck anywhere.
> The nf_conntrack_net->count was == 1

Its caused skb skb fraglist skbs that still hold nf_conn references
on the softnet data defer lists.

setting net.core.skb_defer_max=0 makes the hang disappear for me.

Re: [syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[Eric Dumazet]

On Sat, Dec 13, 2025 at 2:27 PM Florian Westphal <fw@strlen.de> wrote:
>
> Jakub Kicinski <kuba@kernel.org> wrote:
> > On Thu, 11 Dec 2025 10:38:31 -0800 syzbot wrote:
> > > ------------[ cut here ]------------
> > > conntrack cleanup blocked for 60s
> > > WARNING: net/netfilter/nf_conntrack_core.c:2512 at
> >
> > Yes, I was about to comment on the patch which added the warning..
> >
> > There is still a leak somewhere. Running ip_defrag.sh and then load /
> > unload ipvlan repros this (modprobe ipvlan is a quick check if the
> > cleanup thread is wedged, if it is modprobe will hang, if it isn't
> > run ip_defrag.sh, again etc).
> >
> > I looked around last night but couldn't find an skb stuck anywhere.
> > The nf_conntrack_net->count was == 1
>
> Its caused skb skb fraglist skbs that still hold nf_conn references
> on the softnet data defer lists.
>
> setting net.core.skb_defer_max=0 makes the hang disappear for me.

What kind of packets ? TCP ones ?

Re: [syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[Florian Westphal]

Eric Dumazet <edumazet@google.com> wrote:
> > > I looked around last night but couldn't find an skb stuck anywhere.
> > > The nf_conntrack_net->count was == 1
> >
> > Its caused skb skb fraglist skbs that still hold nf_conn references
> > on the softnet data defer lists.
> >
> > setting net.core.skb_defer_max=0 makes the hang disappear for me.
> 
> What kind of packets ? TCP ones ?

UDP, but I can't say yet if thats an udp specific issue or not.
(the packets are generated via ip_defrag.c).

Re: [syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[Eric Dumazet]

On Sat, Dec 13, 2025 at 2:40 PM Florian Westphal <fw@strlen.de> wrote:
>
> Eric Dumazet <edumazet@google.com> wrote:
> > > > I looked around last night but couldn't find an skb stuck anywhere.
> > > > The nf_conntrack_net->count was == 1
> > >
> > > Its caused skb skb fraglist skbs that still hold nf_conn references
> > > on the softnet data defer lists.
> > >
> > > setting net.core.skb_defer_max=0 makes the hang disappear for me.
> >
> > What kind of packets ? TCP ones ?
>
> UDP, but I can't say yet if thats an udp specific issue or not.
> (the packets are generated via ip_defrag.c).

skb_release_head_state() does not follow the fraglist. Oh well.

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index a00808f7be6a1b86c595183f8b131996e3d0afcc..f597769d8c206dc063b53938a18edbe9620101d9
100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1497,7 +1497,9 @@ void napi_consume_skb(struct sk_buff *skb, int budget)

        DEBUG_NET_WARN_ON_ONCE(!in_softirq());

-       if (skb->alloc_cpu != smp_processor_id() && !skb_shared(skb)) {
+       if (skb->alloc_cpu != smp_processor_id() &&
+           !skb_shared(skb) &&
+           !skb_has_frag_list(skb)) {
                skb_release_head_state(skb);
                return skb_attempt_defer_free(skb);
        }

Re: [syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[Florian Westphal]

Eric Dumazet <edumazet@google.com> wrote:
> > UDP, but I can't say yet if thats an udp specific issue or not.
> > (the packets are generated via ip_defrag.c).
> 
> skb_release_head_state() does not follow the fraglist. Oh well.
> 
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index a00808f7be6a1b86c595183f8b131996e3d0afcc..f597769d8c206dc063b53938a18edbe9620101d9
> 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -1497,7 +1497,9 @@ void napi_consume_skb(struct sk_buff *skb, int budget)
> 
>         DEBUG_NET_WARN_ON_ONCE(!in_softirq());
> 
> -       if (skb->alloc_cpu != smp_processor_id() && !skb_shared(skb)) {
> +       if (skb->alloc_cpu != smp_processor_id() &&
> +           !skb_shared(skb) &&
> +           !skb_has_frag_list(skb)) {
>                 skb_release_head_state(skb);
>                 return skb_attempt_defer_free(skb);

There is also:
skb_attempt_defer_free -> skb_attempt_defer_free

Alternatively we could export skb_defer_free_flush or
kick_defer_list_purge() and call that from nf_conntrack
net exit path.

I will investigate more closely on monday, I still don't
understand why fragments are conntracked in the first place.