Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

["Heyne, Maximilian" <mheyne@amazon.de>]

On Fri, Jul 04, 2025 at 05:48:18AM +0000, Kuniyuki Iwashima wrote:
> Netlink has this pattern in some places
> 
>   if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
>   	atomic_add(skb->truesize, &sk->sk_rmem_alloc);
> 
> , which has the same problem fixed by commit 5a465a0da13e ("udp:
> Fix multiple wraparounds of sk->sk_rmem_alloc.").
> 
> For example, if we set INT_MAX to SO_RCVBUFFORCE, the condition
> is always false as the two operands are of int.
> 
> Then, a single socket can eat as many skb as possible until OOM
> happens, and we can see multiple wraparounds of sk->sk_rmem_alloc.
> 
> Let's fix it by using atomic_add_return() and comparing the two
> variables as unsigned int.
> 
> Before:
>   [root@fedora ~]# ss -f netlink
>   Recv-Q      Send-Q Local Address:Port                Peer Address:Port
>   -1668710080 0               rtnl:nl_wraparound/293               *
> 
> After:
>   [root@fedora ~]# ss -f netlink
>   Recv-Q     Send-Q Local Address:Port                Peer Address:Port
>   2147483072 0               rtnl:nl_wraparound/290               *
>   ^
>   `--- INT_MAX - 576
> 
> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> Reported-by: Jason Baron <jbaron@akamai.com>
> Closes: https://lore.kernel.org/netdev/cover.1750285100.git.jbaron@akamai.com/
> Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>

Hi Kuniyuki,

We're seeing soft lockups with this patch in a variety of (stable)
kernel versions.

I was able to reproduce it on a couple of different EC2 instances also
with the latest linux kernel 6.16-rc7 using the following steps:

systemctl start auditd
sudo auditctl -D
sudo auditctl -b 512
sudo auditctl -a always,exit -F arch=b64 -S mmap,munmap,mprotect,brk -k memory_ops
sudo auditctl -e 1

Then execute some programs that call some of these memory functions,
such as repeated calls of "sudo auditctl -s" or logging in via SSH.

These settings are set in a way to create a lot audit messages. Once the
backlog (see auditctl -s) overshoots the backlog_limit, the system soft
lockups:

[  460.056244] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kauditd:32]
[  460.056249] Modules linked in: mousedev(E) nls_ascii(E) nls_cp437(E) sunrpc(E) vfat(E) fat(E) psmouse(E) atkbd(E) libps2(E) vivaldi_fmap(E) i8042(E) serio(E) skx_edac_common(E) button(E) ena(E) ghash_clmulni_intel(E) sch_fq_codel(E) drm(E) i2c_core(E) dm_mod(E) drm_panel_orientation_quirks(E) backlight(E) fuse(E) loop(E) dax(E) configfs(E) dmi_sysfs(E) efivarfs(E)
[  460.056272] CPU: 1 UID: 0 PID: 32 Comm: kauditd Tainted: G            EL      6.16.0-rc7+ #3 PREEMPT(none)
[  460.056275] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[  460.056276] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
[  460.056277] RIP: 0010:_raw_spin_unlock_irqrestore+0x1b/0x30
[  460.056284] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 e8 16 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d b5 23 b6 01 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 e9 14 23 00 00 0f 1f 40 00 90
[  460.056285] RSP: 0018:ffffb762c0123d70 EFLAGS: 00000246
[  460.056287] RAX: 0000000000000001 RBX: ffff9b14c08eafc0 RCX: ffff9b14c3337348
[  460.056288] RDX: ffff9b14c3337348 RSI: 0000000000000246 RDI: ffff9b14c3337340
[  460.056289] RBP: ffff9b14c3337000 R08: ffffffff93cea880 R09: 0000000000000001
[  460.056290] R10: 0000000000000001 R11: 0000000000000080 R12: ffff9b14c1b72500
[  460.056291] R13: ffffb762c0123de0 R14: ffff9b14c3337340 R15: ffff9b14c3337080
[  460.056294] FS:  0000000000000000(0000) GS:ffff9b1563788000(0000) knlGS:0000000000000000
[  460.056296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  460.056297] CR2: 00007f36fd5d21b4 CR3: 000000010241a002 CR4: 00000000007706f0
[  460.056298] PKRU: 55555554
[  460.056299] Call Trace:
[  460.056300]  <TASK>
[  460.056302]  netlink_attachskb+0xb7/0x2f0
[  460.056308]  ? __pfx_default_wake_function+0x10/0x10
[  460.056313]  netlink_unicast+0xea/0x3b0
[  460.056315]  kauditd_send_queue+0xaf/0x170
[  460.056318]  ? __pfx_kauditd_send_multicast_skb+0x10/0x10
[  460.056320]  ? __pfx_kauditd_retry_skb+0x10/0x10
[  460.056321]  kauditd_thread+0x132/0x2b0
[  460.056323]  ? __pfx_autoremove_wake_function+0x10/0x10
[  460.056327]  ? __pfx_kauditd_thread+0x10/0x10
[  460.056328]  kthread+0xfb/0x230
[  460.056331]  ? __pfx_kthread+0x10/0x10
[  460.056332]  ? __pfx_kthread+0x10/0x10
[  460.056334]  ret_from_fork+0x142/0x160
[  460.056338]  ? __pfx_kthread+0x10/0x10
[  460.056339]  ret_from_fork_asm+0x1a/0x30
[  460.056343]  </TASK>
[  469.011800] audit_log_start: 120 callbacks suppressed
[  469.011805] audit: audit_backlog=513 > audit_backlog_limit=512
[  469.013154] audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=512
[  469.013967] audit: backlog limit exceeded
[  469.014617] audit: audit_backlog=513 > audit_backlog_limit=512
[  469.015313] audit: audit_lost=2 audit_rate_limit=0 audit_backlog_limit=512
[  469.016112] audit: backlog limit exceeded

We've bug reports from many users, so the issue is rather wide-spread.

So far I don't know why the commit is causing this issue and will keep
investigating. However, when reverted (together with its 2 follow-up
patches), the issue goes away and host do not lock up.

Thanks,
Maximilian



Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597