* [6.6.y] ipv6: make addrconf_wq single threaded
@ 2025-07-18 10:16 Brett A C Sheffield
2025-07-18 10:22 ` [PATCH] " Brett A C Sheffield
2025-07-21 13:55 ` [6.6.y] " Greg KH
0 siblings, 2 replies; 3+ messages in thread
From: Brett A C Sheffield @ 2025-07-18 10:16 UTC (permalink / raw)
To: stable; +Cc: edumazet, kuba, peterz, mingo
TL;DR - please backport
dfd2ee086a63c730022cb095576a8b3a5a752109 "ipv6: make addrconf_wq single threaded"
to 6.6.y (only).
When testing the stable and LTS kernels this week I noticed one of our tests was failing on the 6.6.y kernels and passing on both newer and older stable kernels as well as mainline.
It is not a perfect test. Running the test in a tight loop, it would occasionally fail on other kernels after thousands of iterations (6k-10k). However, on 6.6.y it usually fails on the first or second run, so I thought it was worth digging a bit.
I've bisected the behaviour change back to:
86bfbb7ce4f67a88df2639198169b685668e7349 "sched/fair: Add lag based placement"
The regression is later fixed by completely unrelated commit:
dfd2ee086a63c730022cb095576a8b3a5a752109 "ipv6: make addrconf_wq single threaded"
ie. a change to the scheduler caused the change in behaviour, and changing ipv6 addrconf to use a single-threaded workqueue seems to have "fixed" it.
The original test is creating three tap interfaces, binding a socket to each and sending a packet on each. It's a simple test just to check that binding a socket to an interface works with our API.
I wrote a simplified test, which just:
- creates a single TAP interface, disables DAD, and brings it up
- times how long before a single IPv6 packet can be sent without error
These are the results for the commit (e0c2ff903c32) before the change:
2025-07-18 07:01:33-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:01:35-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:01:37-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:01:39-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:01:40-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 2306 tries (0.00296s)
2025-07-18 07:01:47-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:01:49-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:01:50-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:01:52-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:01:53-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:01:59-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:02:00-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:02:01-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:05:03-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:05:05-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 2042 tries (0.00266s)
2025-07-18 07:05:07-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:05:09-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:05:11-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:05:12-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00004s)
2025-07-18 07:05:15-00:00 6.5.0-rc2-00017-ge0c2ff903c32 8 bytes sent after 1 tries (0.00002s)
ie. sending mostly succeeds on the first attempt after TAP creation, with occasional outliers.
With 86bfbb7ce4f6 applied, it's not so good:
2025-07-18 09:25:00-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2184 tries (0.00285s)
2025-07-18 09:25:02-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2597 tries (0.00336s)
2025-07-18 09:25:03-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2579 tries (0.00332s)
2025-07-18 09:25:05-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2045 tries (0.00261s)
2025-07-18 09:25:06-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2333 tries (0.00305s)
2025-07-18 09:25:08-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2847 tries (0.00355s)
2025-07-18 09:25:09-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2267 tries (0.00306s)
2025-07-18 09:25:11-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2731 tries (0.00343s)
2025-07-18 09:25:12-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2506 tries (0.00327s)
2025-07-18 09:25:14-00:00 6.5.0-rc2-00018-g86bfbb7ce4f6 8 bytes sent after 2554 tries (0.00338s)
If we apply dfd2ee086a63 here, we see:
2025-07-18 07:06:19-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:06:22-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 1 tries (0.00005s)
2025-07-18 07:06:25-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:06:27-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 44 tries (0.00013s)
2025-07-18 07:06:32-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:06:34-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 42 tries (0.00013s)
2025-07-18 07:06:36-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:06:39-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 32 tries (0.00014s)
2025-07-18 07:06:41-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:06:43-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 67 tries (0.00012s)
2025-07-18 07:06:45-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:06:47-00:00 6.5.0-rc2-00019-gdf3c40c549d8 8 bytes sent after 44 tries (0.00013s)
6.6.98:
2025-07-18 07:09:58-00:00 6.6.98 8 bytes sent after 2107 tries (0.00273s)
2025-07-18 07:10:03-00:00 6.6.98 8 bytes sent after 2597 tries (0.00350s)
2025-07-18 07:10:05-00:00 6.6.98 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:10:07-00:00 6.6.98 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:10:10-00:00 6.6.98 8 bytes sent after 2714 tries (0.00343s)
2025-07-18 07:10:12-00:00 6.6.98 8 bytes sent after 2635 tries (0.00330s)
2025-07-18 07:10:15-00:00 6.6.98 8 bytes sent after 1939 tries (0.00247s)
2025-07-18 07:10:17-00:00 6.6.98 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:10:20-00:00 6.6.98 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:10:22-00:00 6.6.98 8 bytes sent after 2688 tries (0.00343s)
2025-07-18 07:10:24-00:00 6.6.98 8 bytes sent after 2586 tries (0.00334s)
2025-07-18 07:10:26-00:00 6.6.98 8 bytes sent after 2526 tries (0.00335s)
2025-07-18 07:10:28-00:00 6.6.98 8 bytes sent after 2411 tries (0.00311s)
6.6.98 with dfd2ee086a63 applied goes back to much more consistent behaviour:
2025-07-18 08:29:52-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 81 tries (0.00015s)
2025-07-18 08:29:55-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 46 tries (0.00013s)
2025-07-18 08:29:57-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 54 tries (0.00012s)
2025-07-18 08:29:59-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 44 tries (0.00013s)
2025-07-18 08:30:00-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 71 tries (0.00014s)
2025-07-18 08:30:02-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 44 tries (0.00014s)
2025-07-18 08:30:04-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 73 tries (0.00013s)
2025-07-18 08:30:05-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 46 tries (0.00014s)
2025-07-18 08:30:07-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 73 tries (0.00013s)
2025-07-18 08:30:09-00:00 6.6.98-00001-g9a37684fd14f 8 bytes sent after 46 tries (0.00013s)
6.12.39:
2025-07-18 07:20:21-00:00 6.12.39 8 bytes sent after 100 tries (0.00015s)
2025-07-18 07:20:24-00:00 6.12.39 8 bytes sent after 54 tries (0.00014s)
2025-07-18 07:20:26-00:00 6.12.39 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:20:27-00:00 6.12.39 8 bytes sent after 53 tries (0.00014s)
2025-07-18 07:20:28-00:00 6.12.39 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:20:30-00:00 6.12.39 8 bytes sent after 56 tries (0.00013s)
2025-07-18 07:20:31-00:00 6.12.39 8 bytes sent after 1 tries (0.00003s)
2025-07-18 07:20:32-00:00 6.12.39 8 bytes sent after 94 tries (0.00012s)
2025-07-18 07:20:33-00:00 6.12.39 8 bytes sent after 1 tries (0.00002s)
2025-07-18 07:20:34-00:00 6.12.39 8 bytes sent after 60 tries (0.00013s)
mainline:
2025-07-18 06:57:42-00:00 6.16.0-rc6 8 bytes sent after 1 tries (0.0000s)
2025-07-18 06:57:44-00:00 6.16.0-rc6 8 bytes sent after 31 tries (0.0001s)
2025-07-18 06:57:46-00:00 6.16.0-rc6 8 bytes sent after 70 tries (0.0001s)
2025-07-18 06:57:48-00:00 6.16.0-rc6 8 bytes sent after 69 tries (0.0001s)
2025-07-18 06:57:49-00:00 6.16.0-rc6 8 bytes sent after 3 tries (0.0000s)
2025-07-18 06:57:51-00:00 6.16.0-rc6 8 bytes sent after 68 tries (0.0001s)
2025-07-18 06:59:31-00:00 6.16.0-rc6 8 bytes sent after 1 tries (0.00003s)
2025-07-18 06:59:35-00:00 6.16.0-rc6 8 bytes sent after 34 tries (0.00014s)
2025-07-18 06:59:36-00:00 6.16.0-rc6 8 bytes sent after 1 tries (0.00003s)
2025-07-18 06:59:38-00:00 6.16.0-rc6 8 bytes sent after 1 tries (0.00003s)
2025-07-18 06:59:49-00:00 6.16.0-rc6 8 bytes sent after 73 tries (0.00014s)
2025-07-18 06:59:51-00:00 6.16.0-rc6 8 bytes sent after 52 tries (0.00012s)
2025-07-18 06:59:51-00:00 6.16.0-rc6 8 bytes sent after 50 tries (0.00012s)
As dfd2ee086a63c730022cb095576a8b3a5a752109 is a single line change and clearly brings behaviour back in line with other kernels we should backport this to 6.6.y.
Cheers,
Brett
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH] ipv6: make addrconf_wq single threaded
2025-07-18 10:16 [6.6.y] ipv6: make addrconf_wq single threaded Brett A C Sheffield
@ 2025-07-18 10:22 ` Brett A C Sheffield
2025-07-21 13:55 ` [6.6.y] " Greg KH
1 sibling, 0 replies; 3+ messages in thread
From: Brett A C Sheffield @ 2025-07-18 10:22 UTC (permalink / raw)
To: bacs; +Cc: edumazet, kuba, mingo, peterz, stable, David Ahern
From: Brett A C Sheffield (Librecast) <bacs@librecast.net>
[ Upstream commit dfd2ee086a63c730022cb095576a8b3a5a752109 ]
Both addrconf_verify_work() and addrconf_dad_work() acquire rtnl,
there is no point trying to have one thread per cpu.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: David Ahern <dsahern@kernel.org>
Link: https://lore.kernel.org/r/20240201173031.3654257-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Brett A C Sheffield <bacs@librecast.net>
---
net/ipv6/addrconf.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index 231fa4dc6cde..9fc511c0c3c4 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -7388,7 +7388,8 @@ int __init addrconf_init(void)
if (err < 0)
goto out_addrlabel;
- addrconf_wq = create_workqueue("ipv6_addrconf");
+ /* All works using addrconf_wq need to lock rtnl. */
+ addrconf_wq = create_singlethread_workqueue("ipv6_addrconf");
if (!addrconf_wq) {
err = -ENOMEM;
goto out_nowq;
--
2.49.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [6.6.y] ipv6: make addrconf_wq single threaded
2025-07-18 10:16 [6.6.y] ipv6: make addrconf_wq single threaded Brett A C Sheffield
2025-07-18 10:22 ` [PATCH] " Brett A C Sheffield
@ 2025-07-21 13:55 ` Greg KH
1 sibling, 0 replies; 3+ messages in thread
From: Greg KH @ 2025-07-21 13:55 UTC (permalink / raw)
To: Brett A C Sheffield; +Cc: stable, edumazet, kuba, peterz, mingo
On Fri, Jul 18, 2025 at 10:16:04AM +0000, Brett A C Sheffield wrote:
> TL;DR - please backport
> dfd2ee086a63c730022cb095576a8b3a5a752109 "ipv6: make addrconf_wq single threaded"
> to 6.6.y (only).
Now done, thanks!
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2025-07-21 13:55 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-07-18 10:16 [6.6.y] ipv6: make addrconf_wq single threaded Brett A C Sheffield
2025-07-18 10:22 ` [PATCH] " Brett A C Sheffield
2025-07-21 13:55 ` [6.6.y] " Greg KH
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).