* [PATCH net-next v13] net: reduce ARFS flow updates by checking LLC affinity
@ 2026-11-13 12:56 Chuang Wang
2026-11-17 13:24 ` netdev-bot+sashiko
0 siblings, 1 reply; 2+ messages in thread
From: Chuang Wang @ 2026-11-13 12:56 UTC (permalink / raw)
Cc: Chuang Wang, David S. Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Jonathan Corbet, Shuah Khan,
Stanislav Fomichev, Daniel Borkmann, Kuniyuki Iwashima, netdev,
linux-doc, linux-kernel
When an application migrates to a different CPU, RFS records the new
CPU in the global rps_sock_flow_table via rps_record_sock_flow(). On
the next packet of a flow, get_rps_cpu() notices the change and calls
set_rps_cpu() to update the per-queue flow entry. With Accelerated
RFS (ARFS), set_rps_cpu() also invokes the driver's ndo_rx_flow_steer()
callback so that the NIC steers the flow to a queue close to the new
CPU.
Threads that migrate frequently trigger this on every migration. For
drivers like mlx5, each steering call serialises on an internal lock
(arfs_lock), and under heavy load the contention on that lock can
outweigh the benefit of the steering update itself.
This patch introduces a cache-aware update strategy: the hardware
steering update is skipped when the flow's previous target CPU and the
new one share the same Last Level Cache (LLC), as cache locality for
the application is preserved either way. The per-queue flow bookkeeping
(rflow->cpu) is still updated, so RFS keeps steering packets to the CPU
the application is running on. A new sysctl, net.core.arfs_llc_affinity,
is added to toggle this feature (default: off).
Performance Test Results:
The patch was tested in a K8s environment (AMD CPU 128*2, 16-core Pod
with CPU pinning, mlx5 NIC) using brpc[1] echo_server and rpc_press.
rpc_press Commands:
for i in {1..8}; do
./rpc_press -proto=./echo.proto -method=example.EchoService.Echo
-server=<IP>:8000 -input='{"message":"hello"}'
-qps=0 -thread_num=512 -connection_type=pooled &
done
Monitor mlx5e_rx_flow_steer frequency:
/usr/share/bcc/tools/funccount -i 1 mlx5e_rx_flow_steer
Frequency of mlx5e_rx_flow_steer (via funccount[2]):
Before: ~200,000 counts/sec
After: ~10 counts/sec (reduced by ~99%)
[1] https://github.com/apache/brpc/
[2] https://github.com/iovisor/bcc/blob/master/tools/funccount.py
Signed-off-by: Chuang Wang <nashuiliang@gmail.com>
---
v13 -> v12:
- add rflow->filter != RPS_NO_FILTER check in set_rps_cpu, per Simon Horman
v11 -> v12:
- drop the comment block above the LLC check, per Jakub Kicinski
- guard the arfs_llc_affinity declaration, definition and sysctl entry
with CONFIG_RFS_ACCEL
- document the sysctl in Documentation/networking/scaling.rst, per Jakub Kicinski
v10 -> v11:
- rename the sysctl from net.core.rps_feat_llc_affinity to
net.core.arfs_llc_affinity, per Neal Cardwell
v9 -> v10:
- move the LLC check from rps_record_sock_flow() into set_rps_cpu(),
so that only the ARFS hardware steering is skipped while rflow->cpu
is still updated, per Eric Dumazet
- drop the sock_rps_record_flow_hash()/sock_rps_record_flow() exports,
which are no longer needed
v6 -> v9:
- simplify code and fix errors in AI submissions, per Simon Horman
v5 -> v6:
- remove the multi-check 'old_val == new_val', per Xuan Zhuo
- fix 'modpost: "sock_rps_record_flow_hash" [drivers/net/tun.ko]
undefined!', per kernel test robot
- fix 'tcp.c:(.text+0x3e90): undefined reference to
`sock_rps_record_flow'', per kernel test robot
v4 -> v5:
- fix 'modpost: "rps_llc_check" [net/sctp/sctp.ko] undefined!', per
kernel test robot
v3 -> v4:
- add rps_llc_check, per Eric Dumazet
v2 -> v3:
- patch net -> net-next, per Jakub Kicinski
v1 -> v2:
- add rps_feat_llc_affinity; add brpc tests
Documentation/networking/scaling.rst | 9 +++++++++
include/net/rps.h | 4 ++++
net/core/dev.c | 15 +++++++++++++--
net/core/sysctl_net_core.c | 9 +++++++++
4 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/Documentation/networking/scaling.rst b/Documentation/networking/scaling.rst
index 6c9836000a8d..d85fe7a9e9bf 100644
--- a/Documentation/networking/scaling.rst
+++ b/Documentation/networking/scaling.rst
@@ -468,6 +468,15 @@ of CPU to queues is automatically deduced from the IRQ affinities
configured for each receive queue by the driver, so no additional
configuration should be necessary.
+When the flow's previous target CPU and the new target CPU share the
+last level cache (LLC), the hardware steering update can be skipped,
+since the application's cache locality is preserved either way. This
+reduces contention on driver-internal locks when application threads
+migrate frequently between CPUs sharing an LLC. The functionality
+remains disabled until explicitly enabled through::
+
+ /proc/sys/net/core/arfs_llc_affinity
+
Suggested Configuration
~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/include/net/rps.h b/include/net/rps.h
index e33c6a2fa8bb..ff2f490a4e9a 100644
--- a/include/net/rps.h
+++ b/include/net/rps.h
@@ -13,6 +13,10 @@
extern struct static_key_false rps_needed;
extern struct static_key_false rfs_needed;
+#ifdef CONFIG_RFS_ACCEL
+extern struct static_key_false arfs_llc_affinity;
+#endif
+
/*
* This structure holds an RPS map which can be of variable length. The
* map is an array of CPUs.
diff --git a/net/core/dev.c b/net/core/dev.c
index 3b60b15e2175..6d3dd2d9ae82 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -5217,6 +5217,10 @@ EXPORT_SYMBOL(rps_needed);
struct static_key_false rfs_needed __read_mostly;
EXPORT_SYMBOL(rfs_needed);
+#ifdef CONFIG_RFS_ACCEL
+struct static_key_false arfs_llc_affinity __read_mostly;
+#endif
+
static u32 rfs_slot(u32 hash, rps_tag_ptr tag_ptr)
{
return hash_32(hash, rps_tag_to_log(tag_ptr));
@@ -5254,7 +5258,7 @@ static bool rps_flow_is_active(struct rps_dev_flow *rflow,
static struct rps_dev_flow *
set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
- struct rps_dev_flow *rflow, u16 next_cpu, u32 hash)
+ struct rps_dev_flow *rflow, u16 old_cpu, u16 next_cpu, u32 hash)
{
if (next_cpu < nr_cpu_ids) {
u32 head;
@@ -5273,6 +5277,13 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
if (!skb_rx_queue_recorded(skb) || !dev->rx_cpu_rmap ||
!(dev->features & NETIF_F_NTUPLE))
goto out;
+
+ if (static_branch_unlikely(&arfs_llc_affinity) &&
+ old_cpu < nr_cpu_ids && cpu_online(old_cpu) &&
+ cpus_share_cache(old_cpu, next_cpu) &&
+ READ_ONCE(rflow->filter) != RPS_NO_FILTER)
+ goto out;
+
rxq_index = cpu_rmap_lookup_index(dev->rx_cpu_rmap, next_cpu);
if (rxq_index == skb_get_rx_queue(skb))
goto out;
@@ -5402,8 +5413,8 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
(tcpu >= nr_cpu_ids || !cpu_online(tcpu) ||
((int)(READ_ONCE(per_cpu(softnet_data, tcpu).input_queue_head) -
rflow->last_qtail)) >= 0)) {
+ rflow = set_rps_cpu(dev, skb, rflow, tcpu, next_cpu, hash);
tcpu = next_cpu;
- rflow = set_rps_cpu(dev, skb, rflow, next_cpu, hash);
}
if (tcpu < nr_cpu_ids && cpu_online(tcpu)) {
diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c
index 9226b96f2e89..64ae4c07f1b1 100644
--- a/net/core/sysctl_net_core.c
+++ b/net/core/sysctl_net_core.c
@@ -562,6 +562,15 @@ static struct ctl_table net_core_table[] = {
.mode = 0644,
.proc_handler = rps_sock_flow_sysctl
},
+#ifdef CONFIG_RFS_ACCEL
+ {
+ .procname = "arfs_llc_affinity",
+ .data = &arfs_llc_affinity.key,
+ .maxlen = sizeof(arfs_llc_affinity.key),
+ .mode = 0644,
+ .proc_handler = proc_do_static_key
+ },
+#endif
#endif
#ifdef CONFIG_NET_FLOW_LIMIT
{
--
2.47.3
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH net-next v13] net: reduce ARFS flow updates by checking LLC affinity
2026-11-13 12:56 [PATCH net-next v13] net: reduce ARFS flow updates by checking LLC affinity Chuang Wang
@ 2026-11-17 13:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 2+ messages in thread
From: netdev-bot+sashiko @ 2026-11-17 13:24 UTC (permalink / raw)
To: nashuiliang
Cc: davem, edumazet, kuba, pabeni, horms, corbet, skhan, sdf.kernel,
daniel, kuniyu, netdev, linux-doc, linux-kernel
Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 1
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] With net.core.arfs_llc_affinity enabled, the new early `goto
out` in set_rps_cpu() (net/core/dev.c:5301-5305) can skip…
- [Low] Commit message / documentation completeness (baseline: not
confirmed as a defect).
--
net: reduce ARFS flow updates by checking LLC affinity
This patch adds a default-off sysctl, net.core.arfs_llc_affinity. When it
is enabled, set_rps_cpu() does not call ndo_rx_flow_steer() if the flow's
previous RFS target CPU and the new one share an LLC. The per-queue
rflow->cpu is still updated, so software RFS keeps following the
application.
> diff --git a/Documentation/networking/scaling.rst b/Documentation/networking/scaling.rst
> index 6c9836000a8d1..d85fe7a9e9bfb 100644
> --- a/Documentation/networking/scaling.rst
> +++ b/Documentation/networking/scaling.rst
> @@ -468,6 +468,15 @@ of CPU to queues is automatically deduced from the IRQ affinities
> configured for each receive queue by the driver, so no additional
> configuration should be necessary.
>
> +When the flow's previous target CPU and the new target CPU share the
> +last level cache (LLC), the hardware steering update can be skipped,
> +since the application's cache locality is preserved either way. This
> +reduces contention on driver-internal locks when application threads
> +migrate frequently between CPUs sharing an LLC. The functionality
> +remains disabled until explicitly enabled through::
> +
> + /proc/sys/net/core/arfs_llc_affinity
> +
[Severity: Low]
This isn't a bug, but the text gives the condition only as the previous
and new target CPU sharing the LLC. The code in set_rps_cpu() has two more
requirements: old_cpu must be online, and rflow->filter must not be
RPS_NO_FILTER.
Could the documentation mention these? It could also say that after a
skip, packets keep arriving on the old queue and are RPS-forwarded to the
application CPU.
[ ... ]
> diff --git a/net/core/dev.c b/net/core/dev.c
> index a230ecce9327d..15a5c7caaf8e8 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
[ ... ]
> @@ -5274,7 +5278,7 @@ static bool rps_flow_is_active(struct rps_dev_flow *rflow,
>
> static struct rps_dev_flow *
> set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
> - struct rps_dev_flow *rflow, u16 next_cpu, u32 hash)
> + struct rps_dev_flow *rflow, u16 old_cpu, u16 next_cpu, u32 hash)
> {
> if (next_cpu < nr_cpu_ids) {
> u32 head;
> @@ -5293,6 +5297,13 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
> if (!skb_rx_queue_recorded(skb) || !dev->rx_cpu_rmap ||
> !(dev->features & NETIF_F_NTUPLE))
> goto out;
> +
> + if (static_branch_unlikely(&arfs_llc_affinity) &&
> + old_cpu < nr_cpu_ids && cpu_online(old_cpu) &&
> + cpus_share_cache(old_cpu, next_cpu) &&
> + READ_ONCE(rflow->filter) != RPS_NO_FILTER)
> + goto out;
> +
[Severity: Medium]
Can this skip ndo_rx_flow_steer() even when the NIC is not delivering the
flow anywhere near next_cpu?
old_cpu is tcpu from get_rps_cpu(), which is rflow->cpu, the last software
RFS target. That is not necessarily the CPU that services the queue the
hardware filter delivers to.
The rflow->filter != RPS_NO_FILTER test is also not checked against
rflow->hash or rps_flow_is_active(). The existing tmp_rflow check further
down in set_rps_cpu() does both.
set_rps_cpu() writes rflow->cpu on every exit, including the paths where
no hardware steering happened:
q_tag_ptr = READ_ONCE(rxqueue->rps_flow_table);
if (!q_tag_ptr)
goto out;
...
if (hash != READ_ONCE(tmp_rflow->hash) ||
next_cpu == tmp_cpu)
goto out;
...
rc = dev->netdev_ops->ndo_rx_flow_steer(dev, skb,
rxq_index, flow_id);
if (rc < 0)
goto out;
...
out:
...
WRITE_ONCE(rflow->cpu, next_cpu);
On those paths rflow is still the arrival queue's entry. Its filter keeps
the old filter id, but its cpu moves to the new CPU. For example,
mlx5e_rx_flow_steer() can return -ENOMEM, -EPERM or -EPROTONOSUPPORT.
Consider this sequence:
1. Flow F has filter f steering it to queue Q near LLC-A. Q's entry is
{filter = f, cpu = cA}.
2. The application moves to cB in LLC-B. cpus_share_cache(cA, cB) is
false, so steering towards rmap(cB) is attempted, but it fails on one
of the paths above. Q's entry becomes {filter = f, cpu = cB}, while
the NIC still delivers F to Q.
3. The application moves to cB' in LLC-B. cpus_share_cache(cB, cB') is
true and filter == f, so steering is skipped.
4. get_rps_cpu() only calls set_rps_cpu() when rflow->cpu != next_cpu, so
every later move inside LLC-B is skipped as well.
After that, would every packet be RPS-forwarded from Q's CPU in LLC-A to
the application in LLC-B, with the hardware filter never retried? Before
this patch, each migration retried steering. A retry could succeed once
the colliding filter had expired or memory was available again.
The filter id can also be stale or belong to another flow:
- RPS_NO_FILTER is only written at table allocation and when
old_rflow->filter == rc.
- rps_may_expire_flow() only reports expiry to the driver. Nothing clears
rflow->filter after the driver deletes the rule.
- A colliding flow's filter in the same rfs_slot() would also pass this
check.
Changing IRQ affinity also changes the queue-to-CPU mapping without
touching rflow->cpu.
This seems to conflict with the claim in the commit message and
scaling.rst that "cache locality for the application is preserved either
way".
Would it make sense to also require READ_ONCE(rflow->hash) == hash and an
activity check? Another option is to compare next_cpu with the CPU that
services the arrival queue (skb_get_rx_queue(skb)), or with
smp_processor_id(), instead of rflow->cpu.
> rxq_index = cpu_rmap_lookup_index(dev->rx_cpu_rmap, next_cpu);
> if (rxq_index == skb_get_rx_queue(skb))
> goto out;
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261113125643.2255-1-nashuiliang%40gmail.com
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-11-17 13:24 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-13 12:56 [PATCH net-next v13] net: reduce ARFS flow updates by checking LLC affinity Chuang Wang
2026-11-17 13:24 ` netdev-bot+sashiko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox