* [PATCH] net: dst: Avoid false sharing between tclassid and __rcuref
@ 2026-11-16 1:57 Jiebin Sun
2026-11-18 16:58 ` netdev-bot+sashiko
2026-11-18 17:15 ` Eric Dumazet
0 siblings, 2 replies; 3+ messages in thread
From: Jiebin Sun @ 2026-11-16 1:57 UTC (permalink / raw)
To: netdev
Cc: davem, edumazet, kuba, pabeni, horms, linux-kernel, tim.c.chen,
tianyou.li, wangyang.guo, jiebin.sun
On x86-64, dst_entry::tclassid shares the same cache line with
dst_entry::__rcuref. Updates to __rcuref cause false sharing with
tclassid reads, for example in ip_rcv_finish_core() when
CONFIG_IP_ROUTE_CLASSID=y. Memcached perf c2c shows the paths:
read: net_rx_action -> process_backlog -> ip_rcv_finish_core
-> skb_dst(skb)->tclassid (load)
acquire: tcp_write_xmit -> __tcp_transmit_skb -> __dev_queue_xmit
-> skb_dst_force -> dst_hold_safe -> rcuref_get
release: tcp_rcv_established/tcp_add_backlog -> tcp_cleanup_skb
-> skb_dst_drop -> dst_release -> rcuref_put
Move tclassid after lwtstate on 64-bit to avoid false sharing with
__rcuref.
On x86-64, struct dst_entry grows from 136 to 144 bytes. struct
rtable and struct rt6_info embed dst_entry, so their sizes grow
from 176 to 184 and from 224 to 232 bytes, respectively. However,
both struct rtable and struct rt6_info are allocated from slab
caches with SLAB_HWCACHE_ALIGN, so the per-object allocation sizes
remain unchanged at 192 bytes for rtable and 256 bytes for rt6_info.
Memcached perf c2c samples at the class-ID load:
+----------+------------+-------------+
| Kernel | Local HITM | Remote HITM |
+----------+------------+-------------+
| Baseline | 894 | 680 |
| Patched | 15 | 0 |
+----------+------------+-------------+
Results with 64 workers per endpoint (baseline 6628e7d4c6b2).
Improvement is 100 * (patched average / baseline average - 1).
Memcached (5 iterations, average operations/s):
+--------------------+--------------------+
| Case | Improvement (%) |
+--------------------+--------------------+
| default | +27.20 |
| hit | +52.36 |
| balanced | -0.52 |
| large | +64.68 |
+--------------------+--------------------+
Memcached over IPv4 loopback.
The default client command is:
memtier_benchmark -s 127.0.0.1 -p 11221 -P memcache_text
-t 64 -c 1 --pipeline=16 --test-time=60 --ratio=1:100
--data-size=32 --key-maximum=10000000
hit: --key-maximum=1000
balanced: --key-maximum=1000 --ratio=1:1
large: --key-maximum=1000 --data-size=1024
TCP request/reply (5 iterations, average transactions/s):
+----------------------+--------------------+
| Case | Improvement (%) |
+----------------------+--------------------+
| tcp4-rr | +1.85 |
| ipv6-loopback-plain | -0.60 |
| ipv6-loopback-expiry | -0.24 |
| tcp6-rr | -1.60 |
+----------------------+--------------------+
64 pinned netperf clients per case, with 128-byte requests and
replies:
tcp4-rr: netperf -t TCP_RR -l 8 -- -r 128,128
ipv6-loopback-plain: netperf -6 -t TCP_RR -l 8 -- -r 128,128
ipv6-loopback-expiry: netperf -6 -t TCP_RR -l 8 -- -r 128,128
tcp6-rr: netperf -6 -t TCP_RR -l 30 -- -r 128,128
Transfer (5 iterations, average received Mbit/s):
+-------------+--------------------+
| Case | Improvement (%) |
+-------------+--------------------+
| udp4-small | -0.91 |
| udp4-frag | +0.41 |
| tcp4-stream | +0.92 |
| udp6-small | -0.81 |
| udp6-frag | +1.37 |
| tcp6-stream | +3.22 |
| udpxfrm | +4.57 |
| tcpxfrm | -1.23 |
| geneve0 | +0.49 |
| geneve8 | +4.28 |
| geneve32 | +1.53 |
+-------------+--------------------+
64 pinned netperf clients per case. Netperf commands:
udp4-small: netperf -t UDP_STREAM -l 30 -- -m 512
udp4-frag: netperf -t UDP_STREAM -l 8 -- -m 8K
tcp4-stream: netperf -t TCP_STREAM -l 8 -- -m 10K
udp6-small: netperf -6 -t UDP_STREAM -l 8 -- -m 512
udp6-frag: netperf -6 -t UDP_STREAM -l 8 -- -m 8K
tcp6-stream: netperf -6 -t TCP_STREAM -l 8 -- -m 10K
udpxfrm: netperf -t UDP_STREAM -l 8 -- -m 8K
tcpxfrm: netperf -t TCP_STREAM -l 8 -- -m 10K
geneve0: netperf -t TCP_STREAM -l 8 -- -m 10K
geneve8: netperf -t TCP_STREAM -l 8 -- -m 10K
geneve32: netperf -t TCP_STREAM -l 30 -- -m 10K
Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: Tianyou Li <tianyou.li@intel.com>
Reviewed-by: Wangyang Guo <wangyang.guo@intel.com>
Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>
---
include/net/dst.h | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/include/net/dst.h b/include/net/dst.h
index dbedfe72e1fd..f01cfb42f97c 100644
--- a/include/net/dst.h
+++ b/include/net/dst.h
@@ -76,17 +76,17 @@ struct dst_entry {
struct rcu_head rcu_head;
short error;
short __pad;
- __u32 tclassid;
#ifndef CONFIG_64BIT
+ __u32 tclassid;
struct lwtunnel_state *lwtstate;
rcuref_t __rcuref; /* 32-bit offset 64 */
#endif
netdevice_tracker dev_tracker;
/*
- * Used by rtable and rt6_info. Moves lwtstate into the next cache
- * line on 64bit so that lwtstate does not cause false sharing with
- * __rcuref under contention of __rcuref. This also puts the
+ * Used by rtable and rt6_info. Moves lwtstate and tclassid into the
+ * next cache line on 64bit so that they do not cause false sharing
+ * with __rcuref under contention of __rcuref. This also puts the
* frequently accessed members of rtable and rt6_info out of the
* __rcuref cache line.
*/
@@ -94,6 +94,7 @@ struct dst_entry {
struct uncached_list *rt_uncached_list;
#ifdef CONFIG_64BIT
struct lwtunnel_state *lwtstate;
+ __u32 tclassid;
#endif
};
--
2.52.0
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] net: dst: Avoid false sharing between tclassid and __rcuref
2026-11-16 1:57 [PATCH] net: dst: Avoid false sharing between tclassid and __rcuref Jiebin Sun
@ 2026-11-18 16:58 ` netdev-bot+sashiko
2026-11-18 17:15 ` Eric Dumazet
1 sibling, 0 replies; 3+ messages in thread
From: netdev-bot+sashiko @ 2026-11-18 16:58 UTC (permalink / raw)
To: jiebin.sun
Cc: netdev, davem, edumazet, kuba, pabeni, horms, linux-kernel,
tim.c.chen, tianyou.li, wangyang.guo
Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 0
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] The patch moves tclassid after lwtstate in struct dst_entry
(include/net/dst.h), which makes sizeof(struct dst_entry) on x86-64…
--
net: dst: Avoid false sharing between tclassid and __rcuref
On 64-bit builds, this commit moves dst_entry::tclassid to sit after
lwtstate, so it no longer shares a cache line with __rcuref. On x86-64,
struct dst_entry grows from 136 to 144 bytes. The commit message says the
SLAB_HWCACHE_ALIGN caches used for rtable and rt6_info absorb the growth.
> diff --git a/include/net/dst.h b/include/net/dst.h
> index dbedfe72e1fd1..f01cfb42f97c1 100644
> --- a/include/net/dst.h
> +++ b/include/net/dst.h
[ ... ]
> @@ -94,6 +94,7 @@ struct dst_entry {
> struct uncached_list *rt_uncached_list;
> #ifdef CONFIG_64BIT
> struct lwtunnel_state *lwtstate;
> + __u32 tclassid;
> #endif
> };
[Severity: Medium]
The commit message says:
However, both struct rtable and struct rt6_info are allocated from slab
caches with SLAB_HWCACHE_ALIGN, so the per-object allocation sizes
remain unchanged at 192 bytes for rtable and 256 bytes for rt6_info.
Does the same hold for struct metadata_dst? It also embeds dst_entry, but
it is allocated with kmalloc and not from an aligned cache:
include/net/dst_metadata.h:
struct metadata_dst {
struct dst_entry dst;
enum metadata_type type;
union {
struct ip_tunnel_info tun_info;
...
} u;
};
The union is 16-byte aligned because ip_tunnel_info::options[] is marked
__aligned_largest. So the 8-byte growth of dst_entry becomes 16 bytes
here. On x86-64 with CONFIG_DST_CACHE=y, sizeof(struct metadata_dst) goes
from 240 to 256.
metadata_dst_alloc() then allocates that size plus the options length:
net/core/dst.c:metadata_dst_alloc() {
...
md_dst = kmalloc_flex(*md_dst, u.tun_info.options, optslen, flags);
...
}
With the new 256-byte base, does any optslen > 0 now land in kmalloc-512
instead of kmalloc-256?
On a collect_md device, vxlan_rcv() does this for every received packet:
tun_dst = udp_tun_rx_dst(skb, vxlan_get_sk_family(vs), flags,
key32_to_tunnel_id(vni), sizeof(*md));
That reaches udp_tun_rx_dst()->__ip_tun_set_dst()->metadata_dst_alloc()
with GFP_ATOMIC and optslen = sizeof(struct vxlan_metadata), which is 4.
Before this patch the allocation is 244 bytes (kmalloc-256). After it,
the allocation is 260 bytes (kmalloc-512).
geneve_rx() passes gnvh->opt_len * 4. Before the patch, options of up to
16 bytes still fit in kmalloc-256. After it, would any Geneve option at
all push the allocation into kmalloc-512?
The growth also applies to 64-bit kernels built with
CONFIG_IP_ROUTE_CLASSID=n, where nothing reads tclassid on the hot path.
Could the commit message cover the metadata_dst case? Alternatively,
could the layout be arranged so that metadata_dst stays within the
256-byte kmalloc bucket?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116015732.2744688-1-jiebin.sun%40intel.com
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] net: dst: Avoid false sharing between tclassid and __rcuref
2026-11-16 1:57 [PATCH] net: dst: Avoid false sharing between tclassid and __rcuref Jiebin Sun
2026-11-18 16:58 ` netdev-bot+sashiko
@ 2026-11-18 17:15 ` Eric Dumazet
1 sibling, 0 replies; 3+ messages in thread
From: Eric Dumazet @ 2026-11-18 17:15 UTC (permalink / raw)
To: Jiebin Sun
Cc: netdev, davem, kuba, pabeni, horms, linux-kernel, tim.c.chen,
tianyou.li, wangyang.guo
Le lun. 16 nov. 2026 à 02:53, Jiebin Sun <jiebin.sun@intel.com> a écrit :
>
> On x86-64, dst_entry::tclassid shares the same cache line with
> dst_entry::__rcuref. Updates to __rcuref cause false sharing with
> tclassid reads, for example in ip_rcv_finish_core() when
> CONFIG_IP_ROUTE_CLASSID=y. Memcached perf c2c shows the paths:
>
> read: net_rx_action -> process_backlog -> ip_rcv_finish_core
> -> skb_dst(skb)->tclassid (load)
> acquire: tcp_write_xmit -> __tcp_transmit_skb -> __dev_queue_xmit
> -> skb_dst_force -> dst_hold_safe -> rcuref_get
> release: tcp_rcv_established/tcp_add_backlog -> tcp_cleanup_skb
> -> skb_dst_drop -> dst_release -> rcuref_put
>
> Move tclassid after lwtstate on 64-bit to avoid false sharing with
> __rcuref.
>
> On x86-64, struct dst_entry grows from 136 to 144 bytes. struct
> rtable and struct rt6_info embed dst_entry, so their sizes grow
> from 176 to 184 and from 224 to 232 bytes, respectively. However,
> both struct rtable and struct rt6_info are allocated from slab
> caches with SLAB_HWCACHE_ALIGN, so the per-object allocation sizes
> remain unchanged at 192 bytes for rtable and 256 bytes for rt6_info.
>
This leaves a 4-byte hole where tclassid was, and adds 4 bytes of
padding at the end of dst_entry.
As the AI review pointed out, struct metadata_dst is not allocated
from a HWCACHE_ALIGN cache. Its union is 16-byte aligned, so it grows
from 240 to 256 bytes, and metadata_dst_alloc() with any optslen > 0
(vxlan collect_md, geneve options) moves from kmalloc-256 to
kmalloc-512, for every received packet.
I do not think we need to grow dst_entry at all.
dst->expires sits in the first (read-mostly) cache line, but its
fast path readers are all conditional (rt->rt_pmtu for IPv4,
RTF_EXPIRES for IPv6). It can live in the __rcuref cache line.
Swapping it with {error, __pad, tclassid} keeps sizeof(struct dst_entry)
at 136 bytes, keeps __rcuref at offset 64 on both 32-bit and 64-bit
without any new #ifdef, and also moves dst->error out of the
contended cache line.
Something like this (untested):
diff --git a/include/net/dst.h b/include/net/dst.h
--- a/include/net/dst.h
+++ b/include/net/dst.h
@@ -31,7 +31,9 @@ struct dst_entry {
};
struct dst_ops *ops;
unsigned long _metrics;
- unsigned long expires;
+ short error;
+ short __pad;
+ __u32 tclassid;
#ifdef CONFIG_XFRM
struct xfrm_state *xfrm;
#else
@@ -74,9 +76,7 @@ struct dst_entry {
int __use;
unsigned long lastuse;
struct rcu_head rcu_head;
- short error;
- short __pad;
- __u32 tclassid;
+ unsigned long expires;
#ifndef CONFIG_64BIT
struct lwtunnel_state *lwtstate;
rcuref_t __rcuref; /* 32-bit offset 64 */
Could you test this with your memcached setup, and include
pahole output for dst_entry and metadata_dst in the changelog?
> Memcached perf c2c samples at the class-ID load:
>
> +----------+------------+-------------+
> | Kernel | Local HITM | Remote HITM |
> +----------+------------+-------------+
> | Baseline | 894 | 680 |
> | Patched | 15 | 0 |
> +----------+------------+-------------+
>
> Results with 64 workers per endpoint (baseline 6628e7d4c6b2).
> Improvement is 100 * (patched average / baseline average - 1).
>
> Memcached (5 iterations, average operations/s):
>
> +--------------------+--------------------+
> | Case | Improvement (%) |
> +--------------------+--------------------+
> | default | +27.20 |
> | hit | +52.36 |
> | balanced | -0.52 |
> | large | +64.68 |
> +--------------------+--------------------+
>
> Memcached over IPv4 loopback.
Please make clear in the changelog that this is a loopback workload.
__rcuref is written for every packet there because loopback does not
set IFF_XMIT_DST_RELEASE, so __dev_queue_xmit() calls skb_dst_force()
and the receive path reuses the same dst. This is not what happens
with a physical NIC.
> The default client command is:
>
> memtier_benchmark -s 127.0.0.1 -p 11221 -P memcache_text
> -t 64 -c 1 --pipeline=16 --test-time=60 --ratio=1:100
> --data-size=32 --key-maximum=10000000
>
> hit: --key-maximum=1000
> balanced: --key-maximum=1000 --ratio=1:1
> large: --key-maximum=1000 --data-size=1024
>
> TCP request/reply (5 iterations, average transactions/s):
>
> +----------------------+--------------------+
> | Case | Improvement (%) |
> +----------------------+--------------------+
> | tcp4-rr | +1.85 |
> | ipv6-loopback-plain | -0.60 |
> | ipv6-loopback-expiry | -0.24 |
> | tcp6-rr | -1.60 |
> +----------------------+--------------------+
>
> 64 pinned netperf clients per case, with 128-byte requests and
> replies:
>
> tcp4-rr: netperf -t TCP_RR -l 8 -- -r 128,128
> ipv6-loopback-plain: netperf -6 -t TCP_RR -l 8 -- -r 128,128
> ipv6-loopback-expiry: netperf -6 -t TCP_RR -l 8 -- -r 128,128
> tcp6-rr: netperf -6 -t TCP_RR -l 30 -- -r 128,128
>
> Transfer (5 iterations, average received Mbit/s):
These tables are all within noise and do not exercise the contended
path. Please remove them, the changelog would be much easier to read
with only the perf c2c data and the memcached numbers.
Also, please tag the next version for net-next.
Thanks.
pw-bot: cr
>
> +-------------+--------------------+
> | Case | Improvement (%) |
> +-------------+--------------------+
> | udp4-small | -0.91 |
> | udp4-frag | +0.41 |
> | tcp4-stream | +0.92 |
> | udp6-small | -0.81 |
> | udp6-frag | +1.37 |
> | tcp6-stream | +3.22 |
> | udpxfrm | +4.57 |
> | tcpxfrm | -1.23 |
> | geneve0 | +0.49 |
> | geneve8 | +4.28 |
> | geneve32 | +1.53 |
> +-------------+--------------------+
>
> 64 pinned netperf clients per case. Netperf commands:
>
> udp4-small: netperf -t UDP_STREAM -l 30 -- -m 512
> udp4-frag: netperf -t UDP_STREAM -l 8 -- -m 8K
> tcp4-stream: netperf -t TCP_STREAM -l 8 -- -m 10K
> udp6-small: netperf -6 -t UDP_STREAM -l 8 -- -m 512
> udp6-frag: netperf -6 -t UDP_STREAM -l 8 -- -m 8K
> tcp6-stream: netperf -6 -t TCP_STREAM -l 8 -- -m 10K
> udpxfrm: netperf -t UDP_STREAM -l 8 -- -m 8K
> tcpxfrm: netperf -t TCP_STREAM -l 8 -- -m 10K
> geneve0: netperf -t TCP_STREAM -l 8 -- -m 10K
> geneve8: netperf -t TCP_STREAM -l 8 -- -m 10K
> geneve32: netperf -t TCP_STREAM -l 30 -- -m 10K
>
> Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
> Reviewed-by: Tianyou Li <tianyou.li@intel.com>
> Reviewed-by: Wangyang Guo <wangyang.guo@intel.com>
> Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>
> ---
> include/net/dst.h | 9 +++++----
> 1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/include/net/dst.h b/include/net/dst.h
> index dbedfe72e1fd..f01cfb42f97c 100644
> --- a/include/net/dst.h
> +++ b/include/net/dst.h
> @@ -76,17 +76,17 @@ struct dst_entry {
> struct rcu_head rcu_head;
> short error;
> short __pad;
> - __u32 tclassid;
> #ifndef CONFIG_64BIT
> + __u32 tclassid;
> struct lwtunnel_state *lwtstate;
> rcuref_t __rcuref; /* 32-bit offset 64 */
> #endif
> netdevice_tracker dev_tracker;
>
> /*
> - * Used by rtable and rt6_info. Moves lwtstate into the next cache
> - * line on 64bit so that lwtstate does not cause false sharing with
> - * __rcuref under contention of __rcuref. This also puts the
> + * Used by rtable and rt6_info. Moves lwtstate and tclassid into the
> + * next cache line on 64bit so that they do not cause false sharing
> + * with __rcuref under contention of __rcuref. This also puts the
> * frequently accessed members of rtable and rt6_info out of the
> * __rcuref cache line.
> */
> @@ -94,6 +94,7 @@ struct dst_entry {
> struct uncached_list *rt_uncached_list;
> #ifdef CONFIG_64BIT
> struct lwtunnel_state *lwtstate;
> + __u32 tclassid;
> #endif
> };
>
> --
> 2.52.0
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-18 17:15 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 1:57 [PATCH] net: dst: Avoid false sharing between tclassid and __rcuref Jiebin Sun
2026-11-18 16:58 ` netdev-bot+sashiko
2026-11-18 17:15 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox