Re: [PATCH v2 net-next] net: add sysctl to toggle napi_consume_skb() alien skb defer

[Paolo Abeni <pabeni@redhat.com>]

On 3/27/26 4:33 PM, Jason Xing wrote:
> Commit e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs")
> defers freeing of alien SKBs (alloc_cpu != current cpu) via
> skb_attempt_defer_free() on the TX completion path to reduce cross-NUMA
> SLUB spinlock contention to improve multi-queue UDP workloads.
> 
> However, this unconditionally impacts the napi_skb_cache fast recycle
> path for single-flow / few-flow workloads (e.g. AF_XDP benchmarks[1]):
> when the TX completion NAPI CPU differs from the SKB allocation CPU,
> SKBs are deferred instead of being returned to the local napi_skb_cache,
> forcing RX allocations back to the slow slab path.
> 
> The existing net.core.skb_defer_max=0 could disable this, but it is a
> global switch that also disables the defer mechanism in TCP/UDP/MPTCP
> recvmsg paths, losing its positive SLUB locality benefits there. AF_XDP
> can co-exist with other protocols. That's the reason why I gave up
> reusing skb_defer_disable_key. Besides, if the defer path is disabled,
> that means TCP/UDP/MPTCP in process path will trigger directly freeing
> skb with enabling/disabling bottom half(in kfree_skb_napi_cache())
> which could affect others. So my thinking is not to touch this path.
> 
> Add a dedicated sysctl net.core.napi_consume_skb_defer backed by a
> static key to selectively control the alien skb defer feature. Let
> users decide which is the best fit for their own requirements.
> 
> This patch also avoids touching local_bh* pair(in kfree_skb_napi_cache())
> to minimize the overhead.
> 
> [1]: taskset -c 0 ./xdpsock -i enp2s0f1 -q 1 -t -S -s 64
> 1) sysctl -w net.core.napi_consume_skb_defer=1 (as default)
>  sock0@enp2s0f1:1 txonly xdp-skb
>                    pps            pkts           1.00
> rx                 0              0
> tx                 1,851,950      20,397,952
> 
> 2)sysctl -w net.core.napi_consume_skb_defer=0
>  sock0@enp2s0f1:1 txonly xdp-skb
>                    pps            pkts           1.00
> rx                 0              0
> tx                 1,985,067      25,530,432
> 
> For AF_XDP scenario, it turns out to be around 6.6% improvement.

I'm not a big fan of multiple tunables around the same feature, but
possibly here the use-case extends beyond AF_XDP right? Do you observe
some measurable positive delta even with UDP? Possibly even with TCP,
when the bottleneck is the sender?

More data would be helpful.

> Signed-off-by: Jason Xing <kerneljasonxing@gmail.com>
> ---
> V2
> Link: https://lore.kernel.org/all/20260326144249.97213-1-kerneljasonxing@gmail.com/
> 1. reuse proc_do_static_key() (Eric)
> 2. add doc (Stan)
> ---
>  Documentation/admin-guide/sysctl/net.rst | 13 +++++++++++++
>  net/core/net-sysfs.h                     |  1 +
>  net/core/skbuff.c                        |  5 ++++-
>  net/core/sysctl_net_core.c               |  7 +++++++
>  4 files changed, 25 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
> index 0724a793798f..42e06f93306f 100644
> --- a/Documentation/admin-guide/sysctl/net.rst
> +++ b/Documentation/admin-guide/sysctl/net.rst
> @@ -368,6 +368,19 @@ by the cpu which allocated them.
>  
>  Default: 128
>  
> +napi_consume_skb_defer
> +----------------------
> +When set to 1 (default), napi_consume_skb() defers freeing SKBs whose
> +allocation CPU differs from the current CPU via skb_attempt_defer_free().
> +This reduces cross-NUMA SLUB spinlock contention for multi-queue workloads.
> +
> +Setting this to 0 disables the defer path in napi_consume_skb() only,
> +allowing SKBs to be returned to the local napi_skb_cache immediately.
> +This can benefit single-flow or few-flow workloads (e.g. AF_XDP TX)
> +where the defer detour hurts the fast recycle path.

I think it should be clarified that skb_defer_max takes priority, i.e.
no defer with skb_defer_max == 0. Also it would be great if you could
additional extend skb_defer_max documentation noting that 0 disable such
feature.

Thanks,

Paolo