Netdev List
 help / color / mirror / Atom feed
* [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list
@ 2026-10-13 13:02 Sorah Fukumori
  2026-10-16 16:54 ` Chase Hiltz
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Sorah Fukumori @ 2026-10-13 13:02 UTC (permalink / raw)
  To: netdev, davem, edumazet, kuba, pabeni
  Cc: horms, willemb, lixiaoyan, linux-kernel, Sorah Fukumori

skb_gro_receive() chains an skb into the frag_list of the held packet
when its payload cannot be attached as page frags: when its head is
not a page fragment and still holds payload, or when its head and
frags would overflow the frags[] array of the last packet. Since commit
5eddb24901ee ("gro: add support of (hw)gro packets to gro stack"), that
skb may itself be a GSO packet whose linear part holds several
gso_size segments. skb_segment() cannot split the linear part of a
frag_list member across segments, so segmenting the aggregate in
software, e.g. when forwarding it to a device without NETIF_F_FRAGLIST,
hits

  BUG_ON(skb_headlen(list_skb) > len);

Fully linear GSO packets come from generic XDP. When the program does
not support frags, netif_skb_check_for_xdp() linearizes any nonlinear
skb with pskb_expand_head() and skb_linearize() and keeps its GSO
metadata. GRO on the device runs before generic XDP, so each aggregate
passed by the program carries all of its payload in a kmalloc'ed head.
A tunnel that decapsulates these packets and runs GRO again through
gro_cells then builds the bad frag_list. This was first seen on a
router that forwards TCP received over VXLAN, with a non-frags XDP
program attached in SKB mode to the underlying bond because one slave
lacks native XDP:

  kernel BUG at net/core/skbuff.c:4917!
  RIP: 0010:skb_segment+0xbb6/0xf40
  Call Trace:
   tcp_gso_segment
   ipv6_gso_segment
   skb_mac_gso_segment
   __skb_gso_segment
   validate_xmit_skb
   __dev_queue_xmit
   ip6_output
   ip6_sublist_rcv_finish
   ipv6_list_rcv
   netif_receive_skb_list_internal
   gro_complete
   dev_gro_receive
   gro_receive_skb
   gro_cell_poll
   __napi_poll
   net_rx_action

It reproduces with three network namespaces: TCP sent over VXLAN on an
IPv6 veth link to a router namespace that has an XDP_PASS program
without frags support attached to the underlay veth in generic mode
and GRO enabled on its VXLAN device, and forwards to a veth with
tx-scatter-gather-fraglist disabled.

A program with frags support in generic mode gives similar packets.
skb_pp_cow_data() rebuilds the skb with up to about a page of linear
data in a page_pool head_frag and keeps the GSO metadata, so once
frags[] is full the next such packet is chained through frag_list
with more than gso_size bytes in its head as well.

Refuse the merge when the linear part to be chained exceeds the held
packet's gso_size. tcp_gro_receive() then flushes the held packet and
the skb starts a new GRO packet as its head, whose linear length
skb_segment() handles.

Fixes: 5eddb24901ee ("gro: add support of (hw)gro packets to gro stack")
Cc: stable@vger.kernel.org
Assisted-by: claude-opus-5-5
Signed-off-by: Sorah Fukumori <her@sorah.jp>
---

Notes:
    v2:
      - Mention the head_frag path that reaches merge: once frags[] is
        full, e.g. after a frags-capable generic XDP program (Eric Dumazet)
    v1: https://lore.kernel.org/netdev/20261013102453.3272620-1-her@sorah.jp/
    
    Testing, on net b7ad72414c67 in qemu/KVM (x86_64_defconfig plus
    VETH/VXLAN/BPF_SYSCALL built in), using the reproducer described above
    with iperf3 -P 4 for 20s:
      - without this patch: BUG in skb_segment() within ~3s in 4/4 runs
      - with this patch: 8/8 runs completed without a splat
    The same reproducer also hits the BUG on Ubuntu 7.0.0-38-generic and
    on v7.3-rc3. W=1 build of net/core/gro.o under allmodconfig adds no
    warnings. Full allyesconfig/allmodconfig builds and the net selftests
    were not run, and no selftest is included.
    
    The frags variant did not crash an unpatched kernel in the same
    reproducer with an xdp.frags XDP_PASS program, also with RPS on the
    underlay, gro_max_size 196608 or sender gso_max_size 8000-20000: GRO
    batches there are too short to fill frags[]. A kprobe on
    skb_gro_receive() shows the 64K page_pool skbs carrying about 3.4K of
    payload in the head, and the non-frags variant returning -E2BIG from
    the new check about 4000 times per 20s run.
    
    The crash analysis, reproducer and fix were developed with an AI
    coding assistant (Claude); I reviewed the change and the analysis.

 net/core/gro.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/core/gro.c b/net/core/gro.c
index 29b4d02bf519..ff828a3dce3c 100644
--- a/net/core/gro.c
+++ b/net/core/gro.c
@@ -187,6 +187,10 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb)
 	}
 
 merge:
+	/* skb_segment() cannot split the linear part of a frag_list member. */
+	if (unlikely(headlen > offset + skb_shinfo(p)->gso_size))
+		return -E2BIG;
+
 	/* sk ownership - if any - completely transferred to the aggregated packet */
 	skb->destructor = NULL;
 	skb->sk = NULL;

base-commit: b7ad72414c673ce1fa1f463e807cd4f34f8be9bf
-- 
2.55.0


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list
  2026-10-13 13:02 [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list Sorah Fukumori
@ 2026-10-16 16:54 ` Chase Hiltz
  2026-10-16 17:14 ` Eric Dumazet
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Chase Hiltz @ 2026-10-16 16:54 UTC (permalink / raw)
  To: Sorah Fukumori, netdev, davem, edumazet, kuba, pabeni
  Cc: horms, willemb, lixiaoyan, linux-kernel


On 2026-10-13 10:02, Sorah Fukumori wrote:
> skb_gro_receive() chains an skb into the frag_list of the held packet
> when its payload cannot be attached as page frags: when its head is
> not a page fragment and still holds payload, or when its head and
> frags would overflow the frags[] array of the last packet. Since commit
> 5eddb24901ee ("gro: add support of (hw)gro packets to gro stack"), that
> skb may itself be a GSO packet whose linear part holds several
> gso_size segments. skb_segment() cannot split the linear part of a
> frag_list member across segments, so segmenting the aggregate in
> software, e.g. when forwarding it to a device without NETIF_F_FRAGLIST,
> hits
> 
>    BUG_ON(skb_headlen(list_skb) > len);
> 
> Fully linear GSO packets come from generic XDP. When the program does
> not support frags, netif_skb_check_for_xdp() linearizes any nonlinear
> skb with pskb_expand_head() and skb_linearize() and keeps its GSO
> metadata. GRO on the device runs before generic XDP, so each aggregate
> passed by the program carries all of its payload in a kmalloc'ed head.
> A tunnel that decapsulates these packets and runs GRO again through
> gro_cells then builds the bad frag_list. This was first seen on a
> router that forwards TCP received over VXLAN, with a non-frags XDP
> program attached in SKB mode to the underlying bond because one slave
> lacks native XDP:
> 
>    kernel BUG at net/core/skbuff.c:4917!
>    RIP: 0010:skb_segment+0xbb6/0xf40
>    Call Trace:
>     tcp_gso_segment
>     ipv6_gso_segment
>     skb_mac_gso_segment
>     __skb_gso_segment
>     validate_xmit_skb
>     __dev_queue_xmit
>     ip6_output
>     ip6_sublist_rcv_finish
>     ipv6_list_rcv
>     netif_receive_skb_list_internal
>     gro_complete
>     dev_gro_receive
>     gro_receive_skb
>     gro_cell_poll
>     __napi_poll
>     net_rx_action
> 
> It reproduces with three network namespaces: TCP sent over VXLAN on an
> IPv6 veth link to a router namespace that has an XDP_PASS program
> without frags support attached to the underlay veth in generic mode
> and GRO enabled on its VXLAN device, and forwards to a veth with
> tx-scatter-gather-fraglist disabled.
> 
> A program with frags support in generic mode gives similar packets.
> skb_pp_cow_data() rebuilds the skb with up to about a page of linear
> data in a page_pool head_frag and keeps the GSO metadata, so once
> frags[] is full the next such packet is chained through frag_list
> with more than gso_size bytes in its head as well.
> 
> Refuse the merge when the linear part to be chained exceeds the held
> packet's gso_size. tcp_gro_receive() then flushes the held packet and
> the skb starts a new GRO packet as its head, whose linear length
> skb_segment() handles.
> 
> Fixes: 5eddb24901ee ("gro: add support of (hw)gro packets to gro stack")
> Cc: stable@vger.kernel.org
> Assisted-by: claude-opus-5-5
> Signed-off-by: Sorah Fukumori <her@sorah.jp>
> ---
> 
> Notes:
>      v2:
>        - Mention the head_frag path that reaches merge: once frags[] is
>          full, e.g. after a frags-capable generic XDP program (Eric Dumazet)
>      v1: https://lore.kernel.org/netdev/20261013102453.3272620-1-her@sorah.jp/
>      
>      Testing, on net b7ad72414c67 in qemu/KVM (x86_64_defconfig plus
>      VETH/VXLAN/BPF_SYSCALL built in), using the reproducer described above
>      with iperf3 -P 4 for 20s:
>        - without this patch: BUG in skb_segment() within ~3s in 4/4 runs
>        - with this patch: 8/8 runs completed without a splat
>      The same reproducer also hits the BUG on Ubuntu 7.0.0-38-generic and
>      on v7.3-rc3. W=1 build of net/core/gro.o under allmodconfig adds no
>      warnings. Full allyesconfig/allmodconfig builds and the net selftests
>      were not run, and no selftest is included.
>      
>      The frags variant did not crash an unpatched kernel in the same
>      reproducer with an xdp.frags XDP_PASS program, also with RPS on the
>      underlay, gro_max_size 196608 or sender gso_max_size 8000-20000: GRO
>      batches there are too short to fill frags[]. A kprobe on
>      skb_gro_receive() shows the 64K page_pool skbs carrying about 3.4K of
>      payload in the head, and the non-frags variant returning -E2BIG from
>      the new check about 4000 times per 20s run.
>      
>      The crash analysis, reproducer and fix were developed with an AI
>      coding assistant (Claude); I reviewed the change and the analysis.
> 
>   net/core/gro.c | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/net/core/gro.c b/net/core/gro.c
> index 29b4d02bf519..ff828a3dce3c 100644
> --- a/net/core/gro.c
> +++ b/net/core/gro.c
> @@ -187,6 +187,10 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb)
>   	}
>   
>   merge:
> +	/* skb_segment() cannot split the linear part of a frag_list member. */
> +	if (unlikely(headlen > offset + skb_shinfo(p)->gso_size))
> +		return -E2BIG;
> +
>   	/* sk ownership - if any - completely transferred to the aggregated packet */
>   	skb->destructor = NULL;
>   	skb->sk = NULL;
> 
> base-commit: b7ad72414c673ce1fa1f463e807cd4f34f8be9bf

Tested-by: Chase Hiltz <me@chasehil.tz>

I encountered and reproduced the same bug independently on Ubuntu 26.04 running 
the upstream v7.2.6 kernel. Applying the patch on top of v7.2.6 results in the 
crash no longer happening for me.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list
  2026-10-13 13:02 [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list Sorah Fukumori
  2026-10-16 16:54 ` Chase Hiltz
@ 2026-10-16 17:14 ` Eric Dumazet
  2026-10-19  9:27 ` Jason Xing
  2026-10-20  8:40 ` patchwork-bot+netdevbpf
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Dumazet @ 2026-10-16 17:14 UTC (permalink / raw)
  To: Sorah Fukumori
  Cc: netdev, davem, kuba, pabeni, horms, willemb, lixiaoyan,
	linux-kernel

Le mar. 13 oct. 2026 à 15:03, Sorah Fukumori <her@sorah.jp> a écrit :
>
> skb_gro_receive() chains an skb into the frag_list of the held packet
> when its payload cannot be attached as page frags: when its head is
> not a page fragment and still holds payload, or when its head and
> frags would overflow the frags[] array of the last packet. Since commit
> 5eddb24901ee ("gro: add support of (hw)gro packets to gro stack"), that
> skb may itself be a GSO packet whose linear part holds several
> gso_size segments. skb_segment() cannot split the linear part of a
> frag_list member across segments, so segmenting the aggregate in
> software, e.g. when forwarding it to a device without NETIF_F_FRAGLIST,
> hits
>
>   BUG_ON(skb_headlen(list_skb) > len);
>
> Fully linear GSO packets come from generic XDP. When the program does
> not support frags, netif_skb_check_for_xdp() linearizes any nonlinear
> skb with pskb_expand_head() and skb_linearize() and keeps its GSO
> metadata. GRO on the device runs before generic XDP, so each aggregate
> passed by the program carries all of its payload in a kmalloc'ed head.
> A tunnel that decapsulates these packets and runs GRO again through
> gro_cells then builds the bad frag_list. This was first seen on a
> router that forwards TCP received over VXLAN, with a non-frags XDP
> program attached in SKB mode to the underlying bond because one slave
> lacks native XDP:

Reviewed-by: Eric Dumazet <edumazet@kernel.org>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list
  2026-10-13 13:02 [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list Sorah Fukumori
  2026-10-16 16:54 ` Chase Hiltz
  2026-10-16 17:14 ` Eric Dumazet
@ 2026-10-19  9:27 ` Jason Xing
  2026-10-20  8:40 ` patchwork-bot+netdevbpf
  3 siblings, 0 replies; 5+ messages in thread
From: Jason Xing @ 2026-10-19  9:27 UTC (permalink / raw)
  To: Sorah Fukumori
  Cc: netdev, davem, edumazet, kuba, pabeni, horms, willemb, lixiaoyan,
	linux-kernel

On Tue, Oct 13, 2026 at 9:17 PM Sorah Fukumori <her@sorah.jp> wrote:
>
> skb_gro_receive() chains an skb into the frag_list of the held packet
> when its payload cannot be attached as page frags: when its head is
> not a page fragment and still holds payload, or when its head and
> frags would overflow the frags[] array of the last packet. Since commit
> 5eddb24901ee ("gro: add support of (hw)gro packets to gro stack"), that
> skb may itself be a GSO packet whose linear part holds several
> gso_size segments. skb_segment() cannot split the linear part of a
> frag_list member across segments, so segmenting the aggregate in
> software, e.g. when forwarding it to a device without NETIF_F_FRAGLIST,
> hits
>
>   BUG_ON(skb_headlen(list_skb) > len);
>
> Fully linear GSO packets come from generic XDP. When the program does
> not support frags, netif_skb_check_for_xdp() linearizes any nonlinear
> skb with pskb_expand_head() and skb_linearize() and keeps its GSO
> metadata. GRO on the device runs before generic XDP, so each aggregate
> passed by the program carries all of its payload in a kmalloc'ed head.
> A tunnel that decapsulates these packets and runs GRO again through
> gro_cells then builds the bad frag_list. This was first seen on a
> router that forwards TCP received over VXLAN, with a non-frags XDP
> program attached in SKB mode to the underlying bond because one slave
> lacks native XDP:
>
>   kernel BUG at net/core/skbuff.c:4917!
>   RIP: 0010:skb_segment+0xbb6/0xf40
>   Call Trace:
>    tcp_gso_segment
>    ipv6_gso_segment
>    skb_mac_gso_segment
>    __skb_gso_segment
>    validate_xmit_skb
>    __dev_queue_xmit
>    ip6_output
>    ip6_sublist_rcv_finish
>    ipv6_list_rcv
>    netif_receive_skb_list_internal
>    gro_complete
>    dev_gro_receive
>    gro_receive_skb
>    gro_cell_poll
>    __napi_poll
>    net_rx_action
>
> It reproduces with three network namespaces: TCP sent over VXLAN on an
> IPv6 veth link to a router namespace that has an XDP_PASS program
> without frags support attached to the underlay veth in generic mode
> and GRO enabled on its VXLAN device, and forwards to a veth with
> tx-scatter-gather-fraglist disabled.
>
> A program with frags support in generic mode gives similar packets.
> skb_pp_cow_data() rebuilds the skb with up to about a page of linear
> data in a page_pool head_frag and keeps the GSO metadata, so once
> frags[] is full the next such packet is chained through frag_list
> with more than gso_size bytes in its head as well.
>
> Refuse the merge when the linear part to be chained exceeds the held
> packet's gso_size. tcp_gro_receive() then flushes the held packet and
> the skb starts a new GRO packet as its head, whose linear length
> skb_segment() handles.
>
> Fixes: 5eddb24901ee ("gro: add support of (hw)gro packets to gro stack")
> Cc: stable@vger.kernel.org
> Assisted-by: claude-opus-5-5
> Signed-off-by: Sorah Fukumori <her@sorah.jp>

Reviewed-by: Jason Xing <kerneljasonxing@gmail.com>

Thanks!

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list
  2026-10-13 13:02 [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list Sorah Fukumori
                   ` (2 preceding siblings ...)
  2026-10-19  9:27 ` Jason Xing
@ 2026-10-20  8:40 ` patchwork-bot+netdevbpf
  3 siblings, 0 replies; 5+ messages in thread
From: patchwork-bot+netdevbpf @ 2026-10-20  8:40 UTC (permalink / raw)
  To: Sorah Fukumori
  Cc: netdev, davem, edumazet, kuba, pabeni, horms, willemb, lixiaoyan,
	linux-kernel

Hello:

This patch was applied to netdev/net.git (main)
by Paolo Abeni <pabeni@redhat.com>:

On Tue, 13 Oct 2026 22:02:48 +0900 you wrote:
> skb_gro_receive() chains an skb into the frag_list of the held packet
> when its payload cannot be attached as page frags: when its head is
> not a page fragment and still holds payload, or when its head and
> frags would overflow the frags[] array of the last packet. Since commit
> 5eddb24901ee ("gro: add support of (hw)gro packets to gro stack"), that
> skb may itself be a GSO packet whose linear part holds several
> gso_size segments. skb_segment() cannot split the linear part of a
> frag_list member across segments, so segmenting the aggregate in
> software, e.g. when forwarding it to a device without NETIF_F_FRAGLIST,
> hits
> 
> [...]

Here is the summary with links:
  - [net,v2] net: gro: don't chain linear parts larger than gso_size into frag_list
    https://git.kernel.org/netdev/net/c/679f6929a66d

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html



^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-10-20  8:40 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-13 13:02 [PATCH net v2] net: gro: don't chain linear parts larger than gso_size into frag_list Sorah Fukumori
2026-10-16 16:54 ` Chase Hiltz
2026-10-16 17:14 ` Eric Dumazet
2026-10-19  9:27 ` Jason Xing
2026-10-20  8:40 ` patchwork-bot+netdevbpf

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox