* [PATCH net] bonding: fix skb_under_panic in bond_ns_send() over stacked VLANs
@ 2026-07-19 23:21 Xiang Mei (Microsoft)
2026-07-20 17:58 ` Jay Vosburgh
0 siblings, 1 reply; 5+ messages in thread
From: Xiang Mei (Microsoft) @ 2026-07-19 23:21 UTC (permalink / raw)
To: Jay Vosburgh, Andrew Lunn, David S . Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni
Cc: Hangbin Liu, netdev, linux-kernel, AutonomousCodeSecurity,
tgopinath, kys, Xiang Mei (Microsoft)
bond_ns_send() builds an IPv6 Neighbor Solicitation with
ndisc_ns_create(), which reserves exactly LL_RESERVED_SPACE(dev) +
sizeof(struct ipv6hdr) of headroom for the later ip6_nd_hdr() push.
bond_handle_vlan() then inserts the collected VLAN tags into the skb;
each inner tag consumes VLAN_HLEN of that headroom via skb_push(). With
enough stacked VLAN devices between the bond and the ns_ip6_target, the
reserved IPv6 headroom is exhausted, so the subsequent
ndisc_send_skb() -> ip6_nd_hdr() -> skb_push(sizeof(struct ipv6hdr))
underflows past skb->head and hits skb_under_panic().
Restore the required headroom with skb_cow_head() after VLAN insertion
and before handing the skb to ndisc_send_skb(); drop the probe on
allocation failure. For paths that did not exhaust the headroom this is
a no-op, so previously working configurations are unaffected.
skbuff: skb_under_panic: len:84 put:40 head:... data:... tail:0x50 end:0x180 dev:veth0
kernel BUG at net/core/skbuff.c:214!
Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI
Workqueue: bond0 bond_arp_monitor
RIP: 0010:skb_panic+0x142/0x230
Call Trace:
skb_push (net/core/skbuff.c:224)
ndisc_send_skb (net/ipv6/ndisc.c:454 net/ipv6/ndisc.c:506)
bond_ns_send (drivers/net/bonding/bond_main.c:3255)
bond_ns_send_all (drivers/net/bonding/bond_main.c:3313)
bond_arp_monitor (drivers/net/bonding/bond_main.c:3458)
process_one_work (kernel/workqueue.c:3322)
worker_thread (kernel/workqueue.c:3405)
kthread (kernel/kthread.c:436)
Kernel panic - not syncing: Fatal exception
Fixes: 4e24be018eb9 ("bonding: add new parameter ns_targets")
Reported-by: AutonomousCodeSecurity@microsoft.com
Signed-off-by: Xiang Mei (Microsoft) <xmei5@asu.edu>
---
drivers/net/bonding/bond_main.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index e044fc733b8c..3ac3418c9498 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -3251,6 +3251,10 @@ static void bond_ns_send(struct slave *slave, const struct in6_addr *daddr,
addrconf_addr_solict_mult(daddr, &mcaddr);
if (bond_handle_vlan(slave, tags, skb)) {
+ if (skb_cow_head(skb, sizeof(struct ipv6hdr))) {
+ kfree_skb(skb);
+ return;
+ }
slave_update_last_tx(slave);
ndisc_send_skb(skb, &mcaddr, saddr);
}
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH net] bonding: fix skb_under_panic in bond_ns_send() over stacked VLANs 2026-07-19 23:21 [PATCH net] bonding: fix skb_under_panic in bond_ns_send() over stacked VLANs Xiang Mei (Microsoft) @ 2026-07-20 17:58 ` Jay Vosburgh 2026-07-29 23:30 ` Xiang Mei 0 siblings, 1 reply; 5+ messages in thread From: Jay Vosburgh @ 2026-07-20 17:58 UTC (permalink / raw) To: Xiang Mei (Microsoft) Cc: Andrew Lunn, David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Hangbin Liu, netdev, linux-kernel, AutonomousCodeSecurity, tgopinath, kys Xiang Mei (Microsoft) <xmei5@asu.edu> wrote: >bond_ns_send() builds an IPv6 Neighbor Solicitation with >ndisc_ns_create(), which reserves exactly LL_RESERVED_SPACE(dev) + >sizeof(struct ipv6hdr) of headroom for the later ip6_nd_hdr() push. >bond_handle_vlan() then inserts the collected VLAN tags into the skb; >each inner tag consumes VLAN_HLEN of that headroom via skb_push(). With >enough stacked VLAN devices between the bond and the ns_ip6_target, the >reserved IPv6 headroom is exhausted, so the subsequent >ndisc_send_skb() -> ip6_nd_hdr() -> skb_push(sizeof(struct ipv6hdr)) >underflows past skb->head and hits skb_under_panic(). How many stacked VLANs is "enough"? I'm guessing that it's somewhat device dependent, as LL_RESERVED_SPACE includes dev->needed_headroom, but as a ballpark here, was it more in the realm of 3, or 30? Also, why doesn't the skb_push called via bond_handle_vlan -> vlan_insert_tag_set_proto -> vlan_insert_tag -> vlan_insert_inner_tag -> __vlan_inser_inner_tag trigger the skb_under_panic? Does adding one or two more nested VLANs move the panic into the above call path? >Restore the required headroom with skb_cow_head() after VLAN insertion >and before handing the skb to ndisc_send_skb(); drop the probe on >allocation failure. For paths that did not exhaust the headroom this is >a no-op, so previously working configurations are unaffected. > > skbuff: skb_under_panic: len:84 put:40 head:... data:... tail:0x50 end:0x180 dev:veth0 > kernel BUG at net/core/skbuff.c:214! The above text doesn't seem to match with current net-next: pr_emerg("%s: text:%px len:%d put:%d head:%px data:%px tail:%#lx end:%#l x dev:%s\n", I presume you replaced the "head:" and "data:" values with "..."; did you also edit out the "text:" that should precede "len:"? I'm wondering because without the actual values, it's not clear how far beyond head the data pointer went, and thus whether adding more VLANs would move the failure into the VLAN tag code. -J > Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI > Workqueue: bond0 bond_arp_monitor > RIP: 0010:skb_panic+0x142/0x230 > Call Trace: > skb_push (net/core/skbuff.c:224) > ndisc_send_skb (net/ipv6/ndisc.c:454 net/ipv6/ndisc.c:506) > bond_ns_send (drivers/net/bonding/bond_main.c:3255) > bond_ns_send_all (drivers/net/bonding/bond_main.c:3313) > bond_arp_monitor (drivers/net/bonding/bond_main.c:3458) > process_one_work (kernel/workqueue.c:3322) > worker_thread (kernel/workqueue.c:3405) > kthread (kernel/kthread.c:436) > Kernel panic - not syncing: Fatal exception > >Fixes: 4e24be018eb9 ("bonding: add new parameter ns_targets") >Reported-by: AutonomousCodeSecurity@microsoft.com >Signed-off-by: Xiang Mei (Microsoft) <xmei5@asu.edu> >--- > drivers/net/bonding/bond_main.c | 4 ++++ > 1 file changed, 4 insertions(+) > >diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c >index e044fc733b8c..3ac3418c9498 100644 >--- a/drivers/net/bonding/bond_main.c >+++ b/drivers/net/bonding/bond_main.c >@@ -3251,6 +3251,10 @@ static void bond_ns_send(struct slave *slave, const struct in6_addr *daddr, > > addrconf_addr_solict_mult(daddr, &mcaddr); > if (bond_handle_vlan(slave, tags, skb)) { >+ if (skb_cow_head(skb, sizeof(struct ipv6hdr))) { >+ kfree_skb(skb); >+ return; >+ } > slave_update_last_tx(slave); > ndisc_send_skb(skb, &mcaddr, saddr); > } >-- >2.43.0 > --- -Jay Vosburgh, jv@jvosburgh.net ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] bonding: fix skb_under_panic in bond_ns_send() over stacked VLANs 2026-07-20 17:58 ` Jay Vosburgh @ 2026-07-29 23:30 ` Xiang Mei 2026-07-30 3:50 ` Hangbin Liu 0 siblings, 1 reply; 5+ messages in thread From: Xiang Mei @ 2026-07-29 23:30 UTC (permalink / raw) To: Jay Vosburgh Cc: Andrew Lunn, David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Hangbin Liu, netdev, linux-kernel, AutonomousCodeSecurity, tgopinath, kys Thanks for the review, and sorry for the slow reply. On Mon, Jul 20, 2026 at 10:58 AM Jay Vosburgh <jv@jvosburgh.net> wrote: > > Xiang Mei (Microsoft) <xmei5@asu.edu> wrote: > > >bond_ns_send() builds an IPv6 Neighbor Solicitation with > >ndisc_ns_create(), which reserves exactly LL_RESERVED_SPACE(dev) + > >sizeof(struct ipv6hdr) of headroom for the later ip6_nd_hdr() push. > >bond_handle_vlan() then inserts the collected VLAN tags into the skb; > >each inner tag consumes VLAN_HLEN of that headroom via skb_push(). With > >enough stacked VLAN devices between the bond and the ns_ip6_target, the > >reserved IPv6 headroom is exhausted, so the subsequent > >ndisc_send_skb() -> ip6_nd_hdr() -> skb_push(sizeof(struct ipv6hdr)) > >underflows past skb->head and hits skb_under_panic(). > > How many stacked VLANs is "enough"? I'm guessing that it's > somewhat device dependent, as LL_RESERVED_SPACE includes > dev->needed_headroom, but as a ballpark here, was it more in the realm > of 3, or 30? Exactly 6, and it can only ever be 6, because that is also the deepest stack the netdev nesting limit allows. For the veth slave in my reproducer LL_RESERVED_SPACE(veth0) = 16, so ndisc_alloc_skb() reserves 16 + sizeof(struct ipv6hdr) = 56 bytes. With N VLAN devices above bond0, bond_handle_vlan() puts tags[0] in skb->vlan_tci (costs no headroom) and pushes the other N-1 inline at VLAN_HLEN each: headroom before ip6_nd_hdr() = 56 - 4 * (N - 1) (it needs 40) N = 5: 40 -> skb->data lands exactly on skb->head, passes by 0 N = 6: 36 -> 4 bytes short -> skb_under_panic() N >= 7 is unreachable: veth0 -> bond0 -> v0..v5 already exhausts MAX_NEST_DEV (8), so creating v6 fails with -EMLINK in __netdev_upper_dev_link(). Sweeping N = 1..7 on the unpatched kernel panics only at N = 6, and it panics on every boot; with the patch applied the same sweep is clean throughout. You are right that it is device dependent, but only in that first term. The panic needs hlen < 20, i.e. hlen == 16, i.e. hard_header_len + needed_headroom <= 15. For an Ethernet slave that means needed_headroom <= 1, which covers veth and plenty of plain NICs. A slave with needed_headroom >= 2 gets hlen == 32 and cannot hit this at any legal stacking depth. > > Also, why doesn't the skb_push called via bond_handle_vlan -> > vlan_insert_tag_set_proto -> vlan_insert_tag -> vlan_insert_inner_tag -> > __vlan_inser_inner_tag trigger the skb_under_panic? > Because __vlan_insert_inner_tag() grows the head before it pushes: if (skb_cow_head(skb, meta_len + VLAN_HLEN) < 0) return -ENOMEM; skb_push(skb, VLAN_HLEN); so short headroom means a pskb_expand_head() realloc, not an underflow. ip6_nd_hdr()'s skb_push(sizeof(struct ipv6hdr)) is the only unguarded push in the chain, which is why the fault always lands there. > Does adding one or two more nested VLANs move the panic into the > above call path? > No, for both of the reasons above: there is no room to add more (-EMLINK at N = 7), and skb_cow_head() would absorb it if there were. > >Restore the required headroom with skb_cow_head() after VLAN insertion > >and before handing the skb to ndisc_send_skb(); drop the probe on > >allocation failure. For paths that did not exhaust the headroom this is > >a no-op, so previously working configurations are unaffected. > > > > skbuff: skb_under_panic: len:84 put:40 head:... data:... tail:0x50 end:0x180 dev:veth0 > > kernel BUG at net/core/skbuff.c:214! > > The above text doesn't seem to match with current net-next: > > pr_emerg("%s: text:%px len:%d put:%d head:%px data:%px tail:%#lx end:%#l > x dev:%s\n", > > I presume you replaced the "head:" and "data:" values with > "..."; did you also edit out the "text:" that should precede "len:"? > Yes, sorry, I over-trimmed it. Orignal: skbuff: skb_under_panic: text:ffffffff85eef44b len:84 put:40 head:ffff88801400b740 data:ffff88801400b73c tail:0x50 end:0x180 dev:veth0 data is 4 bytes below head, i.e. the N = 6 case above. Now the part that matters more than the panic, and that changes what the right fix is. Following your VLAN-insertion question, I noticed vlan_insert_tag() passes mac_len = ETH_HLEN, i.e. it assumes skb->data is at an Ethernet header. That holds for bond_arp_send(), because arp_create() calls dev_hard_header() first. It does not hold for bond_ns_send(): ndisc_ns_create() leaves both the IPv6 header and L2 to ndisc_send_skb(), so at bond_handle_vlan() time skb->data points at the nd_msg. __vlan_insert_inner_tag() then memmoves the first 12 bytes of the ICMPv6 message down by 4 and writes the tag at payload offset 12, and transport_header is left stale, so the checksum is stored 4 bytes per tag too late. Sniffing the slave's peer confirms it. With two VLANs (one inline tag, VID 11): 60 00 00 00 00 1c 3a ff payload_len 28 (+4) 87 00 00 00 00 00 52 db cksum 0, 0x52db written 4 bytes late 20 01 0d b8 81 00 00 0b the 8100/000b tag sits in the target 00 01 00 00 00 00 00 00 00 00 00 02 So bond NS monitoring over more than one level of VLAN has never worked: no peer can answer these probes. The panic at N = 6 is just where the same bogus insertion also consumes the last of the ipv6hdr reservation. That makes the headroom-only patch the wrong fix: it stops the crash, but bond_handle_vlan() has already mis-tagged the packet by then, so every 2+ VLAN probe stays corrupt. Both symptoms have the same cause, tags inserted before any header exists, so the fix is to build the probe in the order the stack uses: - push the IPv6 header first, so the reservation is consumed up front and the later tag inserts grow the head via skb_vlan_push() instead of eating it; - run the probe through NF_INET_LOCAL_OUT as a bare IPv6 packet, exactly what ndisc_send_skb() feeds the hooks today. This needs a small export (ndisc_attach_dst()) because ip6_route_me_harder() dereferences skb_dst() unconditionally; - only then push the link-layer header and add the VLAN tags, so they land at a real Ethernet header and the target address is untouched. With that applied the N = 1..6 wire capture is clean throughout. I kept LOCAL_OUT deliberately so NS monitoring keeps traversing netfilter the way it does today; dropping it would be simpler, but a silent behaviour change. I am not familiar with the bonding and ndisc internals, so I posted it as an RFC rather than presenting it as settled: https://lore.kernel.org/netdev/20260729230119.2717507-1-xmei5@asu.edu/T/#t Thanks, Xiang > I'm wondering because without the actual values, it's not clear > how far beyond head the data pointer went, and thus whether adding more > VLANs would move the failure into the VLAN tag code. > > -J > > > Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI > > Workqueue: bond0 bond_arp_monitor > > RIP: 0010:skb_panic+0x142/0x230 > > Call Trace: > > skb_push (net/core/skbuff.c:224) > > ndisc_send_skb (net/ipv6/ndisc.c:454 net/ipv6/ndisc.c:506) > > bond_ns_send (drivers/net/bonding/bond_main.c:3255) > > bond_ns_send_all (drivers/net/bonding/bond_main.c:3313) > > bond_arp_monitor (drivers/net/bonding/bond_main.c:3458) > > process_one_work (kernel/workqueue.c:3322) > > worker_thread (kernel/workqueue.c:3405) > > kthread (kernel/kthread.c:436) > > Kernel panic - not syncing: Fatal exception > > > >Fixes: 4e24be018eb9 ("bonding: add new parameter ns_targets") > >Reported-by: AutonomousCodeSecurity@microsoft.com > >Signed-off-by: Xiang Mei (Microsoft) <xmei5@asu.edu> > >--- > > drivers/net/bonding/bond_main.c | 4 ++++ > > 1 file changed, 4 insertions(+) > > > >diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c > >index e044fc733b8c..3ac3418c9498 100644 > >--- a/drivers/net/bonding/bond_main.c > >+++ b/drivers/net/bonding/bond_main.c > >@@ -3251,6 +3251,10 @@ static void bond_ns_send(struct slave *slave, const struct in6_addr *daddr, > > > > addrconf_addr_solict_mult(daddr, &mcaddr); > > if (bond_handle_vlan(slave, tags, skb)) { > >+ if (skb_cow_head(skb, sizeof(struct ipv6hdr))) { > >+ kfree_skb(skb); > >+ return; > >+ } > > slave_update_last_tx(slave); > > ndisc_send_skb(skb, &mcaddr, saddr); > > } > >-- > >2.43.0 > > > > --- > -Jay Vosburgh, jv@jvosburgh.net ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] bonding: fix skb_under_panic in bond_ns_send() over stacked VLANs 2026-07-29 23:30 ` Xiang Mei @ 2026-07-30 3:50 ` Hangbin Liu 2026-07-31 23:57 ` Hangbin Liu 0 siblings, 1 reply; 5+ messages in thread From: Hangbin Liu @ 2026-07-30 3:50 UTC (permalink / raw) To: Xiang Mei Cc: Jay Vosburgh, Andrew Lunn, David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, netdev, linux-kernel, AutonomousCodeSecurity, tgopinath, kys Hi Xiang, On 29.07.2026 16:30, Xiang Mei wrote: >> Does adding one or two more nested VLANs move the panic into the >> above call path? >> > >No, for both of the reasons above: there is no room to add more >(-EMLINK at N = 7), and skb_cow_head() would absorb it if there were. What about the scenario that involves nested VLANs? For example: vlan.1 - vlan.1.2 ... vlan.1..6 - bond - bond.1 - bond.1.2 ... bond.1..6? >Both symptoms have the same cause, tags inserted before any header >exists, so the fix is to build the probe in the order the stack uses: > > - push the IPv6 header first, so the reservation is consumed up front > and the later tag inserts grow the head via skb_vlan_push() instead > of eating it; > > - run the probe through NF_INET_LOCAL_OUT as a bare IPv6 packet, > exactly what ndisc_send_skb() feeds the hooks today. This needs a > small export (ndisc_attach_dst()) because ip6_route_me_harder() > dereferences skb_dst() unconditionally; > > - only then push the link-layer header and add the VLAN tags, so they > land at a real Ethernet header and the target address is untouched. > >With that applied the N = 1..6 wire capture is clean throughout. I kept >LOCAL_OUT deliberately so NS monitoring keeps traversing netfilter the >way it does today; dropping it would be simpler, but a silent behaviour >change. > >I am not familiar with the bonding and ndisc internals, so I posted it as >an RFC rather than presenting it as settled: > > https://lore.kernel.org/netdev/20260729230119.2717507-1-xmei5@asu.edu/T/#t I'm not sure if calling NF_INET_LOCAL_OUT in bond code is a good idea. Waiting for other's opinion. Thanks Hangbin ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] bonding: fix skb_under_panic in bond_ns_send() over stacked VLANs 2026-07-30 3:50 ` Hangbin Liu @ 2026-07-31 23:57 ` Hangbin Liu 0 siblings, 0 replies; 5+ messages in thread From: Hangbin Liu @ 2026-07-31 23:57 UTC (permalink / raw) To: Xiang Mei Cc: Jay Vosburgh, Andrew Lunn, David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, netdev, linux-kernel, AutonomousCodeSecurity, tgopinath, kys On 30.07.2026 11:50, Hangbin Liu wrote: >Hi Xiang, > >On 29.07.2026 16:30, Xiang Mei wrote: >>> Does adding one or two more nested VLANs move the panic into the >>>above call path? >>> >> >>No, for both of the reasons above: there is no room to add more >>(-EMLINK at N = 7), and skb_cow_head() would absorb it if there were. > >What about the scenario that involves nested VLANs? For example: > >vlan.1 - vlan.1.2 ... vlan.1..6 - bond - bond.1 - bond.1.2 ... bond.1..6? Oh, I just remember that the MAX_NEST_DEV is 8. So there could be only 1 VLAN slave with this topology. Thanks Hangbin ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-07-31 23:57 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-07-19 23:21 [PATCH net] bonding: fix skb_under_panic in bond_ns_send() over stacked VLANs Xiang Mei (Microsoft) 2026-07-20 17:58 ` Jay Vosburgh 2026-07-29 23:30 ` Xiang Mei 2026-07-30 3:50 ` Hangbin Liu 2026-07-31 23:57 ` Hangbin Liu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.