* [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device @ 2026-07-16 12:20 xietangxin 2026-07-16 12:53 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: xietangxin @ 2026-07-16 12:20 UTC (permalink / raw) To: David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni Cc: Simon Horman, netdev, linux-kernel, John Fastabend, Jesse Gross, gaoxingwang1, huyizhen [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device Hi all, We encountered a skb_under_panic triggered by toggling NETIF_F_HW_VLAN_CTAG_TX on the lower device while a VLAN device is up and sending traffic. Call trace ========== skbuff: skb_under_panic: text:ffffc0d2900283d8 len:74 put:14 head:ffff334820249c00 data:ffff334820249bfe tail:0x48 end:0xc0 dev:vlan4 ------------[ cut here ]------------ kernel BUG at net/core/skbuff.c:116! Internal error: Oops - BUG: 00000000f2000800 [#1] SMP Call trace: skb_panic+0xcc/0xd0 __skb_checksum+0x0/0x480 eth_header+0x48/0x1a0 vlan_dev_hard_header+0xd0/0x284 neigh_connected_output+0x16c/0x20c ip6_finish_output2+0x4b4/0xd74 __ip6_finish_output.part.0+0x1ac/0x3b0 ip6_finish_output+0x160/0x200 ip6_output+0x13c/0x294 ndisc_send_skb+0x41c/0x6f0 ndisc_send_rs+0xac/0x3b0 addrconf_rs_timer+0x42c/0x660 call_timer_fn+0x54/0x290 expire_timers+0x26c/0x420 Reproducer ========== # Create veth pair (NETIF_F_HW_VLAN_CTAG_TX is ON by default) ip link add veth0 type veth peer name veth1 ip link set veth0 up ip link set veth1 up # Turn off HW VLAN TX offload on lower device ethtool -K veth0 tx-vlan-hw-insert off # Create VLAN device on veth0 # At this point: header_ops = &vlan_header_ops, hard_header_len = 18 ip link add link veth0 name veth0.10 type vlan id 10 reorder_hdr off ip addr add 192.168.10.1/24 dev veth0.10 ip link set veth0.10 up # Turn HW VLAN TX offload back ON on lower device # This triggers NETDEV_FEAT_CHANGE -> vlan_transfer_features() # hard_header_len changes from 18 to 14, but header_ops is NOT updated ethtool -K veth0 tx-vlan-hw-insert on # When a packet is sent through veth0.10 # - skb is allocated based on hard_header_len=14 -> ~16 bytes # - vlan_dev_hard_header() pushes VLAN_HLEN(4) + ETH_HLEN(14) = 18 bytes # - skb_under_panic! Any feedback or guidance would be greatly appreciated. -- Best regards, Tangxin Xie ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device 2026-07-16 12:20 [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device xietangxin @ 2026-07-16 12:53 ` Eric Dumazet 2026-07-16 13:06 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: Eric Dumazet @ 2026-07-16 12:53 UTC (permalink / raw) To: xietangxin Cc: David S . Miller, Jakub Kicinski, Paolo Abeni, Simon Horman, netdev, linux-kernel, John Fastabend, Jesse Gross, gaoxingwang1, huyizhen On Thu, Jul 16, 2026 at 2:20 PM xietangxin <xietangxin@h-partners.com> wrote: > > [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device > > Hi all, > > We encountered a skb_under_panic triggered by toggling > NETIF_F_HW_VLAN_CTAG_TX on the lower device while a VLAN device is > up and sending traffic. > > Call trace > ========== > > skbuff: skb_under_panic: text:ffffc0d2900283d8 len:74 put:14 > head:ffff334820249c00 data:ffff334820249bfe tail:0x48 end:0xc0 dev:vlan4 > ------------[ cut here ]------------ > kernel BUG at net/core/skbuff.c:116! > Internal error: Oops - BUG: 00000000f2000800 [#1] SMP > Call trace: > skb_panic+0xcc/0xd0 > __skb_checksum+0x0/0x480 > eth_header+0x48/0x1a0 > vlan_dev_hard_header+0xd0/0x284 > neigh_connected_output+0x16c/0x20c > ip6_finish_output2+0x4b4/0xd74 > __ip6_finish_output.part.0+0x1ac/0x3b0 > ip6_finish_output+0x160/0x200 > ip6_output+0x13c/0x294 > ndisc_send_skb+0x41c/0x6f0 > ndisc_send_rs+0xac/0x3b0 > addrconf_rs_timer+0x42c/0x660 > call_timer_fn+0x54/0x290 > expire_timers+0x26c/0x420 > > Reproducer > ========== > > # Create veth pair (NETIF_F_HW_VLAN_CTAG_TX is ON by default) > ip link add veth0 type veth peer name veth1 > ip link set veth0 up > ip link set veth1 up > > # Turn off HW VLAN TX offload on lower device > ethtool -K veth0 tx-vlan-hw-insert off > > # Create VLAN device on veth0 > # At this point: header_ops = &vlan_header_ops, hard_header_len = 18 > ip link add link veth0 name veth0.10 type vlan id 10 reorder_hdr off > ip addr add 192.168.10.1/24 dev veth0.10 > ip link set veth0.10 up > > # Turn HW VLAN TX offload back ON on lower device > # This triggers NETDEV_FEAT_CHANGE -> vlan_transfer_features() > # hard_header_len changes from 18 to 14, but header_ops is NOT updated > ethtool -K veth0 tx-vlan-hw-insert on > > # When a packet is sent through veth0.10 > # - skb is allocated based on hard_header_len=14 -> ~16 bytes > # - vlan_dev_hard_header() pushes VLAN_HLEN(4) + ETH_HLEN(14) = 18 bytes > # - skb_under_panic! > > > Any feedback or guidance would be greatly appreciated. This rings a bell, I thought we already fixed this issue long ago :/ I would suggest we always add VLAN_HLEN even if not (yet) needed. diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c index 2b74ed56eb166d52c3351768d9dfedc7b2c8ec2d..d7f90b3b2bb3aadb2c466891720a7f407d0bc34b 100644 --- a/net/8021q/vlan.c +++ b/net/8021q/vlan.c @@ -323,10 +323,7 @@ static void vlan_transfer_features(struct net_device *dev, netif_inherit_tso_max(vlandev, dev); - if (vlan_hw_offload_capable(dev->features, vlan->vlan_proto)) - vlandev->hard_header_len = dev->hard_header_len; - else - vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN; + vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN; #if IS_ENABLED(CONFIG_FCOE) vlandev->fcoe_ddp_xid = dev->fcoe_ddp_xid; A kind-of-similar change is needed in net/8021q/vlan_dev.c ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device 2026-07-16 12:53 ` Eric Dumazet @ 2026-07-16 13:06 ` Eric Dumazet 2026-07-17 6:42 ` xietangxin 0 siblings, 1 reply; 5+ messages in thread From: Eric Dumazet @ 2026-07-16 13:06 UTC (permalink / raw) To: xietangxin Cc: David S . Miller, Jakub Kicinski, Paolo Abeni, Simon Horman, netdev, linux-kernel, John Fastabend, Jesse Gross, gaoxingwang1, huyizhen On Thu, Jul 16, 2026 at 2:53 PM Eric Dumazet <edumazet@google.com> wrote: > > On Thu, Jul 16, 2026 at 2:20 PM xietangxin <xietangxin@h-partners.com> wrote: > > > > [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device > > > > Hi all, > > > > We encountered a skb_under_panic triggered by toggling > > NETIF_F_HW_VLAN_CTAG_TX on the lower device while a VLAN device is > > up and sending traffic. > > > > Call trace > > ========== > > > > skbuff: skb_under_panic: text:ffffc0d2900283d8 len:74 put:14 > > head:ffff334820249c00 data:ffff334820249bfe tail:0x48 end:0xc0 dev:vlan4 > > ------------[ cut here ]------------ > > kernel BUG at net/core/skbuff.c:116! > > Internal error: Oops - BUG: 00000000f2000800 [#1] SMP > > Call trace: > > skb_panic+0xcc/0xd0 > > __skb_checksum+0x0/0x480 > > eth_header+0x48/0x1a0 > > vlan_dev_hard_header+0xd0/0x284 > > neigh_connected_output+0x16c/0x20c > > ip6_finish_output2+0x4b4/0xd74 > > __ip6_finish_output.part.0+0x1ac/0x3b0 > > ip6_finish_output+0x160/0x200 > > ip6_output+0x13c/0x294 > > ndisc_send_skb+0x41c/0x6f0 > > ndisc_send_rs+0xac/0x3b0 > > addrconf_rs_timer+0x42c/0x660 > > call_timer_fn+0x54/0x290 > > expire_timers+0x26c/0x420 > > > > Reproducer > > ========== > > > > # Create veth pair (NETIF_F_HW_VLAN_CTAG_TX is ON by default) > > ip link add veth0 type veth peer name veth1 > > ip link set veth0 up > > ip link set veth1 up > > > > # Turn off HW VLAN TX offload on lower device > > ethtool -K veth0 tx-vlan-hw-insert off > > > > # Create VLAN device on veth0 > > # At this point: header_ops = &vlan_header_ops, hard_header_len = 18 > > ip link add link veth0 name veth0.10 type vlan id 10 reorder_hdr off > > ip addr add 192.168.10.1/24 dev veth0.10 > > ip link set veth0.10 up > > > > # Turn HW VLAN TX offload back ON on lower device > > # This triggers NETDEV_FEAT_CHANGE -> vlan_transfer_features() > > # hard_header_len changes from 18 to 14, but header_ops is NOT updated > > ethtool -K veth0 tx-vlan-hw-insert on > > > > # When a packet is sent through veth0.10 > > # - skb is allocated based on hard_header_len=14 -> ~16 bytes > > # - vlan_dev_hard_header() pushes VLAN_HLEN(4) + ETH_HLEN(14) = 18 bytes > > # - skb_under_panic! > > > > > > Any feedback or guidance would be greatly appreciated. > > This rings a bell, I thought we already fixed this issue long ago :/ > > I would suggest we always add VLAN_HLEN even if not (yet) needed. > > diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c > index 2b74ed56eb166d52c3351768d9dfedc7b2c8ec2d..d7f90b3b2bb3aadb2c466891720a7f407d0bc34b > 100644 > --- a/net/8021q/vlan.c > +++ b/net/8021q/vlan.c > @@ -323,10 +323,7 @@ static void vlan_transfer_features(struct net_device *dev, > > netif_inherit_tso_max(vlandev, dev); > > - if (vlan_hw_offload_capable(dev->features, vlan->vlan_proto)) > - vlandev->hard_header_len = dev->hard_header_len; > - else > - vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN; > + vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN; > > #if IS_ENABLED(CONFIG_FCOE) > vlandev->fcoe_ddp_xid = dev->fcoe_ddp_xid; > > A kind-of-similar change is needed in net/8021q/vlan_dev.c The second part would be a nice cleanup, unless I am missing something. diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c index 7aa3af8b10ead0384bbb2e767883b63a37be5fa2..dfd907e2ede2dcb0b6120e1fa5b6bb4d30bfb28a 100644 --- a/net/8021q/vlan_dev.c +++ b/net/8021q/vlan_dev.c @@ -496,26 +496,6 @@ static const struct header_ops vlan_header_ops = { .parse_protocol = vlan_parse_protocol, }; -static int vlan_passthru_hard_header(struct sk_buff *skb, struct net_device *dev, - unsigned short type, - const void *daddr, const void *saddr, - unsigned int len) -{ - struct vlan_dev_priv *vlan = vlan_dev_priv(dev); - struct net_device *real_dev = vlan->real_dev; - - if (saddr == NULL) - saddr = dev->dev_addr; - - return dev_hard_header(skb, real_dev, type, daddr, saddr, len); -} - -static const struct header_ops vlan_passthru_header_ops = { - .create = vlan_passthru_hard_header, - .parse = eth_header_parse, - .parse_protocol = vlan_parse_protocol, -}; - static const struct device_type vlan_type = { .name = "vlan", }; @@ -575,13 +555,8 @@ static int vlan_dev_init(struct net_device *dev) #endif dev->needed_headroom = real_dev->needed_headroom; - if (vlan_hw_offload_capable(real_dev->features, vlan->vlan_proto)) { - dev->header_ops = &vlan_passthru_header_ops; - dev->hard_header_len = real_dev->hard_header_len; - } else { - dev->header_ops = &vlan_header_ops; - dev->hard_header_len = real_dev->hard_header_len + VLAN_HLEN; - } + dev->header_ops = &vlan_header_ops; + dev->hard_header_len = real_dev->hard_header_len + VLAN_HLEN; dev->netdev_ops = &vlan_netdev_ops; ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device 2026-07-16 13:06 ` Eric Dumazet @ 2026-07-17 6:42 ` xietangxin 2026-07-20 7:29 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: xietangxin @ 2026-07-17 6:42 UTC (permalink / raw) To: Eric Dumazet Cc: David S . Miller, Jakub Kicinski, Paolo Abeni, Simon Horman, netdev, linux-kernel, John Fastabend, Jesse Gross, gaoxingwang1, huyizhen On 7/16/2026 9:06 PM, Eric Dumazet wrote: > On Thu, Jul 16, 2026 at 2:53 PM Eric Dumazet <edumazet@google.com> wrote: >> >> On Thu, Jul 16, 2026 at 2:20 PM xietangxin <xietangxin@h-partners.com> wrote: >>> >>> [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device >>> >>> Hi all, >>> >>> We encountered a skb_under_panic triggered by toggling >>> NETIF_F_HW_VLAN_CTAG_TX on the lower device while a VLAN device is >>> up and sending traffic. >>> >>> Call trace >>> ========== >>> >>> skbuff: skb_under_panic: text:ffffc0d2900283d8 len:74 put:14 >>> head:ffff334820249c00 data:ffff334820249bfe tail:0x48 end:0xc0 dev:vlan4 >>> ------------[ cut here ]------------ >>> kernel BUG at net/core/skbuff.c:116! >>> Internal error: Oops - BUG: 00000000f2000800 [#1] SMP >>> Call trace: >>> skb_panic+0xcc/0xd0 >>> __skb_checksum+0x0/0x480 >>> eth_header+0x48/0x1a0 >>> vlan_dev_hard_header+0xd0/0x284 >>> neigh_connected_output+0x16c/0x20c >>> ip6_finish_output2+0x4b4/0xd74 >>> __ip6_finish_output.part.0+0x1ac/0x3b0 >>> ip6_finish_output+0x160/0x200 >>> ip6_output+0x13c/0x294 >>> ndisc_send_skb+0x41c/0x6f0 >>> ndisc_send_rs+0xac/0x3b0 >>> addrconf_rs_timer+0x42c/0x660 >>> call_timer_fn+0x54/0x290 >>> expire_timers+0x26c/0x420 >>> >>> Reproducer >>> ========== >>> >>> # Create veth pair (NETIF_F_HW_VLAN_CTAG_TX is ON by default) >>> ip link add veth0 type veth peer name veth1 >>> ip link set veth0 up >>> ip link set veth1 up >>> >>> # Turn off HW VLAN TX offload on lower device >>> ethtool -K veth0 tx-vlan-hw-insert off >>> >>> # Create VLAN device on veth0 >>> # At this point: header_ops = &vlan_header_ops, hard_header_len = 18 >>> ip link add link veth0 name veth0.10 type vlan id 10 reorder_hdr off >>> ip addr add 192.168.10.1/24 dev veth0.10 >>> ip link set veth0.10 up >>> >>> # Turn HW VLAN TX offload back ON on lower device >>> # This triggers NETDEV_FEAT_CHANGE -> vlan_transfer_features() >>> # hard_header_len changes from 18 to 14, but header_ops is NOT updated >>> ethtool -K veth0 tx-vlan-hw-insert on >>> >>> # When a packet is sent through veth0.10 >>> # - skb is allocated based on hard_header_len=14 -> ~16 bytes >>> # - vlan_dev_hard_header() pushes VLAN_HLEN(4) + ETH_HLEN(14) = 18 bytes >>> # - skb_under_panic! >>> >>> >>> Any feedback or guidance would be greatly appreciated. >> >> This rings a bell, I thought we already fixed this issue long ago :/ >> >> I would suggest we always add VLAN_HLEN even if not (yet) needed. >> >> diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c >> index 2b74ed56eb166d52c3351768d9dfedc7b2c8ec2d..d7f90b3b2bb3aadb2c466891720a7f407d0bc34b >> 100644 >> --- a/net/8021q/vlan.c >> +++ b/net/8021q/vlan.c >> @@ -323,10 +323,7 @@ static void vlan_transfer_features(struct net_device *dev, >> >> netif_inherit_tso_max(vlandev, dev); >> >> - if (vlan_hw_offload_capable(dev->features, vlan->vlan_proto)) >> - vlandev->hard_header_len = dev->hard_header_len; >> - else >> - vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN; >> + vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN; >> >> #if IS_ENABLED(CONFIG_FCOE) >> vlandev->fcoe_ddp_xid = dev->fcoe_ddp_xid; >> >> A kind-of-similar change is needed in net/8021q/vlan_dev.c > > The second part would be a nice cleanup, unless I am missing something. > > diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c > index 7aa3af8b10ead0384bbb2e767883b63a37be5fa2..dfd907e2ede2dcb0b6120e1fa5b6bb4d30bfb28a > 100644 > --- a/net/8021q/vlan_dev.c > +++ b/net/8021q/vlan_dev.c > @@ -496,26 +496,6 @@ static const struct header_ops vlan_header_ops = { > .parse_protocol = vlan_parse_protocol, > }; > > -static int vlan_passthru_hard_header(struct sk_buff *skb, struct > net_device *dev, > - unsigned short type, > - const void *daddr, const void *saddr, > - unsigned int len) > -{ > - struct vlan_dev_priv *vlan = vlan_dev_priv(dev); > - struct net_device *real_dev = vlan->real_dev; > - > - if (saddr == NULL) > - saddr = dev->dev_addr; > - > - return dev_hard_header(skb, real_dev, type, daddr, saddr, len); > -} > - > -static const struct header_ops vlan_passthru_header_ops = { > - .create = vlan_passthru_hard_header, > - .parse = eth_header_parse, > - .parse_protocol = vlan_parse_protocol, > -}; > - > static const struct device_type vlan_type = { > .name = "vlan", > }; > @@ -575,13 +555,8 @@ static int vlan_dev_init(struct net_device *dev) > #endif > > dev->needed_headroom = real_dev->needed_headroom; > - if (vlan_hw_offload_capable(real_dev->features, vlan->vlan_proto)) { > - dev->header_ops = &vlan_passthru_header_ops; > - dev->hard_header_len = real_dev->hard_header_len; > - } else { > - dev->header_ops = &vlan_header_ops; > - dev->hard_header_len = real_dev->hard_header_len + VLAN_HLEN; > - } > + dev->header_ops = &vlan_header_ops; > + dev->hard_header_len = real_dev->hard_header_len + VLAN_HLEN; > > dev->netdev_ops = &vlan_netdev_ops; Hi Eric, Thanks for the suggestion. I tested both parts of your patch locally and can confirm the skb_under_panic no longer reproduces. your diff is against an older tree. After commit cd1c188db109 ("vlan: defer real device state propagation to netdev_work"), vlan_transfer_features() has been moved to vlan_dev.c. Could you post a formal patch? -- Best regards, Tangxin Xie ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device 2026-07-17 6:42 ` xietangxin @ 2026-07-20 7:29 ` Eric Dumazet 0 siblings, 0 replies; 5+ messages in thread From: Eric Dumazet @ 2026-07-20 7:29 UTC (permalink / raw) To: xietangxin Cc: David S . Miller, Jakub Kicinski, Paolo Abeni, Simon Horman, netdev, linux-kernel, John Fastabend, Jesse Gross, gaoxingwang1, huyizhen On Fri, Jul 17, 2026 at 8:42 AM xietangxin <xietangxin@h-partners.com> wrote: > > > Hi Eric, > > Thanks for the suggestion. I tested both parts of your patch locally > and can confirm the skb_under_panic no longer reproduces. Excellent, thanks for testing. > > your diff is against an older tree. After commit > cd1c188db109 ("vlan: defer real device state propagation to netdev_work"), > vlan_transfer_features() has been moved to vlan_dev.c. Yeah, I had this patch in an old branch, I should have rebased it. > > Could you post a formal patch? Sure thing : https://lore.kernel.org/netdev/20260720072622.3541873-1-edumazet@google.com/T/#u ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-07-20 7:30 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-07-16 12:20 [BUG] vlan: skb_under_panic when toggling NETIF_F_HW_VLAN_CTAG_TX on lower device xietangxin 2026-07-16 12:53 ` Eric Dumazet 2026-07-16 13:06 ` Eric Dumazet 2026-07-17 6:42 ` xietangxin 2026-07-20 7:29 ` Eric Dumazet
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox