Netdev List
 help / color / mirror / Atom feed
From: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
To: Jakub Kicinski <kuba@kernel.org>,  davem@davemloft.net
Cc: netdev@vger.kernel.org,  edumazet@google.com,  pabeni@redhat.com,
	 andrew+netdev@lunn.ch,  horms@kernel.org,
	 michael.chan@broadcom.com,  manishc@marvell.com,
	 joshwash@google.com,  hramamurthy@google.com,
	 tariqt@nvidia.com,  saeedm@nvidia.com,  daniel@iogearbox.net,
	 hawk@kernel.org,  sdf@fomichev.me,  willemb@google.com,
	 przemyslaw.kitszel@intel.com,  bpf@vger.kernel.org,
	 Jakub Kicinski <kuba@kernel.org>
Subject: Re: [PATCH net-next 3/8] net: state the "XDP excludes HW-GRO" rule in the core
Date: Thu, 19 Nov 2026 17:59:08 -0500	[thread overview]
Message-ID: <willemdebruijn.kernel.7756e60d4e06@gmail.com> (raw)
In-Reply-To: <20261118232315.3593731-4-kuba@kernel.org>

Jakub Kicinski wrote:
> XDP has to see the frames as the wire delivered them, so HW-GRO must be
> off while a program is attached. Drivers currently open-code that policy,
> and they diverge: mlx5, gve and virtio_net refuse the attach, bnxt and
> qede clear the feature from ndo_fix_features(), and idpf does nothing
> at all and happily runs RSC under XDP.
> 
> Clear HW-GRO in __netdev_update_features(), and make sure XDP setup
> paths refresh the features.
> 
> Take extra care to make sure the disablement actually works, some
> drivers add the feature back in ndo_fix_features, even when removed
> from wanted features (mlx5 does for header-data-split).
> 
> This change also removes the warn-only check dev_disable_gro_hw()
> performed. On mlx5 with tcp-data-split on, attaching a multi-buf
> generic XDP program reached generic_xdp_install(), whose
> dev_disable_gro_hw() cleared wanted_features, recomputed, saw
> mlx5e_fix_features() put NETIF_F_GRO_HW straight back, and splatted:
> 
>   ------------[ cut here ]------------
>   netdevice: eth0: failed to disable GRO_HW!
>   WARNING: net/core/dev.c:1880 at generic_xdp_install+0x135/0x190, CPU#53: ip/667
>   RIP: 0010:generic_xdp_install+0x17a/0x190
>   Call Trace:
>    <TASK>
>    dev_xdp_install+0x2aa/0x380
>    dev_xdp_attach+0x41f/0x460
>    dev_change_xdp_fd+0xbb/0x110
>    do_setlink+0xfb9/0x1030
>    rtnl_newlink+0x9f5/0xaf0
>    rtnetlink_rcv_msg+0x442/0x490
>    netlink_rcv_skb+0x16a/0x1a0
>    netlink_unicast+0x1fc/0x320
>    netlink_sendmsg+0x2d6/0x380
> 
> and then attached anyway, leaving XDP looking at coalesced frames.
> The device also lost the bit from wanted_features for good, so HW-GRO
> never came back after detach - on mlx5 that took header-data split
> with it, and with it devmem and io_uring zero-copy Rx.
> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
>  .../networking/net_cachelines/net_device.rst  |  1 +
>  include/linux/netdevice.h                     |  6 ++
>  net/core/dev.c                                | 97 ++++++++++++++-----
>  3 files changed, 81 insertions(+), 23 deletions(-)
> 
> diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst
> index 7620d3fcb8dc..25cf02a184ec 100644
> --- a/Documentation/networking/net_cachelines/net_device.rst
> +++ b/Documentation/networking/net_cachelines/net_device.rst
> @@ -169,6 +169,7 @@ unsigned_long:1                     see_all_hwtstamp_requests
>  unsigned_long:1                     change_proto_down
>  unsigned_long:1                     netns_immutable
>  unsigned_long:1                     fcoe_mtu
> +unsigned_long:1                     xdp_pending
>  unsigned_long:1                     xdp_from_upper
>  struct list_head                    net_notifier_list
>  struct macsec_ops*                  macsec_ops
> diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> index 0325c6f2a7d1..049538b13e61 100644
> --- a/include/linux/netdevice.h
> +++ b/include/linux/netdevice.h
> @@ -2531,6 +2531,12 @@ struct net_device {
>  	unsigned long		change_proto_down:1;
>  	unsigned long		netns_immutable:1;
>  	unsigned long		fcoe_mtu:1;
> +	/**
> +	 * @xdp_pending: an XDP program is being installed, but is not in
> +	 *	@xdp_state yet. Most users should access it implicitly via
> +	 *	helpers like dev_xdp_prog_count(), rather than directly.
> +	 */
> +	unsigned long		xdp_pending:1;
>  	/**
>  	 * @xdp_from_upper: the program in @xdp_state was installed by an
>  	 *	upper device with netif_xdp_propagate(); it belongs to the
> diff --git a/net/core/dev.c b/net/core/dev.c
> index d88af07aea29..c0bb5eb9fdea 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -1901,23 +1901,6 @@ void netif_disable_lro(struct net_device *dev)
>  	}
>  }
>  
> -/**
> - *	dev_disable_gro_hw - disable HW Generic Receive Offload on a device
> - *	@dev: device
> - *
> - *	Disable HW Generic Receive Offload (GRO_HW) on a net device.  Must be
> - *	called under RTNL.  This is needed if Generic XDP is installed on
> - *	the device.
> - */
> -static void dev_disable_gro_hw(struct net_device *dev)
> -{
> -	dev->wanted_features &= ~NETIF_F_GRO_HW;
> -	netdev_update_features(dev);
> -
> -	if (unlikely(dev->features & NETIF_F_GRO_HW))
> -		netdev_WARN(dev, "failed to disable GRO_HW!\n");
> -}
> -
>  const char *netdev_cmd_to_name(enum netdev_cmd cmd)
>  {
>  #define N(val) 						\
> @@ -6652,7 +6635,6 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp)
>  		} else if (new && !old) {
>  			static_branch_inc(&generic_xdp_needed_key);
>  			netif_disable_lro(dev);
> -			dev_disable_gro_hw(dev);
>  		}
>  		break;
>  
> @@ -10571,7 +10553,7 @@ u8 dev_xdp_prog_count(struct net_device *dev)
>  	for (i = 0; i < __MAX_XDP_MODE; i++)
>  		if (dev->xdp_state[i].prog || dev->xdp_state[i].link)
>  			count++;
> -	return count;
> +	return count ?: !!dev->xdp_pending;
>  }
>  EXPORT_SYMBOL_GPL(dev_xdp_prog_count);
>  
> @@ -10612,6 +10594,32 @@ static void dev_xdp_set_prog(struct net_device *dev, enum bpf_xdp_mode mode,
>  	dev->xdp_state[mode].prog = prog;
>  }
>  
> +/* Try to clear HW-GRO before the driver reconfigures for XDP.
> + * Driver may add features back in via ndo_fix_features(), due to feature
> + * dependencies. In that case we can't run XDP at all.
> + */
> +static int dev_xdp_clear_rx_agg(struct net_device *dev,
> +				struct netlink_ext_ack *extack)
> +{
> +	if (!(dev->features & NETIF_F_GRO_HW))
> +		return 0;
> +	if (!(dev->hw_features & NETIF_F_GRO_HW)) {
> +		NL_SET_ERR_MSG(extack,
> +			       "unable to disable HW-GRO, fixed on");
> +		return -EOPNOTSUPP;
> +	}
> +
> +	netdev_update_features(dev);
> +
> +	if (dev->features & NETIF_F_GRO_HW) {
> +		NL_SET_ERR_MSG(extack,
> +			       "unable to disable HW-GRO, required by another feature");
> +		return -EBUSY;
> +	}
> +
> +	return 0;
> +}
> +
>  int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
>  {
>  	struct bpf_prog *old_prog;
> @@ -10645,11 +10653,17 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
>  				       "unable to propagate XDP to device using memory provider");
>  			return -EBUSY;
>  		}
> +
> +		dev->xdp_pending = 1;
> +
> +		err = dev_xdp_clear_rx_agg(dev, bpf->extack);
> +		if (err)
> +			goto err_clear_pending;
>  	}
>  
>  	err = dev->netdev_ops->ndo_bpf(dev, bpf);
>  	if (err)
> -		return err;
> +		goto err_clear_pending;
>  	if (bpf->prog)
>  		bpf_prog_inc(bpf->prog);
>  
> @@ -10662,11 +10676,22 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
>  	old_prog = dev_xdp_prog(dev, XDP_MODE_DRV);
>  	bpf_prog_change_xdp(old_prog, bpf->prog);
>  	dev_xdp_set_prog(dev, XDP_MODE_DRV, bpf->prog);
> +	dev->xdp_pending = 0;
>  	dev->xdp_from_upper = !!bpf->prog;
> -	if (old_prog)
> +
> +	if (old_prog) {
>  		bpf_prog_put(old_prog);
> +		/* The last program is gone, HW-GRO may be available again */
> +		if (!bpf->prog)
> +			netdev_update_features(dev);
> +	}
>  
>  	return 0;
> +
> +err_clear_pending:
> +	dev->xdp_pending = 0;
> +	netdev_update_features(dev);
> +	return err;
>  }
>  EXPORT_SYMBOL_GPL(netif_xdp_propagate);
>  
> @@ -10875,19 +10900,36 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack
>  			return -EOPNOTSUPP;
>  		}
>  
> +		if (new_prog) {
> +			dev->xdp_pending = 1;
> +			err = dev_xdp_clear_rx_agg(dev, extack);
> +			if (err)
> +				goto err_clear_pending;
> +		}
> +
>  		err = dev_xdp_install(dev, mode, bpf_op, extack, flags, new_prog);
>  		if (err)
> -			return err;
> +			goto err_clear_pending;
>  	}
>  
>  	if (link)
>  		dev_xdp_set_link(dev, mode, link);
>  	else
>  		dev_xdp_set_prog(dev, mode, new_prog);
> -	if (cur_prog)
> +	dev->xdp_pending = 0;
> +	if (cur_prog) {
>  		bpf_prog_put(cur_prog);
> +		/* The last program is gone, HW-GRO may be available again */
> +		if (!new_prog)
> +			netdev_update_features(dev);
> +	}
>  
>  	return 0;
> +
> +err_clear_pending:
> +	dev->xdp_pending = 0;
> +	netdev_update_features(dev);
> +	return err;
>  }
>  
>  static int dev_xdp_attach_link(struct net_device *dev,
> @@ -10913,6 +10955,7 @@ static int dev_xdp_detach_link(struct net_device *dev,
>  	bpf_op = dev_xdp_bpf_op(dev, mode);
>  	WARN_ON(dev_xdp_install(dev, mode, bpf_op, NULL, 0, NULL));
>  	dev_xdp_set_link(dev, mode, NULL);
> +	netdev_update_features(dev);
>  	return 0;
>  }
>  
> @@ -11385,6 +11428,14 @@ int __netdev_update_features(struct net_device *dev)
>  
>  	features = netdev_get_wanted_features(dev);
>  
> +	/* GRO_HW may be added back in via ndo_fix_features e.g. if driver needs
> +	 * it to honor the HDS config, clear it before the fix features call.
> +	 */
> +	if ((features & NETIF_F_GRO_HW) && dev_xdp_prog_count(dev)) {
> +		netdev_dbg(dev, "Dropping HW-GRO feature since XDP is attached.\n");
> +		features &= ~NETIF_F_GRO_HW;
> +	}
> +

Apparently drivers can enable HW_GRO in some edge case (that can be
plugged itself?):

    The rule is only enforced at attach time (dev_xdp_clear_rx_agg). Later
    recomputes don't re-check after ndo_fix_features. Example:
      1. Attach a multi-buf XDP program to mlx5 with HDS off. HW-GRO goes off.
      2. Enable tcp-data-split. rings.c:253 only refuses this for
         single-buffer XDP, via dev_xdp_sb_prog_count().
      3. The next feature recompute runs mlx5e_fix_features(), and its
         "Keeping HW-GRO enabled" block (en_main.c:4643) sets GRO_HW again,
         after this strip. HW-GRO is now on with XDP attached.
    This already happens today, but once 5/8 removes mlx5's own XDP check
    the core is the only guard, and it misses this case. Either add a
    post-fix_features check here (warn and strip, or refuse the change), or
    make the HDS-enable path in rings.c refuse while any XDP program is
    attached and the driver ties HDS to HW-GRO.

>  	if (dev->netdev_ops->ndo_fix_features)
>  		features = dev->netdev_ops->ndo_fix_features(dev, features);
>  
> -- 
> 2.56.1
> 



  reply	other threads:[~2026-11-19 22:59 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 23:23 [PATCH net-next 0/8] net: adjust HW GRO and LRO feature handling Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 1/8] net: stop making HW-GRO depend on SW GRO Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 2/8] selftests: drv-net: drop the generic XDP hack from the HW GRO setup Jakub Kicinski
2026-11-19 21:40   ` Willem de Bruijn
2026-11-18 23:23 ` [PATCH net-next 3/8] net: state the "XDP excludes HW-GRO" rule in the core Jakub Kicinski
2026-11-19 22:59   ` Willem de Bruijn [this message]
2026-11-18 23:23 ` [PATCH net-next 4/8] net: state the "XDP excludes LRO" " Jakub Kicinski
2026-11-19 23:02   ` Willem de Bruijn
2026-11-18 23:23 ` [PATCH net-next 5/8] eth: mlx5e: drop the now-dead HW-GRO and LRO vs XDP handling Jakub Kicinski
2026-11-19 15:42   ` Dragos Tatulea
2026-11-18 23:23 ` [PATCH net-next 6/8] eth: bnxt_en: don't clear HW-GRO and LRO for XDP in ndo_fix_features Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 7/8] eth: qede: drop the now-dead HW-GRO vs XDP handling Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 8/8] eth: gve: drop the now-dead HW-GRO vs XDP interlock Jakub Kicinski
2026-11-18 23:44   ` Ankit Garg
2026-11-19 22:14   ` Harshitha Ramamurthy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=willemdebruijn.kernel.7756e60d4e06@gmail.com \
    --to=willemdebruijn.kernel@gmail.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=hawk@kernel.org \
    --cc=horms@kernel.org \
    --cc=hramamurthy@google.com \
    --cc=joshwash@google.com \
    --cc=kuba@kernel.org \
    --cc=manishc@marvell.com \
    --cc=michael.chan@broadcom.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=przemyslaw.kitszel@intel.com \
    --cc=saeedm@nvidia.com \
    --cc=sdf@fomichev.me \
    --cc=tariqt@nvidia.com \
    --cc=willemb@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox