From: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
To: Jakub Kicinski <kuba@kernel.org>, davem@davemloft.net
Cc: netdev@vger.kernel.org, edumazet@google.com, pabeni@redhat.com,
andrew+netdev@lunn.ch, horms@kernel.org,
michael.chan@broadcom.com, manishc@marvell.com,
joshwash@google.com, hramamurthy@google.com,
tariqt@nvidia.com, saeedm@nvidia.com, daniel@iogearbox.net,
hawk@kernel.org, sdf@fomichev.me, willemb@google.com,
przemyslaw.kitszel@intel.com, bpf@vger.kernel.org,
Jakub Kicinski <kuba@kernel.org>
Subject: Re: [PATCH net-next 3/8] net: state the "XDP excludes HW-GRO" rule in the core
Date: Thu, 19 Nov 2026 17:59:08 -0500 [thread overview]
Message-ID: <willemdebruijn.kernel.7756e60d4e06@gmail.com> (raw)
In-Reply-To: <20261118232315.3593731-4-kuba@kernel.org>
Jakub Kicinski wrote:
> XDP has to see the frames as the wire delivered them, so HW-GRO must be
> off while a program is attached. Drivers currently open-code that policy,
> and they diverge: mlx5, gve and virtio_net refuse the attach, bnxt and
> qede clear the feature from ndo_fix_features(), and idpf does nothing
> at all and happily runs RSC under XDP.
>
> Clear HW-GRO in __netdev_update_features(), and make sure XDP setup
> paths refresh the features.
>
> Take extra care to make sure the disablement actually works, some
> drivers add the feature back in ndo_fix_features, even when removed
> from wanted features (mlx5 does for header-data-split).
>
> This change also removes the warn-only check dev_disable_gro_hw()
> performed. On mlx5 with tcp-data-split on, attaching a multi-buf
> generic XDP program reached generic_xdp_install(), whose
> dev_disable_gro_hw() cleared wanted_features, recomputed, saw
> mlx5e_fix_features() put NETIF_F_GRO_HW straight back, and splatted:
>
> ------------[ cut here ]------------
> netdevice: eth0: failed to disable GRO_HW!
> WARNING: net/core/dev.c:1880 at generic_xdp_install+0x135/0x190, CPU#53: ip/667
> RIP: 0010:generic_xdp_install+0x17a/0x190
> Call Trace:
> <TASK>
> dev_xdp_install+0x2aa/0x380
> dev_xdp_attach+0x41f/0x460
> dev_change_xdp_fd+0xbb/0x110
> do_setlink+0xfb9/0x1030
> rtnl_newlink+0x9f5/0xaf0
> rtnetlink_rcv_msg+0x442/0x490
> netlink_rcv_skb+0x16a/0x1a0
> netlink_unicast+0x1fc/0x320
> netlink_sendmsg+0x2d6/0x380
>
> and then attached anyway, leaving XDP looking at coalesced frames.
> The device also lost the bit from wanted_features for good, so HW-GRO
> never came back after detach - on mlx5 that took header-data split
> with it, and with it devmem and io_uring zero-copy Rx.
>
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> .../networking/net_cachelines/net_device.rst | 1 +
> include/linux/netdevice.h | 6 ++
> net/core/dev.c | 97 ++++++++++++++-----
> 3 files changed, 81 insertions(+), 23 deletions(-)
>
> diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst
> index 7620d3fcb8dc..25cf02a184ec 100644
> --- a/Documentation/networking/net_cachelines/net_device.rst
> +++ b/Documentation/networking/net_cachelines/net_device.rst
> @@ -169,6 +169,7 @@ unsigned_long:1 see_all_hwtstamp_requests
> unsigned_long:1 change_proto_down
> unsigned_long:1 netns_immutable
> unsigned_long:1 fcoe_mtu
> +unsigned_long:1 xdp_pending
> unsigned_long:1 xdp_from_upper
> struct list_head net_notifier_list
> struct macsec_ops* macsec_ops
> diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> index 0325c6f2a7d1..049538b13e61 100644
> --- a/include/linux/netdevice.h
> +++ b/include/linux/netdevice.h
> @@ -2531,6 +2531,12 @@ struct net_device {
> unsigned long change_proto_down:1;
> unsigned long netns_immutable:1;
> unsigned long fcoe_mtu:1;
> + /**
> + * @xdp_pending: an XDP program is being installed, but is not in
> + * @xdp_state yet. Most users should access it implicitly via
> + * helpers like dev_xdp_prog_count(), rather than directly.
> + */
> + unsigned long xdp_pending:1;
> /**
> * @xdp_from_upper: the program in @xdp_state was installed by an
> * upper device with netif_xdp_propagate(); it belongs to the
> diff --git a/net/core/dev.c b/net/core/dev.c
> index d88af07aea29..c0bb5eb9fdea 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -1901,23 +1901,6 @@ void netif_disable_lro(struct net_device *dev)
> }
> }
>
> -/**
> - * dev_disable_gro_hw - disable HW Generic Receive Offload on a device
> - * @dev: device
> - *
> - * Disable HW Generic Receive Offload (GRO_HW) on a net device. Must be
> - * called under RTNL. This is needed if Generic XDP is installed on
> - * the device.
> - */
> -static void dev_disable_gro_hw(struct net_device *dev)
> -{
> - dev->wanted_features &= ~NETIF_F_GRO_HW;
> - netdev_update_features(dev);
> -
> - if (unlikely(dev->features & NETIF_F_GRO_HW))
> - netdev_WARN(dev, "failed to disable GRO_HW!\n");
> -}
> -
> const char *netdev_cmd_to_name(enum netdev_cmd cmd)
> {
> #define N(val) \
> @@ -6652,7 +6635,6 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp)
> } else if (new && !old) {
> static_branch_inc(&generic_xdp_needed_key);
> netif_disable_lro(dev);
> - dev_disable_gro_hw(dev);
> }
> break;
>
> @@ -10571,7 +10553,7 @@ u8 dev_xdp_prog_count(struct net_device *dev)
> for (i = 0; i < __MAX_XDP_MODE; i++)
> if (dev->xdp_state[i].prog || dev->xdp_state[i].link)
> count++;
> - return count;
> + return count ?: !!dev->xdp_pending;
> }
> EXPORT_SYMBOL_GPL(dev_xdp_prog_count);
>
> @@ -10612,6 +10594,32 @@ static void dev_xdp_set_prog(struct net_device *dev, enum bpf_xdp_mode mode,
> dev->xdp_state[mode].prog = prog;
> }
>
> +/* Try to clear HW-GRO before the driver reconfigures for XDP.
> + * Driver may add features back in via ndo_fix_features(), due to feature
> + * dependencies. In that case we can't run XDP at all.
> + */
> +static int dev_xdp_clear_rx_agg(struct net_device *dev,
> + struct netlink_ext_ack *extack)
> +{
> + if (!(dev->features & NETIF_F_GRO_HW))
> + return 0;
> + if (!(dev->hw_features & NETIF_F_GRO_HW)) {
> + NL_SET_ERR_MSG(extack,
> + "unable to disable HW-GRO, fixed on");
> + return -EOPNOTSUPP;
> + }
> +
> + netdev_update_features(dev);
> +
> + if (dev->features & NETIF_F_GRO_HW) {
> + NL_SET_ERR_MSG(extack,
> + "unable to disable HW-GRO, required by another feature");
> + return -EBUSY;
> + }
> +
> + return 0;
> +}
> +
> int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
> {
> struct bpf_prog *old_prog;
> @@ -10645,11 +10653,17 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
> "unable to propagate XDP to device using memory provider");
> return -EBUSY;
> }
> +
> + dev->xdp_pending = 1;
> +
> + err = dev_xdp_clear_rx_agg(dev, bpf->extack);
> + if (err)
> + goto err_clear_pending;
> }
>
> err = dev->netdev_ops->ndo_bpf(dev, bpf);
> if (err)
> - return err;
> + goto err_clear_pending;
> if (bpf->prog)
> bpf_prog_inc(bpf->prog);
>
> @@ -10662,11 +10676,22 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
> old_prog = dev_xdp_prog(dev, XDP_MODE_DRV);
> bpf_prog_change_xdp(old_prog, bpf->prog);
> dev_xdp_set_prog(dev, XDP_MODE_DRV, bpf->prog);
> + dev->xdp_pending = 0;
> dev->xdp_from_upper = !!bpf->prog;
> - if (old_prog)
> +
> + if (old_prog) {
> bpf_prog_put(old_prog);
> + /* The last program is gone, HW-GRO may be available again */
> + if (!bpf->prog)
> + netdev_update_features(dev);
> + }
>
> return 0;
> +
> +err_clear_pending:
> + dev->xdp_pending = 0;
> + netdev_update_features(dev);
> + return err;
> }
> EXPORT_SYMBOL_GPL(netif_xdp_propagate);
>
> @@ -10875,19 +10900,36 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack
> return -EOPNOTSUPP;
> }
>
> + if (new_prog) {
> + dev->xdp_pending = 1;
> + err = dev_xdp_clear_rx_agg(dev, extack);
> + if (err)
> + goto err_clear_pending;
> + }
> +
> err = dev_xdp_install(dev, mode, bpf_op, extack, flags, new_prog);
> if (err)
> - return err;
> + goto err_clear_pending;
> }
>
> if (link)
> dev_xdp_set_link(dev, mode, link);
> else
> dev_xdp_set_prog(dev, mode, new_prog);
> - if (cur_prog)
> + dev->xdp_pending = 0;
> + if (cur_prog) {
> bpf_prog_put(cur_prog);
> + /* The last program is gone, HW-GRO may be available again */
> + if (!new_prog)
> + netdev_update_features(dev);
> + }
>
> return 0;
> +
> +err_clear_pending:
> + dev->xdp_pending = 0;
> + netdev_update_features(dev);
> + return err;
> }
>
> static int dev_xdp_attach_link(struct net_device *dev,
> @@ -10913,6 +10955,7 @@ static int dev_xdp_detach_link(struct net_device *dev,
> bpf_op = dev_xdp_bpf_op(dev, mode);
> WARN_ON(dev_xdp_install(dev, mode, bpf_op, NULL, 0, NULL));
> dev_xdp_set_link(dev, mode, NULL);
> + netdev_update_features(dev);
> return 0;
> }
>
> @@ -11385,6 +11428,14 @@ int __netdev_update_features(struct net_device *dev)
>
> features = netdev_get_wanted_features(dev);
>
> + /* GRO_HW may be added back in via ndo_fix_features e.g. if driver needs
> + * it to honor the HDS config, clear it before the fix features call.
> + */
> + if ((features & NETIF_F_GRO_HW) && dev_xdp_prog_count(dev)) {
> + netdev_dbg(dev, "Dropping HW-GRO feature since XDP is attached.\n");
> + features &= ~NETIF_F_GRO_HW;
> + }
> +
Apparently drivers can enable HW_GRO in some edge case (that can be
plugged itself?):
The rule is only enforced at attach time (dev_xdp_clear_rx_agg). Later
recomputes don't re-check after ndo_fix_features. Example:
1. Attach a multi-buf XDP program to mlx5 with HDS off. HW-GRO goes off.
2. Enable tcp-data-split. rings.c:253 only refuses this for
single-buffer XDP, via dev_xdp_sb_prog_count().
3. The next feature recompute runs mlx5e_fix_features(), and its
"Keeping HW-GRO enabled" block (en_main.c:4643) sets GRO_HW again,
after this strip. HW-GRO is now on with XDP attached.
This already happens today, but once 5/8 removes mlx5's own XDP check
the core is the only guard, and it misses this case. Either add a
post-fix_features check here (warn and strip, or refuse the change), or
make the HDS-enable path in rings.c refuse while any XDP program is
attached and the driver ties HDS to HW-GRO.
> if (dev->netdev_ops->ndo_fix_features)
> features = dev->netdev_ops->ndo_fix_features(dev, features);
>
> --
> 2.56.1
>
next prev parent reply other threads:[~2026-11-19 22:59 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 23:23 [PATCH net-next 0/8] net: adjust HW GRO and LRO feature handling Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 1/8] net: stop making HW-GRO depend on SW GRO Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 2/8] selftests: drv-net: drop the generic XDP hack from the HW GRO setup Jakub Kicinski
2026-11-19 21:40 ` Willem de Bruijn
2026-11-18 23:23 ` [PATCH net-next 3/8] net: state the "XDP excludes HW-GRO" rule in the core Jakub Kicinski
2026-11-19 22:59 ` Willem de Bruijn [this message]
2026-11-18 23:23 ` [PATCH net-next 4/8] net: state the "XDP excludes LRO" " Jakub Kicinski
2026-11-19 23:02 ` Willem de Bruijn
2026-11-19 23:24 ` sashiko-bot
2026-11-18 23:23 ` [PATCH net-next 5/8] eth: mlx5e: drop the now-dead HW-GRO and LRO vs XDP handling Jakub Kicinski
2026-11-19 15:42 ` Dragos Tatulea
2026-11-18 23:23 ` [PATCH net-next 6/8] eth: bnxt_en: don't clear HW-GRO and LRO for XDP in ndo_fix_features Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 7/8] eth: qede: drop the now-dead HW-GRO vs XDP handling Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 8/8] eth: gve: drop the now-dead HW-GRO vs XDP interlock Jakub Kicinski
2026-11-18 23:44 ` Ankit Garg
2026-11-19 22:14 ` Harshitha Ramamurthy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=willemdebruijn.kernel.7756e60d4e06@gmail.com \
--to=willemdebruijn.kernel@gmail.com \
--cc=andrew+netdev@lunn.ch \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=hawk@kernel.org \
--cc=horms@kernel.org \
--cc=hramamurthy@google.com \
--cc=joshwash@google.com \
--cc=kuba@kernel.org \
--cc=manishc@marvell.com \
--cc=michael.chan@broadcom.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=przemyslaw.kitszel@intel.com \
--cc=saeedm@nvidia.com \
--cc=sdf@fomichev.me \
--cc=tariqt@nvidia.com \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.