From: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
To: Jakub Kicinski <kuba@kernel.org>, davem@davemloft.net
Cc: netdev@vger.kernel.org, edumazet@google.com, pabeni@redhat.com,
andrew+netdev@lunn.ch, horms@kernel.org,
michael.chan@broadcom.com, manishc@marvell.com,
joshwash@google.com, hramamurthy@google.com,
tariqt@nvidia.com, saeedm@nvidia.com, daniel@iogearbox.net,
hawk@kernel.org, sdf@fomichev.me, willemb@google.com,
przemyslaw.kitszel@intel.com, bpf@vger.kernel.org,
Jakub Kicinski <kuba@kernel.org>
Subject: Re: [PATCH net-next 3/8] net: state the "XDP excludes HW-GRO" rule in the core
Date: Thu, 19 Nov 2026 17:59:08 -0500 [thread overview]
Message-ID: <willemdebruijn.kernel.7756e60d4e06@gmail.com> (raw)
In-Reply-To: <20261118232315.3593731-4-kuba@kernel.org>
Jakub Kicinski wrote:
> XDP has to see the frames as the wire delivered them, so HW-GRO must be
> off while a program is attached. Drivers currently open-code that policy,
> and they diverge: mlx5, gve and virtio_net refuse the attach, bnxt and
> qede clear the feature from ndo_fix_features(), and idpf does nothing
> at all and happily runs RSC under XDP.
>
> Clear HW-GRO in __netdev_update_features(), and make sure XDP setup
> paths refresh the features.
>
> Take extra care to make sure the disablement actually works, some
> drivers add the feature back in ndo_fix_features, even when removed
> from wanted features (mlx5 does for header-data-split).
>
> This change also removes the warn-only check dev_disable_gro_hw()
> performed. On mlx5 with tcp-data-split on, attaching a multi-buf
> generic XDP program reached generic_xdp_install(), whose
> dev_disable_gro_hw() cleared wanted_features, recomputed, saw
> mlx5e_fix_features() put NETIF_F_GRO_HW straight back, and splatted:
>
> ------------[ cut here ]------------
> netdevice: eth0: failed to disable GRO_HW!
> WARNING: net/core/dev.c:1880 at generic_xdp_install+0x135/0x190, CPU#53: ip/667
> RIP: 0010:generic_xdp_install+0x17a/0x190
> Call Trace:
> <TASK>
> dev_xdp_install+0x2aa/0x380
> dev_xdp_attach+0x41f/0x460
> dev_change_xdp_fd+0xbb/0x110
> do_setlink+0xfb9/0x1030
> rtnl_newlink+0x9f5/0xaf0
> rtnetlink_rcv_msg+0x442/0x490
> netlink_rcv_skb+0x16a/0x1a0
> netlink_unicast+0x1fc/0x320
> netlink_sendmsg+0x2d6/0x380
>
> and then attached anyway, leaving XDP looking at coalesced frames.
> The device also lost the bit from wanted_features for good, so HW-GRO
> never came back after detach - on mlx5 that took header-data split
> with it, and with it devmem and io_uring zero-copy Rx.
>
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> .../networking/net_cachelines/net_device.rst | 1 +
> include/linux/netdevice.h | 6 ++
> net/core/dev.c | 97 ++++++++++++++-----
> 3 files changed, 81 insertions(+), 23 deletions(-)
>
> diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst
> index 7620d3fcb8dc..25cf02a184ec 100644
> --- a/Documentation/networking/net_cachelines/net_device.rst
> +++ b/Documentation/networking/net_cachelines/net_device.rst
> @@ -169,6 +169,7 @@ unsigned_long:1 see_all_hwtstamp_requests
> unsigned_long:1 change_proto_down
> unsigned_long:1 netns_immutable
> unsigned_long:1 fcoe_mtu
> +unsigned_long:1 xdp_pending
> unsigned_long:1 xdp_from_upper
> struct list_head net_notifier_list
> struct macsec_ops* macsec_ops
> diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> index 0325c6f2a7d1..049538b13e61 100644
> --- a/include/linux/netdevice.h
> +++ b/include/linux/netdevice.h
> @@ -2531,6 +2531,12 @@ struct net_device {
> unsigned long change_proto_down:1;
> unsigned long netns_immutable:1;
> unsigned long fcoe_mtu:1;
> + /**
> + * @xdp_pending: an XDP program is being installed, but is not in
> + * @xdp_state yet. Most users should access it implicitly via
> + * helpers like dev_xdp_prog_count(), rather than directly.
> + */
> + unsigned long xdp_pending:1;
> /**
> * @xdp_from_upper: the program in @xdp_state was installed by an
> * upper device with netif_xdp_propagate(); it belongs to the
> diff --git a/net/core/dev.c b/net/core/dev.c
> index d88af07aea29..c0bb5eb9fdea 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -1901,23 +1901,6 @@ void netif_disable_lro(struct net_device *dev)
> }
> }
>
> -/**
> - * dev_disable_gro_hw - disable HW Generic Receive Offload on a device
> - * @dev: device
> - *
> - * Disable HW Generic Receive Offload (GRO_HW) on a net device. Must be
> - * called under RTNL. This is needed if Generic XDP is installed on
> - * the device.
> - */
> -static void dev_disable_gro_hw(struct net_device *dev)
> -{
> - dev->wanted_features &= ~NETIF_F_GRO_HW;
> - netdev_update_features(dev);
> -
> - if (unlikely(dev->features & NETIF_F_GRO_HW))
> - netdev_WARN(dev, "failed to disable GRO_HW!\n");
> -}
> -
> const char *netdev_cmd_to_name(enum netdev_cmd cmd)
> {
> #define N(val) \
> @@ -6652,7 +6635,6 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp)
> } else if (new && !old) {
> static_branch_inc(&generic_xdp_needed_key);
> netif_disable_lro(dev);
> - dev_disable_gro_hw(dev);
> }
> break;
>
> @@ -10571,7 +10553,7 @@ u8 dev_xdp_prog_count(struct net_device *dev)
> for (i = 0; i < __MAX_XDP_MODE; i++)
> if (dev->xdp_state[i].prog || dev->xdp_state[i].link)
> count++;
> - return count;
> + return count ?: !!dev->xdp_pending;
> }
> EXPORT_SYMBOL_GPL(dev_xdp_prog_count);
>
> @@ -10612,6 +10594,32 @@ static void dev_xdp_set_prog(struct net_device *dev, enum bpf_xdp_mode mode,
> dev->xdp_state[mode].prog = prog;
> }
>
> +/* Try to clear HW-GRO before the driver reconfigures for XDP.
> + * Driver may add features back in via ndo_fix_features(), due to feature
> + * dependencies. In that case we can't run XDP at all.
> + */
> +static int dev_xdp_clear_rx_agg(struct net_device *dev,
> + struct netlink_ext_ack *extack)
> +{
> + if (!(dev->features & NETIF_F_GRO_HW))
> + return 0;
> + if (!(dev->hw_features & NETIF_F_GRO_HW)) {
> + NL_SET_ERR_MSG(extack,
> + "unable to disable HW-GRO, fixed on");
> + return -EOPNOTSUPP;
> + }
> +
> + netdev_update_features(dev);
> +
> + if (dev->features & NETIF_F_GRO_HW) {
> + NL_SET_ERR_MSG(extack,
> + "unable to disable HW-GRO, required by another feature");
> + return -EBUSY;
> + }
> +
> + return 0;
> +}
> +
> int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
> {
> struct bpf_prog *old_prog;
> @@ -10645,11 +10653,17 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
> "unable to propagate XDP to device using memory provider");
> return -EBUSY;
> }
> +
> + dev->xdp_pending = 1;
> +
> + err = dev_xdp_clear_rx_agg(dev, bpf->extack);
> + if (err)
> + goto err_clear_pending;
> }
>
> err = dev->netdev_ops->ndo_bpf(dev, bpf);
> if (err)
> - return err;
> + goto err_clear_pending;
> if (bpf->prog)
> bpf_prog_inc(bpf->prog);
>
> @@ -10662,11 +10676,22 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
> old_prog = dev_xdp_prog(dev, XDP_MODE_DRV);
> bpf_prog_change_xdp(old_prog, bpf->prog);
> dev_xdp_set_prog(dev, XDP_MODE_DRV, bpf->prog);
> + dev->xdp_pending = 0;
> dev->xdp_from_upper = !!bpf->prog;
> - if (old_prog)
> +
> + if (old_prog) {
> bpf_prog_put(old_prog);
> + /* The last program is gone, HW-GRO may be available again */
> + if (!bpf->prog)
> + netdev_update_features(dev);
> + }
>
> return 0;
> +
> +err_clear_pending:
> + dev->xdp_pending = 0;
> + netdev_update_features(dev);
> + return err;
> }
> EXPORT_SYMBOL_GPL(netif_xdp_propagate);
>
> @@ -10875,19 +10900,36 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack
> return -EOPNOTSUPP;
> }
>
> + if (new_prog) {
> + dev->xdp_pending = 1;
> + err = dev_xdp_clear_rx_agg(dev, extack);
> + if (err)
> + goto err_clear_pending;
> + }
> +
> err = dev_xdp_install(dev, mode, bpf_op, extack, flags, new_prog);
> if (err)
> - return err;
> + goto err_clear_pending;
> }
>
> if (link)
> dev_xdp_set_link(dev, mode, link);
> else
> dev_xdp_set_prog(dev, mode, new_prog);
> - if (cur_prog)
> + dev->xdp_pending = 0;
> + if (cur_prog) {
> bpf_prog_put(cur_prog);
> + /* The last program is gone, HW-GRO may be available again */
> + if (!new_prog)
> + netdev_update_features(dev);
> + }
>
> return 0;
> +
> +err_clear_pending:
> + dev->xdp_pending = 0;
> + netdev_update_features(dev);
> + return err;
> }
>
> static int dev_xdp_attach_link(struct net_device *dev,
> @@ -10913,6 +10955,7 @@ static int dev_xdp_detach_link(struct net_device *dev,
> bpf_op = dev_xdp_bpf_op(dev, mode);
> WARN_ON(dev_xdp_install(dev, mode, bpf_op, NULL, 0, NULL));
> dev_xdp_set_link(dev, mode, NULL);
> + netdev_update_features(dev);
> return 0;
> }
>
> @@ -11385,6 +11428,14 @@ int __netdev_update_features(struct net_device *dev)
>
> features = netdev_get_wanted_features(dev);
>
> + /* GRO_HW may be added back in via ndo_fix_features e.g. if driver needs
> + * it to honor the HDS config, clear it before the fix features call.
> + */
> + if ((features & NETIF_F_GRO_HW) && dev_xdp_prog_count(dev)) {
> + netdev_dbg(dev, "Dropping HW-GRO feature since XDP is attached.\n");
> + features &= ~NETIF_F_GRO_HW;
> + }
> +
Apparently drivers can enable HW_GRO in some edge case (that can be
plugged itself?):
The rule is only enforced at attach time (dev_xdp_clear_rx_agg). Later
recomputes don't re-check after ndo_fix_features. Example:
1. Attach a multi-buf XDP program to mlx5 with HDS off. HW-GRO goes off.
2. Enable tcp-data-split. rings.c:253 only refuses this for
single-buffer XDP, via dev_xdp_sb_prog_count().
3. The next feature recompute runs mlx5e_fix_features(), and its
"Keeping HW-GRO enabled" block (en_main.c:4643) sets GRO_HW again,
after this strip. HW-GRO is now on with XDP attached.
This already happens today, but once 5/8 removes mlx5's own XDP check
the core is the only guard, and it misses this case. Either add a
post-fix_features check here (warn and strip, or refuse the change), or
make the HDS-enable path in rings.c refuse while any XDP program is
attached and the driver ties HDS to HW-GRO.
> if (dev->netdev_ops->ndo_fix_features)
> features = dev->netdev_ops->ndo_fix_features(dev, features);
>
> --
> 2.56.1
>
next prev parent reply other threads:[~2026-11-19 22:59 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 23:23 [PATCH net-next 0/8] net: adjust HW GRO and LRO feature handling Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 1/8] net: stop making HW-GRO depend on SW GRO Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 2/8] selftests: drv-net: drop the generic XDP hack from the HW GRO setup Jakub Kicinski
2026-11-19 21:40 ` Willem de Bruijn
2026-11-18 23:23 ` [PATCH net-next 3/8] net: state the "XDP excludes HW-GRO" rule in the core Jakub Kicinski
2026-11-19 22:59 ` Willem de Bruijn [this message]
2026-11-18 23:23 ` [PATCH net-next 4/8] net: state the "XDP excludes LRO" " Jakub Kicinski
2026-11-19 23:02 ` Willem de Bruijn
2026-11-18 23:23 ` [PATCH net-next 5/8] eth: mlx5e: drop the now-dead HW-GRO and LRO vs XDP handling Jakub Kicinski
2026-11-19 15:42 ` Dragos Tatulea
2026-11-18 23:23 ` [PATCH net-next 6/8] eth: bnxt_en: don't clear HW-GRO and LRO for XDP in ndo_fix_features Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 7/8] eth: qede: drop the now-dead HW-GRO vs XDP handling Jakub Kicinski
2026-11-18 23:23 ` [PATCH net-next 8/8] eth: gve: drop the now-dead HW-GRO vs XDP interlock Jakub Kicinski
2026-11-18 23:44 ` Ankit Garg
2026-11-19 22:14 ` Harshitha Ramamurthy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=willemdebruijn.kernel.7756e60d4e06@gmail.com \
--to=willemdebruijn.kernel@gmail.com \
--cc=andrew+netdev@lunn.ch \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=hawk@kernel.org \
--cc=horms@kernel.org \
--cc=hramamurthy@google.com \
--cc=joshwash@google.com \
--cc=kuba@kernel.org \
--cc=manishc@marvell.com \
--cc=michael.chan@broadcom.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=przemyslaw.kitszel@intel.com \
--cc=saeedm@nvidia.com \
--cc=sdf@fomichev.me \
--cc=tariqt@nvidia.com \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox