Netdev List
 help / color / mirror / Atom feed
* [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP
@ 2026-11-19 16:53 Magnus Karlsson
  2026-11-19 16:53 ` [PATCH net-next 1/5] xsk: add USO support for skb path Magnus Karlsson
                   ` (4 more replies)
  0 siblings, 5 replies; 8+ messages in thread
From: Magnus Karlsson @ 2026-11-19 16:53 UTC (permalink / raw)
  To: netdev, magnus.karlsson, magnus.karlsson, maciej.fijalkowski, sdf,
	davem, edumazet, kuba, pabeni, horms, bjorn, kerneljasonxing
  Cc: intel-wired-lan, anthony.l.nguyen, przemyslaw.kitszel,
	jacob.e.keller

This series adds UDP Segmentation Offload (USO) to AF_XDP. An
application submits one Ethernet frame holding a single IP and UDP
header followed by the aggregate UDP payload, and the device emits a
train of MTU-sized UDP datagrams. Patch 1 implements it for copy mode
(the skb path). Patch 2 is an ice preparation that adds Tx descriptor
builders, and patch 3 implements USO for zero-copy mode in the ice
driver.

AF_XDP never segments these packets itself: a request is rejected
unless the device offers USO. For the ice zero-copy path, the
segmentation is done in HW. In copy mode, the packet is passed
unsegmented to the bound device's driver, and as with UDP_SEGMENT
sockets, how the driver carries out the segmentation is up to it.
Stacked devices (VLAN, bonding, macvlan) forward the packet to a lower
device that may segment it in SW.

The design extends the existing Tx metadata support in AF_XDP. One new
flag, XDP_TXMD_FLAGS_USO, is added, and the reserved u32 in the request
part of struct xsk_tx_metadata is split into two u16 fields called
gso_size and network_offset. Zero-copy support is advertised through
the new tx-uso-zc netdev XSK feature.

network_offset is part of the uAPI because csum_start only locates L4,
while segmentation also needs the start of L3: the device rewrites the
IP header of every datagram and needs the L2 and L3 lengths, and on
the skb path it becomes the network header. With it, the kernel does
not need to understand the link layer, and the fields can be reused
later. Any future extension for tunnel segmentation would treat
network_offset as the outer L3 offset and csum_start as the inner L4
offset, adding only the outer L4 and inner L3 offsets.

To request USO, bind the socket in XDP_COPY mode, or in XDP_ZEROCOPY
mode on a driver advertising tx-uso-zc, and set XDP_TXMD_FLAGS_USO |
XDP_TXMD_FLAGS_CHECKSUM in the Tx metadata. Reserve at least 16 bytes
of metadata and set XDP_TX_METADATA on the first descriptor of the
packet.

Submit an Ethernet frame containing one IP header, one UDP header, and
the aggregate UDP payload. Set network_offset to the IP header offset,
csum_start to the UDP header offset, and csum_offset to the UDP
checksum field offset (6 bytes). Header offsets are measured from the
first descriptor's packet-data address, excluding metadata. Set
gso_size to the number of UDP payload bytes per output datagram,
excluding headers; the last datagram may of course be shorter.

USO is only available on Ethernet devices. IPv4 headers may carry
options but must not be fragmented. In IPv6, the UDP header must
directly follow the fixed header; extension headers are not supported.

The IP and UDP lengths must describe the aggregate packet. Initialize
the UDP checksum field with the partial pseudoheader checksum seed for
that packet. The device generates the per-datagram lengths and UDP
checksums and, for IPv4 segmentation, the IP header checksums. You must
supply a valid IPv4 header checksum as well, since a request that fits
in one datagram uses ordinary checksum offload without segmentation.

All headers through UDP must be in the first buffer. Multi-buffer
packets use XDP_USE_SG and XDP_PKT_CONTD, with metadata only on the
first descriptor. Each output IP packet must fit the device MTU, and
requests remain subject to the device's segmentation limits. Query
tx-udp-segmentation through ethtool to check whether skb UDP
segmentation offload is enabled.

USO cannot be combined with XDP_UMEM_TX_SW_CSUM or
XDP_TXMD_FLAGS_TIMESTAMP. Launch time can be combined with USO in copy
mode, but not in zero-copy mode as that has not been implemented on
ice or tested on any other device for that matter.

A USO request that is rejected is dropped: its buffers are completed
and counted in tx_invalid_descs. This is the only indication of the
failure, whatever its cause: invalid metadata or headers, the MTU,
segmentation limits, or a device without the offload. Errors in other
Tx metadata keep failing sendmsg() in copy mode, so the behaviour for
existing non-USO users is unchanged. Zero-copy drivers without
tx-uso-zc do not drop USO requests; they ignore the Tx metadata and
send the frame unsegmented and without offloads. ice behaves the same
way when it has fewer XDP Tx rings than CPUs, as queues then share XDP
Tx rings. This can be supported in a follow up patch series as it
would require quite some changes to the driver. Note that tx-uso-zc is
still reported in this case, since it is a static driver capability.

In both modes, requests that produce more than one datagram need
tx-udp-segmentation to be enabled; tx-uso-zc only reports driver
capability. In zero-copy mode, a packet may use at most
xdp-zc-max-segs descriptors, and ice specifically allows at most eight
buffers per output datagram, counting the copied header.

The header validator works on a raw header copy instead of an skb, so
that it can be shared by the skb and zero-copy paths. It lives in
net/xdp/xsk_tx_uso.c and include/net/xsk_tx_uso.h. In zero-copy mode,
ice copies the headers into kernel-owned memory before validating them
and handing them to the NIC, while the payload stays in the UMEM. I
tried finding an existing packet header validator that works on raw
buffers that could be reused, but I could not find one. Therefore this
small validator was added.

Testing was done on an ice NIC in loopback mode with a USO selftest,
in both copy and zero-copy mode. Patch 5 adds it as
selftests/net/xsk_uso. It checks the segment count, lengths, checksums
and payload of the received datagrams for IPv4 and IPv6, single and
multi-buffer requests, and that invalid requests are rejected. Patch 4
adds KUnit tests for the header validation helpers. Note that patch 5
should really be integrated into either xskxceiver or Stan's
xdp_hw_metadata tests. But as these are currently being modified to
fit into selftests/net and maybe even merged, I kept the USO tests
separate for now.

The tables below show the UDP packet rate achieved by a single socket
on my rather old 2.1 GHz Cascade Lake server with PCIe3, relative to
no offloads at 1472 bytes. Each request carries a UDP payload that is
segmented into 1472-byte datagrams, either in user space (no offloads,
so UDP checksum is also performed in SW using the kernel's
csum_partial() algorithm) or by the NIC (USO). The application and the
queue's interrupt are pinned to the same core, and the socket is
driven by preferred busy polling with napi_defer_hard_irqs=100 and
gro_flush_timeout=2000000, so there are virtually no device
interrupts. Each table's performance numbers are normalized to the
performance for 1472 byte packets with no offloads in that table.

SKB mode - write all packet data for each packet

+---------------+-------------+---------------+
| Payload bytes | No offloads | USO + HW csum |
+---------------+-------------+---------------+
|          1472 |       +0.0% |        +86.2% |
|          4416 |       +2.2% |       +192.1% |
|          7360 |       +2.6% |       +290.0% |
|         10304 |       +2.4% |       +301.5% |
|         13248 |       +2.4% |       +322.2% |
|         16192 |       +2.1% |       +355.6% |
+---------------+-------------+---------------+

SKB mode - write only headers for each packet

+---------------+-------------+---------------+
| Payload bytes | No offloads | USO + HW csum |
+---------------+-------------+---------------+
|          1472 |       +0.0% |        +88.7% |
|          4416 |       +2.3% |       +211.3% |
|          7360 |       +2.5% |       +342.0% |
|         10304 |       +2.6% |       +367.7% |
|         13248 |       +2.8% |       +394.1% |
|         16192 |       +2.8% |       +444.8% |
+---------------+-------------+---------------+

ZC mode - write all packet data for each packet

+---------------+-------------+---------------+
| Payload bytes | No offloads | USO + HW csum |
+---------------+-------------+---------------+
|          1472 |       +0.0% |       +185.6% |
|          4416 |       +4.1% |       +279.6% |
|          7360 |       +4.8% |       +332.1% |
|         10304 |       +4.4% |       +325.5% |
|         13248 |       +3.6% |       +333.4% |
|         16192 |       +3.5% |       +339.1% |
+---------------+-------------+---------------+

ZC mode - write only headers for each packet

+---------------+-------------+---------------+
| Payload bytes | No offloads | USO + HW csum |
+---------------+-------------+---------------+
|          1472 |       +0.0% |       +163.9% |
|          4416 |       +2.9% |       +260.0% |
|          7360 |       +3.1% |       +291.9% |
|         10304 |       +3.6% |       +303.9% |
|         13248 |       +4.2% |       +307.6% |
|         16192 |       +4.1% |       +315.8% |
+---------------+-------------+---------------+

At 1472 bytes the request fits in one datagram, so USO is not
segmenting anything and the gain comes from the checksum offload
alone. As soon as multiple UDP packets have to be produced on the
wire, USO becomes really beneficial.

For the zero-copy tests, my system seems to bottleneck at around 50
Gbps (about 4 Mpps) with 7K payloads or above. The board only has x8
lanes of PCIe3, though the slot is x16. In any case, there is a
benefit to USO even on this system.

Patch series:
1: Implements USO for skb path
2: Adds Tx descriptor builders and a header length check to ice
3: Implements USO for ice zero-copy driver
4: KUnit tests for the USO header helpers
5: selftests/net: AF_XDP USO functional test


Magnus Karlsson (5):
  xsk: add USO support for skb path
  ice: add Tx descriptor builders and header length check
  xsk: ice: add USO zero-copy support
  xsk: add KUnit tests for AF_XDP USO header helpers
  selftests: net: add AF_XDP USO test

 Documentation/netlink/specs/netdev.yaml       |   4 +
 Documentation/networking/xsk-tx-metadata.rst  | 124 +++
 MAINTAINERS                                   |   1 +
 drivers/net/ethernet/intel/ice/ice.h          |  10 +-
 drivers/net/ethernet/intel/ice/ice_ethtool.c  |   2 +
 .../net/ethernet/intel/ice/ice_lan_tx_rx.h    |   1 +
 drivers/net/ethernet/intel/ice/ice_main.c     |  13 +-
 drivers/net/ethernet/intel/ice/ice_sf_eth.c   |   1 +
 drivers/net/ethernet/intel/ice/ice_txrx.c     |  23 +-
 drivers/net/ethernet/intel/ice/ice_txrx.h     |  22 +-
 drivers/net/ethernet/intel/ice/ice_txrx_lib.h |  48 +
 drivers/net/ethernet/intel/ice/ice_xsk.c      | 502 ++++++++-
 drivers/net/ethernet/intel/ice/ice_xsk.h      |  13 +
 include/net/xdp_sock.h                        |   5 +
 include/net/xdp_sock_drv.h                    |  18 +
 include/net/xsk_buff_pool.h                   |   4 +
 include/net/xsk_tx_uso.h                      |  50 +
 include/uapi/linux/if_xdp.h                   |  14 +-
 include/uapi/linux/netdev.h                   |   3 +
 net/core/netdev-genl.c                        |   2 +
 net/xdp/.kunitconfig                          |   6 +
 net/xdp/Kconfig                               |  12 +
 net/xdp/Makefile                              |   2 +
 net/xdp/tests/xsk_tx_uso_kunit.c              | 620 ++++++++++++
 net/xdp/xsk.c                                 | 154 ++-
 net/xdp/xsk_buff_pool.c                       |   6 +
 net/xdp/xsk_queue.h                           |   6 +
 net/xdp/xsk_tx_uso.c                          | 174 ++++
 tools/include/uapi/linux/if_xdp.h             |  14 +-
 tools/include/uapi/linux/netdev.h             |   3 +
 tools/testing/selftests/net/.gitignore        |   1 +
 tools/testing/selftests/net/Makefile          |  10 +-
 tools/testing/selftests/net/xsk_uso.c         | 954 ++++++++++++++++++
 33 files changed, 2751 insertions(+), 71 deletions(-)
 create mode 100644 include/net/xsk_tx_uso.h
 create mode 100644 net/xdp/.kunitconfig
 create mode 100644 net/xdp/tests/xsk_tx_uso_kunit.c
 create mode 100644 net/xdp/xsk_tx_uso.c
 create mode 100644 tools/testing/selftests/net/xsk_uso.c

-- 
2.38.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH net-next 1/5] xsk: add USO support for skb path
  2026-11-19 16:53 [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Magnus Karlsson
@ 2026-11-19 16:53 ` Magnus Karlsson
  2026-11-19 18:47   ` Maciej Fijalkowski
  2026-11-19 16:53 ` [PATCH net-next 2/5] ice: add Tx descriptor builders and header length check Magnus Karlsson
                   ` (3 subsequent siblings)
  4 siblings, 1 reply; 8+ messages in thread
From: Magnus Karlsson @ 2026-11-19 16:53 UTC (permalink / raw)
  To: netdev, magnus.karlsson, magnus.karlsson, maciej.fijalkowski, sdf,
	davem, edumazet, kuba, pabeni, horms, bjorn, kerneljasonxing
  Cc: intel-wired-lan, anthony.l.nguyen, przemyslaw.kitszel,
	jacob.e.keller

Add UDP Segmentation Offload (USO) to the AF_XDP Tx metadata interface
and implement it for copy mode. An application submits one Ethernet
frame holding a single IP and UDP header followed by the aggregate UDP
payload, and the device emits a train of MTU-sized UDP datagrams.

The uAPI adds the XDP_TXMD_FLAGS_USO flag and splits the reserved u32
in the request part of struct xsk_tx_metadata into two u16 fields:
gso_size, the UDP payload bytes per output datagram, and
network_offset, the IP header offset. USO requires
XDP_TXMD_FLAGS_CHECKSUM, with csum_start pointing at the UDP header,
csum_offset at its checksum field and the checksum field seeded with
the partial pseudoheader checksum of the aggregate packet. It cannot
be combined with XDP_TXMD_FLAGS_TIMESTAMP or
XDP_UMEM_TX_SW_CSUM. Launch time is supported and applies to the start
of the segmented packet.

The request is turned into a GSO skb of type SKB_GSO_UDP_L4. Before
transmission, the headers are validated. The link layer is not parsed:
the IP header starts at network_offset and its version selects
unfragmented IPv4 or IPv6 without extension headers, followed by UDP.
The IP and UDP lengths must describe the aggregate packet, every output
datagram must fit the MTU, and the number of segments is limited to
UDP_MAX_SEGMENTS. As skb drivers find L3 through the EtherType, the skb
path also requires the EtherType, after any VLAN tags, to lead to the
same IP header. A request that fits in one datagram is sent as an
ordinary checksum offload request.

network_offset is part of the uAPI because csum_start only locates L4,
while segmentation also needs the start of L3: the device rewrites the
IP header of every datagram and needs the L2 and L3 lengths, and on
the skb path it becomes the network header. With it, the kernel does
not need to understand the link layer, and the fields can be reused
later. Any future tunnel segmentation extension to the uAPI could then
treat network_offset as the outer L3 offset and csum_start as the
inner L4 offset, adding only the outer L4 and inner L3 offsets.

AF_XDP never segments these packets itself. A request that produces more
than one datagram is rejected unless the bound device advertises
NETIF_F_GSO_UDP_L4, and the GSO skb is passed to its driver unsegmented.
As with UDP_SEGMENT sockets, how the segmentation is carried out is up
to the driver. Stacked devices such as VLAN or bonding forward the
packet to a lower device that may segment it in software.

A new pool field, tx_uso_supported, records whether USO requests are
accepted on a pool. It is set only for Ethernet devices (ARPHRD_ETHER)
and is cleared when XDP_UMEM_TX_SW_CSUM is used.

A USO request that fails validation is dropped: its buffers are
completed and counted in tx_invalid_descs. Errors in other Tx metadata
keep failing sendmsg() and leave the descriptor on the Tx ring, so the
existing behaviour for non-USO requests is unchanged. A USO request
that cannot be processed for lack of memory is also left on the Tx
ring, and sendmsg() fails with -EAGAIN, unless part of it was already
released to user space, either by an earlier call or while peeking
further descriptors. That part may already have been reused, so the
request is then dropped.

The header validator works on a raw header copy instead of an skb, so
that the zero-copy path in a later patch can reuse it. It lives in
the new files net/xdp/xsk_tx_uso.c and include/net/xsk_tx_uso.h.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 Documentation/networking/xsk-tx-metadata.rst |  91 +++++++++++++
 MAINTAINERS                                  |   1 +
 include/net/xdp_sock_drv.h                   |  12 ++
 include/net/xsk_buff_pool.h                  |   4 +
 include/net/xsk_tx_uso.h                     |  30 +++++
 include/uapi/linux/if_xdp.h                  |  13 +-
 net/xdp/Makefile                             |   1 +
 net/xdp/xsk.c                                | 130 +++++++++++++++++--
 net/xdp/xsk_buff_pool.c                      |   4 +
 net/xdp/xsk_queue.h                          |   6 +
 net/xdp/xsk_tx_uso.c                         |  77 +++++++++++
 tools/include/uapi/linux/if_xdp.h            |  13 +-
 12 files changed, 371 insertions(+), 11 deletions(-)
 create mode 100644 include/net/xsk_tx_uso.h
 create mode 100644 net/xdp/xsk_tx_uso.c

diff --git a/Documentation/networking/xsk-tx-metadata.rst b/Documentation/networking/xsk-tx-metadata.rst
index ac15e5fddc6b..41cfc5482b15 100644
--- a/Documentation/networking/xsk-tx-metadata.rst
+++ b/Documentation/networking/xsk-tx-metadata.rst
@@ -55,6 +55,9 @@ The flags field enables the particular offload:
   packet for transmission at a pre-determined time called launch time. The
   value of launch time is indicated by ``request.launch_time`` field of
   ``struct xsk_tx_metadata``.
+- ``XDP_TXMD_FLAGS_USO``: requests UDP segmentation offload.
+  ``request.gso_size`` specifies UDP payload bytes per output datagram;
+  ``request.network_offset`` identifies the IP header.
 
 Besides the flags above, in order to trigger the offloads, the first
 packet's ``struct xdp_desc`` descriptor should set ``XDP_TX_METADATA``
@@ -70,6 +73,69 @@ In this case, when running in ``XDP_COPY`` mode, the TX checksum
 is calculated on the CPU. Do not enable this option in production because
 it will negatively affect performance.
 
+UDP Segmentation
+================
+
+To request UDP segmentation offload (USO), bind the socket in
+``XDP_COPY`` mode and set ``XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM``
+in the Tx metadata. Reserve at least 16 bytes of metadata and set
+``XDP_TX_METADATA`` on the first descriptor of the packet.
+
+Submit an Ethernet frame containing one IP header, one UDP header, and the
+aggregate UDP payload. Set ``request.network_offset`` to the IP header
+offset, ``request.csum_start`` to the UDP header offset, and
+``request.csum_offset`` to the UDP checksum field offset (6 bytes).
+Header offsets are measured from the first descriptor's packet-data address,
+excluding metadata. Set ``request.gso_size`` to the number of UDP payload
+bytes per output datagram, excluding headers; the last datagram may be
+shorter.
+
+USO is only available on Ethernet devices. ``request.network_offset``
+locates the IP header, and its version selects IPv4 or IPv6. In copy mode,
+the EtherType, after up to six VLAN tags, must lead to the same IP header,
+since drivers rely on it. IPv4 headers may carry options but must not be
+fragmented. In IPv6, the UDP header must directly follow the fixed header;
+extension headers are not supported.
+
+The IP and UDP lengths must describe the aggregate packet. Initialize the
+UDP checksum field with the partial pseudoheader checksum seed for that
+packet. The device generates the per-datagram lengths and UDP checksums and,
+for IPv4 segmentation, the IP header checksums. Supply a valid IPv4 header
+checksum as well, since a request that fits in one datagram uses ordinary
+checksum offload without segmentation.
+
+All headers through UDP must be in the first buffer. Multi-buffer packets
+use ``XDP_USE_SG`` and ``XDP_PKT_CONTD``, with metadata only on the first
+descriptor. Each output IP packet must fit the device MTU, and requests
+remain subject to the device's segmentation limits. Query
+``tx-udp-segmentation`` through ethtool to check whether skb UDP segmentation
+offload is enabled.
+
+AF_XDP never segments the packet itself. In copy mode, a request that
+produces more than one datagram is rejected unless the bound device
+advertises UDP segmentation offload, and the packet is passed to its driver
+unsegmented. A request that fits in one datagram does not need the offload,
+so its success does not show that the device supports USO. As with
+``UDP_SEGMENT`` sockets, how the segmentation is carried out is up to the
+driver, which may build the per-datagram headers in software. Stacked
+devices such as VLAN, bonding, or macvlan advertise UDP segmentation
+regardless of the lower device and forward the packet through the lower
+device's qdisc, where it is segmented in software if the lower device lacks
+the offload.
+
+A USO request that is rejected is dropped: its buffers are returned
+through the completion ring and counted in the socket's ``tx_invalid_descs``
+statistic. This is the only indication of the failure, whatever its cause:
+invalid metadata or headers, the MTU, segmentation limits, or a device
+without the offload. A request that cannot be processed for lack of memory
+is instead left on the Tx ring, and ``sendmsg()`` fails with ``EAGAIN``,
+unless part of a multi-buffer request was already released to user space;
+it is then dropped. Errors in other Tx metadata also fail ``sendmsg()`` and
+leave the descriptor on the Tx ring.
+
+USO cannot be combined with ``XDP_UMEM_TX_SW_CSUM`` or
+``XDP_TXMD_FLAGS_TIMESTAMP``.
+
 Launch Time
 ===========
 
@@ -127,6 +193,31 @@ cycle time is set to 1 second, the horizon of the launch time ranges
 from 1 second to 2 seconds, depending on where the Qbv cycle is currently
 running.
 
+USO with launch time
+====================
+
+Add ``XDP_TXMD_FLAGS_LAUNCH_TIME`` to the USO and checksum flags and set
+``request.launch_time`` to schedule the segmented packet. This combination
+requires at least 24 bytes of Tx metadata.
+
+The launch time applies to the start of the segmented packet: the first
+output UDP datagram is scheduled, and subsequent datagrams follow according
+to the device's scheduling rules. This does not request per-datagram pacing
+or separate launch times.
+
+The skb path passes one unsegmented GSO skb and its launch time directly to
+the bound device's driver, bypassing that device's qdisc, including ETF.
+On stacked devices, the lower device's qdisc and software segmentation
+may still apply, as described above. The driver and selected queue
+must support UDP segmentation and launch time together. Separate support
+for either offload does not establish support for their combination.
+As with ordinary launch-time requests, drivers may ignore the time when
+scheduling is disabled or unsupported. Acceptance or Tx completion alone
+does not prove that a packet was transmitted at the requested time.
+
+The clock, queue configuration, and scheduling horizon requirements above
+also apply to USO launch-time requests.
+
 Querying Device Capabilities
 ============================
 
diff --git a/MAINTAINERS b/MAINTAINERS
index 40bb77cbc6b0..9b5f6f307d53 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -30588,6 +30588,7 @@ F:	Documentation/networking/af_xdp.rst
 F:	include/net/netns/xdp.h
 F:	include/net/xdp_sock*
 F:	include/net/xsk_buff_pool.h
+F:	include/net/xsk_tx_uso.h
 F:	include/uapi/linux/if_xdp.h
 F:	include/uapi/linux/xdp_diag.h
 F:	net/xdp/
diff --git a/include/net/xdp_sock_drv.h b/include/net/xdp_sock_drv.h
index a40ad3f90f13..6b5bb7b29f87 100644
--- a/include/net/xdp_sock_drv.h
+++ b/include/net/xdp_sock_drv.h
@@ -263,6 +263,7 @@ xsk_buff_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr, u32 options)
 		XDP_TXMD_FLAGS_TIMESTAMP | \
 		XDP_TXMD_FLAGS_CHECKSUM | \
 		XDP_TXMD_FLAGS_LAUNCH_TIME | \
+		XDP_TXMD_FLAGS_USO | \
 	0)
 
 static inline bool
@@ -270,6 +271,10 @@ xsk_buff_valid_tx_metadata(const struct xsk_buff_pool *pool,
 			   const struct xsk_tx_metadata *meta, u64 *flags)
 {
 	*flags = READ_ONCE(meta->flags);
+	if (*flags & XDP_TXMD_FLAGS_USO)
+		if (!pool->tx_uso_supported || pool->tx_metadata_len <
+		    offsetofend(struct xsk_tx_metadata, request.network_offset))
+			return false;
 	if (*flags & XDP_TXMD_FLAGS_LAUNCH_TIME)
 		if (pool->tx_metadata_len <
 		    offsetofend(struct xsk_tx_metadata, request.launch_time))
@@ -277,6 +282,13 @@ xsk_buff_valid_tx_metadata(const struct xsk_buff_pool *pool,
 	return !(*flags & ~XDP_TXMD_FLAGS_VALID);
 }
 
+/* USO requires checksum offload and cannot report a Tx timestamp. */
+static inline bool xsk_tx_uso_flags_valid(u64 flags)
+{
+	return (flags & XDP_TXMD_FLAGS_CHECKSUM) &&
+	       !(flags & XDP_TXMD_FLAGS_TIMESTAMP);
+}
+
 /**
  *  xsk_tx_metadata_request - Evaluate AF_XDP TX metadata at submission
  *  and call appropriate xsk_tx_metadata_ops operation.
diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h
index 02e89350faf9..ea2ff3680efa 100644
--- a/include/net/xsk_buff_pool.h
+++ b/include/net/xsk_buff_pool.h
@@ -89,6 +89,10 @@ struct xsk_buff_pool {
 	bool uses_need_wakeup;
 	bool unaligned;
 	bool tx_sw_csum;
+	/* USO requests accepted: Ethernet device without SW checksum, and
+	 * driver support in zero-copy mode. Set when the pool is bound.
+	 */
+	bool tx_uso_supported;
 	void *addrs;
 	/* Mutual exclusion of the completion ring in the SKB mode.
 	 * Protect: NAPI TX thread and sendmsg error paths in the SKB
diff --git a/include/net/xsk_tx_uso.h b/include/net/xsk_tx_uso.h
new file mode 100644
index 000000000000..f23384077712
--- /dev/null
+++ b/include/net/xsk_tx_uso.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _NET_XSK_TX_USO_H
+#define _NET_XSK_TX_USO_H
+
+#include <linux/errno.h>
+#include <linux/types.h>
+#include <net/xsk_buff_pool.h>
+
+/* Header offsets are relative to the start of the supplied Ethernet frame. */
+struct xsk_tx_uso_info {
+	struct xsk_tx_metadata metadata;
+	u32 len;
+	u16 header_len;
+	u16 gso_segs;
+	__be16 protocol;
+	u8 l4_protocol;
+};
+
+#ifdef CONFIG_XDP_SOCKETS
+int xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso,
+			u32 mtu);
+#else
+static inline int
+xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso, u32 mtu)
+{
+	return -EOPNOTSUPP;
+}
+#endif
+
+#endif
diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
index 50d67df78911..0f7ba4859218 100644
--- a/include/uapi/linux/if_xdp.h
+++ b/include/uapi/linux/if_xdp.h
@@ -134,6 +134,12 @@ struct xdp_options {
  */
 #define XDP_TXMD_FLAGS_LAUNCH_TIME		(1 << 2)
 
+/* Request UDP segmentation offload in XDP_COPY mode. Requires CHECKSUM.
+ * Segment size and IP header offset are in request.
+ * With LAUNCH_TIME, scheduling applies to the start of the segmented packet.
+ */
+#define XDP_TXMD_FLAGS_USO			(1 << 3)
+
 /* AF_XDP offloads request. 'request' union member is consumed by the driver
  * when the packet is being transmitted. 'completion' union member is
  * filled by the driver when the transmit completion arrives.
@@ -149,7 +155,12 @@ struct xsk_tx_metadata {
 			__u16 csum_start;
 			/* Offset from csum_start where checksum should be stored. */
 			__u16 csum_offset;
-			__u32 reserved;
+
+			/* XDP_TXMD_FLAGS_USO */
+			/* UDP payload bytes per output datagram. */
+			__u16 gso_size;
+			/* IP header offset from desc->addr. */
+			__u16 network_offset;
 
 			/* XDP_TXMD_FLAGS_LAUNCH_TIME */
 			/* Launch time in nanosecond against the PTP HW Clock */
diff --git a/net/xdp/Makefile b/net/xdp/Makefile
index 30cdc4315f42..b6109e0e5547 100644
--- a/net/xdp/Makefile
+++ b/net/xdp/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-$(CONFIG_XDP_SOCKETS) += xsk.o xdp_umem.o xsk_queue.o xskmap.o
 obj-$(CONFIG_XDP_SOCKETS) += xsk_buff_pool.o
+obj-$(CONFIG_XDP_SOCKETS) += xsk_tx_uso.o
 obj-$(CONFIG_XDP_SOCKETS_DIAG) += xsk_diag.o
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index f22f9a2cdbbe..14be62b8de5f 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -11,6 +11,8 @@
 
 #define pr_fmt(fmt) "AF_XDP: %s: " fmt, __func__
 
+#include <linux/if_arp.h>
+#include <linux/if_vlan.h>
 #include <linux/if_xdp.h>
 #include <linux/init.h>
 #include <linux/sched/mm.h>
@@ -22,6 +24,7 @@
 #include <linux/net.h>
 #include <linux/netdevice.h>
 #include <linux/rculist.h>
+#include <linux/udp.h>
 #include <linux/uio.h>
 #include <linux/vmalloc.h>
 
@@ -31,6 +34,7 @@
 #include <net/netdev_lock.h>
 #include <net/netdev_rx_queue.h>
 #include <net/xdp.h>
+#include <net/xsk_tx_uso.h>
 
 #include "../core/dev.h"
 
@@ -986,12 +990,62 @@ static void xsk_drop_skb(struct sk_buff *skb)
 	xs->skb = NULL;
 }
 
+static int xsk_skb_uso(struct sk_buff *skb)
+{
+	struct xsk_tx_uso_info uso = {
+		.metadata.request = {
+			.gso_size = skb_shinfo(skb)->gso_size,
+			.csum_start = skb_transport_offset(skb),
+			.network_offset = skb_network_offset(skb),
+		},
+		.header_len = skb_transport_offset(skb) + sizeof(struct udphdr),
+		.len = skb->len,
+	};
+	int depth, err;
+
+	/* Best effort: a bond or team device can change type while the pool
+	 * is bound, also after this check.
+	 */
+	if (READ_ONCE(skb->dev->type) != ARPHRD_ETHER)
+		return -EINVAL;
+	if (!pskb_may_pull(skb, uso.header_len))
+		return -ENOMEM;
+
+	err = xsk_tx_uso_validate(skb->data, &uso, READ_ONCE(skb->dev->mtu));
+	if (err)
+		return err;
+
+	skb_reset_mac_header(skb);
+	skb->protocol = eth_hdr(skb)->h_proto;
+	/* skb drivers find L3 through the EtherType and VLAN tags. Parse them
+	 * from the end of the Ethernet header, not from network_offset.
+	 */
+	skb->mac_len = ETH_HLEN;
+	if (__vlan_get_protocol(skb, skb->protocol, &depth) != uso.protocol ||
+	    depth != skb_network_offset(skb))
+		return -EINVAL;
+	skb_reset_mac_len(skb);
+	if (uso.gso_segs == 1) {
+		skb_gso_reset(skb);
+		return 0;
+	}
+
+	skb_shinfo(skb)->gso_segs = uso.gso_segs;
+	if (netif_needs_gso(skb, netif_skb_features(skb)))
+		return -EOPNOTSUPP;
+
+	return 0;
+}
+
+/* A failed USO request returns -EOVERFLOW, so that the packet is dropped. Other
+ * metadata errors are returned to user space.
+ */
 static int xsk_skb_metadata(struct sk_buff *skb, void *buffer,
 			    struct xdp_desc *desc, struct xsk_buff_pool *pool,
 			    u32 hr)
 {
 	struct xsk_tx_metadata *meta = NULL;
-	u16 csum_start, csum_offset;
+	int err = -EINVAL;
 	u64 flags;
 
 	if (unlikely(pool->tx_metadata_len == 0))
@@ -999,18 +1053,43 @@ static int xsk_skb_metadata(struct sk_buff *skb, void *buffer,
 
 	meta = buffer - pool->tx_metadata_len;
 	if (unlikely(!xsk_buff_valid_tx_metadata(pool, meta, &flags)))
-		return -EINVAL;
+		goto err;
+
+	if (flags & XDP_TXMD_FLAGS_USO) {
+		u16 nhoff = READ_ONCE(meta->request.network_offset);
+		u16 offset = READ_ONCE(meta->request.csum_offset);
+		u16 thoff = READ_ONCE(meta->request.csum_start);
+		u16 size = READ_ONCE(meta->request.gso_size);
+
+		if (!xsk_tx_uso_flags_valid(flags))
+			goto err;
+		if (!size || size == GSO_BY_FRAGS ||
+		    nhoff < ETH_HLEN || thoff < nhoff ||
+		    thoff + sizeof(struct udphdr) > desc->len ||
+		    offset != offsetof(struct udphdr, check) ||
+		    hr + thoff >= U16_MAX)
+			goto err;
+
+		skb_set_network_header(skb, nhoff);
+		skb_set_transport_header(skb, thoff);
+		skb_shinfo(skb)->gso_size = size;
+		skb_shinfo(skb)->gso_type = SKB_GSO_UDP_L4;
+	}
 
 	if (flags & XDP_TXMD_FLAGS_CHECKSUM) {
-		csum_start = READ_ONCE(meta->request.csum_start);
-		csum_offset = READ_ONCE(meta->request.csum_offset);
+		if (flags & XDP_TXMD_FLAGS_USO) {
+			skb->csum_start = skb_transport_offset(skb);
+			skb->csum_offset = offsetof(struct udphdr, check);
+		} else {
+			skb->csum_start = READ_ONCE(meta->request.csum_start);
+			skb->csum_offset = READ_ONCE(meta->request.csum_offset);
+		}
 
-		if (unlikely(csum_start + csum_offset +
+		if (unlikely(skb->csum_start + skb->csum_offset +
 			     sizeof(__sum16) > desc->len))
-			return -EINVAL;
+			goto err;
 
-		skb->csum_start = hr + csum_start;
-		skb->csum_offset = csum_offset;
+		skb->csum_start += hr;
 		skb->ip_summed = CHECKSUM_PARTIAL;
 	}
 
@@ -1021,6 +1100,9 @@ static int xsk_skb_metadata(struct sk_buff *skb, void *buffer,
 	xsk_tx_metadata_to_compl(meta, &skb_shinfo(skb)->xsk_meta);
 
 	return 0;
+
+err:
+	return flags & XDP_TXMD_FLAGS_USO ? -EOVERFLOW : err;
 }
 
 static int xsk_skb_sw_csum(struct sk_buff *skb, struct xsk_buff_pool *pool)
@@ -1108,6 +1190,10 @@ static struct sk_buff *xsk_build_skb_zerocopy(struct xdp_sock *xs,
 	return skb;
 }
 
+/* On -EOVERFLOW, the packet was dropped and its descriptors were consumed and
+ * completed as invalid. Other errors leave the descriptor on the Tx ring for
+ * the application to retry.
+ */
 static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
 				     struct xdp_desc *desc)
 {
@@ -1196,13 +1282,17 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
 	if (skb && !xs->skb)
 		kfree_skb(skb);
 
+	/* Too many descriptors and invalid USO requests drop the packet. Only
+	 * the latter can fail on the first descriptor, which is completed
+	 * here. Other errors are returned so that the application can retry.
+	 */
 	if (err == -EOVERFLOW) {
 		if (xs->skb) {
 			/* Drop the packet */
 			xsk_inc_num_desc(xs->skb);
 			xsk_drop_skb(xs->skb);
 		} else {
-			xsk_cq_cancel_locked(xs->pool, 1);
+			xsk_cq_submit_addr_single_locked(xs->pool, desc);
 			xs->tx->invalid_descs++;
 		}
 		xskq_cons_release(xs->tx);
@@ -1279,6 +1369,28 @@ static int __xsk_generic_xmit(struct sock *sk)
 			continue;
 		}
 
+		if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_L4) {
+			err = xsk_skb_uso(skb);
+			/* Descriptors already released to user space may be
+			 * reused, so the packet can only be put back if none
+			 * of them has been released. Otherwise it is dropped.
+			 */
+			if (err == -ENOMEM &&
+			    xskq_cons_nb_unreleased(xs->tx) >=
+			    xsk_get_num_desc(skb)) {
+				/* Tell user-space to retry the send */
+				xskq_cons_cancel_n(xs->tx, xsk_get_num_desc(skb));
+				xsk_consume_skb(skb);
+				err = -EAGAIN;
+				goto out;
+			}
+			if (err) {
+				xsk_drop_skb(skb);
+				err = 0;
+				continue;
+			}
+		}
+
 		err = xsk_skb_sw_csum(skb, xs->pool);
 		if (unlikely(err)) {
 			xsk_drop_skb(skb);
diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c
index 5ec092f7cb6f..e62f3c2b3362 100644
--- a/net/xdp/xsk_buff_pool.c
+++ b/net/xdp/xsk_buff_pool.c
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 
+#include <linux/if_arp.h>
 #include <linux/netdevice.h>
 #include <net/netdev_lock.h>
 #include <net/xsk_buff_pool.h>
@@ -192,6 +193,8 @@ int xp_assign_dev(struct xsk_buff_pool *pool,
 	err = xsk_reg_pool_at_qid(netdev, pool, queue_id);
 	if (err)
 		return err;
+	pool->tx_uso_supported = !pool->tx_sw_csum &&
+				 netdev->type == ARPHRD_ETHER;
 
 	if (mbuf)
 		pool->umem->flags |= XDP_UMEM_SG_FLAG;
@@ -260,6 +263,7 @@ int xp_assign_dev(struct xsk_buff_pool *pool,
 		goto err_unreg_xsk;
 	}
 	pool->umem->zc = true;
+	pool->tx_uso_supported = false;
 	pool->xdp_zc_max_segs = netdev->xdp_zc_max_segs;
 	return 0;
 
diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h
index 1bc42c8902f4..182cd1cccb74 100644
--- a/net/xdp/xsk_queue.h
+++ b/net/xdp/xsk_queue.h
@@ -390,6 +390,12 @@ static inline void xskq_cons_release(struct xsk_queue *q)
 	q->cached_cons++;
 }
 
+/* Entries consumed locally but not yet released to the producer. */
+static inline u32 xskq_cons_nb_unreleased(struct xsk_queue *q)
+{
+	return q->cached_cons - q->ring->consumer;
+}
+
 static inline void xskq_cons_cancel_n(struct xsk_queue *q, u32 cnt)
 {
 	q->cached_cons -= cnt;
diff --git a/net/xdp/xsk_tx_uso.c b/net/xdp/xsk_tx_uso.c
new file mode 100644
index 000000000000..a7313ade1092
--- /dev/null
+++ b/net/xdp/xsk_tx_uso.c
@@ -0,0 +1,77 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <net/ip.h>
+#include <net/ipv6.h>
+#include <net/udp.h>
+#include <net/xsk_tx_uso.h>
+
+/**
+ * xsk_tx_uso_validate - validate a kernel-owned plain UDP header
+ * @header: kernel-owned buffer containing at least uso->header_len bytes
+ * @uso: packet length and metadata offsets; receives protocol and segment count
+ * @mtu: maximum output IP packet length
+ *
+ * The link-layer header is not parsed. The IP header starts at network_offset
+ * and its version selects IPv4 or IPv6. The caller must provide a nonzero
+ * gso_size other than GSO_BY_FRAGS and header_len equal to transport_offset
+ * plus the UDP header size.
+ *
+ * Return: zero on success or a negative error code.
+ */
+int xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso,
+			u32 mtu)
+{
+	const struct xsk_tx_metadata *meta = &uso->metadata;
+	u32 nhoff = meta->request.network_offset;
+	u32 thoff = meta->request.csum_start;
+	u32 udp_len, payload_len, network_len;
+	const struct udphdr *uh;
+	__be16 protocol;
+
+	if (nhoff < ETH_HLEN || thoff <= nhoff || uso->header_len < thoff ||
+	    uso->len < uso->header_len)
+		return -EINVAL;
+
+	network_len = uso->len - nhoff;
+	switch (((const struct iphdr *)(header + nhoff))->version) {
+	case 4: {
+		const struct iphdr *iph = header + nhoff;
+
+		if (thoff - nhoff < sizeof(*iph) ||
+		    iph->ihl < 5 || iph->ihl * 4 != thoff - nhoff ||
+		    ip_is_fragment(iph) ||
+		    ntohs(iph->tot_len) != network_len)
+			return -EINVAL;
+		protocol = htons(ETH_P_IP);
+		uso->l4_protocol = iph->protocol;
+		break;
+	}
+	case 6: {
+		const struct ipv6hdr *ip6h = header + nhoff;
+
+		if (thoff - nhoff != sizeof(*ip6h) ||
+		    ntohs(ip6h->payload_len) != network_len - sizeof(*ip6h))
+			return -EINVAL;
+		protocol = htons(ETH_P_IPV6);
+		uso->l4_protocol = ip6h->nexthdr;
+		break;
+	}
+	default:
+		return -EPROTONOSUPPORT;
+	}
+	uso->protocol = protocol;
+	udp_len = uso->len - thoff;
+	uh = header + thoff;
+	if (uso->l4_protocol != IPPROTO_UDP || udp_len > U16_MAX ||
+	    ntohs(uh->len) != udp_len)
+		return -EINVAL;
+	payload_len = udp_len - sizeof(*uh);
+	if (payload_len > meta->request.gso_size * UDP_MAX_SEGMENTS ||
+	    uso->header_len - nhoff +
+	    min_t(u32, payload_len, meta->request.gso_size) > mtu)
+		return -EINVAL;
+
+	uso->gso_segs = max_t(u32, 1, DIV_ROUND_UP(payload_len,
+						   meta->request.gso_size));
+	return 0;
+}
+EXPORT_SYMBOL_GPL(xsk_tx_uso_validate);
diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
index 50d67df78911..0f7ba4859218 100644
--- a/tools/include/uapi/linux/if_xdp.h
+++ b/tools/include/uapi/linux/if_xdp.h
@@ -134,6 +134,12 @@ struct xdp_options {
  */
 #define XDP_TXMD_FLAGS_LAUNCH_TIME		(1 << 2)
 
+/* Request UDP segmentation offload in XDP_COPY mode. Requires CHECKSUM.
+ * Segment size and IP header offset are in request.
+ * With LAUNCH_TIME, scheduling applies to the start of the segmented packet.
+ */
+#define XDP_TXMD_FLAGS_USO			(1 << 3)
+
 /* AF_XDP offloads request. 'request' union member is consumed by the driver
  * when the packet is being transmitted. 'completion' union member is
  * filled by the driver when the transmit completion arrives.
@@ -149,7 +155,12 @@ struct xsk_tx_metadata {
 			__u16 csum_start;
 			/* Offset from csum_start where checksum should be stored. */
 			__u16 csum_offset;
-			__u32 reserved;
+
+			/* XDP_TXMD_FLAGS_USO */
+			/* UDP payload bytes per output datagram. */
+			__u16 gso_size;
+			/* IP header offset from desc->addr. */
+			__u16 network_offset;
 
 			/* XDP_TXMD_FLAGS_LAUNCH_TIME */
 			/* Launch time in nanosecond against the PTP HW Clock */
-- 
2.38.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH net-next 2/5] ice: add Tx descriptor builders and header length check
  2026-11-19 16:53 [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Magnus Karlsson
  2026-11-19 16:53 ` [PATCH net-next 1/5] xsk: add USO support for skb path Magnus Karlsson
@ 2026-11-19 16:53 ` Magnus Karlsson
  2026-11-19 16:53 ` [PATCH net-next 3/5] xsk: ice: add USO zero-copy support Magnus Karlsson
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 8+ messages in thread
From: Magnus Karlsson @ 2026-11-19 16:53 UTC (permalink / raw)
  To: netdev, magnus.karlsson, magnus.karlsson, maciej.fijalkowski, sdf,
	davem, edumazet, kuba, pabeni, horms, bjorn, kerneljasonxing
  Cc: intel-wired-lan, anthony.l.nguyen, przemyslaw.kitszel,
	jacob.e.keller

Add ice_build_tx_desc_offset() and ice_build_tso_ctx_qw1() to
ice_txrx_lib.h for the length fields of the Tx data descriptor and the
TSO context descriptor, and use them in ice_tso() and ice_tx_csum().
Add ice_tx_hdr_lens_ok() for the header length limits of the data
descriptor, and use it in ice_features_check().

This prepares for AF_XDP USO in zero-copy mode, which builds the same
descriptors from a copied packet header instead of an skb. The new
check also requires IPLEN to be a multiple of 4 bytes, as the field is
encoded in dwords, and so does the inner IPLEN check for encapsulated
packets. The IPv4 and IPv6 header lengths that reach them always are.
No functional change intended.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 drivers/net/ethernet/intel/ice/ice_main.c     | 12 ++---
 drivers/net/ethernet/intel/ice/ice_txrx.c     | 20 +++-----
 drivers/net/ethernet/intel/ice/ice_txrx_lib.h | 48 +++++++++++++++++++
 3 files changed, 60 insertions(+), 20 deletions(-)

diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index 47c6798c97e8..41bcac642291 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -10,6 +10,7 @@
 #include "ice.h"
 #include "ice_base.h"
 #include "ice_lib.h"
+#include "ice_txrx_lib.h"
 #include "ice_fltr.h"
 #include "ice_dcb_lib.h"
 #include "ice_dcb_nl.h"
@@ -9748,12 +9749,8 @@ ice_features_check(struct sk_buff *skb,
 	if (gso && (skb_shinfo(skb)->gso_size < ICE_TXD_CTX_MIN_MSS))
 		features &= ~NETIF_F_GSO_MASK;
 
-	len = skb_network_offset(skb);
-	if (len > ICE_TXD_MACLEN_MAX || len & 0x1)
-		goto out_rm_features;
-
-	len = skb_network_header_len(skb);
-	if (len > ICE_TXD_IPLEN_MAX || len & 0x1)
+	if (!ice_tx_hdr_lens_ok(skb_network_offset(skb),
+				skb_network_header_len(skb)))
 		goto out_rm_features;
 
 	if (skb->encapsulation) {
@@ -9770,8 +9767,9 @@ ice_features_check(struct sk_buff *skb,
 				goto out_rm_features;
 		}
 
+		/* IPLEN is encoded in dwords, see ice_tx_hdr_lens_ok() */
 		len = skb_inner_network_header_len(skb);
-		if (len > ICE_TXD_IPLEN_MAX || len & 0x1)
+		if (len > ICE_TXD_IPLEN_MAX || len & 3)
 			goto out_rm_features;
 	}
 
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c
index 35a480a1e172..7877c27293a6 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c
@@ -1624,7 +1624,6 @@ int ice_tx_csum(struct ice_tx_buf *first, struct ice_tx_offload_params *off)
 
 	/* compute outer L2 header size */
 	l2_len = ip.hdr - skb->data;
-	offset = (l2_len / 2) << ICE_TX_DESC_LEN_MACLEN_S;
 
 	/* set the tx_flags to indicate the IP protocol type. this is
 	 * required so that checksum header computation below is accurate.
@@ -1736,7 +1735,6 @@ int ice_tx_csum(struct ice_tx_buf *first, struct ice_tx_offload_params *off)
 
 	/* compute inner L3 header size */
 	l3_len = l4.hdr - ip.hdr;
-	offset |= (l3_len / 4) << ICE_TX_DESC_LEN_IPLEN_S;
 
 	if ((tx_ring->netdev->features & NETIF_F_HW_CSUM) &&
 	    !(first->tx_flags & ICE_TX_FLAGS_TSO) &&
@@ -1757,7 +1755,7 @@ int ice_tx_csum(struct ice_tx_buf *first, struct ice_tx_offload_params *off)
 		off->cd_qw1 |= ICE_TX_DESC_DTYPE_CTX;
 		off->cd_gcs_params = gcs_params;
 		/* Fill out CSO info in data descriptors */
-		off->td_offset |= offset;
+		off->td_offset |= ice_build_tx_desc_offset(l2_len, l3_len, 0);
 		off->td_cmd |= cmd;
 		return 1;
 	}
@@ -1767,26 +1765,24 @@ int ice_tx_csum(struct ice_tx_buf *first, struct ice_tx_offload_params *off)
 	case IPPROTO_TCP:
 		/* enable checksum offloads */
 		cmd |= ICE_TX_DESC_CMD_L4T_EOFT_TCP;
-		l4_len = l4.tcp->doff;
-		offset |= l4_len << ICE_TX_DESC_LEN_L4_LEN_S;
+		l4_len = l4.tcp->doff * sizeof(u32);
 		break;
 	case IPPROTO_UDP:
 		/* enable UDP checksum offload */
 		cmd |= ICE_TX_DESC_CMD_L4T_EOFT_UDP;
-		l4_len = (sizeof(struct udphdr) >> 2);
-		offset |= l4_len << ICE_TX_DESC_LEN_L4_LEN_S;
+		l4_len = sizeof(struct udphdr);
 		break;
 	case IPPROTO_SCTP:
 		/* enable SCTP checksum offload */
 		cmd |= ICE_TX_DESC_CMD_L4T_EOFT_SCTP;
-		l4_len = sizeof(struct sctphdr) >> 2;
-		offset |= l4_len << ICE_TX_DESC_LEN_L4_LEN_S;
+		l4_len = sizeof(struct sctphdr);
 		break;
 
 	default:
 		goto checksum_sw_fb;
 	}
 
+	offset = ice_build_tx_desc_offset(l2_len, l3_len, l4_len);
 	off->td_cmd |= cmd;
 	off->td_offset |= offset;
 	return 1;
@@ -1940,10 +1936,8 @@ int ice_tso(struct ice_tx_buf *first, struct ice_tx_offload_params *off)
 	cd_mss = skb_shinfo(skb)->gso_size;
 
 	/* record cdesc_qw1 with TSO parameters */
-	off->cd_qw1 |= (u64)(ICE_TX_DESC_DTYPE_CTX |
-			     (ICE_TX_CTX_DESC_TSO << ICE_TXD_CTX_QW1_CMD_S) |
-			     (cd_tso_len << ICE_TXD_CTX_QW1_TSO_LEN_S) |
-			     (cd_mss << ICE_TXD_CTX_QW1_MSS_S));
+	off->cd_qw1 |= ICE_TX_DESC_DTYPE_CTX |
+		       ice_build_tso_ctx_qw1(cd_tso_len, cd_mss);
 	first->tx_flags |= ICE_TX_FLAGS_TSO;
 	return 1;
 }
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx_lib.h b/drivers/net/ethernet/intel/ice/ice_txrx_lib.h
index f17990b68b62..e9dc17dab2a4 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx_lib.h
+++ b/drivers/net/ethernet/intel/ice/ice_txrx_lib.h
@@ -53,6 +53,54 @@ ice_build_ctob(u64 td_cmd, u64 td_offset, unsigned int size, u64 td_tag)
 			   (td_tag    << ICE_TXD_QW1_L2TAG1_S));
 }
 
+/**
+ * ice_tx_hdr_lens_ok - check header lengths against the Tx descriptor fields
+ * @mac_len: L2 header length in bytes
+ * @ip_len: L3 header length in bytes
+ *
+ * MACLEN is encoded in 2-byte words and IPLEN in 4-byte dwords, in fields of
+ * limited width.
+ *
+ * Return: true if the MACLEN and IPLEN fields of a Tx data descriptor can
+ * hold these lengths.
+ */
+static inline bool ice_tx_hdr_lens_ok(u32 mac_len, u32 ip_len)
+{
+	return mac_len <= ICE_TXD_MACLEN_MAX && !(mac_len & 1) &&
+	       ip_len <= ICE_TXD_IPLEN_MAX && !(ip_len & 3);
+}
+
+/**
+ * ice_build_tx_desc_offset - build the offset field of a Tx data descriptor
+ * @mac_len: L2 header length in bytes
+ * @ip_len: L3 header length in bytes
+ * @l4_len: L4 header length in bytes
+ *
+ * Return: the MACLEN field in 2-byte words, and the IPLEN and L4LEN fields in
+ * 4-byte dwords.
+ */
+static inline u32 ice_build_tx_desc_offset(u32 mac_len, u32 ip_len, u32 l4_len)
+{
+	return (mac_len / 2) << ICE_TX_DESC_LEN_MACLEN_S |
+	       (ip_len / 4) << ICE_TX_DESC_LEN_IPLEN_S |
+	       (l4_len / 4) << ICE_TX_DESC_LEN_L4_LEN_S;
+}
+
+/**
+ * ice_build_tso_ctx_qw1 - build quad word 1 of a TSO context descriptor
+ * @tso_len: payload bytes to segment, excluding the headers
+ * @mss: payload bytes per output segment
+ *
+ * Return: the TSO command with the TSO_LEN and MSS fields. The caller adds the
+ * descriptor type.
+ */
+static inline u64 ice_build_tso_ctx_qw1(u32 tso_len, u16 mss)
+{
+	return (u64)ICE_TX_CTX_DESC_TSO << ICE_TXD_CTX_QW1_CMD_S |
+	       (u64)tso_len << ICE_TXD_CTX_QW1_TSO_LEN_S |
+	       (u64)mss << ICE_TXD_CTX_QW1_MSS_S;
+}
+
 /**
  * ice_build_tstamp_desc - build Tx time stamp descriptor
  * @tx_desc: Tx LAN descriptor index
-- 
2.38.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH net-next 3/5] xsk: ice: add USO zero-copy support
  2026-11-19 16:53 [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Magnus Karlsson
  2026-11-19 16:53 ` [PATCH net-next 1/5] xsk: add USO support for skb path Magnus Karlsson
  2026-11-19 16:53 ` [PATCH net-next 2/5] ice: add Tx descriptor builders and header length check Magnus Karlsson
@ 2026-11-19 16:53 ` Magnus Karlsson
  2026-11-19 16:53 ` [PATCH net-next 4/5] xsk: add KUnit tests for AF_XDP USO header helpers Magnus Karlsson
  2026-11-19 16:53 ` [PATCH net-next 5/5] selftests: net: add AF_XDP USO test Magnus Karlsson
  4 siblings, 0 replies; 8+ messages in thread
From: Magnus Karlsson @ 2026-11-19 16:53 UTC (permalink / raw)
  To: netdev, magnus.karlsson, magnus.karlsson, maciej.fijalkowski, sdf,
	davem, edumazet, kuba, pabeni, horms, bjorn, kerneljasonxing
  Cc: intel-wired-lan, anthony.l.nguyen, przemyslaw.kitszel,
	jacob.e.keller

Add UDP Segmentation Offload (USO) for AF_XDP zero-copy and implement
it in the ice driver. Drivers advertise support with tmo_supports_uso,
which is reported to user space as the tx-uso-zc netdev XSK feature.

The core exports xsk_tx_uso_parse() and xsk_tx_uso_prepare_headers() for
drivers, which snapshot, validate and prepare the packet header, and
xsk_tx_invalid_descs() for accounting dropped requests. Drivers without
tmo_supports_uso keep handling a USO request as invalid metadata, so
xsk_tx_metadata_request() ignores the packet's metadata. As in copy
mode, requests that produce more than one datagram also need
NETIF_F_GSO_UDP_L4 on the device.

Packet headers are copied into kernel-owned memory before they are
validated and handed to the NIC, so that user space cannot change them
afterwards. ice keeps a DMA-mapped header slot for every two
descriptors of the XDP Tx ring, as a USO packet takes at least a
context and a header descriptor. The slots hold only header bytes and
are allocated page by page to avoid a large contiguous
allocation. They belong to the ring: they are allocated when a pool
that can carry USO requests is assigned to the ring, and freed with
the ring. They therefore follow XDP attach and detach, ring rebuilds
and netdev unregistration rather than the pool. Only the copied header
bytes are synced for the device. With fewer XDP rings than CPUs,
queues share XDP rings and would share their slots, so no slots are
allocated and USO requests are ignored for now, though support for
this could be added as a separate patch set in the future. tx-uso-zc
is still reported in this case, as documented. The slots are only
set up when an XDP program is attached. If they cannot be allocated
when a ring is reconfigured, the ring keeps its pool without USO and a
warning is logged, while a bind fails.

When the slots exist, ice_xmit_zc() processes descriptors one packet
at a time. A USO packet gets a context and a header descriptor, then
shares the data-descriptor loop with ordinary packets. The Tx budget
reserves worst-case descriptors per packet. The per-packet path is used
for every pool with a Tx metadata area that can carry USO requests,
whether or not it sends them. Other pools keep the existing batched
fast path. Both paths share the budget, tail, stats and wakeup
handling. If the Tx ring cannot hold a maximum-size packet in this
worst case, such a packet could never be peeked and Tx would stall, so
no header slots are allocated, USO requests are ignored and a warning
is logged. The pool flag is not cleared, as the pool may still fall
back to copy mode. This is only possible when PAGE_SIZE is 64K and the
Tx ring is set to its minimum size.

Packets that exceed the hardware limits are dropped and counted in
tx_invalid_descs through the new xsk_tx_invalid_descs() helper. Their
completions are deferred behind in-flight descriptors to keep the
completion ring in order. Packets without USO are dropped as well if
they would exceed the eight buffers, which is possible when chunks
larger than a data descriptor are split. A batch from the shared-umem
fallback can end inside a packet. That packet is dropped, and so is
its remainder in the next batch.

On completion, a buffer of the existing fast path still stands for one
completion, so that path writes nothing to tx_buf[] and struct
ice_tx_buf keeps its size. Buffers of the USO path carry an explicit
completion count in a union with fields that zero-copy rings do not use.
USO header slots, deferred drops and buffers split into several
descriptors are tracked with a per-ring counter, so ordinary traffic
keeps the fast cleanup path. The header slot of a context descriptor is
found from its position in the ring.

The descriptors are built with the descriptor builders that the
previous patch added for ice_tso().

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 Documentation/netlink/specs/netdev.yaml       |   4 +
 Documentation/networking/xsk-tx-metadata.rst  |  47 +-
 drivers/net/ethernet/intel/ice/ice.h          |  10 +-
 drivers/net/ethernet/intel/ice/ice_ethtool.c  |   2 +
 .../net/ethernet/intel/ice/ice_lan_tx_rx.h    |   1 +
 drivers/net/ethernet/intel/ice/ice_main.c     |   1 +
 drivers/net/ethernet/intel/ice/ice_sf_eth.c   |   1 +
 drivers/net/ethernet/intel/ice/ice_txrx.c     |   3 +
 drivers/net/ethernet/intel/ice/ice_txrx.h     |  22 +-
 drivers/net/ethernet/intel/ice/ice_xsk.c      | 502 ++++++++++++++++--
 drivers/net/ethernet/intel/ice/ice_xsk.h      |  13 +
 include/net/xdp_sock.h                        |   5 +
 include/net/xdp_sock_drv.h                    |   6 +
 include/net/xsk_tx_uso.h                      |  20 +
 include/uapi/linux/if_xdp.h                   |   5 +-
 include/uapi/linux/netdev.h                   |   3 +
 net/core/netdev-genl.c                        |   2 +
 net/xdp/xsk.c                                 |  24 +
 net/xdp/xsk_buff_pool.c                       |   4 +-
 net/xdp/xsk_tx_uso.c                          |  97 ++++
 tools/include/uapi/linux/if_xdp.h             |   5 +-
 tools/include/uapi/linux/netdev.h             |   3 +
 22 files changed, 729 insertions(+), 51 deletions(-)

diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlink/specs/netdev.yaml
index 0de7f93f0c4c..a19512be1798 100644
--- a/Documentation/netlink/specs/netdev.yaml
+++ b/Documentation/netlink/specs/netdev.yaml
@@ -85,6 +85,10 @@ definitions:
         name: tx-launch-time-fifo
         doc: >-
           Launch time HW offload is supported by the driver.
+      -
+        name: tx-uso-zc
+        doc: >-
+          UDP segmentation HW offload is supported for AF_XDP zero-copy.
   -
     name: queue-type
     type: enum
diff --git a/Documentation/networking/xsk-tx-metadata.rst b/Documentation/networking/xsk-tx-metadata.rst
index 41cfc5482b15..1f44fcce3e3d 100644
--- a/Documentation/networking/xsk-tx-metadata.rst
+++ b/Documentation/networking/xsk-tx-metadata.rst
@@ -77,7 +77,8 @@ UDP Segmentation
 ================
 
 To request UDP segmentation offload (USO), bind the socket in
-``XDP_COPY`` mode and set ``XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM``
+``XDP_COPY`` mode, or ``XDP_ZEROCOPY`` mode on a supporting driver, and set
+``XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM``
 in the Tx metadata. Reserve at least 16 bytes of metadata and set
 ``XDP_TX_METADATA`` on the first descriptor of the packet.
 
@@ -127,15 +128,33 @@ A USO request that is rejected is dropped: its buffers are returned
 through the completion ring and counted in the socket's ``tx_invalid_descs``
 statistic. This is the only indication of the failure, whatever its cause:
 invalid metadata or headers, the MTU, segmentation limits, or a device
-without the offload. A request that cannot be processed for lack of memory
-is instead left on the Tx ring, and ``sendmsg()`` fails with ``EAGAIN``,
-unless part of a multi-buffer request was already released to user space;
-it is then dropped. Errors in other Tx metadata also fail ``sendmsg()`` and
-leave the descriptor on the Tx ring.
+without the offload. In zero-copy mode, dropped requests are only counted
+when a single socket transmits on the UMEM. In copy mode, a request that
+cannot be processed for lack of memory is instead left on the Tx ring, and
+``sendmsg()`` fails with ``EAGAIN``, unless part of a multi-buffer request
+was already released to user space; it is then dropped. Errors in other Tx
+metadata also fail ``sendmsg()`` and leave the descriptor on the Tx ring.
 
 USO cannot be combined with ``XDP_UMEM_TX_SW_CSUM`` or
 ``XDP_TXMD_FLAGS_TIMESTAMP``.
 
+For zero-copy mode, query ``tx-uso-zc`` through the netdev netlink interface
+before submitting requests. Supporting drivers copy headers into
+kernel-owned memory for validation and hardware preparation while leaving
+payload bytes in UMEM. The link-layer header before
+``request.network_offset`` is not checked against the IP header and is sent
+as is. Completion returns each submitted buffer address once, regardless of
+the number of generated datagrams. Zero-copy drivers without ``tx-uso-zc``
+do not drop USO requests; they ignore the Tx metadata and send the frame
+unsegmented and without offloads.
+
+In both modes, requests that produce more than one datagram also need
+``tx-udp-segmentation`` to be enabled on the device. In zero-copy mode, a
+packet may use at most ``xdp-zc-max-segs`` descriptors, as reported through
+the netdev netlink interface, and drivers may have further limits. For
+example, ice allows at most eight buffers per output datagram, counting the
+copied header.
+
 Launch Time
 ===========
 
@@ -198,7 +217,8 @@ USO with launch time
 
 Add ``XDP_TXMD_FLAGS_LAUNCH_TIME`` to the USO and checksum flags and set
 ``request.launch_time`` to schedule the segmented packet. This combination
-requires at least 24 bytes of Tx metadata.
+requires at least 24 bytes of Tx metadata and is only supported in copy
+mode; zero-copy mode rejects it.
 
 The launch time applies to the start of the segmented packet: the first
 output UDP datagram is scheduled, and subsequent datagrams follow according
@@ -229,6 +249,19 @@ Query the ``xsk-features`` attribute in
 - ``tx-timestamp``: device supports ``XDP_TXMD_FLAGS_TIMESTAMP``
 - ``tx-checksum``: device supports ``XDP_TXMD_FLAGS_CHECKSUM``
 - ``tx-launch-time-fifo``: device supports ``XDP_TXMD_FLAGS_LAUNCH_TIME``
+- ``tx-uso-zc``: driver supports ``XDP_TXMD_FLAGS_USO`` in zero-copy mode
+
+``tx-uso-zc`` reports driver capability. In both modes, segmentation is
+also switched by the device's ``tx-udp-segmentation`` feature, subject to
+the limits described in the UDP Segmentation section. A driver may also be
+unable to offer USO in some configurations and then ignores USO requests,
+as drivers without ``tx-uso-zc`` do. ice does so when it has fewer XDP Tx
+rings than CPUs, as queues then share XDP Tx rings, when its Tx ring is too
+small for a maximum-size USO packet with the UMEM chunk size, and when it
+cannot allocate its header buffers while reconfiguring a ring with a bound
+socket. The last two are reported in the kernel log. If the header buffers
+cannot be allocated at bind time, the zero-copy bind fails instead, and a
+bind without ``XDP_ZEROCOPY`` falls back to copy mode.
 
 See ``tools/net/ynl/samples/netdev.c`` on how to query this information.
 
diff --git a/drivers/net/ethernet/intel/ice/ice.h b/drivers/net/ethernet/intel/ice/ice.h
index 432da05c48c0..e19929b37e11 100644
--- a/drivers/net/ethernet/intel/ice/ice.h
+++ b/drivers/net/ethernet/intel/ice/ice.h
@@ -833,16 +833,24 @@ static inline void ice_rx_xsk_pool(struct ice_rx_ring *ring)
  * carries a pointer to one of these XDP rings for its own purposes, such as
  * handling XDP_TX action, therefore we can piggyback here on the
  * rx_ring->xdp_ring assignment that was done during XDP rings initialization.
+ * The ring must be stopped, as its USO header slots are matched to the pool.
+ * If they cannot be allocated, the ring keeps the pool without USO.
  */
 static inline void ice_tx_xsk_pool(struct ice_vsi *vsi, u16 qid)
 {
+	struct xsk_buff_pool *pool;
 	struct ice_tx_ring *ring;
 
 	ring = vsi->rx_rings[qid]->xdp_ring;
 	if (!ring)
 		return;
 
-	WRITE_ONCE(ring->xsk_pool, ice_get_xp_from_qid(vsi, qid));
+	pool = ice_get_xp_from_qid(vsi, qid);
+	if (ice_xsk_cfg_uso_slots(ring, pool))
+		netdev_warn(vsi->netdev,
+			    "Failed to allocate USO header buffers, USO requests on queue %u are ignored\n",
+			    qid);
+	WRITE_ONCE(ring->xsk_pool, pool);
 }
 
 /**
diff --git a/drivers/net/ethernet/intel/ice/ice_ethtool.c b/drivers/net/ethernet/intel/ice/ice_ethtool.c
index dffa213734f1..423abdf2960d 100644
--- a/drivers/net/ethernet/intel/ice/ice_ethtool.c
+++ b/drivers/net/ethernet/intel/ice/ice_ethtool.c
@@ -3309,6 +3309,8 @@ ice_set_ringparam(struct net_device *netdev, struct ethtool_ringparam *ring,
 		xdp_rings[i].count = new_tx_cnt;
 		xdp_rings[i].desc = NULL;
 		xdp_rings[i].tx_buf = NULL;
+		xdp_rings[i].xsk_uso_pages = NULL;
+		xdp_rings[i].xsk_uso_nr_pages = 0;
 		err = ice_setup_tx_ring(&xdp_rings[i]);
 		if (err) {
 			while (i--)
diff --git a/drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h b/drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h
index 185672c7e17d..d4d02c2466d2 100644
--- a/drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h
+++ b/drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h
@@ -499,6 +499,7 @@ struct ice_tx_ctx_desc {
 
 #define ICE_TXD_CTX_QW1_MSS_S	50
 #define ICE_TXD_CTX_MIN_MSS	64
+#define ICE_TXD_CTX_MAX_MSS	0x3FFF
 
 #define ICE_TXD_CTX_QW1_VSI_S	50
 #define ICE_TXD_CTX_QW1_VSI_M	(0x3FFULL << ICE_TXD_CTX_QW1_VSI_S)
diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index 41bcac642291..d32a466869a9 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -3464,6 +3464,7 @@ static void ice_set_ops(struct ice_vsi *vsi)
 			       NETDEV_XDP_ACT_XSK_ZEROCOPY |
 			       NETDEV_XDP_ACT_RX_SG;
 	netdev->xdp_zc_max_segs = ICE_MAX_BUF_TXD;
+	ice_xsk_setup_metadata(netdev);
 }
 
 /**
diff --git a/drivers/net/ethernet/intel/ice/ice_sf_eth.c b/drivers/net/ethernet/intel/ice/ice_sf_eth.c
index a730aa368c92..d2a76a3e2cd3 100644
--- a/drivers/net/ethernet/intel/ice/ice_sf_eth.c
+++ b/drivers/net/ethernet/intel/ice/ice_sf_eth.c
@@ -54,6 +54,7 @@ static int ice_sf_cfg_netdev(struct ice_dynamic_port *dyn_port,
 			       NETDEV_XDP_ACT_XSK_ZEROCOPY |
 			       NETDEV_XDP_ACT_RX_SG;
 	netdev->xdp_zc_max_segs = ICE_MAX_BUF_TXD;
+	ice_xsk_setup_metadata(netdev);
 
 	eth_hw_addr_set(netdev, dyn_port->hw_addr);
 	ether_addr_copy(netdev->perm_addr, dyn_port->hw_addr);
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c
index 7877c27293a6..8fb76652837c 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c
@@ -228,6 +228,8 @@ void ice_clean_tx_ring(struct ice_tx_ring *tx_ring)
 
 	tx_ring->next_to_use = 0;
 	tx_ring->next_to_clean = 0;
+	tx_ring->xsk_uso_active = 0;
+	tx_ring->xsk_drain = false;
 
 	if (!tx_ring->netdev)
 		return;
@@ -250,6 +252,7 @@ void ice_free_tx_ring(struct ice_tx_ring *tx_ring)
 	u32 size;
 
 	ice_clean_tx_ring(tx_ring);
+	ice_xsk_free_uso_slots(tx_ring);
 	devm_kfree(tx_ring->dev, tx_ring->tx_buf);
 	tx_ring->tx_buf = NULL;
 
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.h b/drivers/net/ethernet/intel/ice/ice_txrx.h
index 5e517f219379..68cb588d7ed3 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.h
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.h
@@ -76,13 +76,17 @@
 
 /**
  * enum ice_tx_buf_type - type of &ice_tx_buf to act on Tx completion
- * @ICE_TX_BUF_EMPTY: unused OR XSk frame, no action required
+ * @ICE_TX_BUF_EMPTY: unused OR XSk frame, no action required, one completion
  * @ICE_TX_BUF_DUMMY: dummy Flow Director packet, unmap and kfree()
  * @ICE_TX_BUF_FRAG: mapped skb OR &xdp_buff frag, only unmap DMA
  * @ICE_TX_BUF_SKB: &sk_buff, unmap and consume_skb(), update stats
  * @ICE_TX_BUF_XDP_TX: &xdp_buff, unmap and page_frag_free(), stats
  * @ICE_TX_BUF_XDP_XMIT: &xdp_frame, unmap and xdp_return_frame(), stats
  * @ICE_TX_BUF_XSK_TX: &xdp_buff on XSk queue, xsk_buff_free(), stats
+ * @ICE_TX_BUF_XSK_USO: XSk USO context, sync header slot for the CPU
+ * @ICE_TX_BUF_XSK_DROP: XSk frame whose completions differ from its descriptor
+ *                       count, from dropped frames or split buffers
+ * @ICE_TX_BUF_XSK_DESC: XSk metadata-path frame, @xsk_completions completions
  */
 enum ice_tx_buf_type {
 	ICE_TX_BUF_EMPTY	= 0U,
@@ -92,6 +96,9 @@ enum ice_tx_buf_type {
 	ICE_TX_BUF_XDP_TX,
 	ICE_TX_BUF_XDP_XMIT,
 	ICE_TX_BUF_XSK_TX,
+	ICE_TX_BUF_XSK_USO,
+	ICE_TX_BUF_XSK_DROP,
+	ICE_TX_BUF_XSK_DESC,
 };
 
 struct ice_tx_buf {
@@ -109,6 +116,7 @@ struct ice_tx_buf {
 	union {
 		unsigned int gso_segs;
 		unsigned int nr_frags;	/* used for mbuf XDP */
+		u32 xsk_completions;	/* used for XSk metadata path */
 	};
 	u32 tx_flags:12;
 	u32 type:4;			/* &ice_tx_buf_type */
@@ -346,12 +354,22 @@ struct ice_rx_ring {
 	__cacheline_group_end_aligned(cold);
 } ____cacheline_internodealigned_in_smp;
 
+/* One DMA-mapped page of USO header slots. */
+struct ice_xsk_uso_page {
+	void *addr;
+	dma_addr_t dma;
+};
+
 struct ice_tx_ring {
 	__cacheline_group_begin_aligned(read_mostly);
 	void *desc;			/* Descriptor ring memory */
 	struct device *dev;		/* Used for DMA mapping */
 	u8 __iomem *tail;
 	struct ice_tx_buf *tx_buf;
+	struct ice_xsk_uso_page *xsk_uso_pages;	/* USO header slots */
+	u16 xsk_uso_slot_stride;
+	u16 xsk_uso_slots_per_page;
+	u32 xsk_uso_nr_pages;
 
 	struct ice_q_vector *q_vector;	/* Backreference to associated vector */
 	struct net_device *netdev;	/* netdev ring maps to */
@@ -377,6 +395,8 @@ struct ice_tx_ring {
 	u16 next_to_clean;
 
 	u16 xdp_tx_active;
+	u16 xsk_uso_active;
+	bool xsk_drain;			/* drop XSk descriptors up to EOP */
 	spinlock_t tx_lock;
 	__cacheline_group_end_aligned(read_write);
 
diff --git a/drivers/net/ethernet/intel/ice/ice_xsk.c b/drivers/net/ethernet/intel/ice/ice_xsk.c
index 0c1ce3e86a45..7fa1127480b6 100644
--- a/drivers/net/ethernet/intel/ice/ice_xsk.c
+++ b/drivers/net/ethernet/intel/ice/ice_xsk.c
@@ -5,6 +5,7 @@
 #include <linux/unroll.h>
 #include <net/libeth/xdp.h>
 #include <net/xdp_sock_drv.h>
+#include <net/xsk_tx_uso.h>
 #include <net/xdp.h>
 #include "ice.h"
 #include "ice_base.h"
@@ -133,6 +134,22 @@ static int ice_xsk_pool_disable(struct ice_vsi *vsi, u16 qid)
 	return 0;
 }
 
+/* Worst-case HW descriptors per AF_XDP descriptor on the USO path: the data
+ * split into maximum-size buffers plus a context and a header descriptor.
+ */
+static u32 ice_xsk_uso_desc_cost(const struct xsk_buff_pool *pool)
+{
+	return DIV_ROUND_UP(pool->chunk_size, ICE_MAX_DATA_PER_TXD_ALIGNED) + 2;
+}
+
+/* The Tx budget must fit a maximum-size packet, or it is never peeked. */
+static bool ice_xsk_uso_ring_fits(const struct ice_vsi *vsi, u16 count,
+				  const struct xsk_buff_pool *pool)
+{
+	return (count - 1) / ice_xsk_uso_desc_cost(pool) >=
+	       vsi->netdev->xdp_zc_max_segs;
+}
+
 /**
  * ice_xsk_pool_enable - enable a buffer pool region
  * @vsi: Current VSI
@@ -144,6 +161,7 @@ static int ice_xsk_pool_disable(struct ice_vsi *vsi, u16 qid)
 static int
 ice_xsk_pool_enable(struct ice_vsi *vsi, struct xsk_buff_pool *pool, u16 qid)
 {
+	struct ice_tx_ring *xdp_ring;
 	int err;
 
 	if (vsi->type != ICE_VSI_PF && vsi->type != ICE_VSI_SF)
@@ -153,14 +171,129 @@ ice_xsk_pool_enable(struct ice_vsi *vsi, struct xsk_buff_pool *pool, u16 qid)
 	    qid >= vsi->netdev->real_num_tx_queues)
 		return -EINVAL;
 
-	err = xsk_pool_dma_map(pool, ice_pf_to_dev(vsi->back),
-			       ICE_RX_DMA_ATTR);
-	if (err)
-		return err;
+	/* No header slots are allocated then, see ice_xsk_cfg_uso_slots(). The
+	 * pool flag is left alone, as copy mode is a fallback if this fails.
+	 */
+	if (pool->tx_metadata_len && pool->tx_uso_supported &&
+	    !ice_xsk_uso_ring_fits(vsi, vsi->num_tx_desc, pool))
+		netdev_warn(vsi->netdev, "Tx ring too small for USO with %u byte chunks, USO requests on queue %u are ignored\n",
+			    pool->chunk_size, qid);
+
+	/* Allocate the header slots before the pool is DMA mapped and can be
+	 * published to the rings, so that a failure leaves them without it.
+	 * Without XDP, rx_ring->xdp_ring is stale, and the slots are allocated
+	 * when the XDP rings are set up.
+	 */
+	xdp_ring = ice_is_xdp_ena_vsi(vsi) ? vsi->rx_rings[qid]->xdp_ring : NULL;
+	if (xdp_ring) {
+		err = ice_xsk_cfg_uso_slots(xdp_ring, pool);
+		if (err)
+			return err;
+	}
+
+	err = xsk_pool_dma_map(pool, ice_pf_to_dev(vsi->back), ICE_RX_DMA_ATTR);
+	if (err && xdp_ring)
+		ice_xsk_cfg_uso_slots(xdp_ring, NULL);
+	return err;
+}
+
+/**
+ * ice_xsk_free_uso_slots - free the USO header slots of an XDP Tx ring
+ * @ring: XDP Tx ring without outstanding descriptors
+ */
+void ice_xsk_free_uso_slots(struct ice_tx_ring *ring)
+{
+	u32 i;
+
+	if (!ring->xsk_uso_pages)
+		return;
+
+	for (i = 0; i < ring->xsk_uso_nr_pages; i++) {
+		struct ice_xsk_uso_page *page = &ring->xsk_uso_pages[i];
+
+		if (page->addr)
+			dma_free_noncoherent(ring->dev, PAGE_SIZE, page->addr,
+					     page->dma, DMA_TO_DEVICE);
+	}
+	kfree(ring->xsk_uso_pages);
+	ring->xsk_uso_pages = NULL;
+	ring->xsk_uso_nr_pages = 0;
+	ring->xsk_uso_slots_per_page = 0;
+	ring->xsk_uso_slot_stride = 0;
+}
+
+/**
+ * ice_xsk_cfg_uso_slots - match the USO header slots of a ring to its pool
+ * @ring: XDP Tx ring without outstanding descriptors
+ * @pool: buffer pool bound to the ring, or NULL
+ *
+ * Header slots are only needed when USO requests can be submitted. They
+ * belong to the ring, one per two descriptors, and are freed with it. They
+ * are allocated in single pages to avoid a large contiguous allocation.
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+int ice_xsk_cfg_uso_slots(struct ice_tx_ring *ring, struct xsk_buff_pool *pool)
+{
+	u32 stride = ALIGN(NET_IP_ALIGN + XSK_TX_USO_MAX_HEADER,
+			   dma_get_cache_alignment());
+	u32 per_page = PAGE_SIZE / stride;
+	u32 nr_pages = DIV_ROUND_UP(ring->count / 2, per_page);
+	u32 i;
+
+	/* With fewer XDP rings than CPUs, queues share XDP rings and so would
+	 * share their slots. USO requests are then ignored, as by drivers
+	 * without USO support. The same goes for a Tx ring too small for a
+	 * maximum-size USO packet.
+	 */
+	if (!pool || !pool->tx_metadata_len || !pool->tx_uso_supported ||
+	    ring->vsi->num_xdp_txq < num_possible_cpus() ||
+	    !ice_xsk_uso_ring_fits(ring->vsi, ring->count, pool)) {
+		ice_xsk_free_uso_slots(ring);
+		return 0;
+	}
+	if (ring->xsk_uso_pages && ring->xsk_uso_nr_pages == nr_pages)
+		return 0;
+
+	ice_xsk_free_uso_slots(ring);
+	ring->xsk_uso_pages = kcalloc(nr_pages, sizeof(*ring->xsk_uso_pages),
+				      GFP_KERNEL);
+	if (!ring->xsk_uso_pages)
+		return -ENOMEM;
+	ring->xsk_uso_nr_pages = nr_pages;
+
+	for (i = 0; i < nr_pages; i++) {
+		struct ice_xsk_uso_page *page = &ring->xsk_uso_pages[i];
+
+		page->addr = dma_alloc_noncoherent(ring->dev, PAGE_SIZE,
+						   &page->dma, DMA_TO_DEVICE,
+						   GFP_KERNEL);
+		if (!page->addr) {
+			ice_xsk_free_uso_slots(ring);
+			return -ENOMEM;
+		}
+	}
+	ring->xsk_uso_slot_stride = stride;
+	ring->xsk_uso_slots_per_page = per_page;
 
 	return 0;
 }
 
+/* Every USO packet takes a context and a header descriptor, so in-flight
+ * context descriptors are never adjacent and, as the ring size is even, half
+ * the context index is a unique header slot. Return the page and set the slot
+ * offset within it.
+ */
+static struct ice_xsk_uso_page *
+ice_xsk_uso_slot(const struct ice_tx_ring *ring, u16 ctx_idx, u32 *offset)
+{
+	u32 slot = ctx_idx >> 1;
+
+	*offset = slot % ring->xsk_uso_slots_per_page *
+		  ring->xsk_uso_slot_stride;
+	return &ring->xsk_uso_pages[slot / ring->xsk_uso_slots_per_page];
+}
+
 /**
  * ice_realloc_rx_xdp_bufs - reallocate for either XSK or normal buffer
  * @rx_ring: Rx ring
@@ -354,6 +487,36 @@ bool ice_alloc_rx_bufs_zc(struct ice_rx_ring *rx_ring,
 	return __ice_alloc_rx_bufs_zc(rx_ring, xsk_pool, leftover);
 }
 
+static u32 ice_clean_xsk_buf(struct ice_tx_ring *ring, struct ice_tx_buf *buf)
+{
+	u32 completed = buf->xsk_completions;
+	struct ice_xsk_uso_page *page;
+	u32 offset;
+
+	switch (buf->type) {
+	case ICE_TX_BUF_EMPTY:
+		return 1;
+	case ICE_TX_BUF_XSK_TX:
+		xsk_buff_free(buf->xdp);
+		ring->xdp_tx_active--;
+		break;
+	case ICE_TX_BUF_XSK_USO:
+		page = ice_xsk_uso_slot(ring, buf - ring->tx_buf, &offset);
+		dma_sync_single_range_for_cpu(ring->dev, page->dma, offset,
+					      ring->xsk_uso_slot_stride,
+					      DMA_TO_DEVICE);
+		ring->xsk_uso_active--;
+		break;
+	case ICE_TX_BUF_XSK_DROP:
+		ring->xsk_uso_active--;
+		break;
+	default:
+		break;
+	}
+	buf->type = ICE_TX_BUF_EMPTY;
+	return completed;
+}
+
 /**
  * ice_clean_xdp_irq_zc - produce AF_XDP descriptors to CQ
  * @xdp_ring: XDP Tx ring
@@ -365,9 +528,8 @@ static u32 ice_clean_xdp_irq_zc(struct ice_tx_ring *xdp_ring,
 	u16 ntc = xdp_ring->next_to_clean;
 	struct ice_tx_desc *tx_desc;
 	u16 cnt = xdp_ring->count;
-	struct ice_tx_buf *tx_buf;
 	u16 completed_frames = 0;
-	u16 xsk_frames = 0;
+	u32 xsk_frames = 0;
 	u16 last_rs;
 	int i;
 
@@ -384,22 +546,14 @@ static u32 ice_clean_xdp_irq_zc(struct ice_tx_ring *xdp_ring,
 	if (!completed_frames)
 		return 0;
 
-	if (likely(!xdp_ring->xdp_tx_active)) {
+	if (likely(!xdp_ring->xsk_uso_active && !xdp_ring->xdp_tx_active)) {
 		xsk_frames = completed_frames;
 		goto skip;
 	}
 
 	ntc = xdp_ring->next_to_clean;
 	for (i = 0; i < completed_frames; i++) {
-		tx_buf = &xdp_ring->tx_buf[ntc];
-
-		if (tx_buf->type == ICE_TX_BUF_XSK_TX) {
-			tx_buf->type = ICE_TX_BUF_EMPTY;
-			xsk_buff_free(tx_buf->xdp);
-			xdp_ring->xdp_tx_active--;
-		} else {
-			xsk_frames++;
-		}
+		xsk_frames += ice_clean_xsk_buf(xdp_ring, &xdp_ring->tx_buf[ntc]);
 
 		ntc++;
 		if (ntc >= xdp_ring->count)
@@ -469,6 +623,7 @@ static int ice_xmit_xdp_tx_zc(struct xdp_buff *xdp,
 
 		tx_buf->xdp = xdp;
 		tx_buf->type = ICE_TX_BUF_XSK_TX;
+		tx_buf->xsk_completions = 0;
 		tx_desc->buf_addr = cpu_to_le64(dma);
 		tx_desc->cmd_type_offset_bsz = ice_build_ctob(0, 0, size, 0);
 		/* account for each xdp_buff from xsk_buff_pool */
@@ -791,6 +946,272 @@ static void ice_fill_tx_hw_ring(struct ice_tx_ring *xdp_ring,
 		ice_xmit_pkt(xdp_ring, xsk_pool, &descs[i], total_bytes);
 }
 
+static void ice_xsk_data_desc(struct ice_tx_ring *ring, dma_addr_t dma,
+			      u32 len, u32 cmd, u32 offset, u32 completions)
+{
+	struct ice_tx_desc *desc = ICE_TX_DESC(ring, ring->next_to_use);
+	struct ice_tx_buf *buf = &ring->tx_buf[ring->next_to_use];
+
+	buf->type = ICE_TX_BUF_XSK_DESC;
+	buf->xsk_completions = completions;
+	desc->buf_addr = cpu_to_le64(dma);
+	desc->cmd_type_offset_bsz = ice_build_ctob(cmd, offset, len, 0);
+	if (++ring->next_to_use == ring->count)
+		ring->next_to_use = 0;
+}
+
+/* Include the copied header in the eight-buffer limit for every wire segment. */
+static bool ice_xsk_uso_fits(const struct xdp_desc *descs, u32 nr,
+			     const struct xsk_tx_uso_info *uso)
+{
+	u16 network_offset = uso->metadata.request.network_offset;
+	u16 transport_offset = uso->metadata.request.csum_start;
+	u16 gso_size = uso->metadata.request.gso_size;
+	u32 remaining = gso_size, buffers = 1, i;
+
+	/* The core does not know the width of the descriptor length fields. */
+	if (!ice_tx_hdr_lens_ok(network_offset,
+				transport_offset - network_offset) ||
+	    (uso->gso_segs > 1 &&
+	     (gso_size < ICE_TXD_CTX_MIN_MSS || gso_size > ICE_TXD_CTX_MAX_MSS)))
+		return false;
+
+	for (i = 0; i < nr; i++) {
+		u32 len = descs[i].len - (i ? 0 : uso->header_len);
+
+		while (len) {
+			u32 chunk = min_t(u32, len, ICE_MAX_DATA_PER_TXD_ALIGNED);
+			u32 left = chunk;
+
+			len -= chunk;
+			while (left) {
+				u32 bytes = min(left, remaining);
+
+				if (++buffers > ICE_MAX_BUF_TXD)
+					return false;
+				left -= bytes;
+				remaining -= bytes;
+				if (!remaining) {
+					remaining = gso_size;
+					buffers = 1;
+				}
+			}
+		}
+	}
+	return true;
+}
+
+/* A buffer longer than one data descriptor is split, so a packet without USO
+ * can then need more data descriptors than the hardware allows.
+ */
+static bool ice_xsk_bufs_fit(const struct xsk_buff_pool *pool,
+			     const struct xdp_desc *descs, u32 nr)
+{
+	u32 buffers = 0, i;
+
+	if (likely(pool->chunk_size <= ICE_MAX_DATA_PER_TXD_ALIGNED))
+		return true;
+	for (i = 0; i < nr; i++)
+		buffers += DIV_ROUND_UP(descs[i].len,
+					ICE_MAX_DATA_PER_TXD_ALIGNED);
+	return buffers <= ICE_MAX_BUF_TXD;
+}
+
+/* Write the context and header descriptors of a USO packet and return the
+ * command and offset fields for its payload descriptors.
+ */
+static int ice_xsk_xmit_uso_hdr(struct ice_tx_ring *ring,
+				const struct xdp_desc *descs, u32 nr,
+				const struct xsk_tx_uso_info *info, u32 *cmd,
+				u32 *offset)
+{
+	const struct xsk_tx_metadata *meta = &info->metadata;
+	u16 slot_idx = ring->next_to_use;
+	u64 qw1 = ICE_TX_DESC_DTYPE_CTX;
+	struct ice_xsk_uso_page *page;
+	struct ice_tx_ctx_desc *ctx;
+	struct ice_tx_buf *buf;
+	dma_addr_t header_dma;
+	u32 slot_offset;
+	bool eop;
+
+	if (!ice_xsk_uso_fits(descs, nr, info))
+		return -EOPNOTSUPP;
+
+	page = ice_xsk_uso_slot(ring, slot_idx, &slot_offset);
+	header_dma = page->dma + slot_offset + NET_IP_ALIGN;
+
+	xsk_tx_uso_prepare_headers(info, page->addr + slot_offset + NET_IP_ALIGN);
+	dma_sync_single_range_for_device(ring->dev, page->dma, slot_offset,
+					 NET_IP_ALIGN + info->header_len,
+					 DMA_TO_DEVICE);
+
+	*cmd = ICE_TX_DESC_CMD_L4T_EOFT_UDP;
+	if (info->protocol == htons(ETH_P_IP))
+		*cmd |= info->gso_segs > 1 ? ICE_TX_DESC_CMD_IIPT_IPV4_CSUM :
+					  ICE_TX_DESC_CMD_IIPT_IPV4;
+	else
+		*cmd |= ICE_TX_DESC_CMD_IIPT_IPV6;
+	*offset = ice_build_tx_desc_offset(meta->request.network_offset,
+					   meta->request.csum_start -
+					   meta->request.network_offset,
+					   sizeof(struct udphdr));
+
+	/* The context owns the header until the whole submitted batch completes. */
+	buf = &ring->tx_buf[slot_idx];
+	buf->type = ICE_TX_BUF_XSK_USO;
+	buf->xsk_completions = 0;
+	ring->xsk_uso_active++;
+
+	if (info->gso_segs > 1)
+		qw1 |= ice_build_tso_ctx_qw1(info->len - info->header_len,
+					     meta->request.gso_size);
+	ctx = (struct ice_tx_ctx_desc *)ICE_TX_DESC(ring, slot_idx);
+	memset(ctx, 0, sizeof(*ctx));
+	ctx->qw1 = cpu_to_le64(qw1);
+	if (++ring->next_to_use == ring->count)
+		ring->next_to_use = 0;
+
+	eop = info->len == info->header_len;
+	ice_xsk_data_desc(ring, header_dma, info->header_len,
+			  *cmd | (eop ? ICE_TX_DESC_CMD_EOP : 0), *offset,
+			  eop ? nr : 0);
+	return 0;
+}
+
+/* Write the data descriptors of a packet, skipping its first @skip bytes. */
+static void ice_xsk_xmit_descs(struct ice_tx_ring *ring,
+			       struct xsk_buff_pool *pool,
+			       const struct xdp_desc *descs, u32 nr, u32 skip,
+			       u32 cmd, u32 offset)
+{
+	bool split = false;
+	u32 i;
+
+	for (i = 0; i < nr; i++, skip = 0) {
+		u32 len = descs[i].len - skip;
+		dma_addr_t dma;
+
+		if (!len)
+			continue;
+		dma = xsk_buff_raw_get_dma(pool, descs[i].addr) + skip;
+		xsk_buff_raw_dma_sync_for_device(pool, dma, len);
+		while (len) {
+			u32 chunk = min_t(u32, len, ICE_MAX_DATA_PER_TXD_ALIGNED);
+			bool eop = i == nr - 1 && len == chunk;
+
+			split |= len > chunk;
+			ice_xsk_data_desc(ring, dma, chunk,
+					  cmd | (eop ? ICE_TX_DESC_CMD_EOP : 0),
+					  offset, eop ? nr : 0);
+			dma += chunk;
+			len -= chunk;
+		}
+	}
+	/* Split buffers break the one-completion-per-descriptor cleanup
+	 * shortcut, so make the cleanup walk this packet.
+	 */
+	if (unlikely(split)) {
+		u16 last = ring->next_to_use ? ring->next_to_use - 1 : ring->count - 1;
+
+		ring->tx_buf[last].type = ICE_TX_BUF_XSK_DROP;
+		ring->xsk_uso_active++;
+	}
+}
+
+static void ice_xsk_drop_packet(struct ice_tx_ring *ring,
+				struct xsk_buff_pool *pool, u32 nr)
+{
+	xsk_tx_invalid_descs(pool, nr);
+	/* Completion queue entries are published in submission order, so a
+	 * dropped packet must not be completed ahead of descriptors still in
+	 * flight. The ring always keeps one descriptor unused, so equal
+	 * indices mean that nothing is outstanding.
+	 */
+	if (ring->next_to_use != ring->next_to_clean) {
+		u16 last = ring->next_to_use ? ring->next_to_use - 1 : ring->count - 1;
+		struct ice_tx_buf *buf = &ring->tx_buf[last];
+
+		/* XSK_TX buffers already force the slow cleanup path. */
+		if (buf->type == ICE_TX_BUF_EMPTY) {
+			buf->xsk_completions = 1;
+			buf->type = ICE_TX_BUF_XSK_DROP;
+			ring->xsk_uso_active++;
+		} else if (buf->type == ICE_TX_BUF_XSK_DESC) {
+			buf->type = ICE_TX_BUF_XSK_DROP;
+			ring->xsk_uso_active++;
+		}
+		buf->xsk_completions += nr;
+	} else {
+		xsk_tx_completed(pool, nr);
+	}
+}
+
+/* Returns the number of packets queued on the ring. */
+static u32 ice_xmit_zc_metadata(struct ice_tx_ring *ring, struct xsk_buff_pool *pool,
+				u32 count, unsigned int *total_bytes)
+{
+	struct xdp_desc *descs = pool->tx_descs;
+	u32 i = 0, packets = 0;
+
+	/* Drop the rest of a packet whose start was dropped in an earlier batch. */
+	if (unlikely(ring->xsk_drain)) {
+		while (i < count && ring->xsk_drain)
+			ring->xsk_drain = !xsk_is_eop_desc(&descs[i++]);
+		ice_xsk_drop_packet(ring, pool, i);
+	}
+
+	while (i < count) {
+		u32 cmd = 0, offset = 0, skip = 0, slot_offset;
+		u32 len = descs[i].len, nr = 1;
+		struct ice_xsk_uso_page *page;
+		struct xsk_tx_uso_info info;
+		int uso;
+
+		while (!xsk_is_eop_desc(&descs[i + nr - 1]) && i + nr < count)
+			len += descs[i + nr++].len;
+		/* The shared-umem fallback can end a batch inside a packet. */
+		if (unlikely(!xsk_is_eop_desc(&descs[i + nr - 1]))) {
+			ice_xsk_drop_packet(ring, pool, nr);
+			ring->xsk_drain = true;
+			break;
+		}
+		page = ice_xsk_uso_slot(ring, ring->next_to_use, &slot_offset);
+		uso = xsk_tx_uso_parse(pool, &descs[i], nr, &info,
+				       page->addr + slot_offset + NET_IP_ALIGN);
+		if (uso < 0 || (!uso && !ice_xsk_bufs_fit(pool, &descs[i], nr)) ||
+		    (uso && ice_xsk_xmit_uso_hdr(ring, &descs[i], nr, &info,
+						 &cmd, &offset))) {
+			ice_xsk_drop_packet(ring, pool, nr);
+			i += nr;
+			continue;
+		}
+		if (uso) {
+			skip = info.header_len;
+			*total_bytes += (info.gso_segs - 1) * skip;
+			packets += info.gso_segs - 1;
+		}
+		ice_xsk_xmit_descs(ring, pool, &descs[i], nr, skip, cmd, offset);
+		*total_bytes += len;
+		packets++;
+		i += nr;
+	}
+	return packets;
+}
+
+/**
+ * ice_xsk_setup_metadata - register AF_XDP Tx metadata operations
+ * @netdev: net_device
+ */
+void ice_xsk_setup_metadata(struct net_device *netdev)
+{
+	static const struct xsk_tx_metadata_ops ops = {
+		.tmo_supports_uso = true,
+	};
+
+	netdev->xsk_tx_metadata_ops = &ops;
+}
+
 /**
  * ice_xmit_zc - take entries from XSK Tx ring and place them onto HW Tx ring
  * @xdp_ring: XDP ring to produce the HW Tx descriptors on
@@ -801,9 +1222,9 @@ static void ice_fill_tx_hw_ring(struct ice_tx_ring *xdp_ring,
 bool ice_xmit_zc(struct ice_tx_ring *xdp_ring, struct xsk_buff_pool *xsk_pool)
 {
 	struct xdp_desc *descs = xsk_pool->tx_descs;
-	u32 nb_pkts, nb_processed = 0;
+	u32 nb_pkts, nb_processed = 0, packets;
 	unsigned int total_bytes = 0;
-	int budget;
+	u32 budget, cost = 1;
 
 	ice_clean_xdp_irq_zc(xdp_ring, xsk_pool);
 
@@ -811,26 +1232,38 @@ bool ice_xmit_zc(struct ice_tx_ring *xdp_ring, struct xsk_buff_pool *xsk_pool)
 	    !netif_running(xdp_ring->vsi->netdev))
 		return true;
 
-	budget = ICE_DESC_UNUSED(xdp_ring);
-	budget = min_t(u16, budget, ICE_RING_QUARTER(xdp_ring));
+	/* Reserve worst-case descriptors for each potential USO packet. */
+	if (xdp_ring->xsk_uso_pages)
+		cost = ice_xsk_uso_desc_cost(xsk_pool);
+	budget = min_t(u32, ICE_DESC_UNUSED(xdp_ring) / cost,
+		       ICE_RING_QUARTER(xdp_ring));
 
 	nb_pkts = xsk_tx_peek_release_desc_batch(xsk_pool, budget);
 	if (!nb_pkts)
 		return true;
 
-	if (xdp_ring->next_to_use + nb_pkts >= xdp_ring->count) {
-		nb_processed = xdp_ring->count - xdp_ring->next_to_use;
-		ice_fill_tx_hw_ring(xdp_ring, xsk_pool, descs, nb_processed,
-				    &total_bytes);
-		xdp_ring->next_to_use = 0;
-	}
+	if (xdp_ring->xsk_uso_pages) {
+		packets = ice_xmit_zc_metadata(xdp_ring, xsk_pool, nb_pkts,
+					       &total_bytes);
+	} else {
+		if (xdp_ring->next_to_use + nb_pkts >= xdp_ring->count) {
+			nb_processed = xdp_ring->count - xdp_ring->next_to_use;
+			ice_fill_tx_hw_ring(xdp_ring, xsk_pool, descs, nb_processed,
+					    &total_bytes);
+			xdp_ring->next_to_use = 0;
+		}
 
-	ice_fill_tx_hw_ring(xdp_ring, xsk_pool, &descs[nb_processed],
-			    nb_pkts - nb_processed, &total_bytes);
+		ice_fill_tx_hw_ring(xdp_ring, xsk_pool, &descs[nb_processed],
+				    nb_pkts - nb_processed, &total_bytes);
+		packets = nb_pkts;
+	}
 
-	ice_set_rs_bit(xdp_ring);
-	ice_xdp_ring_update_tail(xdp_ring);
-	ice_update_tx_ring_stats(xdp_ring, nb_pkts, total_bytes);
+	/* Metadata packets may all have been dropped. */
+	if (packets) {
+		ice_set_rs_bit(xdp_ring);
+		ice_xdp_ring_update_tail(xdp_ring);
+		ice_update_tx_ring_stats(xdp_ring, packets, total_bytes);
+	}
 
 	if (xsk_uses_need_wakeup(xsk_pool))
 		xsk_set_tx_need_wakeup(xsk_pool);
@@ -939,12 +1372,7 @@ void ice_xsk_clean_xdp_ring(struct ice_tx_ring *xdp_ring)
 	while (ntc != ntu) {
 		struct ice_tx_buf *tx_buf = &xdp_ring->tx_buf[ntc];
 
-		if (tx_buf->type == ICE_TX_BUF_XSK_TX) {
-			tx_buf->type = ICE_TX_BUF_EMPTY;
-			xsk_buff_free(tx_buf->xdp);
-		} else {
-			xsk_frames++;
-		}
+		xsk_frames += ice_clean_xsk_buf(xdp_ring, tx_buf);
 
 		ntc++;
 		if (ntc >= xdp_ring->count)
diff --git a/drivers/net/ethernet/intel/ice/ice_xsk.h b/drivers/net/ethernet/intel/ice/ice_xsk.h
index 5275fcedc9e1..30f759cfaf9b 100644
--- a/drivers/net/ethernet/intel/ice/ice_xsk.h
+++ b/drivers/net/ethernet/intel/ice/ice_xsk.h
@@ -22,6 +22,9 @@ bool ice_xsk_any_rx_ring_ena(struct ice_vsi *vsi);
 void ice_xsk_clean_rx_ring(struct ice_rx_ring *rx_ring);
 void ice_xsk_clean_xdp_ring(struct ice_tx_ring *xdp_ring);
 bool ice_xmit_zc(struct ice_tx_ring *xdp_ring, struct xsk_buff_pool *xsk_pool);
+void ice_xsk_setup_metadata(struct net_device *netdev);
+int ice_xsk_cfg_uso_slots(struct ice_tx_ring *ring, struct xsk_buff_pool *pool);
+void ice_xsk_free_uso_slots(struct ice_tx_ring *ring);
 int ice_realloc_rx_xdp_bufs(struct ice_rx_ring *rx_ring, bool pool_present);
 void ice_qvec_cfg_msix(struct ice_vsi *vsi, struct ice_q_vector *q_vector,
 		       u16 qid);
@@ -75,6 +78,16 @@ ice_xsk_wakeup(struct net_device __always_unused *netdev,
 
 static inline void ice_xsk_clean_rx_ring(struct ice_rx_ring *rx_ring) { }
 static inline void ice_xsk_clean_xdp_ring(struct ice_tx_ring *xdp_ring) { }
+static inline void ice_xsk_setup_metadata(struct net_device *netdev) { }
+
+static inline int
+ice_xsk_cfg_uso_slots(struct ice_tx_ring __always_unused *ring,
+		      struct xsk_buff_pool __always_unused *pool)
+{
+	return 0;
+}
+
+static inline void ice_xsk_free_uso_slots(struct ice_tx_ring *ring) { }
 
 static inline int
 ice_realloc_rx_xdp_bufs(struct ice_rx_ring *rx_ring,
diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h
index 6e70b320b399..7b9f449c275a 100644
--- a/include/net/xdp_sock.h
+++ b/include/net/xdp_sock.h
@@ -113,12 +113,17 @@ struct xdp_sock {
  *     Called when AF_XDP frame requested launch time HW offload support.
  *     launch_time indicates the PTP time at which the device can schedule the
  *     packet for transmission.
+ *
+ * bool tmo_supports_uso
+ *     The driver handles USO using a kernel-owned header snapshot rather than
+ *     the optional metadata callback dispatcher.
  */
 struct xsk_tx_metadata_ops {
 	void	(*tmo_request_timestamp)(void *priv);
 	u64	(*tmo_fill_timestamp)(void *priv);
 	void	(*tmo_request_checksum)(u16 csum_start, u16 csum_offset, void *priv);
 	void	(*tmo_request_launch_time)(u64 launch_time, void *priv);
+	bool	tmo_supports_uso;
 };
 
 #ifdef CONFIG_XDP_SOCKETS
diff --git a/include/net/xdp_sock_drv.h b/include/net/xdp_sock_drv.h
index 6b5bb7b29f87..fb5ffe8cfc52 100644
--- a/include/net/xdp_sock_drv.h
+++ b/include/net/xdp_sock_drv.h
@@ -28,6 +28,7 @@ void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries);
 bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc);
 u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 max);
 void xsk_tx_release(struct xsk_buff_pool *pool);
+void xsk_tx_invalid_descs(struct xsk_buff_pool *pool, u32 nb_descs);
 struct xsk_buff_pool *xsk_get_pool_from_qid(struct net_device *dev,
 					    u16 queue_id);
 void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool);
@@ -387,6 +388,11 @@ static inline void xsk_tx_release(struct xsk_buff_pool *pool)
 {
 }
 
+static inline void xsk_tx_invalid_descs(struct xsk_buff_pool *pool,
+					u32 nb_descs)
+{
+}
+
 static inline struct xsk_buff_pool *
 xsk_get_pool_from_qid(struct net_device *dev, u16 queue_id)
 {
diff --git a/include/net/xsk_tx_uso.h b/include/net/xsk_tx_uso.h
index f23384077712..99c7ef631f59 100644
--- a/include/net/xsk_tx_uso.h
+++ b/include/net/xsk_tx_uso.h
@@ -6,6 +6,8 @@
 #include <linux/types.h>
 #include <net/xsk_buff_pool.h>
 
+#define XSK_TX_USO_MAX_HEADER 256
+
 /* Header offsets are relative to the start of the supplied Ethernet frame. */
 struct xsk_tx_uso_info {
 	struct xsk_tx_metadata metadata;
@@ -19,12 +21,30 @@ struct xsk_tx_uso_info {
 #ifdef CONFIG_XDP_SOCKETS
 int xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso,
 			u32 mtu);
+int xsk_tx_uso_parse(struct xsk_buff_pool *pool, const struct xdp_desc *descs,
+		     u32 nr, struct xsk_tx_uso_info *info, void *header);
+void xsk_tx_uso_prepare_headers(const struct xsk_tx_uso_info *info,
+				void *header);
+
 #else
+
 static inline int
 xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso, u32 mtu)
 {
 	return -EOPNOTSUPP;
 }
+
+static inline int
+xsk_tx_uso_parse(struct xsk_buff_pool *pool, const struct xdp_desc *descs,
+		 u32 nr, struct xsk_tx_uso_info *info, void *header)
+{
+	return 0;
+}
+
+static inline void
+xsk_tx_uso_prepare_headers(const struct xsk_tx_uso_info *info, void *header)
+{
+}
 #endif
 
 #endif
diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
index 0f7ba4859218..c9d99f988b33 100644
--- a/include/uapi/linux/if_xdp.h
+++ b/include/uapi/linux/if_xdp.h
@@ -134,9 +134,10 @@ struct xdp_options {
  */
 #define XDP_TXMD_FLAGS_LAUNCH_TIME		(1 << 2)
 
-/* Request UDP segmentation offload in XDP_COPY mode. Requires CHECKSUM.
+/* Request UDP segmentation offload. Requires CHECKSUM.
  * Segment size and IP header offset are in request.
- * With LAUNCH_TIME, scheduling applies to the start of the segmented packet.
+ * With LAUNCH_TIME (copy mode only), scheduling applies to the start of the
+ * segmented packet.
  */
 #define XDP_TXMD_FLAGS_USO			(1 << 3)
 
diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h
index 9026e0414fbe..bd52893adcac 100644
--- a/include/uapi/linux/netdev.h
+++ b/include/uapi/linux/netdev.h
@@ -62,11 +62,14 @@ enum netdev_xdp_rx_metadata {
  *   driver.
  * @NETDEV_XSK_FLAGS_TX_LAUNCH_TIME_FIFO: Launch time HW offload is supported
  *   by the driver.
+ * @NETDEV_XSK_FLAGS_TX_USO_ZC: UDP segmentation HW offload is supported for
+ *   AF_XDP zero-copy transmission.
  */
 enum netdev_xsk_flags {
 	NETDEV_XSK_FLAGS_TX_TIMESTAMP = 1,
 	NETDEV_XSK_FLAGS_TX_CHECKSUM = 2,
 	NETDEV_XSK_FLAGS_TX_LAUNCH_TIME_FIFO = 4,
+	NETDEV_XSK_FLAGS_TX_USO_ZC = 8,
 };
 
 enum netdev_queue_type {
diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c
index d2ccb1090496..cee2f9d391d1 100644
--- a/net/core/netdev-genl.c
+++ b/net/core/netdev-genl.c
@@ -58,6 +58,8 @@ XDP_METADATA_KFUNC_xxx
 			xsk_features |= NETDEV_XSK_FLAGS_TX_CHECKSUM;
 		if (netdev->xsk_tx_metadata_ops->tmo_request_launch_time)
 			xsk_features |= NETDEV_XSK_FLAGS_TX_LAUNCH_TIME_FIFO;
+		if (netdev->xsk_tx_metadata_ops->tmo_supports_uso)
+			xsk_features |= NETDEV_XSK_FLAGS_TX_USO_ZC;
 	}
 
 	if (nla_put_u32(rsp, NETDEV_A_DEV_IFINDEX, netdev->ifindex) ||
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index 14be62b8de5f..e830bee4d1fc 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -560,6 +560,30 @@ void xsk_tx_release(struct xsk_buff_pool *pool)
 }
 EXPORT_SYMBOL(xsk_tx_release);
 
+/**
+ * xsk_tx_invalid_descs - account Tx descriptors rejected by the driver
+ * @pool: buffer pool the descriptors were consumed from
+ * @nb_descs: number of rejected descriptors
+ *
+ * Report descriptors that were consumed from the Tx ring but rejected by the
+ * driver in the tx_invalid_descs statistic. The driver must still complete
+ * them. With a shared umem the owning socket is unknown, so the descriptors
+ * are only accounted when a single socket transmits on @pool.
+ */
+void xsk_tx_invalid_descs(struct xsk_buff_pool *pool, u32 nb_descs)
+{
+	struct xdp_sock *xs;
+
+	rcu_read_lock();
+	xs = list_first_or_null_rcu(&pool->xsk_tx_list, struct xdp_sock,
+				    tx_list);
+	if (xs && !list_next_or_null_rcu(&pool->xsk_tx_list, &xs->tx_list,
+					 struct xdp_sock, tx_list))
+		xs->tx->invalid_descs += nb_descs;
+	rcu_read_unlock();
+}
+EXPORT_SYMBOL(xsk_tx_invalid_descs);
+
 bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc)
 {
 	bool budget_exhausted = false;
diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c
index e62f3c2b3362..f20f3e84bbd6 100644
--- a/net/xdp/xsk_buff_pool.c
+++ b/net/xdp/xsk_buff_pool.c
@@ -263,7 +263,9 @@ int xp_assign_dev(struct xsk_buff_pool *pool,
 		goto err_unreg_xsk;
 	}
 	pool->umem->zc = true;
-	pool->tx_uso_supported = false;
+	if (!netdev->xsk_tx_metadata_ops ||
+	    !netdev->xsk_tx_metadata_ops->tmo_supports_uso)
+		pool->tx_uso_supported = false;
 	pool->xdp_zc_max_segs = netdev->xdp_zc_max_segs;
 	return 0;
 
diff --git a/net/xdp/xsk_tx_uso.c b/net/xdp/xsk_tx_uso.c
index a7313ade1092..1cec1ed30dd1 100644
--- a/net/xdp/xsk_tx_uso.c
+++ b/net/xdp/xsk_tx_uso.c
@@ -1,7 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0
+#include <net/gso.h>
 #include <net/ip.h>
 #include <net/ipv6.h>
 #include <net/udp.h>
+#include <net/xdp_sock_drv.h>
 #include <net/xsk_tx_uso.h>
 
 /**
@@ -75,3 +77,98 @@ int xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso,
 	return 0;
 }
 EXPORT_SYMBOL_GPL(xsk_tx_uso_validate);
+
+/**
+ * xsk_tx_uso_parse - snapshot and validate an AF_XDP USO packet
+ * @pool: pool owning the validated descriptors
+ * @descs: kernel-owned descriptors, starting at the beginning of a packet
+ * @nr: nonzero number of descriptors, including the final EOP descriptor
+ * @info: receives the snapshotted metadata and the parsed header layout
+ * @header: storage for at least XSK_TX_USO_MAX_HEADER bytes of packet header
+ *
+ * Metadata is snapshotted before using its offsets. The packet header is
+ * copied exactly once, before any parsing. @header must remain valid until
+ * DMA completes if this packet is transmitted.
+ *
+ * Return: 1 for a valid USO request, 0 without USO, or a negative error code.
+ */
+int xsk_tx_uso_parse(struct xsk_buff_pool *pool, const struct xdp_desc *descs,
+		     u32 nr, struct xsk_tx_uso_info *info, void *header)
+{
+	struct xsk_tx_metadata *meta = &info->metadata;
+	struct xsk_tx_metadata *user_meta;
+	u32 header_len, len = 0, i;
+	u64 flags;
+	int err;
+
+	user_meta = xsk_buff_get_metadata(pool, descs->addr, descs->options);
+	if (!user_meta)
+		return 0;
+	memcpy(meta, user_meta,
+	       min_t(size_t, pool->tx_metadata_len, sizeof(*meta)));
+	if (!(meta->flags & XDP_TXMD_FLAGS_USO))
+		return 0;
+	if (!xsk_buff_valid_tx_metadata(pool, meta, &flags) ||
+	    !xsk_tx_uso_flags_valid(flags) ||
+	    (flags & XDP_TXMD_FLAGS_LAUNCH_TIME))
+		return -EOPNOTSUPP;
+
+	if (!meta->request.gso_size || meta->request.gso_size == GSO_BY_FRAGS ||
+	    meta->request.network_offset < ETH_HLEN ||
+	    meta->request.csum_start < meta->request.network_offset ||
+	    meta->request.csum_offset != offsetof(struct udphdr, check))
+		return -EINVAL;
+	header_len = (u32)meta->request.csum_start + sizeof(struct udphdr);
+	if (header_len > XSK_TX_USO_MAX_HEADER || header_len > descs->len)
+		return -EINVAL;
+	for (i = 0; i < nr; i++) {
+		if (check_add_overflow(len, descs[i].len, &len) ||
+		    xsk_is_eop_desc(&descs[i]) != (i == nr - 1))
+			return -EINVAL;
+	}
+
+	info->header_len = header_len;
+	info->len = len;
+	memcpy(header, xsk_buff_raw_get_data(pool, descs->addr), header_len);
+	err = xsk_tx_uso_validate(header, info, READ_ONCE(pool->netdev->mtu));
+	if (err)
+		return err;
+	/* As in copy mode, segmentation follows the device feature. */
+	if (info->gso_segs > 1 &&
+	    !(READ_ONCE(pool->netdev->features) & NETIF_F_GSO_UDP_L4))
+		return -EOPNOTSUPP;
+	return 1;
+}
+EXPORT_SYMBOL_GPL(xsk_tx_uso_parse);
+
+/**
+ * xsk_tx_uso_prepare_headers - prepare the copied header for hardware USO
+ * @info: layout of a request validated by xsk_tx_uso_parse()
+ * @header: header copied by xsk_tx_uso_parse()
+ *
+ * Call once after validation and before syncing the header for DMA. Removes
+ * the aggregate length from the partial UDP checksum seed and clears IP
+ * length/checksum fields for the segmentation engine. Single-datagram headers
+ * are unchanged.
+ */
+void xsk_tx_uso_prepare_headers(const struct xsk_tx_uso_info *info,
+				void *header)
+{
+	const struct xsk_tx_metadata *meta = &info->metadata;
+	struct udphdr *uh = header + meta->request.csum_start;
+	struct iphdr *iph = header + meta->request.network_offset;
+
+	if (info->gso_segs == 1)
+		return;
+
+	if (iph->version == 4) {
+		iph->tot_len = 0;
+		iph->check = 0;
+	} else {
+		((struct ipv6hdr *)iph)->payload_len = 0;
+	}
+	csum_replace_by_diff(&uh->check,
+			     (__force __wsum)htonl(info->len -
+						   meta->request.csum_start));
+}
+EXPORT_SYMBOL_GPL(xsk_tx_uso_prepare_headers);
diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
index 0f7ba4859218..c9d99f988b33 100644
--- a/tools/include/uapi/linux/if_xdp.h
+++ b/tools/include/uapi/linux/if_xdp.h
@@ -134,9 +134,10 @@ struct xdp_options {
  */
 #define XDP_TXMD_FLAGS_LAUNCH_TIME		(1 << 2)
 
-/* Request UDP segmentation offload in XDP_COPY mode. Requires CHECKSUM.
+/* Request UDP segmentation offload. Requires CHECKSUM.
  * Segment size and IP header offset are in request.
- * With LAUNCH_TIME, scheduling applies to the start of the segmented packet.
+ * With LAUNCH_TIME (copy mode only), scheduling applies to the start of the
+ * segmented packet.
  */
 #define XDP_TXMD_FLAGS_USO			(1 << 3)
 
diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/netdev.h
index 9026e0414fbe..bd52893adcac 100644
--- a/tools/include/uapi/linux/netdev.h
+++ b/tools/include/uapi/linux/netdev.h
@@ -62,11 +62,14 @@ enum netdev_xdp_rx_metadata {
  *   driver.
  * @NETDEV_XSK_FLAGS_TX_LAUNCH_TIME_FIFO: Launch time HW offload is supported
  *   by the driver.
+ * @NETDEV_XSK_FLAGS_TX_USO_ZC: UDP segmentation HW offload is supported for
+ *   AF_XDP zero-copy transmission.
  */
 enum netdev_xsk_flags {
 	NETDEV_XSK_FLAGS_TX_TIMESTAMP = 1,
 	NETDEV_XSK_FLAGS_TX_CHECKSUM = 2,
 	NETDEV_XSK_FLAGS_TX_LAUNCH_TIME_FIFO = 4,
+	NETDEV_XSK_FLAGS_TX_USO_ZC = 8,
 };
 
 enum netdev_queue_type {
-- 
2.38.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH net-next 4/5] xsk: add KUnit tests for AF_XDP USO header helpers
  2026-11-19 16:53 [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Magnus Karlsson
                   ` (2 preceding siblings ...)
  2026-11-19 16:53 ` [PATCH net-next 3/5] xsk: ice: add USO zero-copy support Magnus Karlsson
@ 2026-11-19 16:53 ` Magnus Karlsson
  2026-11-19 16:53 ` [PATCH net-next 5/5] selftests: net: add AF_XDP USO test Magnus Karlsson
  4 siblings, 0 replies; 8+ messages in thread
From: Magnus Karlsson @ 2026-11-19 16:53 UTC (permalink / raw)
  To: netdev, magnus.karlsson, magnus.karlsson, maciej.fijalkowski, sdf,
	davem, edumazet, kuba, pabeni, horms, bjorn, kerneljasonxing
  Cc: intel-wired-lan, anthony.l.nguyen, przemyslaw.kitszel,
	jacob.e.keller

Add KUnit tests for the helpers that AF_XDP uses to validate and prepare
USO requests, as these run on headers supplied by user space and are
shared by the copy and zero-copy paths:

- xsk_tx_uso_validate(): IPv4 and IPv6, link-layer headers of different
  lengths, IPv4 options, malformed IP and UDP headers, an IPv6
  destination options header, a non-UDP IPv6 next header, a UDP header
  that does not directly follow the IPv6 header, bad offsets and
  lengths, the MTU and the segment count limit.
  The link layer is not parsed, so the IP header is found at
  network_offset whatever the EtherType says.
- xsk_tx_uso_parse(): the header and metadata are snapshotted, so later
  changes to the UMEM do not reach the copy; invalid metadata, offsets
  and descriptor layouts are rejected; flag combinations that USO does
  not support are rejected; segmentation needs NETIF_F_GSO_UDP_L4 while
  a single datagram does not; requests without USO are left alone.
- xsk_tx_uso_prepare_headers(): the checksum seeds are compared with
  pseudo-header checksums computed independently with
  csum_tcpudp_magic() and csum_ipv6_magic(), and a single datagram
  keeps its headers.
- xsk_buff_valid_tx_metadata() and xsk_tx_uso_flags_valid() for USO.

The tests are in net/xdp/tests/xsk_tx_uso_kunit.c, following the KUnit
naming convention. Run with:

  ./tools/testing/kunit/kunit.py run --kunitconfig=net/xdp

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 net/xdp/.kunitconfig             |   6 +
 net/xdp/Kconfig                  |  12 +
 net/xdp/Makefile                 |   1 +
 net/xdp/tests/xsk_tx_uso_kunit.c | 620 +++++++++++++++++++++++++++++++
 4 files changed, 639 insertions(+)
 create mode 100644 net/xdp/.kunitconfig
 create mode 100644 net/xdp/tests/xsk_tx_uso_kunit.c

diff --git a/net/xdp/.kunitconfig b/net/xdp/.kunitconfig
new file mode 100644
index 000000000000..d88b8e3643f1
--- /dev/null
+++ b/net/xdp/.kunitconfig
@@ -0,0 +1,6 @@
+CONFIG_KUNIT=y
+CONFIG_NET=y
+CONFIG_BPF=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_XDP_SOCKETS=y
+CONFIG_XDP_SOCKETS_USO_KUNIT_TEST=y
diff --git a/net/xdp/Kconfig b/net/xdp/Kconfig
index 71af2febe72a..f230b9049ef0 100644
--- a/net/xdp/Kconfig
+++ b/net/xdp/Kconfig
@@ -14,3 +14,15 @@ config XDP_SOCKETS_DIAG
 	help
 	  Support for PF_XDP sockets monitoring interface used by the ss tool.
 	  If unsure, say Y.
+
+config XDP_SOCKETS_USO_KUNIT_TEST
+	tristate "AF_XDP USO header validation tests" if !KUNIT_ALL_TESTS
+	depends on XDP_SOCKETS && KUNIT
+	default KUNIT_ALL_TESTS
+	help
+	  KUnit tests for the helpers that validate and prepare AF_XDP UDP
+	  segmentation offload (USO) requests in copy and zero-copy mode:
+	  header validation, metadata snapshots, segment limits and the
+	  preparation of headers and checksum seeds for segmentation.
+
+	  If unsure, say N.
diff --git a/net/xdp/Makefile b/net/xdp/Makefile
index b6109e0e5547..aef2e70f7a6f 100644
--- a/net/xdp/Makefile
+++ b/net/xdp/Makefile
@@ -3,3 +3,4 @@ obj-$(CONFIG_XDP_SOCKETS) += xsk.o xdp_umem.o xsk_queue.o xskmap.o
 obj-$(CONFIG_XDP_SOCKETS) += xsk_buff_pool.o
 obj-$(CONFIG_XDP_SOCKETS) += xsk_tx_uso.o
 obj-$(CONFIG_XDP_SOCKETS_DIAG) += xsk_diag.o
+obj-$(CONFIG_XDP_SOCKETS_USO_KUNIT_TEST) += tests/xsk_tx_uso_kunit.o
diff --git a/net/xdp/tests/xsk_tx_uso_kunit.c b/net/xdp/tests/xsk_tx_uso_kunit.c
new file mode 100644
index 000000000000..53d763aa53dc
--- /dev/null
+++ b/net/xdp/tests/xsk_tx_uso_kunit.c
@@ -0,0 +1,620 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <kunit/test.h>
+#include <linux/if_vlan.h>
+#include <net/ip.h>
+#include <net/ip6_checksum.h>
+#include <net/udp.h>
+#include <net/xdp_sock_drv.h>
+#include <net/xsk_tx_uso.h>
+
+#define USO_META_LEN	sizeof(struct xsk_tx_metadata)
+
+static const __be32 uso_saddr = cpu_to_be32(0xc0a80001);
+static const __be32 uso_daddr = cpu_to_be32(0xc0a80002);
+
+/* Build an Ethernet frame with @l2_len bytes of link layer, one IP header and
+ * a UDP header carrying @payload bytes, and describe it in @uso.
+ */
+static void uso_packet(void *header, struct xsk_tx_uso_info *uso,
+		       bool ipv6, u32 l2_len, u32 payload)
+{
+	u32 iplen = ipv6 ? sizeof(struct ipv6hdr) : sizeof(struct iphdr);
+	struct xsk_tx_metadata *meta = &uso->metadata;
+	struct ethhdr *eth = header;
+	struct udphdr *uh;
+
+	memset(header, 0, XSK_TX_USO_MAX_HEADER);
+	memset(uso, 0, sizeof(*uso));
+	meta->request.network_offset = l2_len;
+	meta->request.csum_start = l2_len + iplen;
+	meta->request.csum_offset = offsetof(struct udphdr, check);
+	meta->request.gso_size = 1400;
+	uso->header_len = meta->request.csum_start + sizeof(*uh);
+	uso->len = uso->header_len + payload;
+	eth->h_proto = htons(ipv6 ? ETH_P_IPV6 : ETH_P_IP);
+	if (ipv6) {
+		struct ipv6hdr *ip6h = header + l2_len;
+
+		ip6h->version = 6;
+		ip6h->nexthdr = IPPROTO_UDP;
+		ip6h->payload_len = htons(sizeof(*uh) + payload);
+		ip6h->saddr.s6_addr32[0] = uso_saddr;
+		ip6h->daddr.s6_addr32[0] = uso_daddr;
+	} else {
+		struct iphdr *iph = header + l2_len;
+
+		iph->version = 4;
+		iph->ihl = 5;
+		iph->protocol = IPPROTO_UDP;
+		iph->tot_len = htons(iplen + sizeof(*uh) + payload);
+		iph->saddr = uso_saddr;
+		iph->daddr = uso_daddr;
+	}
+	uh = header + meta->request.csum_start;
+	uh->len = htons(sizeof(*uh) + payload);
+}
+
+/* The header copy, at the alignment that drivers give it. Kept off the stack,
+ * as KASAN builds pad every on-stack buffer.
+ */
+static void *uso_header(struct kunit *test)
+{
+	u8 *storage = kunit_kzalloc(test, NET_IP_ALIGN + XSK_TX_USO_MAX_HEADER,
+				    GFP_KERNEL);
+
+	KUNIT_ASSERT_NOT_NULL(test, storage);
+	return storage + NET_IP_ALIGN;
+}
+
+/* Partial pseudo-header checksum over a UDP length, as an application seeds it. */
+static __sum16 uso_pseudo_csum(void *header, const struct xsk_tx_uso_info *uso,
+			       bool ipv6, u32 udp_len)
+{
+	void *ip = header + uso->metadata.request.network_offset;
+
+	if (ipv6) {
+		struct ipv6hdr *ip6h = ip;
+
+		return ~csum_ipv6_magic(&ip6h->saddr, &ip6h->daddr, udp_len,
+					IPPROTO_UDP, 0);
+	}
+	return ~csum_tcpudp_magic(((struct iphdr *)ip)->saddr,
+				  ((struct iphdr *)ip)->daddr, udp_len,
+				  IPPROTO_UDP, 0);
+}
+
+static void uso_valid(struct kunit *test)
+{
+	static const u32 l2_lens[] = { ETH_HLEN, ETH_HLEN + VLAN_HLEN,
+				       ETH_HLEN + 8 };
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info uso;
+	int ipv6, i;
+
+	for (ipv6 = 0; ipv6 < 2; ipv6++)
+		for (i = 0; i < ARRAY_SIZE(l2_lens); i++) {
+			uso_packet(header, &uso, ipv6, l2_lens[i], 4000);
+			KUNIT_ASSERT_EQ_MSG(test,
+					    xsk_tx_uso_validate(header, &uso, 1500),
+					    0, "ipv6 %d l2 %u", ipv6, l2_lens[i]);
+			KUNIT_EXPECT_EQ(test, uso.gso_segs, 3);
+			KUNIT_EXPECT_EQ(test, ntohs(uso.protocol),
+					ipv6 ? ETH_P_IPV6 : ETH_P_IP);
+			KUNIT_EXPECT_EQ(test, uso.l4_protocol, IPPROTO_UDP);
+			uso_packet(header, &uso, ipv6, l2_lens[i], 0);
+			KUNIT_ASSERT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500),
+					0);
+			KUNIT_EXPECT_EQ(test, uso.gso_segs, 1);
+		}
+}
+
+enum uso_invalid_kind {
+	USO_INVALID_IPV6_VERSION,
+	USO_INVALID_IHL,
+	USO_INVALID_TOT_LEN,
+	USO_INVALID_FRAGMENT,
+	USO_INVALID_PROTOCOL,
+	USO_INVALID_UDP_LEN,
+	USO_INVALID_NETWORK_OFFSET,
+	USO_INVALID_NO_IP_HEADER,
+	USO_INVALID_LEN,
+	/* IPv6 cases from here on. */
+	USO_INVALID_IPV6_NEXTHDR,
+	USO_INVALID_IPV6_EXT_HEADER,
+	USO_INVALID_IPV6_UDP_OFFSET,
+};
+
+struct uso_invalid_case {
+	const char *name;
+	enum uso_invalid_kind kind;
+};
+
+static const struct uso_invalid_case uso_invalid_cases[] = {
+	{ "ipv4_header_with_ipv6_version", USO_INVALID_IPV6_VERSION },
+	{ "short_ihl", USO_INVALID_IHL },
+	{ "short_tot_len", USO_INVALID_TOT_LEN },
+	{ "fragment", USO_INVALID_FRAGMENT },
+	{ "not_udp", USO_INVALID_PROTOCOL },
+	{ "short_udp_len", USO_INVALID_UDP_LEN },
+	{ "short_network_offset", USO_INVALID_NETWORK_OFFSET },
+	{ "no_ip_header", USO_INVALID_NO_IP_HEADER },
+	{ "len_below_header", USO_INVALID_LEN },
+	{ "ipv6_non_udp_nexthdr", USO_INVALID_IPV6_NEXTHDR },
+	{ "ipv6_extension_header", USO_INVALID_IPV6_EXT_HEADER },
+	{ "ipv6_udp_offset_past_header", USO_INVALID_IPV6_UDP_OFFSET },
+};
+
+KUNIT_ARRAY_PARAM_DESC(uso_invalid, uso_invalid_cases, name);
+
+static void uso_invalid(struct kunit *test)
+{
+	const struct uso_invalid_case *param = test->param_value;
+	bool ipv6 = param->kind >= USO_INVALID_IPV6_NEXTHDR;
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info uso;
+	struct iphdr *iph;
+	struct udphdr *uh;
+
+	uso_packet(header, &uso, ipv6, ETH_HLEN, 4000);
+	iph = header + uso.metadata.request.network_offset;
+	uh = header + uso.metadata.request.csum_start;
+	switch (param->kind) {
+	case USO_INVALID_IPV6_VERSION:
+		/* An IPv6 version needs a 40-byte network header. */
+		iph->version = 6;
+		break;
+	case USO_INVALID_IHL:
+		iph->ihl = 4;
+		break;
+	case USO_INVALID_TOT_LEN:
+		iph->tot_len = htons(40);
+		break;
+	case USO_INVALID_FRAGMENT:
+		iph->frag_off = htons(IP_MF);
+		break;
+	case USO_INVALID_PROTOCOL:
+		iph->protocol = IPPROTO_TCP;
+		break;
+	case USO_INVALID_UDP_LEN:
+		uh->len = htons(8);
+		break;
+	case USO_INVALID_NETWORK_OFFSET:
+		uso.metadata.request.network_offset = ETH_HLEN - 1;
+		break;
+	case USO_INVALID_NO_IP_HEADER:
+		uso.metadata.request.network_offset =
+			uso.metadata.request.csum_start;
+		break;
+	case USO_INVALID_LEN:
+		uso.len = uso.header_len - 1;
+		break;
+	case USO_INVALID_IPV6_NEXTHDR:
+		((struct ipv6hdr *)iph)->nexthdr = IPPROTO_DSTOPTS;
+		break;
+	case USO_INVALID_IPV6_EXT_HEADER: {
+		/* An otherwise valid 8-byte destination options header between
+		 * the IPv6 and UDP headers, with csum_start past it.
+		 */
+		struct ipv6hdr *ip6h = (struct ipv6hdr *)iph;
+		u8 *opts = (u8 *)uh;
+
+		memmove(opts + 8, uh, sizeof(*uh));
+		memset(opts, 0, 8);
+		opts[0] = IPPROTO_UDP;		/* next header */
+		opts[2] = IPV6_TLV_PADN;	/* pad the remaining 6 bytes */
+		opts[3] = 4;
+		ip6h->nexthdr = IPPROTO_DSTOPTS;
+		be16_add_cpu(&ip6h->payload_len, 8);
+		uso.metadata.request.csum_start += 8;
+		uso.header_len += 8;
+		uso.len += 8;
+		break;
+	}
+	case USO_INVALID_IPV6_UDP_OFFSET:
+		/* The same layout, but with nexthdr naming UDP, so that only
+		 * the IPv6 header length check can reject it.
+		 */
+		memmove((u8 *)uh + 8, uh, sizeof(*uh));
+		memset(uh, 0, 8);
+		be16_add_cpu(&((struct ipv6hdr *)iph)->payload_len, 8);
+		uso.metadata.request.csum_start += 8;
+		uso.header_len += 8;
+		uso.len += 8;
+		break;
+	}
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), -EINVAL);
+}
+
+static void uso_limits(struct kunit *test)
+{
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info uso;
+
+	uso_packet(header, &uso, false, ETH_HLEN, 8192);
+	uso.metadata.request.gso_size = 64;
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), 0);
+	KUNIT_EXPECT_EQ(test, uso.gso_segs, UDP_MAX_SEGMENTS);
+	uso_packet(header, &uso, false, ETH_HLEN, 8193);
+	uso.metadata.request.gso_size = 64;
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), -EINVAL);
+	uso_packet(header, &uso, false, ETH_HLEN, 4000);
+	uso.metadata.request.gso_size = 1472;
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), 0);
+	uso.metadata.request.gso_size++;
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), -EINVAL);
+	/* A single datagram must fit the MTU as well. */
+	uso_packet(header, &uso, false, ETH_HLEN, 1473);
+	uso.metadata.request.gso_size = 4000;
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), -EINVAL);
+}
+
+static void uso_header_layout(struct kunit *test)
+{
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info uso;
+	struct udphdr *uh;
+	struct iphdr *iph;
+
+	/* IPv4 options are accepted when csum_start points past them. */
+	uso_packet(header, &uso, false, ETH_HLEN, 4000);
+	iph = header + uso.metadata.request.network_offset;
+	memmove(header + uso.metadata.request.csum_start + 4,
+		header + uso.metadata.request.csum_start, sizeof(*uh));
+	memset(header + uso.metadata.request.csum_start, 0, 4);
+	iph->ihl++;
+	iph->tot_len = htons(ntohs(iph->tot_len) + 4);
+	uso.metadata.request.csum_start += 4;
+	uso.header_len += 4;
+	uso.len += 4;
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), 0);
+
+	/* The link layer is not parsed: the IP header is found at
+	 * network_offset whatever the EtherType says. Copy mode checks the
+	 * EtherType separately.
+	 */
+	uso_packet(header, &uso, false, ETH_HLEN + VLAN_HLEN, 4000);
+	((struct ethhdr *)header)->h_proto = htons(ETH_P_ARP);
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), 0);
+	KUNIT_EXPECT_EQ(test, ntohs(uso.protocol), ETH_P_IP);
+
+	/* The IP version selects the protocol. */
+	uso_packet(header, &uso, false, ETH_HLEN, 4000);
+	((struct iphdr *)(header + ETH_HLEN))->version = 5;
+	KUNIT_EXPECT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500),
+			-EPROTONOSUPPORT);
+}
+
+static void uso_prepare_headers(struct kunit *test)
+{
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info uso;
+	int ipv6;
+
+	for (ipv6 = 0; ipv6 < 2; ipv6++) {
+		void *ip = header + ETH_HLEN;
+		struct udphdr *uh;
+		u32 udp_len;
+
+		uso_packet(header, &uso, ipv6, ETH_HLEN, 4000);
+		/* Start from a valid IPv4 checksum, so that clearing it shows. */
+		if (!ipv6) {
+			struct iphdr *iph = ip;
+
+			iph->check = ip_fast_csum(iph, iph->ihl);
+			KUNIT_ASSERT_NE(test, (__force u16)iph->check, 0);
+		}
+		KUNIT_ASSERT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), 0);
+		uh = header + uso.metadata.request.csum_start;
+		udp_len = uso.len - uso.metadata.request.csum_start;
+		uh->check = uso_pseudo_csum(header, &uso, ipv6, udp_len);
+
+		/* The seed must cover the pseudo header without the length. */
+		xsk_tx_uso_prepare_headers(&uso, header);
+		KUNIT_EXPECT_EQ_MSG(test, (__force u16)uh->check,
+				    (__force u16)uso_pseudo_csum(header, &uso, ipv6, 0),
+				    "ipv6 %d", ipv6);
+		if (ipv6) {
+			KUNIT_EXPECT_EQ(test,
+					ntohs(((struct ipv6hdr *)ip)->payload_len), 0);
+		} else {
+			KUNIT_EXPECT_EQ(test, ntohs(((struct iphdr *)ip)->tot_len), 0);
+			KUNIT_EXPECT_EQ(test,
+					(__force u16)((struct iphdr *)ip)->check, 0);
+		}
+	}
+}
+
+/* A single datagram uses ordinary checksum offload, so its headers stay. */
+static void uso_prepare_single(struct kunit *test)
+{
+	u8 *orig = kunit_kzalloc(test, XSK_TX_USO_MAX_HEADER, GFP_KERNEL);
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info uso;
+	struct udphdr *uh;
+
+	KUNIT_ASSERT_NOT_NULL(test, orig);
+	uso_packet(header, &uso, false, ETH_HLEN, 1000);
+	KUNIT_ASSERT_EQ(test, xsk_tx_uso_validate(header, &uso, 1500), 0);
+	KUNIT_ASSERT_EQ(test, uso.gso_segs, 1);
+	uh = header + uso.metadata.request.csum_start;
+	uh->check = uso_pseudo_csum(header, &uso, false, ntohs(uh->len));
+	memcpy(orig, header, uso.header_len);
+	xsk_tx_uso_prepare_headers(&uso, header);
+	KUNIT_EXPECT_MEMEQ(test, header, orig, uso.header_len);
+}
+
+static void uso_metadata_flags(struct kunit *test)
+{
+	struct xsk_tx_metadata meta = {
+		.flags = XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM,
+	};
+	struct xsk_buff_pool *pool;
+	u64 flags;
+
+	pool = kunit_kzalloc(test, sizeof(*pool), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, pool);
+	pool->tx_metadata_len = USO_META_LEN;
+	pool->tx_uso_supported = true;
+
+	KUNIT_ASSERT_TRUE(test, xsk_buff_valid_tx_metadata(pool, &meta, &flags));
+	KUNIT_EXPECT_TRUE(test, xsk_tx_uso_flags_valid(flags));
+
+	/* USO needs checksum offload and cannot report a Tx timestamp. */
+	meta.flags = XDP_TXMD_FLAGS_USO;
+	KUNIT_ASSERT_TRUE(test, xsk_buff_valid_tx_metadata(pool, &meta, &flags));
+	KUNIT_EXPECT_FALSE(test, xsk_tx_uso_flags_valid(flags));
+	meta.flags = XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM |
+		     XDP_TXMD_FLAGS_TIMESTAMP;
+	KUNIT_ASSERT_TRUE(test, xsk_buff_valid_tx_metadata(pool, &meta, &flags));
+	KUNIT_EXPECT_FALSE(test, xsk_tx_uso_flags_valid(flags));
+
+	/* Launch time is allowed here; zero-copy rejects it when parsing. */
+	meta.flags = XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM |
+		     XDP_TXMD_FLAGS_LAUNCH_TIME;
+	KUNIT_ASSERT_TRUE(test, xsk_buff_valid_tx_metadata(pool, &meta, &flags));
+	KUNIT_EXPECT_TRUE(test, xsk_tx_uso_flags_valid(flags));
+
+	/* USO requests are refused on pools that cannot carry them. */
+	meta.flags = XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM;
+	pool->tx_uso_supported = false;
+	KUNIT_EXPECT_FALSE(test, xsk_buff_valid_tx_metadata(pool, &meta, &flags));
+	pool->tx_uso_supported = true;
+	pool->tx_metadata_len =
+		offsetofend(struct xsk_tx_metadata, request.network_offset) - 1;
+	KUNIT_EXPECT_FALSE(test, xsk_buff_valid_tx_metadata(pool, &meta, &flags));
+	pool->tx_metadata_len = USO_META_LEN;
+	meta.flags |= BIT_ULL(63);
+	KUNIT_EXPECT_FALSE(test, xsk_buff_valid_tx_metadata(pool, &meta, &flags));
+}
+
+struct uso_parse_ctx {
+	struct xsk_buff_pool *pool;
+	struct xdp_desc descs[2];
+	u32 nr;
+	struct xsk_tx_metadata meta;
+	struct xsk_tx_uso_info orig;
+	void *orig_header;
+	void *umem_header;
+};
+
+/* Place a packet with @payload bytes in a UMEM, split over two descriptors
+ * when it does not fit in the first 2048 bytes.
+ */
+static void uso_parse_init(struct kunit *test, struct uso_parse_ctx *ctx,
+			   u32 payload)
+{
+	struct xsk_buff_pool *pool;
+
+	memset(ctx, 0, sizeof(*ctx));
+	pool = kunit_kzalloc(test, sizeof(*pool), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, pool);
+	pool->netdev = kunit_kzalloc(test, sizeof(*pool->netdev), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, pool->netdev);
+	pool->netdev->mtu = 1500;
+	pool->netdev->features = NETIF_F_GSO_UDP_L4;
+	pool->tx_metadata_len = USO_META_LEN;
+	pool->tx_uso_supported = true;
+	pool->addrs = kunit_kzalloc(test, 8192, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, pool->addrs);
+	ctx->pool = pool;
+	ctx->orig_header = uso_header(test);
+
+	uso_packet(ctx->orig_header, &ctx->orig, false, ETH_HLEN, payload);
+	ctx->nr = ctx->orig.len > 2048 ? 2 : 1;
+	ctx->descs[0] = (struct xdp_desc) {
+		.addr = 128, .len = min(ctx->orig.len, 2048U),
+		.options = XDP_TX_METADATA | (ctx->nr > 1 ? XDP_PKT_CONTD : 0),
+	};
+	ctx->descs[1] = (struct xdp_desc) {
+		.addr = 4096, .len = ctx->orig.len - ctx->descs[0].len,
+	};
+	ctx->meta = ctx->orig.metadata;
+	ctx->meta.flags = XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM;
+	ctx->umem_header = pool->addrs + ctx->descs[0].addr;
+	memcpy(ctx->umem_header, ctx->orig_header, ctx->orig.header_len);
+}
+
+static struct uso_parse_ctx *uso_parse_ctx(struct kunit *test)
+{
+	struct uso_parse_ctx *ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+
+	KUNIT_ASSERT_NOT_NULL(test, ctx);
+	return ctx;
+}
+
+static int uso_parse(struct uso_parse_ctx *ctx, struct xsk_tx_uso_info *info,
+		     void *header)
+{
+	memcpy(ctx->umem_header - USO_META_LEN, &ctx->meta, USO_META_LEN);
+	return xsk_tx_uso_parse(ctx->pool, ctx->descs, ctx->nr, info, header);
+}
+
+static void uso_snapshot(struct kunit *test)
+{
+	struct uso_parse_ctx *ctx = uso_parse_ctx(test);
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info info;
+	struct udphdr *uh;
+
+	uso_parse_init(test, ctx, 4000);
+	KUNIT_ASSERT_EQ(test, uso_parse(ctx, &info, header), 1);
+	KUNIT_EXPECT_EQ(test, info.len, ctx->orig.len);
+	KUNIT_EXPECT_EQ(test, info.header_len, ctx->orig.header_len);
+	KUNIT_EXPECT_EQ(test, info.gso_segs, 3);
+
+	/* Changes to the UMEM after parsing do not reach the copy. */
+	memset(ctx->umem_header - USO_META_LEN, 0xff,
+	       USO_META_LEN + ctx->orig.header_len);
+	KUNIT_EXPECT_MEMEQ(test, header, ctx->orig_header, ctx->orig.header_len);
+	KUNIT_EXPECT_EQ(test, info.metadata.request.gso_size, 1400);
+
+	/* Preparing the copy leaves the UMEM alone. */
+	xsk_tx_uso_prepare_headers(&info, header);
+	uh = header + info.metadata.request.csum_start;
+	KUNIT_EXPECT_EQ(test, ntohs(((struct iphdr *)(header + ETH_HLEN))->tot_len),
+			0);
+	KUNIT_EXPECT_EQ(test, ntohs(uh->len), 4008);
+	KUNIT_EXPECT_EQ(test, *(u8 *)ctx->umem_header, 0xff);
+}
+
+enum uso_parse_kind {
+	USO_PARSE_NO_GSO_SIZE,
+	USO_PARSE_GSO_BY_FRAGS,
+	USO_PARSE_NO_NETWORK_OFFSET,
+	USO_PARSE_CSUM_START,
+	USO_PARSE_CSUM_OFFSET,
+	USO_PARSE_SPLIT_HEADER,
+	USO_PARSE_EARLY_EOP,
+	USO_PARSE_NO_CSUM,
+	USO_PARSE_TIMESTAMP,
+	USO_PARSE_LAUNCH_TIME,
+	USO_PARSE_BAD_HEADER,
+};
+
+struct uso_parse_case {
+	const char *name;
+	enum uso_parse_kind kind;
+	int err;
+};
+
+static const struct uso_parse_case uso_parse_cases[] = {
+	{ "no_gso_size", USO_PARSE_NO_GSO_SIZE, -EINVAL },
+	{ "gso_by_frags", USO_PARSE_GSO_BY_FRAGS, -EINVAL },
+	{ "no_network_offset", USO_PARSE_NO_NETWORK_OFFSET, -EINVAL },
+	{ "csum_start_past_max_header", USO_PARSE_CSUM_START, -EINVAL },
+	{ "bad_csum_offset", USO_PARSE_CSUM_OFFSET, -EINVAL },
+	{ "header_split_over_buffers", USO_PARSE_SPLIT_HEADER, -EINVAL },
+	{ "early_end_of_packet", USO_PARSE_EARLY_EOP, -EINVAL },
+	{ "no_checksum_flag", USO_PARSE_NO_CSUM, -EOPNOTSUPP },
+	{ "timestamp_flag", USO_PARSE_TIMESTAMP, -EOPNOTSUPP },
+	{ "launch_time_flag", USO_PARSE_LAUNCH_TIME, -EOPNOTSUPP },
+	{ "invalid_header", USO_PARSE_BAD_HEADER, -EINVAL },
+};
+
+KUNIT_ARRAY_PARAM_DESC(uso_parse_invalid, uso_parse_cases, name);
+
+static void uso_parse_invalid(struct kunit *test)
+{
+	const struct uso_parse_case *param = test->param_value;
+	struct uso_parse_ctx *ctx = uso_parse_ctx(test);
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info info;
+
+	uso_parse_init(test, ctx, 4000);
+	switch (param->kind) {
+	case USO_PARSE_NO_GSO_SIZE:
+		ctx->meta.request.gso_size = 0;
+		break;
+	case USO_PARSE_GSO_BY_FRAGS:
+		ctx->meta.request.gso_size = GSO_BY_FRAGS;
+		break;
+	case USO_PARSE_NO_NETWORK_OFFSET:
+		ctx->meta.request.network_offset = 0;
+		break;
+	case USO_PARSE_CSUM_START:
+		ctx->meta.request.csum_start = XSK_TX_USO_MAX_HEADER;
+		break;
+	case USO_PARSE_CSUM_OFFSET:
+		ctx->meta.request.csum_offset = 0;
+		break;
+	case USO_PARSE_SPLIT_HEADER:
+		/* All headers through UDP must be in the first buffer. */
+		ctx->descs[0].len = ctx->orig.header_len - 1;
+		ctx->descs[1].len = ctx->orig.len - ctx->descs[0].len;
+		break;
+	case USO_PARSE_EARLY_EOP:
+		/* Only the last descriptor may end the packet. */
+		ctx->descs[0].options &= ~XDP_PKT_CONTD;
+		break;
+	case USO_PARSE_NO_CSUM:
+		ctx->meta.flags = XDP_TXMD_FLAGS_USO;
+		break;
+	case USO_PARSE_TIMESTAMP:
+		ctx->meta.flags |= XDP_TXMD_FLAGS_TIMESTAMP;
+		break;
+	case USO_PARSE_LAUNCH_TIME:
+		ctx->meta.flags |= XDP_TXMD_FLAGS_LAUNCH_TIME;
+		break;
+	case USO_PARSE_BAD_HEADER:
+		((struct iphdr *)(ctx->umem_header + ETH_HLEN))->protocol =
+			IPPROTO_TCP;
+		break;
+	}
+	KUNIT_EXPECT_EQ(test, uso_parse(ctx, &info, header), param->err);
+}
+
+static void uso_parse_feature(struct kunit *test)
+{
+	struct uso_parse_ctx *ctx = uso_parse_ctx(test);
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info info;
+
+	/* Segmentation follows NETIF_F_GSO_UDP_L4. */
+	uso_parse_init(test, ctx, 4000);
+	ctx->pool->netdev->features = 0;
+	KUNIT_EXPECT_EQ(test, uso_parse(ctx, &info, header), -EOPNOTSUPP);
+
+	/* A single datagram does not need it. */
+	uso_parse_init(test, ctx, 1000);
+	ctx->pool->netdev->features = 0;
+	KUNIT_EXPECT_EQ(test, uso_parse(ctx, &info, header), 1);
+	KUNIT_EXPECT_EQ(test, info.gso_segs, 1);
+}
+
+static void uso_parse_no_uso(struct kunit *test)
+{
+	struct uso_parse_ctx *ctx = uso_parse_ctx(test);
+	void *header = uso_header(test);
+	struct xsk_tx_uso_info info;
+
+	uso_parse_init(test, ctx, 4000);
+	ctx->meta.flags = XDP_TXMD_FLAGS_CHECKSUM;
+	KUNIT_EXPECT_EQ(test, uso_parse(ctx, &info, header), 0);
+	ctx->meta.flags = XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM;
+	ctx->descs[0].options &= ~XDP_TX_METADATA;
+	KUNIT_EXPECT_EQ(test, uso_parse(ctx, &info, header), 0);
+}
+
+static struct kunit_case xsk_tx_uso_cases[] = {
+	KUNIT_CASE(uso_valid),
+	KUNIT_CASE_PARAM(uso_invalid, uso_invalid_gen_params),
+	KUNIT_CASE(uso_limits),
+	KUNIT_CASE(uso_header_layout),
+	KUNIT_CASE(uso_prepare_headers),
+	KUNIT_CASE(uso_prepare_single),
+	KUNIT_CASE(uso_metadata_flags),
+	KUNIT_CASE(uso_snapshot),
+	KUNIT_CASE_PARAM(uso_parse_invalid, uso_parse_invalid_gen_params),
+	KUNIT_CASE(uso_parse_feature),
+	KUNIT_CASE(uso_parse_no_uso),
+	{}
+};
+
+static struct kunit_suite xsk_tx_uso_suite = {
+	.name = "xsk_tx_uso",
+	.test_cases = xsk_tx_uso_cases,
+};
+
+kunit_test_suite(xsk_tx_uso_suite);
+MODULE_DESCRIPTION("KUnit tests for AF_XDP USO header helpers");
+MODULE_LICENSE("GPL");
-- 
2.38.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH net-next 5/5] selftests: net: add AF_XDP USO test
  2026-11-19 16:53 [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Magnus Karlsson
                   ` (3 preceding siblings ...)
  2026-11-19 16:53 ` [PATCH net-next 4/5] xsk: add KUnit tests for AF_XDP USO header helpers Magnus Karlsson
@ 2026-11-19 16:53 ` Magnus Karlsson
  2026-11-19 19:05   ` Maciej Fijalkowski
  4 siblings, 1 reply; 8+ messages in thread
From: Magnus Karlsson @ 2026-11-19 16:53 UTC (permalink / raw)
  To: netdev, magnus.karlsson, magnus.karlsson, maciej.fijalkowski, sdf,
	davem, edumazet, kuba, pabeni, horms, bjorn, kerneljasonxing
  Cc: intel-wired-lan, anthony.l.nguyen, przemyslaw.kitszel,
	jacob.e.keller

Add xsk_uso, a functional test for AF_XDP UDP segmentation offload in
copy and zero-copy mode. It sends USO requests from an AF_XDP socket
and checks the datagrams received on a packet socket: the number of
datagrams, the IP and UDP lengths, the checksums and the payload, for
IPv4 and IPv6, and single and multi-buffer requests. It also checks
that invalid requests are dropped and counted in tx_invalid_descs.

The test needs root and physical NICs with an MTU of at least 1448,
either two connected ports (-i TX_IFACE -r RX_IFACE) or one in NIC
loopback (-l -i IFACE). Virtual devices such as veth pass the packet on
without segmenting it. Zero copy (-z) also needs a native XDP program on
the interface. Standalone checksum offload cases run only with -C, as
they test the existing checksum offload rather than USO. As it needs
real hardware, it is built but not run by default, like xdp_hw_metadata.
It uses the AF_XDP library in net/lib/xsk.

Missing prerequisites are reported as skips. In zero-copy mode, the
device must report xsk-zerocopy and tx-uso-zc, and checksum-only cases
need tx-checksum. In both modes, cases that produce more than one
datagram need tx-udp-segmentation, including the invalid ones, which
would otherwise be dropped for lack of the offload whatever their
headers. Cases that need more descriptors than xdp-zc-max-segs are
skipped, and so are binds that the device refuses with EOPNOTSUPP.
Receive-side UDP GRO (rx-udp-gro-forwarding, rx-gro-list) must be off,
as it would merge the segments again before the packet socket sees them.
USO with launch time must succeed in copy mode and be dropped in
zero-copy mode.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 tools/testing/selftests/net/.gitignore |   1 +
 tools/testing/selftests/net/Makefile   |  10 +-
 tools/testing/selftests/net/xsk_uso.c  | 954 +++++++++++++++++++++++++
 3 files changed, 964 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/net/xsk_uso.c

diff --git a/tools/testing/selftests/net/.gitignore b/tools/testing/selftests/net/.gitignore
index 83d68e2dd4ac..e40a97618cdc 100644
--- a/tools/testing/selftests/net/.gitignore
+++ b/tools/testing/selftests/net/.gitignore
@@ -64,3 +64,4 @@ udpgso_bench_rx
 udpgso_bench_tx
 xdp_hw_metadata
 xdp_hw_metadata.skel.h
+xsk_uso
diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index c80dabbcc907..afe839db2a45 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -219,7 +219,8 @@ TEST_FILES := \
 # end of TEST_FILES
 
 # YNL files, must be before "include ..lib.mk"
-YNL_GEN_FILES := busy_poller
+# xsk_uso needs root and a NIC with loopback or a connected peer; see the source.
+YNL_GEN_FILES := busy_poller xsk_uso
 YNL_GEN_PROGS := \
 	netlink-dumps \
 	tun \
@@ -273,3 +274,10 @@ $(OUTPUT)/xdp_hw_metadata: xdp_hw_metadata.c xdp_metadata.h lib/xsk/xsk.c \
 	$(Q)$(CC) -I$(top_srcdir)/tools/include $(CFLAGS) -I$(OUTPUT) \
 		-I$(SCRATCH_DIR)/include \
 		$(filter %.c,$^) $(BPFOBJ) $(LDLIBS) -lelf -lz -o $@
+
+$(OUTPUT)/xsk_uso: xsk_uso.c lib/xsk/xsk.c lib/xsk/xsk.h $(BPFOBJ) \
+		   $(OUTPUT)/libynl.a
+	$(call msg,CC,,$@)
+	$(Q)$(CC) -I$(top_srcdir)/tools/include $(CFLAGS) \
+		-I$(SCRATCH_DIR)/include \
+		$(filter %.c,$^) $(BPFOBJ) $(OUTPUT)/libynl.a $(LDLIBS) -lelf -lz -o $@
diff --git a/tools/testing/selftests/net/xsk_uso.c b/tools/testing/selftests/net/xsk_uso.c
new file mode 100644
index 000000000000..2e3228e2b2f8
--- /dev/null
+++ b/tools/testing/selftests/net/xsk_uso.c
@@ -0,0 +1,954 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* AF_XDP UDP segmentation offload (USO) test.
+ *
+ * Sends USO requests from an AF_XDP socket in copy or zero-copy mode and
+ * checks the datagrams received on a packet socket: their number, IP and
+ * UDP lengths, checksums and payload, for IPv4 and IPv6, and single and
+ * multi-buffer requests. Invalid requests must be dropped and counted in
+ * tx_invalid_descs.
+ *
+ * Needs root and physical NICs with an MTU of at least 1448, either two
+ * connected ports (-i TX_IFACE -r RX_IFACE) or one in NIC loopback
+ * (-l -i IFACE). Virtual devices such as veth pass the packet on without
+ * segmenting it, so they are skipped. An MTU too large for eight 4K buffers
+ * is also a skip. Zero-copy mode (-z) also needs a native XDP program on the
+ * interface and a driver reporting tx-uso-zc. Requests that produce more than one datagram
+ * need tx-udp-segmentation. Standalone checksum offload cases run only with
+ * -C, and in zero-copy mode only on drivers reporting tx-checksum. Missing
+ * prerequisites are reported as skips.
+ */
+
+#define _GNU_SOURCE
+#include <arpa/inet.h>
+#include <errno.h>
+#include <linux/ethtool.h>
+#include <linux/if_ether.h>
+#include <linux/if_link.h>
+#include <linux/if_packet.h>
+#include <linux/if_xdp.h>
+#include <linux/ipv6.h>
+#include <linux/netdev.h>
+#include <linux/sockios.h>
+#include <linux/udp.h>
+#include <net/if.h>
+#include <netinet/ip.h>
+#include <poll.h>
+#include <stdbool.h>
+#include <assert.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/socket.h>
+#include <time.h>
+#include <unistd.h>
+#include <ynl.h>
+
+#include "kselftest.h"
+#include "lib/xsk/xsk.h"
+#include "netdev-user.h"
+
+#define RING_SIZE 1024
+#define NUM_FRAMES 1024
+#define FRAME_SIZE 4096
+#define META_LEN sizeof(struct xsk_tx_metadata)
+/* Each segment then fills an IPv4 or IPv6 packet of at most 1448 bytes, the
+ * smallest MTU that the test accepts.
+ */
+#define GSO_SIZE 1400
+#define MIN_MTU (sizeof(struct ipv6hdr) + sizeof(struct udphdr) + GSO_SIZE)
+/* Most buffers a test packet uses. In zero-copy mode, cases that need more
+ * descriptors than the driver's xdp-zc-max-segs are skipped.
+ */
+#define MAX_BUFFERS 8
+#define TEST_PORT 49732
+#define TIMEOUT_MS 2000
+
+static_assert(sizeof(struct xsk_tx_metadata) == 24, "metadata size");
+static_assert(offsetof(struct xsk_tx_metadata, request.gso_size) == 12,
+	      "segment size offset");
+static_assert(offsetof(struct xsk_tx_metadata, request.network_offset) == 14,
+	      "network header offset");
+static_assert(offsetof(struct xsk_tx_metadata, request.launch_time) == 16,
+	      "launch time offset");
+static_assert(offsetof(struct xsk_tx_metadata, completion.tx_timestamp) == 8,
+	      "timestamp offset");
+
+enum bad_request {
+	VALID, ZERO_SIZE, FRAGS_SIZE, NO_CHECKSUM, BAD_CSUM_OFFSET,
+	BAD_NETWORK_OFFSET, BAD_UDP_LENGTH, FRAGMENT, BAD_PROTOCOL,
+	EXT_HEADER, LARGE_SEGMENT, SHORT_HEADER, TIMESTAMP, LAUNCH_TIME,
+	SW_CHECKSUM, UNKNOWN_FLAG, BAD_IP_LENGTH, BAD_IP_VERSION, BAD_IHL,
+	BAD_TRANSPORT_OFFSET, TOO_MANY_SEGMENTS,
+};
+
+static const struct test_case {
+	const char *name;
+	bool ipv6;
+	unsigned int payload;
+	unsigned int first_len;
+	enum bad_request bad;
+	bool checksum_only;
+} cases[] = {
+	{ "IPv4 USO", false, 4000, 0, VALID, false },
+	{ "IPv6 USO", true, 4000, 0, VALID, false },
+	{ "IPv4 multi-buffer USO", false, 6000, 2048, VALID, false },
+	{ "IPv6 multi-buffer USO", true, 6000, 2048, VALID, false },
+	{ "single output datagram", false, GSO_SIZE, 0, VALID, false },
+	{ "empty IPv4 datagram", false, 0, 0, VALID, false },
+	{ "empty IPv6 datagram", true, 0, 0, VALID, false },
+	{ "checksum-only compatibility", false, 1000, 0, VALID, true },
+	{ "IPv6 checksum-only", true, 1000, 0, VALID, true },
+	{ "IPv4 multi-buffer checksum-only", false, 1400, 512, VALID, true },
+	{ "IPv6 multi-buffer checksum-only", true, 1400, 512, VALID, true },
+	{ "empty IPv4 checksum-only", false, 0, 0, VALID, true },
+	{ "empty IPv6 checksum-only", true, 0, 0, VALID, true },
+	{ "zero segment size", false, 4000, 0, ZERO_SIZE, false },
+	{ "GSO_BY_FRAGS segment size", false, 4000, 0, FRAGS_SIZE, false },
+	{ "missing checksum flag", false, 6000, 2048, NO_CHECKSUM, false },
+	{ "invalid checksum offset", false, 4000, 0, BAD_CSUM_OFFSET, false },
+	{ "invalid IP offset", false, 4000, 0, BAD_NETWORK_OFFSET, false },
+	{ "invalid UDP length", false, 6000, 2048, BAD_UDP_LENGTH, false },
+	{ "fragmented IPv4", false, 4000, 0, FRAGMENT, false },
+	{ "non-UDP protocol", false, 4000, 0, BAD_PROTOCOL, false },
+	{ "IPv6 extension header", true, 4000, 0, EXT_HEADER, false },
+	{ "segment exceeds MTU", false, 0, 0, LARGE_SEGMENT, false },
+	{ "headers split across buffers", true, 4000, 0, SHORT_HEADER, false },
+	{ "USO with timestamp", false, 4000, 0, TIMESTAMP, false },
+	{ "USO with launch time", false, 4000, 0, LAUNCH_TIME, false },
+	{ "IPv6 USO with launch time", true, 4000, 0, LAUNCH_TIME, false },
+	{ "multi-buffer USO with launch time", false, 6000, 2048, LAUNCH_TIME, false },
+	{ "USO with software checksum", false, 4000, 0, SW_CHECKSUM, false },
+	{ "unknown metadata flag", false, 4000, 0, UNKNOWN_FLAG, false },
+	{ "invalid IPv4 total length", false, 4000, 0, BAD_IP_LENGTH, false },
+	{ "invalid IPv6 payload length", true, 4000, 0, BAD_IP_LENGTH, false },
+	{ "invalid IPv4 version", false, 4000, 0, BAD_IP_VERSION, false },
+	{ "invalid IPv6 version", true, 4000, 0, BAD_IP_VERSION, false },
+	{ "invalid IPv4 header length", false, 4000, 0, BAD_IHL, false },
+	{ "UDP offset mismatch", false, 4000, 0, BAD_TRANSPORT_OFFSET, false },
+	{ "too many segments", false, 4000, 0, TOO_MANY_SEGMENTS, false },
+};
+
+struct xsk {
+	int fd;
+	void *umem;
+	struct xsk_umem *umem_handle;
+	struct xsk_socket *socket;
+	struct xsk_ring_prod fill;
+	struct xsk_ring_prod tx;
+	struct xsk_ring_cons cq;
+};
+
+static unsigned char tx_mac[ETH_ALEN], rx_mac[ETH_ALEN];
+static unsigned char tx_packet[MAX_BUFFERS * FRAME_SIZE];
+static unsigned int mtu;
+static bool zero_copy;
+static bool checksum_tests;
+
+static uint64_t now_ms(void)
+{
+	struct timespec ts;
+
+	if (clock_gettime(CLOCK_MONOTONIC, &ts))
+		ksft_exit_fail_msg("clock_gettime: %s\n", strerror(errno));
+	return ts.tv_sec * 1000ULL + ts.tv_nsec / 1000000;
+}
+
+static uint32_t sum_bytes(const void *data, unsigned int len)
+{
+	const unsigned char *p = data;
+	uint32_t sum = 0;
+
+	while (len > 1) {
+		sum += p[0] * 256 + p[1];
+		p += 2;
+		len -= 2;
+	}
+	if (len)
+		sum += p[0] * 256;
+	return sum;
+}
+
+static uint16_t fold(uint32_t sum)
+{
+	while (sum >> 16)
+		sum = (sum & 0xffff) + (sum >> 16);
+	return sum;
+}
+
+static uint32_t pseudo_sum(const void *ip, bool ipv6, unsigned int udp_len)
+{
+	const struct iphdr *h = ip;
+
+	if (ipv6) {
+		const struct ipv6hdr *h6 = ip;
+
+		return sum_bytes(&h6->saddr, 32) + IPPROTO_UDP + udp_len;
+	}
+	return sum_bytes(&h->saddr, 8) + IPPROTO_UDP + udp_len;
+}
+
+static void get_mac(const char *name, unsigned char *mac)
+{
+	struct ifreq ifr = {};
+	int fd = socket(AF_INET, SOCK_DGRAM, 0);
+
+	if (fd < 0)
+		ksft_exit_fail_msg("socket: %s\n", strerror(errno));
+	snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", name);
+	if (ioctl(fd, SIOCGIFHWADDR, &ifr))
+		ksft_exit_fail_msg("MAC address for %s: %s\n", name, strerror(errno));
+	memcpy(mac, ifr.ifr_hwaddr.sa_data, ETH_ALEN);
+	close(fd);
+}
+
+static unsigned int get_mtu(const char *name)
+{
+	struct ifreq ifr = {};
+	int fd = socket(AF_INET, SOCK_DGRAM, 0);
+
+	snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", name);
+	if (fd < 0 || ioctl(fd, SIOCGIFMTU, &ifr))
+		ksft_exit_fail_msg("MTU of %s: %s\n", name, strerror(errno));
+	close(fd);
+	return ifr.ifr_mtu;
+}
+
+/* Virtual devices have no backing device, and pass GSO packets on whole. */
+static bool is_physical(const char *name)
+{
+	char path[64];
+
+	snprintf(path, sizeof(path), "/sys/class/net/%s/device", name);
+	return !access(path, F_OK);
+}
+
+static int ethtool_ioctl(const char *name, void *cmd)
+{
+	struct ifreq ifr = { .ifr_data = cmd };
+	int fd = socket(AF_INET, SOCK_DGRAM, 0), err;
+
+	if (fd < 0)
+		return -1;
+	snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", name);
+	err = ioctl(fd, SIOCETHTOOL, &ifr);
+	close(fd);
+	return err;
+}
+
+/* Returns whether the netdev feature called @feature is active on @name. */
+static bool feature_active(const char *name, const char *feature)
+{
+	struct ethtool_gfeatures *features = NULL;
+	struct ethtool_gstrings *strings = NULL;
+	struct ethtool_sset_info *sset;
+	unsigned int count, blocks, i;
+	bool active = false;
+
+	sset = calloc(1, sizeof(*sset) + sizeof(sset->data[0]));
+	if (!sset)
+		ksft_exit_fail_msg("calloc: %s\n", strerror(errno));
+	sset->cmd = ETHTOOL_GSSET_INFO;
+	sset->sset_mask = 1ULL << ETH_SS_FEATURES;
+	if (ethtool_ioctl(name, sset))
+		ksft_exit_fail_msg("ETHTOOL_GSSET_INFO: %s\n", strerror(errno));
+	if (!sset->sset_mask)
+		ksft_exit_fail_msg("%s reports no feature strings\n", name);
+	count = sset->data[0];
+	blocks = (count + 31) / 32;
+	strings = calloc(1, sizeof(*strings) + count * ETH_GSTRING_LEN);
+	features = calloc(1, sizeof(*features) +
+			  blocks * sizeof(features->features[0]));
+	if (!strings || !features)
+		ksft_exit_fail_msg("calloc: %s\n", strerror(errno));
+	strings->cmd = ETHTOOL_GSTRINGS;
+	strings->string_set = ETH_SS_FEATURES;
+	strings->len = count;
+	features->cmd = ETHTOOL_GFEATURES;
+	features->size = blocks;
+	if (ethtool_ioctl(name, strings) || ethtool_ioctl(name, features))
+		ksft_exit_fail_msg("ethtool features: %s\n", strerror(errno));
+	for (i = 0; i < count; i++)
+		if (!strncmp((char *)strings->data + i * ETH_GSTRING_LEN, feature,
+			     ETH_GSTRING_LEN)) {
+			active = features->features[i / 32].active & (1U << (i % 32));
+			break;
+		}
+	free(features);
+	free(strings);
+	free(sset);
+	return active;
+}
+
+static void get_xdp_features(unsigned int ifindex, uint64_t *xdp, uint64_t *xsk,
+			     unsigned int *zc_max_segs)
+{
+	struct netdev_dev_get_req *req;
+	struct netdev_dev_get_rsp *rsp;
+	struct ynl_error yerr;
+	struct ynl_sock *ys;
+
+	ys = ynl_sock_create(&ynl_netdev_family, &yerr);
+	if (!ys)
+		ksft_exit_fail_msg("YNL: %s\n", yerr.msg);
+	req = netdev_dev_get_req_alloc();
+	if (!req)
+		ksft_exit_fail_msg("netdev_dev_get_req_alloc failed\n");
+	netdev_dev_get_req_set_ifindex(req, ifindex);
+	rsp = netdev_dev_get(ys, req);
+	netdev_dev_get_req_free(req);
+	if (!rsp)
+		ksft_exit_fail_msg("netdev dev-get: %s\n", ys->err.msg);
+	*xdp = rsp->_present.xdp_features ? rsp->xdp_features : 0;
+	*xsk = rsp->_present.xsk_features ? rsp->xsk_features : 0;
+	*zc_max_segs = rsp->_present.xdp_zc_max_segs ? rsp->xdp_zc_max_segs : 1;
+	netdev_dev_get_rsp_free(rsp);
+	ynl_sock_destroy(ys);
+}
+
+/* Returns 0, or a negative error if the device refuses the bind. */
+static int open_xsk(struct xsk *xsk, unsigned int ifindex, bool sw_csum)
+{
+	struct xsk_socket_config config = {
+		.tx_size = RING_SIZE,
+		.bind_flags = (zero_copy ? XDP_ZEROCOPY : XDP_COPY) | XDP_USE_SG,
+	};
+	struct xsk_umem_config umem_config = {
+		.fill_size = RING_SIZE,
+		.comp_size = RING_SIZE,
+		.frame_size = FRAME_SIZE,
+		.flags = XDP_UMEM_TX_METADATA_LEN,
+		.tx_metadata_len = META_LEN,
+	};
+	uint64_t deadline;
+	int err;
+
+	memset(xsk, 0, sizeof(*xsk));
+	xsk->umem = mmap(NULL, NUM_FRAMES * FRAME_SIZE, PROT_READ | PROT_WRITE,
+			 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (xsk->umem == MAP_FAILED)
+		ksft_exit_fail_msg("mmap UMEM: %s\n", strerror(errno));
+	if (sw_csum)
+		umem_config.flags |= XDP_UMEM_TX_SW_CSUM;
+	err = xsk_umem__create(&xsk->umem_handle, xsk->umem, NUM_FRAMES * FRAME_SIZE,
+			       &xsk->fill, &xsk->cq, &umem_config);
+	if (err)
+		ksft_exit_fail_msg("UMEM creation: %s\n", strerror(-err));
+	/* A previous socket on the queue may still be going away. */
+	deadline = now_ms() + TIMEOUT_MS;
+	while ((err = xsk_socket__create(&xsk->socket, ifindex, 0,
+					 xsk->umem_handle, NULL, &xsk->tx,
+					 &config))) {
+		if (err != -EBUSY || now_ms() >= deadline) {
+			xsk_umem__delete(xsk->umem_handle);
+			munmap(xsk->umem, NUM_FRAMES * FRAME_SIZE);
+			return err;
+		}
+		usleep(10000);
+	}
+	xsk->fd = xsk_socket__fd(xsk->socket);
+	return 0;
+}
+
+static void close_xsk(struct xsk *xsk)
+{
+	int err;
+
+	xsk_socket__delete(xsk->socket);
+	err = xsk_umem__delete(xsk->umem_handle);
+	if (err)
+		ksft_exit_fail_msg("UMEM deletion: %s\n", strerror(-err));
+	munmap(xsk->umem, NUM_FRAMES * FRAME_SIZE);
+}
+
+/* An IPv4 segment size one byte larger than the MTU allows. */
+static unsigned int large_gso_size(void)
+{
+	return mtu - sizeof(struct iphdr) - sizeof(struct udphdr) + 1;
+}
+
+static unsigned int case_payload(const struct test_case *t)
+{
+	/* Two segments, each too large for the MTU. */
+	return t->bad == LARGE_SEGMENT ? 2 * large_gso_size() : t->payload;
+}
+
+static unsigned int
+make_packet(unsigned char *packet, const struct test_case *t, unsigned int id,
+	    struct xsk_tx_metadata *meta)
+{
+	struct ethhdr *eth = (void *)packet;
+	unsigned int ip_len = t->ipv6 ? sizeof(struct ipv6hdr) : sizeof(struct iphdr);
+	unsigned int start = sizeof(*eth) + ip_len;
+	unsigned int udp_len = sizeof(struct udphdr) + case_payload(t);
+	struct udphdr *uh = (void *)(packet + start);
+	void *ip = packet + sizeof(*eth);
+	unsigned int i;
+
+	memset(packet, 0, start + udp_len);
+	memcpy(eth->h_source, tx_mac, ETH_ALEN);
+	memcpy(eth->h_dest, rx_mac, ETH_ALEN);
+	eth->h_proto = htons(t->ipv6 ? ETH_P_IPV6 : ETH_P_IP);
+	if (t->ipv6) {
+		struct ipv6hdr *h = ip;
+
+		h->version = 6;
+		h->nexthdr = IPPROTO_UDP;
+		h->hop_limit = 64;
+		h->payload_len = htons(udp_len);
+		inet_pton(AF_INET6, "2001:db8::1", &h->saddr);
+		inet_pton(AF_INET6, "2001:db8::2", &h->daddr);
+	} else {
+		struct iphdr *h = ip;
+
+		h->version = 4;
+		h->ihl = 5;
+		h->ttl = 64;
+		h->protocol = IPPROTO_UDP;
+		h->tot_len = htons(ip_len + udp_len);
+		h->frag_off = htons(IP_DF);
+		inet_pton(AF_INET, "198.18.0.1", &h->saddr);
+		inet_pton(AF_INET, "198.18.0.2", &h->daddr);
+		h->check = htons((uint16_t)~fold(sum_bytes(h, ip_len)));
+	}
+	uh->source = htons(TEST_PORT);
+	uh->dest = htons(TEST_PORT);
+	uh->len = htons(udp_len);
+	uh->check = htons(fold(pseudo_sum(ip, t->ipv6, udp_len)));
+	for (i = 0; i < udp_len - sizeof(*uh); i++)
+		packet[start + sizeof(*uh) + i] = (i + id * 13) % 251;
+
+	memset(meta, 0, sizeof(*meta));
+	meta->flags = XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM;
+	meta->request.csum_start = start;
+	meta->request.csum_offset = offsetof(struct udphdr, check);
+	meta->request.gso_size = GSO_SIZE;
+	meta->request.network_offset = sizeof(*eth);
+	if (t->checksum_only)
+		meta->flags &= ~XDP_TXMD_FLAGS_USO;
+	switch (t->bad) {
+	case ZERO_SIZE:
+		meta->request.gso_size = 0;
+		break;
+	case FRAGS_SIZE:
+		meta->request.gso_size = UINT16_MAX;
+		break;
+	case NO_CHECKSUM:
+		meta->flags &= ~XDP_TXMD_FLAGS_CHECKSUM;
+		break;
+	case BAD_CSUM_OFFSET:
+		meta->request.csum_offset = 0;
+		break;
+	case BAD_NETWORK_OFFSET:
+		meta->request.network_offset++;
+		break;
+	case BAD_UDP_LENGTH:
+		uh->len = htons(udp_len - 1);
+		break;
+	case FRAGMENT:
+		((struct iphdr *)ip)->frag_off = htons(IP_MF);
+		break;
+	case BAD_PROTOCOL:
+		((struct iphdr *)ip)->protocol = IPPROTO_TCP;
+		break;
+	case EXT_HEADER:
+		((struct ipv6hdr *)ip)->nexthdr = IPPROTO_DSTOPTS;
+		break;
+	case LARGE_SEGMENT:
+		meta->request.gso_size = large_gso_size();
+		break;
+	case TIMESTAMP:
+		meta->flags |= XDP_TXMD_FLAGS_TIMESTAMP;
+		break;
+	case LAUNCH_TIME:
+		meta->flags |= XDP_TXMD_FLAGS_LAUNCH_TIME;
+		break;
+	case UNKNOWN_FLAG:
+		meta->flags |= 1ULL << 63;
+		break;
+	case BAD_IP_LENGTH:
+		if (t->ipv6)
+			((struct ipv6hdr *)ip)->payload_len = htons(udp_len - 1);
+		else
+			((struct iphdr *)ip)->tot_len = htons(ip_len + udp_len - 1);
+		break;
+	case BAD_IP_VERSION:
+		if (t->ipv6)
+			((struct ipv6hdr *)ip)->version = 4;
+		else
+			((struct iphdr *)ip)->version = 6;
+		break;
+	case BAD_IHL:
+		((struct iphdr *)ip)->ihl = 4;
+		break;
+	case BAD_TRANSPORT_OFFSET:
+		meta->request.csum_start += 4;
+		break;
+	case TOO_MANY_SEGMENTS:
+		meta->request.gso_size = 1;
+		break;
+	default:
+		break;
+	}
+	return start + udp_len;
+}
+
+/* EAGAIN means no Tx progress until completions free resources, so return
+ * to the caller to reap them instead of sleeping.
+ */
+static bool kick_tx(struct xsk *xsk)
+{
+	if (sendto(xsk->fd, NULL, 0, MSG_DONTWAIT, NULL, 0) < 0 && errno != EAGAIN) {
+		ksft_print_msg("sendto: %s\n", strerror(errno));
+		return false;
+	}
+	return true;
+}
+
+/* Length of descriptor @n of a packet with @left bytes still to place. */
+static unsigned int desc_len(const struct test_case *t,
+			     const struct xsk_tx_metadata *meta, unsigned int n,
+			     unsigned int left)
+{
+	unsigned int amount = left;
+
+	if (amount > FRAME_SIZE - META_LEN)
+		amount = FRAME_SIZE - META_LEN;
+	if (!n && t->first_len && amount > t->first_len)
+		amount = t->first_len;
+	if (!n && t->bad == SHORT_HEADER)
+		amount = meta->request.csum_start + sizeof(struct udphdr) - 1;
+	return amount;
+}
+
+static unsigned int count_descs(const struct test_case *t,
+				const struct xsk_tx_metadata *meta,
+				unsigned int len)
+{
+	unsigned int n = 0, copied = 0;
+
+	while (copied < len)
+		copied += desc_len(t, meta, n++, len - copied);
+	return n;
+}
+
+/* Submits the packet and waits for its completions. Sets @nr_descs to the
+ * number of descriptors used.
+ */
+static bool transmit(struct xsk *xsk, unsigned char *packet, unsigned int len,
+		     const struct test_case *t, struct xsk_tx_metadata *meta,
+		     unsigned int *nr_descs)
+{
+	uint32_t index;
+	uint64_t deadline = now_ms() + TIMEOUT_MS;
+	uint64_t addrs[RING_SIZE];
+	unsigned int copied = 0, n = 0, completed = 0;
+	bool seen[RING_SIZE] = {};
+
+	while (copied < len) {
+		struct xdp_desc *desc;
+		unsigned int amount = desc_len(t, meta, n, len - copied);
+		uint64_t addr = n * FRAME_SIZE + META_LEN;
+
+		memcpy(xsk->umem + addr, packet + copied, amount);
+		if (!n)
+			memcpy(xsk->umem + addr - META_LEN, meta, META_LEN);
+		if (xsk_ring_prod__reserve(&xsk->tx, 1, &index) != 1) {
+			ksft_print_msg("functional Tx ring reservation failed\n");
+			return false;
+		}
+		desc = xsk_ring_prod__tx_desc(&xsk->tx, index);
+		*desc = (struct xdp_desc) {
+			.addr = addr,
+			.len = amount,
+			.options = (!n && meta->flags ? XDP_TX_METADATA : 0) |
+				   (copied + amount < len ? XDP_PKT_CONTD : 0),
+		};
+		addrs[n++] = addr;
+		copied += amount;
+	}
+	xsk_ring_prod__submit(&xsk->tx, n);
+	*nr_descs = n;
+	while (completed < n) {
+		unsigned int count, i;
+
+		if (!kick_tx(xsk))
+			return false;
+		count = xsk_ring_cons__peek(&xsk->cq, RING_SIZE, &index);
+		for (i = 0; i < count; i++) {
+			uint64_t addr = *xsk_ring_cons__comp_addr(&xsk->cq, index + i);
+			unsigned int j;
+
+			for (j = 0; j < n; j++)
+				if (addr == addrs[j])
+					break;
+			if (j == n || seen[j]) {
+				ksft_print_msg("functional Tx: invalid or duplicate completion\n");
+				return false;
+			}
+			seen[j] = true;
+		}
+		xsk_ring_cons__release(&xsk->cq, count);
+		completed += count;
+		if (completed < n && now_ms() > deadline) {
+			ksft_print_msg("completion timeout\n");
+			return false;
+		}
+	}
+	return true;
+}
+
+/* USO with launch time is copy mode only. */
+static bool valid_request(const struct test_case *t)
+{
+	return t->bad == VALID || (t->bad == LAUNCH_TIME && !zero_copy);
+}
+
+/* Requests that produce more than one datagram, valid or not. Without
+ * tx-udp-segmentation, invalid ones would be dropped for lack of the
+ * offload even if a header check were missing.
+ */
+static bool needs_segmentation(const struct test_case *t)
+{
+	return !t->checksum_only && case_payload(t) > GSO_SIZE;
+}
+
+/* Drop frames that a failed or late case left behind. */
+static void drain_packets(int fd)
+{
+	unsigned char packet[64];
+
+	while (recv(fd, packet, sizeof(packet), MSG_DONTWAIT | MSG_TRUNC) >= 0)
+		;
+}
+
+static bool receive_packets(int fd, const struct test_case *t, unsigned int id)
+{
+	unsigned int count = 0, received = 0;
+	unsigned int expected = valid_request(t) ? t->payload : 0;
+	unsigned int expected_count = 0;
+	uint64_t deadline;
+	unsigned char packet[8192];
+
+	if (valid_request(t)) {
+		expected_count = (expected + GSO_SIZE - 1) / GSO_SIZE;
+		if (t->checksum_only || !expected)
+			expected_count = 1;
+	}
+	deadline = now_ms() + (expected_count ? TIMEOUT_MS : 200);
+	while (now_ms() < deadline) {
+		unsigned int ip_len = t->ipv6 ? sizeof(struct ipv6hdr) : sizeof(struct iphdr);
+		unsigned int start = sizeof(struct ethhdr) + ip_len;
+		struct pollfd pfd = { .fd = fd, .events = POLLIN };
+		struct ethhdr *eth = (void *)packet;
+		struct udphdr *uh = (void *)(packet + start);
+		unsigned int payload, frame_len, max_frame_len, i;
+		ssize_t len;
+
+		if (poll(&pfd, 1, 20) < 0) {
+			ksft_print_msg("receive: poll: %s\n", strerror(errno));
+			return false;
+		}
+		len = recv(fd, packet, sizeof(packet), MSG_DONTWAIT);
+		if (len < 0) {
+			if (errno == EAGAIN)
+				continue;
+			ksft_print_msg("receive: recv: %s\n", strerror(errno));
+			return false;
+		}
+		if ((size_t)len < start + sizeof(*uh) ||
+		    memcmp(eth->h_source, tx_mac, ETH_ALEN) ||
+		    memcmp(eth->h_dest, rx_mac, ETH_ALEN) ||
+		    eth->h_proto != htons(t->ipv6 ? ETH_P_IPV6 : ETH_P_IP) ||
+		    uh->source != htons(TEST_PORT) || uh->dest != htons(TEST_PORT))
+			continue;
+		if (count >= expected_count) {
+			ksft_print_msg("receive: unexpected datagram, %u expected\n",
+				       expected_count);
+			return false;
+		}
+		payload = expected - received;
+		if (!t->checksum_only && payload > GSO_SIZE)
+			payload = GSO_SIZE;
+		frame_len = start + sizeof(*uh) + payload;
+		max_frame_len = frame_len < ETH_ZLEN ? ETH_ZLEN : frame_len;
+		if (ntohs(uh->len) != sizeof(*uh) + payload) {
+			ksft_print_msg("receive: datagram %u: UDP length %u, expected %zu\n",
+				       count, ntohs(uh->len), sizeof(*uh) + payload);
+			return false;
+		}
+		if ((size_t)len < frame_len || (size_t)len > max_frame_len) {
+			ksft_print_msg("receive: datagram %u: frame length %zd, expected %u\n",
+				       count, len, frame_len);
+			return false;
+		}
+		if (!uh->check ||
+		    fold(sum_bytes(uh, sizeof(*uh) + payload) +
+			 pseudo_sum(packet + sizeof(*eth), t->ipv6,
+				    sizeof(*uh) + payload)) != 0xffff) {
+			ksft_print_msg("receive: datagram %u: bad UDP checksum 0x%04x\n",
+				       count, ntohs(uh->check));
+			return false;
+		}
+		if (t->ipv6) {
+			struct ipv6hdr *h = (void *)(packet + sizeof(*eth));
+
+			if (ntohs(h->payload_len) != sizeof(*uh) + payload) {
+				ksft_print_msg("receive: datagram %u: IPv6 len %u, expected %zu\n",
+					       count, ntohs(h->payload_len),
+					       sizeof(*uh) + payload);
+				return false;
+			}
+		} else {
+			struct iphdr *h = (void *)(packet + sizeof(*eth));
+
+			if (ntohs(h->tot_len) != ip_len + sizeof(*uh) + payload) {
+				ksft_print_msg("receive: datagram %u: IPv4 len %u, expected %zu\n",
+					       count, ntohs(h->tot_len),
+					       ip_len + sizeof(*uh) + payload);
+				return false;
+			}
+			if (fold(sum_bytes(h, ip_len)) != 0xffff) {
+				ksft_print_msg("receive: datagram %u: bad IPv4 header checksum\n",
+					       count);
+				return false;
+			}
+		}
+		for (i = 0; i < payload; i++)
+			if (packet[start + sizeof(*uh) + i] !=
+			    (received + i + id * 13) % 251) {
+				ksft_print_msg("receive: datagram %u: bad payload byte %u\n",
+					       count, i);
+				return false;
+			}
+		received += payload;
+		count++;
+		if (count == expected_count)
+			deadline = now_ms() + 100;
+	}
+	if (received != expected || count != expected_count) {
+		ksft_print_msg("receive: %u/%u packets, %u/%u payload bytes\n",
+			       count, expected_count, received, expected);
+		return false;
+	}
+	return true;
+}
+
+/* Opens the default socket for case @i onwards. If that fails, the remaining
+ * cases cannot run.
+ */
+static void restore_xsk(struct xsk *xsk, unsigned int ifindex, unsigned int i)
+{
+	int err = open_xsk(xsk, ifindex, false);
+
+	if (!err)
+		return;
+	for (; i < ARRAY_SIZE(cases); i++)
+		ksft_test_result_error("%s: socket creation: %s\n", cases[i].name,
+				       strerror(-err));
+	ksft_finished();
+}
+
+static void reopen_xsk(struct xsk *xsk, unsigned int ifindex, unsigned int i)
+{
+	close_xsk(xsk);
+	restore_xsk(xsk, ifindex, i);
+}
+
+int main(int argc, char **argv)
+{
+	const char *tx_name = NULL, *rx_name = NULL;
+	struct sockaddr_ll addr = { .sll_family = AF_PACKET, .sll_protocol = htons(ETH_P_ALL) };
+	uint64_t xdp_features = 0, xsk_features = 0;
+	bool loopback = false, udp_segmentation;
+	struct xsk_tx_metadata meta;
+	unsigned int ifindex, i;
+	unsigned int zc_max_segs = 0;
+	bool sw_csum = false;
+	int err;
+	struct xsk xsk;
+	int fd, opt;
+
+	while ((opt = getopt(argc, argv, "i:r:zlC")) != -1) {
+		if (opt == 'i')
+			tx_name = optarg;
+		else if (opt == 'r')
+			rx_name = optarg;
+		else if (opt == 'z')
+			zero_copy = true;
+		else if (opt == 'l')
+			loopback = true;
+		else if (opt == 'C')
+			checksum_tests = true;
+		else
+			ksft_exit_fail_msg("Usage: %s [-z] [-l] [-C] -i TX_IFACE [-r RX_IFACE]\n",
+					   argv[0]);
+	}
+	if (loopback) {
+		if (rx_name && (!tx_name || strcmp(tx_name, rx_name)))
+			ksft_exit_fail_msg("Loopback needs one interface for Tx and Rx\n");
+		rx_name = tx_name;
+	}
+	ksft_print_header();
+	if (!tx_name || !rx_name || geteuid())
+		ksft_exit_skip("needs root and -i TX_IFACE -r RX_IFACE, or -l -i IFACE\n");
+	ifindex = if_nametoindex(tx_name);
+	addr.sll_ifindex = if_nametoindex(rx_name);
+	if (!addr.sll_ifindex || !ifindex ||
+	    (!loopback && !strcmp(tx_name, rx_name)))
+		ksft_exit_fail_msg("Use two connected interfaces, or -l\n");
+	get_mac(tx_name, tx_mac);
+	get_mac(rx_name, rx_mac);
+	if (loopback) {
+		/* Avoid sending the port's own source MAC back into its Rx filter. */
+		tx_mac[0] = (tx_mac[0] | 2) & ~1;
+		tx_mac[ETH_ALEN - 1] ^= 0x80;
+		memset(rx_mac, 0xff, ETH_ALEN);
+	}
+	if (!is_physical(tx_name) || !is_physical(rx_name))
+		ksft_exit_skip("%s and %s must be physical NICs that segment on the wire\n",
+			       tx_name, rx_name);
+	mtu = get_mtu(tx_name);
+	if (mtu < MIN_MTU || get_mtu(rx_name) < MIN_MTU)
+		ksft_exit_skip("%s and %s need an MTU of at least %zu\n",
+			       tx_name, rx_name, MIN_MTU);
+	/* The oversized-segment case must fit the buffers of a packet. */
+	if (sizeof(struct ethhdr) + sizeof(struct iphdr) +
+	    sizeof(struct udphdr) + 2 * large_gso_size() >
+	    MAX_BUFFERS * (FRAME_SIZE - META_LEN))
+		ksft_exit_skip("%s has MTU %u, too large for the test buffers\n",
+			       tx_name, mtu);
+	if (zero_copy) {
+		get_xdp_features(ifindex, &xdp_features, &xsk_features,
+				 &zc_max_segs);
+		if (!(xdp_features & NETDEV_XDP_ACT_XSK_ZEROCOPY))
+			ksft_exit_skip("%s does not support AF_XDP zero-copy\n", tx_name);
+		if (!(xsk_features & NETDEV_XSK_FLAGS_TX_USO_ZC))
+			ksft_exit_skip("%s does not report tx-uso-zc\n", tx_name);
+		if (!xsk_is_in_mode(ifindex, XDP_FLAGS_DRV_MODE))
+			ksft_exit_skip("%s has no native XDP program attached\n", tx_name);
+		if (zc_max_segs < 2)
+			ksft_exit_skip("%s does not support multi-buffer zero-copy\n",
+				       tx_name);
+	}
+	/* The packet socket sees received packets after GRO, which could merge
+	 * the segments back together.
+	 */
+	if (feature_active(rx_name, "rx-udp-gro-forwarding") ||
+	    feature_active(rx_name, "rx-gro-list"))
+		ksft_exit_skip("turn off rx-udp-gro-forwarding and rx-gro-list on %s\n",
+			       rx_name);
+	udp_segmentation = feature_active(tx_name, "tx-udp-segmentation");
+	fd = socket(AF_PACKET, SOCK_RAW | SOCK_NONBLOCK, htons(ETH_P_ALL));
+	if (fd < 0)
+		ksft_exit_fail_msg("packet socket: %s\n", strerror(errno));
+	{
+		int ignore_outgoing = 1;
+
+		if (setsockopt(fd, SOL_PACKET, PACKET_IGNORE_OUTGOING,
+			       &ignore_outgoing, sizeof(ignore_outgoing)) ||
+		    bind(fd, (void *)&addr, sizeof(addr)))
+			ksft_exit_fail_msg("packet socket setup: %s\n", strerror(errno));
+	}
+	err = open_xsk(&xsk, ifindex, false);
+	if (err == -EOPNOTSUPP)
+		ksft_exit_skip("%s does not support this AF_XDP bind\n", tx_name);
+	if (err)
+		ksft_exit_fail_msg("socket creation: %s\n", strerror(-err));
+	ksft_set_plan(ARRAY_SIZE(cases));
+	for (i = 0; i < ARRAY_SIZE(cases); i++) {
+		const struct test_case *t = &cases[i];
+		struct xdp_statistics before, after;
+		socklen_t size = sizeof(before);
+		unsigned int len, descs, expected_invalid = 0;
+		bool ok, want_sw_csum;
+
+		if (!checksum_tests && t->checksum_only) {
+			ksft_test_result_skip("%s: checksum offload cases need -C\n",
+					      t->name);
+			continue;
+		}
+		if (zero_copy && t->checksum_only &&
+		    !(xsk_features & NETDEV_XSK_FLAGS_TX_CHECKSUM)) {
+			ksft_test_result_skip("%s: %s does not report tx-checksum\n",
+					      t->name, tx_name);
+			continue;
+		}
+		if (zero_copy && t->bad == SW_CHECKSUM) {
+			ksft_test_result_skip("%s: software checksum is copy mode only\n",
+					      t->name);
+			continue;
+		}
+		if (needs_segmentation(t) && !udp_segmentation) {
+			ksft_test_result_skip("%s: tx-udp-segmentation is off on %s\n",
+					      t->name, tx_name);
+			continue;
+		}
+		len = make_packet(tx_packet, t, i, &meta);
+		if (zero_copy && count_descs(t, &meta, len) > zc_max_segs) {
+			ksft_test_result_skip("%s: needs more than %u descriptors\n",
+					      t->name, zc_max_segs);
+			continue;
+		}
+		want_sw_csum = t->bad == SW_CHECKSUM;
+		if (sw_csum && !want_sw_csum) {
+			sw_csum = false;
+			reopen_xsk(&xsk, ifindex, i);
+		} else if (want_sw_csum && !sw_csum) {
+			close_xsk(&xsk);
+			err = open_xsk(&xsk, ifindex, true);
+			if (err) {
+				/* open_xsk() released the socket on failure. */
+				if (err == -EOPNOTSUPP)
+					ksft_test_result_skip("%s: %s refuses %s\n", t->name,
+							      tx_name, "XDP_UMEM_TX_SW_CSUM");
+				else
+					ksft_test_result_error("%s: socket creation: %s\n",
+							       t->name, strerror(-err));
+				restore_xsk(&xsk, ifindex, i + 1);
+				continue;
+			}
+			sw_csum = true;
+		}
+		drain_packets(fd);
+		if (getsockopt(xsk.fd, SOL_XDP, XDP_STATISTICS, &before, &size)) {
+			ksft_test_result_error("%s: XDP_STATISTICS: %s\n", t->name,
+					       strerror(errno));
+			continue;
+		}
+		if (!transmit(&xsk, tx_packet, len, t, &meta, &descs)) {
+			ksft_test_result_fail("%s: submission or completion failed\n",
+					      t->name);
+			/* Start the next case from a clean socket. */
+			sw_csum = false;
+			reopen_xsk(&xsk, ifindex, i + 1);
+			continue;
+		}
+		ok = receive_packets(fd, t, i);
+		if (!valid_request(t))
+			expected_invalid = descs;
+		size = sizeof(after);
+		if (getsockopt(xsk.fd, SOL_XDP, XDP_STATISTICS, &after, &size)) {
+			ksft_test_result_error("%s: XDP_STATISTICS: %s\n", t->name,
+					       strerror(errno));
+			continue;
+		}
+		if (after.tx_invalid_descs - before.tx_invalid_descs != expected_invalid) {
+			ksft_print_msg("tx_invalid_descs grew by %llu, expected %u\n",
+				       after.tx_invalid_descs - before.tx_invalid_descs,
+				       expected_invalid);
+			ok = false;
+		}
+		if (!ok && zero_copy && valid_request(t))
+			ksft_print_msg("%s\n%s\n",
+				       "Drivers may ignore USO in some configurations,",
+				       "see Documentation/networking/xsk-tx-metadata.rst");
+		ksft_test_result(ok, "%s\n", t->name);
+	}
+	close_xsk(&xsk);
+	close(fd);
+	ksft_finished();
+}
-- 
2.38.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH net-next 1/5] xsk: add USO support for skb path
  2026-11-19 16:53 ` [PATCH net-next 1/5] xsk: add USO support for skb path Magnus Karlsson
@ 2026-11-19 18:47   ` Maciej Fijalkowski
  0 siblings, 0 replies; 8+ messages in thread
From: Maciej Fijalkowski @ 2026-11-19 18:47 UTC (permalink / raw)
  To: Magnus Karlsson
  Cc: netdev, magnus.karlsson, sdf, davem, edumazet, kuba, pabeni,
	horms, bjorn, kerneljasonxing, intel-wired-lan, anthony.l.nguyen,
	przemyslaw.kitszel, jacob.e.keller

On Thu, Nov 19, 2026 at 05:53:36PM +0100, Magnus Karlsson wrote:
> Add UDP Segmentation Offload (USO) to the AF_XDP Tx metadata interface
> and implement it for copy mode. An application submits one Ethernet
> frame holding a single IP and UDP header followed by the aggregate UDP
> payload, and the device emits a train of MTU-sized UDP datagrams.
> 
> The uAPI adds the XDP_TXMD_FLAGS_USO flag and splits the reserved u32
> in the request part of struct xsk_tx_metadata into two u16 fields:
> gso_size, the UDP payload bytes per output datagram, and
> network_offset, the IP header offset. USO requires
> XDP_TXMD_FLAGS_CHECKSUM, with csum_start pointing at the UDP header,
> csum_offset at its checksum field and the checksum field seeded with
> the partial pseudoheader checksum of the aggregate packet. It cannot
> be combined with XDP_TXMD_FLAGS_TIMESTAMP or
> XDP_UMEM_TX_SW_CSUM. Launch time is supported and applies to the start
> of the segmented packet.
> 
> The request is turned into a GSO skb of type SKB_GSO_UDP_L4. Before
> transmission, the headers are validated. The link layer is not parsed:
> the IP header starts at network_offset and its version selects
> unfragmented IPv4 or IPv6 without extension headers, followed by UDP.
> The IP and UDP lengths must describe the aggregate packet, every output
> datagram must fit the MTU, and the number of segments is limited to
> UDP_MAX_SEGMENTS. As skb drivers find L3 through the EtherType, the skb
> path also requires the EtherType, after any VLAN tags, to lead to the
> same IP header. A request that fits in one datagram is sent as an
> ordinary checksum offload request.
> 
> network_offset is part of the uAPI because csum_start only locates L4,
> while segmentation also needs the start of L3: the device rewrites the
> IP header of every datagram and needs the L2 and L3 lengths, and on
> the skb path it becomes the network header. With it, the kernel does
> not need to understand the link layer, and the fields can be reused
> later. Any future tunnel segmentation extension to the uAPI could then
> treat network_offset as the outer L3 offset and csum_start as the
> inner L4 offset, adding only the outer L4 and inner L3 offsets.
> 
> AF_XDP never segments these packets itself. A request that produces more
> than one datagram is rejected unless the bound device advertises
> NETIF_F_GSO_UDP_L4, and the GSO skb is passed to its driver unsegmented.
> As with UDP_SEGMENT sockets, how the segmentation is carried out is up
> to the driver. Stacked devices such as VLAN or bonding forward the
> packet to a lower device that may segment it in software.
> 
> A new pool field, tx_uso_supported, records whether USO requests are
> accepted on a pool. It is set only for Ethernet devices (ARPHRD_ETHER)
> and is cleared when XDP_UMEM_TX_SW_CSUM is used.
> 
> A USO request that fails validation is dropped: its buffers are
> completed and counted in tx_invalid_descs. Errors in other Tx metadata
> keep failing sendmsg() and leave the descriptor on the Tx ring, so the
> existing behaviour for non-USO requests is unchanged. A USO request
> that cannot be processed for lack of memory is also left on the Tx
> ring, and sendmsg() fails with -EAGAIN, unless part of it was already
> released to user space, either by an earlier call or while peeking
> further descriptors. That part may already have been reused, so the
> request is then dropped.
> 
> The header validator works on a raw header copy instead of an skb, so
> that the zero-copy path in a later patch can reuse it. It lives in
> the new files net/xdp/xsk_tx_uso.c and include/net/xsk_tx_uso.h.
> 
> Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>

Hmm - this looks good to me!
Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>

I only had minor comments/questions.

> ---
>  Documentation/networking/xsk-tx-metadata.rst |  91 +++++++++++++
>  MAINTAINERS                                  |   1 +
>  include/net/xdp_sock_drv.h                   |  12 ++
>  include/net/xsk_buff_pool.h                  |   4 +
>  include/net/xsk_tx_uso.h                     |  30 +++++
>  include/uapi/linux/if_xdp.h                  |  13 +-
>  net/xdp/Makefile                             |   1 +
>  net/xdp/xsk.c                                | 130 +++++++++++++++++--
>  net/xdp/xsk_buff_pool.c                      |   4 +
>  net/xdp/xsk_queue.h                          |   6 +
>  net/xdp/xsk_tx_uso.c                         |  77 +++++++++++
>  tools/include/uapi/linux/if_xdp.h            |  13 +-
>  12 files changed, 371 insertions(+), 11 deletions(-)
>  create mode 100644 include/net/xsk_tx_uso.h
>  create mode 100644 net/xdp/xsk_tx_uso.c
> 
> diff --git a/Documentation/networking/xsk-tx-metadata.rst b/Documentation/networking/xsk-tx-metadata.rst
> index ac15e5fddc6b..41cfc5482b15 100644
> --- a/Documentation/networking/xsk-tx-metadata.rst
> +++ b/Documentation/networking/xsk-tx-metadata.rst
> @@ -55,6 +55,9 @@ The flags field enables the particular offload:
>    packet for transmission at a pre-determined time called launch time. The
>    value of launch time is indicated by ``request.launch_time`` field of
>    ``struct xsk_tx_metadata``.
> +- ``XDP_TXMD_FLAGS_USO``: requests UDP segmentation offload.
> +  ``request.gso_size`` specifies UDP payload bytes per output datagram;
> +  ``request.network_offset`` identifies the IP header.
>  
>  Besides the flags above, in order to trigger the offloads, the first
>  packet's ``struct xdp_desc`` descriptor should set ``XDP_TX_METADATA``
> @@ -70,6 +73,69 @@ In this case, when running in ``XDP_COPY`` mode, the TX checksum
>  is calculated on the CPU. Do not enable this option in production because
>  it will negatively affect performance.
>  
> +UDP Segmentation
> +================
> +
> +To request UDP segmentation offload (USO), bind the socket in
> +``XDP_COPY`` mode and set ``XDP_TXMD_FLAGS_USO | XDP_TXMD_FLAGS_CHECKSUM``
> +in the Tx metadata. Reserve at least 16 bytes of metadata and set
> +``XDP_TX_METADATA`` on the first descriptor of the packet.
> +
> +Submit an Ethernet frame containing one IP header, one UDP header, and the
> +aggregate UDP payload. Set ``request.network_offset`` to the IP header
> +offset, ``request.csum_start`` to the UDP header offset, and
> +``request.csum_offset`` to the UDP checksum field offset (6 bytes).
> +Header offsets are measured from the first descriptor's packet-data address,
> +excluding metadata. Set ``request.gso_size`` to the number of UDP payload
> +bytes per output datagram, excluding headers; the last datagram may be
> +shorter.
> +
> +USO is only available on Ethernet devices. ``request.network_offset``
> +locates the IP header, and its version selects IPv4 or IPv6. In copy mode,
> +the EtherType, after up to six VLAN tags, must lead to the same IP header,
> +since drivers rely on it. IPv4 headers may carry options but must not be
> +fragmented. In IPv6, the UDP header must directly follow the fixed header;
> +extension headers are not supported.
> +
> +The IP and UDP lengths must describe the aggregate packet. Initialize the
> +UDP checksum field with the partial pseudoheader checksum seed for that
> +packet. The device generates the per-datagram lengths and UDP checksums and,
> +for IPv4 segmentation, the IP header checksums. Supply a valid IPv4 header
> +checksum as well, since a request that fits in one datagram uses ordinary
> +checksum offload without segmentation.
> +
> +All headers through UDP must be in the first buffer. Multi-buffer packets
> +use ``XDP_USE_SG`` and ``XDP_PKT_CONTD``, with metadata only on the first
> +descriptor. Each output IP packet must fit the device MTU, and requests
> +remain subject to the device's segmentation limits. Query
> +``tx-udp-segmentation`` through ethtool to check whether skb UDP segmentation
> +offload is enabled.
> +
> +AF_XDP never segments the packet itself. In copy mode, a request that
> +produces more than one datagram is rejected unless the bound device
> +advertises UDP segmentation offload, and the packet is passed to its driver
> +unsegmented. A request that fits in one datagram does not need the offload,
> +so its success does not show that the device supports USO. As with
> +``UDP_SEGMENT`` sockets, how the segmentation is carried out is up to the
> +driver, which may build the per-datagram headers in software. Stacked
> +devices such as VLAN, bonding, or macvlan advertise UDP segmentation
> +regardless of the lower device and forward the packet through the lower
> +device's qdisc, where it is segmented in software if the lower device lacks
> +the offload.
> +
> +A USO request that is rejected is dropped: its buffers are returned
> +through the completion ring and counted in the socket's ``tx_invalid_descs``
> +statistic. This is the only indication of the failure, whatever its cause:
> +invalid metadata or headers, the MTU, segmentation limits, or a device
> +without the offload. A request that cannot be processed for lack of memory
> +is instead left on the Tx ring, and ``sendmsg()`` fails with ``EAGAIN``,
> +unless part of a multi-buffer request was already released to user space;
> +it is then dropped. Errors in other Tx metadata also fail ``sendmsg()`` and
> +leave the descriptor on the Tx ring.
> +
> +USO cannot be combined with ``XDP_UMEM_TX_SW_CSUM`` or
> +``XDP_TXMD_FLAGS_TIMESTAMP``.
> +
>  Launch Time
>  ===========
>  
> @@ -127,6 +193,31 @@ cycle time is set to 1 second, the horizon of the launch time ranges
>  from 1 second to 2 seconds, depending on where the Qbv cycle is currently
>  running.
>  
> +USO with launch time
> +====================
> +
> +Add ``XDP_TXMD_FLAGS_LAUNCH_TIME`` to the USO and checksum flags and set
> +``request.launch_time`` to schedule the segmented packet. This combination
> +requires at least 24 bytes of Tx metadata.
> +
> +The launch time applies to the start of the segmented packet: the first
> +output UDP datagram is scheduled, and subsequent datagrams follow according
> +to the device's scheduling rules. This does not request per-datagram pacing
> +or separate launch times.
> +
> +The skb path passes one unsegmented GSO skb and its launch time directly to
> +the bound device's driver, bypassing that device's qdisc, including ETF.
> +On stacked devices, the lower device's qdisc and software segmentation
> +may still apply, as described above. The driver and selected queue
> +must support UDP segmentation and launch time together. Separate support
> +for either offload does not establish support for their combination.
> +As with ordinary launch-time requests, drivers may ignore the time when
> +scheduling is disabled or unsupported. Acceptance or Tx completion alone
> +does not prove that a packet was transmitted at the requested time.
> +
> +The clock, queue configuration, and scheduling horizon requirements above
> +also apply to USO launch-time requests.
> +
>  Querying Device Capabilities
>  ============================
>  
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 40bb77cbc6b0..9b5f6f307d53 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -30588,6 +30588,7 @@ F:	Documentation/networking/af_xdp.rst
>  F:	include/net/netns/xdp.h
>  F:	include/net/xdp_sock*
>  F:	include/net/xsk_buff_pool.h
> +F:	include/net/xsk_tx_uso.h
>  F:	include/uapi/linux/if_xdp.h
>  F:	include/uapi/linux/xdp_diag.h
>  F:	net/xdp/
> diff --git a/include/net/xdp_sock_drv.h b/include/net/xdp_sock_drv.h
> index a40ad3f90f13..6b5bb7b29f87 100644
> --- a/include/net/xdp_sock_drv.h
> +++ b/include/net/xdp_sock_drv.h
> @@ -263,6 +263,7 @@ xsk_buff_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr, u32 options)
>  		XDP_TXMD_FLAGS_TIMESTAMP | \
>  		XDP_TXMD_FLAGS_CHECKSUM | \
>  		XDP_TXMD_FLAGS_LAUNCH_TIME | \
> +		XDP_TXMD_FLAGS_USO | \
>  	0)
>  
>  static inline bool
> @@ -270,6 +271,10 @@ xsk_buff_valid_tx_metadata(const struct xsk_buff_pool *pool,
>  			   const struct xsk_tx_metadata *meta, u64 *flags)
>  {
>  	*flags = READ_ONCE(meta->flags);
> +	if (*flags & XDP_TXMD_FLAGS_USO)
> +		if (!pool->tx_uso_supported || pool->tx_metadata_len <
> +		    offsetofend(struct xsk_tx_metadata, request.network_offset))
> +			return false;
>  	if (*flags & XDP_TXMD_FLAGS_LAUNCH_TIME)
>  		if (pool->tx_metadata_len <
>  		    offsetofend(struct xsk_tx_metadata, request.launch_time))
> @@ -277,6 +282,13 @@ xsk_buff_valid_tx_metadata(const struct xsk_buff_pool *pool,
>  	return !(*flags & ~XDP_TXMD_FLAGS_VALID);
>  }
>  
> +/* USO requires checksum offload and cannot report a Tx timestamp. */
> +static inline bool xsk_tx_uso_flags_valid(u64 flags)
> +{
> +	return (flags & XDP_TXMD_FLAGS_CHECKSUM) &&
> +	       !(flags & XDP_TXMD_FLAGS_TIMESTAMP);
> +}
> +
>  /**
>   *  xsk_tx_metadata_request - Evaluate AF_XDP TX metadata at submission
>   *  and call appropriate xsk_tx_metadata_ops operation.
> diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h
> index 02e89350faf9..ea2ff3680efa 100644
> --- a/include/net/xsk_buff_pool.h
> +++ b/include/net/xsk_buff_pool.h
> @@ -89,6 +89,10 @@ struct xsk_buff_pool {
>  	bool uses_need_wakeup;
>  	bool unaligned;
>  	bool tx_sw_csum;
> +	/* USO requests accepted: Ethernet device without SW checksum, and
> +	 * driver support in zero-copy mode. Set when the pool is bound.
> +	 */
> +	bool tx_uso_supported;
>  	void *addrs;
>  	/* Mutual exclusion of the completion ring in the SKB mode.
>  	 * Protect: NAPI TX thread and sendmsg error paths in the SKB
> diff --git a/include/net/xsk_tx_uso.h b/include/net/xsk_tx_uso.h
> new file mode 100644
> index 000000000000..f23384077712
> --- /dev/null
> +++ b/include/net/xsk_tx_uso.h
> @@ -0,0 +1,30 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _NET_XSK_TX_USO_H
> +#define _NET_XSK_TX_USO_H
> +
> +#include <linux/errno.h>
> +#include <linux/types.h>
> +#include <net/xsk_buff_pool.h>
> +
> +/* Header offsets are relative to the start of the supplied Ethernet frame. */
> +struct xsk_tx_uso_info {
> +	struct xsk_tx_metadata metadata;
> +	u32 len;
> +	u16 header_len;
> +	u16 gso_segs;
> +	__be16 protocol;
> +	u8 l4_protocol;
> +};
> +
> +#ifdef CONFIG_XDP_SOCKETS
> +int xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso,
> +			u32 mtu);
> +#else
> +static inline int
> +xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso, u32 mtu)
> +{
> +	return -EOPNOTSUPP;
> +}
> +#endif
> +
> +#endif
> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
> index 50d67df78911..0f7ba4859218 100644
> --- a/include/uapi/linux/if_xdp.h
> +++ b/include/uapi/linux/if_xdp.h
> @@ -134,6 +134,12 @@ struct xdp_options {
>   */
>  #define XDP_TXMD_FLAGS_LAUNCH_TIME		(1 << 2)
>  
> +/* Request UDP segmentation offload in XDP_COPY mode. Requires CHECKSUM.
> + * Segment size and IP header offset are in request.
> + * With LAUNCH_TIME, scheduling applies to the start of the segmented packet.
> + */
> +#define XDP_TXMD_FLAGS_USO			(1 << 3)
> +
>  /* AF_XDP offloads request. 'request' union member is consumed by the driver
>   * when the packet is being transmitted. 'completion' union member is
>   * filled by the driver when the transmit completion arrives.
> @@ -149,7 +155,12 @@ struct xsk_tx_metadata {
>  			__u16 csum_start;
>  			/* Offset from csum_start where checksum should be stored. */
>  			__u16 csum_offset;
> -			__u32 reserved;
> +
> +			/* XDP_TXMD_FLAGS_USO */
> +			/* UDP payload bytes per output datagram. */
> +			__u16 gso_size;
> +			/* IP header offset from desc->addr. */
> +			__u16 network_offset;
>  
>  			/* XDP_TXMD_FLAGS_LAUNCH_TIME */
>  			/* Launch time in nanosecond against the PTP HW Clock */
> diff --git a/net/xdp/Makefile b/net/xdp/Makefile
> index 30cdc4315f42..b6109e0e5547 100644
> --- a/net/xdp/Makefile
> +++ b/net/xdp/Makefile
> @@ -1,4 +1,5 @@
>  # SPDX-License-Identifier: GPL-2.0-only
>  obj-$(CONFIG_XDP_SOCKETS) += xsk.o xdp_umem.o xsk_queue.o xskmap.o
>  obj-$(CONFIG_XDP_SOCKETS) += xsk_buff_pool.o
> +obj-$(CONFIG_XDP_SOCKETS) += xsk_tx_uso.o
>  obj-$(CONFIG_XDP_SOCKETS_DIAG) += xsk_diag.o
> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> index f22f9a2cdbbe..14be62b8de5f 100644
> --- a/net/xdp/xsk.c
> +++ b/net/xdp/xsk.c
> @@ -11,6 +11,8 @@
>  
>  #define pr_fmt(fmt) "AF_XDP: %s: " fmt, __func__
>  
> +#include <linux/if_arp.h>
> +#include <linux/if_vlan.h>
>  #include <linux/if_xdp.h>
>  #include <linux/init.h>
>  #include <linux/sched/mm.h>
> @@ -22,6 +24,7 @@
>  #include <linux/net.h>
>  #include <linux/netdevice.h>
>  #include <linux/rculist.h>
> +#include <linux/udp.h>
>  #include <linux/uio.h>
>  #include <linux/vmalloc.h>
>  
> @@ -31,6 +34,7 @@
>  #include <net/netdev_lock.h>
>  #include <net/netdev_rx_queue.h>
>  #include <net/xdp.h>
> +#include <net/xsk_tx_uso.h>
>  
>  #include "../core/dev.h"
>  
> @@ -986,12 +990,62 @@ static void xsk_drop_skb(struct sk_buff *skb)
>  	xs->skb = NULL;
>  }
>  
> +static int xsk_skb_uso(struct sk_buff *skb)
> +{
> +	struct xsk_tx_uso_info uso = {
> +		.metadata.request = {
> +			.gso_size = skb_shinfo(skb)->gso_size,
> +			.csum_start = skb_transport_offset(skb),
> +			.network_offset = skb_network_offset(skb),
> +		},
> +		.header_len = skb_transport_offset(skb) + sizeof(struct udphdr),
> +		.len = skb->len,
> +	};
> +	int depth, err;
> +
> +	/* Best effort: a bond or team device can change type while the pool
> +	 * is bound, also after this check.
> +	 */
> +	if (READ_ONCE(skb->dev->type) != ARPHRD_ETHER)
> +		return -EINVAL;
> +	if (!pskb_may_pull(skb, uso.header_len))
> +		return -ENOMEM;
> +
> +	err = xsk_tx_uso_validate(skb->data, &uso, READ_ONCE(skb->dev->mtu));
> +	if (err)
> +		return err;
> +
> +	skb_reset_mac_header(skb);
> +	skb->protocol = eth_hdr(skb)->h_proto;
> +	/* skb drivers find L3 through the EtherType and VLAN tags. Parse them
> +	 * from the end of the Ethernet header, not from network_offset.
> +	 */
> +	skb->mac_len = ETH_HLEN;
> +	if (__vlan_get_protocol(skb, skb->protocol, &depth) != uso.protocol ||
> +	    depth != skb_network_offset(skb))
> +		return -EINVAL;
> +	skb_reset_mac_len(skb);
> +	if (uso.gso_segs == 1) {
> +		skb_gso_reset(skb);
> +		return 0;
> +	}
> +
> +	skb_shinfo(skb)->gso_segs = uso.gso_segs;
> +	if (netif_needs_gso(skb, netif_skb_features(skb)))
> +		return -EOPNOTSUPP;
> +
> +	return 0;
> +}
> +
> +/* A failed USO request returns -EOVERFLOW, so that the packet is dropped. Other
> + * metadata errors are returned to user space.
> + */
>  static int xsk_skb_metadata(struct sk_buff *skb, void *buffer,
>  			    struct xdp_desc *desc, struct xsk_buff_pool *pool,
>  			    u32 hr)
>  {
>  	struct xsk_tx_metadata *meta = NULL;
> -	u16 csum_start, csum_offset;
> +	int err = -EINVAL;
>  	u64 flags;
>  
>  	if (unlikely(pool->tx_metadata_len == 0))
> @@ -999,18 +1053,43 @@ static int xsk_skb_metadata(struct sk_buff *skb, void *buffer,
>  
>  	meta = buffer - pool->tx_metadata_len;
>  	if (unlikely(!xsk_buff_valid_tx_metadata(pool, meta, &flags)))
> -		return -EINVAL;
> +		goto err;
> +
> +	if (flags & XDP_TXMD_FLAGS_USO) {
> +		u16 nhoff = READ_ONCE(meta->request.network_offset);
> +		u16 offset = READ_ONCE(meta->request.csum_offset);
> +		u16 thoff = READ_ONCE(meta->request.csum_start);
> +		u16 size = READ_ONCE(meta->request.gso_size);
> +
> +		if (!xsk_tx_uso_flags_valid(flags))
> +			goto err;
> +		if (!size || size == GSO_BY_FRAGS ||
> +		    nhoff < ETH_HLEN || thoff < nhoff ||
> +		    thoff + sizeof(struct udphdr) > desc->len ||
> +		    offset != offsetof(struct udphdr, check) ||
> +		    hr + thoff >= U16_MAX)
> +			goto err;
> +
> +		skb_set_network_header(skb, nhoff);
> +		skb_set_transport_header(skb, thoff);
> +		skb_shinfo(skb)->gso_size = size;
> +		skb_shinfo(skb)->gso_type = SKB_GSO_UDP_L4;
> +	}
>  
>  	if (flags & XDP_TXMD_FLAGS_CHECKSUM) {
> -		csum_start = READ_ONCE(meta->request.csum_start);
> -		csum_offset = READ_ONCE(meta->request.csum_offset);
> +		if (flags & XDP_TXMD_FLAGS_USO) {

you're doing it like this as you still want to execute
XDP_TXMD_FLAGS_CHECKSUM which is present for USO, ok.

> +			skb->csum_start = skb_transport_offset(skb);
> +			skb->csum_offset = offsetof(struct udphdr, check);
> +		} else {
> +			skb->csum_start = READ_ONCE(meta->request.csum_start);
> +			skb->csum_offset = READ_ONCE(meta->request.csum_offset);
> +		}
>  
> -		if (unlikely(csum_start + csum_offset +
> +		if (unlikely(skb->csum_start + skb->csum_offset +
>  			     sizeof(__sum16) > desc->len))
> -			return -EINVAL;
> +			goto err;
>  
> -		skb->csum_start = hr + csum_start;
> -		skb->csum_offset = csum_offset;
> +		skb->csum_start += hr;
>  		skb->ip_summed = CHECKSUM_PARTIAL;
>  	}
>  
> @@ -1021,6 +1100,9 @@ static int xsk_skb_metadata(struct sk_buff *skb, void *buffer,
>  	xsk_tx_metadata_to_compl(meta, &skb_shinfo(skb)->xsk_meta);
>  
>  	return 0;
> +
> +err:
> +	return flags & XDP_TXMD_FLAGS_USO ? -EOVERFLOW : err;

could you stick to explicit returns maybe and be consistent with rest of
flags?

>  }
>  
>  static int xsk_skb_sw_csum(struct sk_buff *skb, struct xsk_buff_pool *pool)
> @@ -1108,6 +1190,10 @@ static struct sk_buff *xsk_build_skb_zerocopy(struct xdp_sock *xs,
>  	return skb;
>  }
>  
> +/* On -EOVERFLOW, the packet was dropped and its descriptors were consumed and
> + * completed as invalid. Other errors leave the descriptor on the Tx ring for
> + * the application to retry.
> + */
>  static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
>  				     struct xdp_desc *desc)
>  {
> @@ -1196,13 +1282,17 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
>  	if (skb && !xs->skb)
>  		kfree_skb(skb);
>  
> +	/* Too many descriptors and invalid USO requests drop the packet. Only
> +	 * the latter can fail on the first descriptor, which is completed
> +	 * here. Other errors are returned so that the application can retry.
> +	 */
>  	if (err == -EOVERFLOW) {
>  		if (xs->skb) {
>  			/* Drop the packet */
>  			xsk_inc_num_desc(xs->skb);
>  			xsk_drop_skb(xs->skb);
>  		} else {
> -			xsk_cq_cancel_locked(xs->pool, 1);
> +			xsk_cq_submit_addr_single_locked(xs->pool, desc);
>  			xs->tx->invalid_descs++;
>  		}
>  		xskq_cons_release(xs->tx);
> @@ -1279,6 +1369,28 @@ static int __xsk_generic_xmit(struct sock *sk)
>  			continue;
>  		}
>  
> +		if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_L4) {
> +			err = xsk_skb_uso(skb);
> +			/* Descriptors already released to user space may be
> +			 * reused, so the packet can only be put back if none
> +			 * of them has been released. Otherwise it is dropped.

This is the only part i don't follow.

> +			 */
> +			if (err == -ENOMEM &&
> +			    xskq_cons_nb_unreleased(xs->tx) >=
> +			    xsk_get_num_desc(skb)) {
> +				/* Tell user-space to retry the send */
> +				xskq_cons_cancel_n(xs->tx, xsk_get_num_desc(skb));
> +				xsk_consume_skb(skb);
> +				err = -EAGAIN;
> +				goto out;
> +			}
> +			if (err) {
> +				xsk_drop_skb(skb);
> +				err = 0;
> +				continue;
> +			}
> +		}
> +
>  		err = xsk_skb_sw_csum(skb, xs->pool);
>  		if (unlikely(err)) {
>  			xsk_drop_skb(skb);
> diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c
> index 5ec092f7cb6f..e62f3c2b3362 100644
> --- a/net/xdp/xsk_buff_pool.c
> +++ b/net/xdp/xsk_buff_pool.c
> @@ -1,5 +1,6 @@
>  // SPDX-License-Identifier: GPL-2.0
>  
> +#include <linux/if_arp.h>
>  #include <linux/netdevice.h>
>  #include <net/netdev_lock.h>
>  #include <net/xsk_buff_pool.h>
> @@ -192,6 +193,8 @@ int xp_assign_dev(struct xsk_buff_pool *pool,
>  	err = xsk_reg_pool_at_qid(netdev, pool, queue_id);
>  	if (err)
>  		return err;
> +	pool->tx_uso_supported = !pool->tx_sw_csum &&
> +				 netdev->type == ARPHRD_ETHER;
>  
>  	if (mbuf)
>  		pool->umem->flags |= XDP_UMEM_SG_FLAG;
> @@ -260,6 +263,7 @@ int xp_assign_dev(struct xsk_buff_pool *pool,
>  		goto err_unreg_xsk;
>  	}
>  	pool->umem->zc = true;
> +	pool->tx_uso_supported = false;

I assume this will be lifted on later patch

>  	pool->xdp_zc_max_segs = netdev->xdp_zc_max_segs;
>  	return 0;
>  
> diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h
> index 1bc42c8902f4..182cd1cccb74 100644
> --- a/net/xdp/xsk_queue.h
> +++ b/net/xdp/xsk_queue.h
> @@ -390,6 +390,12 @@ static inline void xskq_cons_release(struct xsk_queue *q)
>  	q->cached_cons++;
>  }
>  
> +/* Entries consumed locally but not yet released to the producer. */
> +static inline u32 xskq_cons_nb_unreleased(struct xsk_queue *q)
> +{
> +	return q->cached_cons - q->ring->consumer;

READ_ONCE(q->ring->consumer) ?

> +}
> +
>  static inline void xskq_cons_cancel_n(struct xsk_queue *q, u32 cnt)
>  {
>  	q->cached_cons -= cnt;
> diff --git a/net/xdp/xsk_tx_uso.c b/net/xdp/xsk_tx_uso.c
> new file mode 100644
> index 000000000000..a7313ade1092
> --- /dev/null
> +++ b/net/xdp/xsk_tx_uso.c
> @@ -0,0 +1,77 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <net/ip.h>
> +#include <net/ipv6.h>
> +#include <net/udp.h>
> +#include <net/xsk_tx_uso.h>
> +
> +/**
> + * xsk_tx_uso_validate - validate a kernel-owned plain UDP header
> + * @header: kernel-owned buffer containing at least uso->header_len bytes
> + * @uso: packet length and metadata offsets; receives protocol and segment count
> + * @mtu: maximum output IP packet length
> + *
> + * The link-layer header is not parsed. The IP header starts at network_offset
> + * and its version selects IPv4 or IPv6. The caller must provide a nonzero
> + * gso_size other than GSO_BY_FRAGS and header_len equal to transport_offset
> + * plus the UDP header size.
> + *
> + * Return: zero on success or a negative error code.
> + */
> +int xsk_tx_uso_validate(const void *header, struct xsk_tx_uso_info *uso,
> +			u32 mtu)
> +{
> +	const struct xsk_tx_metadata *meta = &uso->metadata;
> +	u32 nhoff = meta->request.network_offset;
> +	u32 thoff = meta->request.csum_start;
> +	u32 udp_len, payload_len, network_len;
> +	const struct udphdr *uh;
> +	__be16 protocol;
> +
> +	if (nhoff < ETH_HLEN || thoff <= nhoff || uso->header_len < thoff ||
> +	    uso->len < uso->header_len)
> +		return -EINVAL;
> +
> +	network_len = uso->len - nhoff;
> +	switch (((const struct iphdr *)(header + nhoff))->version) {
> +	case 4: {
> +		const struct iphdr *iph = header + nhoff;
> +
> +		if (thoff - nhoff < sizeof(*iph) ||
> +		    iph->ihl < 5 || iph->ihl * 4 != thoff - nhoff ||
> +		    ip_is_fragment(iph) ||
> +		    ntohs(iph->tot_len) != network_len)

question: i keep on seeing this validation of iphdr - don't we have
somewhere a generic routine for that?

> +			return -EINVAL;
> +		protocol = htons(ETH_P_IP);
> +		uso->l4_protocol = iph->protocol;
> +		break;
> +	}
> +	case 6: {
> +		const struct ipv6hdr *ip6h = header + nhoff;
> +
> +		if (thoff - nhoff != sizeof(*ip6h) ||
> +		    ntohs(ip6h->payload_len) != network_len - sizeof(*ip6h))
> +			return -EINVAL;
> +		protocol = htons(ETH_P_IPV6);
> +		uso->l4_protocol = ip6h->nexthdr;
> +		break;
> +	}
> +	default:
> +		return -EPROTONOSUPPORT;
> +	}
> +	uso->protocol = protocol;
> +	udp_len = uso->len - thoff;
> +	uh = header + thoff;
> +	if (uso->l4_protocol != IPPROTO_UDP || udp_len > U16_MAX ||
> +	    ntohs(uh->len) != udp_len)
> +		return -EINVAL;

nit: i think you should start with this validation?

> +	payload_len = udp_len - sizeof(*uh);
> +	if (payload_len > meta->request.gso_size * UDP_MAX_SEGMENTS ||
> +	    uso->header_len - nhoff +
> +	    min_t(u32, payload_len, meta->request.gso_size) > mtu)
> +		return -EINVAL;
> +
> +	uso->gso_segs = max_t(u32, 1, DIV_ROUND_UP(payload_len,
> +						   meta->request.gso_size));
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(xsk_tx_uso_validate);
> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
> index 50d67df78911..0f7ba4859218 100644
> --- a/tools/include/uapi/linux/if_xdp.h
> +++ b/tools/include/uapi/linux/if_xdp.h
> @@ -134,6 +134,12 @@ struct xdp_options {
>   */
>  #define XDP_TXMD_FLAGS_LAUNCH_TIME		(1 << 2)
>  
> +/* Request UDP segmentation offload in XDP_COPY mode. Requires CHECKSUM.
> + * Segment size and IP header offset are in request.
> + * With LAUNCH_TIME, scheduling applies to the start of the segmented packet.
> + */
> +#define XDP_TXMD_FLAGS_USO			(1 << 3)
> +
>  /* AF_XDP offloads request. 'request' union member is consumed by the driver
>   * when the packet is being transmitted. 'completion' union member is
>   * filled by the driver when the transmit completion arrives.
> @@ -149,7 +155,12 @@ struct xsk_tx_metadata {
>  			__u16 csum_start;
>  			/* Offset from csum_start where checksum should be stored. */
>  			__u16 csum_offset;
> -			__u32 reserved;
> +
> +			/* XDP_TXMD_FLAGS_USO */
> +			/* UDP payload bytes per output datagram. */
> +			__u16 gso_size;
> +			/* IP header offset from desc->addr. */
> +			__u16 network_offset;
>  
>  			/* XDP_TXMD_FLAGS_LAUNCH_TIME */
>  			/* Launch time in nanosecond against the PTP HW Clock */
> -- 
> 2.38.1
> 

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH net-next 5/5] selftests: net: add AF_XDP USO test
  2026-11-19 16:53 ` [PATCH net-next 5/5] selftests: net: add AF_XDP USO test Magnus Karlsson
@ 2026-11-19 19:05   ` Maciej Fijalkowski
  0 siblings, 0 replies; 8+ messages in thread
From: Maciej Fijalkowski @ 2026-11-19 19:05 UTC (permalink / raw)
  To: Magnus Karlsson
  Cc: netdev, magnus.karlsson, sdf, davem, edumazet, kuba, pabeni,
	horms, bjorn, kerneljasonxing, intel-wired-lan, anthony.l.nguyen,
	przemyslaw.kitszel, jacob.e.keller

On Thu, Nov 19, 2026 at 05:53:40PM +0100, Magnus Karlsson wrote:
> Add xsk_uso, a functional test for AF_XDP UDP segmentation offload in
> copy and zero-copy mode. It sends USO requests from an AF_XDP socket
> and checks the datagrams received on a packet socket: the number of
> datagrams, the IP and UDP lengths, the checksums and the payload, for
> IPv4 and IPv6, and single and multi-buffer requests. It also checks
> that invalid requests are dropped and counted in tx_invalid_descs.
> 
> The test needs root and physical NICs with an MTU of at least 1448,
> either two connected ports (-i TX_IFACE -r RX_IFACE) or one in NIC
> loopback (-l -i IFACE). Virtual devices such as veth pass the packet on
> without segmenting it. Zero copy (-z) also needs a native XDP program on
> the interface. Standalone checksum offload cases run only with -C, as
> they test the existing checksum offload rather than USO. As it needs
> real hardware, it is built but not run by default, like xdp_hw_metadata.
> It uses the AF_XDP library in net/lib/xsk.

I assume the end goal for xdp_hw_metadata is to pull in its contents onto
xsk python runner. What is the case for this one?

Also since you're adding support to copy mode i assume veth runs would be
welcome...

Did you try to point AI to my set and ask whether it would be able to
produce python runner for that? Or do we don't want to have this CI'ed ?

> 
> Missing prerequisites are reported as skips. In zero-copy mode, the
> device must report xsk-zerocopy and tx-uso-zc, and checksum-only cases
> need tx-checksum. In both modes, cases that produce more than one
> datagram need tx-udp-segmentation, including the invalid ones, which
> would otherwise be dropped for lack of the offload whatever their
> headers. Cases that need more descriptors than xdp-zc-max-segs are
> skipped, and so are binds that the device refuses with EOPNOTSUPP.
> Receive-side UDP GRO (rx-udp-gro-forwarding, rx-gro-list) must be off,
> as it would merge the segments again before the packet socket sees them.
> USO with launch time must succeed in copy mode and be dropped in
> zero-copy mode.
> 
> Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
> ---
>  tools/testing/selftests/net/.gitignore |   1 +
>  tools/testing/selftests/net/Makefile   |  10 +-
>  tools/testing/selftests/net/xsk_uso.c  | 954 +++++++++++++++++++++++++
>  3 files changed, 964 insertions(+), 1 deletion(-)
>  create mode 100644 tools/testing/selftests/net/xsk_uso.c
> 

...

> +
> +static void get_mac(const char *name, unsigned char *mac)
> +{
> +	struct ifreq ifr = {};
> +	int fd = socket(AF_INET, SOCK_DGRAM, 0);
> +
> +	if (fd < 0)
> +		ksft_exit_fail_msg("socket: %s\n", strerror(errno));
> +	snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", name);
> +	if (ioctl(fd, SIOCGIFHWADDR, &ifr))
> +		ksft_exit_fail_msg("MAC address for %s: %s\n", name, strerror(errno));
> +	memcpy(mac, ifr.ifr_hwaddr.sa_data, ETH_ALEN);
> +	close(fd);
> +}
> +
> +static unsigned int get_mtu(const char *name)
> +{
> +	struct ifreq ifr = {};
> +	int fd = socket(AF_INET, SOCK_DGRAM, 0);
> +
> +	snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", name);
> +	if (fd < 0 || ioctl(fd, SIOCGIFMTU, &ifr))
> +		ksft_exit_fail_msg("MTU of %s: %s\n", name, strerror(errno));
> +	close(fd);
> +	return ifr.ifr_mtu;
> +}

this is a lib content which i'm also adding, so we're gonna clash.

> +
> +/* Virtual devices have no backing device, and pass GSO packets on whole. */
> +static bool is_physical(const char *name)
> +{
> +	char path[64];
> +
> +	snprintf(path, sizeof(path), "/sys/class/net/%s/device", name);
> +	return !access(path, F_OK);
> +}
> +
> +static int ethtool_ioctl(const char *name, void *cmd)
> +{
> +	struct ifreq ifr = { .ifr_data = cmd };
> +	int fd = socket(AF_INET, SOCK_DGRAM, 0), err;
> +
> +	if (fd < 0)
> +		return -1;
> +	snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", name);
> +	err = ioctl(fd, SIOCETHTOOL, &ifr);
> +	close(fd);
> +	return err;
> +}
> +
> +/* Returns whether the netdev feature called @feature is active on @name. */
> +static bool feature_active(const char *name, const char *feature)
> +{
> +	struct ethtool_gfeatures *features = NULL;
> +	struct ethtool_gstrings *strings = NULL;
> +	struct ethtool_sset_info *sset;
> +	unsigned int count, blocks, i;
> +	bool active = false;
> +
> +	sset = calloc(1, sizeof(*sset) + sizeof(sset->data[0]));
> +	if (!sset)
> +		ksft_exit_fail_msg("calloc: %s\n", strerror(errno));
> +	sset->cmd = ETHTOOL_GSSET_INFO;
> +	sset->sset_mask = 1ULL << ETH_SS_FEATURES;
> +	if (ethtool_ioctl(name, sset))
> +		ksft_exit_fail_msg("ETHTOOL_GSSET_INFO: %s\n", strerror(errno));
> +	if (!sset->sset_mask)
> +		ksft_exit_fail_msg("%s reports no feature strings\n", name);
> +	count = sset->data[0];
> +	blocks = (count + 31) / 32;
> +	strings = calloc(1, sizeof(*strings) + count * ETH_GSTRING_LEN);
> +	features = calloc(1, sizeof(*features) +
> +			  blocks * sizeof(features->features[0]));
> +	if (!strings || !features)
> +		ksft_exit_fail_msg("calloc: %s\n", strerror(errno));
> +	strings->cmd = ETHTOOL_GSTRINGS;
> +	strings->string_set = ETH_SS_FEATURES;
> +	strings->len = count;
> +	features->cmd = ETHTOOL_GFEATURES;
> +	features->size = blocks;
> +	if (ethtool_ioctl(name, strings) || ethtool_ioctl(name, features))
> +		ksft_exit_fail_msg("ethtool features: %s\n", strerror(errno));
> +	for (i = 0; i < count; i++)
> +		if (!strncmp((char *)strings->data + i * ETH_GSTRING_LEN, feature,
> +			     ETH_GSTRING_LEN)) {
> +			active = features->features[i / 32].active & (1U << (i % 32));
> +			break;
> +		}
> +	free(features);
> +	free(strings);
> +	free(sset);
> +	return active;
> +}
> +
> +static void get_xdp_features(unsigned int ifindex, uint64_t *xdp, uint64_t *xsk,
> +			     unsigned int *zc_max_segs)
> +{
> +	struct netdev_dev_get_req *req;
> +	struct netdev_dev_get_rsp *rsp;
> +	struct ynl_error yerr;
> +	struct ynl_sock *ys;
> +
> +	ys = ynl_sock_create(&ynl_netdev_family, &yerr);
> +	if (!ys)
> +		ksft_exit_fail_msg("YNL: %s\n", yerr.msg);
> +	req = netdev_dev_get_req_alloc();
> +	if (!req)
> +		ksft_exit_fail_msg("netdev_dev_get_req_alloc failed\n");
> +	netdev_dev_get_req_set_ifindex(req, ifindex);
> +	rsp = netdev_dev_get(ys, req);
> +	netdev_dev_get_req_free(req);
> +	if (!rsp)
> +		ksft_exit_fail_msg("netdev dev-get: %s\n", ys->err.msg);
> +	*xdp = rsp->_present.xdp_features ? rsp->xdp_features : 0;
> +	*xsk = rsp->_present.xsk_features ? rsp->xsk_features : 0;
> +	*zc_max_segs = rsp->_present.xdp_zc_max_segs ? rsp->xdp_zc_max_segs : 1;
> +	netdev_dev_get_rsp_free(rsp);
> +	ynl_sock_destroy(ys);

you really should look at python drv-net infra...

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2026-11-19 19:05 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 16:53 [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Magnus Karlsson
2026-11-19 16:53 ` [PATCH net-next 1/5] xsk: add USO support for skb path Magnus Karlsson
2026-11-19 18:47   ` Maciej Fijalkowski
2026-11-19 16:53 ` [PATCH net-next 2/5] ice: add Tx descriptor builders and header length check Magnus Karlsson
2026-11-19 16:53 ` [PATCH net-next 3/5] xsk: ice: add USO zero-copy support Magnus Karlsson
2026-11-19 16:53 ` [PATCH net-next 4/5] xsk: add KUnit tests for AF_XDP USO header helpers Magnus Karlsson
2026-11-19 16:53 ` [PATCH net-next 5/5] selftests: net: add AF_XDP USO test Magnus Karlsson
2026-11-19 19:05   ` Maciej Fijalkowski

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox