Intel-Wired-Lan Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Paolo Abeni <pabeni@redhat.com>
To: Magnus Karlsson <magnus.karlsson@gmail.com>
Cc: Magnus Karlsson <magnus.karlsson@intel.com>,
	maciej.fijalkowski@intel.com, sdf@fomichev.me,
	davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
	horms@kernel.org, bjorn@kernel.org, kerneljasonxing@gmail.com,
	intel-wired-lan@lists.osuosl.org, anthony.l.nguyen@intel.com,
	przemyslaw.kitszel@intel.com, jacob.e.keller@intel.com
Subject: Re: [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP
Date: Thu, 19 Nov 2026 17:39:22 +0100	[thread overview]
Message-ID: <d2356153-be70-42ff-bfae-e766290e02a3@redhat.com> (raw)
In-Reply-To: <CAJ8uoz25cA_oEidTzwC_sjCH_0YMT2njXbCXwWfeAkMpEUDV8g@mail.gmail.com>

On 11/19/26 16:51, Magnus Karlsson wrote:
> On Thu, 19 Nov 2026 at 16:33, Paolo Abeni <pabeni@redhat.com> wrote:
>>
>> On 11/19/26 15:10, Magnus Karlsson wrote:
>>> This series adds UDP Segmentation Offload (USO) to AF_XDP. An
>>> application submits one Ethernet frame holding a single IP and UDP
>>> header followed by the aggregate UDP payload, and the device emits a
>>> train of MTU-sized UDP datagrams. Patch 1 implements it for copy mode
>>> (the skb path). Patch 2 is an ice preparation that adds Tx descriptor
>>> builders, and patch 3 implements USO for zero-copy mode in the ice
>>> driver.
>>>
>>> AF_XDP never segments these packets itself: a request is rejected
>>> unless the device offers USO. For the ice zero-copy path, the
>>> segmentation is done in HW. In copy mode, the packet is passed
>>> unsegmented to the bound device's driver, and as with UDP_SEGMENT
>>> sockets, how the driver carries out the segmentation is up to it.
>>> Stacked devices (VLAN, bonding, macvlan) forward the packet to a lower
>>> device that may segment it in SW.
>>>
>>> The design extends the existing Tx metadata support in AF_XDP. One new
>>> flag, XDP_TXMD_FLAGS_USO, is added, and the reserved u32 in the request
>>> part of struct xsk_tx_metadata is split into two u16 fields called
>>> gso_size and network_offset. Zero-copy support is advertised through
>>> the new tx-uso-zc netdev XSK feature.
>>>
>>> network_offset is part of the uAPI because csum_start only locates L4,
>>> while segmentation also needs the start of L3: the device rewrites the
>>> IP header of every datagram and needs the L2 and L3 lengths, and on
>>> the skb path it becomes the network header. With it, the kernel does
>>> not need to understand the link layer, and the fields can be reused
>>> later. Any future extension for tunnel segmentation would treat
>>> network_offset as the outer L3 offset and csum_start as the inner L4
>>> offset, adding only the outer L4 and inner L3 offsets.
>>>
>>> To request USO, bind the socket in XDP_COPY mode, or in XDP_ZEROCOPY
>>> mode on a driver advertising tx-uso-zc, and set XDP_TXMD_FLAGS_USO |
>>> XDP_TXMD_FLAGS_CHECKSUM in the Tx metadata. Reserve at least 16 bytes
>>> of metadata and set XDP_TX_METADATA on the first descriptor of the
>>> packet.
>>>
>>> Submit an Ethernet frame containing one IP header, one UDP header, and
>>> the aggregate UDP payload. Set network_offset to the IP header offset,
>>> csum_start to the UDP header offset, and csum_offset to the UDP
>>> checksum field offset (6 bytes). Header offsets are measured from the
>>> first descriptor's packet-data address, excluding metadata. Set
>>> gso_size to the number of UDP payload bytes per output datagram,
>>> excluding headers; the last datagram may of course be shorter.
>>>
>>> USO is only available on Ethernet devices. IPv4 headers may carry
>>> options but must not be fragmented. In IPv6, the UDP header must
>>> directly follow the fixed header; extension headers are not supported.
>>>
>>> The IP and UDP lengths must describe the aggregate packet. Initialize
>>> the UDP checksum field with the partial pseudoheader checksum seed for
>>> that packet. The device generates the per-datagram lengths and UDP
>>> checksums and, for IPv4 segmentation, the IP header checksums. You must
>>> supply a valid IPv4 header checksum as well, since a request that fits
>>> in one datagram uses ordinary checksum offload without segmentation.
>>>
>>> All headers through UDP must be in the first buffer. Multi-buffer
>>> packets use XDP_USE_SG and XDP_PKT_CONTD, with metadata only on the
>>> first descriptor. Each output IP packet must fit the device MTU, and
>>> requests remain subject to the device's segmentation limits. Query
>>> tx-udp-segmentation through ethtool to check whether skb UDP
>>> segmentation offload is enabled.
>>>
>>> USO cannot be combined with XDP_UMEM_TX_SW_CSUM or
>>> XDP_TXMD_FLAGS_TIMESTAMP. Launch time can be combined with USO in copy
>>> mode, but not in zero-copy mode as that has not been implemented on
>>> ice or tested on any other device for that matter.
>>>
>>> A USO request that is rejected is dropped: its buffers are completed
>>> and counted in tx_invalid_descs. This is the only indication of the
>>> failure, whatever its cause: invalid metadata or headers, the MTU,
>>> segmentation limits, or a device without the offload. Errors in other
>>> Tx metadata keep failing sendmsg() in copy mode, so the behaviour for
>>> existing non-USO users is unchanged. Zero-copy drivers without
>>> tx-uso-zc do not drop USO requests; they ignore the Tx metadata and
>>> send the frame unsegmented and without offloads. ice behaves the same
>>> way when it has fewer XDP Tx rings than CPUs, as queues then share XDP
>>> Tx rings. This can be supported in a follow up patch series as it
>>> would require quite some changes to the driver. Note that tx-uso-zc is
>>> still reported in this case, since it is a static driver capability.
>>>
>>> In both modes, requests that produce more than one datagram need
>>> tx-udp-segmentation to be enabled; tx-uso-zc only reports driver
>>> capability. In zero-copy mode, a packet may use at most
>>> xdp-zc-max-segs descriptors, and ice specifically allows at most eight
>>> buffers per output datagram, counting the copied header.
>>>
>>> The header validator works on a raw header copy instead of an skb, so
>>> that it can be shared by the skb and zero-copy paths. It lives in
>>> net/xdp/xsk_tx_uso.c and include/net/xsk_tx_uso.h. In zero-copy mode,
>>> ice copies the headers into kernel-owned memory before validating them
>>> and handing them to the NIC, while the payload stays in the UMEM. I
>>> tried finding an existing packet header validator that works on raw
>>> buffers that could be reused, but I could not find one. Therefore this
>>> small validator was added.
>>>
>>> Testing was done on an ice NIC in loopback mode with a USO selftest,
>>> in both copy and zero-copy mode. Patch 5 adds it as
>>> selftests/net/xsk_uso. It checks the segment count, lengths, checksums
>>> and payload of the received datagrams for IPv4 and IPv6, single and
>>> multi-buffer requests, and that invalid requests are rejected. Patch 4
>>> adds KUnit tests for the header validation helpers. Note that patch 5
>>> should really be integrated into either xskxceiver or Stan's
>>> xdp_hw_metadata tests. But as these are currently being modified to
>>> fit into selftests/net and maybe even merged, I kept the USO tests
>>> separate for now.
>>>
>>> The tables below show the UDP packet rate achieved by a single socket
>>> on my rather old 2.1 GHz Cascade Lake server with PCIe3, relative to
>>> no offloads at 1472 bytes. Each request carries a UDP payload that is
>>> segmented into 1472-byte datagrams, either in user space (no offloads,
>>> so UDP checksum is also performed in SW using the kernel's
>>> csum_partial() algorithm) or by the NIC (USO). The application and the
>>> queue's interrupt are pinned to the same core, and the socket is
>>> driven by preferred busy polling with napi_defer_hard_irqs=100 and
>>> gro_flush_timeout=2000000, so there are virtually no device
>>> interrupts. Each table's performance numbers are normalized to the
>>> performance for 1472 byte packets with no offloads in that table.
>>>
>>> SKB mode - write all packet data for each packet
>>>
>>> +---------------+-------------+---------------+
>>> | Payload bytes | No offloads | USO + HW csum |
>>> +---------------+-------------+---------------+
>>> |          1472 |       +0.0% |        +86.2% |
>>> |          4416 |       +2.2% |       +192.1% |
>>> |          7360 |       +2.6% |       +290.0% |
>>> |         10304 |       +2.4% |       +301.5% |
>>> |         13248 |       +2.4% |       +322.2% |
>>> |         16192 |       +2.1% |       +355.6% |
>>> +---------------+-------------+---------------+
>>>
>>> SKB mode - write only headers for each packet
>>>
>>> +---------------+-------------+---------------+
>>> | Payload bytes | No offloads | USO + HW csum |
>>> +---------------+-------------+---------------+
>>> |          1472 |       +0.0% |        +88.7% |
>>> |          4416 |       +2.3% |       +211.3% |
>>> |          7360 |       +2.5% |       +342.0% |
>>> |         10304 |       +2.6% |       +367.7% |
>>> |         13248 |       +2.8% |       +394.1% |
>>> |         16192 |       +2.8% |       +444.8% |
>>> +---------------+-------------+---------------+
>>>
>>> ZC mode - write all packet data for each packet
>>>
>>> +---------------+-------------+---------------+
>>> | Payload bytes | No offloads | USO + HW csum |
>>> +---------------+-------------+---------------+
>>> |          1472 |       +0.0% |       +185.6% |
>>> |          4416 |       +4.1% |       +279.6% |
>>> |          7360 |       +4.8% |       +332.1% |
>>> |         10304 |       +4.4% |       +325.5% |
>>> |         13248 |       +3.6% |       +333.4% |
>>> |         16192 |       +3.5% |       +339.1% |
>>> +---------------+-------------+---------------+
>>>
>>> ZC mode - write only headers for each packet
>>>
>>> +---------------+-------------+---------------+
>>> | Payload bytes | No offloads | USO + HW csum |
>>> +---------------+-------------+---------------+
>>> |          1472 |       +0.0% |       +163.9% |
>>> |          4416 |       +2.9% |       +260.0% |
>>> |          7360 |       +3.1% |       +291.9% |
>>> |         10304 |       +3.6% |       +303.9% |
>>> |         13248 |       +4.2% |       +307.6% |
>>> |         16192 |       +4.1% |       +315.8% |
>>> +---------------+-------------+---------------+
>>>
>>> At 1472 bytes the request fits in one datagram, so USO is not
>>> segmenting anything and the gain comes from the checksum offload
>>> alone. As soon as multiple UDP packets have to be produced on the
>>> wire, USO becomes really beneficial.
>>>
>>> For the zero-copy tests, my system seems to bottleneck at around 50
>>> Gbps (about 4 Mpps) with 7K payloads or above. The board only has x8
>>> lanes of PCIe3, though the slot is x16. In any case, there is a
>>> benefit to USO even on this system.
>>>
>>> Patch series:
>>> 1: Implements USO for skb path
>>> 2: Adds Tx descriptor builders and a header length check to ice
>>> 3: Implements USO for ice zero-copy driver
>>> 4: KUnit tests for the USO header helpers
>>> 5: selftests/net: AF_XDP USO functional test
>> You forgot to CC: netdev
> 
> Ha, that is what I get for being out of the game for a while! Should I
> send a v2 or just resend? Please advice.
It's not a big deal either way, but the ML has never seen the v1, so a
resend would be better, I think.

/P


  reply	other threads:[~2026-11-19 16:39 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 14:10 [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Magnus Karlsson
2026-11-19 14:10 ` [PATCH net-next 1/5] xsk: add USO support for skb path Magnus Karlsson
2026-11-19 14:10 ` [PATCH net-next 2/5] ice: add Tx descriptor builders and header length check Magnus Karlsson
2026-11-19 14:10 ` [PATCH net-next 3/5] xsk: ice: add USO zero-copy support Magnus Karlsson
2026-11-19 14:59   ` sashiko-bot
2026-11-19 14:10 ` [PATCH net-next 4/5] xsk: add KUnit tests for AF_XDP USO header helpers Magnus Karlsson
2026-11-19 14:10 ` [PATCH net-next 5/5] selftests: net: add AF_XDP USO test Magnus Karlsson
2026-11-19 15:33 ` [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Paolo Abeni
2026-11-19 15:51   ` Magnus Karlsson
2026-11-19 16:39     ` Paolo Abeni [this message]
2026-11-19 16:52       ` Magnus Karlsson
  -- strict thread matches above, loose matches on Subject: below --
2026-11-19 16:53 Magnus Karlsson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d2356153-be70-42ff-bfae-e766290e02a3@redhat.com \
    --to=pabeni@redhat.com \
    --cc=anthony.l.nguyen@intel.com \
    --cc=bjorn@kernel.org \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=horms@kernel.org \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=jacob.e.keller@intel.com \
    --cc=kerneljasonxing@gmail.com \
    --cc=kuba@kernel.org \
    --cc=maciej.fijalkowski@intel.com \
    --cc=magnus.karlsson@gmail.com \
    --cc=magnus.karlsson@intel.com \
    --cc=przemyslaw.kitszel@intel.com \
    --cc=sdf@fomichev.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox