All of lore.kernel.org
 help / color / mirror / Atom feed
From: Paolo Abeni <pabeni@redhat.com>
To: Magnus Karlsson <magnus.karlsson@gmail.com>
Cc: Magnus Karlsson <magnus.karlsson@intel.com>,
	maciej.fijalkowski@intel.com, sdf@fomichev.me,
	davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
	horms@kernel.org, bjorn@kernel.org, kerneljasonxing@gmail.com,
	intel-wired-lan@lists.osuosl.org, anthony.l.nguyen@intel.com,
	przemyslaw.kitszel@intel.com, jacob.e.keller@intel.com
Subject: Re: [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP
Date: Thu, 19 Nov 2026 17:39:22 +0100	[thread overview]
Message-ID: <d2356153-be70-42ff-bfae-e766290e02a3@redhat.com> (raw)
In-Reply-To: <CAJ8uoz25cA_oEidTzwC_sjCH_0YMT2njXbCXwWfeAkMpEUDV8g@mail.gmail.com>

On 11/19/26 16:51, Magnus Karlsson wrote:
> On Thu, 19 Nov 2026 at 16:33, Paolo Abeni <pabeni@redhat.com> wrote:
>>
>> On 11/19/26 15:10, Magnus Karlsson wrote:
>>> This series adds UDP Segmentation Offload (USO) to AF_XDP. An
>>> application submits one Ethernet frame holding a single IP and UDP
>>> header followed by the aggregate UDP payload, and the device emits a
>>> train of MTU-sized UDP datagrams. Patch 1 implements it for copy mode
>>> (the skb path). Patch 2 is an ice preparation that adds Tx descriptor
>>> builders, and patch 3 implements USO for zero-copy mode in the ice
>>> driver.
>>>
>>> AF_XDP never segments these packets itself: a request is rejected
>>> unless the device offers USO. For the ice zero-copy path, the
>>> segmentation is done in HW. In copy mode, the packet is passed
>>> unsegmented to the bound device's driver, and as with UDP_SEGMENT
>>> sockets, how the driver carries out the segmentation is up to it.
>>> Stacked devices (VLAN, bonding, macvlan) forward the packet to a lower
>>> device that may segment it in SW.
>>>
>>> The design extends the existing Tx metadata support in AF_XDP. One new
>>> flag, XDP_TXMD_FLAGS_USO, is added, and the reserved u32 in the request
>>> part of struct xsk_tx_metadata is split into two u16 fields called
>>> gso_size and network_offset. Zero-copy support is advertised through
>>> the new tx-uso-zc netdev XSK feature.
>>>
>>> network_offset is part of the uAPI because csum_start only locates L4,
>>> while segmentation also needs the start of L3: the device rewrites the
>>> IP header of every datagram and needs the L2 and L3 lengths, and on
>>> the skb path it becomes the network header. With it, the kernel does
>>> not need to understand the link layer, and the fields can be reused
>>> later. Any future extension for tunnel segmentation would treat
>>> network_offset as the outer L3 offset and csum_start as the inner L4
>>> offset, adding only the outer L4 and inner L3 offsets.
>>>
>>> To request USO, bind the socket in XDP_COPY mode, or in XDP_ZEROCOPY
>>> mode on a driver advertising tx-uso-zc, and set XDP_TXMD_FLAGS_USO |
>>> XDP_TXMD_FLAGS_CHECKSUM in the Tx metadata. Reserve at least 16 bytes
>>> of metadata and set XDP_TX_METADATA on the first descriptor of the
>>> packet.
>>>
>>> Submit an Ethernet frame containing one IP header, one UDP header, and
>>> the aggregate UDP payload. Set network_offset to the IP header offset,
>>> csum_start to the UDP header offset, and csum_offset to the UDP
>>> checksum field offset (6 bytes). Header offsets are measured from the
>>> first descriptor's packet-data address, excluding metadata. Set
>>> gso_size to the number of UDP payload bytes per output datagram,
>>> excluding headers; the last datagram may of course be shorter.
>>>
>>> USO is only available on Ethernet devices. IPv4 headers may carry
>>> options but must not be fragmented. In IPv6, the UDP header must
>>> directly follow the fixed header; extension headers are not supported.
>>>
>>> The IP and UDP lengths must describe the aggregate packet. Initialize
>>> the UDP checksum field with the partial pseudoheader checksum seed for
>>> that packet. The device generates the per-datagram lengths and UDP
>>> checksums and, for IPv4 segmentation, the IP header checksums. You must
>>> supply a valid IPv4 header checksum as well, since a request that fits
>>> in one datagram uses ordinary checksum offload without segmentation.
>>>
>>> All headers through UDP must be in the first buffer. Multi-buffer
>>> packets use XDP_USE_SG and XDP_PKT_CONTD, with metadata only on the
>>> first descriptor. Each output IP packet must fit the device MTU, and
>>> requests remain subject to the device's segmentation limits. Query
>>> tx-udp-segmentation through ethtool to check whether skb UDP
>>> segmentation offload is enabled.
>>>
>>> USO cannot be combined with XDP_UMEM_TX_SW_CSUM or
>>> XDP_TXMD_FLAGS_TIMESTAMP. Launch time can be combined with USO in copy
>>> mode, but not in zero-copy mode as that has not been implemented on
>>> ice or tested on any other device for that matter.
>>>
>>> A USO request that is rejected is dropped: its buffers are completed
>>> and counted in tx_invalid_descs. This is the only indication of the
>>> failure, whatever its cause: invalid metadata or headers, the MTU,
>>> segmentation limits, or a device without the offload. Errors in other
>>> Tx metadata keep failing sendmsg() in copy mode, so the behaviour for
>>> existing non-USO users is unchanged. Zero-copy drivers without
>>> tx-uso-zc do not drop USO requests; they ignore the Tx metadata and
>>> send the frame unsegmented and without offloads. ice behaves the same
>>> way when it has fewer XDP Tx rings than CPUs, as queues then share XDP
>>> Tx rings. This can be supported in a follow up patch series as it
>>> would require quite some changes to the driver. Note that tx-uso-zc is
>>> still reported in this case, since it is a static driver capability.
>>>
>>> In both modes, requests that produce more than one datagram need
>>> tx-udp-segmentation to be enabled; tx-uso-zc only reports driver
>>> capability. In zero-copy mode, a packet may use at most
>>> xdp-zc-max-segs descriptors, and ice specifically allows at most eight
>>> buffers per output datagram, counting the copied header.
>>>
>>> The header validator works on a raw header copy instead of an skb, so
>>> that it can be shared by the skb and zero-copy paths. It lives in
>>> net/xdp/xsk_tx_uso.c and include/net/xsk_tx_uso.h. In zero-copy mode,
>>> ice copies the headers into kernel-owned memory before validating them
>>> and handing them to the NIC, while the payload stays in the UMEM. I
>>> tried finding an existing packet header validator that works on raw
>>> buffers that could be reused, but I could not find one. Therefore this
>>> small validator was added.
>>>
>>> Testing was done on an ice NIC in loopback mode with a USO selftest,
>>> in both copy and zero-copy mode. Patch 5 adds it as
>>> selftests/net/xsk_uso. It checks the segment count, lengths, checksums
>>> and payload of the received datagrams for IPv4 and IPv6, single and
>>> multi-buffer requests, and that invalid requests are rejected. Patch 4
>>> adds KUnit tests for the header validation helpers. Note that patch 5
>>> should really be integrated into either xskxceiver or Stan's
>>> xdp_hw_metadata tests. But as these are currently being modified to
>>> fit into selftests/net and maybe even merged, I kept the USO tests
>>> separate for now.
>>>
>>> The tables below show the UDP packet rate achieved by a single socket
>>> on my rather old 2.1 GHz Cascade Lake server with PCIe3, relative to
>>> no offloads at 1472 bytes. Each request carries a UDP payload that is
>>> segmented into 1472-byte datagrams, either in user space (no offloads,
>>> so UDP checksum is also performed in SW using the kernel's
>>> csum_partial() algorithm) or by the NIC (USO). The application and the
>>> queue's interrupt are pinned to the same core, and the socket is
>>> driven by preferred busy polling with napi_defer_hard_irqs=100 and
>>> gro_flush_timeout=2000000, so there are virtually no device
>>> interrupts. Each table's performance numbers are normalized to the
>>> performance for 1472 byte packets with no offloads in that table.
>>>
>>> SKB mode - write all packet data for each packet
>>>
>>> +---------------+-------------+---------------+
>>> | Payload bytes | No offloads | USO + HW csum |
>>> +---------------+-------------+---------------+
>>> |          1472 |       +0.0% |        +86.2% |
>>> |          4416 |       +2.2% |       +192.1% |
>>> |          7360 |       +2.6% |       +290.0% |
>>> |         10304 |       +2.4% |       +301.5% |
>>> |         13248 |       +2.4% |       +322.2% |
>>> |         16192 |       +2.1% |       +355.6% |
>>> +---------------+-------------+---------------+
>>>
>>> SKB mode - write only headers for each packet
>>>
>>> +---------------+-------------+---------------+
>>> | Payload bytes | No offloads | USO + HW csum |
>>> +---------------+-------------+---------------+
>>> |          1472 |       +0.0% |        +88.7% |
>>> |          4416 |       +2.3% |       +211.3% |
>>> |          7360 |       +2.5% |       +342.0% |
>>> |         10304 |       +2.6% |       +367.7% |
>>> |         13248 |       +2.8% |       +394.1% |
>>> |         16192 |       +2.8% |       +444.8% |
>>> +---------------+-------------+---------------+
>>>
>>> ZC mode - write all packet data for each packet
>>>
>>> +---------------+-------------+---------------+
>>> | Payload bytes | No offloads | USO + HW csum |
>>> +---------------+-------------+---------------+
>>> |          1472 |       +0.0% |       +185.6% |
>>> |          4416 |       +4.1% |       +279.6% |
>>> |          7360 |       +4.8% |       +332.1% |
>>> |         10304 |       +4.4% |       +325.5% |
>>> |         13248 |       +3.6% |       +333.4% |
>>> |         16192 |       +3.5% |       +339.1% |
>>> +---------------+-------------+---------------+
>>>
>>> ZC mode - write only headers for each packet
>>>
>>> +---------------+-------------+---------------+
>>> | Payload bytes | No offloads | USO + HW csum |
>>> +---------------+-------------+---------------+
>>> |          1472 |       +0.0% |       +163.9% |
>>> |          4416 |       +2.9% |       +260.0% |
>>> |          7360 |       +3.1% |       +291.9% |
>>> |         10304 |       +3.6% |       +303.9% |
>>> |         13248 |       +4.2% |       +307.6% |
>>> |         16192 |       +4.1% |       +315.8% |
>>> +---------------+-------------+---------------+
>>>
>>> At 1472 bytes the request fits in one datagram, so USO is not
>>> segmenting anything and the gain comes from the checksum offload
>>> alone. As soon as multiple UDP packets have to be produced on the
>>> wire, USO becomes really beneficial.
>>>
>>> For the zero-copy tests, my system seems to bottleneck at around 50
>>> Gbps (about 4 Mpps) with 7K payloads or above. The board only has x8
>>> lanes of PCIe3, though the slot is x16. In any case, there is a
>>> benefit to USO even on this system.
>>>
>>> Patch series:
>>> 1: Implements USO for skb path
>>> 2: Adds Tx descriptor builders and a header length check to ice
>>> 3: Implements USO for ice zero-copy driver
>>> 4: KUnit tests for the USO header helpers
>>> 5: selftests/net: AF_XDP USO functional test
>> You forgot to CC: netdev
> 
> Ha, that is what I get for being out of the game for a while! Should I
> send a v2 or just resend? Please advice.
It's not a big deal either way, but the ML has never seen the v1, so a
resend would be better, I think.

/P


  reply	other threads:[~2026-11-19 16:39 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 14:10 [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Magnus Karlsson
2026-11-19 14:10 ` [PATCH net-next 1/5] xsk: add USO support for skb path Magnus Karlsson
2026-11-19 14:10 ` [PATCH net-next 2/5] ice: add Tx descriptor builders and header length check Magnus Karlsson
2026-11-19 14:10 ` [PATCH net-next 3/5] xsk: ice: add USO zero-copy support Magnus Karlsson
2026-11-19 14:59   ` sashiko-bot
2026-11-19 14:10 ` [PATCH net-next 4/5] xsk: add KUnit tests for AF_XDP USO header helpers Magnus Karlsson
2026-11-19 14:10 ` [PATCH net-next 5/5] selftests: net: add AF_XDP USO test Magnus Karlsson
2026-11-19 15:33 ` [PATCH net-next 0/5] xsk: ice: USO support for AF_XDP Paolo Abeni
2026-11-19 15:51   ` Magnus Karlsson
2026-11-19 16:39     ` Paolo Abeni [this message]
2026-11-19 16:52       ` Magnus Karlsson
  -- strict thread matches above, loose matches on Subject: below --
2026-11-19 16:53 Magnus Karlsson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d2356153-be70-42ff-bfae-e766290e02a3@redhat.com \
    --to=pabeni@redhat.com \
    --cc=anthony.l.nguyen@intel.com \
    --cc=bjorn@kernel.org \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=horms@kernel.org \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=jacob.e.keller@intel.com \
    --cc=kerneljasonxing@gmail.com \
    --cc=kuba@kernel.org \
    --cc=maciej.fijalkowski@intel.com \
    --cc=magnus.karlsson@gmail.com \
    --cc=magnus.karlsson@intel.com \
    --cc=przemyslaw.kitszel@intel.com \
    --cc=sdf@fomichev.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.