[PATCH iwl-next 0/3] iavf: defer loading netmem_desc

[PATCH iwl-next 0/3] iavf: defer loading netmem_desc

[Matt Vollrath]

In a few places libeth and the iavf driver inspect which page pool is
attached to a netmem_ref. This forces an immediate load of the
netmem_desc struct for each buffer in the Rx loop.

Defer or eliminate these loads of netmem_desc from the driver fast path
by using the page_pool ref in the first cache line of iavf_ring instead.

There are only two paths out of the Rx loop where netmem_desc needs to
be consumed:
* The "very rare" case of libeth_rx_sync_for_cpu calling
  libeth_rx_recycle_slow and indicating that there was no data. This
  could be similarly factored out, but not by this series.
* GRO merging a frame into an existing aggregate stream. In this case,
  the cold load of netmem_desc may overlap the payload prefetch started
  by iavf_build_skb, which is now no longer dependent on netmem_desc to
  start.

Matt Vollrath (3):
  libeth: add __libeth_rx_sync_for_cpu
  iavf: use __libeth_rx_sync_for_cpu
  iavf: use cached page_pool ref in skb helpers

 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 18 ++++++----
 include/net/libeth/rx.h                     | 37 +++++++++++++++------
 2 files changed, 37 insertions(+), 18 deletions(-)


base-commit: 5c458073553f0ef74f5c8db1bd459c87c722a299
-- 
2.43.0

[PATCH iwl-next 1/3] libeth: add __libeth_rx_sync_for_cpu

[Matt Vollrath]

Deriving a netmem's page pool from the ref requires loading a pointer
from the netmem_desc, which may not be cached. The driver may already
have a cached reference to the correct page pool, so add a bypass.

No functional change for existing callers.

Signed-off-by: Matt Vollrath <tactii@gmail.com>
Assisted-by: Claude:claude-5-fable
---
 include/net/libeth/rx.h | 37 ++++++++++++++++++++++++++-----------
 1 file changed, 26 insertions(+), 11 deletions(-)

diff --git a/include/net/libeth/rx.h b/include/net/libeth/rx.h
index 5d991404845e..fddac084e001 100644
--- a/include/net/libeth/rx.h
+++ b/include/net/libeth/rx.h
@@ -118,36 +118,51 @@ static inline dma_addr_t libeth_rx_alloc(const struct libeth_fq_fp *fq, u32 i)
 void libeth_rx_recycle_slow(netmem_ref netmem);
 
 /**
- * libeth_rx_sync_for_cpu - synchronize or recycle buffer post DMA
+ * __libeth_rx_sync_for_cpu - synchronize or recycle buffer post DMA
+ * @pool: &page_pool the buffer was allocated from
  * @fqe: buffer to process
  * @len: frame length from the descriptor
  *
- * Process the buffer after it's written by HW. The regular path is to
- * synchronize DMA for CPU, but in case of no data it will be immediately
- * recycled back to its PP.
+ * Variant for callers which already know the buffer's pool, typically
+ * their Rx queue's fill queue pool.
  *
  * Return: true when there's data to process, false otherwise.
  */
-static inline bool libeth_rx_sync_for_cpu(const struct libeth_fqe *fqe,
-					  u32 len)
+static inline bool __libeth_rx_sync_for_cpu(const struct page_pool *pool,
+					    const struct libeth_fqe *fqe,
+					    u32 len)
 {
-	netmem_ref netmem = fqe->netmem;
-
 	/* Very rare, but possible case. The most common reason:
 	 * the last fragment contained FCS only, which was then
 	 * stripped by the HW.
 	 */
 	if (unlikely(!len)) {
-		libeth_rx_recycle_slow(netmem);
+		libeth_rx_recycle_slow(fqe->netmem);
 		return false;
 	}
 
-	page_pool_dma_sync_netmem_for_cpu(netmem_get_pp(netmem), netmem,
-					  fqe->offset, len);
+	page_pool_dma_sync_netmem_for_cpu(pool, fqe->netmem, fqe->offset, len);
 
 	return true;
 }
 
+/**
+ * libeth_rx_sync_for_cpu - synchronize or recycle buffer post DMA
+ * @fqe: buffer to process
+ * @len: frame length from the descriptor
+ *
+ * Process the buffer after it's written by HW. The regular path is to
+ * synchronize DMA for CPU, but in case of no data it will be immediately
+ * recycled back to its PP.
+ *
+ * Return: true when there's data to process, false otherwise.
+ */
+static inline bool libeth_rx_sync_for_cpu(const struct libeth_fqe *fqe,
+					  u32 len)
+{
+	return __libeth_rx_sync_for_cpu(netmem_get_pp(fqe->netmem), fqe, len);
+}
+
 /* Converting abstract packet type numbers into a software structure with
  * the packet parameters to do O(1) lookup on Rx.
  */
-- 
2.43.0

[PATCH iwl-next 2/3] iavf: use __libeth_rx_sync_for_cpu

[Matt Vollrath]

The Rx ring already has a reference to its page pool adjacent to fields
touched every iteration. Avoid loading the netmem_desc for sync by
taking the shortcut.

Signed-off-by: Matt Vollrath <tactii@gmail.com>
Assisted-by: Claude:claude-5-fable
---
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/intel/iavf/iavf_txrx.c b/drivers/net/ethernet/intel/iavf/iavf_txrx.c
index c30abf17cf5d..3ae2f0a0ee4d 100644
--- a/drivers/net/ethernet/intel/iavf/iavf_txrx.c
+++ b/drivers/net/ethernet/intel/iavf/iavf_txrx.c
@@ -1387,6 +1387,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 {
 	bool flex = rx_ring->rxdid == VIRTCHNL_RXDID_2_FLEX_SQ_NIC;
 	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
+	const struct page_pool *pp = rx_ring->pp;
 	struct sk_buff *skb = rx_ring->skb;
 	u16 cleaned_count = IAVF_DESC_UNUSED(rx_ring);
 	bool failure = false;
@@ -1424,7 +1425,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 		iavf_trace(clean_rx_irq, rx_ring, rx_desc, skb);
 
 		rx_buffer = &rx_ring->rx_fqes[rx_ring->next_to_clean];
-		if (!libeth_rx_sync_for_cpu(rx_buffer, fields.len))
+		if (!__libeth_rx_sync_for_cpu(pp, rx_buffer, fields.len))
 			goto skip_data;
 
 		/* retrieve a buffer from the ring */
-- 
2.43.0

[PATCH iwl-next 3/3] iavf: use cached page_pool ref in skb helpers

[Matt Vollrath]

Deriving a reference to the page pool from netmem_desc requires loading
the struct. We already have a cached reference to the pool, so use that
instead.

Signed-off-by: Matt Vollrath <tactii@gmail.com>
Assisted-by: Claude:claude-5-fable
---
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/intel/iavf/iavf_txrx.c b/drivers/net/ethernet/intel/iavf/iavf_txrx.c
index 3ae2f0a0ee4d..9dc5f0761d8b 100644
--- a/drivers/net/ethernet/intel/iavf/iavf_txrx.c
+++ b/drivers/net/ethernet/intel/iavf/iavf_txrx.c
@@ -1184,6 +1184,7 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)
 
 /**
  * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff
+ * @pp: page pool the buffer was allocated from
  * @skb: sk_buff to place the data into
  * @rx_buffer: buffer containing page to add
  * @size: packet length from rx_desc
@@ -1193,11 +1194,11 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)
  *
  * The function will then update the page offset.
  **/
-static void iavf_add_rx_frag(struct sk_buff *skb,
+static void iavf_add_rx_frag(const struct page_pool *pp, struct sk_buff *skb,
 			     const struct libeth_fqe *rx_buffer,
 			     unsigned int size)
 {
-	u32 hr = netmem_get_pp(rx_buffer->netmem)->p.offset;
+	u32 hr = pp->p.offset;
 
 	skb_add_rx_frag_netmem(skb, skb_shinfo(skb)->nr_frags,
 			       rx_buffer->netmem, rx_buffer->offset + hr,
@@ -1206,17 +1207,19 @@ static void iavf_add_rx_frag(struct sk_buff *skb,
 
 /**
  * iavf_build_skb - Build skb around an existing buffer
+ * @pp: page pool the buffer was allocated from
  * @rx_buffer: Rx buffer to pull data from
  * @size: size of buffer to add to skb
  *
  * This function builds an skb around an existing Rx buffer, taking care
  * to set up the skb correctly and avoid any memcpy overhead.
  */
-static struct sk_buff *iavf_build_skb(const struct libeth_fqe *rx_buffer,
+static struct sk_buff *iavf_build_skb(const struct page_pool *pp,
+				      const struct libeth_fqe *rx_buffer,
 				      unsigned int size)
 {
 	struct page *buf_page = __netmem_to_page(rx_buffer->netmem);
-	u32 hr = pp_page_to_nmdesc(buf_page)->pp->p.offset;
+	u32 hr = pp->p.offset;
 	struct sk_buff *skb;
 	void *va;
 
@@ -1430,9 +1433,9 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 
 		/* retrieve a buffer from the ring */
 		if (skb)
-			iavf_add_rx_frag(skb, rx_buffer, fields.len);
+			iavf_add_rx_frag(pp, skb, rx_buffer, fields.len);
 		else
-			skb = iavf_build_skb(rx_buffer, fields.len);
+			skb = iavf_build_skb(pp, rx_buffer, fields.len);
 
 		/* exit if we failed to retrieve a buffer */
 		if (!skb) {
-- 
2.43.0

RE: [Intel-wired-lan] [PATCH iwl-next 1/3] libeth: add __libeth_rx_sync_for_cpu

[Loktionov, Aleksandr]

> -----Original Message-----
> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
> Of Matt Vollrath
> Sent: Friday, July 31, 2026 2:41 PM
> To: intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; Lobakin, Aleksander
> <aleksander.lobakin@intel.com>; Nguyen, Anthony L
> <anthony.l.nguyen@intel.com>; Kitszel, Przemyslaw
> <przemyslaw.kitszel@intel.com>; Andrew Lunn <andrew+netdev@lunn.ch>;
> David S . Miller <davem@davemloft.net>; Eric Dumazet
> <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni
> <pabeni@redhat.com>; Simon Horman <horms@kernel.org>; Matt Vollrath
> <tactii@gmail.com>
> Subject: [Intel-wired-lan] [PATCH iwl-next 1/3] libeth: add
> __libeth_rx_sync_for_cpu
> 
> Deriving a netmem's page pool from the ref requires loading a pointer
> from the netmem_desc, which may not be cached. The driver may already
> have a cached reference to the correct page pool, so add a bypass.
> 
> No functional change for existing callers.
> 
> Signed-off-by: Matt Vollrath <tactii@gmail.com>
> Assisted-by: Claude:claude-5-fable
> ---
>  include/net/libeth/rx.h | 37 ++++++++++++++++++++++++++-----------
>  1 file changed, 26 insertions(+), 11 deletions(-)
> 
> diff --git a/include/net/libeth/rx.h b/include/net/libeth/rx.h index
> 5d991404845e..fddac084e001 100644
> --- a/include/net/libeth/rx.h
> +++ b/include/net/libeth/rx.h
> @@ -118,36 +118,51 @@ static inline dma_addr_t libeth_rx_alloc(const
> struct libeth_fq_fp *fq, u32 i)  void
> libeth_rx_recycle_slow(netmem_ref netmem);
> 
>  /**

...

>  /* Converting abstract packet type numbers into a software structure
> with
>   * the packet parameters to do O(1) lookup on Rx.
>   */
> --
> 2.43.0

Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>

RE: [Intel-wired-lan] [PATCH iwl-next 2/3] iavf: use __libeth_rx_sync_for_cpu

[Loktionov, Aleksandr]

> -----Original Message-----
> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
> Of Matt Vollrath
> Sent: Friday, July 31, 2026 2:41 PM
> To: intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; Lobakin, Aleksander
> <aleksander.lobakin@intel.com>; Nguyen, Anthony L
> <anthony.l.nguyen@intel.com>; Kitszel, Przemyslaw
> <przemyslaw.kitszel@intel.com>; Andrew Lunn <andrew+netdev@lunn.ch>;
> David S . Miller <davem@davemloft.net>; Eric Dumazet
> <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni
> <pabeni@redhat.com>; Simon Horman <horms@kernel.org>; Matt Vollrath
> <tactii@gmail.com>
> Subject: [Intel-wired-lan] [PATCH iwl-next 2/3] iavf: use
> __libeth_rx_sync_for_cpu
> 
> The Rx ring already has a reference to its page pool adjacent to
> fields touched every iteration. Avoid loading the netmem_desc for sync
> by taking the shortcut.
> 
> Signed-off-by: Matt Vollrath <tactii@gmail.com>
> Assisted-by: Claude:claude-5-fable
> ---
>  drivers/net/ethernet/intel/iavf/iavf_txrx.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/intel/iavf/iavf_txrx.c
> b/drivers/net/ethernet/intel/iavf/iavf_txrx.c
> index c30abf17cf5d..3ae2f0a0ee4d 100644
> --- a/drivers/net/ethernet/intel/iavf/iavf_txrx.c
> +++ b/drivers/net/ethernet/intel/iavf/iavf_txrx.c
> @@ -1387,6 +1387,7 @@ static int iavf_clean_rx_irq(struct iavf_ring
> *rx_ring, int budget)  {
>  	bool flex = rx_ring->rxdid == VIRTCHNL_RXDID_2_FLEX_SQ_NIC;
>  	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
> +	const struct page_pool *pp = rx_ring->pp;
>  	struct sk_buff *skb = rx_ring->skb;
>  	u16 cleaned_count = IAVF_DESC_UNUSED(rx_ring);
>  	bool failure = false;
> @@ -1424,7 +1425,7 @@ static int iavf_clean_rx_irq(struct iavf_ring
> *rx_ring, int budget)
>  		iavf_trace(clean_rx_irq, rx_ring, rx_desc, skb);
> 
>  		rx_buffer = &rx_ring->rx_fqes[rx_ring->next_to_clean];
> -		if (!libeth_rx_sync_for_cpu(rx_buffer, fields.len))
> +		if (!__libeth_rx_sync_for_cpu(pp, rx_buffer,
> fields.len))
>  			goto skip_data;
> 
>  		/* retrieve a buffer from the ring */
> --
> 2.43.0

Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>

RE: [Intel-wired-lan] [PATCH iwl-next 3/3] iavf: use cached page_pool ref in skb helpers

[Loktionov, Aleksandr]

> -----Original Message-----
> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
> Of Matt Vollrath
> Sent: Friday, July 31, 2026 2:41 PM
> To: intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; Lobakin, Aleksander
> <aleksander.lobakin@intel.com>; Nguyen, Anthony L
> <anthony.l.nguyen@intel.com>; Kitszel, Przemyslaw
> <przemyslaw.kitszel@intel.com>; Andrew Lunn <andrew+netdev@lunn.ch>;
> David S . Miller <davem@davemloft.net>; Eric Dumazet
> <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni
> <pabeni@redhat.com>; Simon Horman <horms@kernel.org>; Matt Vollrath
> <tactii@gmail.com>
> Subject: [Intel-wired-lan] [PATCH iwl-next 3/3] iavf: use cached
> page_pool ref in skb helpers
> 
> Deriving a reference to the page pool from netmem_desc requires
> loading the struct. We already have a cached reference to the pool, so
> use that instead.
> 
> Signed-off-by: Matt Vollrath <tactii@gmail.com>
> Assisted-by: Claude:claude-5-fable
> ---
>  drivers/net/ethernet/intel/iavf/iavf_txrx.c | 15 +++++++++------
>  1 file changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/iavf/iavf_txrx.c
> b/drivers/net/ethernet/intel/iavf/iavf_txrx.c
> index 3ae2f0a0ee4d..9dc5f0761d8b 100644
> --- a/drivers/net/ethernet/intel/iavf/iavf_txrx.c
> +++ b/drivers/net/ethernet/intel/iavf/iavf_txrx.c
> @@ -1184,6 +1184,7 @@ static bool iavf_cleanup_headers(struct
> iavf_ring *rx_ring, struct sk_buff *skb)
> 
>  /**
>   * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff
> + * @pp: page pool the buffer was allocated from
>   * @skb: sk_buff to place the data into
>   * @rx_buffer: buffer containing page to add
>   * @size: packet length from rx_desc
> @@ -1193,11 +1194,11 @@ static bool iavf_cleanup_headers(struct
> iavf_ring *rx_ring, struct sk_buff *skb)
>   *
>   * The function will then update the page offset.
>   **/
> -static void iavf_add_rx_frag(struct sk_buff *skb,
> +static void iavf_add_rx_frag(const struct page_pool *pp, struct
> sk_buff
> +*skb,
>  			     const struct libeth_fqe *rx_buffer,
>  			     unsigned int size)
>  {
> -	u32 hr = netmem_get_pp(rx_buffer->netmem)->p.offset;
> +	u32 hr = pp->p.offset;
> 
>  	skb_add_rx_frag_netmem(skb, skb_shinfo(skb)->nr_frags,
>  			       rx_buffer->netmem, rx_buffer->offset + hr,
> @@ -1206,17 +1207,19 @@ static void iavf_add_rx_frag(struct sk_buff
> *skb,
> 
>  /**
>   * iavf_build_skb - Build skb around an existing buffer
> + * @pp: page pool the buffer was allocated from
>   * @rx_buffer: Rx buffer to pull data from
>   * @size: size of buffer to add to skb
>   *
>   * This function builds an skb around an existing Rx buffer, taking
> care
>   * to set up the skb correctly and avoid any memcpy overhead.
>   */
> -static struct sk_buff *iavf_build_skb(const struct libeth_fqe
> *rx_buffer,
> +static struct sk_buff *iavf_build_skb(const struct page_pool *pp,
> +				      const struct libeth_fqe *rx_buffer,
>  				      unsigned int size)
>  {
>  	struct page *buf_page = __netmem_to_page(rx_buffer->netmem);
> -	u32 hr = pp_page_to_nmdesc(buf_page)->pp->p.offset;
> +	u32 hr = pp->p.offset;
>  	struct sk_buff *skb;
>  	void *va;
> 
> @@ -1430,9 +1433,9 @@ static int iavf_clean_rx_irq(struct iavf_ring
> *rx_ring, int budget)
> 
>  		/* retrieve a buffer from the ring */
>  		if (skb)
> -			iavf_add_rx_frag(skb, rx_buffer, fields.len);
> +			iavf_add_rx_frag(pp, skb, rx_buffer, fields.len);
>  		else
> -			skb = iavf_build_skb(rx_buffer, fields.len);
> +			skb = iavf_build_skb(pp, rx_buffer, fields.len);
> 
>  		/* exit if we failed to retrieve a buffer */
>  		if (!skb) {
> --
> 2.43.0

Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>

Re: [PATCH iwl-next 0/3] iavf: defer loading netmem_desc

[Alexander Lobakin]

From: Matt Vollrath <tactii@gmail.com>
Date: Fri, 31 Jul 2026 08:41:06 -0400

> In a few places libeth and the iavf driver inspect which page pool is
> attached to a netmem_ref. This forces an immediate load of the
> netmem_desc struct for each buffer in the Rx loop.
> 
> Defer or eliminate these loads of netmem_desc from the driver fast path
> by using the page_pool ref in the first cache line of iavf_ring instead.

I was thinking of this when implementing the current design, but:

Do you have any data to prove this actually helps performance?

> 
> There are only two paths out of the Rx loop where netmem_desc needs to
> be consumed:
> * The "very rare" case of libeth_rx_sync_for_cpu calling
>   libeth_rx_recycle_slow and indicating that there was no data. This
>   could be similarly factored out, but not by this series.
> * GRO merging a frame into an existing aggregate stream. In this case,
>   the cold load of netmem_desc may overlap the payload prefetch started
>   by iavf_build_skb, which is now no longer dependent on netmem_desc to
>   start.

Thanks,
Olek

Re: [PATCH iwl-next 0/3] iavf: defer loading netmem_desc

[Matt Vollrath]

On 7/31/26 11:13, Alexander Lobakin wrote:
> From: Matt Vollrath <tactii@gmail.com>
> Date: Fri, 31 Jul 2026 08:41:06 -0400
> 
>> In a few places libeth and the iavf driver inspect which page pool is
>> attached to a netmem_ref. This forces an immediate load of the
>> netmem_desc struct for each buffer in the Rx loop.
>>
>> Defer or eliminate these loads of netmem_desc from the driver fast path
>> by using the page_pool ref in the first cache line of iavf_ring instead.
> 
> I was thinking of this when implementing the current design, but:
> 
> Do you have any data to prove this actually helps performance?

The optimization was profile-driven, but not on an iavf. I was working on
a libeth port of e1000e, because I'm an obsessive optimizer, and this quirk
of the interface fell out. I'm working with a Xeon E3-1240 v3 and I218-V.

With a mirror of the changes in this patch series, on my platform, it's a
net zero overall change in Rx performance. This is not surprising, because
the cold read is just deferred until later. The time saved in
eth_type_trans (where the payload is first read) is instead spent in
skb_gro_receive and pool puts.

The additional thing that does make an improvement, which was omitted from
this series, is prefetching netmem_desc early in the loop. This makes about
22 cycles/packet, 3% difference.

I left it where it is because I don't have hardware to perf this on, and
didn't know if anyone would even be interested in looking at something that
I can't produce data for on the actual hardware. For iavf the prefetch
would be conditional, only when the EOP flag is set, because in this case
netmem_desc will always be needed either for GRO merge or pool put.

I can't guarantee that this will be worth 3% on an iavf platform. In any
case, it'll be a couple weeks before I can work on this again.

> 
>>
>> There are only two paths out of the Rx loop where netmem_desc needs to
>> be consumed:
>> * The "very rare" case of libeth_rx_sync_for_cpu calling
>>    libeth_rx_recycle_slow and indicating that there was no data. This
>>    could be similarly factored out, but not by this series.
>> * GRO merging a frame into an existing aggregate stream. In this case,
>>    the cold load of netmem_desc may overlap the payload prefetch started
>>    by iavf_build_skb, which is now no longer dependent on netmem_desc to
>>    start.

I see now this isn't right, anything going to GRO will end up back in the
pool in the same iteration, and netmem_desc will be loaded. It just
happens later.

> 
> Thanks,
> Olek