Re: igc (i226-V): persistent RX-path cost regression after rx buffer-exhaustion storms

Re: igc (i226-V): persistent RX-path cost regression after rx buffer-exhaustion storms

[Ruinskiy, Dima]

[-- Attachment #1: Type: text/plain, Size: 4200 bytes --]

On 14/10/2026 19:02, Frank Zhang wrote:
> Hello,
> 
> We run a native-XDP forwarding datapath (an eBPF GTP-U user plane) on an
> Intel N100 mini-PC with two i226-V NICs (in-tree igc, firmware 2023:889d,
> 2.5GBASE-T; Ubuntu 24.04, 7.0.0-38-generic, iommu=pt). Small-packet
> capacity is ~2.6 Mpps/port here. We found that after the port is
> over-offered for a few seconds, the RX path keeps a persistent per-packet
> cost regression after the offered load is reduced back to a clean level.
> It never recovers on its own. A comparable storm-and-return test on a
> second box with an ixgbe NIC did not leave a persistent regression, so we
> would appreciate a look from the igc side.

Hello, Frank.
Thank you very much for this report and the in-depth analysis.

Your description makes me think that it is probably not a hardware 
limitation, but more like some inefficiency in the allocator, or 
specifically how the igc driver uses it.

> During the storm __page_frag_cache_drain runs 0.45-0.73M times per 15 s;
> storms that do not trigger this recycling leave layout and performance
> unchanged. Our reading: storm -> buffer recycling churn -> refill from
> fragmented blocks -> scattered ring pages -> per-packet page-metadata
> misses.
> 
> We are not claiming this is igc-exclusive -- ixgbe shares a similar
> RX-buffer lifetime shape; the X520 result above just means we cannot say
> the effect is universal either.

This observation suggests that there may be some secondary factors at 
play. I confess that I am not an expert in the details of how this 
allocations and recycling works and what may affect it. I consulted my 
AI agent, which suggested to add a bit of instrumentation to see which 
places in the code dealing with this are hit, and when.

Maybe this duplicates some of the data you already collected, but just 
in case it will help, the instrumentation patch is attached, with the 
following advice:

Apply 0001-igc-add-rx-page-lifecycle-instrumentation.patch (need to use 
patch, not git am), and collect ethtool -S before storm, during storm, 
and after recovery attempts.

> 
> 3) Recovery (measured)
> 
>    ip link down/up, ethtool -r, FLR, compact_memory: no effect
>    rmmod/modprobe x1: partial
>    rmmod/modprobe x2-3: near baseline, stable
>    reboot: full
>    (local workaround: force fresh ring allocations at refill: full)
Interesting observation. Could you elaborate more on this "local 
workaround: force fresh ring allocations at refill" which successfully 
recovers the performance? How was it implemented? Do you also have a 
patch to share?

> 
> 4) Related observation (happy to split out or drop)
> 
> The default RSS indirection table is block-interleaved
> (igc_setup_mrqc(): tbl[j] = (j * queues) / IGC_RETA_SIZE). With
> correlated tunnel flows (GTP-U flows differing only in low tuple bits)
> and a per-boot random RSS key, we saw the whole workload collapse into
> one 32-bucket block -> a single RX queue (2 cores saturated, capacity
> ~halved). ethtool -X equal N restores the spread; we apply that as a
> workaround. ixgbe initialises this table interleaved. Flagging because it
> materially lowers this box's small-packet ceiling and makes the storm
> above easier to reach. (Also: RSS key get/set was merged in July 2026 --
> thanks; our distro kernel predates it.)
This might be worth to tackle as a separate patch. If ixgbe's default 
initialization is more optimal, then we can consider updating igc (maybe 
also igb, which I believe is similar) to use a similar approach.

> 
> 5) Ask
> 
> - Is this persistent post-storm state known for igc/I226? Any recommended
>    mitigation?
Not something I am aware of. Such persistency is not something expected 
from the hardware, and if it is a software-related inefficiency (which 
it sounds like), then hopefully it can be tackled in the driver/kernel.

> - Would keeping the RX ring pages collectively fresh (allocation
>    strategy, higher-order pages, a local reserve) at refill time be
>    feasible in the driver, or is this considered allocator-side?
Depends on the fix. If you have some workaround that resolves the issue, 
we can think where it naturally belongs.

Thanks!
--Dima

[-- Attachment #2: 0001-igc-add-rx-page-lifecycle-instrumentation.patch --]
[-- Type: text/plain, Size: 6617 bytes --]

diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h
index f1efadbd9f74..81862b7d5b1b 100644
--- a/drivers/net/ethernet/intel/igc/igc.h
+++ b/drivers/net/ethernet/intel/igc/igc.h
@@ -65,6 +65,16 @@ struct igc_rx_queue_stats {
 	u64 drops;
 	u64 csum_err;
 	u64 alloc_failed;
+	u64 page_reuse;
+	u64 page_reuse_failed;
+	u64 page_reuse_failed_dev_page;
+	u64 page_reuse_failed_refcnt;
+	u64 page_restock;
+	u64 page_allocs;
+	u64 page_alloc_map_err;
+	u64 page_drains;
+	u64 resource_allocs;
+	u64 resource_frees;
 };
 
 struct igc_rx_packet_stats {
diff --git a/drivers/net/ethernet/intel/igc/igc_ethtool.c b/drivers/net/ethernet/intel/igc/igc_ethtool.c
index 89fe2788a565..3eff7a0cb0d5 100644
--- a/drivers/net/ethernet/intel/igc/igc_ethtool.c
+++ b/drivers/net/ethernet/intel/igc/igc_ethtool.c
@@ -790,6 +790,16 @@ static void igc_ethtool_get_strings(struct net_device *netdev, u32 stringset,
 			ethtool_sprintf(&p, "rx_queue_%u_drops", i);
 			ethtool_sprintf(&p, "rx_queue_%u_csum_err", i);
 			ethtool_sprintf(&p, "rx_queue_%u_alloc_failed", i);
+			ethtool_sprintf(&p, "rx_queue_%u_page_reuse", i);
+			ethtool_sprintf(&p, "rx_queue_%u_page_reuse_failed", i);
+			ethtool_sprintf(&p, "rx_queue_%u_page_reuse_failed_dev_page", i);
+			ethtool_sprintf(&p, "rx_queue_%u_page_reuse_failed_refcnt", i);
+			ethtool_sprintf(&p, "rx_queue_%u_page_restock", i);
+			ethtool_sprintf(&p, "rx_queue_%u_page_allocs", i);
+			ethtool_sprintf(&p, "rx_queue_%u_page_alloc_map_err", i);
+			ethtool_sprintf(&p, "rx_queue_%u_page_drains", i);
+			ethtool_sprintf(&p, "rx_queue_%u_resource_allocs", i);
+			ethtool_sprintf(&p, "rx_queue_%u_resource_frees", i);
 		}
 		/* BUG_ON(p - data != IGC_STATS_LEN * ETH_GSTRING_LEN); */
 		break;
@@ -864,6 +874,16 @@ static void igc_ethtool_get_stats(struct net_device *netdev,
 			data[i + 2] = ring->rx_stats.drops;
 			data[i + 3] = ring->rx_stats.csum_err;
 			data[i + 4] = ring->rx_stats.alloc_failed;
+				data[i + 5] = ring->rx_stats.page_reuse;
+				data[i + 6] = ring->rx_stats.page_reuse_failed;
+				data[i + 7] = ring->rx_stats.page_reuse_failed_dev_page;
+				data[i + 8] = ring->rx_stats.page_reuse_failed_refcnt;
+				data[i + 9] = ring->rx_stats.page_restock;
+				data[i + 10] = ring->rx_stats.page_allocs;
+				data[i + 11] = ring->rx_stats.page_alloc_map_err;
+				data[i + 12] = ring->rx_stats.page_drains;
+				data[i + 13] = ring->rx_stats.resource_allocs;
+				data[i + 14] = ring->rx_stats.resource_frees;
 		} while (u64_stats_fetch_retry(&ring->rx_syncp, start));
 		i += IGC_RX_QUEUE_STATS_LEN;
 	}
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 019f0c2375d7..3668089319e0 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -495,6 +495,13 @@ static void igc_clean_all_rx_rings(struct igc_adapter *adapter)
 			igc_clean_rx_ring(adapter->rx_ring[i]);
 }
 
+static void igc_rx_ring_stat_inc(struct igc_ring *rx_ring, u64 *stat)
+{
+	u64_stats_update_begin(&rx_ring->rx_syncp);
+	(*stat)++;
+	u64_stats_update_end(&rx_ring->rx_syncp);
+}
+
 /**
  * igc_free_rx_resources - Free Rx Resources
  * @rx_ring: ring to clean the resources from
@@ -503,6 +510,8 @@ static void igc_clean_all_rx_rings(struct igc_adapter *adapter)
  */
 void igc_free_rx_resources(struct igc_ring *rx_ring)
 {
+	igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.resource_frees);
+
 	igc_clean_rx_ring(rx_ring);
 
 	xdp_rxq_info_unreg(&rx_ring->xdp_rxq);
@@ -578,6 +587,7 @@ int igc_setup_rx_resources(struct igc_ring *rx_ring)
 	rx_ring->next_to_alloc = 0;
 	rx_ring->next_to_clean = 0;
 	rx_ring->next_to_use = 0;
+	igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.resource_allocs);
 
 	return 0;
 
@@ -2076,20 +2086,29 @@ static void igc_reuse_rx_page(struct igc_ring *rx_ring,
 	new_buff->pagecnt_bias	= old_buff->pagecnt_bias;
 }
 
-static bool igc_can_reuse_rx_page(struct igc_rx_buffer *rx_buffer,
+static bool igc_can_reuse_rx_page(struct igc_ring *rx_ring,
+				  struct igc_rx_buffer *rx_buffer,
 				  int rx_buffer_pgcnt)
 {
 	unsigned int pagecnt_bias = rx_buffer->pagecnt_bias;
 	struct page *page = rx_buffer->page;
 
 	/* avoid re-using remote and pfmemalloc pages */
-	if (!dev_page_is_reusable(page))
+	if (!dev_page_is_reusable(page)) {
+		igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.page_reuse_failed);
+		igc_rx_ring_stat_inc(rx_ring,
+				     &rx_ring->rx_stats.page_reuse_failed_dev_page);
 		return false;
+	}
 
 #if (PAGE_SIZE < 8192)
 	/* if we are only owner of page we can reuse it */
-	if (unlikely((rx_buffer_pgcnt - pagecnt_bias) > 1))
+	if (unlikely((rx_buffer_pgcnt - pagecnt_bias) > 1)) {
+		igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.page_reuse_failed);
+		igc_rx_ring_stat_inc(rx_ring,
+				     &rx_ring->rx_stats.page_reuse_failed_refcnt);
 		return false;
+	}
 #else
 #define IGC_LAST_OFFSET \
 	(SKB_WITH_OVERHEAD(PAGE_SIZE) - IGC_RXBUFFER_2048)
@@ -2105,8 +2124,11 @@ static bool igc_can_reuse_rx_page(struct igc_rx_buffer *rx_buffer,
 	if (unlikely(pagecnt_bias == 1)) {
 		page_ref_add(page, USHRT_MAX - 1);
 		rx_buffer->pagecnt_bias = USHRT_MAX;
+		igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.page_restock);
 	}
 
+	igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.page_reuse);
+
 	return true;
 }
 
@@ -2175,7 +2197,7 @@ static void igc_put_rx_buffer(struct igc_ring *rx_ring,
 			      struct igc_rx_buffer *rx_buffer,
 			      int rx_buffer_pgcnt)
 {
-	if (igc_can_reuse_rx_page(rx_buffer, rx_buffer_pgcnt)) {
+	if (igc_can_reuse_rx_page(rx_ring, rx_buffer, rx_buffer_pgcnt)) {
 		/* hand second half of page back to the ring */
 		igc_reuse_rx_page(rx_ring, rx_buffer);
 	} else {
@@ -2187,6 +2209,7 @@ static void igc_put_rx_buffer(struct igc_ring *rx_ring,
 				     IGC_RX_DMA_ATTR);
 		__page_frag_cache_drain(rx_buffer->page,
 					rx_buffer->pagecnt_bias);
+		igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.page_drains);
 	}
 
 	/* clear contents of rx_buffer */
@@ -2222,6 +2245,7 @@ static bool igc_alloc_mapped_page(struct igc_ring *rx_ring,
 		set_bit(IGC_RING_FLAG_RX_ALLOC_FAILED, &rx_ring->flags);
 		return false;
 	}
+	igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.page_allocs);
 
 	/* map page for use */
 	dma = dma_map_page_attrs(rx_ring->dev, page, 0,
@@ -2236,6 +2260,7 @@ static bool igc_alloc_mapped_page(struct igc_ring *rx_ring,
 		__free_page(page);
 
 		rx_ring->rx_stats.alloc_failed++;
+		igc_rx_ring_stat_inc(rx_ring, &rx_ring->rx_stats.page_alloc_map_err);
 		set_bit(IGC_RING_FLAG_RX_ALLOC_FAILED, &rx_ring->flags);
 		return false;
 	}