Intel-Wired-Lan Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Jacob Keller <jacob.e.keller@intel.com>
To: Jaroslav Pulchart <jaroslav.pulchart@gooddata.com>
Cc: Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
	Jakub Kicinski <kuba@kernel.org>,
	Przemek Kitszel <przemyslaw.kitszel@intel.com>,
	"intel-wired-lan@lists.osuosl.org"
	<intel-wired-lan@lists.osuosl.org>,
	"Damato, Joe" <jdamato@fastly.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"Nguyen, Anthony L" <anthony.l.nguyen@intel.com>,
	Michal Swiatkowski <michal.swiatkowski@linux.intel.com>,
	"Czapnik, Lukasz" <lukasz.czapnik@intel.com>,
	"Dumazet, Eric" <edumazet@google.com>,
	"Zaki, Ahmed" <ahmed.zaki@intel.com>,
	Martin Karsten <mkarsten@uwaterloo.ca>,
	"Igor Raits" <igor@gooddata.com>,
	Daniel Secik <daniel.secik@gooddata.com>,
	"Zdenek Pesek" <zdenek.pesek@gooddata.com>
Subject: Re: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE driver after upgrade to 6.13.y (regression in commit 492a044508ad)
Date: Tue, 8 Jul 2025 17:50:23 -0700	[thread overview]
Message-ID: <a4b27e11-a3fd-4df0-8dd4-60d1a4aec5a8@intel.com> (raw)
In-Reply-To: <ee05284e-3ab1-482f-a727-981b9fd4e9ee@intel.com>


[-- Attachment #1.1: Type: text/plain, Size: 1346 bytes --]



On 7/7/2025 3:03 PM, Jacob Keller wrote:
> Bad news: my hypothesis was incorrect.
> 
> Good news: I can immediately see the problem if I set MTU to 9K and
> start an iperf3 session and just watch the count of allocations from
> ice_alloc_mapped_pages(). It goes up consistently, so I can quickly tell
> if a change is helping.
> 
> I ported the stats from i40e for tracking the page allocations, and I
> can see that we're allocating new pages despite not actually performing
> releases.
> 
> I don't yet have a good understanding of what causes this, and the logic
> in ice is pretty hard to track...
> 
> I'm going to try the page pool patches myself to see if this test bed
> triggers the same problems. Unfortunately I think I need someone else
> with more experience with the hotpath code to help figure out whats
> going wrong here...

I believe I have isolated this and figured out the issue: With 9K MTU,
sometimes the hardware posts a multi-buffer frame with an extra
descriptor that has a size of 0 bytes with no data in it. When this
happens, our logic for tracking buffers fails to free this buffer. We
then later overwrite the page because we failed to either free or re-use
the page, and our overwriting logic doesn't verify this.

I will have a fix with a more detailed description posted tomorrow.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

  reply	other threads:[~2025-07-09  0:50 UTC|newest]

Thread overview: 49+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-04-14 16:29 [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE driver after upgrade to 6.13.y (regression in commit 492a044508ad) Jaroslav Pulchart
2025-04-14 17:15 ` Paul Menzel
2025-04-15 14:38 ` Przemek Kitszel
2025-04-16  0:53   ` Jakub Kicinski
2025-04-16  7:13     ` Jaroslav Pulchart
2025-04-16 13:48       ` Jakub Kicinski
2025-04-16 16:03         ` Jaroslav Pulchart
2025-04-16 22:44           ` Jakub Kicinski
2025-04-16 22:57             ` Keller, Jacob E
2025-04-16 22:57           ` Keller, Jacob E
2025-04-17  0:13             ` Jakub Kicinski
2025-04-17 17:52               ` Keller, Jacob E
2025-05-21  9:32                 ` Jaroslav Pulchart
2025-05-21 10:50                 ` Jaroslav Pulchart
2025-06-04  8:42                   ` Jaroslav Pulchart
2025-06-25 12:17                     ` Jaroslav Pulchart
2025-06-25 14:03                       ` Przemek Kitszel
2025-06-25 17:51                         ` Jaroslav Pulchart
2025-06-25 20:25                           ` Jakub Kicinski
2025-06-26  7:42                             ` Jaroslav Pulchart
2025-06-30  7:35                               ` Jaroslav Pulchart
2025-06-30 16:02                                 ` Jacob Keller
2025-06-30 17:24                                   ` Jaroslav Pulchart
2025-06-30 18:59                                     ` Jacob Keller
2025-06-30 20:01                                       ` Jaroslav Pulchart
2025-06-30 20:42                                         ` Jacob Keller
2025-06-30 21:56                                         ` Jacob Keller
2025-06-30 23:16                                           ` Jacob Keller
2025-07-01  6:48                                             ` Jaroslav Pulchart
2025-07-01 20:48                                               ` Jacob Keller
2025-07-02  9:48                                                 ` Jaroslav Pulchart
2025-07-02 18:01                                                   ` Jacob Keller
2025-07-02 21:56                                                   ` Jacob Keller
2025-07-03  6:46                                                     ` Jaroslav Pulchart
2025-07-03 16:16                                                       ` Jacob Keller
2025-07-04 19:30                                                         ` Maciej Fijalkowski
2025-07-07 18:32                                                         ` Jacob Keller
2025-07-07 22:03                                                           ` Jacob Keller
2025-07-09  0:50                                                             ` Jacob Keller [this message]
2025-07-09 19:11                                                               ` Jacob Keller
2025-07-09 21:04                                                                 ` Jaroslav Pulchart
2025-07-09 21:15                                                                   ` Jacob Keller
2025-07-11 18:16                                                                     ` Jaroslav Pulchart
2025-07-11 22:30                                                                       ` Jacob Keller
2025-07-14  5:34                                                                         ` Jaroslav Pulchart
2025-06-25 14:53                       ` Paul Menzel
2025-07-04 16:55 ` Michal Kubiak
2025-07-05  7:01   ` Jaroslav Pulchart
2025-07-07 15:37     ` Jaroslav Pulchart

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a4b27e11-a3fd-4df0-8dd4-60d1a4aec5a8@intel.com \
    --to=jacob.e.keller@intel.com \
    --cc=ahmed.zaki@intel.com \
    --cc=anthony.l.nguyen@intel.com \
    --cc=daniel.secik@gooddata.com \
    --cc=edumazet@google.com \
    --cc=igor@gooddata.com \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=jaroslav.pulchart@gooddata.com \
    --cc=jdamato@fastly.com \
    --cc=kuba@kernel.org \
    --cc=lukasz.czapnik@intel.com \
    --cc=maciej.fijalkowski@intel.com \
    --cc=michal.swiatkowski@linux.intel.com \
    --cc=mkarsten@uwaterloo.ca \
    --cc=netdev@vger.kernel.org \
    --cc=przemyslaw.kitszel@intel.com \
    --cc=zdenek.pesek@gooddata.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox