Intel-Wired-Lan Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Jacob Keller <jacob.e.keller@intel.com>
To: Jaroslav Pulchart <jaroslav.pulchart@gooddata.com>
Cc: Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
	Jakub Kicinski <kuba@kernel.org>,
	Przemek Kitszel <przemyslaw.kitszel@intel.com>,
	"intel-wired-lan@lists.osuosl.org"
	<intel-wired-lan@lists.osuosl.org>,
	"Damato, Joe" <jdamato@fastly.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"Nguyen, Anthony L" <anthony.l.nguyen@intel.com>,
	Michal Swiatkowski <michal.swiatkowski@linux.intel.com>,
	"Czapnik, Lukasz" <lukasz.czapnik@intel.com>,
	"Dumazet, Eric" <edumazet@google.com>,
	"Zaki, Ahmed" <ahmed.zaki@intel.com>,
	Martin Karsten <mkarsten@uwaterloo.ca>,
	"Igor Raits" <igor@gooddata.com>,
	Daniel Secik <daniel.secik@gooddata.com>,
	"Zdenek Pesek" <zdenek.pesek@gooddata.com>
Subject: Re: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE driver after upgrade to 6.13.y (regression in commit 492a044508ad)
Date: Wed, 9 Jul 2025 12:11:04 -0700	[thread overview]
Message-ID: <04f3dc70-fd31-4b26-9647-c2f4ed012d8e@intel.com> (raw)
In-Reply-To: <a4b27e11-a3fd-4df0-8dd4-60d1a4aec5a8@intel.com>


[-- Attachment #1.1: Type: text/plain, Size: 1772 bytes --]



On 7/8/2025 5:50 PM, Jacob Keller wrote:
> 
> 
> On 7/7/2025 3:03 PM, Jacob Keller wrote:
>> Bad news: my hypothesis was incorrect.
>>
>> Good news: I can immediately see the problem if I set MTU to 9K and
>> start an iperf3 session and just watch the count of allocations from
>> ice_alloc_mapped_pages(). It goes up consistently, so I can quickly tell
>> if a change is helping.
>>
>> I ported the stats from i40e for tracking the page allocations, and I
>> can see that we're allocating new pages despite not actually performing
>> releases.
>>
>> I don't yet have a good understanding of what causes this, and the logic
>> in ice is pretty hard to track...
>>
>> I'm going to try the page pool patches myself to see if this test bed
>> triggers the same problems. Unfortunately I think I need someone else
>> with more experience with the hotpath code to help figure out whats
>> going wrong here...
> 
> I believe I have isolated this and figured out the issue: With 9K MTU,
> sometimes the hardware posts a multi-buffer frame with an extra
> descriptor that has a size of 0 bytes with no data in it. When this
> happens, our logic for tracking buffers fails to free this buffer. We
> then later overwrite the page because we failed to either free or re-use
> the page, and our overwriting logic doesn't verify this.
> 
> I will have a fix with a more detailed description posted tomorrow.

@Jaroslav, I've posted a fix which I believe should resolve your issue:

https://lore.kernel.org/intel-wired-lan/20250709-jk-ice-fix-rx-mem-leak-v1-1-cfdd7eeea905@intel.com/T/#u

I am reasonably confident it should resolve the issue you reported. If
possible, it would be appreciated if you could test it and report back
to confirm.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

  reply	other threads:[~2025-07-09 19:11 UTC|newest]

Thread overview: 49+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-04-14 16:29 [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE driver after upgrade to 6.13.y (regression in commit 492a044508ad) Jaroslav Pulchart
2025-04-14 17:15 ` Paul Menzel
2025-04-15 14:38 ` Przemek Kitszel
2025-04-16  0:53   ` Jakub Kicinski
2025-04-16  7:13     ` Jaroslav Pulchart
2025-04-16 13:48       ` Jakub Kicinski
2025-04-16 16:03         ` Jaroslav Pulchart
2025-04-16 22:44           ` Jakub Kicinski
2025-04-16 22:57             ` Keller, Jacob E
2025-04-16 22:57           ` Keller, Jacob E
2025-04-17  0:13             ` Jakub Kicinski
2025-04-17 17:52               ` Keller, Jacob E
2025-05-21  9:32                 ` Jaroslav Pulchart
2025-05-21 10:50                 ` Jaroslav Pulchart
2025-06-04  8:42                   ` Jaroslav Pulchart
2025-06-25 12:17                     ` Jaroslav Pulchart
2025-06-25 14:03                       ` Przemek Kitszel
2025-06-25 17:51                         ` Jaroslav Pulchart
2025-06-25 20:25                           ` Jakub Kicinski
2025-06-26  7:42                             ` Jaroslav Pulchart
2025-06-30  7:35                               ` Jaroslav Pulchart
2025-06-30 16:02                                 ` Jacob Keller
2025-06-30 17:24                                   ` Jaroslav Pulchart
2025-06-30 18:59                                     ` Jacob Keller
2025-06-30 20:01                                       ` Jaroslav Pulchart
2025-06-30 20:42                                         ` Jacob Keller
2025-06-30 21:56                                         ` Jacob Keller
2025-06-30 23:16                                           ` Jacob Keller
2025-07-01  6:48                                             ` Jaroslav Pulchart
2025-07-01 20:48                                               ` Jacob Keller
2025-07-02  9:48                                                 ` Jaroslav Pulchart
2025-07-02 18:01                                                   ` Jacob Keller
2025-07-02 21:56                                                   ` Jacob Keller
2025-07-03  6:46                                                     ` Jaroslav Pulchart
2025-07-03 16:16                                                       ` Jacob Keller
2025-07-04 19:30                                                         ` Maciej Fijalkowski
2025-07-07 18:32                                                         ` Jacob Keller
2025-07-07 22:03                                                           ` Jacob Keller
2025-07-09  0:50                                                             ` Jacob Keller
2025-07-09 19:11                                                               ` Jacob Keller [this message]
2025-07-09 21:04                                                                 ` Jaroslav Pulchart
2025-07-09 21:15                                                                   ` Jacob Keller
2025-07-11 18:16                                                                     ` Jaroslav Pulchart
2025-07-11 22:30                                                                       ` Jacob Keller
2025-07-14  5:34                                                                         ` Jaroslav Pulchart
2025-06-25 14:53                       ` Paul Menzel
2025-07-04 16:55 ` Michal Kubiak
2025-07-05  7:01   ` Jaroslav Pulchart
2025-07-07 15:37     ` Jaroslav Pulchart

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=04f3dc70-fd31-4b26-9647-c2f4ed012d8e@intel.com \
    --to=jacob.e.keller@intel.com \
    --cc=ahmed.zaki@intel.com \
    --cc=anthony.l.nguyen@intel.com \
    --cc=daniel.secik@gooddata.com \
    --cc=edumazet@google.com \
    --cc=igor@gooddata.com \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=jaroslav.pulchart@gooddata.com \
    --cc=jdamato@fastly.com \
    --cc=kuba@kernel.org \
    --cc=lukasz.czapnik@intel.com \
    --cc=maciej.fijalkowski@intel.com \
    --cc=michal.swiatkowski@linux.intel.com \
    --cc=mkarsten@uwaterloo.ca \
    --cc=netdev@vger.kernel.org \
    --cc=przemyslaw.kitszel@intel.com \
    --cc=zdenek.pesek@gooddata.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox