* [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory
@ 2023-08-30 9:21 Kalle Valo
2023-08-30 9:55 ` Linux regression tracking (Thorsten Leemhuis)
2023-09-05 16:28 ` Ross Lagerwall
0 siblings, 2 replies; 6+ messages in thread
From: Kalle Valo @ 2023-08-30 9:21 UTC (permalink / raw)
To: Ross Lagerwall, Bjorn Helgaas
Cc: linux-pci, linux-kernel, ath11k, regressions
Hi,
I noticed that starting from v6.5-rc1 my ath11k tests reported several
memory leaks from swapper/0:
unreferenced object 0xffff88810a02b7a8 (size 96):
comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
hex dump (first 32 bytes):
80 b8 02 0a 81 88 ff ff b8 72 07 00 00 c9 ff ff .........r......
c8 b7 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
backtrace:
unreferenced object 0xffff88810a02b880 (size 96):
comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
hex dump (first 32 bytes):
58 b9 02 0a 81 88 ff ff a8 b7 02 0a 81 88 ff ff X...............
a0 b8 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
backtrace:
unreferenced object 0xffff88810a02b958 (size 96):
comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
hex dump (first 32 bytes):
30 ba 02 0a 81 88 ff ff 80 b8 02 0a 81 88 ff ff 0...............
78 b9 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 x...............
backtrace:
unreferenced object 0xffff88810a02ba30 (size 96):
comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
hex dump (first 32 bytes):
08 bb 02 0a 81 88 ff ff 58 b9 02 0a 81 88 ff ff ........X.......
50 ba 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 P...............
backtrace:
unreferenced object 0xffff88810a02bb08 (size 96):
comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
hex dump (first 32 bytes):
e0 bb 02 0a 81 88 ff ff 30 ba 02 0a 81 88 ff ff ........0.......
28 bb 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 (...............
backtrace:
I can easily reproduce this by doing a simple insmod and rmmod of ath11k
and it's dependencies (mac80211, MHI etc). I can reliability reproduce
the leaks but I only see them once after a boot, I need to reboot the
host to see the leaks again. v6.4 has no leaks.
I did a bisect and found the commit below. I verified reverting the
commit makes the leaks go away.
commit e54223275ba1bc6f704a6bab015fcd2ae4f72572
Author: Ross Lagerwall <ross.lagerwall@citrix.com>
AuthorDate: Thu May 25 16:32:48 2023 +0100
Commit: Bjorn Helgaas <bhelgaas@google.com>
CommitDate: Fri Jun 9 15:06:16 2023 -0500
PCI: Release resource invalidated by coalescing
Kalle
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory
2023-08-30 9:21 [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory Kalle Valo
@ 2023-08-30 9:55 ` Linux regression tracking (Thorsten Leemhuis)
2023-08-30 11:31 ` Kalle Valo
2023-09-05 16:28 ` Ross Lagerwall
1 sibling, 1 reply; 6+ messages in thread
From: Linux regression tracking (Thorsten Leemhuis) @ 2023-08-30 9:55 UTC (permalink / raw)
To: Kalle Valo, Ross Lagerwall, Bjorn Helgaas
Cc: linux-pci, linux-kernel, ath11k, regressions
Hi, Thorsten here, the Linux kernel's regression tracker. Partially
top-posting for once, to make this easily accessible to everyone.
Thx for the report. To quote one aspect:
> v6.4 has no leaks.
I pretty sure you mean stock 6.4 here. But FWIW, the culprit was
backported to 6.4.y and 6.1.y, so they latest versions might be affected
as well.
For the rest of this mail:
[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]
On 30.08.23 11:21, Kalle Valo wrote:
>
> I noticed that starting from v6.5-rc1 my ath11k tests reported several
> memory leaks from swapper/0:
>
> unreferenced object 0xffff88810a02b7a8 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> 80 b8 02 0a 81 88 ff ff b8 72 07 00 00 c9 ff ff .........r......
> c8 b7 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
> backtrace:
> unreferenced object 0xffff88810a02b880 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> 58 b9 02 0a 81 88 ff ff a8 b7 02 0a 81 88 ff ff X...............
> a0 b8 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
> backtrace:
> unreferenced object 0xffff88810a02b958 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> 30 ba 02 0a 81 88 ff ff 80 b8 02 0a 81 88 ff ff 0...............
> 78 b9 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 x...............
> backtrace:
> unreferenced object 0xffff88810a02ba30 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> 08 bb 02 0a 81 88 ff ff 58 b9 02 0a 81 88 ff ff ........X.......
> 50 ba 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 P...............
> backtrace:
> unreferenced object 0xffff88810a02bb08 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> e0 bb 02 0a 81 88 ff ff 30 ba 02 0a 81 88 ff ff ........0.......
> 28 bb 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 (...............
> backtrace:
>
> I can easily reproduce this by doing a simple insmod and rmmod of ath11k
> and it's dependencies (mac80211, MHI etc). I can reliability reproduce
> the leaks but I only see them once after a boot, I need to reboot the
> host to see the leaks again. v6.4 has no leaks.
>
> I did a bisect and found the commit below. I verified reverting the
> commit makes the leaks go away.
>
> commit e54223275ba1bc6f704a6bab015fcd2ae4f72572
> Author: Ross Lagerwall <ross.lagerwall@citrix.com>
> AuthorDate: Thu May 25 16:32:48 2023 +0100
> Commit: Bjorn Helgaas <bhelgaas@google.com>
> CommitDate: Fri Jun 9 15:06:16 2023 -0500
>
> PCI: Release resource invalidated by coalescing
To be sure the issue doesn't fall through the cracks unnoticed, I'm
adding it to regzbot, the Linux kernel regression tracking bot:
#regzbot ^introduced e54223275ba1bc6f704a6bab015fcd2ae4f72572
#regzbot title PCI: comm "swapper/0" leaking memory
#regzbot ignore-activity
This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.
Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory
2023-08-30 9:55 ` Linux regression tracking (Thorsten Leemhuis)
@ 2023-08-30 11:31 ` Kalle Valo
0 siblings, 0 replies; 6+ messages in thread
From: Kalle Valo @ 2023-08-30 11:31 UTC (permalink / raw)
To: Linux regression tracking (Thorsten Leemhuis)
Cc: Ross Lagerwall, Bjorn Helgaas, Linux regressions mailing list,
linux-pci, linux-kernel, ath11k
"Linux regression tracking (Thorsten Leemhuis)"
<regressions@leemhuis.info> writes:
> Hi, Thorsten here, the Linux kernel's regression tracker. Partially
> top-posting for once, to make this easily accessible to everyone.
>
> Thx for the report. To quote one aspect:
>
>> v6.4 has no leaks.
>
> I pretty sure you mean stock 6.4 here. But FWIW, the culprit was
> backported to 6.4.y and 6.1.y, so they latest versions might be affected
> as well.
Yes, I mean v6.4 release tagged by Linus:
6995e2de6891 Linux 6.4
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory
2023-08-30 9:21 [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory Kalle Valo
2023-08-30 9:55 ` Linux regression tracking (Thorsten Leemhuis)
@ 2023-09-05 16:28 ` Ross Lagerwall
2023-09-06 8:40 ` Kalle Valo
1 sibling, 1 reply; 6+ messages in thread
From: Ross Lagerwall @ 2023-09-05 16:28 UTC (permalink / raw)
To: Kalle Valo; +Cc: Bjorn Helgaas, linux-pci, linux-kernel, ath11k, regressions
On Wed, Aug 30, 2023 at 10:21 AM Kalle Valo <kvalo@kernel.org> wrote:
>
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
>
> Hi,
>
> I noticed that starting from v6.5-rc1 my ath11k tests reported several
> memory leaks from swapper/0:
>
> unreferenced object 0xffff88810a02b7a8 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> 80 b8 02 0a 81 88 ff ff b8 72 07 00 00 c9 ff ff .........r......
> c8 b7 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
> backtrace:
> unreferenced object 0xffff88810a02b880 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> 58 b9 02 0a 81 88 ff ff a8 b7 02 0a 81 88 ff ff X...............
> a0 b8 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
> backtrace:
> unreferenced object 0xffff88810a02b958 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> 30 ba 02 0a 81 88 ff ff 80 b8 02 0a 81 88 ff ff 0...............
> 78 b9 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 x...............
> backtrace:
> unreferenced object 0xffff88810a02ba30 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> 08 bb 02 0a 81 88 ff ff 58 b9 02 0a 81 88 ff ff ........X.......
> 50 ba 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 P...............
> backtrace:
> unreferenced object 0xffff88810a02bb08 (size 96):
> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> hex dump (first 32 bytes):
> e0 bb 02 0a 81 88 ff ff 30 ba 02 0a 81 88 ff ff ........0.......
> 28 bb 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 (...............
> backtrace:
>
> I can easily reproduce this by doing a simple insmod and rmmod of ath11k
> and it's dependencies (mac80211, MHI etc). I can reliability reproduce
> the leaks but I only see them once after a boot, I need to reboot the
> host to see the leaks again. v6.4 has no leaks.
>
> I did a bisect and found the commit below. I verified reverting the
> commit makes the leaks go away.
>
> commit e54223275ba1bc6f704a6bab015fcd2ae4f72572
> Author: Ross Lagerwall <ross.lagerwall@citrix.com>
> AuthorDate: Thu May 25 16:32:48 2023 +0100
> Commit: Bjorn Helgaas <bhelgaas@google.com>
> CommitDate: Fri Jun 9 15:06:16 2023 -0500
>
> PCI: Release resource invalidated by coalescing
>
> Kalle
>
Hi Kalle,
I can't reproduce the leak by loading/unloading the ath11k module. I suspect
that the leak is always there when PCI resources are coalesced but
kmemleak doesn't notice until ath11k is loaded.
Can you please try the following to confirm it fixes it?
Ross
8<-----------------------
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 8bac3ce02609..907c873473e2 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -998,6 +998,7 @@ static int pci_register_host_bridge(struct
pci_host_bridge *bridge)
res = window->res;
if (!res->flags && !res->start && !res->end) {
release_resource(res);
+ resource_list_destroy_entry(window);
continue;
}
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory
2023-09-05 16:28 ` Ross Lagerwall
@ 2023-09-06 8:40 ` Kalle Valo
2023-09-06 11:03 ` Ross Lagerwall
0 siblings, 1 reply; 6+ messages in thread
From: Kalle Valo @ 2023-09-06 8:40 UTC (permalink / raw)
To: Ross Lagerwall
Cc: Bjorn Helgaas, linux-pci, linux-kernel, ath11k, regressions
Ross Lagerwall <ross.lagerwall@cloud.com> writes:
> On Wed, Aug 30, 2023 at 10:21 AM Kalle Valo <kvalo@kernel.org> wrote:
>
>>
>> I noticed that starting from v6.5-rc1 my ath11k tests reported several
>> memory leaks from swapper/0:
>>
>> unreferenced object 0xffff88810a02b7a8 (size 96):
>> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
>> hex dump (first 32 bytes):
>> 80 b8 02 0a 81 88 ff ff b8 72 07 00 00 c9 ff ff .........r......
>> c8 b7 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
>> backtrace:
>> unreferenced object 0xffff88810a02b880 (size 96):
>> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
>> hex dump (first 32 bytes):
>> 58 b9 02 0a 81 88 ff ff a8 b7 02 0a 81 88 ff ff X...............
>> a0 b8 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
>> backtrace:
>> unreferenced object 0xffff88810a02b958 (size 96):
>> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
>> hex dump (first 32 bytes):
>> 30 ba 02 0a 81 88 ff ff 80 b8 02 0a 81 88 ff ff 0...............
>> 78 b9 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 x...............
>> backtrace:
>> unreferenced object 0xffff88810a02ba30 (size 96):
>> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
>> hex dump (first 32 bytes):
>> 08 bb 02 0a 81 88 ff ff 58 b9 02 0a 81 88 ff ff ........X.......
>> 50 ba 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 P...............
>> backtrace:
>> unreferenced object 0xffff88810a02bb08 (size 96):
>> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
>> hex dump (first 32 bytes):
>> e0 bb 02 0a 81 88 ff ff 30 ba 02 0a 81 88 ff ff ........0.......
>> 28 bb 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 (...............
>> backtrace:
>>
>> I can easily reproduce this by doing a simple insmod and rmmod of ath11k
>> and it's dependencies (mac80211, MHI etc). I can reliability reproduce
>> the leaks but I only see them once after a boot, I need to reboot the
>> host to see the leaks again. v6.4 has no leaks.
>>
>> I did a bisect and found the commit below. I verified reverting the
>> commit makes the leaks go away.
>>
>> commit e54223275ba1bc6f704a6bab015fcd2ae4f72572
>> Author: Ross Lagerwall <ross.lagerwall@citrix.com>
>> AuthorDate: Thu May 25 16:32:48 2023 +0100
>> Commit: Bjorn Helgaas <bhelgaas@google.com>
>> CommitDate: Fri Jun 9 15:06:16 2023 -0500
>>
>> PCI: Release resource invalidated by coalescing
>
> Hi Kalle,
>
> I can't reproduce the leak by loading/unloading the ath11k module. I suspect
> that the leak is always there when PCI resources are coalesced but
> kmemleak doesn't notice until ath11k is loaded.
>
> Can you please try the following to confirm it fixes it?
I run various tests with your patch and I don't see leaks anymore. I
also veried that without your patch I see the leak immediately.
Thanks for fixing this so quickly, it would good to have this fix in
v6.6 if possible.
Tested-by: Kalle Valo <kvalo@kernel.org>
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory
2023-09-06 8:40 ` Kalle Valo
@ 2023-09-06 11:03 ` Ross Lagerwall
0 siblings, 0 replies; 6+ messages in thread
From: Ross Lagerwall @ 2023-09-06 11:03 UTC (permalink / raw)
To: Kalle Valo; +Cc: Bjorn Helgaas, linux-pci, linux-kernel, ath11k, regressions
On Wed, Sep 6, 2023 at 9:40 AM Kalle Valo <kvalo@kernel.org> wrote:
>
> Ross Lagerwall <ross.lagerwall@cloud.com> writes:
>
> > On Wed, Aug 30, 2023 at 10:21 AM Kalle Valo <kvalo@kernel.org> wrote:
> >
> >>
> >> I noticed that starting from v6.5-rc1 my ath11k tests reported several
> >> memory leaks from swapper/0:
> >>
> >> unreferenced object 0xffff88810a02b7a8 (size 96):
> >> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> >> hex dump (first 32 bytes):
> >> 80 b8 02 0a 81 88 ff ff b8 72 07 00 00 c9 ff ff .........r......
> >> c8 b7 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
> >> backtrace:
> >> unreferenced object 0xffff88810a02b880 (size 96):
> >> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> >> hex dump (first 32 bytes):
> >> 58 b9 02 0a 81 88 ff ff a8 b7 02 0a 81 88 ff ff X...............
> >> a0 b8 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 ................
> >> backtrace:
> >> unreferenced object 0xffff88810a02b958 (size 96):
> >> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> >> hex dump (first 32 bytes):
> >> 30 ba 02 0a 81 88 ff ff 80 b8 02 0a 81 88 ff ff 0...............
> >> 78 b9 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 x...............
> >> backtrace:
> >> unreferenced object 0xffff88810a02ba30 (size 96):
> >> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> >> hex dump (first 32 bytes):
> >> 08 bb 02 0a 81 88 ff ff 58 b9 02 0a 81 88 ff ff ........X.......
> >> 50 ba 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 P...............
> >> backtrace:
> >> unreferenced object 0xffff88810a02bb08 (size 96):
> >> comm "swapper/0", pid 1, jiffies 4294671838 (age 98.120s)
> >> hex dump (first 32 bytes):
> >> e0 bb 02 0a 81 88 ff ff 30 ba 02 0a 81 88 ff ff ........0.......
> >> 28 bb 02 0a 81 88 ff ff 00 00 00 00 00 00 00 00 (...............
> >> backtrace:
> >>
> >> I can easily reproduce this by doing a simple insmod and rmmod of ath11k
> >> and it's dependencies (mac80211, MHI etc). I can reliability reproduce
> >> the leaks but I only see them once after a boot, I need to reboot the
> >> host to see the leaks again. v6.4 has no leaks.
> >>
> >> I did a bisect and found the commit below. I verified reverting the
> >> commit makes the leaks go away.
> >>
> >> commit e54223275ba1bc6f704a6bab015fcd2ae4f72572
> >> Author: Ross Lagerwall <ross.lagerwall@citrix.com>
> >> AuthorDate: Thu May 25 16:32:48 2023 +0100
> >> Commit: Bjorn Helgaas <bhelgaas@google.com>
> >> CommitDate: Fri Jun 9 15:06:16 2023 -0500
> >>
> >> PCI: Release resource invalidated by coalescing
> >
> > Hi Kalle,
> >
> > I can't reproduce the leak by loading/unloading the ath11k module. I suspect
> > that the leak is always there when PCI resources are coalesced but
> > kmemleak doesn't notice until ath11k is loaded.
> >
> > Can you please try the following to confirm it fixes it?
>
> I run various tests with your patch and I don't see leaks anymore. I
> also veried that without your patch I see the leak immediately.
>
> Thanks for fixing this so quickly, it would good to have this fix in
> v6.6 if possible.
>
> Tested-by: Kalle Valo <kvalo@kernel.org>
>
Thanks, I will send out a proper patch now.
Ross
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-09-06 11:03 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-08-30 9:21 [regression v6.5-rc1] PCI: comm "swapper/0" leaking memory Kalle Valo
2023-08-30 9:55 ` Linux regression tracking (Thorsten Leemhuis)
2023-08-30 11:31 ` Kalle Valo
2023-09-05 16:28 ` Ross Lagerwall
2023-09-06 8:40 ` Kalle Valo
2023-09-06 11:03 ` Ross Lagerwall
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).