* memleak in libfs report
@ 2023-10-11 15:15 Vlad Buslov
2023-10-11 15:34 ` Chuck Lever III
0 siblings, 1 reply; 6+ messages in thread
From: Vlad Buslov @ 2023-10-11 15:15 UTC (permalink / raw)
To: Chuck Lever; +Cc: linux-fsdevel, Christian Brauner, Gal Pressman
Hello Chuck,
We have been getting memleaks in offset_ctx->xa in our networking tests:
unreferenced object 0xffff8881004cd080 (size 576):
comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s)
hex dump (first 32 bytes):
00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L.....
backtrace:
[<000000000f554608>] xas_alloc+0x306/0x430
[<0000000075537d52>] xas_create+0x4b4/0xc80
[<00000000a927aab2>] xas_store+0x73/0x1680
[<0000000020a61203>] __xa_alloc+0x1d8/0x2d0
[<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310
[<000000001032332c>] simple_offset_add+0xd8/0x170
[<0000000073229fad>] shmem_mknod+0xbf/0x180
[<00000000242520ce>] vfs_mknod+0x3b0/0x5c0
[<000000001ef218dd>] unix_bind+0x2c2/0xdb0
[<0000000009b9a8dd>] __sys_bind+0x127/0x1e0
[<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0
[<00000000b8a767c7>] do_syscall_64+0x3d/0x90
[<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
It looks like those may be caused by recent commit 6faddda69f62 ("libfs:
Add directory operations for stable offsets") but we don't have a proper
reproduction, just sometimes arbitrary getting the memleak complains
during/after the regression run.
Regards,
Vlad
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: memleak in libfs report 2023-10-11 15:15 memleak in libfs report Vlad Buslov @ 2023-10-11 15:34 ` Chuck Lever III 2023-10-11 15:52 ` Vlad Buslov 0 siblings, 1 reply; 6+ messages in thread From: Chuck Lever III @ 2023-10-11 15:34 UTC (permalink / raw) To: Vlad Buslov; +Cc: linux-fsdevel, Christian Brauner, Gal Pressman > On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote: > > Hello Chuck, > > We have been getting memleaks in offset_ctx->xa in our networking tests: > > unreferenced object 0xffff8881004cd080 (size 576): > comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s) > hex dump (first 32 bytes): > 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L..... > backtrace: > [<000000000f554608>] xas_alloc+0x306/0x430 > [<0000000075537d52>] xas_create+0x4b4/0xc80 > [<00000000a927aab2>] xas_store+0x73/0x1680 > [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0 > [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310 > [<000000001032332c>] simple_offset_add+0xd8/0x170 > [<0000000073229fad>] shmem_mknod+0xbf/0x180 > [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0 > [<000000001ef218dd>] unix_bind+0x2c2/0xdb0 > [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0 > [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0 > [<00000000b8a767c7>] do_syscall_64+0x3d/0x90 > [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0 > > It looks like those may be caused by recent commit 6faddda69f62 ("libfs: > Add directory operations for stable offsets") That sounds plausible. > but we don't have a proper > reproduction, just sometimes arbitrary getting the memleak complains > during/after the regression run. If the leak is a trickle rather than a flood, than can you take some time to see if you can narrow down a reproducer? If it's a flood, I can look at this immediately. -- Chuck Lever ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report 2023-10-11 15:34 ` Chuck Lever III @ 2023-10-11 15:52 ` Vlad Buslov 2023-10-11 16:06 ` Chuck Lever III 0 siblings, 1 reply; 6+ messages in thread From: Vlad Buslov @ 2023-10-11 15:52 UTC (permalink / raw) To: Chuck Lever III; +Cc: linux-fsdevel, Christian Brauner, Gal Pressman On Wed 11 Oct 2023 at 15:34, Chuck Lever III <chuck.lever@oracle.com> wrote: >> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote: >> >> Hello Chuck, >> >> We have been getting memleaks in offset_ctx->xa in our networking tests: >> >> unreferenced object 0xffff8881004cd080 (size 576): >> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s) >> hex dump (first 32 bytes): >> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L..... >> backtrace: >> [<000000000f554608>] xas_alloc+0x306/0x430 >> [<0000000075537d52>] xas_create+0x4b4/0xc80 >> [<00000000a927aab2>] xas_store+0x73/0x1680 >> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0 >> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310 >> [<000000001032332c>] simple_offset_add+0xd8/0x170 >> [<0000000073229fad>] shmem_mknod+0xbf/0x180 >> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0 >> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0 >> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0 >> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0 >> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90 >> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0 >> >> It looks like those may be caused by recent commit 6faddda69f62 ("libfs: >> Add directory operations for stable offsets") > > That sounds plausible. > > >> but we don't have a proper >> reproduction, just sometimes arbitrary getting the memleak complains >> during/after the regression run. > > If the leak is a trickle rather than a flood, than can you take > some time to see if you can narrow down a reproducer? If it's a > flood, I can look at this immediately. No, it is not a flood, we are not getting setups ran out of memory during testing or anything. However, I don't have any good idea how to narrow down the repro since as you can see from memleak trace it is a result of some syscall performed by systemd and none of our tests do anything more advanced with it than 'systemctl restart ovs-vswitchd'. Basically it is a setup with Fedora and an upstream kernel that executes bunch of network offload tests with Open vSwitch, iproute2 tc, Linux bridge, etc. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report 2023-10-11 15:52 ` Vlad Buslov @ 2023-10-11 16:06 ` Chuck Lever III 2023-10-22 23:28 ` Chuck Lever III 0 siblings, 1 reply; 6+ messages in thread From: Chuck Lever III @ 2023-10-11 16:06 UTC (permalink / raw) To: Vlad Buslov; +Cc: linux-fsdevel, Christian Brauner, Gal Pressman > On Oct 11, 2023, at 11:52 AM, Vlad Buslov <vladbu@nvidia.com> wrote: > > On Wed 11 Oct 2023 at 15:34, Chuck Lever III <chuck.lever@oracle.com> wrote: >>> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote: >>> >>> Hello Chuck, >>> >>> We have been getting memleaks in offset_ctx->xa in our networking tests: >>> >>> unreferenced object 0xffff8881004cd080 (size 576): >>> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s) >>> hex dump (first 32 bytes): >>> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >>> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L..... >>> backtrace: >>> [<000000000f554608>] xas_alloc+0x306/0x430 >>> [<0000000075537d52>] xas_create+0x4b4/0xc80 >>> [<00000000a927aab2>] xas_store+0x73/0x1680 >>> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0 >>> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310 >>> [<000000001032332c>] simple_offset_add+0xd8/0x170 >>> [<0000000073229fad>] shmem_mknod+0xbf/0x180 >>> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0 >>> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0 >>> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0 >>> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0 >>> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90 >>> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0 >>> >>> It looks like those may be caused by recent commit 6faddda69f62 ("libfs: >>> Add directory operations for stable offsets") >> >> That sounds plausible. >> >> >>> but we don't have a proper >>> reproduction, just sometimes arbitrary getting the memleak complains >>> during/after the regression run. >> >> If the leak is a trickle rather than a flood, than can you take >> some time to see if you can narrow down a reproducer? If it's a >> flood, I can look at this immediately. > > No, it is not a flood, we are not getting setups ran out of memory > during testing or anything. However, I don't have any good idea how to > narrow down the repro since as you can see from memleak trace it is a > result of some syscall performed by systemd and none of our tests do > anything more advanced with it than 'systemctl restart ovs-vswitchd'. > Basically it is a setup with Fedora and an upstream kernel that executes > bunch of network offload tests with Open vSwitch, iproute2 tc, Linux > bridge, etc. OK, I'll see what I can do for a reproducer. Thank you for the report. -- Chuck Lever ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report 2023-10-11 16:06 ` Chuck Lever III @ 2023-10-22 23:28 ` Chuck Lever III 2023-10-23 7:07 ` Vlad Buslov 0 siblings, 1 reply; 6+ messages in thread From: Chuck Lever III @ 2023-10-22 23:28 UTC (permalink / raw) To: Vlad Buslov Cc: linux-fsdevel, Christian Brauner, Gal Pressman, Hugh Dickins, Andrew Morton, linux-mm [ ... adding shmem maintainers ... ] > On Oct 11, 2023, at 12:06 PM, Chuck Lever III <chuck.lever@oracle.com> wrote: > >> On Oct 11, 2023, at 11:52 AM, Vlad Buslov <vladbu@nvidia.com> wrote: >> >> On Wed 11 Oct 2023 at 15:34, Chuck Lever III <chuck.lever@oracle.com> wrote: >>>> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote: >>>> >>>> Hello Chuck, >>>> >>>> We have been getting memleaks in offset_ctx->xa in our networking tests: >>>> >>>> unreferenced object 0xffff8881004cd080 (size 576): >>>> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s) >>>> hex dump (first 32 bytes): >>>> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >>>> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L..... >>>> backtrace: >>>> [<000000000f554608>] xas_alloc+0x306/0x430 >>>> [<0000000075537d52>] xas_create+0x4b4/0xc80 >>>> [<00000000a927aab2>] xas_store+0x73/0x1680 >>>> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0 >>>> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310 >>>> [<000000001032332c>] simple_offset_add+0xd8/0x170 >>>> [<0000000073229fad>] shmem_mknod+0xbf/0x180 >>>> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0 >>>> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0 >>>> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0 >>>> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0 >>>> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90 >>>> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0 >>>> >>>> It looks like those may be caused by recent commit 6faddda69f62 ("libfs: >>>> Add directory operations for stable offsets") >>> >>> That sounds plausible. >>> >>> >>>> but we don't have a proper >>>> reproduction, just sometimes arbitrary getting the memleak complains >>>> during/after the regression run. >>> >>> If the leak is a trickle rather than a flood, than can you take >>> some time to see if you can narrow down a reproducer? If it's a >>> flood, I can look at this immediately. >> >> No, it is not a flood, we are not getting setups ran out of memory >> during testing or anything. However, I don't have any good idea how to >> narrow down the repro since as you can see from memleak trace it is a >> result of some syscall performed by systemd and none of our tests do >> anything more advanced with it than 'systemctl restart ovs-vswitchd'. >> Basically it is a setup with Fedora and an upstream kernel that executes >> bunch of network offload tests with Open vSwitch, iproute2 tc, Linux >> bridge, etc. > > OK, I'll see what I can do for a reproducer. Thank you for the > report. I've had kmemleak enabled on several systems for a week, and there have been no tmpfs-related leaks detected. That suggests we don't have a problem with normal workloads. My next step is to go look at the ovs-vswitchd.service unit to see if there are any leads there. We might ask Lennart or the VSwitch folks if they have any suggestions too. Meantime, can I ask that you open a bug on bugzilla.kernel.org where we can collect troubleshooting information? Looks like "Memory Management / Other" is appropriate for shmem, and Hugh or Andrew can re-assign ownership to me. -- Chuck Lever ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report 2023-10-22 23:28 ` Chuck Lever III @ 2023-10-23 7:07 ` Vlad Buslov 0 siblings, 0 replies; 6+ messages in thread From: Vlad Buslov @ 2023-10-23 7:07 UTC (permalink / raw) To: Chuck Lever III Cc: linux-fsdevel, Christian Brauner, Gal Pressman, Hugh Dickins, Andrew Morton, linux-mm On Sun 22 Oct 2023 at 23:28, Chuck Lever III <chuck.lever@oracle.com> wrote: > [ ... adding shmem maintainers ... ] > >> On Oct 11, 2023, at 12:06 PM, Chuck Lever III <chuck.lever@oracle.com> wrote: >> >>> On Oct 11, 2023, at 11:52 AM, Vlad Buslov <vladbu@nvidia.com> wrote: >>> >>> On Wed 11 Oct 2023 at 15:34, Chuck Lever III <chuck.lever@oracle.com> wrote: >>>>> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote: >>>>> >>>>> Hello Chuck, >>>>> >>>>> We have been getting memleaks in offset_ctx->xa in our networking tests: >>>>> >>>>> unreferenced object 0xffff8881004cd080 (size 576): >>>>> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s) >>>>> hex dump (first 32 bytes): >>>>> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >>>>> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L..... >>>>> backtrace: >>>>> [<000000000f554608>] xas_alloc+0x306/0x430 >>>>> [<0000000075537d52>] xas_create+0x4b4/0xc80 >>>>> [<00000000a927aab2>] xas_store+0x73/0x1680 >>>>> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0 >>>>> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310 >>>>> [<000000001032332c>] simple_offset_add+0xd8/0x170 >>>>> [<0000000073229fad>] shmem_mknod+0xbf/0x180 >>>>> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0 >>>>> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0 >>>>> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0 >>>>> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0 >>>>> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90 >>>>> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0 >>>>> >>>>> It looks like those may be caused by recent commit 6faddda69f62 ("libfs: >>>>> Add directory operations for stable offsets") >>>> >>>> That sounds plausible. >>>> >>>> >>>>> but we don't have a proper >>>>> reproduction, just sometimes arbitrary getting the memleak complains >>>>> during/after the regression run. >>>> >>>> If the leak is a trickle rather than a flood, than can you take >>>> some time to see if you can narrow down a reproducer? If it's a >>>> flood, I can look at this immediately. >>> >>> No, it is not a flood, we are not getting setups ran out of memory >>> during testing or anything. However, I don't have any good idea how to >>> narrow down the repro since as you can see from memleak trace it is a >>> result of some syscall performed by systemd and none of our tests do >>> anything more advanced with it than 'systemctl restart ovs-vswitchd'. >>> Basically it is a setup with Fedora and an upstream kernel that executes >>> bunch of network offload tests with Open vSwitch, iproute2 tc, Linux >>> bridge, etc. >> >> OK, I'll see what I can do for a reproducer. Thank you for the >> report. > > I've had kmemleak enabled on several systems for a week, and there > have been no tmpfs-related leaks detected. That suggests we don't > have a problem with normal workloads. > > My next step is to go look at the ovs-vswitchd.service unit to > see if there are any leads there. We might ask Lennart or the > VSwitch folks if they have any suggestions too. > > Meantime, can I ask that you open a bug on bugzilla.kernel.org > where we can collect troubleshooting information? Looks like > "Memory Management / Other" is appropriate for shmem, and Hugh > or Andrew can re-assign ownership to me. Thanks for investigating this. Bug created: https://bugzilla.kernel.org/show_bug.cgi?id=218039 ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-10-23 7:08 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-10-11 15:15 memleak in libfs report Vlad Buslov 2023-10-11 15:34 ` Chuck Lever III 2023-10-11 15:52 ` Vlad Buslov 2023-10-11 16:06 ` Chuck Lever III 2023-10-22 23:28 ` Chuck Lever III 2023-10-23 7:07 ` Vlad Buslov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).