* memleak in libfs report
@ 2023-10-11 15:15 Vlad Buslov
2023-10-11 15:34 ` Chuck Lever III
0 siblings, 1 reply; 6+ messages in thread
From: Vlad Buslov @ 2023-10-11 15:15 UTC (permalink / raw)
To: Chuck Lever; +Cc: linux-fsdevel, Christian Brauner, Gal Pressman
Hello Chuck,
We have been getting memleaks in offset_ctx->xa in our networking tests:
unreferenced object 0xffff8881004cd080 (size 576):
comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s)
hex dump (first 32 bytes):
00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L.....
backtrace:
[<000000000f554608>] xas_alloc+0x306/0x430
[<0000000075537d52>] xas_create+0x4b4/0xc80
[<00000000a927aab2>] xas_store+0x73/0x1680
[<0000000020a61203>] __xa_alloc+0x1d8/0x2d0
[<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310
[<000000001032332c>] simple_offset_add+0xd8/0x170
[<0000000073229fad>] shmem_mknod+0xbf/0x180
[<00000000242520ce>] vfs_mknod+0x3b0/0x5c0
[<000000001ef218dd>] unix_bind+0x2c2/0xdb0
[<0000000009b9a8dd>] __sys_bind+0x127/0x1e0
[<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0
[<00000000b8a767c7>] do_syscall_64+0x3d/0x90
[<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
It looks like those may be caused by recent commit 6faddda69f62 ("libfs:
Add directory operations for stable offsets") but we don't have a proper
reproduction, just sometimes arbitrary getting the memleak complains
during/after the regression run.
Regards,
Vlad
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report
2023-10-11 15:15 memleak in libfs report Vlad Buslov
@ 2023-10-11 15:34 ` Chuck Lever III
2023-10-11 15:52 ` Vlad Buslov
0 siblings, 1 reply; 6+ messages in thread
From: Chuck Lever III @ 2023-10-11 15:34 UTC (permalink / raw)
To: Vlad Buslov; +Cc: linux-fsdevel, Christian Brauner, Gal Pressman
> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote:
>
> Hello Chuck,
>
> We have been getting memleaks in offset_ctx->xa in our networking tests:
>
> unreferenced object 0xffff8881004cd080 (size 576):
> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s)
> hex dump (first 32 bytes):
> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L.....
> backtrace:
> [<000000000f554608>] xas_alloc+0x306/0x430
> [<0000000075537d52>] xas_create+0x4b4/0xc80
> [<00000000a927aab2>] xas_store+0x73/0x1680
> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0
> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310
> [<000000001032332c>] simple_offset_add+0xd8/0x170
> [<0000000073229fad>] shmem_mknod+0xbf/0x180
> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0
> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0
> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0
> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0
> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90
> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
>
> It looks like those may be caused by recent commit 6faddda69f62 ("libfs:
> Add directory operations for stable offsets")
That sounds plausible.
> but we don't have a proper
> reproduction, just sometimes arbitrary getting the memleak complains
> during/after the regression run.
If the leak is a trickle rather than a flood, than can you take
some time to see if you can narrow down a reproducer? If it's a
flood, I can look at this immediately.
--
Chuck Lever
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report
2023-10-11 15:34 ` Chuck Lever III
@ 2023-10-11 15:52 ` Vlad Buslov
2023-10-11 16:06 ` Chuck Lever III
0 siblings, 1 reply; 6+ messages in thread
From: Vlad Buslov @ 2023-10-11 15:52 UTC (permalink / raw)
To: Chuck Lever III; +Cc: linux-fsdevel, Christian Brauner, Gal Pressman
On Wed 11 Oct 2023 at 15:34, Chuck Lever III <chuck.lever@oracle.com> wrote:
>> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote:
>>
>> Hello Chuck,
>>
>> We have been getting memleaks in offset_ctx->xa in our networking tests:
>>
>> unreferenced object 0xffff8881004cd080 (size 576):
>> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s)
>> hex dump (first 32 bytes):
>> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L.....
>> backtrace:
>> [<000000000f554608>] xas_alloc+0x306/0x430
>> [<0000000075537d52>] xas_create+0x4b4/0xc80
>> [<00000000a927aab2>] xas_store+0x73/0x1680
>> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0
>> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310
>> [<000000001032332c>] simple_offset_add+0xd8/0x170
>> [<0000000073229fad>] shmem_mknod+0xbf/0x180
>> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0
>> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0
>> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0
>> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0
>> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90
>> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
>>
>> It looks like those may be caused by recent commit 6faddda69f62 ("libfs:
>> Add directory operations for stable offsets")
>
> That sounds plausible.
>
>
>> but we don't have a proper
>> reproduction, just sometimes arbitrary getting the memleak complains
>> during/after the regression run.
>
> If the leak is a trickle rather than a flood, than can you take
> some time to see if you can narrow down a reproducer? If it's a
> flood, I can look at this immediately.
No, it is not a flood, we are not getting setups ran out of memory
during testing or anything. However, I don't have any good idea how to
narrow down the repro since as you can see from memleak trace it is a
result of some syscall performed by systemd and none of our tests do
anything more advanced with it than 'systemctl restart ovs-vswitchd'.
Basically it is a setup with Fedora and an upstream kernel that executes
bunch of network offload tests with Open vSwitch, iproute2 tc, Linux
bridge, etc.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report
2023-10-11 15:52 ` Vlad Buslov
@ 2023-10-11 16:06 ` Chuck Lever III
2023-10-22 23:28 ` Chuck Lever III
0 siblings, 1 reply; 6+ messages in thread
From: Chuck Lever III @ 2023-10-11 16:06 UTC (permalink / raw)
To: Vlad Buslov; +Cc: linux-fsdevel, Christian Brauner, Gal Pressman
> On Oct 11, 2023, at 11:52 AM, Vlad Buslov <vladbu@nvidia.com> wrote:
>
> On Wed 11 Oct 2023 at 15:34, Chuck Lever III <chuck.lever@oracle.com> wrote:
>>> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote:
>>>
>>> Hello Chuck,
>>>
>>> We have been getting memleaks in offset_ctx->xa in our networking tests:
>>>
>>> unreferenced object 0xffff8881004cd080 (size 576):
>>> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s)
>>> hex dump (first 32 bytes):
>>> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L.....
>>> backtrace:
>>> [<000000000f554608>] xas_alloc+0x306/0x430
>>> [<0000000075537d52>] xas_create+0x4b4/0xc80
>>> [<00000000a927aab2>] xas_store+0x73/0x1680
>>> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0
>>> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310
>>> [<000000001032332c>] simple_offset_add+0xd8/0x170
>>> [<0000000073229fad>] shmem_mknod+0xbf/0x180
>>> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0
>>> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0
>>> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0
>>> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0
>>> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90
>>> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
>>>
>>> It looks like those may be caused by recent commit 6faddda69f62 ("libfs:
>>> Add directory operations for stable offsets")
>>
>> That sounds plausible.
>>
>>
>>> but we don't have a proper
>>> reproduction, just sometimes arbitrary getting the memleak complains
>>> during/after the regression run.
>>
>> If the leak is a trickle rather than a flood, than can you take
>> some time to see if you can narrow down a reproducer? If it's a
>> flood, I can look at this immediately.
>
> No, it is not a flood, we are not getting setups ran out of memory
> during testing or anything. However, I don't have any good idea how to
> narrow down the repro since as you can see from memleak trace it is a
> result of some syscall performed by systemd and none of our tests do
> anything more advanced with it than 'systemctl restart ovs-vswitchd'.
> Basically it is a setup with Fedora and an upstream kernel that executes
> bunch of network offload tests with Open vSwitch, iproute2 tc, Linux
> bridge, etc.
OK, I'll see what I can do for a reproducer. Thank you for the
report.
--
Chuck Lever
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report
2023-10-11 16:06 ` Chuck Lever III
@ 2023-10-22 23:28 ` Chuck Lever III
2023-10-23 7:07 ` Vlad Buslov
0 siblings, 1 reply; 6+ messages in thread
From: Chuck Lever III @ 2023-10-22 23:28 UTC (permalink / raw)
To: Vlad Buslov
Cc: linux-fsdevel, Christian Brauner, Gal Pressman, Hugh Dickins,
Andrew Morton, linux-mm
[ ... adding shmem maintainers ... ]
> On Oct 11, 2023, at 12:06 PM, Chuck Lever III <chuck.lever@oracle.com> wrote:
>
>> On Oct 11, 2023, at 11:52 AM, Vlad Buslov <vladbu@nvidia.com> wrote:
>>
>> On Wed 11 Oct 2023 at 15:34, Chuck Lever III <chuck.lever@oracle.com> wrote:
>>>> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote:
>>>>
>>>> Hello Chuck,
>>>>
>>>> We have been getting memleaks in offset_ctx->xa in our networking tests:
>>>>
>>>> unreferenced object 0xffff8881004cd080 (size 576):
>>>> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s)
>>>> hex dump (first 32 bytes):
>>>> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>>> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L.....
>>>> backtrace:
>>>> [<000000000f554608>] xas_alloc+0x306/0x430
>>>> [<0000000075537d52>] xas_create+0x4b4/0xc80
>>>> [<00000000a927aab2>] xas_store+0x73/0x1680
>>>> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0
>>>> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310
>>>> [<000000001032332c>] simple_offset_add+0xd8/0x170
>>>> [<0000000073229fad>] shmem_mknod+0xbf/0x180
>>>> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0
>>>> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0
>>>> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0
>>>> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0
>>>> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90
>>>> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
>>>>
>>>> It looks like those may be caused by recent commit 6faddda69f62 ("libfs:
>>>> Add directory operations for stable offsets")
>>>
>>> That sounds plausible.
>>>
>>>
>>>> but we don't have a proper
>>>> reproduction, just sometimes arbitrary getting the memleak complains
>>>> during/after the regression run.
>>>
>>> If the leak is a trickle rather than a flood, than can you take
>>> some time to see if you can narrow down a reproducer? If it's a
>>> flood, I can look at this immediately.
>>
>> No, it is not a flood, we are not getting setups ran out of memory
>> during testing or anything. However, I don't have any good idea how to
>> narrow down the repro since as you can see from memleak trace it is a
>> result of some syscall performed by systemd and none of our tests do
>> anything more advanced with it than 'systemctl restart ovs-vswitchd'.
>> Basically it is a setup with Fedora and an upstream kernel that executes
>> bunch of network offload tests with Open vSwitch, iproute2 tc, Linux
>> bridge, etc.
>
> OK, I'll see what I can do for a reproducer. Thank you for the
> report.
I've had kmemleak enabled on several systems for a week, and there
have been no tmpfs-related leaks detected. That suggests we don't
have a problem with normal workloads.
My next step is to go look at the ovs-vswitchd.service unit to
see if there are any leads there. We might ask Lennart or the
VSwitch folks if they have any suggestions too.
Meantime, can I ask that you open a bug on bugzilla.kernel.org
where we can collect troubleshooting information? Looks like
"Memory Management / Other" is appropriate for shmem, and Hugh
or Andrew can re-assign ownership to me.
--
Chuck Lever
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memleak in libfs report
2023-10-22 23:28 ` Chuck Lever III
@ 2023-10-23 7:07 ` Vlad Buslov
0 siblings, 0 replies; 6+ messages in thread
From: Vlad Buslov @ 2023-10-23 7:07 UTC (permalink / raw)
To: Chuck Lever III
Cc: linux-fsdevel, Christian Brauner, Gal Pressman, Hugh Dickins,
Andrew Morton, linux-mm
On Sun 22 Oct 2023 at 23:28, Chuck Lever III <chuck.lever@oracle.com> wrote:
> [ ... adding shmem maintainers ... ]
>
>> On Oct 11, 2023, at 12:06 PM, Chuck Lever III <chuck.lever@oracle.com> wrote:
>>
>>> On Oct 11, 2023, at 11:52 AM, Vlad Buslov <vladbu@nvidia.com> wrote:
>>>
>>> On Wed 11 Oct 2023 at 15:34, Chuck Lever III <chuck.lever@oracle.com> wrote:
>>>>> On Oct 11, 2023, at 11:15 AM, Vlad Buslov <vladbu@nvidia.com> wrote:
>>>>>
>>>>> Hello Chuck,
>>>>>
>>>>> We have been getting memleaks in offset_ctx->xa in our networking tests:
>>>>>
>>>>> unreferenced object 0xffff8881004cd080 (size 576):
>>>>> comm "systemd", pid 1, jiffies 4294893373 (age 1992.864s)
>>>>> hex dump (first 32 bytes):
>>>>> 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>>>> 38 5c 7c 02 81 88 ff ff 98 d0 4c 00 81 88 ff ff 8\|.......L.....
>>>>> backtrace:
>>>>> [<000000000f554608>] xas_alloc+0x306/0x430
>>>>> [<0000000075537d52>] xas_create+0x4b4/0xc80
>>>>> [<00000000a927aab2>] xas_store+0x73/0x1680
>>>>> [<0000000020a61203>] __xa_alloc+0x1d8/0x2d0
>>>>> [<00000000ae300af2>] __xa_alloc_cyclic+0xf1/0x310
>>>>> [<000000001032332c>] simple_offset_add+0xd8/0x170
>>>>> [<0000000073229fad>] shmem_mknod+0xbf/0x180
>>>>> [<00000000242520ce>] vfs_mknod+0x3b0/0x5c0
>>>>> [<000000001ef218dd>] unix_bind+0x2c2/0xdb0
>>>>> [<0000000009b9a8dd>] __sys_bind+0x127/0x1e0
>>>>> [<000000003c949fbb>] __x64_sys_bind+0x6e/0xb0
>>>>> [<00000000b8a767c7>] do_syscall_64+0x3d/0x90
>>>>> [<000000006132ae0d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
>>>>>
>>>>> It looks like those may be caused by recent commit 6faddda69f62 ("libfs:
>>>>> Add directory operations for stable offsets")
>>>>
>>>> That sounds plausible.
>>>>
>>>>
>>>>> but we don't have a proper
>>>>> reproduction, just sometimes arbitrary getting the memleak complains
>>>>> during/after the regression run.
>>>>
>>>> If the leak is a trickle rather than a flood, than can you take
>>>> some time to see if you can narrow down a reproducer? If it's a
>>>> flood, I can look at this immediately.
>>>
>>> No, it is not a flood, we are not getting setups ran out of memory
>>> during testing or anything. However, I don't have any good idea how to
>>> narrow down the repro since as you can see from memleak trace it is a
>>> result of some syscall performed by systemd and none of our tests do
>>> anything more advanced with it than 'systemctl restart ovs-vswitchd'.
>>> Basically it is a setup with Fedora and an upstream kernel that executes
>>> bunch of network offload tests with Open vSwitch, iproute2 tc, Linux
>>> bridge, etc.
>>
>> OK, I'll see what I can do for a reproducer. Thank you for the
>> report.
>
> I've had kmemleak enabled on several systems for a week, and there
> have been no tmpfs-related leaks detected. That suggests we don't
> have a problem with normal workloads.
>
> My next step is to go look at the ovs-vswitchd.service unit to
> see if there are any leads there. We might ask Lennart or the
> VSwitch folks if they have any suggestions too.
>
> Meantime, can I ask that you open a bug on bugzilla.kernel.org
> where we can collect troubleshooting information? Looks like
> "Memory Management / Other" is appropriate for shmem, and Hugh
> or Andrew can re-assign ownership to me.
Thanks for investigating this. Bug created:
https://bugzilla.kernel.org/show_bug.cgi?id=218039
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-10-23 7:08 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-11 15:15 memleak in libfs report Vlad Buslov
2023-10-11 15:34 ` Chuck Lever III
2023-10-11 15:52 ` Vlad Buslov
2023-10-11 16:06 ` Chuck Lever III
2023-10-22 23:28 ` Chuck Lever III
2023-10-23 7:07 ` Vlad Buslov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).