Re: [lttng-dev] 回复： 回复： 回复： 回复： shm leak in traced application?

[Mathieu Desnoyers via lttng-dev <lttng-dev@lists.lttng.org>]

[-- Attachment #1.1: Type: text/plain, Size: 6427 bytes --]

Hi Zhenyu, 

Can you try this fix please ? 

https://review.lttng.org/c/lttng-ust/+/7530 

And let me know how it goes. 

Thanks, 

Mathieu 

----- On Mar 9, 2022, at 11:37 AM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: 

> When this happpens, is the process holding a single (or very few) shm file
> references, or references to many
> shm files ?

> I wonder if you end up in a scenario where an application very frequently
> performs exec(), and therefore
> sometimes the exec() will happen in the window between the unix socket file
> descriptor reception and
> call to fcntl FD_CLOEXEC.

> Thanks,

> Mathieu

> ----- On Mar 8, 2022, at 8:29 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:

>> Thanks a lot for reply. I do not reply it in bug tracker since I have not gotten
>> a reliable way to reproduce the leak case.

>>> ------------------------------------------------------------------
>>> 发件人：Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>>> 发送时间：2022年3月8日(星期二) 23:26
>>> 收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
>>> 抄 送：Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>; lttng-dev
>>> <lttng-dev@lists.lttng.org>
>>> 主 题：Re: [lttng-dev] 回复： 回复： 回复： shm leak in traced application?

>>> ----- On Mar 8, 2022, at 12:18 AM, lttng-dev lttng-dev@lists.lttng.org wrote:

>>> > Hi,
>>> > In shm_object_table_append_shm()/alloc_shm()， why not calling FD_CLOEXEC fcntl()
>>> > to shmfds? I guess this omission leads to shm fds leak.

>>> Those file descriptors are created when received by ustcomm_recv_fds_unix_sock,
>>> and
>>> immediately after creation they are set as FD_CLOEXEC.

>>> We should continue this discussion in the bug tracker as suggested by Jonathan.
>>> It would greatly help if you can provide a small reproducer.

>>> Thanks,

>>> Mathieu

>>> > Thanks
>>> > zhenyu.ren

>>> >> ------------------------------------------------------------------
>>> >> 发件人：Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com>
>>> >> 发送时间：2022年2月25日(星期五) 22:31
>>> >> 收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
>>> >> 抄 送：lttng-dev <lttng-dev@lists.lttng.org>
>>> >> 主 题：Re: [lttng-dev] 回复： 回复： shm leak in traced application?

>>> >> Hi zhenyu.ren,

>>> >> Please open a bug on our bug tracker and provide a reproducer against the latest
>>> >> stable version (2.13.x).

>>> >> https://bugs.lttng.org/

>>> >> Please follow the guidelines: https://bugs.lttng.org/#Bug-reporting-guidelines

>>> >> Cheers

>>> >> On Fri, Feb 25, 2022 at 12:47:34PM +0800, zhenyu.ren via lttng-dev wrote:
>>> >> > Hi, lttng-dev team
>>> >>> When lttng-sessiond exits, the ust applications should call
>>> >>> lttng_ust_objd_table_owner_cleanup() and clean up all shm resource(unmap and
>>> >>> close). Howerver I do find that the ust applications keep opening "all" of the
>>> >> > shm fds("/dev/shm/ust-shm-consumer-81132 (deleted)") and do NOT free shm.
>>> >>> If we run lttng-sessiond again, ust applications can get a new piece of shm and
>>> >>> a new list of shm fds so double shm usages. Then if we kill lttng-sessiond,
>>> >>> what the mostlikely happened is ust applications close the new list of shm fds
>>> >>> and free new shm resource but keeping old shm still. In other word, we can not
>>> >> > free this piece of shm unless we killing ust applications!!!
>>> >>> So Is there any possilbe that ust applications failed calling
>>> >>> lttng_ust_objd_table_owner_cleanup()? Do you have ever see this problem? Do you
>>> >>> have any advice to free the shm without killling ust applications(I tried to
>>> >> > dig into kernel shm_open and /dev/shm, but not found any ideas)?

>>> >> > Thanks in advance
>>> >> > zhenyu.ren

>>> >> > ------------------------------------------------------------------
>>> >> > 发件人：zhenyu.ren via lttng-dev <lttng-dev@lists.lttng.org>
>>> >> > 发送时间：2022年2月23日(星期三) 23:09
>>> >> > 收件人：lttng-dev <lttng-dev@lists.lttng.org>
>>> >> > 主 题：[lttng-dev] 回复： shm leak in traced application?

>>> >>> >"I found these items also exist in a traced application which is a long-time
>>> >> > >running daemon"
>>> >> > Even if lttng-sessiond has been killed!!

>>> >> > Thanks
>>> >> > zhenyu.ren
>>> >> > ------------------------------------------------------------------
>>> >> > 发件人：zhenyu.ren via lttng-dev <lttng-dev@lists.lttng.org>
>>> >> > 发送时间：2022年2月23日(星期三) 22:44
>>> >> > 收件人：lttng-dev <lttng-dev@lists.lttng.org>
>>> >> > 主 题：[lttng-dev] shm leak in traced application?

>>> >> > Hi,
>>> >>> There are many items such as "/dev/shm/ust-shm-consumer-81132 (deleted)" exist
>>> >>> in lttng-sessiond fd spaces. I know it is the result of shm_open() and
>>> >> > shm_unlnik() in create_posix_shm().
>>> >>> However, today, I found these items also exist in a traced application which is
>>> >>> a long-time running daemon. The most important thing I found is that there
>>> >> > seems no reliable way to release share memory.
>>> >>> I tried to kill lttng-sessiond but not always release share memory. Sometimes I
>>> >>> need to kill the traced application to free share memory....But it is not a
>>> >> > good idea to kill these applications.
>>> >> > My questions are:
>>> >>> 1. Is there any way to release share memory without killing any traced
>>> >> > application?
>>> >>> 2. Is it normal that many items such as "/dev/shm/ust-shm-consumer-81132
>>> >> > (deleted)" exist in the traced application?

>>> >> > Thanks
>>> >> > zhenyu.ren

>>> >> > _______________________________________________
>>> >> > lttng-dev mailing list
>>> >> > lttng-dev@lists.lttng.org
>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>> >> --
>>> >> Jonathan Rajotte-Julien
>>> >> EfficiOS
>>> > _______________________________________________
>>> > lttng-dev mailing list
>>> > lttng-dev@lists.lttng.org
>>> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 11943 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev