* Re: lttng-dev Digest, Vol 203, Issue 7 [not found] <mailman.58.1741790192.282.lttng-dev@lists.lttng.org> @ 2025-03-17 6:29 ` Gour DEV via lttng-dev 2025-03-20 20:51 ` Kienan Stewart via lttng-dev 0 siblings, 1 reply; 5+ messages in thread From: Gour DEV via lttng-dev @ 2025-03-17 6:29 UTC (permalink / raw) To: lttng-dev, Kienan Stewart [-- Attachment #1.1: Type: text/plain, Size: 15034 bytes --] Hi, Kienan Sorry for the late reply. Looks like in buster the memory is allocated by lttng-consumerd reserved I buster, the rss is less than the VIRT root@localhost:~# COLUMNS=500 top -b -n 1 | grep lttng PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4095 root 20 0 1003188 31256 4660 S 0.0 0.1 0:03.81 lttng-sessiond 4096 root 20 0 44260 796 0 S 0.0 0.0 0:00.01 lttng-runas 4440 root 20 0 5236020 10224 8756 S 0.0 0.0 2:56.25 lttng-consumerd -- here the VIRT is much more higher than RSS 4443 root 20 0 48048 540 0 S 0.0 0.0 0:00.12 lttng-runas In bookworm the VIRT and RES are nearly the same only. root@edgecore-40XKE-j2-101-32:~# COLUMNS=500 top -b -n 1 | grep lttng PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4382 root 20 0 1098824 42600 8436 S 0.0 0.1 0:08.87 lttng-sessiond 4403 root 20 0 48928 2116 996 S 0.0 0.0 0:00.00 lttng-runas 5171 root 20 0 9879764 8.9g 8.9g S 0.0 28.7 108:23.53 lttng-consumerd -- here the VRIT is nearly equal to RSS 5173 root 20 0 3680 1028 680 S 0.0 0.0 0:00.88 lttng-runas Looks like lttng consumerd is allocating and reserving those pages, when any instrumented application starts. I am attaching the lttng status output in the mail, please do tell me if you need any more information regarding this. These is how we used to create the lttng channels and enable event which is same in both buster and bookworm, (number of channels might differ) def enable_channel(channels, session, subbuf_size, subbuf_num): for c in channels: call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size', str(subbuf_size), '--num-subbuf', str(subbuf_num),], stdout=devnull, stderr=subprocess.STDOUT) def enable_events(traces, session): for t in traces: if 'log-level-only' in t: log_opt = '--loglevel-only=' + t['log-level-only'] elif 'log-level' in t: log_opt = '--loglevel=' + t['log-level'] else: log_opt = '' else: call(['lttng', 'enable-event', '-u', t['name'], '-c', t['channel'], '-s', session], stdout=devnull, stderr=subprocess.STDOUT) Thank You. Lakshya On Wed, Mar 12, 2025 at 8:06 PM <lttng-dev-request@lists.lttng.org> wrote: > Send lttng-dev mailing list submissions to > lttng-dev@lists.lttng.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > or, via email, send a message with subject or body 'help' to > lttng-dev-request@lists.lttng.org > > You can reach the person managing the list at > lttng-dev-owner@lists.lttng.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of lttng-dev digest..." > > > Today's Topics: > > 1. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > (Kienan Stewart) > 2. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > (Gour DEV) > 3. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > (Kienan Stewart) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 11 Mar 2025 14:55:21 -0400 > From: Kienan Stewart <kstewart@efficios.com> > To: Gour DEV <lakshyagour10@gmail.com>, lttng-dev@lists.lttng.org > Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > Message-ID: <38dab5ef-f106-4e57-9e36-b4b30015c019@efficios.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > Hi Lakshya, > > On 3/11/25 12:25 PM, Gour DEV wrote: > > Hi, Kienan > > > > here is the requested output > > > > root@localhost:~# top -b -n 1 | grep lttng > > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 8:39.93 > > lttng-c+ > > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 0:07.25 > > lttng-s+ > > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 > > lttng-r+ > > 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 > > This top output for `localhost` seems very different than the output for > `localhost` in your previous message. > > > > lttng-r+ > > root@localhost:~# nrpco > > bash: nrpco: command not found > > root@localhost:~# nproc > > 16 > > root@localhost:~# cat /sys/devices/system/cpu/possible > > 0-15 > > > > You indicated the bookworm machine has 32 cores, this is showing 16. If > you're comparing a 16 core machine to a 32 core machine, it is very > normal that the memory usage is higher on the 32 core machine. > > > > > Most of the process are running as asorcs user but some are running > as root. > > So you have two users with instrumented applications. > > > Given the discrepancies in the information provided I'm finding it a bit > hard to understand what you're looking at. > > > In general, a channel's shared memory footprint can be estimated with[1]: > > (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * > (nUIDs or nPIDs) > > Note that the sub-buffer sizes you are using get rounded to the nearest > larger power of 2. See [2]. > > thanks, > kienan > > [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes > [2]: > https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size > > > ------------------------------ > > Message: 2 > Date: Wed, 12 Mar 2025 14:49:07 +0530 > From: Gour DEV <lakshyagour10@gmail.com> > To: Kienan Stewart <kstewart@efficios.com> > Cc: lttng-dev@lists.lttng.org > Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > Message-ID: > <CAE9Jrzg7qsabhPiO-0=B1DY3bVo-3FYu2tiJR2Bmb= > nqOHNZMw@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, Kienan > > I am attaching an screen recording of the behaviour I am seeing in this > mail. The behaviour is same irrespective of the device i use, sorry for > miscommunication in the npocs output (I assumed it was 32), but other than > that all outputs are same (except the hostname as there are multiple > devices with same lttng config but this memory cosumption is seen on all > the devices). > > I had few question > > 1. Does lltng allocated all the memory it needs and mark it as dirty in ram > when any process which links/uses lttng-ust runs? (here i tried with one > process but it is same for any of my process) > 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * > (nUIDs or nPIDs) > > How do we calculate uid in the system is it all uids in the system? is it > equal to `cat /etc/passwd | wc -l` ? > > I will put my calculations according to the above estimate based on all the > channel i am creating > > (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to > `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the > start_lttng.py please do correct me if am wrong here.] > > But since there are only two users which uses lttng i think the correct > estimate would be > (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB > > Please do correct me If I am wrong calculations here. > > Now, there are a few things here, according to my output lttng is using 11G > which is much more higher than the what is configured. > > I am attaching the lttng status and the file which is uses to create the > lttng sessions. > > > > Thank You. > > > > https://drive.google.com/file/d/1tS_ZWEsXDpHZXfWzZHXmWcT0igiIOIaa/view?usp=sharing > -- recording of the behaviour which is seen > > https://drive.google.com/file/d/1PrU31oyEw1n9tKETlUtmNGO50s6ywx7p/view?usp=sharing > -- the file which is used to create lttng sessions > > > On Wed, Mar 12, 2025 at 12:25?AM Kienan Stewart <kstewart@efficios.com> > wrote: > > > Hi Lakshya, > > > > On 3/11/25 12:25 PM, Gour DEV wrote: > > > Hi, Kienan > > > > > > here is the requested output > > > > > > root@localhost:~# top -b -n 1 | grep lttng > > > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 > 8:39.93 > > > lttng-c+ > > > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 > 0:07.25 > > > lttng-s+ > > > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 > 0:00.00 > > > lttng-r+ > > > 4843 root 20 0 3680 1160 816 S 0.0 0.0 > 0:00.23 > > > > This top output for `localhost` seems very different than the output for > > `localhost` in your previous message. > > > > > > > lttng-r+ > > > root@localhost:~# nrpco > > > bash: nrpco: command not found > > > root@localhost:~# nproc > > > 16 > > > root@localhost:~# cat /sys/devices/system/cpu/possible > > > 0-15 > > > > > > > You indicated the bookworm machine has 32 cores, this is showing 16. If > > you're comparing a 16 core machine to a 32 core machine, it is very > > normal that the memory usage is higher on the 32 core machine. > > > > > > > > Most of the process are running as asorcs user but some are running > > as root. > > > > So you have two users with instrumented applications. > > > > > > Given the discrepancies in the information provided I'm finding it a bit > > hard to understand what you're looking at. > > > > > > In general, a channel's shared memory footprint can be estimated with[1]: > > > > (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * > > (nUIDs or nPIDs) > > > > Note that the sub-buffer sizes you are using get rounded to the nearest > > larger power of 2. See [2]. > > > > thanks, > > kienan > > > > [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes > > [2]: > > https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://lists.lttng.org/pipermail/lttng-dev/attachments/20250312/57f240d8/attachment-0001.htm > > > > ------------------------------ > > Message: 3 > Date: Wed, 12 Mar 2025 10:36:28 -0400 > From: Kienan Stewart <kstewart@efficios.com> > To: Gour DEV <lakshyagour10@gmail.com>, lttng-dev@lists.lttng.org > Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > Message-ID: <0f819583-ea8e-468e-9102-e1410d886a6f@efficios.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > Hi Lakshya, > > On 3/12/25 5:03 AM, Gour DEV wrote: > > Hi, Kienan > > > > I am attaching an screen recording of the behaviour I am seeing in this > > mail. The behaviour is same irrespective of the device i use, sorry for > > miscommunication in the npocs output (I assumed it was 32), but other > than > > that all outputs are same (except the hostname as there are multiple > > devices with same lttng config but this memory cosumption is seen on all > > the devices). > > > > I had few question > > > > 1. Does lltng allocated all the memory it needs and mark it as dirty in > ram > > when any process which links/uses lttng-ust runs? (here i tried with one > > process but it is same for any of my process) > > I believe the shared memory for per-CPU data structures is allocated > when an instrumented application connects. There is no pre-allocation > for each possible UID on the system. > > You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to > see when the connection happens[1]. > > > 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * > > (nUIDs or nPIDs) > > > > How do we calculate uid in the system is it all uids in the system? is it > > equal to `cat /etc/passwd | wc -l` ? > > nUIDs is the number of distinct UIDs running instrumented applications. > > > > > I will put my calculations according to the above estimate based on all > the > > channel i am creating > > > > (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to > > `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the > > start_lttng.py please do correct me if am wrong here.] > > > > But since there are only two users which uses lttng i think the correct > > estimate would be > > (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB > > The estimate I gave is per-channel. > > small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel > per-UID > medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel per-UID > large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID > > Now, you said you have 0 small channels, 6 medium channels, and 16 large > channels in your session. (Note: I see your script differs from these > stated channel counts). > > small: 0 * 1.0625 MiB = 0 MiB per-UID > medium: 6 * 17 MiB = 102 MiB per-UID > large: 16 * 272 MiB = 4352 MiB per-UID > > And if you're running instrumented applications with 2 users: > > small: 0 MiB * 2 = 0 MiB with 2 UIDs > medium: 102 MiB * 2 = 204 MiB with 2 UIDs > large: 4352 MiB * 2 = 8704 MiB with 2 UIDs > > Now this is just an estimation for the per-CPU ring buffers only, and > you numbers aren't hugely off so without analyzing your specific system > it doesn't seem to be that strange to me. > > If I take the number of channels I see in your script, it becomes: > > small: 0 MiB with 2 UIDs > medium: 136 MiB with 2 UIDs > large: 7616 MiB with 2 UIDs > > total: 7.57 GiB with 2 UIDs > > > > > Please do correct me If I am wrong calculations here. > > > > Now, there are a few things here, according to my output lttng is using > 11G > > which is much more higher than the what is configured. > > > > I have no idea what 'service start spyder' is doing. Maybe it's running > instrumented applications with an extra user that you didn't expect? I > can't help you with that aspect of your system. > > The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs > so maybe? > > I'd recommend you read your verbose sessiond log so see which > applications are connecting and with which UIDs. > > > I am attaching the lttng status and the file which is uses to create the > > lttng sessions. > > > > > > > > Thank You. > > > > In any case, the information you have given to date hasn't demonstrated > to me in a tangible manner that you are seeing a difference related to > the version of LTTng being used. > > thanks, > kienan > > [1]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > ------------------------------ > > End of lttng-dev Digest, Vol 203, Issue 7 > ***************************************** > [-- Attachment #1.2: Type: text/html, Size: 25507 bytes --] [-- Attachment #2: lttng_status_bookwomr --] [-- Type: application/octet-stream, Size: 13403 bytes --] Recording session default: [active snapshot] === Domain: User space === Buffering scheme: per-user Tracked process attributes Virtual process IDs: all Virtual user IDs: all Virtual group IDs: all Channels: ------------- - l2rib-all: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: l2rib* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - bgp-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: bgp:bgp_error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - fib-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: fib* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - stp-all: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: stp* (loglevel <= TRACE_DEBUG_LINE (13)) (type: tracepoint) [enabled] - mac-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 262144 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: mac* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - fib-info: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 262144 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: fib* (loglevel == TRACE_DEBUG_PROCESS (9)) (type: tracepoint) [enabled] - dpal-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: bcm* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] lxk* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] dpal* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - bgp-events: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: bgp:bgp_ev_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_netlink_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_warn (loglevel <= TRACE_WARNING (4)) (type: tracepoint) [enabled] bgp:bgp_info (loglevel <= TRACE_INFO (6)) (type: tracepoint) [enabled] bgp:bgp_cleanup_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_nbr_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_ha_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_nexthop_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_nbrall_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_label_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - ldp-all: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: ldp* (loglevel <= TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - rib-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: rib:rib_error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - other-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: mbroker* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] kernel_spiel:error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] ifmgr:ifmgr_error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] vrrp:error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] adj:adj_error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - mac-err: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 262144 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: mac* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - other-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: mbroker* (loglevel == TRACE_DEBUG_MODULE (10)) (type: tracepoint) [enabled] dra* (loglevel == TRACE_DEBUG_MODULE (10)) (type: tracepoint) [enabled] vrrp:dpal_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] adj:nl_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] vrrp:ev_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] ifmgr:startup_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rad* (loglevel == TRACE_DEBUG_MODULE (10)) (type: tracepoint) [enabled] kernel_spiel:warn (loglevel <= TRACE_WARNING (4)) (type: tracepoint) [enabled] kernel_spiel:debug (loglevel <= TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] mproxy* (loglevel == TRACE_DEBUG_MODULE (10)) (type: tracepoint) [enabled] vrrp:intf_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] ifmgr:ifmgr_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] adj:adj_ipv6 (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] adj:adj_ipv4 (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] ifmgr:if_state (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] adj:startup_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - bgp-updates: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: bgp:bgp_rpol_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_nlri_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_attrs_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_rib_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_tunnel_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_import_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_updatein_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_updateout_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_export_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - dpal-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: bcm* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] lxk* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] dpal* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - fib-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: fib* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - rib-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: rib:rib_tunnel_termination_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_sid_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v6_igp_rnh_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_4v4path_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v4_tnl_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v4rt_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_ilm_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v6rt_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:startup_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v4_igp_rnh_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib_6v6path_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v6_tnl_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - l2f-all: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 262144 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 us Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Recording event rules: l2f* (loglevel <= TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] [-- Attachment #3: lttng_status_buster --] [-- Type: application/octet-stream, Size: 13139 bytes --] Tracing session default: [active snapshot] Trace path: === Domain: UST global === Buffer type: per UID Channels: ------------- - fib-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: fib* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - fib-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: fib* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - bgp-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: bgp:bgp_error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - dpal-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: bcm* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] dpal* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] lxk* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - mac-err: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 262144 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: mac* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - mac-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 262144 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: mac* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - bgp-events: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: bgp:bgp_label_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_netlink_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_cleanup_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_nbrall_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_nexthop_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_ha_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_ev_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_nbr_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_info (loglevel <= TRACE_INFO (6)) (type: tracepoint) [enabled] bgp:bgp_warn (loglevel <= TRACE_WARNING (4)) (type: tracepoint) [enabled] - fib-info: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 262144 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: fib* (loglevel == TRACE_DEBUG_PROCESS (9)) (type: tracepoint) [enabled] - stp-all: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: stp* (loglevel <= TRACE_DEBUG_LINE (13)) (type: tracepoint) [enabled] - dpal-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: bcm* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] dpal* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] lxk* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - ldp-all: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: ldp* (loglevel <= TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - other-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: kernel_spiel:warn (loglevel <= TRACE_WARNING (4)) (type: tracepoint) [enabled] adj:nl_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] vrrp:dpal_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] adj:adj_ipv6 (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] vrrp:intf_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] adj:startup_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] kernel_spiel:debug (loglevel <= TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] adj:adj_ipv4 (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rad* (loglevel == TRACE_DEBUG_MODULE (10)) (type: tracepoint) [enabled] dra* (loglevel == TRACE_DEBUG_MODULE (10)) (type: tracepoint) [enabled] mbroker* (loglevel == TRACE_DEBUG_MODULE (10)) (type: tracepoint) [enabled] ifmgr:startup_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] mproxy* (loglevel == TRACE_DEBUG_MODULE (10)) (type: tracepoint) [enabled] ifmgr:ifmgr_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] ifmgr:if_state (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] vrrp:ev_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - other-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: vrrp:error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] mbroker* (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] kernel_spiel:error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] adj:adj_error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] ifmgr:ifmgr_error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - rib-error: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: rib:rib_error (loglevel <= TRACE_ERR (3)) (type: tracepoint) [enabled] - l2rib-all: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: l2rib* (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - l2f-all: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 262144 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: l2f* (loglevel <= TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - bgp-updates: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: bgp:bgp_rib_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_rpol_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_export_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_tunnel_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_updateout_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_attrs_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_nlri_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_import_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] bgp:bgp_updatein_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] - rib-event: [enabled] Attributes: Event-loss mode: overwrite Sub-buffer size: 4194304 bytes Sub-buffer count: 4 Switch timer: inactive Read timer: inactive Monitor timer: 1000000 µs Trace file count: 1 per stream Trace file size: unlimited Output mode: mmap Statistics: None Event rules: rib:rib_v4_tnl_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_ilm_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v6_igp_rnh_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v6_tnl_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:startup_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v6rt_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib_6v6path_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v4rt_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_4v4path_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_v4_igp_rnh_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_tunnel_termination_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] rib:rib_sid_debug (loglevel == TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) [enabled] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: lttng-dev Digest, Vol 203, Issue 7 2025-03-17 6:29 ` lttng-dev Digest, Vol 203, Issue 7 Gour DEV via lttng-dev @ 2025-03-20 20:51 ` Kienan Stewart via lttng-dev 2025-03-21 12:52 ` Gour DEV via lttng-dev 0 siblings, 1 reply; 5+ messages in thread From: Kienan Stewart via lttng-dev @ 2025-03-20 20:51 UTC (permalink / raw) To: Gour DEV, lttng-dev Hi Lakshya, I did some digging around. What you are seeing is the result of the switching to MAP_POPULATE by default in LTTng-UST 2.12[1] in commit 4d4838b ("Use MAP_POPULATE to reduce pagefault when available"). The purpose of this change is to avoid taking page faults which tracing, reducing first-event in a page latency. In the master branch, this feature has been made configurable for users who don't want to pre-populate the pages and would rather take page faults while tracing[2]. Here is an example from LTTng master with map populate per possible CPU: ``` export LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible # Create session, channels, start tracing, and run test app # top -n1 -b | grep -E '(MiB|COMMAND|lttng)' MiB Mem : 32768.0 total, 21883.7 free, 1456.0 used, 9428.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 31312.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 301117 debian 20 0 880176 2760 2760 S 0.0 0.0 0:00.04 lttng-sessiond 301118 debian 20 0 43856 1376 1376 S 0.0 0.0 0:00.01 lttng-runas 301133 debian 20 0 718616 263456 263456 S 0.0 0.8 0:00.17 lttng-consumerd 301135 debian 20 0 6996 0 0 S 0.0 0.0 0:00.05 lttng-runas # cat /proc/$(pgrep lttng-sessiond)/statm lttng-sessiond: 220044 690 690 345 0 29900 0 # pmap $(pgrep lttng-sessiond) | grep total total 880176K # smem -P lttng-sessiond PID User Command Swap USS PSS RSS 301118 debian lttng-sessiond --daemonize 0 344 881 2236 301117 debian lttng-sessiond --daemonize 0 5676 6683 9276 301201 debian /usr/bin/python3 /usr/bin/s 0 8636 10086 12936 # /proc/PID/statm for lttng-consumerd lttng-consumerd: 1749 0 0 129 0 130 0 # pmap lttng-consumerd-pid | grep total total kB 6996 1700 472 # smem -P lttng-consumerd PID User Command Swap USS PSS RSS 301135 debian lttng-consumerd -u --consu 0 280 563 1700 301211 debian /usr/bin/python3 /usr/bin/s 0 10048 11501 14404 301133 debian lttng-consumerd -u --consu 0 262376 263177 265480 # smem -m | grep -i ust /dev/shm/lttng-ust-wait-8-1000 1 4 4 /dev/shm/shm-ust-consumer-301133 1 260756 260756 ``` When using the none policy: ``` # export LTTNG_UST_MAP_POPULATE_POLICY=none # as above Running test app UID 0 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 21875 0 9434 0 0 39 636 1105 2496 0 1 99 0 0 MiB Mem : 32768.0 total, 21875.0 free, 1458.2 used, 9434.7 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 31309.8 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 301616 debian 20 0 880176 2756 2756 S 0.0 0.0 0:00.04 lttng-sessiond 301617 debian 20 0 43856 1392 1392 S 0.0 0.0 0:00.01 lttng-runas 301632 debian 20 0 718612 5416 5416 S 0.0 0.0 0:00.17 lttng-consumerd 301634 debian 20 0 6992 0 0 S 0.0 0.0 0:00.05 lttng-runas lttng-sessiond: 220044 689 689 345 0 29900 0 total 880176K PID User Command Swap USS PSS RSS 301617 debian lttng-sessiond --daemonize 0 344 862 2188 301616 debian lttng-sessiond --daemonize 0 5784 6759 9328 301700 debian /usr/bin/python3 /usr/bin/s 0 8632 10079 12928 lttng-consumerd: 1748 0 0 129 0 129 0 total kB 6992 1580 468 PID User Command Swap USS PSS RSS 301634 debian lttng-consumerd -u --consu 0 276 536 1580 301632 debian lttng-consumerd -u --consu 0 5672 6433 8652 301710 debian /usr/bin/python3 /usr/bin/s 0 9996 11449 14328 /dev/shm/lttng-ust-wait-8-1000 1 4 4 /dev/shm/shm-ust-consumer-301632 1 4048 4048 ``` thanks, kienan [1]: https://github.com/lttng/lttng-ust/commit/4d4838bad480d48424bddc686f5ad0089e28ac94 [2]: https://github.com/lttng/lttng-ust/commit/97572c0438845cee953ebd3e39615f78bfa405a7 On 3/17/25 2:29 AM, Gour DEV wrote: > Hi, Kienan > > Sorry for the late reply. > > Looks like in buster the memory is allocated by lttng-consumerd reserved > > I buster, the rss is less than the VIRT > root@localhost:~# COLUMNS=500 top -b -n 1 | grep lttng > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 4095 root 20 0 1003188 31256 4660 S 0.0 0.1 0:03.81 > lttng-sessiond > 4096 root 20 0 44260 796 0 S 0.0 0.0 0:00.01 > lttng-runas > 4440 root 20 0 5236020 10224 8756 S 0.0 0.0 2:56.25 > lttng-consumerd -- here the VIRT is much more higher than RSS > 4443 root 20 0 48048 540 0 S 0.0 0.0 0:00.12 > lttng-runas > > > > In bookworm the VIRT and RES are nearly the same only. > root@edgecore-40XKE-j2-101-32:~# COLUMNS=500 top -b -n 1 | grep lttng > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 4382 root 20 0 1098824 42600 8436 S 0.0 0.1 0:08.87 > lttng-sessiond > 4403 root 20 0 48928 2116 996 S 0.0 0.0 0:00.00 > lttng-runas > 5171 root 20 0 9879764 8.9g 8.9g S 0.0 28.7 108:23.53 > lttng-consumerd -- here the VRIT is nearly equal to RSS > 5173 root 20 0 3680 1028 680 S 0.0 0.0 0:00.88 > lttng-runas > > > Looks like lttng consumerd is allocating and reserving those pages, when > any instrumented application starts. > > I am attaching the lttng status output in the mail, please do tell me if > you need any more information regarding this. > > > These is how we used to create the lttng channels and enable event which is > same in both buster and bookworm, (number of channels might differ) > > def enable_channel(channels, session, subbuf_size, subbuf_num): > for c in channels: > call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size', > str(subbuf_size), '--num-subbuf', str(subbuf_num),], > stdout=devnull, stderr=subprocess.STDOUT) > > > def enable_events(traces, session): > for t in traces: > if 'log-level-only' in t: > log_opt = '--loglevel-only=' + t['log-level-only'] > elif 'log-level' in t: > log_opt = '--loglevel=' + t['log-level'] > else: > log_opt = '' > > else: > call(['lttng', 'enable-event', '-u', t['name'], '-c', t['channel'], > '-s', session], stdout=devnull, stderr=subprocess.STDOUT) > > > Thank You. > Lakshya > > > > > On Wed, Mar 12, 2025 at 8:06 PM <lttng-dev-request@lists.lttng.org> wrote: > >> Send lttng-dev mailing list submissions to >> lttng-dev@lists.lttng.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> or, via email, send a message with subject or body 'help' to >> lttng-dev-request@lists.lttng.org >> >> You can reach the person managing the list at >> lttng-dev-owner@lists.lttng.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of lttng-dev digest..." >> >> >> Today's Topics: >> >> 1. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >> (Kienan Stewart) >> 2. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >> (Gour DEV) >> 3. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >> (Kienan Stewart) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 11 Mar 2025 14:55:21 -0400 >> From: Kienan Stewart <kstewart@efficios.com> >> To: Gour DEV <lakshyagour10@gmail.com>, lttng-dev@lists.lttng.org >> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >> Message-ID: <38dab5ef-f106-4e57-9e36-b4b30015c019@efficios.com> >> Content-Type: text/plain; charset=UTF-8; format=flowed >> >> Hi Lakshya, >> >> On 3/11/25 12:25 PM, Gour DEV wrote: >> > Hi, Kienan >> > >> > here is the requested output >> > >> > root@localhost:~# top -b -n 1 | grep lttng >> > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 8:39.93 >> > lttng-c+ >> > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 0:07.25 >> > lttng-s+ >> > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 >> > lttng-r+ >> > 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 >> >> This top output for `localhost` seems very different than the output for >> `localhost` in your previous message. >> >> >> > lttng-r+ >> > root@localhost:~# nrpco >> > bash: nrpco: command not found >> > root@localhost:~# nproc >> > 16 >> > root@localhost:~# cat /sys/devices/system/cpu/possible >> > 0-15 >> > >> >> You indicated the bookworm machine has 32 cores, this is showing 16. If >> you're comparing a 16 core machine to a 32 core machine, it is very >> normal that the memory usage is higher on the 32 core machine. >> >> > >> > Most of the process are running as asorcs user but some are running >> as root. >> >> So you have two users with instrumented applications. >> >> >> Given the discrepancies in the information provided I'm finding it a bit >> hard to understand what you're looking at. >> >> >> In general, a channel's shared memory footprint can be estimated with[1]: >> >> (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * >> (nUIDs or nPIDs) >> >> Note that the sub-buffer sizes you are using get rounded to the nearest >> larger power of 2. See [2]. >> >> thanks, >> kienan >> >> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes >> [2]: >> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size >> >> >> ------------------------------ >> >> Message: 2 >> Date: Wed, 12 Mar 2025 14:49:07 +0530 >> From: Gour DEV <lakshyagour10@gmail.com> >> To: Kienan Stewart <kstewart@efficios.com> >> Cc: lttng-dev@lists.lttng.org >> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >> Message-ID: >> <CAE9Jrzg7qsabhPiO-0=B1DY3bVo-3FYu2tiJR2Bmb= >> nqOHNZMw@mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hi, Kienan >> >> I am attaching an screen recording of the behaviour I am seeing in this >> mail. The behaviour is same irrespective of the device i use, sorry for >> miscommunication in the npocs output (I assumed it was 32), but other than >> that all outputs are same (except the hostname as there are multiple >> devices with same lttng config but this memory cosumption is seen on all >> the devices). >> >> I had few question >> >> 1. Does lltng allocated all the memory it needs and mark it as dirty in ram >> when any process which links/uses lttng-ust runs? (here i tried with one >> process but it is same for any of my process) >> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * >> (nUIDs or nPIDs) >> >> How do we calculate uid in the system is it all uids in the system? is it >> equal to `cat /etc/passwd | wc -l` ? >> >> I will put my calculations according to the above estimate based on all the >> channel i am creating >> >> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to >> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the >> start_lttng.py please do correct me if am wrong here.] >> >> But since there are only two users which uses lttng i think the correct >> estimate would be >> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB >> >> Please do correct me If I am wrong calculations here. >> >> Now, there are a few things here, according to my output lttng is using 11G >> which is much more higher than the what is configured. >> >> I am attaching the lttng status and the file which is uses to create the >> lttng sessions. >> >> >> >> Thank You. >> >> >> >> https://drive.google.com/file/d/1tS_ZWEsXDpHZXfWzZHXmWcT0igiIOIaa/view?usp=sharing >> -- recording of the behaviour which is seen >> >> https://drive.google.com/file/d/1PrU31oyEw1n9tKETlUtmNGO50s6ywx7p/view?usp=sharing >> -- the file which is used to create lttng sessions >> >> >> On Wed, Mar 12, 2025 at 12:25?AM Kienan Stewart <kstewart@efficios.com> >> wrote: >> >>> Hi Lakshya, >>> >>> On 3/11/25 12:25 PM, Gour DEV wrote: >>> > Hi, Kienan >>> > >>> > here is the requested output >>> > >>> > root@localhost:~# top -b -n 1 | grep lttng >>> > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 >> 8:39.93 >>> > lttng-c+ >>> > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 >> 0:07.25 >>> > lttng-s+ >>> > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 >> 0:00.00 >>> > lttng-r+ >>> > 4843 root 20 0 3680 1160 816 S 0.0 0.0 >> 0:00.23 >>> >>> This top output for `localhost` seems very different than the output for >>> `localhost` in your previous message. >>> >>> >>> > lttng-r+ >>> > root@localhost:~# nrpco >>> > bash: nrpco: command not found >>> > root@localhost:~# nproc >>> > 16 >>> > root@localhost:~# cat /sys/devices/system/cpu/possible >>> > 0-15 >>> > >>> >>> You indicated the bookworm machine has 32 cores, this is showing 16. If >>> you're comparing a 16 core machine to a 32 core machine, it is very >>> normal that the memory usage is higher on the 32 core machine. >>> >>> > >>> > Most of the process are running as asorcs user but some are running >>> as root. >>> >>> So you have two users with instrumented applications. >>> >>> >>> Given the discrepancies in the information provided I'm finding it a bit >>> hard to understand what you're looking at. >>> >>> >>> In general, a channel's shared memory footprint can be estimated with[1]: >>> >>> (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * >>> (nUIDs or nPIDs) >>> >>> Note that the sub-buffer sizes you are using get rounded to the nearest >>> larger power of 2. See [2]. >>> >>> thanks, >>> kienan >>> >>> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes >>> [2]: >>> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size >>> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: < >> https://lists.lttng.org/pipermail/lttng-dev/attachments/20250312/57f240d8/attachment-0001.htm >>> >> >> ------------------------------ >> >> Message: 3 >> Date: Wed, 12 Mar 2025 10:36:28 -0400 >> From: Kienan Stewart <kstewart@efficios.com> >> To: Gour DEV <lakshyagour10@gmail.com>, lttng-dev@lists.lttng.org >> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >> Message-ID: <0f819583-ea8e-468e-9102-e1410d886a6f@efficios.com> >> Content-Type: text/plain; charset=UTF-8; format=flowed >> >> Hi Lakshya, >> >> On 3/12/25 5:03 AM, Gour DEV wrote: >>> Hi, Kienan >>> >>> I am attaching an screen recording of the behaviour I am seeing in this >>> mail. The behaviour is same irrespective of the device i use, sorry for >>> miscommunication in the npocs output (I assumed it was 32), but other >> than >>> that all outputs are same (except the hostname as there are multiple >>> devices with same lttng config but this memory cosumption is seen on all >>> the devices). >>> >>> I had few question >>> >>> 1. Does lltng allocated all the memory it needs and mark it as dirty in >> ram >>> when any process which links/uses lttng-ust runs? (here i tried with one >>> process but it is same for any of my process) >> >> I believe the shared memory for per-CPU data structures is allocated >> when an instrumented application connects. There is no pre-allocation >> for each possible UID on the system. >> >> You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to >> see when the connection happens[1]. >> >>> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * >>> (nUIDs or nPIDs) >>> >>> How do we calculate uid in the system is it all uids in the system? is it >>> equal to `cat /etc/passwd | wc -l` ? >> >> nUIDs is the number of distinct UIDs running instrumented applications. >> >>> >>> I will put my calculations according to the above estimate based on all >> the >>> channel i am creating >>> >>> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to >>> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the >>> start_lttng.py please do correct me if am wrong here.] >>> >>> But since there are only two users which uses lttng i think the correct >>> estimate would be >>> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB >> >> The estimate I gave is per-channel. >> >> small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel >> per-UID >> medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel per-UID >> large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID >> >> Now, you said you have 0 small channels, 6 medium channels, and 16 large >> channels in your session. (Note: I see your script differs from these >> stated channel counts). >> >> small: 0 * 1.0625 MiB = 0 MiB per-UID >> medium: 6 * 17 MiB = 102 MiB per-UID >> large: 16 * 272 MiB = 4352 MiB per-UID >> >> And if you're running instrumented applications with 2 users: >> >> small: 0 MiB * 2 = 0 MiB with 2 UIDs >> medium: 102 MiB * 2 = 204 MiB with 2 UIDs >> large: 4352 MiB * 2 = 8704 MiB with 2 UIDs >> >> Now this is just an estimation for the per-CPU ring buffers only, and >> you numbers aren't hugely off so without analyzing your specific system >> it doesn't seem to be that strange to me. >> >> If I take the number of channels I see in your script, it becomes: >> >> small: 0 MiB with 2 UIDs >> medium: 136 MiB with 2 UIDs >> large: 7616 MiB with 2 UIDs >> >> total: 7.57 GiB with 2 UIDs >> >>> >>> Please do correct me If I am wrong calculations here. >>> >>> Now, there are a few things here, according to my output lttng is using >> 11G >>> which is much more higher than the what is configured. >>> >> >> I have no idea what 'service start spyder' is doing. Maybe it's running >> instrumented applications with an extra user that you didn't expect? I >> can't help you with that aspect of your system. >> >> The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs >> so maybe? >> >> I'd recommend you read your verbose sessiond log so see which >> applications are connecting and with which UIDs. >> >>> I am attaching the lttng status and the file which is uses to create the >>> lttng sessions. >>> >>> >>> >>> Thank You. >>> >> >> In any case, the information you have given to date hasn't demonstrated >> to me in a tangible manner that you are seeing a difference related to >> the version of LTTng being used. >> >> thanks, >> kienan >> >> [1]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables >> >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev@lists.lttng.org >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> >> ------------------------------ >> >> End of lttng-dev Digest, Vol 203, Issue 7 >> ***************************************** >> > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: lttng-dev Digest, Vol 203, Issue 7 2025-03-20 20:51 ` Kienan Stewart via lttng-dev @ 2025-03-21 12:52 ` Gour DEV via lttng-dev 2025-03-21 13:14 ` Kienan Stewart via lttng-dev 0 siblings, 1 reply; 5+ messages in thread From: Gour DEV via lttng-dev @ 2025-03-21 12:52 UTC (permalink / raw) To: Kienan Stewart, lttng-dev [-- Attachment #1: Type: text/plain, Size: 22781 bytes --] Thanks Kienan for digging this and explaining the behaviour, Looks like this option is not present in the version I am using v2.13.5 and the MAP_POPULATE is enabled by default there /* memory_map: mmap */ memory_map = mmap(NULL, memory_map_size, PROT_READ | PROT_WRITE, MAP_SHARED | LTTNG_MAP_POPULATE, shmfd, 0); I have updated to the master branch and have verified by setting the LTTNG_UST_MAP_POPULATE_POLICY=none, and saw that now the RSS consumption is not very high now. I think using LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible is better because here I will definitely know how much amount of memory lttng will use at start only, and allocate memory accordingly to my servers. Also, I thought using --buffers-global <https://lttng.org/man/1/lttng-enable-channel/v2.8/#doc-opt--buffers-global> would be better for me as I could know how much memory will be required but looks like that option is only available for kernel tracking, it would be very good if this option could be available for user buffering also, but that could come later. But again, thanks for the help Kienan. Regards Lakshya. On Fri, Mar 21, 2025 at 2:21 AM Kienan Stewart <kstewart@efficios.com> wrote: > Hi Lakshya, > > I did some digging around. What you are seeing is the result of the > switching to MAP_POPULATE by default in LTTng-UST 2.12[1] in commit > 4d4838b ("Use MAP_POPULATE to reduce pagefault when available"). > > The purpose of this change is to avoid taking page faults which tracing, > reducing first-event in a page latency. > > In the master branch, this feature has been made configurable for users > who don't want to pre-populate the pages and would rather take page > faults while tracing[2]. > > Here is an example from LTTng master with map populate per possible CPU: > > ``` > export LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible > > # Create session, channels, start tracing, and run test app > # top -n1 -b | grep -E '(MiB|COMMAND|lttng)' > MiB Mem : 32768.0 total, 21883.7 free, 1456.0 used, 9428.3 > buff/cache > > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 31312.0 avail > Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 301117 debian 20 0 880176 2760 2760 S 0.0 0.0 0:00.04 > lttng-sessiond > > 301118 debian 20 0 43856 1376 1376 S 0.0 0.0 0:00.01 > lttng-runas > > 301133 debian 20 0 718616 263456 263456 S 0.0 0.8 0:00.17 > lttng-consumerd > > 301135 debian 20 0 6996 0 0 S 0.0 0.0 0:00.05 > lttng-runas > > # cat /proc/$(pgrep lttng-sessiond)/statm > lttng-sessiond: 220044 690 690 345 0 29900 0 > > > > # pmap $(pgrep lttng-sessiond) | grep total > > total 880176K > > # smem -P lttng-sessiond > > > PID User Command Swap USS PSS > RSS > > 301118 debian lttng-sessiond --daemonize 0 344 881 > 2236 > > 301117 debian lttng-sessiond --daemonize 0 5676 6683 > 9276 > > 301201 debian /usr/bin/python3 /usr/bin/s 0 8636 10086 > 12936 > > > # /proc/PID/statm for lttng-consumerd > > > > lttng-consumerd: 1749 0 0 129 0 130 0 > > # pmap lttng-consumerd-pid | grep total > total kB 6996 1700 472 > > > > # smem -P lttng-consumerd > PID User Command Swap USS PSS > RSS > > 301135 debian lttng-consumerd -u --consu 0 280 563 > 1700 > > 301211 debian /usr/bin/python3 /usr/bin/s 0 10048 11501 > 14404 > > 301133 debian lttng-consumerd -u --consu 0 262376 263177 > 265480 > > # smem -m | grep -i ust > > /dev/shm/lttng-ust-wait-8-1000 1 4 4 > /dev/shm/shm-ust-consumer-301133 1 260756 260756 > ``` > > When using the none policy: > > ``` > # export LTTNG_UST_MAP_POPULATE_POLICY=none > # as above > > Running test app UID 0 > > > procs -----------memory---------- ---swap-- -----io---- -system-- > ------cpu----- > > r b swpd free buff cache si so bi bo in cs us sy > id wa st > > 1 0 0 21875 0 9434 0 0 39 636 1105 2496 0 1 > 99 0 0 > > MiB Mem : 32768.0 total, 21875.0 free, 1458.2 used, 9434.7 > buff/cache > > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 31309.8 avail > Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 301616 debian 20 0 880176 2756 2756 S 0.0 0.0 0:00.04 > lttng-sessiond > > 301617 debian 20 0 43856 1392 1392 S 0.0 0.0 0:00.01 > lttng-runas > > 301632 debian 20 0 718612 5416 5416 S 0.0 0.0 0:00.17 > lttng-consumerd > > 301634 debian 20 0 6992 0 0 S 0.0 0.0 0:00.05 > lttng-runas > > lttng-sessiond: 220044 689 689 345 0 29900 0 > > > total 880176K > > > PID User Command Swap USS PSS > RSS > 301617 debian lttng-sessiond --daemonize 0 344 862 > 2188 > 301616 debian lttng-sessiond --daemonize 0 5784 6759 > 9328 > 301700 debian /usr/bin/python3 /usr/bin/s 0 8632 10079 > 12928 > > lttng-consumerd: 1748 0 0 129 0 129 0 > total kB 6992 1580 468 > PID User Command Swap USS PSS > RSS > 301634 debian lttng-consumerd -u --consu 0 276 536 > 1580 > 301632 debian lttng-consumerd -u --consu 0 5672 6433 > 8652 > 301710 debian /usr/bin/python3 /usr/bin/s 0 9996 11449 > 14328 > > /dev/shm/lttng-ust-wait-8-1000 1 4 4 > /dev/shm/shm-ust-consumer-301632 1 4048 4048 > ``` > > thanks, > kienan > > [1]: > > https://github.com/lttng/lttng-ust/commit/4d4838bad480d48424bddc686f5ad0089e28ac94 > [2]: > > https://github.com/lttng/lttng-ust/commit/97572c0438845cee953ebd3e39615f78bfa405a7 > > On 3/17/25 2:29 AM, Gour DEV wrote: > > Hi, Kienan > > > > Sorry for the late reply. > > > > Looks like in buster the memory is allocated by lttng-consumerd reserved > > > > I buster, the rss is less than the VIRT > > root@localhost:~# COLUMNS=500 top -b -n 1 | grep lttng > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 4095 root 20 0 1003188 31256 4660 S 0.0 0.1 0:03.81 > > lttng-sessiond > > 4096 root 20 0 44260 796 0 S 0.0 0.0 0:00.01 > > lttng-runas > > 4440 root 20 0 5236020 10224 8756 S 0.0 0.0 2:56.25 > > lttng-consumerd -- here the VIRT is much more higher than RSS > > 4443 root 20 0 48048 540 0 S 0.0 0.0 0:00.12 > > lttng-runas > > > > > > > > In bookworm the VIRT and RES are nearly the same only. > > root@edgecore-40XKE-j2-101-32:~# COLUMNS=500 top -b -n 1 | grep lttng > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 4382 root 20 0 1098824 42600 8436 S 0.0 0.1 0:08.87 > > lttng-sessiond > > 4403 root 20 0 48928 2116 996 S 0.0 0.0 0:00.00 > > lttng-runas > > 5171 root 20 0 9879764 8.9g 8.9g S 0.0 28.7 108:23.53 > > lttng-consumerd -- here the VRIT is nearly equal to RSS > > 5173 root 20 0 3680 1028 680 S 0.0 0.0 0:00.88 > > lttng-runas > > > > > > Looks like lttng consumerd is allocating and reserving those pages, when > > any instrumented application starts. > > > > I am attaching the lttng status output in the mail, please do tell me if > > you need any more information regarding this. > > > > > > These is how we used to create the lttng channels and enable event which > is > > same in both buster and bookworm, (number of channels might differ) > > > > def enable_channel(channels, session, subbuf_size, subbuf_num): > > for c in channels: > > call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size', > > str(subbuf_size), '--num-subbuf', str(subbuf_num),], > > stdout=devnull, stderr=subprocess.STDOUT) > > > > > > def enable_events(traces, session): > > for t in traces: > > if 'log-level-only' in t: > > log_opt = '--loglevel-only=' + t['log-level-only'] > > elif 'log-level' in t: > > log_opt = '--loglevel=' + t['log-level'] > > else: > > log_opt = '' > > > > else: > > call(['lttng', 'enable-event', '-u', t['name'], '-c', t['channel'], > > '-s', session], stdout=devnull, stderr=subprocess.STDOUT) > > > > > > Thank You. > > Lakshya > > > > > > > > > > On Wed, Mar 12, 2025 at 8:06 PM <lttng-dev-request@lists.lttng.org> > wrote: > > > >> Send lttng-dev mailing list submissions to > >> lttng-dev@lists.lttng.org > >> > >> To subscribe or unsubscribe via the World Wide Web, visit > >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > >> or, via email, send a message with subject or body 'help' to > >> lttng-dev-request@lists.lttng.org > >> > >> You can reach the person managing the list at > >> lttng-dev-owner@lists.lttng.org > >> > >> When replying, please edit your Subject line so it is more specific > >> than "Re: Contents of lttng-dev digest..." > >> > >> > >> Today's Topics: > >> > >> 1. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > >> (Kienan Stewart) > >> 2. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > >> (Gour DEV) > >> 3. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > >> (Kienan Stewart) > >> > >> > >> ---------------------------------------------------------------------- > >> > >> Message: 1 > >> Date: Tue, 11 Mar 2025 14:55:21 -0400 > >> From: Kienan Stewart <kstewart@efficios.com> > >> To: Gour DEV <lakshyagour10@gmail.com>, lttng-dev@lists.lttng.org > >> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > >> Message-ID: <38dab5ef-f106-4e57-9e36-b4b30015c019@efficios.com> > >> Content-Type: text/plain; charset=UTF-8; format=flowed > >> > >> Hi Lakshya, > >> > >> On 3/11/25 12:25 PM, Gour DEV wrote: > >> > Hi, Kienan > >> > > >> > here is the requested output > >> > > >> > root@localhost:~# top -b -n 1 | grep lttng > >> > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 > 8:39.93 > >> > lttng-c+ > >> > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 > 0:07.25 > >> > lttng-s+ > >> > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 > 0:00.00 > >> > lttng-r+ > >> > 4843 root 20 0 3680 1160 816 S 0.0 0.0 > 0:00.23 > >> > >> This top output for `localhost` seems very different than the output for > >> `localhost` in your previous message. > >> > >> > >> > lttng-r+ > >> > root@localhost:~# nrpco > >> > bash: nrpco: command not found > >> > root@localhost:~# nproc > >> > 16 > >> > root@localhost:~# cat /sys/devices/system/cpu/possible > >> > 0-15 > >> > > >> > >> You indicated the bookworm machine has 32 cores, this is showing 16. If > >> you're comparing a 16 core machine to a 32 core machine, it is very > >> normal that the memory usage is higher on the 32 core machine. > >> > >> > > >> > Most of the process are running as asorcs user but some are running > >> as root. > >> > >> So you have two users with instrumented applications. > >> > >> > >> Given the discrepancies in the information provided I'm finding it a bit > >> hard to understand what you're looking at. > >> > >> > >> In general, a channel's shared memory footprint can be estimated > with[1]: > >> > >> (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * > >> (nUIDs or nPIDs) > >> > >> Note that the sub-buffer sizes you are using get rounded to the nearest > >> larger power of 2. See [2]. > >> > >> thanks, > >> kienan > >> > >> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes > >> [2]: > >> > https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size > >> > >> > >> ------------------------------ > >> > >> Message: 2 > >> Date: Wed, 12 Mar 2025 14:49:07 +0530 > >> From: Gour DEV <lakshyagour10@gmail.com> > >> To: Kienan Stewart <kstewart@efficios.com> > >> Cc: lttng-dev@lists.lttng.org > >> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > >> Message-ID: > >> <CAE9Jrzg7qsabhPiO-0=B1DY3bVo-3FYu2tiJR2Bmb= > >> nqOHNZMw@mail.gmail.com> > >> Content-Type: text/plain; charset="utf-8" > >> > >> Hi, Kienan > >> > >> I am attaching an screen recording of the behaviour I am seeing in this > >> mail. The behaviour is same irrespective of the device i use, sorry for > >> miscommunication in the npocs output (I assumed it was 32), but other > than > >> that all outputs are same (except the hostname as there are multiple > >> devices with same lttng config but this memory cosumption is seen on all > >> the devices). > >> > >> I had few question > >> > >> 1. Does lltng allocated all the memory it needs and mark it as dirty in > ram > >> when any process which links/uses lttng-ust runs? (here i tried with one > >> process but it is same for any of my process) > >> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * > >> (nUIDs or nPIDs) > >> > >> How do we calculate uid in the system is it all uids in the system? is > it > >> equal to `cat /etc/passwd | wc -l` ? > >> > >> I will put my calculations according to the above estimate based on all > the > >> channel i am creating > >> > >> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal > to > >> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on > the > >> start_lttng.py please do correct me if am wrong here.] > >> > >> But since there are only two users which uses lttng i think the correct > >> estimate would be > >> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB > >> > >> Please do correct me If I am wrong calculations here. > >> > >> Now, there are a few things here, according to my output lttng is using > 11G > >> which is much more higher than the what is configured. > >> > >> I am attaching the lttng status and the file which is uses to create the > >> lttng sessions. > >> > >> > >> > >> Thank You. > >> > >> > >> > >> > https://drive.google.com/file/d/1tS_ZWEsXDpHZXfWzZHXmWcT0igiIOIaa/view?usp=sharing > >> -- recording of the behaviour which is seen > >> > >> > https://drive.google.com/file/d/1PrU31oyEw1n9tKETlUtmNGO50s6ywx7p/view?usp=sharing > >> -- the file which is used to create lttng sessions > >> > >> > >> On Wed, Mar 12, 2025 at 12:25?AM Kienan Stewart <kstewart@efficios.com> > >> wrote: > >> > >>> Hi Lakshya, > >>> > >>> On 3/11/25 12:25 PM, Gour DEV wrote: > >>> > Hi, Kienan > >>> > > >>> > here is the requested output > >>> > > >>> > root@localhost:~# top -b -n 1 | grep lttng > >>> > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 > >> 8:39.93 > >>> > lttng-c+ > >>> > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 > >> 0:07.25 > >>> > lttng-s+ > >>> > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 > >> 0:00.00 > >>> > lttng-r+ > >>> > 4843 root 20 0 3680 1160 816 S 0.0 0.0 > >> 0:00.23 > >>> > >>> This top output for `localhost` seems very different than the output > for > >>> `localhost` in your previous message. > >>> > >>> > >>> > lttng-r+ > >>> > root@localhost:~# nrpco > >>> > bash: nrpco: command not found > >>> > root@localhost:~# nproc > >>> > 16 > >>> > root@localhost:~# cat /sys/devices/system/cpu/possible > >>> > 0-15 > >>> > > >>> > >>> You indicated the bookworm machine has 32 cores, this is showing 16. If > >>> you're comparing a 16 core machine to a 32 core machine, it is very > >>> normal that the memory usage is higher on the 32 core machine. > >>> > >>> > > >>> > Most of the process are running as asorcs user but some are running > >>> as root. > >>> > >>> So you have two users with instrumented applications. > >>> > >>> > >>> Given the discrepancies in the information provided I'm finding it a > bit > >>> hard to understand what you're looking at. > >>> > >>> > >>> In general, a channel's shared memory footprint can be estimated > with[1]: > >>> > >>> (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * > >>> (nUIDs or nPIDs) > >>> > >>> Note that the sub-buffer sizes you are using get rounded to the nearest > >>> larger power of 2. See [2]. > >>> > >>> thanks, > >>> kienan > >>> > >>> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes > >>> [2]: > >>> > https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size > >>> > >> -------------- next part -------------- > >> An HTML attachment was scrubbed... > >> URL: < > >> > https://lists.lttng.org/pipermail/lttng-dev/attachments/20250312/57f240d8/attachment-0001.htm > >>> > >> > >> ------------------------------ > >> > >> Message: 3 > >> Date: Wed, 12 Mar 2025 10:36:28 -0400 > >> From: Kienan Stewart <kstewart@efficios.com> > >> To: Gour DEV <lakshyagour10@gmail.com>, lttng-dev@lists.lttng.org > >> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 > >> Message-ID: <0f819583-ea8e-468e-9102-e1410d886a6f@efficios.com> > >> Content-Type: text/plain; charset=UTF-8; format=flowed > >> > >> Hi Lakshya, > >> > >> On 3/12/25 5:03 AM, Gour DEV wrote: > >>> Hi, Kienan > >>> > >>> I am attaching an screen recording of the behaviour I am seeing in this > >>> mail. The behaviour is same irrespective of the device i use, sorry for > >>> miscommunication in the npocs output (I assumed it was 32), but other > >> than > >>> that all outputs are same (except the hostname as there are multiple > >>> devices with same lttng config but this memory cosumption is seen on > all > >>> the devices). > >>> > >>> I had few question > >>> > >>> 1. Does lltng allocated all the memory it needs and mark it as dirty in > >> ram > >>> when any process which links/uses lttng-ust runs? (here i tried with > one > >>> process but it is same for any of my process) > >> > >> I believe the shared memory for per-CPU data structures is allocated > >> when an instrumented application connects. There is no pre-allocation > >> for each possible UID on the system. > >> > >> You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to > >> see when the connection happens[1]. > >> > >>> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * > >>> (nUIDs or nPIDs) > >>> > >>> How do we calculate uid in the system is it all uids in the system? is > it > >>> equal to `cat /etc/passwd | wc -l` ? > >> > >> nUIDs is the number of distinct UIDs running instrumented applications. > >> > >>> > >>> I will put my calculations according to the above estimate based on all > >> the > >>> channel i am creating > >>> > >>> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal > to > >>> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on > the > >>> start_lttng.py please do correct me if am wrong here.] > >>> > >>> But since there are only two users which uses lttng i think the correct > >>> estimate would be > >>> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB > >> > >> The estimate I gave is per-channel. > >> > >> small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel > >> per-UID > >> medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel > per-UID > >> large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID > >> > >> Now, you said you have 0 small channels, 6 medium channels, and 16 large > >> channels in your session. (Note: I see your script differs from these > >> stated channel counts). > >> > >> small: 0 * 1.0625 MiB = 0 MiB per-UID > >> medium: 6 * 17 MiB = 102 MiB per-UID > >> large: 16 * 272 MiB = 4352 MiB per-UID > >> > >> And if you're running instrumented applications with 2 users: > >> > >> small: 0 MiB * 2 = 0 MiB with 2 UIDs > >> medium: 102 MiB * 2 = 204 MiB with 2 UIDs > >> large: 4352 MiB * 2 = 8704 MiB with 2 UIDs > >> > >> Now this is just an estimation for the per-CPU ring buffers only, and > >> you numbers aren't hugely off so without analyzing your specific system > >> it doesn't seem to be that strange to me. > >> > >> If I take the number of channels I see in your script, it becomes: > >> > >> small: 0 MiB with 2 UIDs > >> medium: 136 MiB with 2 UIDs > >> large: 7616 MiB with 2 UIDs > >> > >> total: 7.57 GiB with 2 UIDs > >> > >>> > >>> Please do correct me If I am wrong calculations here. > >>> > >>> Now, there are a few things here, according to my output lttng is using > >> 11G > >>> which is much more higher than the what is configured. > >>> > >> > >> I have no idea what 'service start spyder' is doing. Maybe it's running > >> instrumented applications with an extra user that you didn't expect? I > >> can't help you with that aspect of your system. > >> > >> The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs > >> so maybe? > >> > >> I'd recommend you read your verbose sessiond log so see which > >> applications are connecting and with which UIDs. > >> > >>> I am attaching the lttng status and the file which is uses to create > the > >>> lttng sessions. > >>> > >>> > >>> > >>> Thank You. > >>> > >> > >> In any case, the information you have given to date hasn't demonstrated > >> to me in a tangible manner that you are seeing a difference related to > >> the version of LTTng being used. > >> > >> thanks, > >> kienan > >> > >> [1]: > https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables > >> > >> > >> ------------------------------ > >> > >> Subject: Digest Footer > >> > >> _______________________________________________ > >> lttng-dev mailing list > >> lttng-dev@lists.lttng.org > >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > >> > >> > >> ------------------------------ > >> > >> End of lttng-dev Digest, Vol 203, Issue 7 > >> ***************************************** > >> > > > > [-- Attachment #2: Type: text/html, Size: 32238 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: lttng-dev Digest, Vol 203, Issue 7 2025-03-21 12:52 ` Gour DEV via lttng-dev @ 2025-03-21 13:14 ` Kienan Stewart via lttng-dev 2025-03-21 14:10 ` Olivier Dion via lttng-dev 0 siblings, 1 reply; 5+ messages in thread From: Kienan Stewart via lttng-dev @ 2025-03-21 13:14 UTC (permalink / raw) To: Gour DEV, lttng-dev Hi Lakshya, On 3/21/25 8:52 AM, Gour DEV wrote: > Thanks Kienan for digging this and explaining the behaviour, > > Looks like this option is not present in the version I am using v2.13.5 and > the MAP_POPULATE is enabled by default there > /* memory_map: mmap */ > memory_map = mmap(NULL, memory_map_size, PROT_READ | PROT_WRITE, > MAP_SHARED | LTTNG_MAP_POPULATE, shmfd, 0); > > I have updated to the master branch and have verified by setting the > LTTNG_UST_MAP_POPULATE_POLICY=none, and saw that now the RSS consumption is > not very high now. > > I think using LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible is better because > here I will definitely know how much amount of memory lttng will use at > start only, and allocate memory accordingly to my servers. > > Also, I thought using --buffers-global > <https://lttng.org/man/1/lttng-enable-channel/v2.8/#doc-opt--buffers-global> > would > be better for me as I could know how much memory will be required but looks > like that option is only available for kernel tracking, it would be very > good if this option could be available for user buffering also, but that > could come later. > Global buffers for UST is planned for the next release. The trade-off of using it is that tracing overhead will increase (as the buffers are no longer per-CPU), but in exchange for a potentially lower memory footprint particularly in situations where not all possible CPUs are online. > > But again, thanks for the help Kienan. > My pleasure! thanks, kienan > Regards > Lakshya. > > > > > > > On Fri, Mar 21, 2025 at 2:21 AM Kienan Stewart <kstewart@efficios.com> > wrote: > >> Hi Lakshya, >> >> I did some digging around. What you are seeing is the result of the >> switching to MAP_POPULATE by default in LTTng-UST 2.12[1] in commit >> 4d4838b ("Use MAP_POPULATE to reduce pagefault when available"). >> >> The purpose of this change is to avoid taking page faults which tracing, >> reducing first-event in a page latency. >> >> In the master branch, this feature has been made configurable for users >> who don't want to pre-populate the pages and would rather take page >> faults while tracing[2]. >> >> Here is an example from LTTng master with map populate per possible CPU: >> >> ``` >> export LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible >> >> # Create session, channels, start tracing, and run test app >> # top -n1 -b | grep -E '(MiB|COMMAND|lttng)' >> MiB Mem : 32768.0 total, 21883.7 free, 1456.0 used, 9428.3 >> buff/cache >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 31312.0 avail >> Mem >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> >> 301117 debian 20 0 880176 2760 2760 S 0.0 0.0 0:00.04 >> lttng-sessiond >> >> 301118 debian 20 0 43856 1376 1376 S 0.0 0.0 0:00.01 >> lttng-runas >> >> 301133 debian 20 0 718616 263456 263456 S 0.0 0.8 0:00.17 >> lttng-consumerd >> >> 301135 debian 20 0 6996 0 0 S 0.0 0.0 0:00.05 >> lttng-runas >> >> # cat /proc/$(pgrep lttng-sessiond)/statm >> lttng-sessiond: 220044 690 690 345 0 29900 0 >> >> >> >> # pmap $(pgrep lttng-sessiond) | grep total >> >> total 880176K >> >> # smem -P lttng-sessiond >> >> >> PID User Command Swap USS PSS >> RSS >> >> 301118 debian lttng-sessiond --daemonize 0 344 881 >> 2236 >> >> 301117 debian lttng-sessiond --daemonize 0 5676 6683 >> 9276 >> >> 301201 debian /usr/bin/python3 /usr/bin/s 0 8636 10086 >> 12936 >> >> >> # /proc/PID/statm for lttng-consumerd >> >> >> >> lttng-consumerd: 1749 0 0 129 0 130 0 >> >> # pmap lttng-consumerd-pid | grep total >> total kB 6996 1700 472 >> >> >> >> # smem -P lttng-consumerd >> PID User Command Swap USS PSS >> RSS >> >> 301135 debian lttng-consumerd -u --consu 0 280 563 >> 1700 >> >> 301211 debian /usr/bin/python3 /usr/bin/s 0 10048 11501 >> 14404 >> >> 301133 debian lttng-consumerd -u --consu 0 262376 263177 >> 265480 >> >> # smem -m | grep -i ust >> >> /dev/shm/lttng-ust-wait-8-1000 1 4 4 >> /dev/shm/shm-ust-consumer-301133 1 260756 260756 >> ``` >> >> When using the none policy: >> >> ``` >> # export LTTNG_UST_MAP_POPULATE_POLICY=none >> # as above >> >> Running test app UID 0 >> >> >> procs -----------memory---------- ---swap-- -----io---- -system-- >> ------cpu----- >> >> r b swpd free buff cache si so bi bo in cs us sy >> id wa st >> >> 1 0 0 21875 0 9434 0 0 39 636 1105 2496 0 1 >> 99 0 0 >> >> MiB Mem : 32768.0 total, 21875.0 free, 1458.2 used, 9434.7 >> buff/cache >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 31309.8 avail >> Mem >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> >> 301616 debian 20 0 880176 2756 2756 S 0.0 0.0 0:00.04 >> lttng-sessiond >> >> 301617 debian 20 0 43856 1392 1392 S 0.0 0.0 0:00.01 >> lttng-runas >> >> 301632 debian 20 0 718612 5416 5416 S 0.0 0.0 0:00.17 >> lttng-consumerd >> >> 301634 debian 20 0 6992 0 0 S 0.0 0.0 0:00.05 >> lttng-runas >> >> lttng-sessiond: 220044 689 689 345 0 29900 0 >> >> >> total 880176K >> >> >> PID User Command Swap USS PSS >> RSS >> 301617 debian lttng-sessiond --daemonize 0 344 862 >> 2188 >> 301616 debian lttng-sessiond --daemonize 0 5784 6759 >> 9328 >> 301700 debian /usr/bin/python3 /usr/bin/s 0 8632 10079 >> 12928 >> >> lttng-consumerd: 1748 0 0 129 0 129 0 >> total kB 6992 1580 468 >> PID User Command Swap USS PSS >> RSS >> 301634 debian lttng-consumerd -u --consu 0 276 536 >> 1580 >> 301632 debian lttng-consumerd -u --consu 0 5672 6433 >> 8652 >> 301710 debian /usr/bin/python3 /usr/bin/s 0 9996 11449 >> 14328 >> >> /dev/shm/lttng-ust-wait-8-1000 1 4 4 >> /dev/shm/shm-ust-consumer-301632 1 4048 4048 >> ``` >> >> thanks, >> kienan >> >> [1]: >> >> https://github.com/lttng/lttng-ust/commit/4d4838bad480d48424bddc686f5ad0089e28ac94 >> [2]: >> >> https://github.com/lttng/lttng-ust/commit/97572c0438845cee953ebd3e39615f78bfa405a7 >> >> On 3/17/25 2:29 AM, Gour DEV wrote: >>> Hi, Kienan >>> >>> Sorry for the late reply. >>> >>> Looks like in buster the memory is allocated by lttng-consumerd reserved >>> >>> I buster, the rss is less than the VIRT >>> root@localhost:~# COLUMNS=500 top -b -n 1 | grep lttng >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >>> 4095 root 20 0 1003188 31256 4660 S 0.0 0.1 0:03.81 >>> lttng-sessiond >>> 4096 root 20 0 44260 796 0 S 0.0 0.0 0:00.01 >>> lttng-runas >>> 4440 root 20 0 5236020 10224 8756 S 0.0 0.0 2:56.25 >>> lttng-consumerd -- here the VIRT is much more higher than RSS >>> 4443 root 20 0 48048 540 0 S 0.0 0.0 0:00.12 >>> lttng-runas >>> >>> >>> >>> In bookworm the VIRT and RES are nearly the same only. >>> root@edgecore-40XKE-j2-101-32:~# COLUMNS=500 top -b -n 1 | grep lttng >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >>> 4382 root 20 0 1098824 42600 8436 S 0.0 0.1 0:08.87 >>> lttng-sessiond >>> 4403 root 20 0 48928 2116 996 S 0.0 0.0 0:00.00 >>> lttng-runas >>> 5171 root 20 0 9879764 8.9g 8.9g S 0.0 28.7 108:23.53 >>> lttng-consumerd -- here the VRIT is nearly equal to RSS >>> 5173 root 20 0 3680 1028 680 S 0.0 0.0 0:00.88 >>> lttng-runas >>> >>> >>> Looks like lttng consumerd is allocating and reserving those pages, when >>> any instrumented application starts. >>> >>> I am attaching the lttng status output in the mail, please do tell me if >>> you need any more information regarding this. >>> >>> >>> These is how we used to create the lttng channels and enable event which >> is >>> same in both buster and bookworm, (number of channels might differ) >>> >>> def enable_channel(channels, session, subbuf_size, subbuf_num): >>> for c in channels: >>> call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size', >>> str(subbuf_size), '--num-subbuf', str(subbuf_num),], >>> stdout=devnull, stderr=subprocess.STDOUT) >>> >>> >>> def enable_events(traces, session): >>> for t in traces: >>> if 'log-level-only' in t: >>> log_opt = '--loglevel-only=' + t['log-level-only'] >>> elif 'log-level' in t: >>> log_opt = '--loglevel=' + t['log-level'] >>> else: >>> log_opt = '' >>> >>> else: >>> call(['lttng', 'enable-event', '-u', t['name'], '-c', t['channel'], >>> '-s', session], stdout=devnull, stderr=subprocess.STDOUT) >>> >>> >>> Thank You. >>> Lakshya >>> >>> >>> >>> >>> On Wed, Mar 12, 2025 at 8:06 PM <lttng-dev-request@lists.lttng.org> >> wrote: >>> >>>> Send lttng-dev mailing list submissions to >>>> lttng-dev@lists.lttng.org >>>> >>>> To subscribe or unsubscribe via the World Wide Web, visit >>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >>>> or, via email, send a message with subject or body 'help' to >>>> lttng-dev-request@lists.lttng.org >>>> >>>> You can reach the person managing the list at >>>> lttng-dev-owner@lists.lttng.org >>>> >>>> When replying, please edit your Subject line so it is more specific >>>> than "Re: Contents of lttng-dev digest..." >>>> >>>> >>>> Today's Topics: >>>> >>>> 1. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >>>> (Kienan Stewart) >>>> 2. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >>>> (Gour DEV) >>>> 3. Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >>>> (Kienan Stewart) >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> >>>> Message: 1 >>>> Date: Tue, 11 Mar 2025 14:55:21 -0400 >>>> From: Kienan Stewart <kstewart@efficios.com> >>>> To: Gour DEV <lakshyagour10@gmail.com>, lttng-dev@lists.lttng.org >>>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >>>> Message-ID: <38dab5ef-f106-4e57-9e36-b4b30015c019@efficios.com> >>>> Content-Type: text/plain; charset=UTF-8; format=flowed >>>> >>>> Hi Lakshya, >>>> >>>> On 3/11/25 12:25 PM, Gour DEV wrote: >>>> > Hi, Kienan >>>> > >>>> > here is the requested output >>>> > >>>> > root@localhost:~# top -b -n 1 | grep lttng >>>> > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 >> 8:39.93 >>>> > lttng-c+ >>>> > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 >> 0:07.25 >>>> > lttng-s+ >>>> > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 >> 0:00.00 >>>> > lttng-r+ >>>> > 4843 root 20 0 3680 1160 816 S 0.0 0.0 >> 0:00.23 >>>> >>>> This top output for `localhost` seems very different than the output for >>>> `localhost` in your previous message. >>>> >>>> >>>> > lttng-r+ >>>> > root@localhost:~# nrpco >>>> > bash: nrpco: command not found >>>> > root@localhost:~# nproc >>>> > 16 >>>> > root@localhost:~# cat /sys/devices/system/cpu/possible >>>> > 0-15 >>>> > >>>> >>>> You indicated the bookworm machine has 32 cores, this is showing 16. If >>>> you're comparing a 16 core machine to a 32 core machine, it is very >>>> normal that the memory usage is higher on the 32 core machine. >>>> >>>> > >>>> > Most of the process are running as asorcs user but some are running >>>> as root. >>>> >>>> So you have two users with instrumented applications. >>>> >>>> >>>> Given the discrepancies in the information provided I'm finding it a bit >>>> hard to understand what you're looking at. >>>> >>>> >>>> In general, a channel's shared memory footprint can be estimated >> with[1]: >>>> >>>> (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * >>>> (nUIDs or nPIDs) >>>> >>>> Note that the sub-buffer sizes you are using get rounded to the nearest >>>> larger power of 2. See [2]. >>>> >>>> thanks, >>>> kienan >>>> >>>> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes >>>> [2]: >>>> >> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size >>>> >>>> >>>> ------------------------------ >>>> >>>> Message: 2 >>>> Date: Wed, 12 Mar 2025 14:49:07 +0530 >>>> From: Gour DEV <lakshyagour10@gmail.com> >>>> To: Kienan Stewart <kstewart@efficios.com> >>>> Cc: lttng-dev@lists.lttng.org >>>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >>>> Message-ID: >>>> <CAE9Jrzg7qsabhPiO-0=B1DY3bVo-3FYu2tiJR2Bmb= >>>> nqOHNZMw@mail.gmail.com> >>>> Content-Type: text/plain; charset="utf-8" >>>> >>>> Hi, Kienan >>>> >>>> I am attaching an screen recording of the behaviour I am seeing in this >>>> mail. The behaviour is same irrespective of the device i use, sorry for >>>> miscommunication in the npocs output (I assumed it was 32), but other >> than >>>> that all outputs are same (except the hostname as there are multiple >>>> devices with same lttng config but this memory cosumption is seen on all >>>> the devices). >>>> >>>> I had few question >>>> >>>> 1. Does lltng allocated all the memory it needs and mark it as dirty in >> ram >>>> when any process which links/uses lttng-ust runs? (here i tried with one >>>> process but it is same for any of my process) >>>> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * >>>> (nUIDs or nPIDs) >>>> >>>> How do we calculate uid in the system is it all uids in the system? is >> it >>>> equal to `cat /etc/passwd | wc -l` ? >>>> >>>> I will put my calculations according to the above estimate based on all >> the >>>> channel i am creating >>>> >>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal >> to >>>> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on >> the >>>> start_lttng.py please do correct me if am wrong here.] >>>> >>>> But since there are only two users which uses lttng i think the correct >>>> estimate would be >>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB >>>> >>>> Please do correct me If I am wrong calculations here. >>>> >>>> Now, there are a few things here, according to my output lttng is using >> 11G >>>> which is much more higher than the what is configured. >>>> >>>> I am attaching the lttng status and the file which is uses to create the >>>> lttng sessions. >>>> >>>> >>>> >>>> Thank You. >>>> >>>> >>>> >>>> >> https://drive.google.com/file/d/1tS_ZWEsXDpHZXfWzZHXmWcT0igiIOIaa/view?usp=sharing >>>> -- recording of the behaviour which is seen >>>> >>>> >> https://drive.google.com/file/d/1PrU31oyEw1n9tKETlUtmNGO50s6ywx7p/view?usp=sharing >>>> -- the file which is used to create lttng sessions >>>> >>>> >>>> On Wed, Mar 12, 2025 at 12:25?AM Kienan Stewart <kstewart@efficios.com> >>>> wrote: >>>> >>>>> Hi Lakshya, >>>>> >>>>> On 3/11/25 12:25 PM, Gour DEV wrote: >>>>> > Hi, Kienan >>>>> > >>>>> > here is the requested output >>>>> > >>>>> > root@localhost:~# top -b -n 1 | grep lttng >>>>> > 4841 root 20 0 11.5g 11.0g 11.0g S 5.9 35.4 >>>> 8:39.93 >>>>> > lttng-c+ >>>>> > 4824 root 20 0 1098824 26456 5380 S 0.0 0.1 >>>> 0:07.25 >>>>> > lttng-s+ >>>>> > 4825 root 20 0 48872 2188 1012 S 0.0 0.0 >>>> 0:00.00 >>>>> > lttng-r+ >>>>> > 4843 root 20 0 3680 1160 816 S 0.0 0.0 >>>> 0:00.23 >>>>> >>>>> This top output for `localhost` seems very different than the output >> for >>>>> `localhost` in your previous message. >>>>> >>>>> >>>>> > lttng-r+ >>>>> > root@localhost:~# nrpco >>>>> > bash: nrpco: command not found >>>>> > root@localhost:~# nproc >>>>> > 16 >>>>> > root@localhost:~# cat /sys/devices/system/cpu/possible >>>>> > 0-15 >>>>> > >>>>> >>>>> You indicated the bookworm machine has 32 cores, this is showing 16. If >>>>> you're comparing a 16 core machine to a 32 core machine, it is very >>>>> normal that the memory usage is higher on the 32 core machine. >>>>> >>>>> > >>>>> > Most of the process are running as asorcs user but some are running >>>>> as root. >>>>> >>>>> So you have two users with instrumented applications. >>>>> >>>>> >>>>> Given the discrepancies in the information provided I'm finding it a >> bit >>>>> hard to understand what you're looking at. >>>>> >>>>> >>>>> In general, a channel's shared memory footprint can be estimated >> with[1]: >>>>> >>>>> (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * >>>>> (nUIDs or nPIDs) >>>>> >>>>> Note that the sub-buffer sizes you are using get rounded to the nearest >>>>> larger power of 2. See [2]. >>>>> >>>>> thanks, >>>>> kienan >>>>> >>>>> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes >>>>> [2]: >>>>> >> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size >>>>> >>>> -------------- next part -------------- >>>> An HTML attachment was scrubbed... >>>> URL: < >>>> >> https://lists.lttng.org/pipermail/lttng-dev/attachments/20250312/57f240d8/attachment-0001.htm >>>>> >>>> >>>> ------------------------------ >>>> >>>> Message: 3 >>>> Date: Wed, 12 Mar 2025 10:36:28 -0400 >>>> From: Kienan Stewart <kstewart@efficios.com> >>>> To: Gour DEV <lakshyagour10@gmail.com>, lttng-dev@lists.lttng.org >>>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10 >>>> Message-ID: <0f819583-ea8e-468e-9102-e1410d886a6f@efficios.com> >>>> Content-Type: text/plain; charset=UTF-8; format=flowed >>>> >>>> Hi Lakshya, >>>> >>>> On 3/12/25 5:03 AM, Gour DEV wrote: >>>>> Hi, Kienan >>>>> >>>>> I am attaching an screen recording of the behaviour I am seeing in this >>>>> mail. The behaviour is same irrespective of the device i use, sorry for >>>>> miscommunication in the npocs output (I assumed it was 32), but other >>>> than >>>>> that all outputs are same (except the hostname as there are multiple >>>>> devices with same lttng config but this memory cosumption is seen on >> all >>>>> the devices). >>>>> >>>>> I had few question >>>>> >>>>> 1. Does lltng allocated all the memory it needs and mark it as dirty in >>>> ram >>>>> when any process which links/uses lttng-ust runs? (here i tried with >> one >>>>> process but it is same for any of my process) >>>> >>>> I believe the shared memory for per-CPU data structures is allocated >>>> when an instrumented application connects. There is no pre-allocation >>>> for each possible UID on the system. >>>> >>>> You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to >>>> see when the connection happens[1]. >>>> >>>>> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * >>>>> (nUIDs or nPIDs) >>>>> >>>>> How do we calculate uid in the system is it all uids in the system? is >> it >>>>> equal to `cat /etc/passwd | wc -l` ? >>>> >>>> nUIDs is the number of distinct UIDs running instrumented applications. >>>> >>>>> >>>>> I will put my calculations according to the above estimate based on all >>>> the >>>>> channel i am creating >>>>> >>>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal >> to >>>>> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on >> the >>>>> start_lttng.py please do correct me if am wrong here.] >>>>> >>>>> But since there are only two users which uses lttng i think the correct >>>>> estimate would be >>>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB >>>> >>>> The estimate I gave is per-channel. >>>> >>>> small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel >>>> per-UID >>>> medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel >> per-UID >>>> large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID >>>> >>>> Now, you said you have 0 small channels, 6 medium channels, and 16 large >>>> channels in your session. (Note: I see your script differs from these >>>> stated channel counts). >>>> >>>> small: 0 * 1.0625 MiB = 0 MiB per-UID >>>> medium: 6 * 17 MiB = 102 MiB per-UID >>>> large: 16 * 272 MiB = 4352 MiB per-UID >>>> >>>> And if you're running instrumented applications with 2 users: >>>> >>>> small: 0 MiB * 2 = 0 MiB with 2 UIDs >>>> medium: 102 MiB * 2 = 204 MiB with 2 UIDs >>>> large: 4352 MiB * 2 = 8704 MiB with 2 UIDs >>>> >>>> Now this is just an estimation for the per-CPU ring buffers only, and >>>> you numbers aren't hugely off so without analyzing your specific system >>>> it doesn't seem to be that strange to me. >>>> >>>> If I take the number of channels I see in your script, it becomes: >>>> >>>> small: 0 MiB with 2 UIDs >>>> medium: 136 MiB with 2 UIDs >>>> large: 7616 MiB with 2 UIDs >>>> >>>> total: 7.57 GiB with 2 UIDs >>>> >>>>> >>>>> Please do correct me If I am wrong calculations here. >>>>> >>>>> Now, there are a few things here, according to my output lttng is using >>>> 11G >>>>> which is much more higher than the what is configured. >>>>> >>>> >>>> I have no idea what 'service start spyder' is doing. Maybe it's running >>>> instrumented applications with an extra user that you didn't expect? I >>>> can't help you with that aspect of your system. >>>> >>>> The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs >>>> so maybe? >>>> >>>> I'd recommend you read your verbose sessiond log so see which >>>> applications are connecting and with which UIDs. >>>> >>>>> I am attaching the lttng status and the file which is uses to create >> the >>>>> lttng sessions. >>>>> >>>>> >>>>> >>>>> Thank You. >>>>> >>>> >>>> In any case, the information you have given to date hasn't demonstrated >>>> to me in a tangible manner that you are seeing a difference related to >>>> the version of LTTng being used. >>>> >>>> thanks, >>>> kienan >>>> >>>> [1]: >> https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables >>>> >>>> >>>> ------------------------------ >>>> >>>> Subject: Digest Footer >>>> >>>> _______________________________________________ >>>> lttng-dev mailing list >>>> lttng-dev@lists.lttng.org >>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >>>> >>>> >>>> ------------------------------ >>>> >>>> End of lttng-dev Digest, Vol 203, Issue 7 >>>> ***************************************** >>>> >>> >> >> > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: lttng-dev Digest, Vol 203, Issue 7 2025-03-21 13:14 ` Kienan Stewart via lttng-dev @ 2025-03-21 14:10 ` Olivier Dion via lttng-dev 0 siblings, 0 replies; 5+ messages in thread From: Olivier Dion via lttng-dev @ 2025-03-21 14:10 UTC (permalink / raw) To: Kienan Stewart, Gour DEV, lttng-dev On Fri, 21 Mar 2025, Kienan Stewart via lttng-dev <lttng-dev@lists.lttng.org> wrote: > Hi Lakshya, > > On 3/21/25 8:52 AM, Gour DEV wrote: >> Thanks Kienan for digging this and explaining the behaviour, [...] >> Also, I thought using --buffers-global >> <https://lttng.org/man/1/lttng-enable-channel/v2.8/#doc-opt--buffers-global> >> would >> be better for me as I could know how much memory will be required but looks >> like that option is only available for kernel tracking, it would be very >> good if this option could be available for user buffering also, but that >> could come later. >> > > Global buffers for UST is planned for the next release. The trade-off of > using it is that tracing overhead will increase (as the buffers are no > longer per-CPU), but in exchange for a potentially lower memory > footprint particularly in situations where not all possible CPUs are online. > Little correction here. `--buffers-global' is the ownership of the buffers (UID vs PID vs system). The next release introduces the buffers allocation policies `per-{cpu,channel}'. Therefore `--buffers-global' will still be only available for the kernel domain, AFAIK. User-space however will be able to allocate a single ring-buffer for a given channel (actually it depends on the buffer ownership UID vs PID), thus reducing the amount of memory used, for that channel, by a factor proportional to the number of CPUs possible on the system. [...] Thanks, Olivier -- Olivier Dion EfficiOS Inc. https://www.efficios.com ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2025-03-21 14:10 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <mailman.58.1741790192.282.lttng-dev@lists.lttng.org>
2025-03-17 6:29 ` lttng-dev Digest, Vol 203, Issue 7 Gour DEV via lttng-dev
2025-03-20 20:51 ` Kienan Stewart via lttng-dev
2025-03-21 12:52 ` Gour DEV via lttng-dev
2025-03-21 13:14 ` Kienan Stewart via lttng-dev
2025-03-21 14:10 ` Olivier Dion via lttng-dev
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox