[RFC] tracing: aggregate ring-buffer usage statistics

[RFC] tracing: aggregate ring-buffer usage statistics

[Xiang Gao]

Hi,

I would like to clarify the motivation for a tracing-specific memory
usage interface.

The original requirement comes from Android lost-RAM attribution.
Ftrace ring-buffer data pages are not represented as a separate memory
category, so userspace memory diagnostics may treat them as lost RAM.

The memory can be spread across the global trace array, dynamically
created tracing instances, and snapshot buffers. Userspace can read
buffer_total_size_kb for the global trace array and each instance, but
it has to discover and sum all instances, and snapshot buffer capacity
is not exposed through the same per-instance total.

On one Android device, the global buffer and 24 tracing instances
accounted for 395,664 kB of ring-buffer data capacity. This makes the
missing attribution significant for low-memory diagnostics.

The core requirement is a stable tracing-specific interface that allows
userspace to obtain the aggregate tracing buffer usage without walking
all instances. It does not need to be part of /proc/meminfo or
show_mem().

One possible interface would be under:

  /sys/kernel/tracing/trace_stats/

It could provide an aggregate buffer value together with a breakdown
for main and snapshot buffers. Per-CPU statistics could also be added
if useful.

Thanks,
Xiang

Re: [RFC] tracing: aggregate ring-buffer usage statistics

[Steven Rostedt]

On Tue, 11 Aug 2026 19:48:18 +0800
Xiang Gao <gxxa03070307@gmail.com> wrote:

> One possible interface would be under:
> 
>   /sys/kernel/tracing/trace_stats/
> 
> It could provide an aggregate buffer value together with a breakdown
> for main and snapshot buffers. Per-CPU statistics could also be added
> if useful.

As I previous said. I'm OK with adding a file there.

-- Steve

答复: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[高翔]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]


Thanks, Steve.

For the initial implementation, I plan to add:

  /sys/kernel/tracing/trace_stats/total_memory_kb

The file would use an extensible key/value format. Initially it would
report the ring-buffer data capacity:

  total:                   395664
  ring_buffer:             395664
  ring_buffer_live:        395544
  ring_buffer_snapshot:       120

Here, ring_buffer is the sum of ring_buffer_live and
ring_buffer_snapshot. The values cover the global trace array and all
tracing instances across all tracing CPUs.

Additional accounted categories, such as event memory, could be appended
later and included in total. Per-CPU statistics could be exposed
separately under trace_stats/per_cpu/cpuN/ to avoid mixing aggregate and
per-CPU values in one file.

Userspace would parse fields by name and ignore unknown fields, so new
categories can be added without changing the existing format.

Does this initial format look reasonable?

Thanks,
Xiang



________________________________
发件人: Steven Rostedt <rostedt@goodmis.org>
发送时间: 2026年8月11日 22:08:30
收件人: Xiang Gao
抄送: Masami Hiramatsu; Mathieu Desnoyers; Lorenzo Stoakes; gao xu; 印闯; linux-trace-kernel@vger.kernel.org; linux-kernel@vger.kernel.org; linux-mm@kvack.org; 高翔
主题: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[外部邮件] 此邮件来源于小米公司外部，请谨慎处理。若对邮件安全性存疑，请将邮件转发给misec@xiaomi.com进行反馈

On Tue, 11 Aug 2026 19:48:18 +0800
Xiang Gao <gxxa03070307@gmail.com> wrote:

> One possible interface would be under:
>
>   /sys/kernel/tracing/trace_stats/
>
> It could provide an aggregate buffer value together with a breakdown
> for main and snapshot buffers. Per-CPU statistics could also be added
> if useful.

As I previous said. I'm OK with adding a file there.

-- Steve

[-- Attachment #2: Type: text/html, Size: 3508 bytes --]

Re: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[Steven Rostedt]

On Wed, 12 Aug 2026 13:21:29 +0000
高翔 <gaoxiang17@xiaomi.com> wrote:

> Thanks, Steve.
> 
> For the initial implementation, I plan to add:
> 
>   /sys/kernel/tracing/trace_stats/total_memory_kb
> 
> The file would use an extensible key/value format. Initially it would
> report the ring-buffer data capacity:
> 
>   total:                   395664
>   ring_buffer:             395664
>   ring_buffer_live:        395544
>   ring_buffer_snapshot:       120

You can rename the above to:

   total:
   buffers_mem:
   snapshot_buffers_mem:

No need to use "ring" and it keeps consistent with "buffer_size_kb".

I added the "mem" part to distinguish the difference between these numbers
and the buffer_size_kb, as these numbers are about total memory whereas
the "buffer_size_kb" is about usable memory.

There's already a total for all memory listed. We don't need to have a
total for the "live" and "snapshot".

> 
> Here, ring_buffer is the sum of ring_buffer_live and
> ring_buffer_snapshot. The values cover the global trace array and all
> tracing instances across all tracing CPUs.
> 
> Additional accounted categories, such as event memory, could be appended
> later and included in total. Per-CPU statistics could be exposed
> separately under trace_stats/per_cpu/cpuN/ to avoid mixing aggregate and
> per-CPU values in one file.

Agreed, this file should exist in the cpuN/ directories and be only for
those instances.

> 
> Userspace would parse fields by name and ignore unknown fields, so new
> categories can be added without changing the existing format.

Right. I would expect additional items to be added here.

> 
> Does this initial format look reasonable?
> 

Yes, thanks for doing this.

-- Steve

答复: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[高翔]

[-- Attachment #1: Type: text/plain, Size: 2866 bytes --]

Thanks, Steve.

Understood.

I will add:

  /sys/kernel/tracing/trace_stats/total_memory_kb

The file will report:

  total:
  buffers_mem:
  snapshot_buffers_mem:

The values will cover the main and snapshot buffers of the global trace
array and all tracing instances across all tracing CPUs.

For the per-CPU view, I will follow the existing tracefs layout and add:

  /sys/kernel/tracing/trace_stats/per_cpu/cpuN/total_memory_kb

Each per-CPU file will report only the corresponding CPU's buffers
across the global trace array and all tracing instances.

Thanks,
Xiang



________________________________
发件人: Steven Rostedt <rostedt@goodmis.org>
发送时间: 2026年8月12日 23:47:04
收件人: 高翔
抄送: Xiang Gao; Masami Hiramatsu; Mathieu Desnoyers; Lorenzo Stoakes; gao xu; 印闯; linux-trace-kernel@vger.kernel.org; linux-kernel@vger.kernel.org; linux-mm@kvack.org
主题: Re: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[外部邮件] 此邮件来源于小米公司外部，请谨慎处理。若对邮件安全性存疑，请将邮件转发给misec@xiaomi.com进行反馈

On Wed, 12 Aug 2026 13:21:29 +0000
高翔 <gaoxiang17@xiaomi.com> wrote:

> Thanks, Steve.
>
> For the initial implementation, I plan to add:
>
>   /sys/kernel/tracing/trace_stats/total_memory_kb
>
> The file would use an extensible key/value format. Initially it would
> report the ring-buffer data capacity:
>
>   total:                   395664
>   ring_buffer:             395664
>   ring_buffer_live:        395544
>   ring_buffer_snapshot:       120

You can rename the above to:

   total:
   buffers_mem:
   snapshot_buffers_mem:

No need to use "ring" and it keeps consistent with "buffer_size_kb".

I added the "mem" part to distinguish the difference between these numbers
and the buffer_size_kb, as these numbers are about total memory whereas
the "buffer_size_kb" is about usable memory.

There's already a total for all memory listed. We don't need to have a
total for the "live" and "snapshot".

>
> Here, ring_buffer is the sum of ring_buffer_live and
> ring_buffer_snapshot. The values cover the global trace array and all
> tracing instances across all tracing CPUs.
>
> Additional accounted categories, such as event memory, could be appended
> later and included in total. Per-CPU statistics could be exposed
> separately under trace_stats/per_cpu/cpuN/ to avoid mixing aggregate and
> per-CPU values in one file.

Agreed, this file should exist in the cpuN/ directories and be only for
those instances.

>
> Userspace would parse fields by name and ignore unknown fields, so new
> categories can be added without changing the existing format.

Right. I would expect additional items to be added here.

>
> Does this initial format look reasonable?
>

Yes, thanks for doing this.

-- Steve

[-- Attachment #2: Type: text/html, Size: 4895 bytes --]

答复: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[高翔]

[-- Attachment #1: Type: text/plain, Size: 3790 bytes --]

Hi, Steve.


One accounting detail I would like to clarify before implementing this:
should buffers_mem and snapshot_buffers_mem include all memory owned by
the buffers, including slab-allocated ring-buffer metadata and reader
pages, or only the full pages backing the ring-buffer data?


My original lost-RAM use case primarily needs the non-slab page memory,
since slab is already accounted for separately. But I understand
"buffers_mem" and "total memory" may be intended to include both page
and slab allocations.


Thanks,
Xiang



________________________________
发件人: 高翔
发送时间: 2026年8月13日 20:47:44
收件人: Steven Rostedt
抄送: Xiang Gao; Masami Hiramatsu; Mathieu Desnoyers; Lorenzo Stoakes; gao xu; 印闯; linux-trace-kernel@vger.kernel.org; linux-kernel@vger.kernel.org; linux-mm@kvack.org
主题: 答复: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics


Thanks, Steve.

Understood.

I will add:

  /sys/kernel/tracing/trace_stats/total_memory_kb

The file will report:

  total:
  buffers_mem:
  snapshot_buffers_mem:

The values will cover the main and snapshot buffers of the global trace
array and all tracing instances across all tracing CPUs.

For the per-CPU view, I will follow the existing tracefs layout and add:

  /sys/kernel/tracing/trace_stats/per_cpu/cpuN/total_memory_kb

Each per-CPU file will report only the corresponding CPU's buffers
across the global trace array and all tracing instances.

Thanks,
Xiang



________________________________
发件人: Steven Rostedt <rostedt@goodmis.org>
发送时间: 2026年8月12日 23:47:04
收件人: 高翔
抄送: Xiang Gao; Masami Hiramatsu; Mathieu Desnoyers; Lorenzo Stoakes; gao xu; 印闯; linux-trace-kernel@vger.kernel.org; linux-kernel@vger.kernel.org; linux-mm@kvack.org
主题: Re: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[外部邮件] 此邮件来源于小米公司外部，请谨慎处理。若对邮件安全性存疑，请将邮件转发给misec@xiaomi.com进行反馈

On Wed, 12 Aug 2026 13:21:29 +0000
高翔 <gaoxiang17@xiaomi.com> wrote:

> Thanks, Steve.
>
> For the initial implementation, I plan to add:
>
>   /sys/kernel/tracing/trace_stats/total_memory_kb
>
> The file would use an extensible key/value format. Initially it would
> report the ring-buffer data capacity:
>
>   total:                   395664
>   ring_buffer:             395664
>   ring_buffer_live:        395544
>   ring_buffer_snapshot:       120

You can rename the above to:

   total:
   buffers_mem:
   snapshot_buffers_mem:

No need to use "ring" and it keeps consistent with "buffer_size_kb".

I added the "mem" part to distinguish the difference between these numbers
and the buffer_size_kb, as these numbers are about total memory whereas
the "buffer_size_kb" is about usable memory.

There's already a total for all memory listed. We don't need to have a
total for the "live" and "snapshot".

>
> Here, ring_buffer is the sum of ring_buffer_live and
> ring_buffer_snapshot. The values cover the global trace array and all
> tracing instances across all tracing CPUs.
>
> Additional accounted categories, such as event memory, could be appended
> later and included in total. Per-CPU statistics could be exposed
> separately under trace_stats/per_cpu/cpuN/ to avoid mixing aggregate and
> per-CPU values in one file.

Agreed, this file should exist in the cpuN/ directories and be only for
those instances.

>
> Userspace would parse fields by name and ignore unknown fields, so new
> categories can be added without changing the existing format.

Right. I would expect additional items to be added here.

>
> Does this initial format look reasonable?
>

Yes, thanks for doing this.

-- Steve

[-- Attachment #2: Type: text/html, Size: 6396 bytes --]

答复: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[高翔]

[-- Attachment #1: Type: text/plain, Size: 4475 bytes --]

Hi Steve,


Just a gentle ping. I have the RFC patch ready for
trace_stats/total_memory_kb, with the buffers_mem and
snapshot_buffers_mem breakdown.


Would you prefer I post the RFC patch for review, or is there anything
you would like to clarify about the accounting scope first?


Thanks,
Xiang



________________________________
发件人: 高翔
发送时间: 2026年8月15日 12:33:14
收件人: Steven Rostedt
抄送: Xiang Gao; Masami Hiramatsu; Mathieu Desnoyers; Lorenzo Stoakes; gao xu; 印闯; linux-trace-kernel@vger.kernel.org; linux-kernel@vger.kernel.org; linux-mm@kvack.org
主题: 答复: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics


Hi, Steve.


One accounting detail I would like to clarify before implementing this:
should buffers_mem and snapshot_buffers_mem include all memory owned by
the buffers, including slab-allocated ring-buffer metadata and reader
pages, or only the full pages backing the ring-buffer data?


My original lost-RAM use case primarily needs the non-slab page memory,
since slab is already accounted for separately. But I understand
"buffers_mem" and "total memory" may be intended to include both page
and slab allocations.


Thanks,
Xiang



________________________________
发件人: 高翔
发送时间: 2026年8月13日 20:47:44
收件人: Steven Rostedt
抄送: Xiang Gao; Masami Hiramatsu; Mathieu Desnoyers; Lorenzo Stoakes; gao xu; 印闯; linux-trace-kernel@vger.kernel.org; linux-kernel@vger.kernel.org; linux-mm@kvack.org
主题: 答复: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics


Thanks, Steve.

Understood.

I will add:

  /sys/kernel/tracing/trace_stats/total_memory_kb

The file will report:

  total:
  buffers_mem:
  snapshot_buffers_mem:

The values will cover the main and snapshot buffers of the global trace
array and all tracing instances across all tracing CPUs.

For the per-CPU view, I will follow the existing tracefs layout and add:

  /sys/kernel/tracing/trace_stats/per_cpu/cpuN/total_memory_kb

Each per-CPU file will report only the corresponding CPU's buffers
across the global trace array and all tracing instances.

Thanks,
Xiang



________________________________
发件人: Steven Rostedt <rostedt@goodmis.org>
发送时间: 2026年8月12日 23:47:04
收件人: 高翔
抄送: Xiang Gao; Masami Hiramatsu; Mathieu Desnoyers; Lorenzo Stoakes; gao xu; 印闯; linux-trace-kernel@vger.kernel.org; linux-kernel@vger.kernel.org; linux-mm@kvack.org
主题: Re: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[外部邮件] 此邮件来源于小米公司外部，请谨慎处理。若对邮件安全性存疑，请将邮件转发给misec@xiaomi.com进行反馈

On Wed, 12 Aug 2026 13:21:29 +0000
高翔 <gaoxiang17@xiaomi.com> wrote:

> Thanks, Steve.
>
> For the initial implementation, I plan to add:
>
>   /sys/kernel/tracing/trace_stats/total_memory_kb
>
> The file would use an extensible key/value format. Initially it would
> report the ring-buffer data capacity:
>
>   total:                   395664
>   ring_buffer:             395664
>   ring_buffer_live:        395544
>   ring_buffer_snapshot:       120

You can rename the above to:

   total:
   buffers_mem:
   snapshot_buffers_mem:

No need to use "ring" and it keeps consistent with "buffer_size_kb".

I added the "mem" part to distinguish the difference between these numbers
and the buffer_size_kb, as these numbers are about total memory whereas
the "buffer_size_kb" is about usable memory.

There's already a total for all memory listed. We don't need to have a
total for the "live" and "snapshot".

>
> Here, ring_buffer is the sum of ring_buffer_live and
> ring_buffer_snapshot. The values cover the global trace array and all
> tracing instances across all tracing CPUs.
>
> Additional accounted categories, such as event memory, could be appended
> later and included in total. Per-CPU statistics could be exposed
> separately under trace_stats/per_cpu/cpuN/ to avoid mixing aggregate and
> per-CPU values in one file.

Agreed, this file should exist in the cpuN/ directories and be only for
those instances.

>
> Userspace would parse fields by name and ignore unknown fields, so new
> categories can be added without changing the existing format.

Right. I would expect additional items to be added here.

>
> Does this initial format look reasonable?
>

Yes, thanks for doing this.

-- Steve

[-- Attachment #2: Type: text/html, Size: 7626 bytes --]

Re: [External Mail]Re: [RFC] tracing: aggregate ring-buffer usage statistics

[Steven Rostedt]

On Tue, 18 Aug 2026 13:30:26 +0000
高翔 <gaoxiang17@xiaomi.com> wrote:

> Hi Steve,
> 
> 
> Just a gentle ping. I have the RFC patch ready for
> trace_stats/total_memory_kb, with the buffers_mem and
> snapshot_buffers_mem breakdown.
> 
> 
> Would you prefer I post the RFC patch for review, or is there anything
> you would like to clarify about the accounting scope first?

I'm traveling this week. Can you send another RFC next week.

Thanks,

-- Steve