* [PATCH v4 3/3] tracing: Update funcgraph-retval documentation
From: Donglin Peng @ 2025-12-15 3:41 UTC (permalink / raw)
To: rostedt
Cc: mhiramat, linux-trace-kernel, bpf, linux-kernel, pengdonglin,
Xiaoqin Zhang
In-Reply-To: <20251215034153.2367756-1-dolinux.peng@gmail.com>
From: pengdonglin <pengdonglin@xiaomi.com>
The existing documentation for funcgraph-retval is outdated and partially
incorrect, as it describes limitations that have now been resolved.
Recent changes (e.g., using BTF to obtain function return types) have
addressed key issues:
1. Return values are now printed only for non-void functions.
2. Values are trimmed to the correct width of the return type, avoiding
garbage data from high bits.
Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Xiaoqin Zhang <zhangxiaoqin@xiaomi.com>
Signed-off-by: pengdonglin <pengdonglin@xiaomi.com>
---
Documentation/trace/ftrace.rst | 88 +++++++++++++++++++---------------
1 file changed, 50 insertions(+), 38 deletions(-)
diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst
index d1f313a5f4ad..b231e80e6a4f 100644
--- a/Documentation/trace/ftrace.rst
+++ b/Documentation/trace/ftrace.rst
@@ -1454,6 +1454,10 @@ Options for function_graph tracer:
printed in hexadecimal format. By default, this option
is off.
+ funcgraph-retaddr
+ When set, the return address will always be printed.
+ By default, this option is off.
+
sleep-time
When running function graph tracer, to include
the time a task schedules out in its function.
@@ -2800,7 +2804,7 @@ It is default disabled.
0) 2.861 us | } /* putname() */
The return value of each traced function can be displayed after
-an equal sign "=". When encountering system call failures, it
+an equal sign "ret =". When encountering system call failures, it
can be very helpful to quickly locate the function that first
returns an error code.
@@ -2810,16 +2814,16 @@ returns an error code.
Example with funcgraph-retval::
1) | cgroup_migrate() {
- 1) 0.651 us | cgroup_migrate_add_task(); /* = 0xffff93fcfd346c00 */
+ 1) 0.651 us | cgroup_migrate_add_task(); /* ret=0xffff93fcfd346c00 */
1) | cgroup_migrate_execute() {
1) | cpu_cgroup_can_attach() {
1) | cgroup_taskset_first() {
- 1) 0.732 us | cgroup_taskset_next(); /* = 0xffff93fc8fb20000 */
- 1) 1.232 us | } /* cgroup_taskset_first = 0xffff93fc8fb20000 */
- 1) 0.380 us | sched_rt_can_attach(); /* = 0x0 */
- 1) 2.335 us | } /* cpu_cgroup_can_attach = -22 */
- 1) 4.369 us | } /* cgroup_migrate_execute = -22 */
- 1) 7.143 us | } /* cgroup_migrate = -22 */
+ 1) 0.732 us | cgroup_taskset_next(); /* ret=0xffff93fc8fb20000 */
+ 1) 1.232 us | } /* cgroup_taskset_first ret=0xffff93fc8fb20000 */
+ 1) 0.380 us | sched_rt_can_attach(); /* ret=0x0 */
+ 1) 2.335 us | } /* cpu_cgroup_can_attach ret=-22 */
+ 1) 4.369 us | } /* cgroup_migrate_execute ret=-22 */
+ 1) 7.143 us | } /* cgroup_migrate ret=-22 */
The above example shows that the function cpu_cgroup_can_attach
returned the error code -22 firstly, then we can read the code
@@ -2836,37 +2840,41 @@ printed in hexadecimal format.
Example with funcgraph-retval-hex::
1) | cgroup_migrate() {
- 1) 0.651 us | cgroup_migrate_add_task(); /* = 0xffff93fcfd346c00 */
+ 1) 0.651 us | cgroup_migrate_add_task(); /* ret=0xffff93fcfd346c00 */
1) | cgroup_migrate_execute() {
1) | cpu_cgroup_can_attach() {
1) | cgroup_taskset_first() {
- 1) 0.732 us | cgroup_taskset_next(); /* = 0xffff93fc8fb20000 */
- 1) 1.232 us | } /* cgroup_taskset_first = 0xffff93fc8fb20000 */
- 1) 0.380 us | sched_rt_can_attach(); /* = 0x0 */
- 1) 2.335 us | } /* cpu_cgroup_can_attach = 0xffffffea */
- 1) 4.369 us | } /* cgroup_migrate_execute = 0xffffffea */
- 1) 7.143 us | } /* cgroup_migrate = 0xffffffea */
-
-At present, there are some limitations when using the funcgraph-retval
-option, and these limitations will be eliminated in the future:
-
-- Even if the function return type is void, a return value will still
- be printed, and you can just ignore it.
-
-- Even if return values are stored in multiple registers, only the
- value contained in the first register will be recorded and printed.
- To illustrate, in the x86 architecture, eax and edx are used to store
- a 64-bit return value, with the lower 32 bits saved in eax and the
- upper 32 bits saved in edx. However, only the value stored in eax
- will be recorded and printed.
-
-- In certain procedure call standards, such as arm64's AAPCS64, when a
- type is smaller than a GPR, it is the responsibility of the consumer
- to perform the narrowing, and the upper bits may contain UNKNOWN values.
- Therefore, it is advisable to check the code for such cases. For instance,
- when using a u8 in a 64-bit GPR, bits [63:8] may contain arbitrary values,
- especially when larger types are truncated, whether explicitly or implicitly.
- Here are some specific cases to illustrate this point:
+ 1) 0.732 us | cgroup_taskset_next(); /* ret=0xffff93fc8fb20000 */
+ 1) 1.232 us | } /* cgroup_taskset_first ret=0xffff93fc8fb20000 */
+ 1) 0.380 us | sched_rt_can_attach(); /* ret=0x0 */
+ 1) 2.335 us | } /* cpu_cgroup_can_attach ret=0xffffffea */
+ 1) 4.369 us | } /* cgroup_migrate_execute ret=0xffffffea */
+ 1) 7.143 us | } /* cgroup_migrate ret=0xffffffea */
+
+Note that there are some limitations when using the funcgraph-retval
+option:
+
+- If CONFIG_DEBUG_INFO_BTF is disabled (n), a return value is printed even for
+ functions with a void return type. When CONFIG_DEBUG_INFO_BTF is enabled (y),
+ the return value is printed only for non-void functions.
+
+- If a return value occupies multiple registers, only the value in the first
+ register is recorded and printed. For example, on the x86 architecture, a
+ 64-bit return value is stored across eax (lower 32 bits) and edx (upper 32 bits),
+ but only the contents of eax are captured. If CONFIG_DEBUG_INFO_BTF is enabled,
+ the suffix "(trunc)" is appended to the printed value to indicate that the
+ output may be truncated because high-order register contents are omitted.
+
+- Under certain procedure-call standards (e.g., arm64's AAPCS64), when the return
+ type is smaller than a general-purpose register (GPR), the caller is responsible
+ for narrowing the value; the upper bits of the register may contain undefined data.
+ For instance, when a u8 is returned in 64-bit GPR, bits [63:8] can hold arbitrary
+ values, especially when larger types are truncated (explicitly or implicitly). It
+ is therefore advisable to inspect the code in such cases. If CONFIG_DEBUG_INFO_BTF
+ is enabled (y), the return value is automatically trimmed to the width of the return
+ type.
+
+ The following examples illustrate the behavior:
**Case One**:
@@ -2885,7 +2893,9 @@ option, and these limitations will be eliminated in the future:
RET
If you pass 0x123456789abcdef to this function and want to narrow it,
- it may be recorded as 0x123456789abcdef instead of 0xef.
+ it may be recorded as 0x123456789abcdef instead of 0xef. When
+ CONFIG_DEBUG_INFO_BTF is enabled, the value will be correctly truncated
+ to 0xef based on the size constraints of the u8 type.
**Case Two**:
@@ -2910,7 +2920,9 @@ option, and these limitations will be eliminated in the future:
RET
When passing 0x2_0000_0000 to it, the return value may be recorded as
- 0x2_0000_0000 instead of 0.
+ 0x2_0000_0000 instead of 0. When CONFIG_DEBUG_INFO_BTF is enabled, the
+ value will be correctly truncated to 0 based on the size constraints of
+ the int type.
You can put some comments on specific functions by using
trace_printk() For example, if you want to put a comment inside
--
2.34.1
^ permalink raw reply related
* [PATCH v4 2/3] fgraph: Enhance funcgraph-retval with BTF-based type-aware output
From: Donglin Peng @ 2025-12-15 3:41 UTC (permalink / raw)
To: rostedt
Cc: mhiramat, linux-trace-kernel, bpf, linux-kernel, pengdonglin,
Xiaoqin Zhang
In-Reply-To: <20251215034153.2367756-1-dolinux.peng@gmail.com>
From: pengdonglin <pengdonglin@xiaomi.com>
The current funcgraph-retval implementation suffers from two accuracy
issues:
1. Void-returning functions still print a return value, creating
misleading noise in the trace output.
2. For functions returning narrower types (e.g., char, short), the
displayed value can be incorrect because high bits of the register
may contain undefined data.
This patch addresses both problems by leveraging BTF to obtain the exact
return type of each traced kernel function. The key changes are:
1. Void function filtering: Functions with void return type no longer
display any return value in the trace output, eliminating unnecessary
clutter.
2. Type-aware value formatting: The return value is now properly truncated
to match the actual width of the return type before being displayed.
Additionally, the value is formatted according to its type for better
human readability.
Here is an output comparison:
Before:
# perf ftrace -G vfs_read --graph-opts retval
...
1) | touch_atime() {
1) | atime_needs_update() {
1) 0.069 us | make_vfsuid(); /* ret=0x0 */
1) 0.067 us | make_vfsgid(); /* ret=0x0 */
1) | current_time() {
1) 0.197 us | ktime_get_coarse_real_ts64_mg(); /* ret=0x187f886aec3ed6f5 */
1) 0.352 us | } /* current_time ret=0x69380753 */
1) 0.792 us | } /* atime_needs_update ret=0x0 */
1) 0.937 us | } /* touch_atime ret=0x0 */
After:
# perf ftrace -G vfs_read --graph-opts retval
...
2) | touch_atime() {
2) | atime_needs_update() {
2) 0.070 us | make_vfsuid(); /* ret=0x0 */
2) 0.070 us | make_vfsgid(); /* ret=0x0 */
2) | current_time() {
2) 0.162 us | ktime_get_coarse_real_ts64_mg();
2) 0.312 us | } /* current_time ret=0x69380649(trunc) */
2) 0.753 us | } /* atime_needs_update ret=false */
2) 0.899 us | } /* touch_atime */
Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Xiaoqin Zhang <zhangxiaoqin@xiaomi.com>
Signed-off-by: pengdonglin <pengdonglin@xiaomi.com>
---
kernel/trace/trace_functions_graph.c | 124 ++++++++++++++++++++++++---
1 file changed, 111 insertions(+), 13 deletions(-)
diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
index 17c75cf2348e..46b66b1cfc16 100644
--- a/kernel/trace/trace_functions_graph.c
+++ b/kernel/trace/trace_functions_graph.c
@@ -15,6 +15,7 @@
#include "trace.h"
#include "trace_output.h"
+#include "trace_btf.h"
/* When set, irq functions might be ignored */
static int ftrace_graph_skip_irqs;
@@ -120,6 +121,13 @@ enum {
FLAGS_FILL_END = 3 << TRACE_GRAPH_PRINT_FILL_SHIFT,
};
+enum {
+ RETVAL_FMT_HEX = BIT(0),
+ RETVAL_FMT_DEC = BIT(1),
+ RETVAL_FMT_BOOL = BIT(2),
+ RETVAL_FMT_TRUNC = BIT(3),
+};
+
static void
print_graph_duration(struct trace_array *tr, unsigned long long duration,
struct trace_seq *s, u32 flags);
@@ -865,6 +873,73 @@ static void print_graph_retaddr(struct trace_seq *s, struct fgraph_retaddr_ent_e
#if defined(CONFIG_FUNCTION_GRAPH_RETVAL) || defined(CONFIG_FUNCTION_GRAPH_RETADDR)
+static void trim_retval(unsigned long func, unsigned long *retval, bool *print_retval,
+ int *fmt)
+{
+ const struct btf_type *t;
+ char name[KSYM_NAME_LEN];
+ struct btf *btf;
+ u32 v, msb;
+ int kind;
+
+ if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
+ return;
+
+ if (lookup_symbol_name(func, name))
+ return;
+
+ t = btf_find_func_proto(name, &btf);
+ if (IS_ERR_OR_NULL(t))
+ return;
+
+ t = btf_type_skip_modifiers(btf, t->type, NULL);
+ kind = t ? BTF_INFO_KIND(t->info) : BTF_KIND_UNKN;
+ switch (kind) {
+ case BTF_KIND_UNKN:
+ *print_retval = false;
+ break;
+ case BTF_KIND_STRUCT:
+ case BTF_KIND_UNION:
+ case BTF_KIND_ENUM:
+ case BTF_KIND_ENUM64:
+ if (kind == BTF_KIND_STRUCT || kind == BTF_KIND_UNION)
+ *fmt = RETVAL_FMT_HEX;
+ else
+ *fmt = RETVAL_FMT_DEC;
+
+ if (t->size > sizeof(unsigned long)) {
+ *fmt |= RETVAL_FMT_TRUNC;
+ } else {
+ msb = BITS_PER_BYTE * t->size - 1;
+ *retval &= GENMASK(msb, 0);
+ }
+ break;
+ case BTF_KIND_INT:
+ v = *(u32 *)(t + 1);
+ if (BTF_INT_ENCODING(v) == BTF_INT_BOOL) {
+ *fmt = RETVAL_FMT_BOOL;
+ msb = 0;
+ } else {
+ if (BTF_INT_ENCODING(v) == BTF_INT_SIGNED)
+ *fmt = RETVAL_FMT_DEC;
+ else
+ *fmt = RETVAL_FMT_HEX;
+
+ if (t->size > sizeof(unsigned long)) {
+ *fmt |= RETVAL_FMT_TRUNC;
+ msb = BITS_PER_LONG - 1;
+ } else {
+ msb = BTF_INT_BITS(v) - 1;
+ }
+ }
+ *retval &= GENMASK(msb, 0);
+ break;
+ default:
+ *fmt = RETVAL_FMT_HEX;
+ break;
+ }
+}
+
static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entry *entry,
struct ftrace_graph_ret *graph_ret, void *func,
u32 opt_flags, u32 trace_flags, int args_size)
@@ -873,7 +948,7 @@ static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entr
unsigned long retval = 0;
bool print_retaddr = false;
bool print_retval = false;
- bool hex_format = !!(opt_flags & TRACE_GRAPH_PRINT_RETVAL_HEX);
+ int retval_fmt = 0;
#ifdef CONFIG_FUNCTION_GRAPH_RETVAL
retval = graph_ret->retval;
@@ -884,17 +959,35 @@ static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entr
print_retaddr = !!(opt_flags & TRACE_GRAPH_PRINT_RETADDR);
#endif
- if (print_retval && retval && !hex_format) {
- /* Check if the return value matches the negative format */
- if (IS_ENABLED(CONFIG_64BIT) && (retval & BIT(31)) &&
- (((u64)retval) >> 32) == 0) {
- err_code = sign_extend64(retval, 31);
- } else {
- err_code = retval;
+ if (print_retval) {
+ int fmt = RETVAL_FMT_HEX;
+
+ trim_retval((unsigned long)func, &retval, &print_retval, &fmt);
+ if (print_retval) {
+ if (opt_flags & TRACE_GRAPH_PRINT_RETVAL_HEX)
+ retval_fmt = RETVAL_FMT_HEX;
+
+ if (retval && retval_fmt != RETVAL_FMT_HEX) {
+ /* Check if the return value matches the negative format */
+ if (IS_ENABLED(CONFIG_64BIT) && (retval & BIT(31)) &&
+ (((u64)retval) >> 32) == 0) {
+ err_code = sign_extend64(retval, 31);
+ } else {
+ err_code = retval;
+ }
+
+ if (!IS_ERR_VALUE(err_code))
+ err_code = 0;
+ }
+
+ if (retval_fmt == RETVAL_FMT_HEX) {
+ retval_fmt |= (fmt & RETVAL_FMT_TRUNC);
+ } else {
+ if (err_code && fmt & RETVAL_FMT_HEX)
+ fmt = (fmt & ~RETVAL_FMT_HEX) | RETVAL_FMT_DEC;
+ retval_fmt = fmt;
+ }
}
-
- if (!IS_ERR_VALUE(err_code))
- err_code = 0;
}
if (entry) {
@@ -921,10 +1014,15 @@ static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entr
trace_flags, false);
if (print_retval) {
- if (hex_format || (err_code == 0))
+ if (retval_fmt & RETVAL_FMT_HEX)
trace_seq_printf(s, " ret=0x%lx", retval);
+ else if (retval_fmt & RETVAL_FMT_BOOL)
+ trace_seq_printf(s, " ret=%s", retval ? "true" : "false");
else
- trace_seq_printf(s, " ret=%ld", err_code);
+ trace_seq_printf(s, " ret=%ld", err_code ?: retval);
+
+ if (retval_fmt & RETVAL_FMT_TRUNC)
+ trace_seq_printf(s, "(trunc)");
}
if (!entry || print_retval || print_retaddr)
--
2.34.1
^ permalink raw reply related
* [PATCH v4 1/3] ftrace: Build trace_btf.c when CONFIG_DEBUG_INFO_BTF is enabled
From: Donglin Peng @ 2025-12-15 3:41 UTC (permalink / raw)
To: rostedt
Cc: mhiramat, linux-trace-kernel, bpf, linux-kernel, pengdonglin,
Xiaoqin Zhang
In-Reply-To: <20251215034153.2367756-1-dolinux.peng@gmail.com>
From: pengdonglin <pengdonglin@xiaomi.com>
The trace_btf.c file provides BTF helper functions used by the ftrace
subsystem. This change makes its compilation solely dependent on
CONFIG_DEBUG_INFO_BTF, allowing features like funcgraph-retval to also
utilize these helpers.
Additionally, the redundant dependency on CONFIG_PROBE_EVENTS_BTF_ARGS
is removed, as CONFIG_DEBUG_INFO_BTF already depends on
CONFIG_BPF_SYSCALL.
Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Xiaoqin Zhang <zhangxiaoqin@xiaomi.com>
Signed-off-by: pengdonglin <pengdonglin@xiaomi.com>
---
kernel/trace/Kconfig | 2 +-
kernel/trace/Makefile | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index e1214b9dc990..653c1fcefa4c 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -755,7 +755,7 @@ config FPROBE_EVENTS
config PROBE_EVENTS_BTF_ARGS
depends on HAVE_FUNCTION_ARG_ACCESS_API
depends on FPROBE_EVENTS || KPROBE_EVENTS
- depends on DEBUG_INFO_BTF && BPF_SYSCALL
+ depends on DEBUG_INFO_BTF
bool "Support BTF function arguments for probe events"
default y
help
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index fc5dcc888e13..6c4bf5a6c4f3 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -116,7 +116,7 @@ obj-$(CONFIG_KGDB_KDB) += trace_kdb.o
endif
obj-$(CONFIG_DYNAMIC_EVENTS) += trace_dynevent.o
obj-$(CONFIG_PROBE_EVENTS) += trace_probe.o
-obj-$(CONFIG_PROBE_EVENTS_BTF_ARGS) += trace_btf.o
+obj-$(CONFIG_DEBUG_INFO_BTF) += trace_btf.o
obj-$(CONFIG_UPROBE_EVENTS) += trace_uprobe.o
obj-$(CONFIG_BOOTTIME_TRACING) += trace_boot.o
obj-$(CONFIG_FTRACE_RECORD_RECURSION) += trace_recursion_record.o
--
2.34.1
^ permalink raw reply related
* [PATCH v4 0/3] Use BTF to trim return values
From: Donglin Peng @ 2025-12-15 3:41 UTC (permalink / raw)
To: rostedt; +Cc: mhiramat, linux-trace-kernel, bpf, linux-kernel, pengdonglin
From: pengdonglin <pengdonglin@xiaomi.com>
This patch series addresses two limitations of the funcgraph-retval feature:
1. Void-returning functions still print a return value, creating misleading
noise in the trace output.
2. For functions returning narrower types (e.g., char, short), the displayed
value can be incorrect because high bits of the register may contain
undefined data.
By leveraging BTF to obtain precise return type information, we now:
1. Void function filtering: Functions with void return type no longer
display any return value in the trace output, eliminating unnecessary
clutter.
2. Type-aware value formatting: The return value is now properly truncated to
match the actual width of the return type before being displayed.
Additionally, the value is formatted according to its type for better human
readability.
Here is an output comparison:
Before:
# perf ftrace -G vfs_read --graph-opts retval
...
1) | touch_atime() {
1) | atime_needs_update() {
1) 0.069 us | make_vfsuid(); /* ret=0x0 */
1) 0.067 us | make_vfsgid(); /* ret=0x0 */
1) | current_time() {
1) 0.197 us | ktime_get_coarse_real_ts64_mg(); /* ret=0x187f886aec3ed6f5 */
1) 0.352 us | } /* current_time ret=0x69380753 */
1) 0.792 us | } /* atime_needs_update ret=0x0 */
1) 0.937 us | } /* touch_atime ret=0x0 */
After:
# perf ftrace -G vfs_read --graph-opts retval
...
2) | touch_atime() {
2) | atime_needs_update() {
2) 0.070 us | make_vfsuid(); /* ret=0x0 */
2) 0.070 us | make_vfsgid(); /* ret=0x0 */
2) | current_time() {
2) 0.162 us | ktime_get_coarse_real_ts64_mg();
2) 0.312 us | } /* current_time ret=0x69380649(trunc) */
2) 0.753 us | } /* atime_needs_update ret=false */
2) 0.899 us | } /* touch_atime */
Note: enabling funcgraph-retval now adds overhead due to repeated btf_find_by_name_kind()
calls during trace output. A separate series [1] optimizes this function with
binary search (O(log n) vs current O(n)), which will greatly reduce the impact.
Here is a performance comparison:
1. Original funcgraph-retval:
# time cat trace | wc -l
101024
real 0m0.682s
user 0m0.000s
sys 0m0.695s
2. Enhanced funcgraph-retval:
# time cat trace | wc -l
99326
real 0m12.886s
user 0m0.010s
sys 0m12.680s
3. Enhanced funcgraph-retval + optimizined btf_find_by_name_kind:
# time cat trace | wc -l
102922
real 0m0.794s
user 0m0.000s
sys 0m0.810s
We can see that after optimizing btf_find_by_name_kind, the overhead
becomes negligible.
Changelog:
v4:
- Build trace_btf.c only when CONFIG_DEBUG_INFO_BTF is enabled
- Remove the redundant dependency of CONFIG_PROBE_EVENTS_BTF_ARGS
- Update related documentation and the cover letter
v3:
- Link: https://lore.kernel.org/all/20251209121349.525641-1-dolinux.peng@gmail.com/
- Print the return value based on its type for human readability, thanks Masami
- Update documentation and cover letter
v2:
- Link: https://lore.kernel.org/all/20251208131917.2444620-1-dolinux.peng@gmail.com/
- Update the funcgraph-retval documentation
- Revise the cover letter
v1:
- Link: https://lore.kernel.org/all/20251207142742.229924-1-dolinux.peng@gmail.com/
[1] https://lore.kernel.org/all/20251208062353.1702672-1-dolinux.peng@gmail.com/
pengdonglin (3):
ftrace: Build trace_btf.c when CONFIG_DEBUG_INFO_BTF is enabled
fgraph: Enhance funcgraph-retval with BTF-based type-aware output
tracing: Update funcgraph-retval documentation
Documentation/trace/ftrace.rst | 88 +++++++++++--------
kernel/trace/Kconfig | 2 +-
kernel/trace/Makefile | 2 +-
kernel/trace/trace_functions_graph.c | 124 ++++++++++++++++++++++++---
4 files changed, 163 insertions(+), 53 deletions(-)
--
2.34.1
^ permalink raw reply
* Re: [PATCH v3 1/2] fgraph: Enhance funcgraph-retval with BTF-based type-aware output
From: kernel test robot @ 2025-12-14 1:14 UTC (permalink / raw)
To: Donglin Peng, rostedt
Cc: oe-kbuild-all, mhiramat, linux-trace-kernel, bpf, linux-kernel,
pengdonglin, Xiaoqin Zhang
In-Reply-To: <20251209121349.525641-2-dolinux.peng@gmail.com>
Hi Donglin,
kernel test robot noticed the following build errors:
[auto build test ERROR on trace/for-next]
[also build test ERROR on linus/master v6.18 next-20251212]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Donglin-Peng/fgraph-Enhance-funcgraph-retval-with-BTF-based-type-aware-output/20251209-201633
base: https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link: https://lore.kernel.org/r/20251209121349.525641-2-dolinux.peng%40gmail.com
patch subject: [PATCH v3 1/2] fgraph: Enhance funcgraph-retval with BTF-based type-aware output
config: arm-randconfig-002-20251214 (https://download.01.org/0day-ci/archive/20251214/202512140850.JdD1lPmn-lkp@intel.com/config)
compiler: arm-linux-gnueabi-gcc (GCC) 10.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251214/202512140850.JdD1lPmn-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202512140850.JdD1lPmn-lkp@intel.com/
All errors (new ones prefixed by >>):
arm-linux-gnueabi-ld: kernel/trace/trace_functions_graph.o: in function `trim_retval':
>> kernel/trace/trace_functions_graph.c:888: undefined reference to `btf_find_func_proto'
vim +888 kernel/trace/trace_functions_graph.c
872
873 static void trim_retval(unsigned long func, unsigned long *retval, bool *print_retval,
874 int *fmt)
875 {
876 const struct btf_type *t;
877 char name[KSYM_NAME_LEN];
878 struct btf *btf;
879 u32 v, msb;
880 int kind;
881
882 if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
883 return;
884
885 if (lookup_symbol_name(func, name))
886 return;
887
> 888 t = btf_find_func_proto(name, &btf);
889 if (IS_ERR_OR_NULL(t))
890 return;
891
892 t = btf_type_skip_modifiers(btf, t->type, NULL);
893 kind = t ? BTF_INFO_KIND(t->info) : BTF_KIND_UNKN;
894 switch (kind) {
895 case BTF_KIND_UNKN:
896 *print_retval = false;
897 break;
898 case BTF_KIND_STRUCT:
899 case BTF_KIND_UNION:
900 case BTF_KIND_ENUM:
901 case BTF_KIND_ENUM64:
902 if (kind == BTF_KIND_STRUCT || kind == BTF_KIND_UNION)
903 *fmt = RETVAL_FMT_HEX;
904 else
905 *fmt = RETVAL_FMT_DEC;
906
907 if (t->size > sizeof(unsigned long)) {
908 *fmt |= RETVAL_FMT_TRUNC;
909 } else {
910 msb = BITS_PER_BYTE * t->size - 1;
911 *retval &= GENMASK(msb, 0);
912 }
913 break;
914 case BTF_KIND_INT:
915 v = *(u32 *)(t + 1);
916 if (BTF_INT_ENCODING(v) == BTF_INT_BOOL) {
917 *fmt = RETVAL_FMT_BOOL;
918 msb = 0;
919 } else {
920 if (BTF_INT_ENCODING(v) == BTF_INT_SIGNED)
921 *fmt = RETVAL_FMT_DEC;
922 else
923 *fmt = RETVAL_FMT_HEX;
924
925 if (t->size > sizeof(unsigned long)) {
926 *fmt |= RETVAL_FMT_TRUNC;
927 msb = BITS_PER_LONG - 1;
928 } else {
929 msb = BTF_INT_BITS(v) - 1;
930 }
931 }
932 *retval &= GENMASK(msb, 0);
933 break;
934 default:
935 *fmt = RETVAL_FMT_HEX;
936 break;
937 }
938 }
939
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* [PATCH v10 1/3] lib: Introduce hierarchical per-cpu counters
From: Mathieu Desnoyers @ 2025-12-13 18:56 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
In-Reply-To: <20251213185608.3418096-1-mathieu.desnoyers@efficios.com>
* Motivation
The purpose of this hierarchical split-counter scheme is to:
- Minimize contention when incrementing and decrementing counters,
- Provide fast access to a sum approximation,
- Provide a sum approximation with an acceptable accuracy level when
scaling to many-core systems.
- Provide approximate and precise comparison of two counters, and
between a counter and a value.
It aims at fixing the per-mm RSS tracking which has become too
inaccurate for OOM killer purposes on large many-core systems [1].
* Design
The hierarchical per-CPU counters propagate a sum approximation through
a N-way tree. When reaching the batch size, the carry is propagated
through a binary tree which consists of logN(nr_cpu_ids) levels. The
batch size for each level is twice the batch size of the prior level.
Example propagation diagram with 8 cpus through a binary tree:
Level 0: 0 1 2 3 4 5 6 7
| / | / | / | /
| / | / | / | /
| / | / | / | /
Level 1: 0 1 2 3
| / | /
| / | /
| / | /
Level 2: 0 1
| /
| /
| /
Level 3: 0
For a binary tree, the maximum inaccuracy is bound by:
batch_size * log2(nr_cpus) * nr_cpus
which evolves with O(n*log(n)) as the number of CPUs increases.
For a N-way tree, the maximum inaccuracy can be pre-calculated
based on the the N-arity of each level and the batch size.
Link: https://lore.kernel.org/lkml/20250331223516.7810-2-sweettea-kernel@dorminy.me/ # [1]
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
---
Changes since v9:
- Introduce kerneldoc documentation.
- Document structure fields.
- Remove inline from percpu_counter_tree_add().
- Reject batch_size==1 which makes no sense.
- Fix copy-paste bug in percpu_counter_tree_precise_compare()
(wrong inequality comparison).
- Rename "inaccuracy" to "accuracy", which makes it easier to
document.
- Track accuracy limits more precisely. In two's complement
signed integers, the range before a n-bit underflow is one
unit larger than the range before a n-bit overflow. This sums
for all the counters within the tree. Therefore the "under"
vs "over" accuracy is not symmetrical.
Changes since v8:
- Remove migrate guard from the fast path. It does not
matter through which path the carry is propagated up
the tree.
- Rebase on top of v6.18-rc6.
- Introduce percpu_counter_tree_init_many and
percpu_counter_tree_destroy_many APIs.
- Move tree items allocation to the caller.
- Introduce percpu_counter_tree_items_size().
- Move percpu_counter_tree_subsystem_init() call before mm_core_init()
so percpu_counter_tree_items_size() is initialized before it is used.
Changes since v7:
- Explicitly initialize the subsystem from start_kernel() right
after mm_core_init() so it is up and running before the creation of
the first mm at boot.
- Remove the module.h include which is not needed with the explicit
initialization.
- Only consider levels>0 items for order={0,1} nr_items. No
functional change except to allocate only the amount of memory
which is strictly needed.
- Introduce positive precise sum API to handle a scenario where an
unlucky precise sum iteration would hit negative counter values
concurrently with counter updates.
Changes since v5:
- Introduce percpu_counter_tree_approximate_sum_positive.
- Introduce !CONFIG_SMP static inlines for UP build.
- Remove percpu_counter_tree_set_bias from the public API and make it
static.
Changes since v3:
- Add gfp flags to init function.
Changes since v2:
- Introduce N-way tree to reduce tree depth on larger systems.
Changes since v1:
- Remove percpu_counter_tree_precise_sum_unbiased from public header,
make this function static,
- Introduce precise and approximate comparisons between two counters,
- Reorder the struct percpu_counter_tree fields,
- Introduce approx_sum field, which points to the approximate sum
for the percpu_counter_tree_approximate_sum() fast path.
---
include/linux/percpu_counter_tree.h | 242 ++++++++++
init/main.c | 2 +
lib/Makefile | 1 +
lib/percpu_counter_tree.c | 705 ++++++++++++++++++++++++++++
4 files changed, 950 insertions(+)
create mode 100644 include/linux/percpu_counter_tree.h
create mode 100644 lib/percpu_counter_tree.c
diff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h
new file mode 100644
index 000000000000..0daf09e08111
--- /dev/null
+++ b/include/linux/percpu_counter_tree.h
@@ -0,0 +1,242 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* SPDX-FileCopyrightText: 2025 Mathieu Desnoyers <mathieu.desnoyers@efficios.com> */
+
+#ifndef _PERCPU_COUNTER_TREE_H
+#define _PERCPU_COUNTER_TREE_H
+
+#include <linux/preempt.h>
+#include <linux/atomic.h>
+#include <linux/percpu.h>
+
+#ifdef CONFIG_SMP
+
+struct percpu_counter_tree_level_item {
+ atomic_t count; /*
+ * Count the number of carry fort this tree item.
+ * The carry counter is kept at the order of the
+ * carry accounted for at this tree level.
+ */
+} ____cacheline_aligned_in_smp;
+
+struct percpu_counter_tree {
+ /* Fast-path fields. */
+ unsigned int __percpu *level0; /* Pointer to per-CPU split counters (tree level 0). */
+ unsigned int level0_bit_mask; /* Bit mask to apply to detect carry propagation from tree level 0. */
+ union {
+ unsigned int *i; /* Approximate sum for single-CPU topology. */
+ atomic_t *a; /* Approximate sum for SMP topology. */
+ } approx_sum;
+ int bias; /* Bias to apply to counter precise and approximate values. */
+
+ /* Slow-path fields. */
+ struct percpu_counter_tree_level_item *items; /* Array of tree items for levels 1 to N. */
+ unsigned int batch_size; /*
+ * The batch size is the increment step at level 0 which
+ * triggers a carry propagation. The batch size is required
+ * to be greater than 1, and a power of 2.
+ */
+ /*
+ * The tree approximate sum is guaranteed to be within this accuracy range:
+ * (precise_sum - approx_accuracy_range.under) <= approx_sum <= (precise_sum + approx_accuracy_range.over).
+ * This accuracy is derived from the hardware topology and the tree batch_size.
+ * The "under" accuracy is larger than the "over" accuracy because the negative range of a
+ * two's complement signed integer is one unit larger than the positive range. This delta
+ * is summed for each tree item, which leads to a significantly larger "under" accuracy range
+ * compared to the "over" accuracy range.
+ */
+ struct {
+ unsigned int under;
+ unsigned int over;
+ } approx_accuracy_range;
+};
+
+size_t percpu_counter_tree_items_size(void);
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned int batch_size, gfp_t gfp_flags);
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned int batch_size, gfp_t gfp_flags);
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters);
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, int inc);
+int percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, int v);
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, int v);
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, int v);
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned int *under, unsigned int *over);
+int percpu_counter_tree_subsystem_init(void);
+
+/**
+ * percpu_counter_tree_approximate_sum() - Return approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the approximate sum is fast, but it is only accurate within
+ * the bounds delimited by percpu_counter_tree_approximate_accuracy_range().
+ * This is meant to be used when speed is preferred over accuracy.
+ *
+ * Return: The current approximate counter sum.
+ */
+static inline
+int percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ unsigned int v;
+
+ if (!counter->level0_bit_mask)
+ v = READ_ONCE(*counter->approx_sum.i);
+ else
+ v = atomic_read(counter->approx_sum.a);
+ return (int) (v + (unsigned int)READ_ONCE(counter->bias));
+}
+
+#else /* !CONFIG_SMP */
+
+struct percpu_counter_tree_level_item;
+
+struct percpu_counter_tree {
+ atomic_t count;
+};
+
+static inline
+size_t percpu_counter_tree_items_size(void)
+{
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned int batch_size, gfp_t gfp_flags)
+{
+ for (unsigned int i = 0; i < nr_counters; i++)
+ atomic_set(&counters[i].count, 0);
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned int batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);
+}
+
+static inline
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters)
+{
+}
+
+static inline
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+}
+
+static inline
+int percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return atomic_read(&counter->count);
+}
+
+static inline
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ int count_a = percpu_counter_tree_precise_sum(a),
+ count_b = percpu_counter_tree_precise_sum(b);
+
+ if (count_a == count_b)
+ return 0;
+ if (count_a < count_b)
+ return -1;
+ return 1;
+}
+
+static inline
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, int v)
+{
+ int count = percpu_counter_tree_precise_sum(counter);
+
+ if (count == v)
+ return 0;
+ if (count < v)
+ return -1;
+ return 1;
+}
+
+static inline
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ return percpu_counter_tree_precise_compare(a, b);
+}
+
+static inline
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, int v)
+{
+ return percpu_counter_tree_precise_compare_value(counter, v);
+}
+
+static inline
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, int v)
+{
+ atomic_set(&counter->count, v);
+}
+
+static inline
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned int *under, unsigned int *over)
+{
+ *under = 0;
+ *over = 0;
+}
+
+static inline
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, int inc)
+{
+ atomic_add(inc, &counter->count);
+}
+
+static inline
+int percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum(counter);
+}
+
+static inline
+int percpu_counter_tree_subsystem_init(void)
+{
+ return 0;
+}
+
+#endif /* CONFIG_SMP */
+
+/**
+ * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return an approximate counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive approximate counter sum.
+ */
+static inline
+int percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)
+{
+ int v = percpu_counter_tree_approximate_sum(counter);
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return a precise counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive precise counter sum.
+ */
+static inline
+int percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)
+{
+ int v = percpu_counter_tree_precise_sum(counter);
+ return v > 0 ? v : 0;
+}
+
+#endif /* _PERCPU_COUNTER_TREE_H */
diff --git a/init/main.c b/init/main.c
index 07a3116811c5..8487489b9d00 100644
--- a/init/main.c
+++ b/init/main.c
@@ -104,6 +104,7 @@
#include <linux/pidfs.h>
#include <linux/ptdump.h>
#include <linux/time_namespace.h>
+#include <linux/percpu_counter_tree.h>
#include <net/net_namespace.h>
#include <asm/io.h>
@@ -968,6 +969,7 @@ void start_kernel(void)
vfs_caches_init_early();
sort_main_extable();
trap_init();
+ percpu_counter_tree_subsystem_init();
mm_core_init();
maple_tree_init();
poking_init();
diff --git a/lib/Makefile b/lib/Makefile
index 1ab2c4be3b66..767dc178a55c 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -179,6 +179,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o
obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o
obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o
obj-$(CONFIG_SMP) += percpu_counter.o
+obj-$(CONFIG_SMP) += percpu_counter_tree.o
obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o
diff --git a/lib/percpu_counter_tree.c b/lib/percpu_counter_tree.c
new file mode 100644
index 000000000000..9b7ae7df5530
--- /dev/null
+++ b/lib/percpu_counter_tree.c
@@ -0,0 +1,705 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2025 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+/*
+ * Split Counters With Tree Approximation Propagation
+ *
+ * * Propagation diagram when reaching batch size thresholds (± batch size):
+ *
+ * Example diagram for 8 CPUs:
+ *
+ * log2(8) = 3 levels
+ *
+ * At each level, each pair propagates its values to the next level when
+ * reaching the batch size thresholds.
+ *
+ * Counters at levels 0, 1, 2 can be kept on a single byte ([-128 .. +127] range),
+ * although it may be relevant to keep them on 32-bit counters for
+ * simplicity. (complexity vs memory footprint tradeoff)
+ *
+ * Counter at level 3 can be kept on a 32-bit counter.
+ *
+ * Level 0: 0 1 2 3 4 5 6 7
+ * | / | / | / | /
+ * | / | / | / | /
+ * | / | / | / | /
+ * Level 1: 0 1 2 3
+ * | / | /
+ * | / | /
+ * | / | /
+ * Level 2: 0 1
+ * | /
+ * | /
+ * | /
+ * Level 3: 0
+ *
+ * * Approximation accuracy:
+ *
+ * BATCH(level N): Level N batch size.
+ *
+ * Example for BATCH(level 0) = 32.
+ *
+ * BATCH(level 0) = 32
+ * BATCH(level 1) = 64
+ * BATCH(level 2) = 128
+ * BATCH(level N) = BATCH(level 0) * 2^N
+ *
+ * per-counter global
+ * accuracy accuracy
+ * Level 0: [ -32 .. +31] ±256 (8 * 32)
+ * Level 1: [ -64 .. +63] ±256 (4 * 64)
+ * Level 2: [-128 .. +127] ±256 (2 * 128)
+ * Total: ------ ±768 (log2(nr_cpu_ids) * BATCH(level 0) * nr_cpu_ids)
+ *
+ * Note that the global accuracy can be calculated more precisely
+ * by taking into account that the positive accuracy range is
+ * 31 rather than 32.
+ *
+ * -----
+ *
+ * Approximate Sum Carry Propagation
+ *
+ * Let's define a number of counter bits for each level, e.g.:
+ *
+ * log2(BATCH(level 0)) = log2(32) = 5
+ *
+ * nr_bit value_mask range
+ * Level 0: 5 bits v 0 .. +31
+ * Level 1: 1 bit (v & ~((1UL << 5) - 1)) 0 .. +63
+ * Level 2: 1 bit (v & ~((1UL << 6) - 1)) 0 .. +127
+ * Level 3: 25 bits (v & ~((1UL << 7) - 1)) 0 .. 2^32-1
+ *
+ * Note: Use a full 32-bit per-cpu counter at level 0 to allow precise sum.
+ *
+ * Note: Use cacheline aligned counters at levels above 0 to prevent false sharing.
+ * If memory footprint is an issue, a specialized allocator could be used
+ * to eliminate padding.
+ *
+ * Example with expanded values:
+ *
+ * counter_add(counter, inc):
+ *
+ * if (!inc)
+ * return;
+ *
+ * res = percpu_add_return(counter @ Level 0, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b00011111); // Clear used bits
+ * // xor bit 5: underflow
+ * if ((inc ^ orig ^ res) & 0b00100000)
+ * inc -= 0b00100000;
+ * } else {
+ * inc &= ~0b00011111; // Clear used bits
+ * // xor bit 5: overflow
+ * if ((inc ^ orig ^ res) & 0b00100000)
+ * inc += 0b00100000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * res = atomic_add_return(counter @ Level 1, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b00111111); // Clear used bits
+ * // xor bit 6: underflow
+ * if ((inc ^ orig ^ res) & 0b01000000)
+ * inc -= 0b01000000;
+ * } else {
+ * inc &= ~0b00111111; // Clear used bits
+ * // xor bit 6: overflow
+ * if ((inc ^ orig ^ res) & 0b01000000)
+ * inc += 0b01000000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * res = atomic_add_return(counter @ Level 2, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b01111111); // Clear used bits
+ * // xor bit 7: underflow
+ * if ((inc ^ orig ^ res) & 0b10000000)
+ * inc -= 0b10000000;
+ * } else {
+ * inc &= ~0b01111111; // Clear used bits
+ * // xor bit 7: overflow
+ * if ((inc ^ orig ^ res) & 0b10000000)
+ * inc += 0b10000000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * atomic_add(counter @ Level 3, inc);
+ */
+
+#include <linux/percpu_counter_tree.h>
+#include <linux/cpumask.h>
+#include <linux/percpu.h>
+#include <linux/atomic.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/math.h>
+
+#define MAX_NR_LEVELS 5
+
+/*
+ * The counter configuration is selected at boot time based on the
+ * hardware topology.
+ */
+struct counter_config {
+ unsigned int nr_items; /*
+ * nr_items is the number of items in the tree for levels 1
+ * up to and including the final level (approximate sum).
+ * It excludes the level 0 per-CPU counters.
+ */
+ unsigned char nr_levels; /*
+ * nr_levels is the number of hierarchical counter tree levels.
+ * It excludes the final level (approximate sum).
+ */
+ unsigned char n_arity_order[MAX_NR_LEVELS]; /*
+ * n-arity of tree nodes for each level from
+ * 0 to (nr_levels - 1).
+ */
+};
+
+static const struct counter_config per_nr_cpu_order_config[] = {
+ [0] = { .nr_items = 0, .nr_levels = 0, .n_arity_order = { 0 } },
+ [1] = { .nr_items = 1, .nr_levels = 1, .n_arity_order = { 1 } },
+ [2] = { .nr_items = 3, .nr_levels = 2, .n_arity_order = { 1, 1 } },
+ [3] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 1, 1, 1 } },
+ [4] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 2, 1, 1 } },
+ [5] = { .nr_items = 11, .nr_levels = 3, .n_arity_order = { 2, 2, 1 } },
+ [6] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 2, 2, 2 } },
+ [7] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 3, 2, 2 } },
+ [8] = { .nr_items = 37, .nr_levels = 3, .n_arity_order = { 3, 3, 2 } },
+ [9] = { .nr_items = 73, .nr_levels = 3, .n_arity_order = { 3, 3, 3 } },
+ [10] = { .nr_items = 149, .nr_levels = 4, .n_arity_order = { 3, 3, 2, 2 } },
+ [11] = { .nr_items = 293, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 2 } },
+ [12] = { .nr_items = 585, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 3 } },
+ [13] = { .nr_items = 1173, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 2, 2 } },
+ [14] = { .nr_items = 2341, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 2 } },
+ [15] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 3 } },
+ [16] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 4, 3, 3, 3, 3 } },
+ [17] = { .nr_items = 8777, .nr_levels = 5, .n_arity_order = { 4, 4, 3, 3, 3 } },
+ [18] = { .nr_items = 17481, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 3, 3 } },
+ [19] = { .nr_items = 34953, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 3 } },
+ [20] = { .nr_items = 69905, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 4 } },
+};
+
+static const struct counter_config *counter_config; /* Hierarchical counter configuration for the hardware topology. */
+static unsigned int nr_cpus_order, /* Order of nr_cpu_ids. */
+ accuracy_multiplier; /* Calculate accuracy for a given batch size (multiplication factor). */
+
+static
+int __percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned int batch_size, gfp_t gfp_flags,
+ unsigned int __percpu *level0,
+ struct percpu_counter_tree_level_item *items)
+{
+ /* Batch size must be greater than 1, and a power of 2. */
+ if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1))))
+ return -EINVAL;
+ counter->batch_size = batch_size;
+ counter->bias = 0;
+ counter->level0 = level0;
+ counter->items = items;
+ if (!nr_cpus_order) {
+ counter->approx_sum.i = per_cpu_ptr(counter->level0, 0);
+ counter->level0_bit_mask = 0;
+ } else {
+ counter->approx_sum.a = &counter->items[counter_config->nr_items - 1].count;
+ counter->level0_bit_mask = 1UL << get_count_order(batch_size);
+ }
+ /*
+ * Each tree item signed integer has a negative range which is
+ * one unit greater than the positive range.
+ */
+ counter->approx_accuracy_range.under = batch_size * accuracy_multiplier;
+ counter->approx_accuracy_range.over = (batch_size - 1) * accuracy_multiplier;
+ return 0;
+}
+
+/**
+ * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.
+ * @counters: An array of @nr_counters counters to initialize.
+ * Their memory is provided by the caller.
+ * @items: Pointer to memory area where to store tree items.
+ * This memory is provided by the caller.
+ * Its size needs to be at least @nr_counters * percpu_counter_tree_items_size().
+ * @nr_counters: The number of counter trees to initialize
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, and a power of 2.
+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.
+ *
+ * Initialize many per-CPU counter trees using a single per-CPU
+ * allocator invocation for @nr_counters counters.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned int batch_size, gfp_t gfp_flags)
+{
+ void __percpu *level0, *level0_iter;
+ size_t counter_size, items_size = 0;
+ void *items_iter;
+ unsigned int i;
+ int ret;
+
+ counter_size = ALIGN(sizeof(*counters), __alignof__(*counters));
+ level0 = __alloc_percpu_gfp(nr_counters * counter_size,
+ __alignof__(*counters), gfp_flags);
+ if (!level0)
+ return -ENOMEM;
+ if (nr_cpus_order) {
+ items_size = percpu_counter_tree_items_size();
+ memset(items, 0, items_size * nr_counters);
+ }
+ level0_iter = level0;
+ items_iter = items;
+ for (i = 0; i < nr_counters; i++) {
+ ret = __percpu_counter_tree_init(&counters[i], batch_size, gfp_flags, level0_iter, items_iter);
+ if (ret)
+ goto free_level0;
+ level0_iter += counter_size;
+ if (nr_cpus_order)
+ items_iter += items_size;
+ }
+ return 0;
+
+free_level0:
+ free_percpu(level0);
+ return ret;
+}
+
+/**
+ * percpu_counter_tree_init() - Initialize one per-CPU counter tree.
+ * @counter: Counter to initialize.
+ * Its memory is provided by the caller.
+ * @items: Pointer to memory area where to store tree items.
+ * This memory is provided by the caller.
+ * Its size needs to be at least percpu_counter_tree_items_size().
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, and a power of 2.
+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.
+ *
+ * Initialize one per-CPU counter tree.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned int batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);
+}
+
+/**
+ * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.
+ * @counters: Array of counters trees to destroy.
+ * @nr_counters: The number of counter trees to destroy.
+ *
+ * Release internal resources allocated for @nr_counters per-CPU counter trees.
+ */
+
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters, unsigned int nr_counters)
+{
+ free_percpu(counters->level0);
+}
+
+/**
+ * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.
+ * @counter: Counter to destroy.
+ *
+ * Release internal resources allocated for one per-CPU counter tree.
+ */
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_destroy_many(counter, 1);
+}
+
+static
+int percpu_counter_tree_carry(int orig, int res, int inc, unsigned int bit_mask)
+{
+ if (inc < 0) {
+ inc = -(-inc & ~(bit_mask - 1));
+ /*
+ * xor bit_mask: underflow.
+ *
+ * If inc has bit set, decrement an additional bit if
+ * there is _no_ bit transition between orig and res.
+ * Else, inc has bit cleared, decrement an additional
+ * bit if there is a bit transition between orig and
+ * res.
+ */
+ if ((inc ^ orig ^ res) & bit_mask)
+ inc -= bit_mask;
+ } else {
+ inc &= ~(bit_mask - 1);
+ /*
+ * xor bit_mask: overflow.
+ *
+ * If inc has bit set, increment an additional bit if
+ * there is _no_ bit transition between orig and res.
+ * Else, inc has bit cleared, increment an additional
+ * bit if there is a bit transition between orig and
+ * res.
+ */
+ if ((inc ^ orig ^ res) & bit_mask)
+ inc += bit_mask;
+ }
+ return inc;
+}
+
+/*
+ * It does not matter through which path the carry propagates up the
+ * tree, therefore there is no need to disable preemption because the
+ * cpu number is only used to favor cache locality.
+ */
+static
+void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, int inc)
+{
+ unsigned int level_items, nr_levels = counter_config->nr_levels,
+ level, n_arity_order, bit_mask;
+ struct percpu_counter_tree_level_item *item = counter->items;
+ unsigned int cpu = raw_smp_processor_id();
+
+ WARN_ON_ONCE(!nr_cpus_order); /* Should never be called for 1 cpu. */
+
+ n_arity_order = counter_config->n_arity_order[0];
+ bit_mask = counter->level0_bit_mask << n_arity_order;
+ level_items = 1U << (nr_cpus_order - n_arity_order);
+
+ for (level = 1; level < nr_levels; level++) {
+ atomic_t *count = &item[cpu & (level_items - 1)].count;
+ unsigned int orig, res;
+
+ res = atomic_add_return_relaxed(inc, count);
+ orig = res - inc;
+ inc = percpu_counter_tree_carry(orig, res, inc, bit_mask);
+ if (likely(!inc))
+ return;
+ item += level_items;
+ n_arity_order = counter_config->n_arity_order[level];
+ level_items >>= n_arity_order;
+ bit_mask <<= n_arity_order;
+ }
+ atomic_add(inc, counter->approx_sum.a);
+}
+
+/**
+ * percpu_counter_tree_add() - Add to a per-CPU counter tree.
+ * @counter: Counter added to.
+ * @inc: Increment value (either positive or negative).
+ *
+ * Add @inc to a per-CPU counter tree. This is a fast-path which will
+ * typically increment per-CPU counters as long as there is no carry
+ * greater or equal to the counter tree batch size.
+ */
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, int inc)
+{
+ unsigned int bit_mask = counter->level0_bit_mask, orig, res;
+
+ res = this_cpu_add_return(*counter->level0, inc);
+ orig = res - inc;
+ inc = percpu_counter_tree_carry(orig, res, inc, bit_mask);
+ if (likely(!inc))
+ return;
+ percpu_counter_tree_add_slowpath(counter, inc);
+}
+
+
+static
+int percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)
+{
+ unsigned int sum = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ sum += *per_cpu_ptr(counter->level0, cpu);
+ return (int) sum;
+}
+
+/**
+ * percpu_counter_tree_precise_sum() - Return precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the precise sum is relatively expensive because it needs to
+ * iterate over all CPUs.
+ * This is meant to be used when accuracy is preferred over speed.
+ *
+ * Return: The current precise counter sum.
+ */
+int percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter->bias);
+}
+
+static
+int compare_delta(int delta, unsigned int accuracy_neg, unsigned int accuracy_pos)
+{
+ if (delta >= 0) {
+ if (delta <= accuracy_pos)
+ return 0;
+ else
+ return 1;
+ } else {
+ if (-delta <= accuracy_neg)
+ return 0;
+ else
+ return -1;
+ }
+}
+
+/**
+ * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate an approximate comparison of two counter trees.
+ * This approximation comparison is fast, and provides an accurate
+ * answer if the counters are found to be either less than or greater
+ * than the other. However, if the approximated comparison returns
+ * 0, the counters respective sums are found to be within the two
+ * counters accuracy range.
+ *
+ * Return:
+ * * %0 - Counters @a and @b do not differ by more than the sum of their respective
+ * accuracy ranges.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ return compare_delta(percpu_counter_tree_approximate_sum(a) - percpu_counter_tree_approximate_sum(b),
+ a->approx_accuracy_range.over + b->approx_accuracy_range.under,
+ a->approx_accuracy_range.under + b->approx_accuracy_range.over);
+}
+
+/**
+ * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate an approximate comparison of a counter tree against a given value.
+ * This approximation comparison is fast, and provides an accurate
+ * answer if the counter is found to be either less than or greater
+ * than the value. However, if the approximated comparison returns
+ * 0, the value is within the counter accuracy range.
+ *
+ * Return:
+ * * %0 - The value @v is within the accuracy range of the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, int v)
+{
+ return compare_delta(v - percpu_counter_tree_approximate_sum(counter),
+ counter->approx_accuracy_range.under,
+ counter->approx_accuracy_range.over);
+}
+
+/**
+ * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate a precise comparison of two counter trees.
+ * As an optimization, it uses the approximate counter comparison
+ * to quickly compare counters which are far apart. Only cases where
+ * counter sums are within the accuracy range require precise counter
+ * sums.
+ *
+ * Return:
+ * * %0 - Counters are equal.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ int count_a = percpu_counter_tree_approximate_sum(a),
+ count_b = percpu_counter_tree_approximate_sum(b);
+ unsigned int accuracy_a, accuracy_b;
+ int delta = count_a - count_b;
+ int res;
+
+ res = compare_delta(delta,
+ a->approx_accuracy_range.over + b->approx_accuracy_range.under,
+ a->approx_accuracy_range.under + b->approx_accuracy_range.over);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /*
+ * The approximated comparison is within the accuracy range, therefore at least one
+ * precise sum is needed. Sum the counter which has the largest accuracy first.
+ */
+ if (delta >= 0) {
+ accuracy_a = a->approx_accuracy_range.under;
+ accuracy_b = b->approx_accuracy_range.over;
+ } else {
+ accuracy_a = a->approx_accuracy_range.over;
+ accuracy_b = b->approx_accuracy_range.under;
+ }
+ if (accuracy_b < accuracy_a) {
+ count_a = percpu_counter_tree_precise_sum(a);
+ res = compare_delta(count_a - count_b,
+ b->approx_accuracy_range.under,
+ b->approx_accuracy_range.over);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_b = percpu_counter_tree_precise_sum(b);
+ } else {
+ count_b = percpu_counter_tree_precise_sum(b);
+ res = compare_delta(count_a - count_b,
+ a->approx_accuracy_range.over,
+ a->approx_accuracy_range.under);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_a = percpu_counter_tree_precise_sum(a);
+ }
+ if (count_a - count_b < 0)
+ return -1;
+ if (count_a - count_b > 0)
+ return 1;
+ return 0;
+}
+
+/**
+ * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate a precise comparison of a counter tree against a given value.
+ * As an optimization, it uses the approximate counter comparison
+ * to quickly identify whether the counter and value are far apart.
+ * Only cases where the value is within the counter accuracy range
+ * require a precise counter sum.
+ *
+ * Return:
+ * * %0 - The value @v is equal to the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, int v)
+{
+ int count = percpu_counter_tree_approximate_sum(counter);
+ int res;
+
+ res = compare_delta(v - count,
+ counter->approx_accuracy_range.under,
+ counter->approx_accuracy_range.over);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /* Precise sum is required. */
+ count = percpu_counter_tree_precise_sum(counter);
+ if (v - count < 0)
+ return -1;
+ if (v - count > 0)
+ return 1;
+ return 0;
+}
+
+static
+void percpu_counter_tree_set_bias(struct percpu_counter_tree *counter, int bias)
+{
+ WRITE_ONCE(counter->bias, bias);
+}
+
+/**
+ * percpu_counter_tree_set - Set the counter tree sum to a given value.
+ * @counter: Counter to set.
+ * @v: Value to set.
+ *
+ * Set the counter sum to a given value. It can be useful for instance
+ * to reset the counter sum to 0. Note that even after setting the
+ * counter sum to a given value, the counter sum approximation can
+ * return any value within the accuracy range around that value.
+ */
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, int v)
+{
+ percpu_counter_tree_set_bias(counter,
+ v - percpu_counter_tree_precise_sum_unbiased(counter));
+}
+
+/**
+ * percpu_counter_tree_approximate_accuracy_range - Query the accuracy range for a counter tree.
+ * @counter: Counter to query.
+ * @under: Pointer where to store the to the accuracy range below the approximation.
+ * @over: Pointer where to store the to the accuracy range above the approximation.
+ *
+ * Query the accuracy range limits for the counter.
+ * Because of two's complement binary representation, the "under" range is typically
+ * slightly larger than the "over" range.
+ * Those values are derived from the hardware topology and the counter tree batch size.
+ * They are invariant for a given counter tree.
+ * Using this function should not be typically required, see the following functions instead:
+ * * percpu_counter_tree_approximate_compare(),
+ * * percpu_counter_tree_approximate_compare_value(),
+ * * percpu_counter_tree_precise_compare(),
+ * * percpu_counter_tree_precise_compare_value().
+ */
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned int *under, unsigned int *over)
+{
+ *under = counter->approx_accuracy_range.under;
+ *over = counter->approx_accuracy_range.over;
+}
+
+/*
+ * percpu_counter_tree_items_size - Query the size required for counter tree items.
+ *
+ * Query the size of the memory area required to hold the counter tree
+ * items. This depends on the hardware topology and is invariant after
+ * boot.
+ *
+ * Return: Size required to hold tree items.
+ */
+size_t percpu_counter_tree_items_size(void)
+{
+ if (!nr_cpus_order)
+ return 0;
+ return counter_config->nr_items * sizeof(struct percpu_counter_tree_level_item);
+}
+
+static void __init calculate_accuracy_topology(void)
+{
+ unsigned int nr_levels = counter_config->nr_levels, level;
+ unsigned int level_items = 1U << nr_cpus_order;
+ unsigned int batch_size = 1;
+
+ for (level = 0; level < nr_levels; level++) {
+ unsigned int n_arity_order = counter_config->n_arity_order[level];
+
+ /*
+ * The accuracy multiplier is derived from a batch size of 1
+ * to speed up calculating the accuracy at tree initialization.
+ */
+ accuracy_multiplier += batch_size * level_items;
+ batch_size <<= n_arity_order;
+ level_items >>= n_arity_order;
+ }
+}
+
+int __init percpu_counter_tree_subsystem_init(void)
+{
+ nr_cpus_order = get_count_order(nr_cpu_ids);
+ if (WARN_ON_ONCE(nr_cpus_order >= ARRAY_SIZE(per_nr_cpu_order_config))) {
+ printk(KERN_ERR "Unsupported number of CPUs (%u)\n", nr_cpu_ids);
+ return -1;
+ }
+ counter_config = &per_nr_cpu_order_config[nr_cpus_order];
+ calculate_accuracy_topology();
+ return 0;
+}
--
2.39.5
^ permalink raw reply related
* [PATCH v10 2/3] mm: Fix OOM killer inaccuracy on large many-core systems
From: Mathieu Desnoyers @ 2025-12-13 18:56 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
In-Reply-To: <20251213185608.3418096-1-mathieu.desnoyers@efficios.com>
Use hierarchical per-cpu counters for rss tracking to fix the per-mm RSS
tracking which has become too inaccurate for OOM killer purposes on
large many-core systems.
The following rss tracking issues were noted by Sweet Tea Dorminy [1],
which lead to picking wrong tasks as OOM kill target:
Recently, several internal services had an RSS usage regression as part of a
kernel upgrade. Previously, they were on a pre-6.2 kernel and were able to
read RSS statistics in a backup watchdog process to monitor and decide if
they'd overrun their memory budget. Now, however, a representative service
with five threads, expected to use about a hundred MB of memory, on a 250-cpu
machine had memory usage tens of megabytes different from the expected amount
-- this constituted a significant percentage of inaccuracy, causing the
watchdog to act.
This was a result of commit f1a7941243c1 ("mm: convert mm's rss stats
into percpu_counter") [1]. Previously, the memory error was bounded by
64*nr_threads pages, a very livable megabyte. Now, however, as a result of
scheduler decisions moving the threads around the CPUs, the memory error could
be as large as a gigabyte.
This is a really tremendous inaccuracy for any few-threaded program on a
large machine and impedes monitoring significantly. These stat counters are
also used to make OOM killing decisions, so this additional inaccuracy could
make a big difference in OOM situations -- either resulting in the wrong
process being killed, or in less memory being returned from an OOM-kill than
expected.
Here is a (possibly incomplete) list of the prior approaches that were
used or proposed, along with their downside:
1) Per-thread rss tracking: large error on many-thread processes.
2) Per-CPU counters: up to 12% slower for short-lived processes and 9%
increased system time in make test workloads [1]. Moreover, the
inaccuracy increases with O(n^2) with the number of CPUs.
3) Per-NUMA-node counters: requires atomics on fast-path (overhead),
error is high with systems that have lots of NUMA nodes (32 times
the number of NUMA nodes).
The approach proposed here is to replace this by the hierarchical
per-cpu counters, which bounds the inaccuracy based on the system
topology with O(N*logN).
commit 82241a83cd15 ("mm: fix the inaccurate memory statistics issue for
users") introduced get_mm_counter_sum() for precise /proc memory status
queries. Implement it with percpu_counter_tree_precise_sum() since it
is not a fast path and precision is preferred over speed.
* Testing results:
Test hardware: 2 sockets AMD EPYC 9654 96-Core Processor (384 logical CPUs total)
Methodology:
Comparing the current upstream implementation with the hierarchical
counters is done by keeping both implementations wired up in parallel,
and running a single-process, single-threaded program which hops
randomly across CPUs in the system, calling mmap(2) and munmap(2) on
random CPUs, keeping track of an array of allocated mappings, randomly
choosing entries to either map or unmap.
get_mm_counter() is instrumented to compare the upstream counter
approximation to the precise value, and print the delta when going over
a given threshold. The delta of the hierarchical counter approximation
to the precise value is also printed for comparison.
After a few minutes running this test, the upstream implementation
counter approximation reaches a 1GB delta from the
precise value, compared to 80MB delta with the hierarchical counter.
The hierarchical counter provides a guaranteed maximum approximation
inaccuracy of 192MB on that hardware topology.
* Fast path implementation comparison
The new inline percpu_counter_tree_add() uses a this_cpu_add_return()
for the fast path (under a certain allocation size threshold). Above
that, it calls a slow path which "trickles up" the carry to upper level
counters with atomic_add_return.
In comparison, the upstream counters implementation calls
percpu_counter_add_batch which uses this_cpu_try_cmpxchg() on the fast
path, and does a raw_spin_lock_irqsave above a certain threshold.
The hierarchical implementation is therefore expected to have less
contention on mid-sized allocations than the upstream counters because
the atomic counters tracking those bits are only shared across nearby
CPUs. In comparison, the upstream counters immediately use a global
spinlock when reaching the threshold.
* Benchmarks
Using will-it-scale page_fault1 benchmarks to compare the upstream
counters to the hierarchical counters. This is done with hyperthreading
disabled. The speedup is within the standard deviation of the upstream
runs, so the overhead is not significant.
upstream hierarchical speedup
page_fault1_processes -s 100 -t 1 614783 615558 +0.1%
page_fault1_threads -s 100 -t 1 612788 612447 -0.1%
page_fault1_processes -s 100 -t 96 37994977 37932035 -0.2%
page_fault1_threads -s 100 -t 96 2484130 2504860 +0.8%
page_fault1_processes -s 100 -t 192 71262917 71118830 -0.2%
page_fault1_threads -s 100 -t 192 2446437 2469296 +0.1%
Link: https://lore.kernel.org/lkml/20250331223516.7810-2-sweettea-kernel@dorminy.me/ # [1]
Link: https://lore.kernel.org/lkml/20250704150226.47980-1-mathieu.desnoyers@efficios.com/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
---
Changes since v8:
- Use percpu_counter_tree_init_many and
percpu_counter_tree_destroy_many APIs.
- Remove percpu tree items allocation. Extend mm_struct size to include
rss items. Those are handled through the new helpers
get_rss_stat_items() and get_rss_stat_items_size() and passed
as parameter to percpu_counter_tree_init_many().
Changes since v7:
- Use precise sum positive API to handle a scenario where an unlucky
precise sum iteration would observe negative counter values due to
concurrent updates.
Changes since v6:
- Rebased on v6.18-rc3.
- Implement get_mm_counter_sum as percpu_counter_tree_precise_sum for
/proc virtual files memory state queries.
Changes since v5:
- Use percpu_counter_tree_approximate_sum_positive.
Change since v4:
- get_mm_counter needs to return 0 or a positive value.
get_mm_counter_sum -> precise sum positive
---
include/linux/mm.h | 28 +++++++++++++++++++++++-----
include/linux/mm_types.h | 4 ++--
include/trace/events/kmem.h | 2 +-
kernel/fork.c | 24 ++++++++++++++----------
4 files changed, 40 insertions(+), 18 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 7c79b3369b82..cd81fa8924eb 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2681,38 +2681,56 @@ static inline bool get_user_page_fast_only(unsigned long addr,
{
return get_user_pages_fast_only(addr, 1, gup_flags, pagep) == 1;
}
+
+static inline size_t get_rss_stat_items_size(void)
+{
+ return percpu_counter_tree_items_size() * NR_MM_COUNTERS;
+}
+
+static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct mm_struct *mm)
+{
+ unsigned long ptr = (unsigned long)mm;
+
+ ptr += offsetof(struct mm_struct, cpu_bitmap);
+ /* Skip cpu_bitmap */
+ ptr += cpumask_size();
+ /* Skip mm_cidmask */
+ ptr += mm_cid_size();
+ return (struct percpu_counter_tree_level_item *)ptr;
+}
+
/*
* per-process(per-mm_struct) statistics.
*/
static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
{
- return percpu_counter_read_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
}
static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)
{
- return percpu_counter_sum_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
}
void mm_trace_rss_stat(struct mm_struct *mm, int member);
static inline void add_mm_counter(struct mm_struct *mm, int member, long value)
{
- percpu_counter_add(&mm->rss_stat[member], value);
+ percpu_counter_tree_add(&mm->rss_stat[member], value);
mm_trace_rss_stat(mm, member);
}
static inline void inc_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_inc(&mm->rss_stat[member]);
+ percpu_counter_tree_add(&mm->rss_stat[member], 1);
mm_trace_rss_stat(mm, member);
}
static inline void dec_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_dec(&mm->rss_stat[member]);
+ percpu_counter_tree_add(&mm->rss_stat[member], -1);
mm_trace_rss_stat(mm, member);
}
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 90e5790c318f..adb2f227bac7 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -18,7 +18,7 @@
#include <linux/page-flags-layout.h>
#include <linux/workqueue.h>
#include <linux/seqlock.h>
-#include <linux/percpu_counter.h>
+#include <linux/percpu_counter_tree.h>
#include <linux/types.h>
#include <linux/bitmap.h>
@@ -1119,7 +1119,7 @@ struct mm_struct {
unsigned long saved_e_flags;
#endif
- struct percpu_counter rss_stat[NR_MM_COUNTERS];
+ struct percpu_counter_tree rss_stat[NR_MM_COUNTERS];
struct linux_binfmt *binfmt;
diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index 7f93e754da5c..91c81c44f884 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -442,7 +442,7 @@ TRACE_EVENT(rss_stat,
__entry->mm_id = mm_ptr_to_hash(mm);
__entry->curr = !!(current->mm == mm);
__entry->member = member;
- __entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member])
+ __entry->size = (percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member])
<< PAGE_SHIFT);
),
diff --git a/kernel/fork.c b/kernel/fork.c
index 3da0f08615a9..5430d63c9e2d 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -133,6 +133,11 @@
*/
#define MAX_THREADS FUTEX_TID_MASK
+/*
+ * Batch size of rss stat approximation
+ */
+#define RSS_STAT_BATCH_SIZE 32
+
/*
* Protected counters by write_lock_irq(&tasklist_lock)
*/
@@ -583,14 +588,12 @@ static void check_mm(struct mm_struct *mm)
"Please make sure 'struct resident_page_types[]' is updated as well");
for (i = 0; i < NR_MM_COUNTERS; i++) {
- long x = percpu_counter_sum(&mm->rss_stat[i]);
-
- if (unlikely(x)) {
- pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n",
- mm, resident_page_types[i], x,
+ if (unlikely(percpu_counter_tree_precise_compare_value(&mm->rss_stat[i], 0) != 0))
+ pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%d Comm:%s Pid:%d\n",
+ mm, resident_page_types[i],
+ percpu_counter_tree_precise_sum(&mm->rss_stat[i]),
current->comm,
task_pid_nr(current));
- }
}
if (mm_pgtables_bytes(mm))
@@ -688,7 +691,7 @@ void __mmdrop(struct mm_struct *mm)
put_user_ns(mm->user_ns);
mm_pasid_drop(mm);
mm_destroy_cid(mm);
- percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
+ percpu_counter_tree_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
free_mm(mm);
}
@@ -1083,8 +1086,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
if (mm_alloc_cid(mm, p))
goto fail_cid;
- if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT,
- NR_MM_COUNTERS))
+ if (percpu_counter_tree_init_many(mm->rss_stat, get_rss_stat_items(mm),
+ NR_MM_COUNTERS, RSS_STAT_BATCH_SIZE,
+ GFP_KERNEL_ACCOUNT))
goto fail_pcpu;
mm->user_ns = get_user_ns(user_ns);
@@ -2964,7 +2968,7 @@ void __init mm_cache_init(void)
* dynamically sized based on the maximum CPU number this system
* can have, taking hotplug into account (nr_cpu_ids).
*/
- mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size();
+ mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size() + get_rss_stat_items_size();
mm_cachep = kmem_cache_create_usercopy("mm_struct",
mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
--
2.39.5
^ permalink raw reply related
* [PATCH v10 3/3] mm: Implement precise OOM killer task selection
From: Mathieu Desnoyers @ 2025-12-13 18:56 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
In-Reply-To: <20251213185608.3418096-1-mathieu.desnoyers@efficios.com>
Use the hierarchical tree counter approximation to implement the OOM
killer task selection with a 2-pass algorithm. The first pass selects
the process that has the highest badness points approximation, and the
second pass compares each process using the current max badness points
approximation.
The second pass uses an approximate comparison to eliminate all processes
which are below the current max badness points approximation accuracy
range.
Summing the per-CPU counters to calculate the precise badness of tasks
is only required for tasks with an approximate badness within the
accuracy range of the current max points value.
Limit to 16 the maximum number of badness sums allowed for an OOM killer
task selection before falling back to the approximated comparison. This
ensures bounded execution time for scenarios where many tasks have
badness within the accuracy of the maximum badness approximation.
Tested with the following script:
#!/bin/sh
for a in $(seq 1 10); do (tail /dev/zero &); done
sleep 5
for a in $(seq 1 10); do (tail /dev/zero &); done
sleep 2
for a in $(seq 1 10); do (tail /dev/zero &); done
echo "Waiting for tasks to finish"
wait
Results: OOM kill order on a 128GB memory system
================================================
* systemd and sd-pam are chosen first due to their oom_score_ajd:100:
Out of memory: Killed process 3502 (systemd) total-vm:20096kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:72kB oom_score_adj:100
Out of memory: Killed process 3503 ((sd-pam)) total-vm:21432kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:76kB oom_score_adj:100
* The first batch of 10 processes are gradually killed, consecutively
picking the one that uses the most memory. The fact that we are
freeing memory from the previous processes increases the threshold
at which the remaining processes of that group are killed. Processes
from the second and third batches of 10 processes have time to start
before we complete killing the first 10 processes:
Out of memory: Killed process 3703 (tail) total-vm:6591280kB, anon-rss:6578176kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:12936kB oom_score_adj:0
Out of memory: Killed process 3705 (tail) total-vm:6731716kB, anon-rss:6709248kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13212kB oom_score_adj:0
Out of memory: Killed process 3707 (tail) total-vm:6977216kB, anon-rss:6946816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13692kB oom_score_adj:0
Out of memory: Killed process 3699 (tail) total-vm:7205640kB, anon-rss:7184384kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14136kB oom_score_adj:0
Out of memory: Killed process 3713 (tail) total-vm:7463204kB, anon-rss:7438336kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14644kB oom_score_adj:0
Out of memory: Killed process 3701 (tail) total-vm:7739204kB, anon-rss:7716864kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15180kB oom_score_adj:0
Out of memory: Killed process 3709 (tail) total-vm:8050176kB, anon-rss:8028160kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15792kB oom_score_adj:0
Out of memory: Killed process 3711 (tail) total-vm:8362236kB, anon-rss:8339456kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16404kB oom_score_adj:0
Out of memory: Killed process 3715 (tail) total-vm:8649360kB, anon-rss:8634368kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16972kB oom_score_adj:0
Out of memory: Killed process 3697 (tail) total-vm:8951788kB, anon-rss:8929280kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:17560kB oom_score_adj:0
* Even though there is a 2 seconds delay between the 2nd and 3rd batches
those appear to execute in mixed order. Therefore, let's consider them
as a single batch of 20 processes. We are hitting oom at a lower
memory threshold because at this point the 20 remaining proceses are
running rather than the previous 10. The process with highest memory
usage is selected for oom, thus making room for the remaining
processes so they can use more memory before they fill the available
memory, thus explaining why the memory use for selected processes
gradually increases, until all system memory is used by the last one:
Out of memory: Killed process 3731 (tail) total-vm:7089868kB, anon-rss:7077888kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13912kB oom_score_adj:0
Out of memory: Killed process 3721 (tail) total-vm:7417248kB, anon-rss:7405568kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14556kB oom_score_adj:0
Out of memory: Killed process 3729 (tail) total-vm:7795864kB, anon-rss:7766016kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15300kB oom_score_adj:0
Out of memory: Killed process 3723 (tail) total-vm:8259620kB, anon-rss:8224768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16208kB oom_score_adj:0
Out of memory: Killed process 3737 (tail) total-vm:8695984kB, anon-rss:8667136kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:17060kB oom_score_adj:0
Out of memory: Killed process 3735 (tail) total-vm:9295980kB, anon-rss:9265152kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:18240kB oom_score_adj:0
Out of memory: Killed process 3727 (tail) total-vm:9907900kB, anon-rss:9895936kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:19428kB oom_score_adj:0
Out of memory: Killed process 3719 (tail) total-vm:10631248kB, anon-rss:10600448kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:20844kB oom_score_adj:0
Out of memory: Killed process 3733 (tail) total-vm:11341720kB, anon-rss:11321344kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22232kB oom_score_adj:0
Out of memory: Killed process 3725 (tail) total-vm:12348124kB, anon-rss:12320768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:24204kB oom_score_adj:0
Out of memory: Killed process 3759 (tail) total-vm:12978888kB, anon-rss:12967936kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:25440kB oom_score_adj:0
Out of memory: Killed process 3751 (tail) total-vm:14386412kB, anon-rss:14352384kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:28196kB oom_score_adj:0
Out of memory: Killed process 3741 (tail) total-vm:16153168kB, anon-rss:16130048kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31652kB oom_score_adj:0
Out of memory: Killed process 3753 (tail) total-vm:18414856kB, anon-rss:18391040kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:36076kB oom_score_adj:0
Out of memory: Killed process 3745 (tail) total-vm:21389456kB, anon-rss:21356544kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:41904kB oom_score_adj:0
Out of memory: Killed process 3747 (tail) total-vm:25659348kB, anon-rss:25632768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:50260kB oom_score_adj:0
Out of memory: Killed process 3755 (tail) total-vm:32030820kB, anon-rss:32006144kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:62720kB oom_score_adj:0
Out of memory: Killed process 3743 (tail) total-vm:42648456kB, anon-rss:42614784kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:83504kB oom_score_adj:0
Out of memory: Killed process 3757 (tail) total-vm:63971028kB, anon-rss:63938560kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:125228kB oom_score_adj:0
Out of memory: Killed process 3749 (tail) total-vm:127799660kB, anon-rss:127778816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:250140kB oom_score_adj:0
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
---
fs/proc/base.c | 2 +-
include/linux/mm.h | 34 +++++++++++++++++----
include/linux/oom.h | 12 +++++++-
mm/oom_kill.c | 72 +++++++++++++++++++++++++++++++++++++--------
4 files changed, 101 insertions(+), 19 deletions(-)
diff --git a/fs/proc/base.c b/fs/proc/base.c
index 6299878e3d97..fa48411835ba 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -589,7 +589,7 @@ static int proc_oom_score(struct seq_file *m, struct pid_namespace *ns,
unsigned long points = 0;
long badness;
- badness = oom_badness(task, totalpages);
+ badness = oom_badness(task, totalpages, false, NULL, NULL);
/*
* Special case OOM_SCORE_ADJ_MIN for all others scale the
* badness value into [0, 2000] range which we have been
diff --git a/include/linux/mm.h b/include/linux/mm.h
index cd81fa8924eb..84a67036d010 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2702,14 +2702,32 @@ static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct m
/*
* per-process(per-mm_struct) statistics.
*/
+static inline unsigned long __get_mm_counter(struct mm_struct *mm, int member, bool approximate,
+ unsigned int *accuracy_under, unsigned int *accuracy_over)
+{
+ if (approximate) {
+ if (accuracy_under && accuracy_over) {
+ unsigned int under, over;
+
+ percpu_counter_tree_approximate_accuracy_range(&mm->rss_stat[member], &under, &over);
+ *accuracy_under += under;
+ *accuracy_over += over;
+ }
+ return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
+ } else {
+ return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
+ }
+}
+
static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
{
- return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
+ return __get_mm_counter(mm, member, true, NULL, NULL);
}
+
static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)
{
- return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
+ return get_mm_counter(mm, member);
}
void mm_trace_rss_stat(struct mm_struct *mm, int member);
@@ -2750,11 +2768,17 @@ static inline int mm_counter(struct folio *folio)
return mm_counter_file(folio);
}
+static inline unsigned long __get_mm_rss(struct mm_struct *mm, bool approximate,
+ unsigned int *accuracy_under, unsigned int *accuracy_over)
+{
+ return __get_mm_counter(mm, MM_FILEPAGES, approximate, accuracy_under, accuracy_over) +
+ __get_mm_counter(mm, MM_ANONPAGES, approximate, accuracy_under, accuracy_over) +
+ __get_mm_counter(mm, MM_SHMEMPAGES, approximate, accuracy_under, accuracy_over);
+}
+
static inline unsigned long get_mm_rss(struct mm_struct *mm)
{
- return get_mm_counter(mm, MM_FILEPAGES) +
- get_mm_counter(mm, MM_ANONPAGES) +
- get_mm_counter(mm, MM_SHMEMPAGES);
+ return __get_mm_rss(mm, true, NULL, NULL);
}
static inline unsigned long get_mm_hiwater_rss(struct mm_struct *mm)
diff --git a/include/linux/oom.h b/include/linux/oom.h
index 7b02bc1d0a7e..ef3919e4c7f2 100644
--- a/include/linux/oom.h
+++ b/include/linux/oom.h
@@ -48,6 +48,13 @@ struct oom_control {
unsigned long totalpages;
struct task_struct *chosen;
long chosen_points;
+ bool approximate;
+ /*
+ * Number of precise badness points sums performed by this task
+ * selection.
+ */
+ int nr_precise;
+ unsigned long accuracy_under;
/* Used to print the constraint info. */
enum oom_constraint constraint;
@@ -97,7 +104,10 @@ static inline vm_fault_t check_stable_address_space(struct mm_struct *mm)
}
long oom_badness(struct task_struct *p,
- unsigned long totalpages);
+ unsigned long totalpages,
+ bool approximate,
+ unsigned int *accuracy_under,
+ unsigned int *accuracy_over);
extern bool out_of_memory(struct oom_control *oc);
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index c145b0feecc1..e3f6ca500701 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -53,6 +53,14 @@
#define CREATE_TRACE_POINTS
#include <trace/events/oom.h>
+/*
+ * Maximum number of badness sums allowed before using an approximated
+ * comparison. This ensures bounded execution time for scenarios where
+ * many tasks have badness within the accuracy of the maximum badness
+ * approximation.
+ */
+static int max_precise_badness_sums = 16;
+
static int sysctl_panic_on_oom;
static int sysctl_oom_kill_allocating_task;
static int sysctl_oom_dump_tasks = 1;
@@ -194,12 +202,16 @@ static bool should_dump_unreclaim_slab(void)
* oom_badness - heuristic function to determine which candidate task to kill
* @p: task struct of which task we should calculate
* @totalpages: total present RAM allowed for page allocation
+ * @approximate: whether the value can be approximated
+ * @accuracy_under: accuracy of the badness value approximation (under value)
+ * @accuracy_over: accuracy of the badness value approximation (over value)
*
* The heuristic for determining which task to kill is made to be as simple and
* predictable as possible. The goal is to return the highest value for the
* task consuming the most memory to avoid subsequent oom failures.
*/
-long oom_badness(struct task_struct *p, unsigned long totalpages)
+long oom_badness(struct task_struct *p, unsigned long totalpages, bool approximate,
+ unsigned int *accuracy_under, unsigned int *accuracy_over)
{
long points;
long adj;
@@ -228,7 +240,8 @@ long oom_badness(struct task_struct *p, unsigned long totalpages)
* The baseline for the badness score is the proportion of RAM that each
* task's rss, pagetable and swap space use.
*/
- points = get_mm_rss(p->mm) + get_mm_counter(p->mm, MM_SWAPENTS) +
+ points = __get_mm_rss(p->mm, approximate, accuracy_under, accuracy_over) +
+ __get_mm_counter(p->mm, MM_SWAPENTS, approximate, accuracy_under, accuracy_over) +
mm_pgtables_bytes(p->mm) / PAGE_SIZE;
task_unlock(p);
@@ -309,6 +322,7 @@ static enum oom_constraint constrained_alloc(struct oom_control *oc)
static int oom_evaluate_task(struct task_struct *task, void *arg)
{
struct oom_control *oc = arg;
+ unsigned int accuracy_under = 0, accuracy_over = 0;
long points;
if (oom_unkillable_task(task))
@@ -339,9 +353,28 @@ static int oom_evaluate_task(struct task_struct *task, void *arg)
goto select;
}
- points = oom_badness(task, oc->totalpages);
- if (points == LONG_MIN || points < oc->chosen_points)
- goto next;
+ points = oom_badness(task, oc->totalpages, true, &accuracy_under, &accuracy_over);
+ if (oc->approximate) {
+ if (points == LONG_MIN || points < oc->chosen_points)
+ goto next;
+ } else {
+ /*
+ * Eliminate processes which are below the chosen
+ * points accuracy range with an approximation.
+ */
+ if (points == LONG_MIN || (long)(points + accuracy_over + oc->accuracy_under - oc->chosen_points) < 0)
+ goto next;
+
+ if (oc->nr_precise < max_precise_badness_sums) {
+ accuracy_under = 0;
+ accuracy_over = 0;
+ oc->nr_precise++;
+ /* Precise evaluation. */
+ points = oom_badness(task, oc->totalpages, false, NULL, NULL);
+ if (points == LONG_MIN || (long)(points + oc->accuracy_under - oc->chosen_points) < 0)
+ goto next;
+ }
+ }
select:
if (oc->chosen)
@@ -349,6 +382,7 @@ static int oom_evaluate_task(struct task_struct *task, void *arg)
get_task_struct(task);
oc->chosen = task;
oc->chosen_points = points;
+ oc->accuracy_under = accuracy_under;
next:
return 0;
abort:
@@ -358,14 +392,8 @@ static int oom_evaluate_task(struct task_struct *task, void *arg)
return 1;
}
-/*
- * Simple selection loop. We choose the process with the highest number of
- * 'points'. In case scan was aborted, oc->chosen is set to -1.
- */
-static void select_bad_process(struct oom_control *oc)
+static void select_bad_process_iter(struct oom_control *oc)
{
- oc->chosen_points = LONG_MIN;
-
if (is_memcg_oom(oc))
mem_cgroup_scan_tasks(oc->memcg, oom_evaluate_task, oc);
else {
@@ -379,6 +407,26 @@ static void select_bad_process(struct oom_control *oc)
}
}
+/*
+ * Simple selection loop. We choose the process with the highest number of
+ * 'points'. In case scan was aborted, oc->chosen is set to -1.
+ */
+static void select_bad_process(struct oom_control *oc)
+{
+ oc->chosen_points = LONG_MIN;
+ oc->accuracy_under = 0;
+ oc->nr_precise = 0;
+
+ /* Approximate scan. */
+ oc->approximate = true;
+ select_bad_process_iter(oc);
+ if (oc->chosen == (void *)-1UL)
+ return;
+ /* Precise scan. */
+ oc->approximate = false;
+ select_bad_process_iter(oc);
+}
+
static int dump_task(struct task_struct *p, void *arg)
{
struct oom_control *oc = arg;
--
2.39.5
^ permalink raw reply related
* Re: [PATCH v6 4/6] perf script: Display PERF_RECORD_CALLCHAIN_DEFERRED
From: Jens Remus @ 2025-12-12 12:11 UTC (permalink / raw)
To: Namhyung Kim, Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Mathieu Desnoyers, linux-trace-kernel, bpf, Heiko Carstens,
Vasily Gorbik
In-Reply-To: <20251120234804.156340-5-namhyung@kernel.org>
Hello Namhyung,
following is an observation from my attempt to enable unwind user fp on
s390 using s390 back chain instead of frame pointer and relaxing the
s390-specific IP validation check.
When capturing call graphs of a Java application the list of "unwound"
user space IPs may contain invalid entries, such as 0x0, 0xdeaddeaf,
and 0xffffffffffffff. IPs that exceed PERF_CONTEXT_MAX, such as the
latter, cause perf not to display any deferred (or merged) call chain.
Note that this is not caused by your patch series.
While re-adding the s390-specific IP checks would "hide" those, I found
that the call graphs look good otherwise. That is the back chain seems
to be intact. It is just the user space application (e.g. Java JRE) not
correctly adhering to the ABI and saving the return address to the
specified location on the stack, causing bogus IPs to be reported.
Could perf be improved to handle those user space IPs that exceed
PERF_CONTEXT_MAX?
Is there otherwise guidance how unwind user and/or the s390
implementation should deal with such IPs? Should it stop taking the
deferred calltrace? Should it substitute those with e.g 0, so that
perf can display them?
Sample for IP == deaddeaf (perf displays this correctly):
java 1084 33.086790: DEFERRED CALLCHAIN [cookie: 2000001f9]
3ff983f071c java.lang.String CryptoBench.crypt(java.lang.String)+0x89c (/tmp/perf-1082.map)
3ff983ff894 void CryptoBench.execute()+0x94 (/tmp/perf-1082.map)
! --> deaddeaf [unknown] ([unknown])
3ff97e37900 StubRoutines (initial stubs)+0x80 (/tmp/perf-1082.map)
3ff97e41080 Interpreter+0x3300 (/tmp/perf-1082.map)
3ffae2d11de JavaCalls::call(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x3e (/usr/lib/jvm/.../libjvm.so)
3ffae38df92 jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*) [clone .constprop.1]+0x242 (/usr/lib/jvm/.../libjvm.so)
3ffae390e86 jni_CallStaticVoidMethod+0x116 (/usr/lib/jvm/.../libjvm.so)
3ffaf08b07e JavaMain+0x113e (/usr/lib/jvm/.../libjli.so)
3ffaf08e040 ThreadJavaMain+0x20 (/usr/lib/jvm/.../libjli.so)
3ffaedabbd8 start_thread+0x198 (/usr/lib64/libc.so.6)
3ffaee2b950 thread_start+0x10 (/usr/lib64/libc.so.6)
Sample for IP == 0 (perf displays this correctly):
java 1084 33.021987: DEFERRED CALLCHAIN [cookie: 20000017b]
3ff983f067c java.lang.String CryptoBench.crypt(java.lang.String)+0x7fc (/tmp/perf-1082.map)
3ff9098aa88 void CryptoBench.execute()+0x748 (/tmp/perf-1082.map)
! --> 0 [unknown] ([unknown])
3ff97e37900 StubRoutines (initial stubs)+0x80 (/tmp/perf-1082.map)
3ff97e41080 Interpreter+0x3300 (/tmp/perf-1082.map)
3ffae2d11de JavaCalls::call(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x3e (/usr/lib/jvm/.../libjvm.so)
3ffae38df92 jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*) [clone .constprop.1]+0x242 (/usr/lib/jvm/.../libjvm.so)
3ffae390e86 jni_CallStaticVoidMethod+0x116 (/usr/lib/jvm/.../libjvm.so)
3ffaf08b07e JavaMain+0x113e (/usr/lib/jvm/.../libjli.so)
3ffaf08e040 ThreadJavaMain+0x20 (/usr/lib/jvm/.../libjli.so)
3ffaedabbd8 start_thread+0x198 (/usr/lib64/libc.so.6)
3ffaee2b950 thread_start+0x10 (/usr/lib64/libc.so.6)
Note that for the IP==0 case I am thinking about adding a common unwind
user check, to stop taking the deferred calltrace.
Sample for IP == ffffffffffffff (perf does not display any call chain):
# perf script
...
java 1084 44.004346: 1001001 task-clock:ppp:
...
[next entry]
# perf script --no-merge-callchain
...
java 1084 44.004346: 1001001 task-clock:ppp:
400000079 (cookie) ([unknown])
java 1084 44.004348: DEFERRED CALLCHAIN [cookie: 400000079]
...
[next entry]
# perf report -D
...
44004346257 0x17718 [0x40]: PERF_RECORD_SAMPLE(IP, 0x2): 1082/1084: 0x3ffa3e413aa period: 1001001 addr: 0
... FP chain: nr:2
..... 0: fffffffffffffd80
..... 1: 0000000400000079
...... (deferred)
... thread: java:1084
...... dso: /tmp/perf-1082.map
0x17758@perf.data [0xd0]: event: 22
.
. ... raw event: size 208 bytes
. 0000: 00 00 00 16 00 02 00 d0 00 00 00 04 00 00 00 79 ...............y
. 0010: 00 00 00 00 00 00 00 15 00 00 03 ff a3 e4 13 aa ................
. 0020: 00 00 03 ff 38 09 e2 d0 00 00 03 ff 38 09 e1 30 ....8.......8..0
. 0030: 00 00 03 ff b9 5f df 68 00 00 00 00 00 00 00 00 ....._.h........
. 0040: 00 00 03 ff b9 5f e1 28 00 00 03 ff b9 5f e1 d0 ....._.(....._..
. 0050: 00 57 80 88 8e 76 47 a5 00 00 03 ff a3 e4 37 f2 .W...vG.......7.
. 0060: ff ff ff ff ff ff ff ff 00 00 03 ff a3 e4 a1 fc ................
. 0070: 00 00 00 00 00 00 00 00 00 00 03 ff a3 e3 79 00 ..............y.
. 0080: 00 00 03 ff a3 e4 10 80 00 00 03 ff b9 dd 11 de ................
. 0090: 00 00 03 ff b9 e8 df 92 00 00 03 ff b9 e9 0e 86 ................
. 00a0: 00 00 03 ff ba b8 b0 7e 00 00 03 ff ba b8 e0 40 .......~.......@
. 00b0: 00 00 03 ff ba 8a bb d8 00 00 03 ff ba 92 b9 50 ...............P
. 00c0: 00 00 04 3a 00 00 04 3c 00 00 00 0a 3e dd 13 c0 ...:...<....>...
44004348864 0x17758 [0xd0]: PERF_RECORD_CALLCHAIN_DEFERRED(IP, 0x2): 1082/1084: 0x400000079
... FP chain: nr:21
..... 0: 000003ffa3e413aa
..... 1: 000003ff3809e2d0
..... 2: 000003ff3809e130
..... 3: 000003ffb95fdf68
..... 4: 0000000000000000
..... 5: 000003ffb95fe128
..... 6: 000003ffb95fe1d0
..... 7: 005780888e7647a5
..... 8: 000003ffa3e437f2
..... 9: ffffffffffffffff <-- !
..... 10: 000003ffa3e4a1fc
..... 11: 0000000000000000
..... 12: 000003ffa3e37900
..... 13: 000003ffa3e41080
..... 14: 000003ffb9dd11de
..... 15: 000003ffb9e8df92
..... 16: 000003ffb9e90e86
..... 17: 000003ffbab8b07e
..... 18: 000003ffbab8e040
..... 19: 000003ffba8abbd8
..... 20: 000003ffba92b950
: unhandled!
...
[next entry]
On 11/21/2025 12:48 AM, Namhyung Kim wrote:
> Handle the deferred callchains in the script output.
>
> $ perf script
> ...
> pwd 2312 121.163435: 249113 cpu/cycles/P:
> ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms])
> ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms])
> ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms])
> ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms])
> ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms])
> ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms])
> ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kallsyms])
> b00000006 (cookie) ([unknown])
>
> pwd 2312 121.163447: DEFERRED CALLCHAIN [cookie: b00000006]
> 7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
> 7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
> 7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
> diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
> +static int process_deferred_sample_event(const struct perf_tool *tool,
> + union perf_event *event,
> + struct perf_sample *sample,
> + struct evsel *evsel,
> + struct machine *machine)
> +{
> + perf_sample__fprintf_start(scr, sample, al.thread, evsel,
> + PERF_RECORD_CALLCHAIN_DEFERRED, fp);
> + fprintf(fp, "DEFERRED CALLCHAIN [cookie: %llx]",
> + (unsigned long long)event->callchain_deferred.cookie);
> +
> + if (PRINT_FIELD(IP)) {
> + struct callchain_cursor *cursor = NULL;
> +
> + if (symbol_conf.use_callchain && sample->callchain) {
> + cursor = get_tls_callchain_cursor();
> + if (thread__resolve_callchain(al.thread, cursor, evsel,
> + sample, NULL, NULL,
> + scripting_max_stack)) {
thread__resolve_callchain()
calls __thread__resolve_callchain()
calls thread__resolve_callchain_sample():
for (i = first_call, nr_entries = 0;
i < chain_nr && nr_entries < max_stack; i++) {
...
ip = chain->ips[j];
if (ip < PERF_CONTEXT_MAX) <-- IP=ff..ff is greater than PERF_CONTEXT_MAX
++nr_entries;
...
err = add_callchain_ip(thread, cursor, parent,
root_al, &cpumode, ip,
false, NULL, NULL, 0, symbols);
if (err)
return (err < 0) ? err : 0;
calls add_callchain_ip:
if (ip >= PERF_CONTEXT_MAX) {
switch (ip) {
case PERF_CONTEXT_HV:
*cpumode = PERF_RECORD_MISC_HYPERVISOR;
break;
case PERF_CONTEXT_KERNEL:
*cpumode = PERF_RECORD_MISC_KERNEL;
break;
case PERF_CONTEXT_USER:
case PERF_CONTEXT_USER_DEFERRED:
*cpumode = PERF_RECORD_MISC_USER;
break;
default:
pr_debug("invalid callchain context: " <-- IP=ff..ff reaches default case
"%"PRId64"\n", (s64) ip);
/*
* It seems the callchain is corrupted.
* Discard all.
*/
callchain_cursor_reset(cursor);
err = 1;
goto out;
}
> + pr_info("cannot resolve deferred callchains\n");
> + cursor = NULL;
> + }
> + }
> +
> + fputc(cursor ? '\n' : ' ', fp);
> + sample__fprintf_sym(sample, &al, 0, output[type].print_ip_opts,
> + cursor, symbol_conf.bt_stop_list, fp);
> + }
Thanks and regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
+49-7031-16-1128 Office
jremus@de.ibm.com
IBM
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* Re: [PATCH v6 5/6] perf tools: Merge deferred user callchains
From: Jens Remus @ 2025-12-12 11:48 UTC (permalink / raw)
To: Namhyung Kim, Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <86593213-8a48-412a-a601-7992cc1660c6@linux.ibm.com>
Hello Namhyung,
sorry for the fuss!
On 12/12/2025 12:16 PM, Jens Remus wrote:
> On 11/21/2025 12:48 AM, Namhyung Kim wrote:
>> Save samples with deferred callchains in a separate list and deliver
>> them after merging the user callchains. If users don't want to merge
>> they can set tool->merge_deferred_callchains to false to prevent the
>> behavior.
>
>> diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c
>
>> +int sample__merge_deferred_callchain(struct perf_sample *sample_orig,
>> + struct perf_sample *sample_callchain)
>> +{
>> + u64 nr_orig = sample_orig->callchain->nr - 1;
>> + u64 nr_deferred = sample_callchain->callchain->nr;
>> + struct ip_callchain *callchain;
>> +
>> + if (sample_orig->callchain->nr < 2) {
>> + sample_orig->deferred_callchain = false;
>> + return -EINVAL;
>> + }
>> +
>> + callchain = calloc(1 + nr_orig + nr_deferred, sizeof(u64));
>> + if (callchain == NULL) {
>> + sample_orig->deferred_callchain = false;
>> + return -ENOMEM;
>> + }
>> +
>> + callchain->nr = nr_orig + nr_deferred;
>> + /* copy original including PERF_CONTEXT_USER_DEFERRED (but the cookie) */
>> + memcpy(callchain->ips, sample_orig->callchain->ips, nr_orig * sizeof(u64));
>> + /* copy deferred user callchains */
>> + memcpy(&callchain->ips[nr_orig], sample_callchain->callchain->ips,
>> + nr_deferred * sizeof(u64));
>> +
>> + sample_orig->callchain = callchain;
>
> Hope you don't mind my naive question, as I don't have a clue about perf:
>
> Doesn't the sample_orig->callchain storage need to be free'd prior to
> assigning the newly allocated one? Or is that just part of a large
> block that got allocated in one piece? How is then the one allocated
> here ever free'd?
Never mind, I found that it is getting free'd in
evlist__deliver_deferred_callchain().
>
>> + return 0;
>> +}
> Thanks and regards,
> Jens
Regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
+49-7031-16-1128 Office
jremus@de.ibm.com
IBM
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* Re: [PATCH v6 5/6] perf tools: Merge deferred user callchains
From: Jens Remus @ 2025-12-12 11:16 UTC (permalink / raw)
To: Namhyung Kim, Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120234804.156340-6-namhyung@kernel.org>
Hello Namhyung!
On 11/21/2025 12:48 AM, Namhyung Kim wrote:
> Save samples with deferred callchains in a separate list and deliver
> them after merging the user callchains. If users don't want to merge
> they can set tool->merge_deferred_callchains to false to prevent the
> behavior.
> diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c
> +int sample__merge_deferred_callchain(struct perf_sample *sample_orig,
> + struct perf_sample *sample_callchain)
> +{
> + u64 nr_orig = sample_orig->callchain->nr - 1;
> + u64 nr_deferred = sample_callchain->callchain->nr;
> + struct ip_callchain *callchain;
> +
> + if (sample_orig->callchain->nr < 2) {
> + sample_orig->deferred_callchain = false;
> + return -EINVAL;
> + }
> +
> + callchain = calloc(1 + nr_orig + nr_deferred, sizeof(u64));
> + if (callchain == NULL) {
> + sample_orig->deferred_callchain = false;
> + return -ENOMEM;
> + }
> +
> + callchain->nr = nr_orig + nr_deferred;
> + /* copy original including PERF_CONTEXT_USER_DEFERRED (but the cookie) */
> + memcpy(callchain->ips, sample_orig->callchain->ips, nr_orig * sizeof(u64));
> + /* copy deferred user callchains */
> + memcpy(&callchain->ips[nr_orig], sample_callchain->callchain->ips,
> + nr_deferred * sizeof(u64));
> +
> + sample_orig->callchain = callchain;
Hope you don't mind my naive question, as I don't have a clue about perf:
Doesn't the sample_orig->callchain storage need to be free'd prior to
assigning the newly allocated one? Or is that just part of a large
block that got allocated in one piece? How is then the one allocated
here ever free'd?
> + return 0;
> +}
Thanks and regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
+49-7031-16-1128 Office
jremus@de.ibm.com
IBM
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* [PATCH v2 1/2] x86/tlb/trace: Export the TLB_REMOTE_WRONG_CPU enum
From: Tal Zussman @ 2025-12-12 9:08 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers, Ingo Molnar,
Rik van Riel, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
x86, Will Deacon, Aneesh Kumar K.V, Nick Piggin, Peter Zijlstra,
David Hildenbrand
Cc: linux-kernel, linux-trace-kernel, linux-mm, linux-arch,
Tal Zussman, David Hildenbrand
In-Reply-To: <20251212-tlb-trace-fix-v2-0-d322e0ad9b69@columbia.edu>
When the TLB_REMOTE_WRONG_CPU enum was introduced for the tlb_flush
tracepoint, the enum was not exported to userspace. Add it to the
appropriate macro definition to enable parsing by userspace tools, as
per [0].
[0] Link: https://lore.kernel.org/all/20150403013802.220157513@goodmis.org
Fixes: 2815a56e4b72 ("x86/mm/tlb: Add tracepoint for TLB flush IPI to stale CPU")
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
include/trace/events/tlb.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/include/trace/events/tlb.h b/include/trace/events/tlb.h
index b4d8e7dc38f8..725a75720a23 100644
--- a/include/trace/events/tlb.h
+++ b/include/trace/events/tlb.h
@@ -13,7 +13,8 @@
EM( TLB_REMOTE_SHOOTDOWN, "remote shootdown" ) \
EM( TLB_LOCAL_SHOOTDOWN, "local shootdown" ) \
EM( TLB_LOCAL_MM_SHOOTDOWN, "local mm shootdown" ) \
- EMe( TLB_REMOTE_SEND_IPI, "remote ipi send" )
+ EM( TLB_REMOTE_SEND_IPI, "remote ipi send" ) \
+ EMe( TLB_REMOTE_WRONG_CPU, "remote wrong CPU" )
/*
* First define the enums in TLB_FLUSH_REASON to be exported to userspace
--
2.39.5
^ permalink raw reply related
* [PATCH v2 2/2] mm: Remove tlb_flush_reason::NR_TLB_FLUSH_REASONS
From: Tal Zussman @ 2025-12-12 9:08 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers, Ingo Molnar,
Rik van Riel, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
x86, Will Deacon, Aneesh Kumar K.V, Nick Piggin, Peter Zijlstra,
David Hildenbrand
Cc: linux-kernel, linux-trace-kernel, linux-mm, linux-arch,
Tal Zussman, David Hildenbrand
In-Reply-To: <20251212-tlb-trace-fix-v2-0-d322e0ad9b69@columbia.edu>
This has been unused since it was added 11 years ago in commit
d17d8f9dedb9 ("x86/mm: Add tracepoints for TLB flushes").
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
include/linux/mm_types.h | 1 -
1 file changed, 1 deletion(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 9f6de068295d..42af2292951d 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1631,7 +1631,6 @@ enum tlb_flush_reason {
TLB_LOCAL_MM_SHOOTDOWN,
TLB_REMOTE_SEND_IPI,
TLB_REMOTE_WRONG_CPU,
- NR_TLB_FLUSH_REASONS,
};
/**
--
2.39.5
^ permalink raw reply related
* [PATCH v2 0/2] x86, mm: minor tlb_flush tracepoint adjustments
From: Tal Zussman @ 2025-12-12 9:08 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers, Ingo Molnar,
Rik van Riel, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
x86, Will Deacon, Aneesh Kumar K.V, Nick Piggin, Peter Zijlstra,
David Hildenbrand
Cc: linux-kernel, linux-trace-kernel, linux-mm, linux-arch,
Tal Zussman, David Hildenbrand
One minor fix and one minor cleanup related to the tlb_flush tracepoint.
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
Changes in v2:
- Propagated tags (thanks David, Rik, Steven!)
- Added MMU Gather maintainers to recipients
- Rebased on master
- Link to v1: https://lore.kernel.org/r/20250528-tlb-trace-fix-v1-0-2e94c58f450d@columbia.edu
---
Tal Zussman (2):
x86/tlb/trace: Export the TLB_REMOTE_WRONG_CPU enum
mm: Remove tlb_flush_reason::NR_TLB_FLUSH_REASONS
include/linux/mm_types.h | 1 -
include/trace/events/tlb.h | 3 ++-
2 files changed, 2 insertions(+), 2 deletions(-)
---
base-commit: d358e5254674b70f34c847715ca509e46eb81e6f
change-id: 20250528-tlb-trace-fix-eb9b6fc7bb26
Best regards,
--
Tal Zussman <tz2294@columbia.edu>
^ permalink raw reply
* Re: [RFC PATCH v3 17/17] s390/unwind_user/fp: Enable back chain unwinding of user space
From: Jens Remus @ 2025-12-12 9:21 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-s390, bpf, x86,
Steven Rostedt
Cc: Heiko Carstens, Vasily Gorbik, Ilya Leoshkevich, Josh Poimboeuf,
Masami Hiramatsu, Mathieu Desnoyers, Peter Zijlstra, Ingo Molnar,
Jiri Olsa, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Andrii Nakryiko, Indu Bhagat, Jose E. Marchesi,
Beau Belgrave, Linus Torvalds, Andrew Morton, Florian Weimer,
Kees Cook, Carlos O'Donell, Sam James, Dylan Hatch
In-Reply-To: <20251208171559.2029709-18-jremus@linux.ibm.com>
On 12/8/2025 6:15 PM, Jens Remus wrote:
...
> Leverage the unwind user fp infrastructure to enable unwinding of user
> space using back chain. Enable HAVE_UNWIND_USER_FP and provide a s390-
> specific implementation of unwind_user_fp_get_frame(), which uses the
> back chain.
> diff --git a/arch/s390/include/asm/unwind_user.h b/arch/s390/include/asm/unwind_user.h
> +static inline int unwind_user_fp_get_frame(struct unwind_user_state *state,
> + struct unwind_user_frame *frame)
> +{
> + struct stack_frame_user __user *sf;
> + unsigned long __user *ra_addr;
> + unsigned long sp;
> +
> + sf = (void __user *)state->sp;
> +
> + /*
> + * In topmost frame check whether IP in early prologue, RA and SP
> + * registers saved, and no new stack frame allocated.
> + */
> + if (state->topmost) {
> + unsigned long ra, ra_reg;
> +
> + ra_addr = (unsigned long __user *)&sf->gprs[8];
> + if (__get_user(ra, ra_addr))
> + return -EINVAL;
> + if (__get_user(sp, (unsigned long __user *)&sf->gprs[9]))
> + return -EINVAL;
> + if (unwind_user_get_ra_reg(&ra_reg))
> + return -EINVAL;
> + if (ra == ra_reg && sp == state->sp)
> + goto done;
> + }
I realized that this additional heuristic is flawed:
The topmost function may be past prologue, have allocated a new stack
frame, and called a function. The callee may have saved its RA and SP
registers in the current stack frame, so that after the return from
function call, the heuristic would erroneously assume that the topmost
function is in early prologue and use the callee's RA and SP.
Instead of erroneously skipping the caller it might erroneously insert
a callee as caller. I'll remove it again in the next version.
> +
> + if (__get_user(sp, (unsigned long __user *)&sf->back_chain))
> + return -EINVAL;
> + if (!sp && ip_within_vdso(state->ip)) {
> + /*
> + * Assume non-standard vDSO user wrapper stack frame.
> + * See vDSO user wrapper code for details.
> + */
> + struct stack_frame_vdso_wrapper *sf_vdso = (void __user *)sf;
> +
> + ra_addr = (unsigned long __user *)&sf_vdso->return_address;
> + sf = (void __user *)((unsigned long)sf + STACK_FRAME_VDSO_OVERHEAD);
> + if (__get_user(sp, (unsigned long __user *)&sf->back_chain))
> + return -EINVAL;
> + } else if (!sp) {
> + /*
> + * Assume outermost frame reached. unwind_user_next_common()
> + * disregards all other fields in outermost frame.
> + */
> + frame->outermost = false;
frame->outermost = true;
> + return 0;
> + } else {
> + /*
> + * Assume IP past prologue and new stack frame allocated.
> + * Follow back chain, which then equals the SP at entry.
> + * Skips caller if wrong in topmost frame.
> + */
> + sf = (void __user *)sp;
> + ra_addr = (unsigned long __user *)&sf->gprs[8];
> + }
> +
> +done:
> + frame->cfa_off = sp - state->sp + 160;
> + frame->sp_off = -160;
> + frame->fp.loc = UNWIND_USER_LOC_UNKNOWN; /* Cannot unwind FP. */
> + frame->use_fp = false;
> + frame->ra.loc = UNWIND_USER_LOC_STACK;
> + frame->ra.offset = (unsigned long)ra_addr - (state->sp + frame->cfa_off);
> + frame->outermost = false;
> +
> + return 0;
> +}
> +#define unwind_user_fp_get_frame unwind_user_fp_get_frame
Regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
+49-7031-16-1128 Office
jremus@de.ibm.com
IBM
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* Re: [RFC PATCH v3 14/17] s390/unwind_user/sframe: Enable HAVE_UNWIND_USER_SFRAME
From: Jens Remus @ 2025-12-12 8:13 UTC (permalink / raw)
To: Heiko Carstens
Cc: linux-kernel, linux-trace-kernel, linux-s390, bpf, x86,
Steven Rostedt, Vasily Gorbik, Ilya Leoshkevich, Josh Poimboeuf,
Masami Hiramatsu, Mathieu Desnoyers, Peter Zijlstra, Ingo Molnar,
Jiri Olsa, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Andrii Nakryiko, Indu Bhagat, Jose E. Marchesi,
Beau Belgrave, Linus Torvalds, Andrew Morton, Florian Weimer,
Kees Cook, Carlos O'Donell, Sam James, Dylan Hatch
In-Reply-To: <20251210151012.40732B79-hca@linux.ibm.com>
Hello Heiko,
thank you for the feedback!
On 12/10/2025 4:10 PM, Heiko Carstens wrote:
> On Mon, Dec 08, 2025 at 06:15:56PM +0100, Jens Remus wrote:
>> +static inline int __s390_get_dwarf_fpr(unsigned long *val, int regnum)
>> +{
>> + switch (regnum) {
>> + case 16:
>> + fpu_std(0, (freg_t *)val);
>> + break;
>> + case 17:
>> + fpu_std(2, (freg_t *)val);
>> + break;
>> + case 18:
>> + fpu_std(4, (freg_t *)val);
>> + break;
>> + case 19:
>> + fpu_std(6, (freg_t *)val);
>> + break;
>> + case 20:
>> + fpu_std(1, (freg_t *)val);
>> + break;
>
> IIRC, I mentioned this already last time. But it is not correct to access user
> space floating point register contents like this. Due to in-kernel fpu/vector
> register usage the user space register contents may have been saved away to
> the per-thread vxrs save area, and registers may have been used for in-kernel
> usage instead.
> Read: the above code could access lazy register contents of in-kernel usage.
>
> Change the above to something like:
>
> struct fpu *fpu = ¤t->thread.ufpu;
>
> save_user_fpu_regs();
> switch (regnum) {
> case 16: return fpu->vxrs[0].high;
> case 17: return fpu->vxrs[2].high;
> case 18: return fpu->vxrs[4].high;
> case 19: return fpu->vxrs[6].high;
> case 20: return fpu->vxrs[1].high;
> ...
>
> save_user_fpu_regs() will write all user space fpu/vector register contents to
> the per-thread save area (if not already saved), and then it is possible to
> read contents from there.
Thanks! I have changed the code accordingly. Works fine.
> I'll see if I can provide something better for this use case, since this code
> needs to access only the first 16 registers; so no need to write contents of
> all registers to the save area.
Ok.
Regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
+49-7031-16-1128 Office
jremus@de.ibm.com
IBM
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* Re: [PATCH v3 2/2] tracing: Update funcgraph-retval documentation
From: Donglin Peng @ 2025-12-12 2:14 UTC (permalink / raw)
To: rostedt
Cc: mhiramat, linux-trace-kernel, bpf, linux-kernel, pengdonglin,
Xiaoqin Zhang
In-Reply-To: <20251209121349.525641-3-dolinux.peng@gmail.com>
On Tue, Dec 9, 2025 at 8:14 PM Donglin Peng <dolinux.peng@gmail.com> wrote:
>
> From: pengdonglin <pengdonglin@xiaomi.com>
>
> The existing documentation for funcgraph-retval is outdated and partially
> incorrect, as it describes limitations that have now been resolved.
>
> Recent changes (e.g., using BTF to obtain function return types) have
> addressed key issues:
> 1. Return values are now printed only for non-void functions.
> 2. Values are trimmed to the correct width of the return type, avoiding
> garbage data from high bits.
>
> Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: Xiaoqin Zhang <zhangxiaoqin@xiaomi.com>
> Signed-off-by: pengdonglin <pengdonglin@xiaomi.com>
> ---
> Documentation/trace/ftrace.rst | 78 ++++++++++++++++++++--------------
> 1 file changed, 45 insertions(+), 33 deletions(-)
>
> diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst
> index d1f313a5f4ad..03c8c433c803 100644
> --- a/Documentation/trace/ftrace.rst
> +++ b/Documentation/trace/ftrace.rst
> @@ -1454,6 +1454,10 @@ Options for function_graph tracer:
> printed in hexadecimal format. By default, this option
> is off.
>
> + funcgraph-retaddr
> + When set, the return address will always be printed.
> + By default, this option is off.
> +
> sleep-time
> When running function graph tracer, to include
> the time a task schedules out in its function.
> @@ -2800,7 +2804,7 @@ It is default disabled.
> 0) 2.861 us | } /* putname() */
>
> The return value of each traced function can be displayed after
> -an equal sign "=". When encountering system call failures, it
> +an equal sign "ret =". When encountering system call failures, it
> can be very helpful to quickly locate the function that first
> returns an error code.
>
> @@ -2810,16 +2814,16 @@ returns an error code.
> Example with funcgraph-retval::
>
> 1) | cgroup_migrate() {
> - 1) 0.651 us | cgroup_migrate_add_task(); /* = 0xffff93fcfd346c00 */
> + 1) 0.651 us | cgroup_migrate_add_task(); /* ret=0xffff93fcfd346c00 */
> 1) | cgroup_migrate_execute() {
> 1) | cpu_cgroup_can_attach() {
> 1) | cgroup_taskset_first() {
> - 1) 0.732 us | cgroup_taskset_next(); /* = 0xffff93fc8fb20000 */
> - 1) 1.232 us | } /* cgroup_taskset_first = 0xffff93fc8fb20000 */
> - 1) 0.380 us | sched_rt_can_attach(); /* = 0x0 */
> - 1) 2.335 us | } /* cpu_cgroup_can_attach = -22 */
> - 1) 4.369 us | } /* cgroup_migrate_execute = -22 */
> - 1) 7.143 us | } /* cgroup_migrate = -22 */
> + 1) 0.732 us | cgroup_taskset_next(); /* ret=0xffff93fc8fb20000 */
> + 1) 1.232 us | } /* cgroup_taskset_first ret=0xffff93fc8fb20000 */
> + 1) 0.380 us | sched_rt_can_attach(); /* ret=0x0 */
> + 1) 2.335 us | } /* cpu_cgroup_can_attach ret=-22 */
> + 1) 4.369 us | } /* cgroup_migrate_execute ret=-22 */
> + 1) 7.143 us | } /* cgroup_migrate ret=-22 */
>
> The above example shows that the function cpu_cgroup_can_attach
> returned the error code -22 firstly, then we can read the code
> @@ -2836,37 +2840,41 @@ printed in hexadecimal format.
> Example with funcgraph-retval-hex::
>
> 1) | cgroup_migrate() {
> - 1) 0.651 us | cgroup_migrate_add_task(); /* = 0xffff93fcfd346c00 */
> + 1) 0.651 us | cgroup_migrate_add_task(); /* ret=0xffff93fcfd346c00 */
> 1) | cgroup_migrate_execute() {
> 1) | cpu_cgroup_can_attach() {
> 1) | cgroup_taskset_first() {
> - 1) 0.732 us | cgroup_taskset_next(); /* = 0xffff93fc8fb20000 */
> - 1) 1.232 us | } /* cgroup_taskset_first = 0xffff93fc8fb20000 */
> - 1) 0.380 us | sched_rt_can_attach(); /* = 0x0 */
> - 1) 2.335 us | } /* cpu_cgroup_can_attach = 0xffffffea */
> - 1) 4.369 us | } /* cgroup_migrate_execute = 0xffffffea */
> + 1) 0.732 us | cgroup_taskset_next(); /* ret=0xffff93fc8fb20000 */
> + 1) 1.232 us | } /* cgroup_taskset_first ret=0xffff93fc8fb20000 */
> + 1) 0.380 us | sched_rt_can_attach(); /* ret=0x0 */
> + 1) 2.335 us | } /* cpu_cgroup_can_attach ret=0xffffffea */
> + 1) 4.369 us | } /* cgroup_migrate_execute ret=0xffffffea */
> 1) 7.143 us | } /* cgroup_migrate = 0xffffffea */
My bad for forgetting to update the above line and will fix it in the
next version.
>
> -At present, there are some limitations when using the funcgraph-retval
> -option, and these limitations will be eliminated in the future:
> +Note that there are some limitations when using the funcgraph-retval
> +option:
> +
> +- If CONFIG_DEBUG_INFO_BTF is disabled (n), a return value is printed even for
> + functions with a void return type. When CONFIG_DEBUG_INFO_BTF is enabled (y),
> + the return value is printed only for non-void functions.
>
> -- Even if the function return type is void, a return value will still
> - be printed, and you can just ignore it.
> +- If a return value occupies multiple registers, only the value in the first
> + register is recorded and printed. For example, on the x86 architecture, a
> + 64-bit return value is stored across eax (lower 32 bits) and edx (upper 32 bits),
> + but only the contents of eax are captured. If CONFIG_DEBUG_INFO_BTF is enabled,
> + the suffix "(trunc)" is appended to the printed value to indicate that the
> + output may be truncated because high-order register contents are omitted.
>
> -- Even if return values are stored in multiple registers, only the
> - value contained in the first register will be recorded and printed.
> - To illustrate, in the x86 architecture, eax and edx are used to store
> - a 64-bit return value, with the lower 32 bits saved in eax and the
> - upper 32 bits saved in edx. However, only the value stored in eax
> - will be recorded and printed.
> +- Under certain procedure-call standards (e.g., arm64's AAPCS64), when the return
> + type is smaller than a general-purpose register (GPR), the caller is responsible
> + for narrowing the value; the upper bits of the register may contain undefined data.
> + For instance, when a u8 is returned in 64-bit GPR, bits [63:8] can hold arbitrary
> + values, especially when larger types are truncated (explicitly or implicitly). It
> + is therefore advisable to inspect the code in such cases. If CONFIG_DEBUG_INFO_BTF
> + is enabled (y), the return value is automatically trimmed to the width of the return
> + type.
>
> -- In certain procedure call standards, such as arm64's AAPCS64, when a
> - type is smaller than a GPR, it is the responsibility of the consumer
> - to perform the narrowing, and the upper bits may contain UNKNOWN values.
> - Therefore, it is advisable to check the code for such cases. For instance,
> - when using a u8 in a 64-bit GPR, bits [63:8] may contain arbitrary values,
> - especially when larger types are truncated, whether explicitly or implicitly.
> - Here are some specific cases to illustrate this point:
> + The following examples illustrate the behavior:
>
> **Case One**:
>
> @@ -2885,7 +2893,9 @@ option, and these limitations will be eliminated in the future:
> RET
>
> If you pass 0x123456789abcdef to this function and want to narrow it,
> - it may be recorded as 0x123456789abcdef instead of 0xef.
> + it may be recorded as 0x123456789abcdef instead of 0xef. When
> + CONFIG_DEBUG_INFO_BTF is enabled, the value will be correctly truncated
> + to 0xef based on the size constraints of the u8 type.
>
> **Case Two**:
>
> @@ -2910,7 +2920,9 @@ option, and these limitations will be eliminated in the future:
> RET
>
> When passing 0x2_0000_0000 to it, the return value may be recorded as
> - 0x2_0000_0000 instead of 0.
> + 0x2_0000_0000 instead of 0. When CONFIG_DEBUG_INFO_BTF is enabled, the
> + value will be correctly truncated to 0 based on the size constraints of
> + the int type.
>
> You can put some comments on specific functions by using
> trace_printk() For example, if you want to put a comment inside
> --
> 2.34.1
>
^ permalink raw reply
* [PATCH AUTOSEL 6.18-6.6] MIPS: ftrace: Fix memory corruption when kernel is located beyond 32 bits
From: Sasha Levin @ 2025-12-12 2:09 UTC (permalink / raw)
To: patches, stable
Cc: Gregory CLEMENT, Thomas Bogendoerfer, Sasha Levin, rostedt,
mhiramat, linux-kernel, linux-trace-kernel
In-Reply-To: <20251212020903.4153935-1-sashal@kernel.org>
From: Gregory CLEMENT <gregory.clement@bootlin.com>
[ Upstream commit 36dac9a3dda1f2bae343191bc16b910c603cac25 ]
Since commit e424054000878 ("MIPS: Tracing: Reduce the overhead of
dynamic Function Tracer"), the macro UASM_i_LA_mostly has been used,
and this macro can generate more than 2 instructions. At the same
time, the code in ftrace assumes that no more than 2 instructions can
be generated, which is why it stores them in an int[2] array. However,
as previously noted, the macro UASM_i_LA_mostly (and now UASM_i_LA)
causes a buffer overflow when _mcount is beyond 32 bits. This leads to
corruption of the variables located in the __read_mostly section.
This corruption was observed because the variable
__cpu_primary_thread_mask was corrupted, causing a hang very early
during boot.
This fix prevents the corruption by avoiding the generation of
instructions if they could exceed 2 instructions in
length. Fortunately, insn_la_mcount is only used if the instrumented
code is located outside the kernel code section, so dynamic ftrace can
still be used, albeit in a more limited scope. This is still
preferable to corrupting memory and/or crashing the kernel.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
LLM Generated explanations, may be completely bogus:
## COMPREHENSIVE ANALYSIS
### 1. COMMIT MESSAGE ANALYSIS
- **Subject**: Contains "Fix" and "memory corruption" - strong
indicators of a serious bug fix
- **Body**: Clearly explains the buffer overflow mechanism and observed
real-world impact
- **References**: Cites commit e424054000878 as the source of the bug
(introduced in 2010, kernel 2.6.35)
- **No explicit Cc: stable tag**, but the nature of the fix makes it
appropriate for stable
### 2. TECHNICAL ROOT CAUSE
The bug is in `ftrace_dyn_arch_init_insns()`:
1. **Buffer size**: `insn_la_mcount[2]` is declared as a 2-element array
(8 bytes)
2. **UASM_i_LA behavior**: When address is NOT in 32-bit compat space,
it can generate up to 5-6 instructions:
- `lui` (always)
- `daddiu` (conditional)
- `dsll` (conditional)
- `daddiu` (conditional)
- `dsll` (conditional)
- final `daddiu`/`addiu`
3. **Overflow**: Writing more than 2 instructions overwrites adjacent
`__read_mostly` variables
4. **Observed impact**: Corruption of `__cpu_primary_thread_mask`
causing boot hang
### 3. FIX MECHANISM
The fix adds two defensive checks:
1. **In `ftrace_dyn_arch_init_insns()`**: Only generate instructions if
`uasm_in_compat_space_p(MCOUNT_ADDR)` - otherwise warn and skip
2. **In `ftrace_make_call()`**: Return `-EFAULT` if `insn_la_mcount`
would be needed but wasn't generated
This gracefully degrades functionality rather than corrupting memory.
### 4. STABLE KERNEL CRITERIA ASSESSMENT
| Criterion | Assessment |
|-----------|------------|
| Obviously correct | ✅ Simple defensive check before buffer write |
| Fixes real bug | ✅ Memory corruption causing boot hang |
| Important issue | ✅ System crash/hang - very severe |
| Small and contained | ✅ Single file, ~30 lines changed |
| No new features | ✅ Actually reduces functionality in edge cases |
| No new APIs | ✅ Purely internal change |
### 5. DEPENDENCY CHECK
- **`uasm_in_compat_space_p()`**: Exists since kernel 2.6.x (commit
e30ec4525d473)
- **Bug source commit**: e424054000878 from 2010 (2.6.35)
- **Dependencies**: None - fix is self-contained
### 6. RISK vs BENEFIT
**Risk**: Very LOW
- Defensive check - prevents execution rather than changing behavior
- Worst case: ftrace doesn't work for code outside kernel text on 64-bit
MIPS with addresses beyond 32 bits
- Architecture-specific (MIPS only)
**Benefit**: HIGH
- Prevents memory corruption that causes boot hangs
- Bug has existed since 2010 - affects all stable kernels
- Observable real-world failure
### 7. USER IMPACT
- **Affected users**: MIPS 64-bit users with kernel loaded at addresses
beyond 32 bits
- **Severity**: Critical - boot hang due to memory corruption
- **Reproducibility**: Deterministic when conditions are met (not a
race)
### 8. CONCERNS
- **No explicit Cc: stable@vger.kernel.org tag**: However, the commit
clearly fixes a serious memory corruption bug
- **Partial functionality loss**: Some ftrace capabilities reduced for
64-bit addresses, but this is far preferable to memory corruption
### CONCLUSION
This commit is an excellent stable backport candidate:
1. **Fixes a serious bug**: Memory corruption causing system boot hangs
2. **Minimal risk**: Defensive check that gracefully degrades
functionality
3. **Self-contained**: Single file change with no dependencies on new
code
4. **Long-standing bug**: Affects all kernels since 2.6.35
5. **MIPS maintainer signed-off**: Thomas Bogendoerfer approved the fix
The fix is small, surgical, and meets all stable kernel criteria. The
trade-off (reduced ftrace functionality in edge cases vs memory
corruption) strongly favors the fix.
**YES**
arch/mips/kernel/ftrace.c | 25 +++++++++++++++++++++----
1 file changed, 21 insertions(+), 4 deletions(-)
diff --git a/arch/mips/kernel/ftrace.c b/arch/mips/kernel/ftrace.c
index f39e85fd58fa9..b15615b285690 100644
--- a/arch/mips/kernel/ftrace.c
+++ b/arch/mips/kernel/ftrace.c
@@ -54,10 +54,20 @@ static inline void ftrace_dyn_arch_init_insns(void)
u32 *buf;
unsigned int v1;
- /* la v1, _mcount */
- v1 = 3;
- buf = (u32 *)&insn_la_mcount[0];
- UASM_i_LA(&buf, v1, MCOUNT_ADDR);
+ /* If we are not in compat space, the number of generated
+ * instructions will exceed the maximum expected limit of 2.
+ * To prevent buffer overflow, we avoid generating them.
+ * insn_la_mcount will not be used later in ftrace_make_call.
+ */
+ if (uasm_in_compat_space_p(MCOUNT_ADDR)) {
+ /* la v1, _mcount */
+ v1 = 3;
+ buf = (u32 *)&insn_la_mcount[0];
+ UASM_i_LA(&buf, v1, MCOUNT_ADDR);
+ } else {
+ pr_warn("ftrace: mcount address beyond 32 bits is not supported (%lX)\n",
+ MCOUNT_ADDR);
+ }
/* jal (ftrace_caller + 8), jump over the first two instruction */
buf = (u32 *)&insn_jal_ftrace_caller;
@@ -189,6 +199,13 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
unsigned int new;
unsigned long ip = rec->ip;
+ /* When the code to patch does not belong to the kernel code
+ * space, we must use insn_la_mcount. However, if MCOUNT_ADDR
+ * is not in compat space, insn_la_mcount is not usable.
+ */
+ if (!core_kernel_text(ip) && !uasm_in_compat_space_p(MCOUNT_ADDR))
+ return -EFAULT;
+
new = core_kernel_text(ip) ? insn_jal_ftrace_caller : insn_la_mcount[0];
#ifdef CONFIG_64BIT
--
2.51.0
^ permalink raw reply related
* Re: [PATCH] tracing: Fix error handling in event_hist_trigger_parse
From: Steven Rostedt @ 2025-12-12 1:45 UTC (permalink / raw)
To: Miaoqian Lin
Cc: Masami Hiramatsu, Mathieu Desnoyers, Tom Zanussi, linux-kernel,
linux-trace-kernel, stable
In-Reply-To: <20251211100058.2381268-1-linmq006@gmail.com>
On Thu, 11 Dec 2025 14:00:58 +0400
Miaoqian Lin <linmq006@gmail.com> wrote:
> @@ -6902,7 +6902,7 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops,
>
> remove_hist_vars(hist_data);
>
> - kfree(trigger_data);
> + trigger_data_free(trigger_data);
>
> destroy_hist_data(hist_data);
> goto out;
The above code has this:
out_free:
event_trigger_reset_filter(cmd_ops, trigger_data);
remove_hist_vars(hist_data);
kfree(trigger_data);
destroy_hist_data(hist_data);
goto out;
Where we have;
void event_trigger_reset_filter(struct event_command *cmd_ops,
struct event_trigger_data *trigger_data)
{
if (cmd_ops->set_filter)
cmd_ops->set_filter(NULL, trigger_data, NULL);
}
And trigger_data_free() starts with:
void trigger_data_free(struct event_trigger_data *data)
{
if (data->cmd_ops->set_filter)
data->cmd_ops->set_filter(NULL, data, NULL);
thus it looks like the current code is an open coded version of
trigger_data_free() without synchronization (as it isn't needed here).
Thus, I believe this is more of a clean up and not a fix (something to
go into the next merge window and not the current -rc release).
And the code can be changed to also remove the event_trigger_reset_filter()
call.
-- Steve
^ permalink raw reply
* Re: [PATCH v3 08/13] sched: Export hidden tracepoints to modules
From: Phil Auld @ 2025-12-11 22:36 UTC (permalink / raw)
To: Gabriele Monaco
Cc: linux-kernel, Steven Rostedt, Nam Cao, Ingo Molnar,
Peter Zijlstra, Tomas Glozar, Juri Lelli, Clark Williams,
John Kacur, linux-trace-kernel
In-Reply-To: <20251205133552.GA400972@pauld.westford.csb>
Hi Peter,
On Fri, Dec 05, 2025 at 08:35:52AM -0500 Phil Auld wrote:
> On Fri, Dec 05, 2025 at 02:16:16PM +0100 Gabriele Monaco wrote:
> > The tracepoints sched_entry, sched_exit and sched_set_need_resched
> > are not exported to tracefs as trace events, this allows only kernel
> > code to access them. Helper modules like [1] can be used to still have
> > the tracepoints available to ftrace for debugging purposes, but they do
> > rely on the tracepoints being exported.
> >
> > Export the 3 not exported tracepoints.
> > Note that sched_set_state is already exported as the macro is called
> > from modules.
> >
> > [1] - https://github.com/qais-yousef/sched_tp.git
> >
> > Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
>
> This makes sense.
>
> Reviewed-by: Phil Auld <pauld@redhat.com>
>
Any chance we can get this one in even if the rest of the series
needs more time?
Without this, out of tree modules fail to link.
I don't know if it's worth adding:
Fixes: adcc3bfa8806 ("sched: Adapt sched tracepoints for RV task model")
Cheers,
Phil
>
> Cheers,
> Phil
>
> > ---
> > kernel/sched/core.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 41ba0be16911..5516778c19eb 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -119,6 +119,9 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_util_est_cfs_tp);
> > EXPORT_TRACEPOINT_SYMBOL_GPL(sched_util_est_se_tp);
> > EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp);
> > EXPORT_TRACEPOINT_SYMBOL_GPL(sched_compute_energy_tp);
> > +EXPORT_TRACEPOINT_SYMBOL_GPL(sched_entry_tp);
> > +EXPORT_TRACEPOINT_SYMBOL_GPL(sched_exit_tp);
> > +EXPORT_TRACEPOINT_SYMBOL_GPL(sched_set_need_resched_tp);
> >
> > DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
> > DEFINE_PER_CPU(struct rnd_state, sched_rnd_state);
> > --
> > 2.52.0
> >
> >
>
> --
>
--
^ permalink raw reply
* [RFC 5/5] ext4: mark group extend fast-commit ineligible
From: Li Chen @ 2025-12-11 11:51 UTC (permalink / raw)
To: Theodore Ts'o, Andreas Dilger, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, linux-ext4, linux-kernel,
linux-trace-kernel
Cc: Li Chen
In-Reply-To: <20251211115146.897420-1-me@linux.beauty>
Fast commits only log operations that have dedicated replay support.
EXT4_IOC_GROUP_EXTEND grows the filesystem to the end of the last
block group and updates the same on-disk metadata without going
through the fast commit tracking paths.
In practice these operations are rare and usually followed by further
updates, but mixing them into a fast commit makes the overall
semantics harder to reason about and risks replay gaps if new call
sites appear.
Teach ext4 to mark the filesystem fast-commit ineligible when
EXT4_IOC_GROUP_EXTEND grows the filesystem.
This forces those transactions to fall back to a full commit,
ensuring that the group extension changes are captured by the normal
journal rather than partially encoded in fast commit TLVs.
This change should not affect common workloads but makes online
resize via GROUP_EXTEND safer and easier to reason about under fast
commit.
Testing:
1. prepare:
dd if=/dev/zero of=/root/fc_resize.img bs=1M count=0 seek=256
mkfs.ext4 -O fast_commit -F /root/fc_resize.img
mkdir -p /mnt/fc_resize && mount -t ext4 -o loop /root/fc_resize.img /mnt/fc_resize
2. Extended the filesystem to the end of the last block group using a
helper that calls EXT4_IOC_GROUP_EXTEND on the mounted filesystem
and checked fc_info:
./group_extend_helper /mnt/fc_resize
cat /proc/fs/ext4/loop0/fc_info
shows the "Resize" ineligible reason increased.
3. Fsynced a file on the resized filesystem and confirmed that the fast
commit ineligible counter incremented for the resize transaction:
touch /mnt/fc_resize/file
/root/fsync_file /mnt/fc_resize/file
sync
cat /proc/fs/ext4/loop0/fc_info
Signed-off-by: Li Chen <me@linux.beauty>
---
fs/ext4/ioctl.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/fs/ext4/ioctl.c b/fs/ext4/ioctl.c
index 57b47b9843f3..ce92652f8332 100644
--- a/fs/ext4/ioctl.c
+++ b/fs/ext4/ioctl.c
@@ -1608,6 +1608,8 @@ static long __ext4_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
err = ext4_group_extend(sb, EXT4_SB(sb)->s_es, n_blocks_count);
if (EXT4_SB(sb)->s_journal) {
+ ext4_fc_mark_ineligible(sb, EXT4_FC_REASON_RESIZE,
+ NULL);
jbd2_journal_lock_updates(EXT4_SB(sb)->s_journal);
err2 = jbd2_journal_flush(EXT4_SB(sb)->s_journal, 0);
jbd2_journal_unlock_updates(EXT4_SB(sb)->s_journal);
--
2.51.0
^ permalink raw reply related
* [RFC 4/5] ext4: mark group add fast-commit ineligible
From: Li Chen @ 2025-12-11 11:51 UTC (permalink / raw)
To: Theodore Ts'o, Andreas Dilger, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, linux-ext4, linux-kernel,
linux-trace-kernel
Cc: Li Chen
In-Reply-To: <20251211115146.897420-1-me@linux.beauty>
Fast commits only log operations that have dedicated replay support.
Online resize via EXT4_IOC_GROUP_ADD updates the superblock and group
descriptor metadata without going through the fast commit tracking
paths.
In practice these operations are rare and usually followed by further
updates, but mixing them into a fast commit makes the overall
semantics harder to reason about and risks replay gaps if new call
sites appear.
Teach ext4 to mark the filesystem fast-commit ineligible when
ext4_ioctl_group_add() adds new block groups.
This forces those transactions to fall back to a full commit,
ensuring that the filesystem geometry updates are captured by the
normal journal rather than partially encoded in fast commit TLVs.
This change should not affect common workloads but makes online
resize via GROUP_ADD safer and easier to reason about under fast
commit.
Testing:
1. prepare:
dd if=/dev/zero of=/root/fc_resize.img bs=1M count=0 seek=256
mkfs.ext4 -O fast_commit -F /root/fc_resize.img
mkdir -p /mnt/fc_resize && mount -t ext4 -o loop /root/fc_resize.img /mnt/fc_resize
2. Ran a helper that issues EXT4_IOC_GROUP_ADD on the mounted
filesystem and checked the resize ineligible reason:
./group_add_helper /mnt/fc_resize
cat /proc/fs/ext4/loop0/fc_info
shows "Resize": > 0.
3. Fsynced a file on the resized filesystem and verified that the fast
commit stats report at least one ineligible commit:
touch /mnt/fc_resize/file
/root/fsync_file /mnt/fc_resize/file
sync
cat /proc/fs/ext4/loop0/fc_info
shows fc stats ineligible > 0.
Signed-off-by: Li Chen <me@linux.beauty>
---
fs/ext4/ioctl.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/fs/ext4/ioctl.c b/fs/ext4/ioctl.c
index a93a7baae990..57b47b9843f3 100644
--- a/fs/ext4/ioctl.c
+++ b/fs/ext4/ioctl.c
@@ -966,6 +966,7 @@ static long ext4_ioctl_group_add(struct file *file,
err = ext4_group_add(sb, input);
if (EXT4_SB(sb)->s_journal) {
+ ext4_fc_mark_ineligible(sb, EXT4_FC_REASON_RESIZE, NULL);
jbd2_journal_lock_updates(EXT4_SB(sb)->s_journal);
err2 = jbd2_journal_flush(EXT4_SB(sb)->s_journal, 0);
jbd2_journal_unlock_updates(EXT4_SB(sb)->s_journal);
--
2.51.0
^ permalink raw reply related
* [RFC 3/5] ext4: mark move extents fast-commit ineligible
From: Li Chen @ 2025-12-11 11:51 UTC (permalink / raw)
To: Theodore Ts'o, Andreas Dilger, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, linux-ext4, linux-kernel,
linux-trace-kernel
Cc: Li Chen
In-Reply-To: <20251211115146.897420-1-me@linux.beauty>
Fast commits only log operations that have dedicated replay support.
EXT4_IOC_MOVE_EXT swaps extents between regular files and may copy
data, rewriting the affected inodes' block mapping layout without
going through the fast commit tracking paths.
In practice these operations are rare and usually followed by further
updates, but mixing them into a fast commit makes the overall
semantics harder to reason about and risks replay gaps if new call
sites appear.
Teach ext4 to mark the filesystem fast-commit ineligible for the
journal transactions used by move_extent_per_page() when
EXT4_IOC_MOVE_EXT runs.
This forces those transactions to fall back to a full commit,
ensuring that these multi-inode extent swaps are captured by the
normal journal rather than partially encoded in fast commit TLVs.
This change should not affect common workloads but makes online
defragmentation safer and easier to reason about under fast commit.
Testing:
1. prepare:
dd if=/dev/zero of=/root/fc_move.img bs=1M count=0 seek=256
mkfs.ext4 -O fast_commit -F /root/fc_move.img
mkdir -p /mnt/fc_move && mount -t ext4 -o loop \
/root/fc_move.img /mnt/fc_move
2. Created two files, ran EXT4_IOC_MOVE_EXT via e4defrag, and checked
the ineligible reason statistics:
fallocate -l 64M /mnt/fc_move/file1
cp /mnt/fc_move/file1 /mnt/fc_move/file2
e4defrag /mnt/fc_move/file1
cat /proc/fs/ext4/loop0/fc_info
shows "Move extents": > 0 and fc stats ineligible > 0.
Signed-off-by: Li Chen <me@linux.beauty>
---
fs/ext4/fast_commit.c | 1 +
fs/ext4/fast_commit.h | 1 +
fs/ext4/move_extent.c | 1 +
include/trace/events/ext4.h | 4 +++-
4 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/fs/ext4/fast_commit.c b/fs/ext4/fast_commit.c
index 242b69e5fe13..0ef2154a2b1f 100644
--- a/fs/ext4/fast_commit.c
+++ b/fs/ext4/fast_commit.c
@@ -2304,6 +2304,7 @@ static const char * const fc_ineligible_reasons[] = {
[EXT4_FC_REASON_ENCRYPTED_FILENAME] = "Encrypted filename",
[EXT4_FC_REASON_MIGRATE] = "Inode format migration",
[EXT4_FC_REASON_VERITY] = "fs-verity enable",
+ [EXT4_FC_REASON_MOVE_EXT] = "Move extents",
};
int ext4_fc_info_show(struct seq_file *seq, void *v)
diff --git a/fs/ext4/fast_commit.h b/fs/ext4/fast_commit.h
index 20f65135208f..2f77a37fb101 100644
--- a/fs/ext4/fast_commit.h
+++ b/fs/ext4/fast_commit.h
@@ -99,6 +99,7 @@ enum {
EXT4_FC_REASON_ENCRYPTED_FILENAME,
EXT4_FC_REASON_MIGRATE,
EXT4_FC_REASON_VERITY,
+ EXT4_FC_REASON_MOVE_EXT,
EXT4_FC_REASON_MAX
};
diff --git a/fs/ext4/move_extent.c b/fs/ext4/move_extent.c
index 4b091c21908f..5a5e91078528 100644
--- a/fs/ext4/move_extent.c
+++ b/fs/ext4/move_extent.c
@@ -287,6 +287,7 @@ move_extent_per_page(struct file *o_filp, struct inode *donor_inode,
*err = PTR_ERR(handle);
return 0;
}
+ ext4_fc_mark_ineligible(sb, EXT4_FC_REASON_MOVE_EXT, handle);
orig_blk_offset = orig_page_offset * blocks_per_page +
data_offset_in_page;
diff --git a/include/trace/events/ext4.h b/include/trace/events/ext4.h
index 224ab12ee83f..56e60080e759 100644
--- a/include/trace/events/ext4.h
+++ b/include/trace/events/ext4.h
@@ -104,6 +104,7 @@ TRACE_DEFINE_ENUM(EXT4_FC_REASON_INODE_JOURNAL_DATA);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_ENCRYPTED_FILENAME);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_MIGRATE);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_VERITY);
+TRACE_DEFINE_ENUM(EXT4_FC_REASON_MOVE_EXT);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_MAX);
#define show_fc_reason(reason) \
@@ -119,7 +120,8 @@ TRACE_DEFINE_ENUM(EXT4_FC_REASON_MAX);
{ EXT4_FC_REASON_INODE_JOURNAL_DATA, "INODE_JOURNAL_DATA"}, \
{ EXT4_FC_REASON_ENCRYPTED_FILENAME, "ENCRYPTED_FILENAME"}, \
{ EXT4_FC_REASON_MIGRATE, "MIGRATE"}, \
- { EXT4_FC_REASON_VERITY, "VERITY"})
+ { EXT4_FC_REASON_VERITY, "VERITY"}, \
+ { EXT4_FC_REASON_MOVE_EXT, "MOVE_EXT"})
TRACE_DEFINE_ENUM(CR_POWER2_ALIGNED);
TRACE_DEFINE_ENUM(CR_GOAL_LEN_FAST);
--
2.51.0
^ permalink raw reply related
* [RFC 2/5] ext4: mark fs-verity enable fast-commit ineligible
From: Li Chen @ 2025-12-11 11:51 UTC (permalink / raw)
To: Theodore Ts'o, Andreas Dilger, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, linux-ext4, linux-kernel,
linux-trace-kernel
Cc: Li Chen
In-Reply-To: <20251211115146.897420-1-me@linux.beauty>
Fast commits only log operations that have dedicated replay support.
Enabling fs-verity builds a Merkle tree and updates inode and orphan
state in ways that are not described by the fast commit replay tags.
In practice these operations are rare and usually followed by further
updates, but mixing them into a fast commit makes the overall
semantics harder to reason about and risks replay gaps if new call
sites appear.
Teach ext4 to mark the filesystem fast-commit ineligible when
ext4_end_enable_verity() starts its journal transaction.
This forces that transaction to fall back to a full commit, ensuring
that the fs-verity enable changes are captured by the normal journal
rather than partially encoded in fast commit TLVs.
This change should not affect common workloads but makes fs-verity
enable safer and easier to reason about under fast commit.
Testing:
1. prepare:
dd if=/dev/zero of=/root/fc_verity.img bs=1M count=0 seek=128
mkfs.ext4 -O fast_commit,verity -F /root/fc_verity.img
mkdir -p /mnt/fc_verity && mount -t ext4 -o loop /root/fc_verity.img /mnt/fc_verity
2. Enabled fs-verity on a file and verified reason accounting:
echo "data" > /mnt/fc_verity/verityfile
/root/enable_verity /mnt/fc_verity/verityfile
sync
tail -n 1 /proc/fs/ext4/loop0/fc_info
"fs-verity enable": 1
3. Enabled fs-verity on a second file, fsynced it, and checked that the
ineligible commit counter is updated too:
echo "data2" > /mnt/fc_verity/verityfile2
/root/enable_verity /mnt/fc_verity/verityfile2
/root/fsync_file /mnt/fc_verity/verityfile2
sync
/proc/fs/ext4/loop0/fc_info shows "fs-verity enable" incremented and
fc stats ineligible increased accordingly.
Signed-off-by: Li Chen <me@linux.beauty>
---
fs/ext4/fast_commit.c | 1 +
fs/ext4/fast_commit.h | 1 +
fs/ext4/verity.c | 2 ++
include/trace/events/ext4.h | 4 +++-
4 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/fs/ext4/fast_commit.c b/fs/ext4/fast_commit.c
index afb28b3e52bb..242b69e5fe13 100644
--- a/fs/ext4/fast_commit.c
+++ b/fs/ext4/fast_commit.c
@@ -2303,6 +2303,7 @@ static const char * const fc_ineligible_reasons[] = {
[EXT4_FC_REASON_INODE_JOURNAL_DATA] = "Data journalling",
[EXT4_FC_REASON_ENCRYPTED_FILENAME] = "Encrypted filename",
[EXT4_FC_REASON_MIGRATE] = "Inode format migration",
+ [EXT4_FC_REASON_VERITY] = "fs-verity enable",
};
int ext4_fc_info_show(struct seq_file *seq, void *v)
diff --git a/fs/ext4/fast_commit.h b/fs/ext4/fast_commit.h
index be3b84a74c32..20f65135208f 100644
--- a/fs/ext4/fast_commit.h
+++ b/fs/ext4/fast_commit.h
@@ -98,6 +98,7 @@ enum {
EXT4_FC_REASON_INODE_JOURNAL_DATA,
EXT4_FC_REASON_ENCRYPTED_FILENAME,
EXT4_FC_REASON_MIGRATE,
+ EXT4_FC_REASON_VERITY,
EXT4_FC_REASON_MAX
};
diff --git a/fs/ext4/verity.c b/fs/ext4/verity.c
index b0acb0c50313..6115a365d491 100644
--- a/fs/ext4/verity.c
+++ b/fs/ext4/verity.c
@@ -231,6 +231,8 @@ static int ext4_end_enable_verity(struct file *filp, const void *desc,
goto cleanup;
}
+ ext4_fc_mark_ineligible(inode->i_sb, EXT4_FC_REASON_VERITY, handle);
+
err = ext4_orphan_del(handle, inode);
if (err)
goto stop_and_cleanup;
diff --git a/include/trace/events/ext4.h b/include/trace/events/ext4.h
index 8f75d41ae5ef..224ab12ee83f 100644
--- a/include/trace/events/ext4.h
+++ b/include/trace/events/ext4.h
@@ -103,6 +103,7 @@ TRACE_DEFINE_ENUM(EXT4_FC_REASON_FALLOC_RANGE);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_INODE_JOURNAL_DATA);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_ENCRYPTED_FILENAME);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_MIGRATE);
+TRACE_DEFINE_ENUM(EXT4_FC_REASON_VERITY);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_MAX);
#define show_fc_reason(reason) \
@@ -117,7 +118,8 @@ TRACE_DEFINE_ENUM(EXT4_FC_REASON_MAX);
{ EXT4_FC_REASON_FALLOC_RANGE, "FALLOC_RANGE"}, \
{ EXT4_FC_REASON_INODE_JOURNAL_DATA, "INODE_JOURNAL_DATA"}, \
{ EXT4_FC_REASON_ENCRYPTED_FILENAME, "ENCRYPTED_FILENAME"}, \
- { EXT4_FC_REASON_MIGRATE, "MIGRATE"})
+ { EXT4_FC_REASON_MIGRATE, "MIGRATE"}, \
+ { EXT4_FC_REASON_VERITY, "VERITY"})
TRACE_DEFINE_ENUM(CR_POWER2_ALIGNED);
TRACE_DEFINE_ENUM(CR_GOAL_LEN_FAST);
--
2.51.0
^ permalink raw reply related
* [RFC 1/5] ext4: mark inode format migration fast-commit ineligible
From: Li Chen @ 2025-12-11 11:51 UTC (permalink / raw)
To: Theodore Ts'o, Andreas Dilger, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, linux-ext4, linux-kernel,
linux-trace-kernel
Cc: Li Chen
In-Reply-To: <20251211115146.897420-1-me@linux.beauty>
Fast commits only log operations that have dedicated replay support.
Inode format migration (indirect<->extent layout changes via
EXT4_IOC_MIGRATE or toggling EXT4_EXTENTS_FL) rewrites the block mapping
representation without going through the fast commit tracking paths.
In practice these migrations are rare and usually followed by further
updates, but mixing them into a fast commit makes the overall semantics
harder to reason about and risks replay gaps if new call sites appear.
Teach ext4 to mark the filesystem fast-commit ineligible when
ext4_ext_migrate() or ext4_ind_migrate() start their journal transactions.
This forces those transactions to fall back to a full commit, ensuring
that the entire inode layout change is captured by the normal journal
rather than partially encoded in fast commit TLVs. This change should
not affect common workloads but makes format migrations safer and easier
to reason about under fast commit.
Testing:
1. prepare:
dd if=/dev/zero of=/root/fc.img bs=1M count=0 seek=128
mkfs.ext4 -O fast_commit -F /root/fc.img
mkdir -p /mnt/fc && mount -t ext4 -o loop /root/fc.img /mnt/fc
2. Created a test file and toggled the extents flag to exercise both
ext4_ind_migrate() and ext4_ext_migrate():
touch /mnt/fc/migtest
chattr -e /mnt/fc/migtest
chattr +e /mnt/fc/migtest
3. Verified fast-commit ineligible statistics:
tail -n 1 /proc/fs/ext4/loop0/fc_info
"Inode format migration": 2
Signed-off-by: Li Chen <me@linux.beauty>
---
fs/ext4/fast_commit.c | 1 +
fs/ext4/fast_commit.h | 1 +
fs/ext4/migrate.c | 12 ++++++++++++
include/trace/events/ext4.h | 4 +++-
4 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/fs/ext4/fast_commit.c b/fs/ext4/fast_commit.c
index fa66b08de999..afb28b3e52bb 100644
--- a/fs/ext4/fast_commit.c
+++ b/fs/ext4/fast_commit.c
@@ -2302,6 +2302,7 @@ static const char * const fc_ineligible_reasons[] = {
[EXT4_FC_REASON_FALLOC_RANGE] = "Falloc range op",
[EXT4_FC_REASON_INODE_JOURNAL_DATA] = "Data journalling",
[EXT4_FC_REASON_ENCRYPTED_FILENAME] = "Encrypted filename",
+ [EXT4_FC_REASON_MIGRATE] = "Inode format migration",
};
int ext4_fc_info_show(struct seq_file *seq, void *v)
diff --git a/fs/ext4/fast_commit.h b/fs/ext4/fast_commit.h
index 3bd534e4dbbf..be3b84a74c32 100644
--- a/fs/ext4/fast_commit.h
+++ b/fs/ext4/fast_commit.h
@@ -97,6 +97,7 @@ enum {
EXT4_FC_REASON_FALLOC_RANGE,
EXT4_FC_REASON_INODE_JOURNAL_DATA,
EXT4_FC_REASON_ENCRYPTED_FILENAME,
+ EXT4_FC_REASON_MIGRATE,
EXT4_FC_REASON_MAX
};
diff --git a/fs/ext4/migrate.c b/fs/ext4/migrate.c
index 1b0dfd963d3f..96ab95167bd6 100644
--- a/fs/ext4/migrate.c
+++ b/fs/ext4/migrate.c
@@ -449,6 +449,12 @@ int ext4_ext_migrate(struct inode *inode)
retval = PTR_ERR(handle);
goto out_unlock;
}
+ /*
+ * This operation rewrites the inode's block mapping layout
+ * (indirect to extents) and is not tracked in the fast commit
+ * log, so disable fast commits for this transaction.
+ */
+ ext4_fc_mark_ineligible(inode->i_sb, EXT4_FC_REASON_MIGRATE, handle);
goal = (((inode->i_ino - 1) / EXT4_INODES_PER_GROUP(inode->i_sb)) *
EXT4_INODES_PER_GROUP(inode->i_sb)) + 1;
owner[0] = i_uid_read(inode);
@@ -630,6 +636,12 @@ int ext4_ind_migrate(struct inode *inode)
ret = PTR_ERR(handle);
goto out_unlock;
}
+ /*
+ * This operation rewrites the inode's block mapping layout
+ * (extents to indirect blocks) and is not tracked in the fast
+ * commit log, so disable fast commits for this transaction.
+ */
+ ext4_fc_mark_ineligible(inode->i_sb, EXT4_FC_REASON_MIGRATE, handle);
down_write(&EXT4_I(inode)->i_data_sem);
ret = ext4_ext_check_inode(inode);
diff --git a/include/trace/events/ext4.h b/include/trace/events/ext4.h
index a374e7ea7e57..8f75d41ae5ef 100644
--- a/include/trace/events/ext4.h
+++ b/include/trace/events/ext4.h
@@ -102,6 +102,7 @@ TRACE_DEFINE_ENUM(EXT4_FC_REASON_RENAME_DIR);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_FALLOC_RANGE);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_INODE_JOURNAL_DATA);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_ENCRYPTED_FILENAME);
+TRACE_DEFINE_ENUM(EXT4_FC_REASON_MIGRATE);
TRACE_DEFINE_ENUM(EXT4_FC_REASON_MAX);
#define show_fc_reason(reason) \
@@ -115,7 +116,8 @@ TRACE_DEFINE_ENUM(EXT4_FC_REASON_MAX);
{ EXT4_FC_REASON_RENAME_DIR, "RENAME_DIR"}, \
{ EXT4_FC_REASON_FALLOC_RANGE, "FALLOC_RANGE"}, \
{ EXT4_FC_REASON_INODE_JOURNAL_DATA, "INODE_JOURNAL_DATA"}, \
- { EXT4_FC_REASON_ENCRYPTED_FILENAME, "ENCRYPTED_FILENAME"})
+ { EXT4_FC_REASON_ENCRYPTED_FILENAME, "ENCRYPTED_FILENAME"}, \
+ { EXT4_FC_REASON_MIGRATE, "MIGRATE"})
TRACE_DEFINE_ENUM(CR_POWER2_ALIGNED);
TRACE_DEFINE_ENUM(CR_GOAL_LEN_FAST);
--
2.51.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox