* [PATCH v3 0/2] Replace trace_cpu_frequency with trace_policy_frequency
From: Samuel Wu @ 2025-12-01 20:24 UTC (permalink / raw)
To: Huang Rui, Gautham R. Shenoy, Mario Limonciello, Perry Yuan,
Jonathan Corbet, Rafael J. Wysocki, Viresh Kumar, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Srinivas Pandruvada,
Len Brown, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Ian Rogers,
Adrian Hunter, James Clark
Cc: christian.loehle, Samuel Wu, kernel-team, linux-pm, linux-doc,
linux-kernel, linux-trace-kernel, bpf, linux-perf-users
This series replaces the cpu_frequency trace event with a new trace event,
policy_frequency. Since by definition all CPUs in a policy are of the same
frequency, we can emit a frequency change per policy instead of per CPU.
This saves some compute and memory from the kernel side, while simplifying
analysis from the post-processing of the trace log side.
Any process that relied on cpu_frequency trace event needs to switch to the
new policy_frequency trace event in order to maintain functionality. The
decision of replacing instead of adding the trace event is intentional. Since
emitting once per policy instead of once per CPU is anyways a semantics change
that would require a tooling update, the trace event was also appropriately
renamed. The presence of the policy_frequency event in a trace log is a clear
and obvious signal for tooling to determine kernel version and which trace
event to parse.
1/2: Replaces trace_cpu_frequency with trace_policy_frequency
2/2: Corresponding documentation patch that updates references to
cpu_frequency with policy_frequency
Changes in v3:
- Resending v2 properly (accidentally ommited cover letter in v2)
Changes in v2:
- Replaced trace_cpu_frequency with trace_policy_frequency (per Christian
and Viresh)
- Updated references to cpu_frequency in documentation with
policy_frequency
- v1 link: https://lore.kernel.org/all/20251112235154.2974902-1-wusamuel@google.com
Samuel Wu (2):
cpufreq: Replace trace_cpu_frequency with trace_policy_frequency
cpufreq: Documentation update for trace_policy_frequency
Documentation/admin-guide/pm/amd-pstate.rst | 10 ++++----
Documentation/admin-guide/pm/intel_pstate.rst | 14 +++++------
Documentation/trace/events-power.rst | 2 +-
drivers/cpufreq/cpufreq.c | 14 ++---------
drivers/cpufreq/intel_pstate.c | 6 +++--
include/trace/events/power.h | 24 ++++++++++++++++---
kernel/trace/power-traces.c | 2 +-
samples/bpf/cpustat_kern.c | 8 +++----
samples/bpf/cpustat_user.c | 6 ++---
tools/perf/builtin-timechart.c | 12 +++++-----
10 files changed, 54 insertions(+), 44 deletions(-)
--
2.52.0.107.ga0afd4fd5b-goog
^ permalink raw reply
* [PATCH v2 2/2] cpufreq: Documentation update for trace_policy_frequency
From: Samuel Wu @ 2025-12-01 20:13 UTC (permalink / raw)
To: Huang Rui, Gautham R. Shenoy, Mario Limonciello, Perry Yuan,
Jonathan Corbet, Rafael J. Wysocki, Viresh Kumar, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Srinivas Pandruvada,
Len Brown, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Ian Rogers,
Adrian Hunter, James Clark
Cc: christian.loehle, Samuel Wu, kernel-team, linux-pm, linux-doc,
linux-kernel, linux-trace-kernel, bpf, linux-perf-users
Documentation update corresponding to replace the cpu_frequency trace
event with the policy_frequency trace event.
Signed-off-by: Samuel Wu <wusamuel@google.com>
---
Documentation/admin-guide/pm/amd-pstate.rst | 10 +++++-----
Documentation/admin-guide/pm/intel_pstate.rst | 14 +++++++-------
Documentation/trace/events-power.rst | 2 +-
3 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/Documentation/admin-guide/pm/amd-pstate.rst b/Documentation/admin-guide/pm/amd-pstate.rst
index e1771f2225d5..e110854ece88 100644
--- a/Documentation/admin-guide/pm/amd-pstate.rst
+++ b/Documentation/admin-guide/pm/amd-pstate.rst
@@ -503,8 +503,8 @@ Trace Events
--------------
There are two static trace events that can be used for ``amd-pstate``
-diagnostics. One of them is the ``cpu_frequency`` trace event generally used
-by ``CPUFreq``, and the other one is the ``amd_pstate_perf`` trace event
+diagnostics. One of them is the ``policy_frequency`` trace event generally
+used by ``CPUFreq``, and the other one is the ``amd_pstate_perf`` trace event
specific to ``amd-pstate``. The following sequence of shell commands can
be used to enable them and see their output (if the kernel is
configured to support event tracing). ::
@@ -531,9 +531,9 @@ configured to support event tracing). ::
<idle>-0 [003] d.s.. 4995.980971: amd_pstate_perf: amd_min_perf=85 amd_des_perf=85 amd_max_perf=166 cpu_id=3 changed=false fast_switch=true
<idle>-0 [011] d.s.. 4995.980996: amd_pstate_perf: amd_min_perf=85 amd_des_perf=85 amd_max_perf=166 cpu_id=11 changed=false fast_switch=true
-The ``cpu_frequency`` trace event will be triggered either by the ``schedutil`` scaling
-governor (for the policies it is attached to), or by the ``CPUFreq`` core (for the
-policies with other scaling governors).
+The ``policy_frequency`` trace event will be triggered either by the
+``schedutil`` scaling governor (for the policies it is attached to), or by the
+``CPUFreq`` core (for the policies with other scaling governors).
Tracer Tool
diff --git a/Documentation/admin-guide/pm/intel_pstate.rst b/Documentation/admin-guide/pm/intel_pstate.rst
index fde967b0c2e0..274c9208f342 100644
--- a/Documentation/admin-guide/pm/intel_pstate.rst
+++ b/Documentation/admin-guide/pm/intel_pstate.rst
@@ -822,23 +822,23 @@ Trace Events
------------
There are two static trace events that can be used for ``intel_pstate``
-diagnostics. One of them is the ``cpu_frequency`` trace event generally used
-by ``CPUFreq``, and the other one is the ``pstate_sample`` trace event specific
-to ``intel_pstate``. Both of them are triggered by ``intel_pstate`` only if
-it works in the :ref:`active mode <active_mode>`.
+diagnostics. One of them is the ``policy_frequency`` trace event generally
+used by ``CPUFreq``, and the other one is the ``pstate_sample`` trace event
+specific to ``intel_pstate``. Both of them are triggered by ``intel_pstate``
+only if it works in the :ref:`active mode <active_mode>`.
The following sequence of shell commands can be used to enable them and see
their output (if the kernel is generally configured to support event tracing)::
# cd /sys/kernel/tracing/
# echo 1 > events/power/pstate_sample/enable
- # echo 1 > events/power/cpu_frequency/enable
+ # echo 1 > events/power/policy_frequency/enable
# cat trace
gnome-terminal--4510 [001] ..s. 1177.680733: pstate_sample: core_busy=107 scaled=94 from=26 to=26 mperf=1143818 aperf=1230607 tsc=29838618 freq=2474476
- cat-5235 [002] ..s. 1177.681723: cpu_frequency: state=2900000 cpu_id=2
+ cat-5235 [002] ..s. 1177.681723: policy_frequency: state=2900000 cpu_id=2 policy_cpus=04
If ``intel_pstate`` works in the :ref:`passive mode <passive_mode>`, the
-``cpu_frequency`` trace event will be triggered either by the ``schedutil``
+``policy_frequency`` trace event will be triggered either by the ``schedutil``
scaling governor (for the policies it is attached to), or by the ``CPUFreq``
core (for the policies with other scaling governors).
diff --git a/Documentation/trace/events-power.rst b/Documentation/trace/events-power.rst
index f45bf11fa88d..f013c74b932f 100644
--- a/Documentation/trace/events-power.rst
+++ b/Documentation/trace/events-power.rst
@@ -26,8 +26,8 @@ cpufreq.
::
cpu_idle "state=%lu cpu_id=%lu"
- cpu_frequency "state=%lu cpu_id=%lu"
cpu_frequency_limits "min=%lu max=%lu cpu_id=%lu"
+ policy_frequency "state=%lu cpu_id=%lu policy_cpus=%*pb"
A suspend event is used to indicate the system going in and out of the
suspend mode:
--
2.52.0.107.ga0afd4fd5b-goog
^ permalink raw reply related
* [PATCH v2 1/2] cpufreq: Replace trace_cpu_frequency with trace_policy_frequency
From: Samuel Wu @ 2025-12-01 20:13 UTC (permalink / raw)
To: Huang Rui, Gautham R. Shenoy, Mario Limonciello, Perry Yuan,
Jonathan Corbet, Rafael J. Wysocki, Viresh Kumar, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Srinivas Pandruvada,
Len Brown, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Ian Rogers,
Adrian Hunter, James Clark
Cc: christian.loehle, Samuel Wu, kernel-team, linux-pm, linux-doc,
linux-kernel, linux-trace-kernel, bpf, linux-perf-users
The existing cpu_frequency trace_event can be verbose, emitting a nearly
identical trace event for every CPU in the policy even when their
frequencies are identical.
This patch replaces the cpu_frequency trace event with policy_frequency
trace event, a more efficient alternative. From the kernel's
perspective, emitting a trace event once per policy instead of once per
cpu saves some memory and is less overhead. From the post-processing
perspective, analysis of the trace log is simplified without any loss of
information.
Signed-off-by: Samuel Wu <wusamuel@google.com>
---
drivers/cpufreq/cpufreq.c | 14 ++------------
drivers/cpufreq/intel_pstate.c | 6 ++++--
include/trace/events/power.h | 24 +++++++++++++++++++++---
kernel/trace/power-traces.c | 2 +-
samples/bpf/cpustat_kern.c | 8 ++++----
samples/bpf/cpustat_user.c | 6 +++---
tools/perf/builtin-timechart.c | 12 ++++++------
7 files changed, 41 insertions(+), 31 deletions(-)
diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
index 4472bb1ec83c..dd3f08f3b958 100644
--- a/drivers/cpufreq/cpufreq.c
+++ b/drivers/cpufreq/cpufreq.c
@@ -309,8 +309,6 @@ static void cpufreq_notify_transition(struct cpufreq_policy *policy,
struct cpufreq_freqs *freqs,
unsigned int state)
{
- int cpu;
-
BUG_ON(irqs_disabled());
if (cpufreq_disabled())
@@ -344,10 +342,7 @@ static void cpufreq_notify_transition(struct cpufreq_policy *policy,
adjust_jiffies(CPUFREQ_POSTCHANGE, freqs);
pr_debug("FREQ: %u - CPUs: %*pbl\n", freqs->new,
cpumask_pr_args(policy->cpus));
-
- for_each_cpu(cpu, policy->cpus)
- trace_cpu_frequency(freqs->new, cpu);
-
+ trace_policy_frequency(freqs->new, policy->cpu, policy->cpus);
srcu_notifier_call_chain(&cpufreq_transition_notifier_list,
CPUFREQ_POSTCHANGE, freqs);
@@ -2201,7 +2196,6 @@ unsigned int cpufreq_driver_fast_switch(struct cpufreq_policy *policy,
unsigned int target_freq)
{
unsigned int freq;
- int cpu;
target_freq = clamp_val(target_freq, policy->min, policy->max);
freq = cpufreq_driver->fast_switch(policy, target_freq);
@@ -2213,11 +2207,7 @@ unsigned int cpufreq_driver_fast_switch(struct cpufreq_policy *policy,
arch_set_freq_scale(policy->related_cpus, freq,
arch_scale_freq_ref(policy->cpu));
cpufreq_stats_record_transition(policy, freq);
-
- if (trace_cpu_frequency_enabled()) {
- for_each_cpu(cpu, policy->cpus)
- trace_cpu_frequency(freq, cpu);
- }
+ trace_policy_frequency(freq, policy->cpu, policy->cpus);
return freq;
}
diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c
index ec4abe374573..9724b5d19d83 100644
--- a/drivers/cpufreq/intel_pstate.c
+++ b/drivers/cpufreq/intel_pstate.c
@@ -2297,7 +2297,8 @@ static int hwp_get_cpu_scaling(int cpu)
static void intel_pstate_set_pstate(struct cpudata *cpu, int pstate)
{
- trace_cpu_frequency(pstate * cpu->pstate.scaling, cpu->cpu);
+ trace_policy_frequency(pstate * cpu->pstate.scaling, cpu->cpu,
+ cpumask_of(cpu->cpu));
cpu->pstate.current_pstate = pstate;
/*
* Generally, there is no guarantee that this code will always run on
@@ -2587,7 +2588,8 @@ static void intel_pstate_adjust_pstate(struct cpudata *cpu)
target_pstate = get_target_pstate(cpu);
target_pstate = intel_pstate_prepare_request(cpu, target_pstate);
- trace_cpu_frequency(target_pstate * cpu->pstate.scaling, cpu->cpu);
+ trace_policy_frequency(target_pstate * cpu->pstate.scaling, cpu->cpu,
+ cpumask_of(cpu->cpu));
intel_pstate_update_pstate(cpu, target_pstate);
sample = &cpu->sample;
diff --git a/include/trace/events/power.h b/include/trace/events/power.h
index 370f8df2fdb4..317098ffdd5f 100644
--- a/include/trace/events/power.h
+++ b/include/trace/events/power.h
@@ -182,11 +182,29 @@ TRACE_EVENT(pstate_sample,
{ PM_EVENT_RECOVER, "recover" }, \
{ PM_EVENT_POWEROFF, "poweroff" })
-DEFINE_EVENT(cpu, cpu_frequency,
+TRACE_EVENT(policy_frequency,
- TP_PROTO(unsigned int frequency, unsigned int cpu_id),
+ TP_PROTO(unsigned int frequency, unsigned int cpu_id,
+ const struct cpumask *policy_cpus),
- TP_ARGS(frequency, cpu_id)
+ TP_ARGS(frequency, cpu_id, policy_cpus),
+
+ TP_STRUCT__entry(
+ __field(u32, state)
+ __field(u32, cpu_id)
+ __cpumask(cpumask)
+ ),
+
+ TP_fast_assign(
+ __entry->state = frequency;
+ __entry->cpu_id = cpu_id;
+ __assign_cpumask(cpumask, policy_cpus);
+ ),
+
+ TP_printk("state=%lu cpu_id=%lu policy_cpus=%*pb",
+ (unsigned long)__entry->state,
+ (unsigned long)__entry->cpu_id,
+ cpumask_pr_args((struct cpumask *)__get_dynamic_array(cpumask)))
);
TRACE_EVENT(cpu_frequency_limits,
diff --git a/kernel/trace/power-traces.c b/kernel/trace/power-traces.c
index f2fe33573e54..a537e68a6878 100644
--- a/kernel/trace/power-traces.c
+++ b/kernel/trace/power-traces.c
@@ -16,5 +16,5 @@
EXPORT_TRACEPOINT_SYMBOL_GPL(suspend_resume);
EXPORT_TRACEPOINT_SYMBOL_GPL(cpu_idle);
-EXPORT_TRACEPOINT_SYMBOL_GPL(cpu_frequency);
+EXPORT_TRACEPOINT_SYMBOL_GPL(policy_frequency);
diff --git a/samples/bpf/cpustat_kern.c b/samples/bpf/cpustat_kern.c
index 7ec7143e2757..f485de0f89b2 100644
--- a/samples/bpf/cpustat_kern.c
+++ b/samples/bpf/cpustat_kern.c
@@ -75,9 +75,9 @@ struct {
} pstate_duration SEC(".maps");
/*
- * The trace events for cpu_idle and cpu_frequency are taken from:
+ * The trace events for cpu_idle and policy_frequency are taken from:
* /sys/kernel/tracing/events/power/cpu_idle/format
- * /sys/kernel/tracing/events/power/cpu_frequency/format
+ * /sys/kernel/tracing/events/power/policy_frequency/format
*
* These two events have same format, so define one common structure.
*/
@@ -162,7 +162,7 @@ int bpf_prog1(struct cpu_args *ctx)
*/
if (ctx->state != (u32)-1) {
- /* record pstate after have first cpu_frequency event */
+ /* record pstate after have first policy_frequency event */
if (!*pts)
return 0;
@@ -208,7 +208,7 @@ int bpf_prog1(struct cpu_args *ctx)
return 0;
}
-SEC("tracepoint/power/cpu_frequency")
+SEC("tracepoint/power/policy_frequency")
int bpf_prog2(struct cpu_args *ctx)
{
u64 *pts, *cstate, *pstate, cur_ts, delta;
diff --git a/samples/bpf/cpustat_user.c b/samples/bpf/cpustat_user.c
index 356f756cba0d..f7e81f702358 100644
--- a/samples/bpf/cpustat_user.c
+++ b/samples/bpf/cpustat_user.c
@@ -143,12 +143,12 @@ static int cpu_stat_inject_cpu_idle_event(void)
/*
* It's possible to have no any frequency change for long time and cannot
- * get ftrace event 'trace_cpu_frequency' for long period, this introduces
+ * get ftrace event 'trace_policy_frequency' for long period, this introduces
* big deviation for pstate statistics.
*
* To solve this issue, below code forces to set 'scaling_max_freq' to 208MHz
- * for triggering ftrace event 'trace_cpu_frequency' and then recovery back to
- * the maximum frequency value 1.2GHz.
+ * for triggering ftrace event 'trace_policy_frequency' and then recovery back
+ * to the maximum frequency value 1.2GHz.
*/
static int cpu_stat_inject_cpu_frequency_event(void)
{
diff --git a/tools/perf/builtin-timechart.c b/tools/perf/builtin-timechart.c
index 22050c640dfa..3ef1a2fd0493 100644
--- a/tools/perf/builtin-timechart.c
+++ b/tools/perf/builtin-timechart.c
@@ -612,10 +612,10 @@ process_sample_cpu_idle(struct timechart *tchart __maybe_unused,
}
static int
-process_sample_cpu_frequency(struct timechart *tchart,
- struct evsel *evsel,
- struct perf_sample *sample,
- const char *backtrace __maybe_unused)
+process_sample_policy_frequency(struct timechart *tchart,
+ struct evsel *evsel,
+ struct perf_sample *sample,
+ const char *backtrace __maybe_unused)
{
u32 state = evsel__intval(evsel, sample, "state");
u32 cpu_id = evsel__intval(evsel, sample, "cpu_id");
@@ -1541,7 +1541,7 @@ static int __cmd_timechart(struct timechart *tchart, const char *output_name)
{
const struct evsel_str_handler power_tracepoints[] = {
{ "power:cpu_idle", process_sample_cpu_idle },
- { "power:cpu_frequency", process_sample_cpu_frequency },
+ { "power:policy_frequency", process_sample_policy_frequency },
{ "sched:sched_wakeup", process_sample_sched_wakeup },
{ "sched:sched_switch", process_sample_sched_switch },
#ifdef SUPPORT_OLD_POWER_EVENTS
@@ -1804,7 +1804,7 @@ static int timechart__record(struct timechart *tchart, int argc, const char **ar
unsigned int backtrace_args_no = ARRAY_SIZE(backtrace_args);
const char * const power_args[] = {
- "-e", "power:cpu_frequency",
+ "-e", "power:policy_frequency",
"-e", "power:cpu_idle",
};
unsigned int power_args_nr = ARRAY_SIZE(power_args);
--
2.52.0.107.ga0afd4fd5b-goog
^ permalink raw reply related
* Re: [PATCH 2/3] kernel.h: move VERIFY_OCTAL_PERMISSIONS() to sysfs.h
From: Andy Shevchenko @ 2025-12-01 20:00 UTC (permalink / raw)
To: Randy Dunlap, Jonathan Cameron
Cc: Yury Norov, Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Ingo Molnar, Jani Nikula, Joonas Lahtinen, Rodrigo Vivi,
Tvrtko Ursulin, Petr Pavlu, Daniel Gomez, Greg Kroah-Hartman,
Rafael J. Wysocki, Danilo Krummrich, Andrew Morton, linux-kernel,
intel-gfx, dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <fd755bbf-50a8-46f7-bff1-61cc625118a9@infradead.org>
On Mon, Dec 01, 2025 at 11:51:24AM -0800, Randy Dunlap wrote:
> On 11/30/25 11:38 AM, Andy Shevchenko wrote:
> > On Sun, Nov 30, 2025 at 12:42:35PM -0500, Yury Norov wrote:
>
> >> This series was tested by 0-day and LKP. 0-day runs allyesconfig,
> >
> > AFAICS in the below no configuration had been tested against allYESconfig.
> > All of them are allNOconfig.
> >
> >> as far as I know. It only sends email in case of errors. LKP is OK, find the
> >> report below.
> >
> >> All but XFS include it via linux/module.h -> linux/moduleparam.h path.
> >> XFS has a linkage layer: xfs.h -> xfs_linux.h-> linux/module.h, so
> >> it's pretty much the same.
> >>
> >> I think, module.h inclusion path is OK for this macro and definitely
> >> better than kernel.h. Notice, none of them, except for vgpu_dbg,
> >> include kernel.h directly.
> >
> > Ideally those (especially and in the first place headers) should follow IWYU
> > principle and avoid indirect (non-guaranteed) inclusions.
>
> Can you (or anyone) get IWYU (software) to work?
> I tried it a few months ago but didn't have the correct magic
> incantation for it.
> (no specifics at the moment)
You should talk to Jonathan Cameron (Cc'ed), he was able to run it to some
extent. AFAIR the state of affairs is that it gives a lot of low-level headers
that we should not really go too deep to (at least for now). That means the
carefully crafted map of guarantees needs to be provided (e.g., if we include
bitmap.h, bitops.h and/or bits.h are guaranteed, so no need to be included).
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply
* Re: [PATCH 2/3] kernel.h: move VERIFY_OCTAL_PERMISSIONS() to sysfs.h
From: Randy Dunlap @ 2025-12-01 19:51 UTC (permalink / raw)
To: Andy Shevchenko, Yury Norov
Cc: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers, Ingo Molnar,
Jani Nikula, Joonas Lahtinen, Rodrigo Vivi, Tvrtko Ursulin,
Petr Pavlu, Daniel Gomez, Greg Kroah-Hartman, Rafael J. Wysocki,
Danilo Krummrich, Andrew Morton, linux-kernel, intel-gfx,
dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <aSydSI-h3KZiYBn6@smile.fi.intel.com>
On 11/30/25 11:38 AM, Andy Shevchenko wrote:
> On Sun, Nov 30, 2025 at 12:42:35PM -0500, Yury Norov wrote:
>> This series was tested by 0-day and LKP. 0-day runs allyesconfig,
>
> AFAICS in the below no configuration had been tested against allYESconfig.
> All of them are allNOconfig.
>
>> as far as I know. It only sends email in case of errors. LKP is OK, find the
>> report below.
>
>> All but XFS include it via linux/module.h -> linux/moduleparam.h path.
>> XFS has a linkage layer: xfs.h -> xfs_linux.h-> linux/module.h, so
>> it's pretty much the same.
>>
>> I think, module.h inclusion path is OK for this macro and definitely
>> better than kernel.h. Notice, none of them, except for vgpu_dbg,
>> include kernel.h directly.
>
> Ideally those (especially and in the first place headers) should follow IWYU
> principle and avoid indirect (non-guaranteed) inclusions.
Can you (or anyone) get IWYU (software) to work?
I tried it a few months ago but didn't have the correct magic
incantation for it.
(no specifics at the moment)
--
~Randy
^ permalink raw reply
* Re: [PATCH bpf 0/2] bpf: fix bpf_d_path() helper prototype
From: Matt Bobrowski @ 2025-12-01 19:22 UTC (permalink / raw)
To: Shuran Liu
Cc: song, bpf, ast, daniel, andrii, martin.lau, eddyz87,
yonghong.song, john.fastabend, kpsingh, sdf, haoluo, jolsa,
rostedt, mhiramat, mathieu.desnoyers, linux-kernel,
linux-trace-kernel
In-Reply-To: <20251201143813.5212-1-electronlsr@gmail.com>
On Mon, Dec 01, 2025 at 10:38:11PM +0800, Shuran Liu wrote:
> Hi,
>
> this series fixes a verifier regression for bpf_d_path() introduced by
> commit 37cce22dbd51 ("bpf: verifier: Refactor helper access type
> tracking") and adds a small selftest to exercise the helper from an
> LSM program.
>
> Commit 37cce22dbd51 started distinguishing read vs write accesses
> performed by helpers. bpf_d_path()'s buffer argument was left as
> ARG_PTR_TO_MEM without MEM_WRITE, so the verifier could incorrectly
> assume that the buffer contents are unchanged across the helper call
> and base its optimizations on this wrong assumption.
>
> In practice this showed up as a misbehaving LSM BPF program that calls
> bpf_d_path() and then does a simple prefix comparison on the returned
> path: the program would sometimes take the "mismatch" branch even
> though both bytes being compared were actually equal.
FTR, I strongly encourage any new BPF LSM implementation to consider
using the newer BPF kfunc alternative instead, being
bpf_path_d_path().
> Patch 1 fixes bpf_d_path()'s helper prototype by marking the buffer
> argument as ARG_PTR_TO_MEM | MEM_WRITE, so that the verifier correctly
> models the write to the caller-provided buffer.
This is the correct thing to do, appreciate you sending through the
fix.
> Patch 2 adds a minimal selftest under tools/testing/selftests/bpf that
> hooks bprm_check_security, calls bpf_d_path() on a binary under /tmp/,
> and verifies that the prefix comparison on the returned path keeps
> working.
Makes sense to add a test for this regression, but please also see my
comments against this patch.
> On my local setup, tools/testing/selftests/bpf does not build fully
> due to unrelated tests using newer helpers. I validated this series by
> manually reproducing the issue with a small LSM program and by
> building and running only the new d_path_lsm test on kernels with and
> without patch 1 applied.
>
> Thanks,
> Shuran Liu
>
> Shuran Liu (2):
> bpf: mark bpf_d_path() buffer as writeable
> selftests/bpf: add regression test for bpf_d_path()
>
> kernel/trace/bpf_trace.c | 2 +-
> .../selftests/bpf/prog_tests/d_path_lsm.c | 27 ++++++++++++
> .../selftests/bpf/progs/d_path_lsm.bpf.c | 43 +++++++++++++++++++
> 3 files changed, 71 insertions(+), 1 deletion(-)
> create mode 100644 tools/testing/selftests/bpf/prog_tests/d_path_lsm.c
> create mode 100644 tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c
>
> --
> 2.52.0
>
^ permalink raw reply
* Re: [PATCH 2/3] kernel.h: move VERIFY_OCTAL_PERMISSIONS() to sysfs.h
From: Andy Shevchenko @ 2025-12-01 19:20 UTC (permalink / raw)
To: Petr Pavlu
Cc: Yury Norov (NVIDIA), Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Randy Dunlap, Ingo Molnar, Jani Nikula,
Joonas Lahtinen, Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
In-Reply-To: <c45058d5-d690-4731-85d1-434971c16f92@suse.com>
On Mon, Dec 01, 2025 at 08:01:23PM +0100, Petr Pavlu wrote:
> On 11/29/25 8:53 PM, Yury Norov (NVIDIA) wrote:
...
> > -#include <linux/kernel.h>
> > +#include <linux/sysfs.h>
>
> If you are removing the kernel.h include from
> include/linux/moduleparam.h, I think it would be good to update the file
> to ensure that all necessary includes are now listed directly.
>
> The following items are present in moduleparam.h:
>
> * __UNIQUE_ID(), __used(), __section(), __aligned(), __always_unused()
> -> linux/compiler.h,
> * THIS_MODULE -> linux/init.h,
> * __stringify() -> linux/stringify.h,
> * u8, s8, u16, ... -> linux/types.h,
> * static_assert() -> linux/build_bug.h,
> * VERIFY_OCTAL_PERMISSIONS() -> linux/sysfs.h,
> * ARRAY_SIZE() -> linux/array_size.h.
>
> I suggest then updating the includes in include/linux/moduleparam.h to:
>
> #include <linux/array_size.h>
> #include <linux/build_bug.h>
> #include <linux/compiler.h>
> #include <linux/init.h>
> #include <linux/stringify.h>
> #include <linux/sysfs.h>
> #include <linux/types.h>
Good point. And since we are not adding some top-level ones, this shouldn't
be worse (in terms of potential cyclic dependencies).
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply
* Re: [PATCH bpf 2/2] selftests/bpf: add regression test for bpf_d_path()
From: Matt Bobrowski @ 2025-12-01 19:16 UTC (permalink / raw)
To: Shuran Liu
Cc: song, bpf, ast, daniel, andrii, martin.lau, eddyz87,
yonghong.song, john.fastabend, kpsingh, sdf, haoluo, jolsa,
rostedt, mhiramat, mathieu.desnoyers, linux-kernel,
linux-trace-kernel, Zesen Liu, Peili Gao, Haoran Ni
In-Reply-To: <20251201143813.5212-3-electronlsr@gmail.com>
On Mon, Dec 01, 2025 at 10:38:13PM +0800, Shuran Liu wrote:
> Add a simple LSM BPF program and a corresponding test_progs test case
> to exercise bpf_d_path() and ensure that prefix comparisons on the
> returned path keep working.
>
n> The LSM program hooks bprm_check_security, calls bpf_d_path() on the
> binary being executed, and compares the returned path against the
> "/tmp/" prefix. The result is recorded in an array map.
>
> The user space test runs /tmp/bpf_d_path_test (copied from /bin/true)
> and checks that the BPF program records a successful prefix match.
>
> Without the preceding fix to bpf_d_path()'s helper prototype, the
> test can fail due to the verifier incorrectly assuming that the
> buffer contents are unchanged across the helper call and misoptimizing
> the program. With the fix applied, the test passes.
>
> Co-developed-by: Zesen Liu <ftyg@live.com>
> Signed-off-by: Zesen Liu <ftyg@live.com>
> Co-developed-by: Peili Gao <gplhust955@gmail.com>
> Signed-off-by: Peili Gao <gplhust955@gmail.com>
> Co-developed-by: Haoran Ni <haoran.ni.cs@gmail.com>
> Signed-off-by: Haoran Ni <haoran.ni.cs@gmail.com>
> Signed-off-by: Shuran Liu <electronlsr@gmail.com>
> ---
> .../selftests/bpf/prog_tests/d_path_lsm.c | 27 ++++++++++++
> .../selftests/bpf/progs/d_path_lsm.bpf.c | 43 +++++++++++++++++++
> 2 files changed, 70 insertions(+)
> create mode 100644 tools/testing/selftests/bpf/prog_tests/d_path_lsm.c
> create mode 100644 tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c
>
> diff --git a/tools/testing/selftests/bpf/prog_tests/d_path_lsm.c b/tools/testing/selftests/bpf/prog_tests/d_path_lsm.c
> new file mode 100644
> index 000000000000..92aad744ed12
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/d_path_lsm.c
I don't see why adding yet another new bpf_d_path() related test to
prog_tests is warranted here. Why not simply incorporate this
additional test case into the preexisting bpf_d_path() related
prog_tests source file i.e. tools/testing/selftests/bpf/d_path.c?
> @@ -0,0 +1,27 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +#include <test_progs.h>
> +#include "d_path_lsm.skel.h"
> +
> +void test_d_path_lsm(void)
> +{
> + struct d_path_lsm *skel = NULL;
> + int err, map_fd, key = 0, val = 0;
> +
> + skel = d_path_lsm__open_and_load();
> + if (!ASSERT_OK_PTR(skel, "open_and_load"))
> + return;
> +
> + err = d_path_lsm__attach(skel);
> + if (!ASSERT_OK(err, "attach"))
> + goto out;
> +
> + system("cp /bin/true /tmp/bpf_d_path_test 2>/dev/null || :");
> + system("/tmp/bpf_d_path_test >/dev/null 2>&1");
> +
> + map_fd = bpf_map__fd(skel->maps.result);
> + err = bpf_map_lookup_elem(map_fd, &key, &val);
> + ASSERT_OK(err, "lookup_result");
> + ASSERT_EQ(val, 1, "prefix_match");
> +out:
> + d_path_lsm__destroy(skel);
> +}
>
> diff --git a/tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c b/tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c
> new file mode 100644
> index 000000000000..36f9ff37e817
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c
> @@ -0,0 +1,43 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +#include "vmlinux.h"
> +#include <bpf/bpf_helpers.h>
> +#include <bpf/bpf_tracing.h>
> +
> +char LICENSE[] SEC("license") = "GPL";
> +
> +#define FILENAME_MAX_SIZE 256
> +#define TARGET_DIR "/tmp/"
> +#define TARGET_DIR_LEN 5
> +
> +struct {
> + __uint(type, BPF_MAP_TYPE_ARRAY);
> + __uint(max_entries, 1);
> + __type(key, int);
> + __type(value, int);
> +} result SEC(".maps");
> +
> +SEC("lsm/bprm_check_security")
> +int BPF_PROG(d_path_lsm_prog, struct linux_binprm *bprm)
> +{
> + char path[FILENAME_MAX_SIZE] = {};
> + long len;
> + int key = 0;
> + int val = 0;
> +
> + len = bpf_d_path(&bprm->file->f_path, path, sizeof(path));
> + if (len < 0)
> + return 0;
> +
> +#pragma unroll
> + for (int i = 0; i < TARGET_DIR_LEN; i++) {
> + if ((u8)path[i] != (u8)TARGET_DIR[i]) {
> + val = -1; /* mismatch */
> + bpf_map_update_elem(&result, &key, &val, BPF_ANY);
> + return 0;
> + }
> + }
> +
> + val = 1; /* prefix match */
> + bpf_map_update_elem(&result, &key, &val, BPF_ANY);
> + return 0;
Will this not flake, like, maybe a lot? Mismatches are being reported
for every non-matched prefix. Meaning, other threads that are racing
alongside your system(3) invocations and going through
security_bprm_check() could very well reset your BPF_MAP_TYPE_ARRAY
element value back to -1 before your userspace code even has a chance
to assert it? Perhaps you can make this test a little more
deterministic by filtering by the expected PID?
^ permalink raw reply
* [PATCH V3 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
From: Shivank Garg @ 2025-12-01 18:56 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, shivankg, Branden Moore
In-Reply-To: <20251201185604.210634-6-shivankg@amd.com>
When MADV_COLLAPSE is called on file-backed mappings (e.g., executable
text sections), the pages may still be dirty from recent writes.
collapse_file() will trigger async writeback and fail with
SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).
MADV_COLLAPSE is a synchronous operation where userspace expects
immediate results. If the collapse fails due to dirty pages, perform
synchronous writeback on the specific range and retry once.
This avoids spurious failures for freshly written executables while
avoiding unnecessary synchronous I/O for mappings that are already clean.
Reported-by: Branden Moore <Branden.Moore@amd.com>
Closes: https://lore.kernel.org/all/4e26fe5e-7374-467c-a333-9dd48f85d7cc@amd.com
Fixes: 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
mm/khugepaged.c | 41 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 219dfa2e523c..7a12e9ef30b4 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -22,6 +22,7 @@
#include <linux/dax.h>
#include <linux/ksm.h>
#include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
#include <asm/tlb.h>
#include "internal.h"
@@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
hend = end & HPAGE_PMD_MASK;
for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
+ bool retried = false;
int result = SCAN_FAIL;
if (!mmap_locked) {
+retry:
cond_resched();
mmap_read_lock(mm);
mmap_locked = true;
@@ -2819,6 +2822,44 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
if (!mmap_locked)
*lock_dropped = true;
+ /*
+ * If the file-backed VMA has dirty pages, the scan triggers
+ * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK.
+ * Since MADV_COLLAPSE is sync, we force sync writeback and
+ * retry once.
+ */
+ if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) {
+ /*
+ * File scan drops the lock. We must re-acquire it to
+ * safely inspect the VMA and hold the file reference.
+ */
+ if (!mmap_locked) {
+ cond_resched();
+ mmap_read_lock(mm);
+ mmap_locked = true;
+ result = hugepage_vma_revalidate(mm, addr, false, &vma, cc);
+ if (result != SCAN_SUCCEED)
+ goto handle_result;
+ }
+
+ if (!vma_is_anonymous(vma) && vma->vm_file &&
+ mapping_can_writeback(vma->vm_file->f_mapping)) {
+ struct file *file = get_file(vma->vm_file);
+ pgoff_t pgoff = linear_page_index(vma, addr);
+ loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+ loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+ mmap_read_unlock(mm);
+ mmap_locked = false;
+ *lock_dropped = true;
+ filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+ fput(file);
+ retried = true;
+ goto retry;
+ }
+ }
+
+
handle_result:
switch (result) {
case SCAN_SUCCEED:
--
2.43.0
^ permalink raw reply related
* [PATCH V3 1/2] mm/khugepaged: map dirty/writeback pages failures to EAGAIN
From: Shivank Garg @ 2025-12-01 18:56 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, shivankg, Branden Moore
In-Reply-To: <20251201185604.210634-6-shivankg@amd.com>
When collapse_file encounters dirty or writeback pages in file-backed
mappings, it currently returns SCAN_FAIL which maps to -EINVAL. This is
misleading as EINVAL suggests invalid arguments, whereas dirty/writeback
pages represent transient conditions that may resolve on retry.
Introduce SCAN_PAGE_DIRTY_OR_WRITEBACK to cover both dirty and writeback
states, mapping it to -EAGAIN. For MADV_COLLAPSE, this provides userspace
with a clear signal that retry may succeed after writeback completes.
For khugepaged, this is harmless as it will naturally revisit the range
during periodic scans after async writeback completes.
Reported-by: Branden Moore <Branden.Moore@amd.com>
Closes: https://lore.kernel.org/all/4e26fe5e-7374-467c-a333-9dd48f85d7cc@amd.com
Fixes: 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
Reviewed-by: Dev Jain <dev.jain@arm.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
include/trace/events/huge_memory.h | 3 ++-
mm/khugepaged.c | 8 +++++---
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
index 4cde53b45a85..4e41bff31888 100644
--- a/include/trace/events/huge_memory.h
+++ b/include/trace/events/huge_memory.h
@@ -37,7 +37,8 @@
EM( SCAN_PAGE_HAS_PRIVATE, "page_has_private") \
EM( SCAN_STORE_FAILED, "store_failed") \
EM( SCAN_COPY_MC, "copy_poisoned_page") \
- EMe(SCAN_PAGE_FILLED, "page_filled")
+ EM( SCAN_PAGE_FILLED, "page_filled") \
+ EMe(SCAN_PAGE_DIRTY_OR_WRITEBACK, "page_dirty_or_writeback")
#undef EM
#undef EMe
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 97d1b2824386..219dfa2e523c 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -58,6 +58,7 @@ enum scan_result {
SCAN_STORE_FAILED,
SCAN_COPY_MC,
SCAN_PAGE_FILLED,
+ SCAN_PAGE_DIRTY_OR_WRITEBACK,
};
#define CREATE_TRACE_POINTS
@@ -1967,11 +1968,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
*/
xas_unlock_irq(&xas);
filemap_flush(mapping);
- result = SCAN_FAIL;
+ result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
goto xa_unlocked;
} else if (folio_test_writeback(folio)) {
xas_unlock_irq(&xas);
- result = SCAN_FAIL;
+ result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
goto xa_unlocked;
} else if (folio_trylock(folio)) {
folio_get(folio);
@@ -2018,7 +2019,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
* folio is dirty because it hasn't been flushed
* since first write.
*/
- result = SCAN_FAIL;
+ result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
goto out_unlock;
}
@@ -2747,6 +2748,7 @@ static int madvise_collapse_errno(enum scan_result r)
case SCAN_PAGE_LRU:
case SCAN_DEL_PAGE_LRU:
case SCAN_PAGE_FILLED:
+ case SCAN_PAGE_DIRTY_OR_WRITEBACK:
return -EAGAIN;
/*
* Other: Trying again likely not to succeed / error intrinsic to
--
2.43.0
^ permalink raw reply related
* [PATCH V3 0/2] mm/khugepaged: fix dirty page handling for MADV_COLLAPSE
From: Shivank Garg @ 2025-12-01 18:56 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, shivankg
MADV_COLLAPSE on file-backed mappings fails with -EINVAL when TEXT pages
are dirty. This affects scenarios like package/container updates or
executing binaries immediately after writing them, etc.
The issue is that collapse_file() triggers async writeback and returns
SCAN_FAIL (maps to -EINVAL), expecting khugepaged to revisit later. But
MADV_COLLAPSE is synchronous and userspace expects immediate success or
a clear retry signal.
Reproduction:
- Compile or copy 2MB-aligned executable to XFS/ext4 FS
- Call MADV_COLLAPSE on .text section
- First call fails with -EINVAL (text pages dirty from copy)
- Second call succeeds (async writeback completed)
Issue Report:
https://lore.kernel.org/all/4e26fe5e-7374-467c-a333-9dd48f85d7cc@amd.com
Changelog:
V3:
- Reordered patches: Enum definition comes first as the retry logic depends on it
- Renamed SCAN_PAGE_NOT_CLEAN to SCAN_PAGE_DIRTY_OR_WRITEBACK (Dev, Lance, David)
- Changed writeback logic: Only trigger synchronous writeback and retry if the
initial collapse attempt failed specifically due to dirty/writeback pages,
rather than blindly flushing all file-backed VMAs (David)
- Added proper file reference counting (get_file/fput) around the unlock window
to prevent UAF (Lance)
V2:
- https://lore.kernel.org/all/20251120065043.41738-6-shivankg@amd.com
- Move writeback to madvise_collapse() (better abstraction, proper
mmap_lock handling and does VMA revalidation after I/O) (Lorenzo)
- Rename to SCAN_PAGE_DIRTY to SCAN_PAGE_NOT_CLEAN and extend its use
for all dirty/writeback folio cases that previously returned incorrect
results (Dev)
V1: https://lore.kernel.org/all/20251110113254.77822-1-shivankg@amd.com
Thanks,
Shivank Garg (2):
mm/khugepaged: map dirty/writeback pages failures to EAGAIN
mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
include/trace/events/huge_memory.h | 3 +-
mm/khugepaged.c | 49 ++++++++++++++++++++++++++++--
2 files changed, 48 insertions(+), 4 deletions(-)
base-commit: 2178727587e1eaa930b8266377119ed6043067df
--
2.43.0
^ permalink raw reply
* Re: [PATCH 2/3] kernel.h: move VERIFY_OCTAL_PERMISSIONS() to sysfs.h
From: Petr Pavlu @ 2025-12-01 19:01 UTC (permalink / raw)
To: Yury Norov (NVIDIA)
Cc: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Andy Shevchenko, Randy Dunlap, Ingo Molnar, Jani Nikula,
Joonas Lahtinen, Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
In-Reply-To: <20251129195304.204082-3-yury.norov@gmail.com>
On 11/29/25 8:53 PM, Yury Norov (NVIDIA) wrote:
> The macro is related to sysfs, but is defined in kernel.h. Move it to
> the proper header, and unload the generic kernel.h.
>
> Signed-off-by: Yury Norov (NVIDIA) <yury.norov@gmail.com>
> ---
> include/linux/kernel.h | 12 ------------
> include/linux/moduleparam.h | 2 +-
> include/linux/sysfs.h | 13 +++++++++++++
> 3 files changed, 14 insertions(+), 13 deletions(-)
>
> [...]
> diff --git a/include/linux/moduleparam.h b/include/linux/moduleparam.h
> index 6907aedc4f74..4e390a84a8bc 100644
> --- a/include/linux/moduleparam.h
> +++ b/include/linux/moduleparam.h
> @@ -4,7 +4,7 @@
> /* (C) Copyright 2001, 2002 Rusty Russell IBM Corporation */
> #include <linux/init.h>
> #include <linux/stringify.h>
> -#include <linux/kernel.h>
> +#include <linux/sysfs.h>
If you are removing the kernel.h include from
include/linux/moduleparam.h, I think it would be good to update the file
to ensure that all necessary includes are now listed directly.
The following items are present in moduleparam.h:
* __UNIQUE_ID(), __used(), __section(), __aligned(), __always_unused()
-> linux/compiler.h,
* THIS_MODULE -> linux/init.h,
* __stringify() -> linux/stringify.h,
* u8, s8, u16, ... -> linux/types.h,
* static_assert() -> linux/build_bug.h,
* VERIFY_OCTAL_PERMISSIONS() -> linux/sysfs.h,
* ARRAY_SIZE() -> linux/array_size.h.
I suggest then updating the includes in include/linux/moduleparam.h to:
#include <linux/array_size.h>
#include <linux/build_bug.h>
#include <linux/compiler.h>
#include <linux/init.h>
#include <linux/stringify.h>
#include <linux/sysfs.h>
#include <linux/types.h>
--
Thanks,
Petr
^ permalink raw reply
* Re: [PATCH v13 mm-new 16/16] Documentation: mm: update the admin guide for mTHP collapse
From: Randy Dunlap @ 2025-12-01 18:52 UTC (permalink / raw)
To: Nico Pache, linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, hughd, richard.weiyang, lance.yang, vbabka,
rppt, jannh, pfalcato, Bagas Sanjaya
In-Reply-To: <20251201174627.23295-17-npache@redhat.com>
On 12/1/25 9:46 AM, Nico Pache wrote:
> Now that we can collapse to mTHPs lets update the admin guide to
> reflect these changes and provide proper guidance on how to utilize it.
>
> Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
> Documentation/admin-guide/mm/transhuge.rst | 48 +++++++++++++---------
> 1 file changed, 28 insertions(+), 20 deletions(-)
>
> diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> index d396d1bfb274..87bcfa80886a 100644
> --- a/Documentation/admin-guide/mm/transhuge.rst
> +++ b/Documentation/admin-guide/mm/transhuge.rst
> @@ -63,7 +63,8 @@ often.
> THP can be enabled system wide or restricted to certain tasks or even
> memory ranges inside task's address space. Unless THP is completely
> disabled, there is ``khugepaged`` daemon that scans memory and
> -collapses sequences of basic pages into PMD-sized huge pages.
> +collapses sequences of basic pages into huge pages of either PMD size
> +or mTHP sizes, if the system is configured to do so
so.
>
> The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
> interface and using madvise(2) and prctl(2) system calls.
> @@ -219,10 +220,10 @@ this behaviour by writing 0 to shrink_underused, and enable it by writing
> echo 0 > /sys/kernel/mm/transparent_hugepage/shrink_underused
> echo 1 > /sys/kernel/mm/transparent_hugepage/shrink_underused
>
> -khugepaged will be automatically started when PMD-sized THP is enabled
> +khugepaged will be automatically started when any THP size is enabled
> (either of the per-size anon control or the top-level control are set
> to "always" or "madvise"), and it'll be automatically shutdown when
> -PMD-sized THP is disabled (when both the per-size anon control and the
> +all THP sizes are disabled (when both the per-size anon control and the
> top-level control are "never")
"never").
>
> process THP controls
[snip]
--
~Randy
^ permalink raw reply
* Re: [PATCH bpf 1/2] bpf: mark bpf_d_path() buffer as writeable
From: Matt Bobrowski @ 2025-12-01 18:48 UTC (permalink / raw)
To: Shuran Liu
Cc: song, bpf, ast, daniel, andrii, martin.lau, eddyz87,
yonghong.song, john.fastabend, kpsingh, sdf, haoluo, jolsa,
rostedt, mhiramat, mathieu.desnoyers, linux-kernel,
linux-trace-kernel, Zesen Liu, Peili Gao, Haoran Ni
In-Reply-To: <20251201143813.5212-2-electronlsr@gmail.com>
On Mon, Dec 01, 2025 at 10:38:12PM +0800, Shuran Liu wrote:
> Commit 37cce22dbd51 ("bpf: verifier: Refactor helper access type
> tracking") started distinguishing read vs write accesses performed by
> helpers.
>
> The second argument of bpf_d_path() is a pointer to a buffer that the
> helper fills with the resulting path. However, its prototype currently
> uses ARG_PTR_TO_MEM without MEM_WRITE.
>
> Before 37cce22dbd51, helper accesses were conservatively treated as
> potential writes, so this mismatch did not cause issues. Since that
> commit, the verifier may incorrectly assume that the buffer contents
> are unchanged across the helper call and base its optimizations on this
> wrong assumption. This can lead to misbehaviour in BPF programs that
> read back the buffer, such as prefix comparisons on the returned path.
>
> Fix this by marking the second argument of bpf_d_path() as
> ARG_PTR_TO_MEM | MEM_WRITE so that the verifier correctly models the
> write to the caller-provided buffer.
>
> Fixes: 37cce22dbd51 ("bpf: verifier: Refactor helper access type tracking")
> Co-developed-by: Zesen Liu <ftyg@live.com>
> Signed-off-by: Zesen Liu <ftyg@live.com>
> Co-developed-by: Peili Gao <gplhust955@gmail.com>
> Signed-off-by: Peili Gao <gplhust955@gmail.com>
> Co-developed-by: Haoran Ni <haoran.ni.cs@gmail.com>
> Signed-off-by: Haoran Ni <haoran.ni.cs@gmail.com>
> Signed-off-by: Shuran Liu <electronlsr@gmail.com>
Reviewed-by: Matt Bobrowski <mattbobrowski@google.com>
> ---
> kernel/trace/bpf_trace.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> index 4f87c16d915a..49e0bdaa7a1b 100644
> --- a/kernel/trace/bpf_trace.c
> +++ b/kernel/trace/bpf_trace.c
> @@ -965,7 +965,7 @@ static const struct bpf_func_proto bpf_d_path_proto = {
> .ret_type = RET_INTEGER,
> .arg1_type = ARG_PTR_TO_BTF_ID,
> .arg1_btf_id = &bpf_d_path_btf_ids[0],
> - .arg2_type = ARG_PTR_TO_MEM,
> + .arg2_type = ARG_PTR_TO_MEM | MEM_WRITE,
> .arg3_type = ARG_CONST_SIZE_OR_ZERO,
> .allowed = bpf_d_path_allowed,
> };
> --
> 2.52.0
>
^ permalink raw reply
* Re: [PATCH v13 mm-new 10/16] khugepaged: add per-order mTHP collapse failure statistics
From: Randy Dunlap @ 2025-12-01 18:38 UTC (permalink / raw)
To: Nico Pache, linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, hughd, richard.weiyang, lance.yang, vbabka,
rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-11-npache@redhat.com>
On 12/1/25 9:46 AM, Nico Pache wrote:
> Add three new mTHP statistics to track collapse failures for different
> orders when encountering swap PTEs, excessive none PTEs, and shared PTEs:
>
> - collapse_exceed_swap_pte: Increment when mTHP collapse fails due to swap
> PTEs
>
> - collapse_exceed_none_pte: Counts when mTHP collapse fails due to
> exceeding the none PTE threshold for the given order
>
> - collapse_exceed_shared_pte: Counts when mTHP collapse fails due to shared
> PTEs
>
> These statistics complement the existing THP_SCAN_EXCEED_* events by
> providing per-order granularity for mTHP collapse attempts. The stats are
> exposed via sysfs under
> `/sys/kernel/mm/transparent_hugepage/hugepages-*/stats/` for each
> supported hugepage size.
>
> As we currently dont support collapsing mTHPs that contain a swap or
> shared entry, those statistics keep track of how often we are
> encountering failed mTHP collapses due to these restrictions.
>
> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
> Documentation/admin-guide/mm/transhuge.rst | 24 ++++++++++++++++++++++
> include/linux/huge_mm.h | 3 +++
> mm/huge_memory.c | 7 +++++++
> mm/khugepaged.c | 16 ++++++++++++---
> 4 files changed, 47 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> index c51932e6275d..d396d1bfb274 100644
> --- a/Documentation/admin-guide/mm/transhuge.rst
> +++ b/Documentation/admin-guide/mm/transhuge.rst
> @@ -714,6 +714,30 @@ nr_anon_partially_mapped
> an anonymous THP as "partially mapped" and count it here, even though it
> is not actually partially mapped anymore.
>
> +collapse_exceed_none_pte
> + The number of collapse attempts that failed due to exceeding the
> + max_ptes_none threshold. For mTHP collapse, Currently only max_ptes_none
currently
> + values of 0 and (HPAGE_PMD_NR - 1) are supported. Any other value will
> + emit a warning and no mTHP collapse will be attempted. khugepaged will
> + try to collapse to the largest enabled (m)THP size, if it fails, it will
size. If
or
size; if
> + try the next lower enabled mTHP size. This counter records the number of
> + times a collapse attempt was skipped for exceeding the max_ptes_none
> + threshold, and khugepaged will move on to the next available mTHP size.
> +
> +collapse_exceed_swap_pte
> + The number of anonymous mTHP pte ranges which were unable to collapse due
PTE
> + to containing at least one swap PTE. Currently khugepaged does not
> + support collapsing mTHP regions that contain a swap PTE. This counter can
> + be used to monitor the number of khugepaged mTHP collapses that failed
> + due to the presence of a swap PTE.
> +
> +collapse_exceed_shared_pte
> + The number of anonymous mTHP pte ranges which were unable to collapse due
PTE
> + to containing at least one shared PTE. Currently khugepaged does not
> + support collapsing mTHP pte ranges that contain a shared PTE. This
PTE
> + counter can be used to monitor the number of khugepaged mTHP collapses
> + that failed due to the presence of a shared PTE.
> +
> As the system ages, allocating huge pages may be expensive as the
> system uses memory compaction to copy data around memory to free a
> huge page for use. There are some counters in ``/proc/vmstat`` to help
--
~Randy
^ permalink raw reply
* [PATCH v13 mm-new 16/16] Documentation: mm: update the admin guide for mTHP collapse
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato, Bagas Sanjaya
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
Now that we can collapse to mTHPs lets update the admin guide to
reflect these changes and provide proper guidance on how to utilize it.
Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
Documentation/admin-guide/mm/transhuge.rst | 48 +++++++++++++---------
1 file changed, 28 insertions(+), 20 deletions(-)
diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
index d396d1bfb274..87bcfa80886a 100644
--- a/Documentation/admin-guide/mm/transhuge.rst
+++ b/Documentation/admin-guide/mm/transhuge.rst
@@ -63,7 +63,8 @@ often.
THP can be enabled system wide or restricted to certain tasks or even
memory ranges inside task's address space. Unless THP is completely
disabled, there is ``khugepaged`` daemon that scans memory and
-collapses sequences of basic pages into PMD-sized huge pages.
+collapses sequences of basic pages into huge pages of either PMD size
+or mTHP sizes, if the system is configured to do so
The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
interface and using madvise(2) and prctl(2) system calls.
@@ -219,10 +220,10 @@ this behaviour by writing 0 to shrink_underused, and enable it by writing
echo 0 > /sys/kernel/mm/transparent_hugepage/shrink_underused
echo 1 > /sys/kernel/mm/transparent_hugepage/shrink_underused
-khugepaged will be automatically started when PMD-sized THP is enabled
+khugepaged will be automatically started when any THP size is enabled
(either of the per-size anon control or the top-level control are set
to "always" or "madvise"), and it'll be automatically shutdown when
-PMD-sized THP is disabled (when both the per-size anon control and the
+all THP sizes are disabled (when both the per-size anon control and the
top-level control are "never")
process THP controls
@@ -264,11 +265,6 @@ support the following arguments::
Khugepaged controls
-------------------
-.. note::
- khugepaged currently only searches for opportunities to collapse to
- PMD-sized THP and no attempt is made to collapse to other THP
- sizes.
-
khugepaged runs usually at low frequency so while one may not want to
invoke defrag algorithms synchronously during the page faults, it
should be worth invoking defrag at least in khugepaged. However it's
@@ -296,11 +292,11 @@ allocation failure to throttle the next allocation attempt::
The khugepaged progress can be seen in the number of pages collapsed (note
that this counter may not be an exact count of the number of pages
collapsed, since "collapsed" could mean multiple things: (1) A PTE mapping
-being replaced by a PMD mapping, or (2) All 4K physical pages replaced by
-one 2M hugepage. Each may happen independently, or together, depending on
-the type of memory and the failures that occur. As such, this value should
-be interpreted roughly as a sign of progress, and counters in /proc/vmstat
-consulted for more accurate accounting)::
+being replaced by a PMD mapping, or (2) physical pages replaced by one
+hugepage of various sizes (PMD-sized or mTHP). Each may happen independently,
+or together, depending on the type of memory and the failures that occur.
+As such, this value should be interpreted roughly as a sign of progress,
+and counters in /proc/vmstat consulted for more accurate accounting)::
/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed
@@ -308,16 +304,19 @@ for each pass::
/sys/kernel/mm/transparent_hugepage/khugepaged/full_scans
-``max_ptes_none`` specifies how many extra small pages (that are
-not already mapped) can be allocated when collapsing a group
-of small pages into one large page::
+``max_ptes_none`` specifies how many empty (none/zero) pages are allowed
+when collapsing a group of small pages into one large page::
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
-A higher value leads to use additional memory for programs.
-A lower value leads to gain less thp performance. Value of
-max_ptes_none can waste cpu time very little, you can
-ignore it.
+For PMD-sized THP collapse, this directly limits the number of empty pages
+allowed in the 2MB region. For mTHP collapse, only 0 or (HPAGE_PMD_NR - 1)
+are supported. Any other value will emit a warning and no mTHP collapse
+will be attempted.
+
+A higher value allows more empty pages, potentially leading to more memory
+usage but better THP performance. A lower value is more conservative and
+may result in fewer THP collapses.
``max_ptes_swap`` specifies how many pages can be brought in from
swap when collapsing a group of pages into a transparent huge page::
@@ -337,6 +336,15 @@ that THP is shared. Exceeding the number would block the collapse::
A higher value may increase memory footprint for some workloads.
+.. note::
+ For mTHP collapse, khugepaged does not support collapsing regions that
+ contain shared or swapped out pages, as this could lead to continuous
+ promotion to higher orders. The collapse will fail if any shared or
+ swapped PTEs are encountered during the scan.
+
+ Currently, madvise_collapse only supports collapsing to PMD-sized THPs
+ and does not attempt mTHP collapses.
+
Boot parameters
===============
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 15/16] khugepaged: run khugepaged for all orders
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
From: Baolin Wang <baolin.wang@linux.alibaba.com>
If any order (m)THP is enabled we should allow running khugepaged to
attempt scanning and collapsing mTHPs. In order for khugepaged to operate
when only mTHP sizes are specified in sysfs, we must modify the predicate
function that determines whether it ought to run to do so.
This function is currently called hugepage_pmd_enabled(), this patch
renames it to hugepage_enabled() and updates the logic to check to
determine whether any valid orders may exist which would justify
khugepaged running.
We must also update collapse_allowable_orders() to check all orders if
the vma is anonymous and the collapse is khugepaged.
After this patch khugepaged mTHP collapse is fully enabled.
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 30 ++++++++++++++++++------------
1 file changed, 18 insertions(+), 12 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index daf39aab7e86..ae524220304a 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -441,23 +441,23 @@ static inline int collapse_test_exit_or_disable(struct mm_struct *mm)
mm_flags_test(MMF_DISABLE_THP_COMPLETELY, mm);
}
-static bool hugepage_pmd_enabled(void)
+static bool hugepage_enabled(void)
{
/*
* We cover the anon, shmem and the file-backed case here; file-backed
* hugepages, when configured in, are determined by the global control.
- * Anon pmd-sized hugepages are determined by the pmd-size control.
+ * Anon hugepages are determined by its per-size mTHP control.
* Shmem pmd-sized hugepages are also determined by its pmd-size control,
* except when the global shmem_huge is set to SHMEM_HUGE_DENY.
*/
if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) &&
hugepage_global_enabled())
return true;
- if (test_bit(PMD_ORDER, &huge_anon_orders_always))
+ if (READ_ONCE(huge_anon_orders_always))
return true;
- if (test_bit(PMD_ORDER, &huge_anon_orders_madvise))
+ if (READ_ONCE(huge_anon_orders_madvise))
return true;
- if (test_bit(PMD_ORDER, &huge_anon_orders_inherit) &&
+ if (READ_ONCE(huge_anon_orders_inherit) &&
hugepage_global_enabled())
return true;
if (IS_ENABLED(CONFIG_SHMEM) && shmem_hpage_pmd_enabled())
@@ -536,8 +536,14 @@ static unsigned int collapse_max_ptes_none(unsigned int order, bool full_scan)
static unsigned long collapse_allowable_orders(struct vm_area_struct *vma,
vm_flags_t vm_flags, bool is_khugepaged)
{
+ unsigned long orders;
enum tva_type tva_flags = is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE;
- unsigned long orders = BIT(HPAGE_PMD_ORDER);
+
+ /* If khugepaged is scanning an anonymous vma, allow mTHP collapse */
+ if (is_khugepaged && vma_is_anonymous(vma))
+ orders = THP_ORDERS_ALL_ANON;
+ else
+ orders = BIT(HPAGE_PMD_ORDER);
return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders);
}
@@ -546,7 +552,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma,
vm_flags_t vm_flags)
{
if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) &&
- hugepage_pmd_enabled()) {
+ hugepage_enabled()) {
if (collapse_allowable_orders(vma, vm_flags, /*is_khugepaged=*/true))
__khugepaged_enter(vma->vm_mm);
}
@@ -2890,7 +2896,7 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,
static int khugepaged_has_work(void)
{
- return !list_empty(&khugepaged_scan.mm_head) && hugepage_pmd_enabled();
+ return !list_empty(&khugepaged_scan.mm_head) && hugepage_enabled();
}
static int khugepaged_wait_event(void)
@@ -2963,7 +2969,7 @@ static void khugepaged_wait_work(void)
return;
}
- if (hugepage_pmd_enabled())
+ if (hugepage_enabled())
wait_event_freezable(khugepaged_wait, khugepaged_wait_event());
}
@@ -2994,7 +3000,7 @@ static void set_recommended_min_free_kbytes(void)
int nr_zones = 0;
unsigned long recommended_min;
- if (!hugepage_pmd_enabled()) {
+ if (!hugepage_enabled()) {
calculate_min_free_kbytes();
goto update_wmarks;
}
@@ -3044,7 +3050,7 @@ int start_stop_khugepaged(void)
int err = 0;
mutex_lock(&khugepaged_mutex);
- if (hugepage_pmd_enabled()) {
+ if (hugepage_enabled()) {
if (!khugepaged_thread)
khugepaged_thread = kthread_run(khugepaged, NULL,
"khugepaged");
@@ -3070,7 +3076,7 @@ int start_stop_khugepaged(void)
void khugepaged_min_free_kbytes_update(void)
{
mutex_lock(&khugepaged_mutex);
- if (hugepage_pmd_enabled() && khugepaged_thread)
+ if (hugepage_enabled() && khugepaged_thread)
set_recommended_min_free_kbytes();
mutex_unlock(&khugepaged_mutex);
}
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 14/16] khugepaged: avoid unnecessary mTHP collapse attempts
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
There are cases where, if an attempted collapse fails, all subsequent
orders are guaranteed to also fail. Avoid these collapse attempts by
bailing out early.
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 35 ++++++++++++++++++++++++++++++++++-
1 file changed, 34 insertions(+), 1 deletion(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index fcb927cd5456..daf39aab7e86 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1503,9 +1503,42 @@ static int mthp_collapse(struct mm_struct *mm, unsigned long address,
ret = collapse_huge_page(mm, collapse_address, referenced,
unmapped, cc, mmap_locked,
order);
- if (ret == SCAN_SUCCEED) {
+
+ switch (ret) {
+ /* Cases were we continue to next collapse candidate */
+ case SCAN_SUCCEED:
collapsed += nr_pte_entries;
+ fallthrough;
+ case SCAN_PTE_MAPPED_HUGEPAGE:
continue;
+ /* Cases were lower orders might still succeed */
+ case SCAN_LACK_REFERENCED_PAGE:
+ case SCAN_EXCEED_NONE_PTE:
+ case SCAN_EXCEED_SWAP_PTE:
+ case SCAN_EXCEED_SHARED_PTE:
+ case SCAN_PAGE_LOCK:
+ case SCAN_PAGE_COUNT:
+ case SCAN_PAGE_LRU:
+ case SCAN_PAGE_NULL:
+ case SCAN_DEL_PAGE_LRU:
+ case SCAN_PTE_NON_PRESENT:
+ case SCAN_PTE_UFFD_WP:
+ case SCAN_ALLOC_HUGE_PAGE_FAIL:
+ goto next_order;
+ /* Cases were no further collapse is possible */
+ case SCAN_CGROUP_CHARGE_FAIL:
+ case SCAN_COPY_MC:
+ case SCAN_ADDRESS_RANGE:
+ case SCAN_NO_PTE_TABLE:
+ case SCAN_ANY_PROCESS:
+ case SCAN_VMA_NULL:
+ case SCAN_VMA_CHECK:
+ case SCAN_SCAN_ABORT:
+ case SCAN_PAGE_ANON:
+ case SCAN_PMD_MAPPED:
+ case SCAN_FAIL:
+ default:
+ return collapsed;
}
}
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 13/16] khugepaged: Introduce mTHP collapse support
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
Enable khugepaged to collapse to mTHP orders. This patch implements the
main scanning logic using a bitmap to track occupied pages and a stack
structure that allows us to find optimal collapse sizes.
Previous to this patch, PMD collapse had 3 main phases, a light weight
scanning phase (mmap_read_lock) that determines a potential PMD
collapse, a alloc phase (mmap unlocked), then finally heavier collapse
phase (mmap_write_lock).
To enabled mTHP collapse we make the following changes:
During PMD scan phase, track occupied pages in a bitmap. When mTHP
orders are enabled, we remove the restriction of max_ptes_none during the
scan phase to avoid missing potential mTHP collapse candidates. Once we
have scanned the full PMD range and updated the bitmap to track occupied
pages, we use the bitmap to find the optimal mTHP size.
Implement collapse_scan_bitmap() to perform binary recursion on the bitmap
and determine the best eligible order for the collapse. A stack structure
is used instead of traditional recursion to manage the search. The
algorithm recursively splits the bitmap into smaller chunks to find the
highest order mTHPs that satisfy the collapse criteria. We start by
attempting the PMD order, then moved on the consecutively lower orders
(mTHP collapse). The stack maintains a pair of variables (offset, order),
indicating the number of PTEs from the start of the PMD, and the order of
the potential collapse candidate.
The algorithm for consuming the bitmap works as such:
1) push (0, HPAGE_PMD_ORDER) onto the stack
2) pop the stack
3) check if the number of set bits in that (offset,order) pair
statisfy the max_ptes_none threshold for that order
4) if yes, attempt collapse
5) if no (or collapse fails), push two new stack items representing
the left and right halves of the current bitmap range, at the
next lower order
6) repeat at step (2) until stack is empty.
Below is a diagram representing the algorithm and stack items:
offset mid_offset
| |
| |
v v
____________________________________
| PTE Page Table |
--------------------------------------
<-------><------->
order-1 order-1
We currently only support mTHP collapse for max_ptes_none values of 0
and HPAGE_PMD_NR - 1. resulting in the following behavior:
- max_ptes_none=0: Never introduce new empty pages during collapse
- max_ptes_none=HPAGE_PMD_NR-1: Always try collapse to the highest
available mTHP order
Any other max_ptes_none value will emit a warning and skip mTHP collapse
attempts. There should be no behavior change for PMD collapse.
Once we determine what mTHP sizes fits best in that PMD range a collapse
is attempted. A minimum collapse order of 2 is used as this is the lowest
order supported by anon memory as defined by THP_ORDERS_ALL_ANON.
mTHP collapses reject regions containing swapped out or shared pages.
This is because adding new entries can lead to new none pages, and these
may lead to constant promotion into a higher order (m)THP. A similar
issue can occur with "max_ptes_none > HPAGE_PMD_NR/2" due to a collapse
introducing at least 2x the number of pages, and on a future scan will
satisfy the promotion condition once again. This issue is prevented via
the collapse_max_ptes_none() function which imposes the max_ptes_none
restrictions above.
Currently madv_collapse is not supported and will only attempt PMD
collapse.
We can also remove the check for is_khugepaged inside the PMD scan as
the collapse_max_ptes_none() function handles this logic now.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 183 +++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 175 insertions(+), 8 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 33b70ca070b4..fcb927cd5456 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -92,6 +92,32 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);
static struct kmem_cache *mm_slot_cache __ro_after_init;
+#define KHUGEPAGED_MIN_MTHP_ORDER 2
+/*
+ * The maximum number of mTHP ranges that can be stored on the stack.
+ * This is calculated based on the number of PTE entries in a PTE page table
+ * and the minimum mTHP order.
+ *
+ * ilog2(MAX_PTRS_PER_PTE) is log2 of the maximum number of PTE entries.
+ * This gives you the PMD_ORDER, and is needed in place of HPAGE_PMD_ORDER due
+ * to restrictions of some architectures (ie ppc64le).
+ *
+ * At most there will be 1 << (PMD_ORDER - KHUGEPAGED_MIN_MTHP_ORDER) mTHP ranges
+ */
+#define MTHP_STACK_SIZE (1UL << (ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER))
+
+/*
+ * Defines a range of PTE entries in a PTE page table which are being
+ * considered for (m)THP collapse.
+ *
+ * @offset: the offset of the first PTE entry in a PMD range.
+ * @order: the order of the PTE entries being considered for collapse.
+ */
+struct mthp_range {
+ u16 offset;
+ u8 order;
+};
+
struct collapse_control {
bool is_khugepaged;
@@ -100,6 +126,11 @@ struct collapse_control {
/* nodemask for allocation fallback */
nodemask_t alloc_nmask;
+
+ /* bitmap used for mTHP collapse */
+ DECLARE_BITMAP(mthp_bitmap, MAX_PTRS_PER_PTE);
+ DECLARE_BITMAP(mthp_bitmap_mask, MAX_PTRS_PER_PTE);
+ struct mthp_range mthp_bitmap_stack[MTHP_STACK_SIZE];
};
/**
@@ -1375,6 +1406,121 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long start_addr,
return result;
}
+static void mthp_stack_push(struct collapse_control *cc, int *stack_size,
+ u16 offset, u8 order)
+{
+ const int size = *stack_size;
+ struct mthp_range *stack = &cc->mthp_bitmap_stack[size];
+
+ VM_WARN_ON_ONCE(size >= MTHP_STACK_SIZE);
+ stack->order = order;
+ stack->offset = offset;
+ (*stack_size)++;
+}
+
+static struct mthp_range mthp_stack_pop(struct collapse_control *cc, int *stack_size)
+{
+ const int size = *stack_size;
+
+ VM_WARN_ON_ONCE(size <= 0);
+ (*stack_size)--;
+ return cc->mthp_bitmap_stack[size - 1];
+}
+
+static unsigned int mthp_nr_occupied_pte_entries(struct collapse_control *cc,
+ u16 offset, unsigned long nr_pte_entries)
+{
+ bitmap_zero(cc->mthp_bitmap_mask, HPAGE_PMD_NR);
+ bitmap_set(cc->mthp_bitmap_mask, offset, nr_pte_entries);
+ return bitmap_weight_and(cc->mthp_bitmap, cc->mthp_bitmap_mask, HPAGE_PMD_NR);
+}
+
+/*
+ * mthp_collapse() consumes the bitmap that is generated during
+ * collapse_scan_pmd() to determine what regions and mTHP orders fit best.
+ *
+ * Each bit in cc->mthp_bitmap represents a single occupied (!none/zero) page.
+ * A stack structure cc->mthp_bitmap_stack is used to check different regions
+ * of the bitmap for collapse eligibility. The stack maintains a pair of
+ * variables (offset, order), indicating the number of PTEs from the start of
+ * the PMD, and the order of the potential collapse candidate respectively. We
+ * start at the PMD order and check if it is eligible for collapse; if not, we
+ * add two entries to the stack at a lower order to represent the left and right
+ * halves of the PTE page table we are examining.
+ *
+ * offset mid_offset
+ * | |
+ * | |
+ * v v
+ * --------------------------------------
+ * | cc->mthp_bitmap |
+ * --------------------------------------
+ * <-------><------->
+ * order-1 order-1
+ *
+ * For each of these, we determine how many PTE entries are occupied in the
+ * range of PTE entries we propose to collapse, then we compare this to a
+ * threshold number of PTE entries which would need to be occupied for a
+ * collapse to be permitted at that order (accounting for max_ptes_none).
+
+ * If a collapse is permitted, we attempt to collapse the PTE range into a
+ * mTHP.
+ */
+static int mthp_collapse(struct mm_struct *mm, unsigned long address,
+ int referenced, int unmapped, struct collapse_control *cc,
+ bool *mmap_locked, unsigned long enabled_orders)
+{
+ unsigned int max_ptes_none, nr_occupied_ptes;
+ struct mthp_range range;
+ unsigned long collapse_address;
+ int collapsed = 0, stack_size = 0;
+ unsigned long nr_pte_entries;
+ u16 offset;
+ u8 order;
+
+ mthp_stack_push(cc, &stack_size, 0, HPAGE_PMD_ORDER);
+
+ while (stack_size > 0) {
+ range = mthp_stack_pop(cc, &stack_size);
+ order = range.order;
+ offset = range.offset;
+ nr_pte_entries = 1UL << order;
+
+ if (!test_bit(order, &enabled_orders))
+ goto next_order;
+
+ max_ptes_none = collapse_max_ptes_none(order, !cc->is_khugepaged);
+
+ if (max_ptes_none == -EINVAL)
+ return collapsed;
+
+ nr_occupied_ptes = mthp_nr_occupied_pte_entries(cc, offset, nr_pte_entries);
+
+ if (nr_occupied_ptes >= nr_pte_entries - max_ptes_none) {
+ int ret;
+
+ collapse_address = address + offset * PAGE_SIZE;
+ ret = collapse_huge_page(mm, collapse_address, referenced,
+ unmapped, cc, mmap_locked,
+ order);
+ if (ret == SCAN_SUCCEED) {
+ collapsed += nr_pte_entries;
+ continue;
+ }
+ }
+
+next_order:
+ if (order > KHUGEPAGED_MIN_MTHP_ORDER) {
+ const u8 next_order = order - 1;
+ const u16 mid_offset = offset + (nr_pte_entries / 2);
+
+ mthp_stack_push(cc, &stack_size, mid_offset, next_order);
+ mthp_stack_push(cc, &stack_size, offset, next_order);
+ }
+ }
+ return collapsed;
+}
+
static int collapse_scan_pmd(struct mm_struct *mm,
struct vm_area_struct *vma,
unsigned long start_addr, bool *mmap_locked,
@@ -1382,11 +1528,15 @@ static int collapse_scan_pmd(struct mm_struct *mm,
{
pmd_t *pmd;
pte_t *pte, *_pte;
+ int i;
int result = SCAN_FAIL, referenced = 0;
- int none_or_zero = 0, shared = 0;
+ int none_or_zero = 0, shared = 0, nr_collapsed = 0;
struct page *page = NULL;
+ unsigned int max_ptes_none;
struct folio *folio = NULL;
unsigned long addr;
+ unsigned long enabled_orders;
+ bool full_scan = true;
spinlock_t *ptl;
int node = NUMA_NO_NODE, unmapped = 0;
@@ -1396,22 +1546,34 @@ static int collapse_scan_pmd(struct mm_struct *mm,
if (result != SCAN_SUCCEED)
goto out;
+ bitmap_zero(cc->mthp_bitmap, HPAGE_PMD_NR);
memset(cc->node_load, 0, sizeof(cc->node_load));
nodes_clear(cc->alloc_nmask);
+
+ enabled_orders = collapse_allowable_orders(vma, vma->vm_flags, cc->is_khugepaged);
+
+ /*
+ * If PMD is the only enabled order, enforce max_ptes_none, otherwise
+ * scan all pages to populate the bitmap for mTHP collapse.
+ */
+ if (cc->is_khugepaged && enabled_orders == BIT(HPAGE_PMD_ORDER))
+ full_scan = false;
+ max_ptes_none = collapse_max_ptes_none(HPAGE_PMD_ORDER, full_scan);
+
pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl);
if (!pte) {
result = SCAN_NO_PTE_TABLE;
goto out;
}
- for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR;
- _pte++, addr += PAGE_SIZE) {
+ for (i = 0; i < HPAGE_PMD_NR; i++) {
+ _pte = pte + i;
+ addr = start_addr + i * PAGE_SIZE;
pte_t pteval = ptep_get(_pte);
if (pte_none_or_zero(pteval)) {
++none_or_zero;
if (!userfaultfd_armed(vma) &&
- (!cc->is_khugepaged ||
- none_or_zero <= khugepaged_max_ptes_none)) {
+ none_or_zero <= max_ptes_none) {
continue;
} else {
result = SCAN_EXCEED_NONE_PTE;
@@ -1479,6 +1641,8 @@ static int collapse_scan_pmd(struct mm_struct *mm,
}
}
+ /* Set bit for occupied pages */
+ bitmap_set(cc->mthp_bitmap, i, 1);
/*
* Record which node the original page is from and save this
* information to cc->node_load[].
@@ -1535,9 +1699,12 @@ static int collapse_scan_pmd(struct mm_struct *mm,
out_unmap:
pte_unmap_unlock(pte, ptl);
if (result == SCAN_SUCCEED) {
- result = collapse_huge_page(mm, start_addr, referenced,
- unmapped, cc, mmap_locked,
- HPAGE_PMD_ORDER);
+ nr_collapsed = mthp_collapse(mm, start_addr, referenced, unmapped,
+ cc, mmap_locked, enabled_orders);
+ if (nr_collapsed > 0)
+ result = SCAN_SUCCEED;
+ else
+ result = SCAN_FAIL;
}
out:
trace_mm_khugepaged_scan_pmd(mm, folio, referenced,
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 12/16] khugepaged: introduce collapse_allowable_orders helper function
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
Add collapse_allowable_orders() to generalize THP order eligibility. The
function determines which THP orders are permitted based on collapse
context (khugepaged vs madv_collapse).
This consolidates collapse configuration logic and provides a clean
interface for future mTHP collapse support where the orders may be
different.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 049da0305440..33b70ca070b4 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -501,12 +501,22 @@ static unsigned int collapse_max_ptes_none(unsigned int order, bool full_scan)
return -EINVAL;
}
+/* Check what orders are allowed based on the vma and collapse type */
+static unsigned long collapse_allowable_orders(struct vm_area_struct *vma,
+ vm_flags_t vm_flags, bool is_khugepaged)
+{
+ enum tva_type tva_flags = is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE;
+ unsigned long orders = BIT(HPAGE_PMD_ORDER);
+
+ return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders);
+}
+
void khugepaged_enter_vma(struct vm_area_struct *vma,
vm_flags_t vm_flags)
{
if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) &&
hugepage_pmd_enabled()) {
- if (thp_vma_allowable_order(vma, vm_flags, TVA_KHUGEPAGED, PMD_ORDER))
+ if (collapse_allowable_orders(vma, vm_flags, /*is_khugepaged=*/true))
__khugepaged_enter(vma->vm_mm);
}
}
@@ -2606,7 +2616,7 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,
progress++;
break;
}
- if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_KHUGEPAGED, PMD_ORDER)) {
+ if (!collapse_allowable_orders(vma, vma->vm_flags, /*is_khugepaged=*/true)) {
skip:
progress++;
continue;
@@ -2912,7 +2922,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
BUG_ON(vma->vm_start > start);
BUG_ON(vma->vm_end < end);
- if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_FORCED_COLLAPSE, PMD_ORDER))
+ if (!collapse_allowable_orders(vma, vma->vm_flags, /*is_khugepaged=*/false))
return -EINVAL;
cc = kmalloc(sizeof(*cc), GFP_KERNEL);
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 11/16] khugepaged: improve tracepoints for mTHP orders
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
Add the order to the mm_collapse_huge_page<_swapin,_isolate> tracepoints to
give better insight into what order is being operated at for.
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
include/trace/events/huge_memory.h | 34 +++++++++++++++++++-----------
mm/khugepaged.c | 9 ++++----
2 files changed, 27 insertions(+), 16 deletions(-)
diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
index 4cde53b45a85..0b8c26035aea 100644
--- a/include/trace/events/huge_memory.h
+++ b/include/trace/events/huge_memory.h
@@ -87,40 +87,44 @@ TRACE_EVENT(mm_khugepaged_scan_pmd,
TRACE_EVENT(mm_collapse_huge_page,
- TP_PROTO(struct mm_struct *mm, int isolated, int status),
+ TP_PROTO(struct mm_struct *mm, int isolated, int status, unsigned int order),
- TP_ARGS(mm, isolated, status),
+ TP_ARGS(mm, isolated, status, order),
TP_STRUCT__entry(
__field(struct mm_struct *, mm)
__field(int, isolated)
__field(int, status)
+ __field(unsigned int, order)
),
TP_fast_assign(
__entry->mm = mm;
__entry->isolated = isolated;
__entry->status = status;
+ __entry->order = order;
),
- TP_printk("mm=%p, isolated=%d, status=%s",
+ TP_printk("mm=%p, isolated=%d, status=%s order=%u",
__entry->mm,
__entry->isolated,
- __print_symbolic(__entry->status, SCAN_STATUS))
+ __print_symbolic(__entry->status, SCAN_STATUS),
+ __entry->order)
);
TRACE_EVENT(mm_collapse_huge_page_isolate,
TP_PROTO(struct folio *folio, int none_or_zero,
- int referenced, int status),
+ int referenced, int status, unsigned int order),
- TP_ARGS(folio, none_or_zero, referenced, status),
+ TP_ARGS(folio, none_or_zero, referenced, status, order),
TP_STRUCT__entry(
__field(unsigned long, pfn)
__field(int, none_or_zero)
__field(int, referenced)
__field(int, status)
+ __field(unsigned int, order)
),
TP_fast_assign(
@@ -128,26 +132,30 @@ TRACE_EVENT(mm_collapse_huge_page_isolate,
__entry->none_or_zero = none_or_zero;
__entry->referenced = referenced;
__entry->status = status;
+ __entry->order = order;
),
- TP_printk("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, status=%s",
+ TP_printk("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, status=%s order=%u",
__entry->pfn,
__entry->none_or_zero,
__entry->referenced,
- __print_symbolic(__entry->status, SCAN_STATUS))
+ __print_symbolic(__entry->status, SCAN_STATUS),
+ __entry->order)
);
TRACE_EVENT(mm_collapse_huge_page_swapin,
- TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret),
+ TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret,
+ unsigned int order),
- TP_ARGS(mm, swapped_in, referenced, ret),
+ TP_ARGS(mm, swapped_in, referenced, ret, order),
TP_STRUCT__entry(
__field(struct mm_struct *, mm)
__field(int, swapped_in)
__field(int, referenced)
__field(int, ret)
+ __field(unsigned int, order)
),
TP_fast_assign(
@@ -155,13 +163,15 @@ TRACE_EVENT(mm_collapse_huge_page_swapin,
__entry->swapped_in = swapped_in;
__entry->referenced = referenced;
__entry->ret = ret;
+ __entry->order = order;
),
- TP_printk("mm=%p, swapped_in=%d, referenced=%d, ret=%d",
+ TP_printk("mm=%p, swapped_in=%d, referenced=%d, ret=%d, order=%u",
__entry->mm,
__entry->swapped_in,
__entry->referenced,
- __entry->ret)
+ __entry->ret,
+ __entry->order)
);
TRACE_EVENT(mm_khugepaged_scan_file,
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index efb8a47af65a..049da0305440 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -734,13 +734,13 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
} else {
result = SCAN_SUCCEED;
trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
- referenced, result);
+ referenced, result, order);
return result;
}
out:
release_pte_pages(pte, _pte, compound_pagelist);
trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
- referenced, result);
+ referenced, result, order);
return result;
}
@@ -1136,7 +1136,8 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,
result = SCAN_SUCCEED;
out:
- trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, result);
+ trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, result,
+ order);
return result;
}
@@ -1360,7 +1361,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long start_addr,
WARN_ON_ONCE(*mmap_locked);
if (folio)
folio_put(folio);
- trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result);
+ trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result, order);
return result;
}
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 10/16] khugepaged: add per-order mTHP collapse failure statistics
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
Add three new mTHP statistics to track collapse failures for different
orders when encountering swap PTEs, excessive none PTEs, and shared PTEs:
- collapse_exceed_swap_pte: Increment when mTHP collapse fails due to swap
PTEs
- collapse_exceed_none_pte: Counts when mTHP collapse fails due to
exceeding the none PTE threshold for the given order
- collapse_exceed_shared_pte: Counts when mTHP collapse fails due to shared
PTEs
These statistics complement the existing THP_SCAN_EXCEED_* events by
providing per-order granularity for mTHP collapse attempts. The stats are
exposed via sysfs under
`/sys/kernel/mm/transparent_hugepage/hugepages-*/stats/` for each
supported hugepage size.
As we currently dont support collapsing mTHPs that contain a swap or
shared entry, those statistics keep track of how often we are
encountering failed mTHP collapses due to these restrictions.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
Documentation/admin-guide/mm/transhuge.rst | 24 ++++++++++++++++++++++
include/linux/huge_mm.h | 3 +++
mm/huge_memory.c | 7 +++++++
mm/khugepaged.c | 16 ++++++++++++---
4 files changed, 47 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
index c51932e6275d..d396d1bfb274 100644
--- a/Documentation/admin-guide/mm/transhuge.rst
+++ b/Documentation/admin-guide/mm/transhuge.rst
@@ -714,6 +714,30 @@ nr_anon_partially_mapped
an anonymous THP as "partially mapped" and count it here, even though it
is not actually partially mapped anymore.
+collapse_exceed_none_pte
+ The number of collapse attempts that failed due to exceeding the
+ max_ptes_none threshold. For mTHP collapse, Currently only max_ptes_none
+ values of 0 and (HPAGE_PMD_NR - 1) are supported. Any other value will
+ emit a warning and no mTHP collapse will be attempted. khugepaged will
+ try to collapse to the largest enabled (m)THP size, if it fails, it will
+ try the next lower enabled mTHP size. This counter records the number of
+ times a collapse attempt was skipped for exceeding the max_ptes_none
+ threshold, and khugepaged will move on to the next available mTHP size.
+
+collapse_exceed_swap_pte
+ The number of anonymous mTHP pte ranges which were unable to collapse due
+ to containing at least one swap PTE. Currently khugepaged does not
+ support collapsing mTHP regions that contain a swap PTE. This counter can
+ be used to monitor the number of khugepaged mTHP collapses that failed
+ due to the presence of a swap PTE.
+
+collapse_exceed_shared_pte
+ The number of anonymous mTHP pte ranges which were unable to collapse due
+ to containing at least one shared PTE. Currently khugepaged does not
+ support collapsing mTHP pte ranges that contain a shared PTE. This
+ counter can be used to monitor the number of khugepaged mTHP collapses
+ that failed due to the presence of a shared PTE.
+
As the system ages, allocating huge pages may be expensive as the
system uses memory compaction to copy data around memory to free a
huge page for use. There are some counters in ``/proc/vmstat`` to help
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index f93365e182b4..1082b78e794d 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -144,6 +144,9 @@ enum mthp_stat_item {
MTHP_STAT_SPLIT_DEFERRED,
MTHP_STAT_NR_ANON,
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED,
+ MTHP_STAT_COLLAPSE_EXCEED_SWAP,
+ MTHP_STAT_COLLAPSE_EXCEED_NONE,
+ MTHP_STAT_COLLAPSE_EXCEED_SHARED,
__MTHP_STAT_COUNT
};
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index c1e1e91b0e61..b4d9b3ac9a7c 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -639,6 +639,10 @@ DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);
DEFINE_MTHP_STAT_ATTR(split_deferred, MTHP_STAT_SPLIT_DEFERRED);
DEFINE_MTHP_STAT_ATTR(nr_anon, MTHP_STAT_NR_ANON);
DEFINE_MTHP_STAT_ATTR(nr_anon_partially_mapped, MTHP_STAT_NR_ANON_PARTIALLY_MAPPED);
+DEFINE_MTHP_STAT_ATTR(collapse_exceed_swap_pte, MTHP_STAT_COLLAPSE_EXCEED_SWAP);
+DEFINE_MTHP_STAT_ATTR(collapse_exceed_none_pte, MTHP_STAT_COLLAPSE_EXCEED_NONE);
+DEFINE_MTHP_STAT_ATTR(collapse_exceed_shared_pte, MTHP_STAT_COLLAPSE_EXCEED_SHARED);
+
static struct attribute *anon_stats_attrs[] = {
&anon_fault_alloc_attr.attr,
@@ -655,6 +659,9 @@ static struct attribute *anon_stats_attrs[] = {
&split_deferred_attr.attr,
&nr_anon_attr.attr,
&nr_anon_partially_mapped_attr.attr,
+ &collapse_exceed_swap_pte_attr.attr,
+ &collapse_exceed_none_pte_attr.attr,
+ &collapse_exceed_shared_pte_attr.attr,
NULL,
};
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index b2ea56c9bb42..efb8a47af65a 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -604,7 +604,9 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
continue;
} else {
result = SCAN_EXCEED_NONE_PTE;
- count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
+ if (!is_mthp_order(order))
+ count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
+ count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_NONE);
goto out;
}
}
@@ -634,10 +636,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
* shared may cause a future higher order collapse on a
* rescan of the same range.
*/
- if (is_mthp_order(order) || (cc->is_khugepaged &&
- shared > khugepaged_max_ptes_shared)) {
+ if (is_mthp_order(order)) {
+ result = SCAN_EXCEED_SHARED_PTE;
+ count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_SHARED);
+ goto out;
+ }
+
+ if (cc->is_khugepaged &&
+ shared > khugepaged_max_ptes_shared) {
result = SCAN_EXCEED_SHARED_PTE;
count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
+ count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_SHARED);
goto out;
}
}
@@ -1086,6 +1095,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,
* range.
*/
if (is_mthp_order(order)) {
+ count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_SWAP);
pte_unmap(pte);
mmap_read_unlock(mm);
result = SCAN_EXCEED_SWAP_PTE;
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 09/16] khugepaged: skip collapsing mTHP to smaller orders
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
khugepaged may try to collapse a mTHP to a smaller mTHP, resulting in
some pages being unmapped. Skip these cases until we have a way to check
if its ok to collapse to a smaller mTHP size (like in the case of a
partially mapped folio).
This patch is inspired by Dev Jain's work on khugepaged mTHP support [1].
[1] https://lore.kernel.org/lkml/20241216165105.56185-11-dev.jain@arm.com/
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: David Hildenbrand <david@redhat.com>
Co-developed-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index b41f68bbac80..b2ea56c9bb42 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -641,6 +641,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
goto out;
}
}
+ /*
+ * TODO: In some cases of partially-mapped folios, we'd actually
+ * want to collapse.
+ */
+ if (is_mthp_order(order) && folio_order(folio) >= order) {
+ result = SCAN_PTE_MAPPED_HUGEPAGE;
+ goto out;
+ }
if (folio_test_large(folio)) {
struct folio *f;
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 08/16] khugepaged: generalize collapse_huge_page for mTHP collapse
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
Pass an order and offset to collapse_huge_page to support collapsing anon
memory to arbitrary orders within a PMD. order indicates what mTHP size we
are attempting to collapse to, and offset indicates were in the PMD to
start the collapse attempt.
For non-PMD collapse we must leave the anon VMA write locked until after
we collapse the mTHP-- in the PMD case all the pages are isolated, but in
the mTHP case this is not true, and we must keep the lock to prevent
changes to the VMA from occurring.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 112 ++++++++++++++++++++++++++++++------------------
1 file changed, 71 insertions(+), 41 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index f425238d5d4f..b41f68bbac80 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1155,45 +1155,54 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,
return SCAN_SUCCEED;
}
-static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
- int referenced, int unmapped,
- struct collapse_control *cc)
+static int collapse_huge_page(struct mm_struct *mm, unsigned long start_addr,
+ int referenced, int unmapped, struct collapse_control *cc,
+ bool *mmap_locked, unsigned int order)
{
LIST_HEAD(compound_pagelist);
pmd_t *pmd, _pmd;
- pte_t *pte;
+ pte_t *pte = NULL;
pgtable_t pgtable;
struct folio *folio;
spinlock_t *pmd_ptl, *pte_ptl;
int result = SCAN_FAIL;
struct vm_area_struct *vma;
struct mmu_notifier_range range;
+ bool anon_vma_locked = false;
+ const unsigned long nr_pages = 1UL << order;
+ const unsigned long pmd_address = start_addr & HPAGE_PMD_MASK;
- VM_BUG_ON(address & ~HPAGE_PMD_MASK);
+ VM_WARN_ON_ONCE(pmd_address & ~HPAGE_PMD_MASK);
/*
* Before allocating the hugepage, release the mmap_lock read lock.
* The allocation can take potentially a long time if it involves
* sync compaction, and we do not need to hold the mmap_lock during
* that. We will recheck the vma after taking it again in write mode.
+ * If collapsing mTHPs we may have already released the read_lock.
*/
- mmap_read_unlock(mm);
+ if (*mmap_locked) {
+ mmap_read_unlock(mm);
+ *mmap_locked = false;
+ }
- result = alloc_charge_folio(&folio, mm, cc, HPAGE_PMD_ORDER);
+ result = alloc_charge_folio(&folio, mm, cc, order);
if (result != SCAN_SUCCEED)
goto out_nolock;
mmap_read_lock(mm);
- result = hugepage_vma_revalidate(mm, address, true, &vma, cc,
- HPAGE_PMD_ORDER);
+ *mmap_locked = true;
+ result = hugepage_vma_revalidate(mm, pmd_address, true, &vma, cc, order);
if (result != SCAN_SUCCEED) {
mmap_read_unlock(mm);
+ *mmap_locked = false;
goto out_nolock;
}
- result = find_pmd_or_thp_or_none(mm, address, &pmd);
+ result = find_pmd_or_thp_or_none(mm, pmd_address, &pmd);
if (result != SCAN_SUCCEED) {
mmap_read_unlock(mm);
+ *mmap_locked = false;
goto out_nolock;
}
@@ -1203,13 +1212,16 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
* released when it fails. So we jump out_nolock directly in
* that case. Continuing to collapse causes inconsistency.
*/
- result = __collapse_huge_page_swapin(mm, vma, address, pmd,
- referenced, HPAGE_PMD_ORDER);
- if (result != SCAN_SUCCEED)
+ result = __collapse_huge_page_swapin(mm, vma, start_addr, pmd,
+ referenced, order);
+ if (result != SCAN_SUCCEED) {
+ *mmap_locked = false;
goto out_nolock;
+ }
}
mmap_read_unlock(mm);
+ *mmap_locked = false;
/*
* Prevent all access to pagetables with the exception of
* gup_fast later handled by the ptep_clear_flush and the VM
@@ -1219,20 +1231,20 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
* mmap_lock.
*/
mmap_write_lock(mm);
- result = hugepage_vma_revalidate(mm, address, true, &vma, cc,
- HPAGE_PMD_ORDER);
+ result = hugepage_vma_revalidate(mm, pmd_address, true, &vma, cc, order);
if (result != SCAN_SUCCEED)
goto out_up_write;
/* check if the pmd is still valid */
vma_start_write(vma);
- result = check_pmd_still_valid(mm, address, pmd);
+ result = check_pmd_still_valid(mm, pmd_address, pmd);
if (result != SCAN_SUCCEED)
goto out_up_write;
anon_vma_lock_write(vma->anon_vma);
+ anon_vma_locked = true;
- mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address,
- address + HPAGE_PMD_SIZE);
+ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, start_addr,
+ start_addr + (PAGE_SIZE << order));
mmu_notifier_invalidate_range_start(&range);
pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */
@@ -1244,24 +1256,21 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
* Parallel GUP-fast is fine since GUP-fast will back off when
* it detects PMD is changed.
*/
- _pmd = pmdp_collapse_flush(vma, address, pmd);
+ _pmd = pmdp_collapse_flush(vma, pmd_address, pmd);
spin_unlock(pmd_ptl);
mmu_notifier_invalidate_range_end(&range);
tlb_remove_table_sync_one();
- pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl);
+ pte = pte_offset_map_lock(mm, &_pmd, start_addr, &pte_ptl);
if (pte) {
- result = __collapse_huge_page_isolate(vma, address, pte, cc,
- HPAGE_PMD_ORDER,
- &compound_pagelist);
+ result = __collapse_huge_page_isolate(vma, start_addr, pte, cc,
+ order, &compound_pagelist);
spin_unlock(pte_ptl);
} else {
result = SCAN_NO_PTE_TABLE;
}
if (unlikely(result != SCAN_SUCCEED)) {
- if (pte)
- pte_unmap(pte);
spin_lock(pmd_ptl);
BUG_ON(!pmd_none(*pmd));
/*
@@ -1271,21 +1280,21 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
*/
pmd_populate(mm, pmd, pmd_pgtable(_pmd));
spin_unlock(pmd_ptl);
- anon_vma_unlock_write(vma->anon_vma);
goto out_up_write;
}
/*
- * All pages are isolated and locked so anon_vma rmap
- * can't run anymore.
+ * For PMD collapse all pages are isolated and locked so anon_vma
+ * rmap can't run anymore. For mTHP collapse we must hold the lock
*/
- anon_vma_unlock_write(vma->anon_vma);
+ if (!is_mthp_order(order)) {
+ anon_vma_unlock_write(vma->anon_vma);
+ anon_vma_locked = false;
+ }
result = __collapse_huge_page_copy(pte, folio, pmd, _pmd,
- vma, address, pte_ptl,
- HPAGE_PMD_ORDER,
- &compound_pagelist);
- pte_unmap(pte);
+ vma, start_addr, pte_ptl,
+ order, &compound_pagelist);
if (unlikely(result != SCAN_SUCCEED))
goto out_up_write;
@@ -1295,20 +1304,42 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
* write.
*/
__folio_mark_uptodate(folio);
- pgtable = pmd_pgtable(_pmd);
+ if (is_mthp_order(order)) { /* mTHP collapse */
+ pte_t mthp_pte = mk_pte(folio_page(folio, 0), vma->vm_page_prot);
- spin_lock(pmd_ptl);
- BUG_ON(!pmd_none(*pmd));
- pgtable_trans_huge_deposit(mm, pmd, pgtable);
- map_anon_folio_pmd_nopf(folio, pmd, vma, address);
+ mthp_pte = maybe_mkwrite(pte_mkdirty(mthp_pte), vma);
+ spin_lock(pmd_ptl);
+ WARN_ON_ONCE(!pmd_none(*pmd));
+ folio_ref_add(folio, nr_pages - 1);
+ folio_add_new_anon_rmap(folio, vma, start_addr, RMAP_EXCLUSIVE);
+ folio_add_lru_vma(folio, vma);
+ set_ptes(vma->vm_mm, start_addr, pte, mthp_pte, nr_pages);
+ update_mmu_cache_range(NULL, vma, start_addr, pte, nr_pages);
+
+ smp_wmb(); /* make PTEs visible before PMD. See pmd_install() */
+ pmd_populate(mm, pmd, pmd_pgtable(_pmd));
+ } else { /* PMD collapse */
+ pgtable = pmd_pgtable(_pmd);
+
+ spin_lock(pmd_ptl);
+ WARN_ON_ONCE(!pmd_none(*pmd));
+ pgtable_trans_huge_deposit(mm, pmd, pgtable);
+ map_anon_folio_pmd_nopf(folio, pmd, vma, pmd_address);
+ }
spin_unlock(pmd_ptl);
folio = NULL;
result = SCAN_SUCCEED;
out_up_write:
+ if (anon_vma_locked)
+ anon_vma_unlock_write(vma->anon_vma);
+ if (pte)
+ pte_unmap(pte);
mmap_write_unlock(mm);
+ *mmap_locked = false;
out_nolock:
+ WARN_ON_ONCE(*mmap_locked);
if (folio)
folio_put(folio);
trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result);
@@ -1476,9 +1507,8 @@ static int collapse_scan_pmd(struct mm_struct *mm,
pte_unmap_unlock(pte, ptl);
if (result == SCAN_SUCCEED) {
result = collapse_huge_page(mm, start_addr, referenced,
- unmapped, cc);
- /* collapse_huge_page will return with the mmap_lock released */
- *mmap_locked = false;
+ unmapped, cc, mmap_locked,
+ HPAGE_PMD_ORDER);
}
out:
trace_mm_khugepaged_scan_pmd(mm, folio, referenced,
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 07/16] khugepaged: introduce collapse_max_ptes_none helper function
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
The current mechanism for determining mTHP collapse scales the
khugepaged_max_ptes_none value based on the target order. This
introduces an undesirable feedback loop, or "creep", when max_ptes_none
is set to a value greater than HPAGE_PMD_NR / 2.
With this configuration, a successful collapse to order N will populate
enough pages to satisfy the collapse condition on order N+1 on the next
scan. This leads to unnecessary work and memory churn.
To fix this issue introduce a helper function that will limit mTHP
collapse support to two max_ptes_none values, 0 and HPAGE_PMD_NR - 1.
This effectively supports two modes:
- max_ptes_none=0: never introduce new none-pages for mTHP collapse.
- max_ptes_none=511 (on 4k pagesz): Always collapse to the highest
available mTHP order.
This removes the possiblilty of "creep", while not modifying any uAPI
expectations. A warning will be emitted if any non-supported
max_ptes_none value is configured with mTHP enabled.
The limits can be ignored by passing full_scan=true, this is useful for
madvise_collapse (which ignores limits), or in the case of
collapse_scan_pmd(), allows the full PMD to be scanned when mTHP
collapse is available.
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 42 insertions(+), 1 deletion(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 8dab49c53128..f425238d5d4f 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -463,6 +463,44 @@ void __khugepaged_enter(struct mm_struct *mm)
wake_up_interruptible(&khugepaged_wait);
}
+/**
+ * collapse_max_ptes_none - Calculate maximum allowed empty PTEs for collapse
+ * @order: The folio order being collapsed to
+ * @full_scan: Whether this is a full scan (ignore limits)
+ *
+ * For madvise-triggered collapses (full_scan=true), all limits are bypassed
+ * and allow up to HPAGE_PMD_NR - 1 empty PTEs.
+ *
+ * For PMD-sized collapses (order == HPAGE_PMD_ORDER), use the configured
+ * khugepaged_max_ptes_none value.
+ *
+ * For mTHP collapses, we currently only support khugepaged_max_pte_none values
+ * of 0 or (HPAGE_PMD_NR - 1). Any other value will emit a warning and no mTHP
+ * collapse will be attempted
+ *
+ * Return: Maximum number of empty PTEs allowed for the collapse operation
+ */
+static unsigned int collapse_max_ptes_none(unsigned int order, bool full_scan)
+{
+ /* ignore max_ptes_none limits */
+ if (full_scan)
+ return HPAGE_PMD_NR - 1;
+
+ if (!is_mthp_order(order))
+ return khugepaged_max_ptes_none;
+
+ /* Zero/non-present collapse disabled. */
+ if (!khugepaged_max_ptes_none)
+ return 0;
+
+ if (khugepaged_max_ptes_none == HPAGE_PMD_NR - 1)
+ return (1 << order) - 1;
+
+ pr_warn_once("mTHP collapse only supports max_ptes_none values of 0 or %d\n",
+ HPAGE_PMD_NR - 1);
+ return -EINVAL;
+}
+
void khugepaged_enter_vma(struct vm_area_struct *vma,
vm_flags_t vm_flags)
{
@@ -550,7 +588,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
pte_t *_pte;
int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0;
const unsigned long nr_pages = 1UL << order;
- int max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order);
+ int max_ptes_none = collapse_max_ptes_none(order, !cc->is_khugepaged);
+
+ if (max_ptes_none == -EINVAL)
+ goto out;
for (_pte = pte; _pte < pte + nr_pages;
_pte++, addr += PAGE_SIZE) {
--
2.51.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox