* Re: 答复: [External Mail]Re: [PATCH v4] dma-buf: add some tracepoints to debug.
From: Christian König @ 2025-12-01 8:24 UTC (permalink / raw)
To: 高翔, Barry Song, Xiang Gao
Cc: sumit.semwal@linaro.org, rostedt@goodmis.org, mhiramat@kernel.org,
linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org,
linux-kernel@vger.kernel.org, mathieu.desnoyers@efficios.com,
dhowells@redhat.com, kuba@kernel.org, brauner@kernel.org,
akpm@linux-foundation.org, linux-trace-kernel@vger.kernel.org
In-Reply-To: <d7269943f2c742bdb4d019ec074e5a8c@xiaomi.com>
On 12/1/25 02:44, 高翔 wrote:
>> BTW We don't have any identification for an attachment, so when multiple devices attach to the same DMA-buf the trace would be quite meaningless. We should probably fix that.
> At present, from the perspective of processes, trace still makes sense.
Yeah, it probably do in most use cases. I just want to cover all use cases.
Maybe just use %p to print the attachment pointer until we have something better.
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *发件人:* 高翔
> *发送时间:* 2025年11月28日 21:14:14
> *收件人:* Christian König; Barry Song; Xiang Gao
> *抄送:* sumit.semwal@linaro.org; rostedt@goodmis.org; mhiramat@kernel.org; linux-media@vger.kernel.org; dri-devel@lists.freedesktop.org; linux-kernel@vger.kernel.org; mathieu.desnoyers@efficios.com; dhowells@redhat.com; kuba@kernel.org; brauner@kernel.org; akpm@linux-foundation.org; linux-trace-kernel@vger.kernel.org
> *主题:* 答复: [External Mail]Re: [PATCH v4] dma-buf: add some tracepoints to debug.
>
>
>> Thinking more about it I was wondering if we really need the userspace assigned name in the tracepoint in the first place.
>> The ino should be sufficient to uniquely identify each dma-buf, tracing the name should only be printed if it's changing or otherwise we spam the tracelog quite a bit with it.
>
> Many threads use dmabuf "qcom,system". If there is a userspace name, we can immediately find the dmabuf I'm using. It might indeed not be necessary in other scenarios.
The DMA-buf "name" is an userspace assigned name and *not* useable for what you suggest here.
E.g. it is usually something like "texture", "shader", "dummy", "global"....
Could be that you have a driver or use case where it is something more specific and we can keep it where it makes sense for you, but I strongly suggest to always print the ino first since that one is guaranteed to be unique.
Regards,
Christian.
>
>
>> BTW We don't have any identification for an attachment, so when multiple devices attach to the same DMA-buf the trace would be quite meaningless. We should probably fix that.
>
> This requires some thought.
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *发件人:* Christian König <christian.koenig@amd.com>
> *发送时间:* 2025年11月28日 17:39:31
> *收件人:* Barry Song; Xiang Gao
> *抄送:* sumit.semwal@linaro.org; rostedt@goodmis.org; mhiramat@kernel.org; linux-media@vger.kernel.org; dri-devel@lists.freedesktop.org; linux-kernel@vger.kernel.org; mathieu.desnoyers@efficios.com; dhowells@redhat.com; kuba@kernel.org; brauner@kernel.org; akpm@linux-foundation.org; linux-trace-kernel@vger.kernel.org; 高翔
> *主题:* [External Mail]Re: [PATCH v4] dma-buf: add some tracepoints to debug.
>
> [外部邮件] 此邮件来源于小米公司外部，请谨慎处理。若对邮件安全性存疑，请将邮件转发给misec@xiaomi.com进行反馈
>
> On 11/28/25 10:31, Barry Song wrote:
>> On Fri, Nov 28, 2025 at 4:53 PM Xiang Gao <gxxa03070307@gmail.com> wrote:
>>>
>>> From: gaoxiang17 <gaoxiang17@xiaomi.com>
>>>
>>> I want to track the status of dmabuf in real time in the production environment.
>>> But now we can only check it by traversing the fd in the process or dmabuf_list.
>>
>> might be:
>>
>> Since we can only inspect dmabuf by iterating over process FDs or the
>> dmabuf_list, we need to add our own tracepoints to track its status in
>> real time in production.
>>
>>>
>>> For example:
>>> binder:3031_4-18348 [005] ...1. 545.568275: dma_buf_export: exp_name=qcom,system name=(null) size=217088 ino=2750
>>> binder:3031_4-18348 [005] ...1. 545.568284: dma_buf_fd: exp_name=qcom,system name=(null) size=217088 ino=2750 fd=8
>>> binder:3031_4-18348 [005] ...1. 545.568399: dma_buf_mmap_internal: exp_name=qcom,system name=qcom,system size=28672 ino=2751
>>> kworker/5:1-130 [005] ...1. 545.570193: dma_buf_put: exp_name=qcom,system name=ab size=28672 ino=2751
>>> RenderThread-18891 [005] ...1. 545.602994: dma_buf_get: exp_name=qcom,system name=ab size=217088 ino=2750 fd=1087
>>> RenderThread-9409 [000] .n.1. 545.745004: dma_buf_dynamic_attach: exp_name=qcom,system name=ab size=217088 ino=2750 is_dynamic=0 dev_name=kgsl-3d0
>>> RenderThread-9409 [005] ...1. 545.747802: dma_buf_detach: exp_name=qcom,system name=bq size=12771328 ino=2764 is_dynamic=0 dev_name=kgsl-3d0
>>>
>>> Signed-off-by: Xiang Gao <gaoxiang17@xiaomi.com>
>>> ---
>>> drivers/dma-buf/dma-buf.c | 48 +++++++++-
>>> include/trace/events/dma_buf.h | 160 +++++++++++++++++++++++++++++++++
>>> 2 files changed, 207 insertions(+), 1 deletion(-)
>>> create mode 100644 include/trace/events/dma_buf.h
>>>
>>> diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
>>> index 2bcf9ceca997..6e04e12f149e 100644
>>> --- a/drivers/dma-buf/dma-buf.c
>>> +++ b/drivers/dma-buf/dma-buf.c
>>> @@ -35,6 +35,9 @@
>>>
>>> #include "dma-buf-sysfs-stats.h"
>>>
>>> +#define CREATE_TRACE_POINTS
>>> +#include <trace/events/dma_buf.h>
>>> +
>>> static inline int is_dma_buf_file(struct file *);
>>>
>>> static DEFINE_MUTEX(dmabuf_list_mutex);
>>> @@ -220,6 +223,11 @@ static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma)
>>> dmabuf->size >> PAGE_SHIFT)
>>> return -EINVAL;
>>>
>>> + if (trace_dma_buf_mmap_internal_enabled()) {
>>> + guard(spinlock)(&dmabuf->name_lock);
>>> + trace_dma_buf_mmap_internal(dmabuf);
>>> + }
>>> +
>>> return dmabuf->ops->mmap(dmabuf, vma);
>>> }
>>>
>>> @@ -745,6 +753,11 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
>>>
>>> __dma_buf_list_add(dmabuf);
>>>
>>> + if (trace_dma_buf_export_enabled()) {
>>> + guard(spinlock)(&dmabuf->name_lock);
>>> + trace_dma_buf_export(dmabuf);
>>> + }
>>> +
>>
>> perhaps a macro similar to the one below
>>
>> #define DMA_BUF_TRACE(FUNC, ...) \
>> do { \
>> if (FUNC##_enabled()) { \
>> guard(spinlock)(&dmabuf->name_lock); \
>> FUNC(__VA_ARGS__); \
>> } \
>> } while (0)
>>
>>
>> This would help us avoid duplicating a lot of code.
>>
>> could be:
>> DMA_BUF_TRACE(trace_dma_buf_put, dmabuf);
>> DMA_BUF_TRACE(trace_dma_buf_attach, dmabuf, dev);
>>
>> ?
>
> Thinking more about it I was wondering if we really need the userspace assigned name in the tracepoint in the first place.
>
> The ino should be sufficient to uniquely identify each dma-buf, tracing the name should only be printed if it's changing or otherwise we spam the tracelog quite a bit with it.
>
> BTW We don't have any identification for an attachment, so when multiple devices attach to the same DMA-buf the trace would be quite meaningless. We should probably fix that.
>
> Regards,
> Christian.
>
^ permalink raw reply
* Re: [PATCH 1/3] kernel.h: drop STACK_MAGIC macro
From: Christophe Leroy (CS GROUP) @ 2025-12-01 9:38 UTC (permalink / raw)
To: Yury Norov (NVIDIA), Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Andy Shevchenko, Randy Dunlap, Ingo Molnar,
Jani Nikula, Joonas Lahtinen, Rodrigo Vivi, Tvrtko Ursulin,
Petr Pavlu, Daniel Gomez, Greg Kroah-Hartman, Rafael J. Wysocki,
Danilo Krummrich, Andrew Morton, linux-kernel, intel-gfx,
dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <20251129195304.204082-2-yury.norov@gmail.com>
Le 29/11/2025 à 20:53, Yury Norov (NVIDIA) a écrit :
> The macro is only used by i915. Move it to a local header and drop from
> the kernel.h.
At the begining of the git history we have:
$ git grep STACK_MAGIC 1da177e4c3f41
1da177e4c3f41:arch/h8300/kernel/traps.c: if (STACK_MAGIC !=
*(unsigned long *)((unsigned long)current+PAGE_SIZE))
1da177e4c3f41:arch/m68k/mac/macints.c: if (STACK_MAGIC !=
*(unsigned long *)current->kernel_stack_page)
1da177e4c3f41:include/linux/kernel.h:#define STACK_MAGIC 0xdeadbeef
Would be good to know the history of its usage over time.
I see:
- Removed from m68k by 3cd53b14e7c4 ("m68k/mac: Improve NMI handler")
- Removed from h8300 by 1c4b5ecb7ea1 ("remove the h8300 architecture")
- Started being used in i915 selftest by 250f8c8140ac ("drm/i915/gtt:
Read-only pages for insert_entries on bdw+")
Christophe
>
> Signed-off-by: Yury Norov (NVIDIA) <yury.norov@gmail.com>
> ---
> drivers/gpu/drm/i915/i915_utils.h | 2 ++
> include/linux/kernel.h | 2 --
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/i915_utils.h b/drivers/gpu/drm/i915/i915_utils.h
> index a0c892e4c40d..6c197e968305 100644
> --- a/drivers/gpu/drm/i915/i915_utils.h
> +++ b/drivers/gpu/drm/i915/i915_utils.h
> @@ -32,6 +32,8 @@
> #include <linux/workqueue.h>
> #include <linux/sched/clock.h>
>
> +#define STACK_MAGIC 0xdeadbeef
> +
> #ifdef CONFIG_X86
> #include <asm/hypervisor.h>
> #endif
> diff --git a/include/linux/kernel.h b/include/linux/kernel.h
> index 5b46924fdff5..61d63c57bc2d 100644
> --- a/include/linux/kernel.h
> +++ b/include/linux/kernel.h
> @@ -40,8 +40,6 @@
>
> #include <uapi/linux/kernel.h>
>
> -#define STACK_MAGIC 0xdeadbeef
> -
> struct completion;
> struct user;
>
^ permalink raw reply
* Re: [PATCH 3/3] tracing: move tracing declarations from kernel.h to a dedicated header
From: david laight @ 2025-12-01 10:16 UTC (permalink / raw)
To: Andy Shevchenko
Cc: Yury Norov, Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Randy Dunlap, Ingo Molnar, Jani Nikula, Joonas Lahtinen,
Rodrigo Vivi, Tvrtko Ursulin, Petr Pavlu, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
In-Reply-To: <aS0CgnvRfQtam0uU@smile.fi.intel.com>
On Mon, 1 Dec 2025 04:50:42 +0200
Andy Shevchenko <andriy.shevchenko@linux.intel.com> wrote:
> On Sun, Nov 30, 2025 at 11:09:25PM +0000, david laight wrote:
> > On Sun, 30 Nov 2025 21:44:46 +0200
> > Andy Shevchenko <andriy.shevchenko@linux.intel.com> wrote:
>
> ...
> kernel.h elimination (in the form it exists right now) is very fruitful.
> However, you may help with the (say) ioctl.h or whatever you consider
> really fruitful, we all will thank you (no jokes).
>
This is the first #include path for ioctl.h
In file included from ../include/asm-generic/ioctl.h:5,
from ./arch/x86/include/generated/uapi/asm/ioctl.h:1,
from ../include/uapi/linux/ioctl.h:5,
from ../include/uapi/linux/random.h:12,
from ../include/linux/random.h:10,
from ../include/linux/nodemask.h:94,
from ../include/linux/numa.h:6,
from ../include/linux/cpumask.h:17,
from ../arch/x86/include/asm/paravirt.h:21,
from ../arch/x86/include/asm/irqflags.h:102,
from ../include/linux/irqflags.h:18,
from ../include/linux/spinlock.h:59,
from ../include/linux/swait.h:7,
from ../include/linux/completion.h:12,
from ../include/linux/crypto.h:15,
from ../arch/x86/kernel/asm-offsets.c:9:
Get past that and sched.h => processor.h => cpuid/api.h also
gets you to paravirt.h.
I suspect a lot of headers get pulled in like that.
David
^ permalink raw reply
* [PATCH v5] dma-buf: add some tracepoints to debug.
From: Xiang Gao @ 2025-12-01 11:21 UTC (permalink / raw)
To: sumit.semwal, christian.koenig, rostedt, mhiramat
Cc: linux-media, dri-devel, linux-kernel, mathieu.desnoyers, dhowells,
kuba, brauner, akpm, linux-trace-kernel, gaoxiang17
From: gaoxiang17 <gaoxiang17@xiaomi.com>
Since we can only inspect dmabuf by iterating over process FDs or the
dmabuf_list, we need to add our own tracepoints to track its status in
real time in production.
For example:
binder:3016_1-3102 [006] ...1. 255.126521: dma_buf_export: exp_name=qcom,system size=12685312 ino=2738
binder:3016_1-3102 [006] ...1. 255.126528: dma_buf_fd: exp_name=qcom,system size=12685312 ino=2738 fd=8
binder:3016_1-3102 [006] ...1. 255.126642: dma_buf_mmap_internal: exp_name=qcom,system size=28672 ino=2739
kworker/6:1-86 [006] ...1. 255.127194: dma_buf_put: exp_name=qcom,system size=12685312 ino=2738
RenderThread-9293 [006] ...1. 316.618179: dma_buf_get: exp_name=qcom,system size=12771328 ino=2762 fd=176
RenderThread-9293 [006] ...1. 316.618195: dma_buf_dynamic_attach: exp_name=qcom,system size=12771328 ino=2762 attachment:ffffff880a18dd00 is_dynamic=0 dev_name=kgsl-3d0
RenderThread-9293 [006] ...1. 318.878220: dma_buf_detach: exp_name=qcom,system size=12771328 ino=2762 attachment:ffffff880a18dd00 is_dynamic=0 dev_name=kgsl-3d0
Signed-off-by: Xiang Gao <gaoxiang17@xiaomi.com>
---
drivers/dma-buf/dma-buf.c | 34 +++++++-
include/trace/events/dma_buf.h | 154 +++++++++++++++++++++++++++++++++
2 files changed, 187 insertions(+), 1 deletion(-)
create mode 100644 include/trace/events/dma_buf.h
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index 2bcf9ceca997..fc05e2114922 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -35,6 +35,17 @@
#include "dma-buf-sysfs-stats.h"
+#define CREATE_TRACE_POINTS
+#include <trace/events/dma_buf.h>
+
+#define DMA_BUF_TRACE(FUNC, ...) \
+ do { \
+ if (FUNC##_enabled()) { \
+ guard(spinlock)(&dmabuf->name_lock); \
+ FUNC(__VA_ARGS__); \
+ } \
+ } while (0)
+
static inline int is_dma_buf_file(struct file *);
static DEFINE_MUTEX(dmabuf_list_mutex);
@@ -220,6 +231,8 @@ static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma)
dmabuf->size >> PAGE_SHIFT)
return -EINVAL;
+ DMA_BUF_TRACE(trace_dma_buf_mmap_internal, dmabuf);
+
return dmabuf->ops->mmap(dmabuf, vma);
}
@@ -745,6 +758,8 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
__dma_buf_list_add(dmabuf);
+ DMA_BUF_TRACE(trace_dma_buf_export, dmabuf);
+
return dmabuf;
err_dmabuf:
@@ -779,6 +794,8 @@ int dma_buf_fd(struct dma_buf *dmabuf, int flags)
fd_install(fd, dmabuf->file);
+ DMA_BUF_TRACE(trace_dma_buf_fd, dmabuf, fd);
+
return fd;
}
EXPORT_SYMBOL_NS_GPL(dma_buf_fd, "DMA_BUF");
@@ -794,6 +811,7 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_fd, "DMA_BUF");
struct dma_buf *dma_buf_get(int fd)
{
struct file *file;
+ struct dma_buf *dmabuf;
file = fget(fd);
@@ -805,7 +823,11 @@ struct dma_buf *dma_buf_get(int fd)
return ERR_PTR(-EINVAL);
}
- return file->private_data;
+ dmabuf = file->private_data;
+
+ DMA_BUF_TRACE(trace_dma_buf_get, dmabuf, fd);
+
+ return dmabuf;
}
EXPORT_SYMBOL_NS_GPL(dma_buf_get, "DMA_BUF");
@@ -825,6 +847,8 @@ void dma_buf_put(struct dma_buf *dmabuf)
return;
fput(dmabuf->file);
+
+ DMA_BUF_TRACE(trace_dma_buf_put, dmabuf);
}
EXPORT_SYMBOL_NS_GPL(dma_buf_put, "DMA_BUF");
@@ -979,6 +1003,9 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev,
list_add(&attach->node, &dmabuf->attachments);
dma_resv_unlock(dmabuf->resv);
+ DMA_BUF_TRACE(trace_dma_buf_dynamic_attach, dmabuf, attach,
+ dma_buf_attachment_is_dynamic(attach), dev);
+
return attach;
err_attach:
@@ -1023,6 +1050,9 @@ void dma_buf_detach(struct dma_buf *dmabuf, struct dma_buf_attachment *attach)
if (dmabuf->ops->detach)
dmabuf->ops->detach(dmabuf, attach);
+ DMA_BUF_TRACE(trace_dma_buf_detach, dmabuf, attach,
+ dma_buf_attachment_is_dynamic(attach), attach->dev);
+
kfree(attach);
}
EXPORT_SYMBOL_NS_GPL(dma_buf_detach, "DMA_BUF");
@@ -1488,6 +1518,8 @@ int dma_buf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma,
vma_set_file(vma, dmabuf->file);
vma->vm_pgoff = pgoff;
+ DMA_BUF_TRACE(trace_dma_buf_mmap, dmabuf);
+
return dmabuf->ops->mmap(dmabuf, vma);
}
EXPORT_SYMBOL_NS_GPL(dma_buf_mmap, "DMA_BUF");
diff --git a/include/trace/events/dma_buf.h b/include/trace/events/dma_buf.h
new file mode 100644
index 000000000000..81ee4d05979c
--- /dev/null
+++ b/include/trace/events/dma_buf.h
@@ -0,0 +1,154 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM dma_buf
+
+#if !defined(_TRACE_DMA_BUF_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_DMA_BUF_H
+
+#include <linux/dma-buf.h>
+#include <linux/tracepoint.h>
+
+DECLARE_EVENT_CLASS(dma_buf,
+
+ TP_PROTO(struct dma_buf *dmabuf),
+
+ TP_ARGS(dmabuf),
+
+ TP_STRUCT__entry(
+ __string(exp_name, dmabuf->exp_name)
+ __field(size_t, size)
+ __field(ino_t, ino)
+ ),
+
+ TP_fast_assign(
+ __assign_str(exp_name);
+ __entry->size = dmabuf->size;
+ __entry->ino = dmabuf->file->f_inode->i_ino;
+ ),
+
+ TP_printk("exp_name=%s size=%zu ino=%lu",
+ __get_str(exp_name),
+ __entry->size,
+ __entry->ino)
+);
+
+DECLARE_EVENT_CLASS(dma_buf_attach_dev,
+
+ TP_PROTO(struct dma_buf *dmabuf, struct dma_buf_attachment *attach, bool is_dynamic, struct device *dev),
+
+ TP_ARGS(dmabuf, attach, is_dynamic, dev),
+
+ TP_STRUCT__entry(
+ __string(dev_name, dev_name(dev))
+ __string(exp_name, dmabuf->exp_name)
+ __field(size_t, size)
+ __field(ino_t, ino)
+ __field(struct dma_buf_attachment *, attach)
+ __field(bool, is_dynamic)
+ ),
+
+ TP_fast_assign(
+ __assign_str(dev_name);
+ __assign_str(exp_name);
+ __entry->size = dmabuf->size;
+ __entry->ino = dmabuf->file->f_inode->i_ino;
+ __entry->is_dynamic = is_dynamic;
+ __entry->attach = attach;
+ ),
+
+ TP_printk("exp_name=%s size=%zu ino=%lu attachment:%p is_dynamic=%d dev_name=%s",
+ __get_str(exp_name),
+ __entry->size,
+ __entry->ino,
+ __entry->attach,
+ __entry->is_dynamic,
+ __get_str(dev_name))
+);
+
+DECLARE_EVENT_CLASS(dma_buf_fd,
+
+ TP_PROTO(struct dma_buf *dmabuf, int fd),
+
+ TP_ARGS(dmabuf, fd),
+
+ TP_STRUCT__entry(
+ __string(exp_name, dmabuf->exp_name)
+ __field(size_t, size)
+ __field(ino_t, ino)
+ __field(int, fd)
+ ),
+
+ TP_fast_assign(
+ __assign_str(exp_name);
+ __entry->size = dmabuf->size;
+ __entry->ino = dmabuf->file->f_inode->i_ino;
+ __entry->fd = fd;
+ ),
+
+ TP_printk("exp_name=%s size=%zu ino=%lu fd=%d",
+ __get_str(exp_name),
+ __entry->size,
+ __entry->ino,
+ __entry->fd)
+);
+
+DEFINE_EVENT(dma_buf, dma_buf_export,
+
+ TP_PROTO(struct dma_buf *dmabuf),
+
+ TP_ARGS(dmabuf)
+);
+
+DEFINE_EVENT(dma_buf, dma_buf_mmap_internal,
+
+ TP_PROTO(struct dma_buf *dmabuf),
+
+ TP_ARGS(dmabuf)
+);
+
+DEFINE_EVENT(dma_buf, dma_buf_mmap,
+
+ TP_PROTO(struct dma_buf *dmabuf),
+
+ TP_ARGS(dmabuf)
+);
+
+DEFINE_EVENT(dma_buf, dma_buf_put,
+
+ TP_PROTO(struct dma_buf *dmabuf),
+
+ TP_ARGS(dmabuf)
+);
+
+DEFINE_EVENT(dma_buf_attach_dev, dma_buf_dynamic_attach,
+
+ TP_PROTO(struct dma_buf *dmabuf, struct dma_buf_attachment *attach, bool is_dynamic, struct device *dev),
+
+ TP_ARGS(dmabuf, attach, is_dynamic, dev)
+);
+
+DEFINE_EVENT(dma_buf_attach_dev, dma_buf_detach,
+
+ TP_PROTO(struct dma_buf *dmabuf, struct dma_buf_attachment *attach, bool is_dynamic, struct device *dev),
+
+ TP_ARGS(dmabuf, attach, is_dynamic, dev)
+);
+
+DEFINE_EVENT(dma_buf_fd, dma_buf_fd,
+
+ TP_PROTO(struct dma_buf *dmabuf, int fd),
+
+ TP_ARGS(dmabuf, fd)
+);
+
+DEFINE_EVENT(dma_buf_fd, dma_buf_get,
+
+ TP_PROTO(struct dma_buf *dmabuf, int fd),
+
+ TP_ARGS(dmabuf, fd)
+);
+
+#endif /* _TRACE_DMA_BUF_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
--
2.34.1
^ permalink raw reply related
* [PATCH v2 0/3] unwind_user: Cleanups
From: Jens Remus @ 2025-12-01 12:54 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, x86, Steven Rostedt,
Peter Zijlstra
Cc: Jens Remus, Josh Poimboeuf, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Mathieu Desnoyers,
Indu Bhagat, Jose E. Marchesi, Heiko Carstens, Vasily Gorbik,
Ilya Leoshkevich
This patch series applies on top of Peter Zijlstras' latest unwind user
enhancements (and perf deferred callchain support) on his tip perf/core
branch:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/core
Which has already been merged to tip/master and linux-next/master.
Patch 1 enhances a few comments in unwind_user_next_common().
Patch 2 gets rid of an ifdef in unwind_user_next_fp() by moving it to
linux/unwind_user.h. Additionally it provides a common fallback for
unwind_user_at_function_start().
Patch 3 ensures the x86 unwind_user_word_size() implementation is
available whenever config option UNWIND_USER is enabled, as it is
required by unwind user in general and is not specific to its FP
unwind method.
Regards,
Jens
Jens Remus (3):
unwind_user: Enhance comments on get CFA, FP, and RA
unwind_user/fp: Use dummies instead of ifdef
x86/unwind_user: Guard unwind_user_word_size() by UNWIND_USER
arch/x86/include/asm/unwind_user.h | 29 +++++++++++++++++------------
include/linux/unwind_user.h | 18 ++++++++++++++++--
kernel/unwind/user.c | 12 ++++--------
3 files changed, 37 insertions(+), 22 deletions(-)
--
2.51.0
^ permalink raw reply
* [PATCH v2 1/3] unwind_user: Enhance comments on get CFA, FP, and RA
From: Jens Remus @ 2025-12-01 12:54 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, x86, Steven Rostedt,
Peter Zijlstra
Cc: Jens Remus, Josh Poimboeuf, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Mathieu Desnoyers,
Indu Bhagat, Jose E. Marchesi, Heiko Carstens, Vasily Gorbik,
Ilya Leoshkevich
In-Reply-To: <20251201125416.1239568-1-jremus@linux.ibm.com>
Move the comment "Get the Canonical Frame Address (CFA)" to the top
of the sequence of statements that actually get the CFA. Reword the
comment "Find the Return Address (RA)" to "Get ...", as the statements
actually get the RA. Add a respective comment to the statements that
get the FP. This will be useful once future commits extend the logic
to get the RA and FP.
While at it align the comment on the "stack going in wrong direction"
check to the following one on the "address is word aligned" check.
Signed-off-by: Jens Remus <jremus@linux.ibm.com>
---
kernel/unwind/user.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/unwind/user.c b/kernel/unwind/user.c
index 39e270789444..0ca434f86e73 100644
--- a/kernel/unwind/user.c
+++ b/kernel/unwind/user.c
@@ -31,6 +31,7 @@ static int unwind_user_next_common(struct unwind_user_state *state,
{
unsigned long cfa, fp, ra;
+ /* Get the Canonical Frame Address (CFA) */
if (frame->use_fp) {
if (state->fp < state->sp)
return -EINVAL;
@@ -38,11 +39,9 @@ static int unwind_user_next_common(struct unwind_user_state *state,
} else {
cfa = state->sp;
}
-
- /* Get the Canonical Frame Address (CFA) */
cfa += frame->cfa_off;
- /* stack going in wrong direction? */
+ /* Make sure that stack is not going in wrong direction */
if (cfa <= state->sp)
return -EINVAL;
@@ -50,10 +49,11 @@ static int unwind_user_next_common(struct unwind_user_state *state,
if (cfa & (state->ws - 1))
return -EINVAL;
- /* Find the Return Address (RA) */
+ /* Get the Return Address (RA) */
if (get_user_word(&ra, cfa, frame->ra_off, state->ws))
return -EINVAL;
+ /* Get the Frame Pointer (FP) */
if (frame->fp_off && get_user_word(&fp, cfa, frame->fp_off, state->ws))
return -EINVAL;
--
2.51.0
^ permalink raw reply related
* [PATCH v2 2/3] unwind_user/fp: Use dummies instead of ifdef
From: Jens Remus @ 2025-12-01 12:54 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, x86, Steven Rostedt,
Peter Zijlstra
Cc: Jens Remus, Josh Poimboeuf, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Mathieu Desnoyers,
Indu Bhagat, Jose E. Marchesi, Heiko Carstens, Vasily Gorbik,
Ilya Leoshkevich
In-Reply-To: <20251201125416.1239568-1-jremus@linux.ibm.com>
This simplifies the code. unwind_user_next_fp() does not need to
return -EINVAL if config option HAVE_UNWIND_USER_FP is disabled, as
unwind_user_start() will then not select this unwind method and
unwind_user_next() will therefore not call it.
Provide (1) a dummy definition of ARCH_INIT_USER_FP_FRAME, if the unwind
user method HAVE_UNWIND_USER_FP is not enabled, (2) a common fallback
definition of unwind_user_at_function_start() which returns false, and
(3) a common dummy definition of ARCH_INIT_USER_FP_ENTRY_FRAME.
Note that enabling the config option HAVE_UNWIND_USER_FP without
defining ARCH_INIT_USER_FP_FRAME triggers a compile error, which is
helpful when implementing support for this unwind user method in an
architecture. Enabling the config option when providing an arch-
specific unwind_user_at_function_start() definition makes it necessary
to also provide an arch-specific ARCH_INIT_USER_FP_ENTRY_FRAME
definition.
Signed-off-by: Jens Remus <jremus@linux.ibm.com>
---
Notes (jremus):
Changes in v2:
- Add parameter ws to ARCH_INIT_USER_{FP_FRAME|FP_ENTRY_FRAME}.
- Provide common fallback of unwind_user_at_function_start().
- Provide common dummy of ARCH_INIT_USER_FP_ENTRY_FRAME.
- Reword commit message accordingly.
arch/x86/include/asm/unwind_user.h | 1 +
include/linux/unwind_user.h | 18 ++++++++++++++++--
kernel/unwind/user.c | 4 ----
3 files changed, 17 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/unwind_user.h b/arch/x86/include/asm/unwind_user.h
index 12064284bc4e..971ffe937d50 100644
--- a/arch/x86/include/asm/unwind_user.h
+++ b/arch/x86/include/asm/unwind_user.h
@@ -35,6 +35,7 @@ static inline bool unwind_user_at_function_start(struct pt_regs *regs)
{
return is_uprobe_at_func_entry(regs);
}
+#define unwind_user_at_function_start unwind_user_at_function_start
#endif /* CONFIG_HAVE_UNWIND_USER_FP */
diff --git a/include/linux/unwind_user.h b/include/linux/unwind_user.h
index 7f7282516bf5..e3e65c2955cf 100644
--- a/include/linux/unwind_user.h
+++ b/include/linux/unwind_user.h
@@ -5,8 +5,22 @@
#include <linux/unwind_user_types.h>
#include <asm/unwind_user.h>
-#ifndef ARCH_INIT_USER_FP_FRAME
- #define ARCH_INIT_USER_FP_FRAME
+#ifndef CONFIG_HAVE_UNWIND_USER_FP
+
+#define ARCH_INIT_USER_FP_FRAME(ws)
+
+#endif /* !CONFIG_HAVE_UNWIND_USER_FP */
+
+#ifndef ARCH_INIT_USER_FP_ENTRY_FRAME
+#define ARCH_INIT_USER_FP_ENTRY_FRAME(ws)
+#endif
+
+#ifndef unwind_user_at_function_start
+static inline bool unwind_user_at_function_start(struct pt_regs *regs)
+{
+ return false;
+}
+#define unwind_user_at_function_start unwind_user_at_function_start
#endif
int unwind_user(struct unwind_stacktrace *trace, unsigned int max_entries);
diff --git a/kernel/unwind/user.c b/kernel/unwind/user.c
index 0ca434f86e73..90ab3c1a205e 100644
--- a/kernel/unwind/user.c
+++ b/kernel/unwind/user.c
@@ -67,7 +67,6 @@ static int unwind_user_next_common(struct unwind_user_state *state,
static int unwind_user_next_fp(struct unwind_user_state *state)
{
-#ifdef CONFIG_HAVE_UNWIND_USER_FP
struct pt_regs *regs = task_pt_regs(current);
if (state->topmost && unwind_user_at_function_start(regs)) {
@@ -81,9 +80,6 @@ static int unwind_user_next_fp(struct unwind_user_state *state)
ARCH_INIT_USER_FP_FRAME(state->ws)
};
return unwind_user_next_common(state, &fp_frame);
-#else
- return -EINVAL;
-#endif
}
static int unwind_user_next(struct unwind_user_state *state)
--
2.51.0
^ permalink raw reply related
* [PATCH v2 3/3] x86/unwind_user: Guard unwind_user_word_size() by UNWIND_USER
From: Jens Remus @ 2025-12-01 12:54 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, x86, Steven Rostedt,
Peter Zijlstra
Cc: Jens Remus, Josh Poimboeuf, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Mathieu Desnoyers,
Indu Bhagat, Jose E. Marchesi, Heiko Carstens, Vasily Gorbik,
Ilya Leoshkevich
In-Reply-To: <20251201125416.1239568-1-jremus@linux.ibm.com>
The unwind user framework in general requires an architecture-specific
implementation of unwind_user_word_size() to be present for any unwind
method, whether that is fp or a future other method, such as potentially
sframe.
Guard unwind_user_word_size() by the availability of the UNWIND_USER
framework instead of the specific HAVE_UNWIND_USER_FP method.
This facilitates to selectively disable HAVE_UNWIND_USER_FP on x86
(e.g. for test purposes) once a new unwind method is added to unwind
user.
Signed-off-by: Jens Remus <jremus@linux.ibm.com>
---
arch/x86/include/asm/unwind_user.h | 28 ++++++++++++++++------------
1 file changed, 16 insertions(+), 12 deletions(-)
diff --git a/arch/x86/include/asm/unwind_user.h b/arch/x86/include/asm/unwind_user.h
index 971ffe937d50..e1137eb67ab2 100644
--- a/arch/x86/include/asm/unwind_user.h
+++ b/arch/x86/include/asm/unwind_user.h
@@ -2,6 +2,22 @@
#ifndef _ASM_X86_UNWIND_USER_H
#define _ASM_X86_UNWIND_USER_H
+#ifdef CONFIG_UNWIND_USER
+
+static inline int unwind_user_word_size(struct pt_regs *regs)
+{
+ /* We can't unwind VM86 stacks */
+ if (regs->flags & X86_VM_MASK)
+ return 0;
+#ifdef CONFIG_X86_64
+ if (!user_64bit_mode(regs))
+ return sizeof(int);
+#endif
+ return sizeof(long);
+}
+
+#endif /* CONFIG_UNWIND_USER */
+
#ifdef CONFIG_HAVE_UNWIND_USER_FP
#include <asm/ptrace.h>
@@ -19,18 +35,6 @@
.fp_off = 0, \
.use_fp = false,
-static inline int unwind_user_word_size(struct pt_regs *regs)
-{
- /* We can't unwind VM86 stacks */
- if (regs->flags & X86_VM_MASK)
- return 0;
-#ifdef CONFIG_X86_64
- if (!user_64bit_mode(regs))
- return sizeof(int);
-#endif
- return sizeof(long);
-}
-
static inline bool unwind_user_at_function_start(struct pt_regs *regs)
{
return is_uprobe_at_func_entry(regs);
--
2.51.0
^ permalink raw reply related
* [PATCH bpf 0/2] bpf: fix bpf_d_path() helper prototype
From: Shuran Liu @ 2025-12-01 14:38 UTC (permalink / raw)
To: song, mattbobrowski, bpf
Cc: ast, daniel, andrii, martin.lau, eddyz87, yonghong.song,
john.fastabend, kpsingh, sdf, haoluo, jolsa, rostedt, mhiramat,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, electronlsr
Hi,
this series fixes a verifier regression for bpf_d_path() introduced by
commit 37cce22dbd51 ("bpf: verifier: Refactor helper access type
tracking") and adds a small selftest to exercise the helper from an
LSM program.
Commit 37cce22dbd51 started distinguishing read vs write accesses
performed by helpers. bpf_d_path()'s buffer argument was left as
ARG_PTR_TO_MEM without MEM_WRITE, so the verifier could incorrectly
assume that the buffer contents are unchanged across the helper call
and base its optimizations on this wrong assumption.
In practice this showed up as a misbehaving LSM BPF program that calls
bpf_d_path() and then does a simple prefix comparison on the returned
path: the program would sometimes take the "mismatch" branch even
though both bytes being compared were actually equal.
Patch 1 fixes bpf_d_path()'s helper prototype by marking the buffer
argument as ARG_PTR_TO_MEM | MEM_WRITE, so that the verifier correctly
models the write to the caller-provided buffer.
Patch 2 adds a minimal selftest under tools/testing/selftests/bpf that
hooks bprm_check_security, calls bpf_d_path() on a binary under /tmp/,
and verifies that the prefix comparison on the returned path keeps
working.
On my local setup, tools/testing/selftests/bpf does not build fully
due to unrelated tests using newer helpers. I validated this series by
manually reproducing the issue with a small LSM program and by
building and running only the new d_path_lsm test on kernels with and
without patch 1 applied.
Thanks,
Shuran Liu
Shuran Liu (2):
bpf: mark bpf_d_path() buffer as writeable
selftests/bpf: add regression test for bpf_d_path()
kernel/trace/bpf_trace.c | 2 +-
.../selftests/bpf/prog_tests/d_path_lsm.c | 27 ++++++++++++
.../selftests/bpf/progs/d_path_lsm.bpf.c | 43 +++++++++++++++++++
3 files changed, 71 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/d_path_lsm.c
create mode 100644 tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c
--
2.52.0
^ permalink raw reply
* [PATCH bpf 1/2] bpf: mark bpf_d_path() buffer as writeable
From: Shuran Liu @ 2025-12-01 14:38 UTC (permalink / raw)
To: song, mattbobrowski, bpf
Cc: ast, daniel, andrii, martin.lau, eddyz87, yonghong.song,
john.fastabend, kpsingh, sdf, haoluo, jolsa, rostedt, mhiramat,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, electronlsr,
Zesen Liu, Peili Gao, Haoran Ni
In-Reply-To: <20251201143813.5212-1-electronlsr@gmail.com>
Commit 37cce22dbd51 ("bpf: verifier: Refactor helper access type
tracking") started distinguishing read vs write accesses performed by
helpers.
The second argument of bpf_d_path() is a pointer to a buffer that the
helper fills with the resulting path. However, its prototype currently
uses ARG_PTR_TO_MEM without MEM_WRITE.
Before 37cce22dbd51, helper accesses were conservatively treated as
potential writes, so this mismatch did not cause issues. Since that
commit, the verifier may incorrectly assume that the buffer contents
are unchanged across the helper call and base its optimizations on this
wrong assumption. This can lead to misbehaviour in BPF programs that
read back the buffer, such as prefix comparisons on the returned path.
Fix this by marking the second argument of bpf_d_path() as
ARG_PTR_TO_MEM | MEM_WRITE so that the verifier correctly models the
write to the caller-provided buffer.
Fixes: 37cce22dbd51 ("bpf: verifier: Refactor helper access type tracking")
Co-developed-by: Zesen Liu <ftyg@live.com>
Signed-off-by: Zesen Liu <ftyg@live.com>
Co-developed-by: Peili Gao <gplhust955@gmail.com>
Signed-off-by: Peili Gao <gplhust955@gmail.com>
Co-developed-by: Haoran Ni <haoran.ni.cs@gmail.com>
Signed-off-by: Haoran Ni <haoran.ni.cs@gmail.com>
Signed-off-by: Shuran Liu <electronlsr@gmail.com>
---
kernel/trace/bpf_trace.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 4f87c16d915a..49e0bdaa7a1b 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -965,7 +965,7 @@ static const struct bpf_func_proto bpf_d_path_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &bpf_d_path_btf_ids[0],
- .arg2_type = ARG_PTR_TO_MEM,
+ .arg2_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.allowed = bpf_d_path_allowed,
};
--
2.52.0
^ permalink raw reply related
* [PATCH bpf 2/2] selftests/bpf: add regression test for bpf_d_path()
From: Shuran Liu @ 2025-12-01 14:38 UTC (permalink / raw)
To: song, mattbobrowski, bpf
Cc: ast, daniel, andrii, martin.lau, eddyz87, yonghong.song,
john.fastabend, kpsingh, sdf, haoluo, jolsa, rostedt, mhiramat,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, electronlsr,
Zesen Liu, Peili Gao, Haoran Ni
In-Reply-To: <20251201143813.5212-1-electronlsr@gmail.com>
Add a simple LSM BPF program and a corresponding test_progs test case
to exercise bpf_d_path() and ensure that prefix comparisons on the
returned path keep working.
The LSM program hooks bprm_check_security, calls bpf_d_path() on the
binary being executed, and compares the returned path against the
"/tmp/" prefix. The result is recorded in an array map.
The user space test runs /tmp/bpf_d_path_test (copied from /bin/true)
and checks that the BPF program records a successful prefix match.
Without the preceding fix to bpf_d_path()'s helper prototype, the
test can fail due to the verifier incorrectly assuming that the
buffer contents are unchanged across the helper call and misoptimizing
the program. With the fix applied, the test passes.
Co-developed-by: Zesen Liu <ftyg@live.com>
Signed-off-by: Zesen Liu <ftyg@live.com>
Co-developed-by: Peili Gao <gplhust955@gmail.com>
Signed-off-by: Peili Gao <gplhust955@gmail.com>
Co-developed-by: Haoran Ni <haoran.ni.cs@gmail.com>
Signed-off-by: Haoran Ni <haoran.ni.cs@gmail.com>
Signed-off-by: Shuran Liu <electronlsr@gmail.com>
---
.../selftests/bpf/prog_tests/d_path_lsm.c | 27 ++++++++++++
.../selftests/bpf/progs/d_path_lsm.bpf.c | 43 +++++++++++++++++++
2 files changed, 70 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/d_path_lsm.c
create mode 100644 tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c
diff --git a/tools/testing/selftests/bpf/prog_tests/d_path_lsm.c b/tools/testing/selftests/bpf/prog_tests/d_path_lsm.c
new file mode 100644
index 000000000000..92aad744ed12
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/d_path_lsm.c
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <test_progs.h>
+#include "d_path_lsm.skel.h"
+
+void test_d_path_lsm(void)
+{
+ struct d_path_lsm *skel = NULL;
+ int err, map_fd, key = 0, val = 0;
+
+ skel = d_path_lsm__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+
+ err = d_path_lsm__attach(skel);
+ if (!ASSERT_OK(err, "attach"))
+ goto out;
+
+ system("cp /bin/true /tmp/bpf_d_path_test 2>/dev/null || :");
+ system("/tmp/bpf_d_path_test >/dev/null 2>&1");
+
+ map_fd = bpf_map__fd(skel->maps.result);
+ err = bpf_map_lookup_elem(map_fd, &key, &val);
+ ASSERT_OK(err, "lookup_result");
+ ASSERT_EQ(val, 1, "prefix_match");
+out:
+ d_path_lsm__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c b/tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c
new file mode 100644
index 000000000000..36f9ff37e817
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/d_path_lsm.bpf.c
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char LICENSE[] SEC("license") = "GPL";
+
+#define FILENAME_MAX_SIZE 256
+#define TARGET_DIR "/tmp/"
+#define TARGET_DIR_LEN 5
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, int);
+} result SEC(".maps");
+
+SEC("lsm/bprm_check_security")
+int BPF_PROG(d_path_lsm_prog, struct linux_binprm *bprm)
+{
+ char path[FILENAME_MAX_SIZE] = {};
+ long len;
+ int key = 0;
+ int val = 0;
+
+ len = bpf_d_path(&bprm->file->f_path, path, sizeof(path));
+ if (len < 0)
+ return 0;
+
+#pragma unroll
+ for (int i = 0; i < TARGET_DIR_LEN; i++) {
+ if ((u8)path[i] != (u8)TARGET_DIR[i]) {
+ val = -1; /* mismatch */
+ bpf_map_update_elem(&result, &key, &val, BPF_ANY);
+ return 0;
+ }
+ }
+
+ val = 1; /* prefix match */
+ bpf_map_update_elem(&result, &key, &val, BPF_ANY);
+ return 0;
+}
--
2.52.0
^ permalink raw reply related
* Re: [PATCH v2 1/2] rv: Convert to use lock guard
From: Steven Rostedt @ 2025-12-01 15:24 UTC (permalink / raw)
To: Gabriele Monaco
Cc: Nam Cao, Masami Hiramatsu, Mathieu Desnoyers, linux-trace-kernel,
linux-kernel
In-Reply-To: <2e46a7dc417119bb7e331ff14253b0ff678d697e.camel@redhat.com>
On Wed, 26 Nov 2025 16:22:17 +0100
Gabriele Monaco <gmonaco@redhat.com> wrote:
> I can send it quickly since you already had a look at those two patches.
> It isn't a big problem you won't have time, though. It can stay for the next
> round.
Did you send them?
-- Steve
^ permalink raw reply
* Re: [PATCH v2 1/2] rv: Convert to use lock guard
From: Gabriele Monaco @ 2025-12-01 15:28 UTC (permalink / raw)
To: Steven Rostedt
Cc: Nam Cao, Masami Hiramatsu, Mathieu Desnoyers, linux-trace-kernel,
linux-kernel
In-Reply-To: <20251201102447.53f33be5@gandalf.local.home>
On Mon, 2025-12-01 at 10:24 -0500, Steven Rostedt wrote:
> On Wed, 26 Nov 2025 16:22:17 +0100
> Gabriele Monaco <gmonaco@redhat.com> wrote:
>
> > I can send it quickly since you already had a look at those two patches.
> > It isn't a big problem you won't have time, though. It can stay for the next
> > round.
>
> Did you send them?
I'm stupid, I sent it [1] but the script somehow didn't add you in the To: field
and I didn't notice..
I'd say since the merge window is open, this change can wait.
Sorry for that.
Gabriele
[1] - https://lore.kernel.org/lkml/20251126152253.464350-1-gmonaco@redhat.com
^ permalink raw reply
* Re: [PATCH v2 1/2] rv: Convert to use lock guard
From: Nam Cao @ 2025-12-01 15:28 UTC (permalink / raw)
To: Steven Rostedt, Gabriele Monaco
Cc: Masami Hiramatsu, Mathieu Desnoyers, linux-trace-kernel,
linux-kernel
In-Reply-To: <20251201102447.53f33be5@gandalf.local.home>
Steven Rostedt <rostedt@goodmis.org> writes:
> On Wed, 26 Nov 2025 16:22:17 +0100
> Gabriele Monaco <gmonaco@redhat.com> wrote:
>
>> I can send it quickly since you already had a look at those two patches.
>> It isn't a big problem you won't have time, though. It can stay for the next
>> round.
>
> Did you send them?
Looks like Gabriele forgot to include you as a recipient:
https://lore.kernel.org/lkml/20251126152253.464350-1-gmonaco@redhat.com/
Nam
^ permalink raw reply
* Re: [PATCH 3/3] tracing: move tracing declarations from kernel.h to a dedicated header
From: Andy Shevchenko @ 2025-12-01 15:37 UTC (permalink / raw)
To: david laight
Cc: Yury Norov, Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Randy Dunlap, Ingo Molnar, Jani Nikula, Joonas Lahtinen,
Rodrigo Vivi, Tvrtko Ursulin, Petr Pavlu, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
In-Reply-To: <20251201101658.0b5ab68e@pumpkin>
On Mon, Dec 01, 2025 at 10:16:58AM +0000, david laight wrote:
> On Mon, 1 Dec 2025 04:50:42 +0200
> Andy Shevchenko <andriy.shevchenko@linux.intel.com> wrote:
> > On Sun, Nov 30, 2025 at 11:09:25PM +0000, david laight wrote:
> > > On Sun, 30 Nov 2025 21:44:46 +0200
> > > Andy Shevchenko <andriy.shevchenko@linux.intel.com> wrote:
...
> > kernel.h elimination (in the form it exists right now) is very fruitful.
> > However, you may help with the (say) ioctl.h or whatever you consider
> > really fruitful, we all will thank you (no jokes).
> >
>
> This is the first #include path for ioctl.h
>
> In file included from ../include/asm-generic/ioctl.h:5,
> from ./arch/x86/include/generated/uapi/asm/ioctl.h:1,
> from ../include/uapi/linux/ioctl.h:5,
> from ../include/uapi/linux/random.h:12,
> from ../include/linux/random.h:10,
> from ../include/linux/nodemask.h:94,
> from ../include/linux/numa.h:6,
> from ../include/linux/cpumask.h:17,
> from ../arch/x86/include/asm/paravirt.h:21,
> from ../arch/x86/include/asm/irqflags.h:102,
> from ../include/linux/irqflags.h:18,
> from ../include/linux/spinlock.h:59,
> from ../include/linux/swait.h:7,
> from ../include/linux/completion.h:12,
> from ../include/linux/crypto.h:15,
> from ../arch/x86/kernel/asm-offsets.c:9:
>
> Get past that and sched.h => processor.h => cpuid/api.h also
> gets you to paravirt.h.
> I suspect a lot of headers get pulled in like that.
And several headers like ioctl.h that is "pull half of everything".
device.h, for example.
So, you can start untangling them piece by piece.
Not sure how [1] is actual right now, but I believe plenty of those
can still be used.
[1]: https://lwn.net/ml/linux-kernel/YdIfz+LMewetSaEB@gmail.com/
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply
* Re: [PATCH v8 26/28] KVM: arm64: Add hyp_enter/hyp_exit events to pKVM hyp
From: Vincent Donnefort @ 2025-12-01 16:04 UTC (permalink / raw)
To: Marc Zyngier
Cc: rostedt, mhiramat, mathieu.desnoyers, linux-trace-kernel,
oliver.upton, joey.gouly, suzuki.poulose, yuzenghui, kvmarm,
linux-arm-kernel, jstultz, qperret, will, aneesh.kumar,
kernel-team, linux-kernel
In-Reply-To: <87345vfiqj.wl-maz@kernel.org>
On Sun, Nov 30, 2025 at 07:00:52PM +0000, Marc Zyngier wrote:
> On Fri, 07 Nov 2025 09:38:38 +0000,
> Vincent Donnefort <vdonnefort@google.com> wrote:
> >
> > The hyp_enter and hyp_exit events are logged by the hypervisor any time
> > it is entered and exited.
> >
> > Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
> >
> > diff --git a/arch/arm64/include/asm/kvm_hypevents.h b/arch/arm64/include/asm/kvm_hypevents.h
> > index d6e033c96c52..ce3953bc884a 100644
> > --- a/arch/arm64/include/asm/kvm_hypevents.h
> > +++ b/arch/arm64/include/asm/kvm_hypevents.h
> > @@ -7,4 +7,21 @@
> > #include <nvhe/trace.h>
> > #endif
> >
> > +HYP_EVENT(hyp_enter,
> > + HE_PROTO(void),
> > + HE_STRUCT(
> > + ),
> > + HE_ASSIGN(
> > + ),
> > + HE_PRINTK()
> > +);
> > +
> > +HYP_EVENT(hyp_exit,
> > + HE_PROTO(void),
> > + HE_STRUCT(
> > + ),
> > + HE_ASSIGN(
> > + ),
> > + HE_PRINTK()
> > +);
> > #endif
> > diff --git a/arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h b/arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h
> > new file mode 100644
> > index 000000000000..4b69d33e4f2d
> > --- /dev/null
> > +++ b/arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h
> > @@ -0,0 +1,13 @@
> > +/* SPDX-License-Identifier: GPL-2.0-only */
> > +
> > +#include <asm/kvm_hypevents.h>
> > +
> > +#include <linux/arm-smccc.h>
> > +
> > +#undef arm_smccc_1_1_smc
> > +#define arm_smccc_1_1_smc(...) \
> > + do { \
> > + trace_hyp_exit(); \
> > + __arm_smccc_1_1(SMCCC_SMC_INST, __VA_ARGS__); \
> > + trace_hyp_enter(); \
> > + } while (0)
>
> Huh. No, please. Don't override existing kernel constructs behind
> everybody's back. I know that tracing folks love to play that sort of
> game, but I really don't want any of that here.
And you haven't seen what I had for arm_smccc_1_2_smc()
>
> Just define a wrapper that is KVM-specific, and use that.
>
> Then from a semantic perspective, an SMC call, an ERET back to EL1
> host, and an ERET back to EL1 guest are all very different events that
> you lump under the "exit" category. That's not right. Same thing for
> "enter".
The idea of those events is just to know when we enter and exit the hypervisor,
purely from a timing point of view. In ACK, we already have other separated
events that can give the "cause" for entering or exiting the hypervisor. I
haven't added them to that series because I thought it was big enough.
So happy to add now or do a follow-up, once we're done with that one?
>
> Thanks,
>
> M.
>
> --
> Jazz isn't dead. It just smells funny.
^ permalink raw reply
* Re: [PATCH v8 26/28] KVM: arm64: Add hyp_enter/hyp_exit events to pKVM hyp
From: Marc Zyngier @ 2025-12-01 16:33 UTC (permalink / raw)
To: Vincent Donnefort
Cc: rostedt, mhiramat, mathieu.desnoyers, linux-trace-kernel,
oliver.upton, joey.gouly, suzuki.poulose, yuzenghui, kvmarm,
linux-arm-kernel, jstultz, qperret, will, aneesh.kumar,
kernel-team, linux-kernel
In-Reply-To: <aS28dyrKgZaOmMNr@google.com>
On Mon, 01 Dec 2025 16:04:07 +0000,
Vincent Donnefort <vdonnefort@google.com> wrote:
>
> On Sun, Nov 30, 2025 at 07:00:52PM +0000, Marc Zyngier wrote:
> > On Fri, 07 Nov 2025 09:38:38 +0000,
> > Vincent Donnefort <vdonnefort@google.com> wrote:
> > >
> > > The hyp_enter and hyp_exit events are logged by the hypervisor any time
> > > it is entered and exited.
> > >
> > > Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
> > >
> > > diff --git a/arch/arm64/include/asm/kvm_hypevents.h b/arch/arm64/include/asm/kvm_hypevents.h
> > > index d6e033c96c52..ce3953bc884a 100644
> > > --- a/arch/arm64/include/asm/kvm_hypevents.h
> > > +++ b/arch/arm64/include/asm/kvm_hypevents.h
> > > @@ -7,4 +7,21 @@
> > > #include <nvhe/trace.h>
> > > #endif
> > >
> > > +HYP_EVENT(hyp_enter,
> > > + HE_PROTO(void),
> > > + HE_STRUCT(
> > > + ),
> > > + HE_ASSIGN(
> > > + ),
> > > + HE_PRINTK()
> > > +);
> > > +
> > > +HYP_EVENT(hyp_exit,
> > > + HE_PROTO(void),
> > > + HE_STRUCT(
> > > + ),
> > > + HE_ASSIGN(
> > > + ),
> > > + HE_PRINTK()
> > > +);
> > > #endif
> > > diff --git a/arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h b/arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h
> > > new file mode 100644
> > > index 000000000000..4b69d33e4f2d
> > > --- /dev/null
> > > +++ b/arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h
> > > @@ -0,0 +1,13 @@
> > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > +
> > > +#include <asm/kvm_hypevents.h>
> > > +
> > > +#include <linux/arm-smccc.h>
> > > +
> > > +#undef arm_smccc_1_1_smc
> > > +#define arm_smccc_1_1_smc(...) \
> > > + do { \
> > > + trace_hyp_exit(); \
> > > + __arm_smccc_1_1(SMCCC_SMC_INST, __VA_ARGS__); \
> > > + trace_hyp_enter(); \
> > > + } while (0)
> >
> > Huh. No, please. Don't override existing kernel constructs behind
> > everybody's back. I know that tracing folks love to play that sort of
> > game, but I really don't want any of that here.
>
> And you haven't seen what I had for arm_smccc_1_2_smc()
I have the feeling that I don't want to see it at all!
> > Just define a wrapper that is KVM-specific, and use that.
> >
> > Then from a semantic perspective, an SMC call, an ERET back to EL1
> > host, and an ERET back to EL1 guest are all very different events that
> > you lump under the "exit" category. That's not right. Same thing for
> > "enter".
>
> The idea of those events is just to know when we enter and exit the hypervisor,
> purely from a timing point of view. In ACK, we already have other separated
> events that can give the "cause" for entering or exiting the hypervisor. I
> haven't added them to that series because I thought it was big enough.
>
> So happy to add now or do a follow-up, once we're done with that one?
Fixed trace points are ABI AFAIC, so I'd rather we make them useful
from the very start. What you have here is not very interesting,
because it makes all entry and exit points equal -- they very much
aren't.
So get rid of these and implement the real thing.
Thanks,
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply
* [PATCH v13 mm-new 00/16] khugepaged: mTHP support
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
The following series provides khugepaged with the capability to collapse
anonymous memory regions to mTHPs.
To achieve this we generalize the khugepaged functions to no longer depend
on PMD_ORDER. Then during the PMD scan, we use a bitmap to track individual
pages that are occupied (!none/zero). After the PMD scan is done, we use
the bitmap to find the optimal mTHP sizes for the PMD range. The
restriction on max_ptes_none is removed during the scan, to make sure we
account for the whole PMD range in the bitmap. When no mTHP size is
enabled, the legacy behavior of khugepaged is maintained.
We currently only support max_ptes_none values of 0 or HPAGE_PMD_NR - 1
(ie 511). If any other value is specified, the kernel will emit a warning
and no mTHP collapse will be attempted. If a mTHP collapse is attempted,
but contains swapped out, or shared pages, we don't perform the collapse.
It is now also possible to collapse to mTHPs without requiring the PMD THP
size to be enabled. These limitiations are to prevent collapse "creep"
behavior. This prevents constantly promoting mTHPs to the next available
size, which would occur because a collapse introduces more non-zero pages
that would satisfy the promotion condition on subsequent scans.
Patch 1: Refactor/rename hpage_collapse
Patch 2: Refactoring to combine madvise_collapse and khugepaged
Patch 3-8: Generalize khugepaged functions for arbitrary orders and
introduce some helper functions
Patch 9: skip collapsing mTHP to smaller orders
Patch 10-11: Add per-order mTHP statistics and tracepoints
Patch 12: Introduce collapse_allowable_orders
Patch 13-15: Introduce bitmap and mTHP collapse support, fully enabled
Patch 16: Documentation
---------
Testing
---------
- Built for x86_64, aarch64, ppc64le, and s390x
- selftests mm
- I created a test script that I used to push khugepaged to its limits
while monitoring a number of stats and tracepoints. The code is
available here[1] (Run in legacy mode for these changes and set mthp
sizes to inherit)
The summary from my testings was that there was no significant
regression noticed through this test. In some cases my changes had
better collapse latencies, and was able to scan more pages in the same
amount of time/work, but for the most part the results were consistent.
- redis testing. I tested these changes along with my defer changes
(see followup [2] post for more details). We've decided to get the mTHP
changes merged first before attempting the defer series.
- some basic testing on 64k page size.
- lots of general use.
V13 Changes:
- Lots of minor nits, cleanups, comments, and renames
- Bitmap function simplification and more helpers (Wei, Lorenzo)
- Max_ptes_none (0 or 511) restriction
- commit description expansion
- list all reachable enum values in mthp_collapse()
- Fix ppc64 compile error due to using HPAGE_PMD_ORDER (replace with
ilog2(MAX_PTRS_PER_PTE))
V12: https://lore.kernel.org/lkml/20251022183717.70829-1-npache@redhat.com/
V11: https://lore.kernel.org/lkml/20250912032810.197475-1-npache@redhat.com/
V10: https://lore.kernel.org/lkml/20250819134205.622806-1-npache@redhat.com/
V9 : https://lore.kernel.org/lkml/20250714003207.113275-1-npache@redhat.com/
V8 : https://lore.kernel.org/lkml/20250702055742.102808-1-npache@redhat.com/
V7 : https://lore.kernel.org/lkml/20250515032226.128900-1-npache@redhat.com/
V6 : https://lore.kernel.org/lkml/20250515030312.125567-1-npache@redhat.com/
V5 : https://lore.kernel.org/lkml/20250428181218.85925-1-npache@redhat.com/
V4 : https://lore.kernel.org/lkml/20250417000238.74567-1-npache@redhat.com/
V3 : https://lore.kernel.org/lkml/20250414220557.35388-1-npache@redhat.com/
V2 : https://lore.kernel.org/lkml/20250211003028.213461-1-npache@redhat.com/
V1 : https://lore.kernel.org/lkml/20250108233128.14484-1-npache@redhat.com/
A big thanks to everyone that has reviewed, tested, and participated in
the development process. Its been a great experience working with all of
you on this endeavour.
[1] - https://gitlab.com/npache/khugepaged_mthp_test
[2] - https://lore.kernel.org/lkml/20250515033857.132535-1-npache@redhat.com/
Baolin Wang (1):
khugepaged: run khugepaged for all orders
Dev Jain (1):
khugepaged: generalize alloc_charge_folio()
Nico Pache (14):
khugepaged: rename hpage_collapse_* to collapse_*
introduce collapse_single_pmd to unify khugepaged and madvise_collapse
khugepaged: generalize hugepage_vma_revalidate for mTHP support
khugepaged: introduce is_mthp_order helper
khugepaged: generalize __collapse_huge_page_* for mTHP support
khugepaged: introduce collapse_max_ptes_none helper function
khugepaged: generalize collapse_huge_page for mTHP collapse
khugepaged: skip collapsing mTHP to smaller orders
khugepaged: add per-order mTHP collapse failure statistics
khugepaged: improve tracepoints for mTHP orders
khugepaged: introduce collapse_allowable_orders helper function
khugepaged: Introduce mTHP collapse support
khugepaged: avoid unnecessary mTHP collapse attempts
Documentation: mm: update the admin guide for mTHP collapse
Documentation/admin-guide/mm/transhuge.rst | 80 ++-
include/linux/huge_mm.h | 5 +
include/trace/events/huge_memory.h | 34 +-
mm/huge_memory.c | 11 +
mm/khugepaged.c | 696 +++++++++++++++------
mm/mremap.c | 2 +-
6 files changed, 618 insertions(+), 210 deletions(-)
--
2.51.1
^ permalink raw reply
* [PATCH v13 mm-new 01/16] khugepaged: rename hpage_collapse_* to collapse_*
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
The hpage_collapse functions describe functions used by madvise_collapse
and khugepaged. remove the unnecessary hpage prefix to shorten the
function name.
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Acked-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 73 ++++++++++++++++++++++++-------------------------
mm/mremap.c | 2 +-
2 files changed, 37 insertions(+), 38 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 97d1b2824386..959be77f2e65 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -394,14 +394,14 @@ void __init khugepaged_destroy(void)
kmem_cache_destroy(mm_slot_cache);
}
-static inline int hpage_collapse_test_exit(struct mm_struct *mm)
+static inline int collapse_test_exit(struct mm_struct *mm)
{
return atomic_read(&mm->mm_users) == 0;
}
-static inline int hpage_collapse_test_exit_or_disable(struct mm_struct *mm)
+static inline int collapse_test_exit_or_disable(struct mm_struct *mm)
{
- return hpage_collapse_test_exit(mm) ||
+ return collapse_test_exit(mm) ||
mm_flags_test(MMF_DISABLE_THP_COMPLETELY, mm);
}
@@ -435,7 +435,7 @@ void __khugepaged_enter(struct mm_struct *mm)
int wakeup;
/* __khugepaged_exit() must not run from under us */
- VM_BUG_ON_MM(hpage_collapse_test_exit(mm), mm);
+ VM_BUG_ON_MM(collapse_test_exit(mm), mm);
if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm)))
return;
@@ -489,7 +489,7 @@ void __khugepaged_exit(struct mm_struct *mm)
} else if (slot) {
/*
* This is required to serialize against
- * hpage_collapse_test_exit() (which is guaranteed to run
+ * collapse_test_exit() (which is guaranteed to run
* under mmap sem read mode). Stop here (after we return all
* pagetables will be destroyed) until khugepaged has finished
* working on the pagetables under the mmap_lock.
@@ -579,7 +579,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
folio = page_folio(page);
VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
- /* See hpage_collapse_scan_pmd(). */
+ /* See collapse_scan_pmd(). */
if (folio_maybe_mapped_shared(folio)) {
++shared;
if (cc->is_khugepaged &&
@@ -830,7 +830,7 @@ struct collapse_control khugepaged_collapse_control = {
.is_khugepaged = true,
};
-static bool hpage_collapse_scan_abort(int nid, struct collapse_control *cc)
+static bool collapse_scan_abort(int nid, struct collapse_control *cc)
{
int i;
@@ -865,7 +865,7 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void)
}
#ifdef CONFIG_NUMA
-static int hpage_collapse_find_target_node(struct collapse_control *cc)
+static int collapse_find_target_node(struct collapse_control *cc)
{
int nid, target_node = 0, max_value = 0;
@@ -884,7 +884,7 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc)
return target_node;
}
#else
-static int hpage_collapse_find_target_node(struct collapse_control *cc)
+static int collapse_find_target_node(struct collapse_control *cc)
{
return 0;
}
@@ -905,7 +905,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :
TVA_FORCED_COLLAPSE;
- if (unlikely(hpage_collapse_test_exit_or_disable(mm)))
+ if (unlikely(collapse_test_exit_or_disable(mm)))
return SCAN_ANY_PROCESS;
*vmap = vma = find_vma(mm, address);
@@ -978,7 +978,7 @@ static int check_pmd_still_valid(struct mm_struct *mm,
/*
* Bring missing pages in from swap, to complete THP collapse.
- * Only done if hpage_collapse_scan_pmd believes it is worthwhile.
+ * Only done if khugepaged_scan_pmd believes it is worthwhile.
*
* Called and returns without pte mapped or spinlocks held.
* Returns result: if not SCAN_SUCCEED, mmap_lock has been released.
@@ -1065,7 +1065,7 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,
{
gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
GFP_TRANSHUGE);
- int node = hpage_collapse_find_target_node(cc);
+ int node = collapse_find_target_node(cc);
struct folio *folio;
folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, &cc->alloc_nmask);
@@ -1244,10 +1244,10 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
return result;
}
-static int hpage_collapse_scan_pmd(struct mm_struct *mm,
- struct vm_area_struct *vma,
- unsigned long start_addr, bool *mmap_locked,
- struct collapse_control *cc)
+static int collapse_scan_pmd(struct mm_struct *mm,
+ struct vm_area_struct *vma,
+ unsigned long start_addr, bool *mmap_locked,
+ struct collapse_control *cc)
{
pmd_t *pmd;
pte_t *pte, *_pte;
@@ -1355,7 +1355,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
* hit record.
*/
node = folio_nid(folio);
- if (hpage_collapse_scan_abort(node, cc)) {
+ if (collapse_scan_abort(node, cc)) {
result = SCAN_SCAN_ABORT;
goto out_unmap;
}
@@ -1421,7 +1421,7 @@ static void collect_mm_slot(struct mm_slot *slot)
lockdep_assert_held(&khugepaged_mm_lock);
- if (hpage_collapse_test_exit(mm)) {
+ if (collapse_test_exit(mm)) {
/* free mm_slot */
hash_del(&slot->hash);
list_del(&slot->mm_node);
@@ -1769,7 +1769,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
if (find_pmd_or_thp_or_none(mm, addr, &pmd) != SCAN_SUCCEED)
continue;
- if (hpage_collapse_test_exit(mm))
+ if (collapse_test_exit(mm))
continue;
if (!file_backed_vma_is_retractable(vma))
@@ -2285,9 +2285,9 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
return result;
}
-static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,
- struct file *file, pgoff_t start,
- struct collapse_control *cc)
+static int collapse_scan_file(struct mm_struct *mm, unsigned long addr,
+ struct file *file, pgoff_t start,
+ struct collapse_control *cc)
{
struct folio *folio = NULL;
struct address_space *mapping = file->f_mapping;
@@ -2342,7 +2342,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,
}
node = folio_nid(folio);
- if (hpage_collapse_scan_abort(node, cc)) {
+ if (collapse_scan_abort(node, cc)) {
result = SCAN_SCAN_ABORT;
folio_put(folio);
break;
@@ -2392,7 +2392,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,
return result;
}
-static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
+static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,
struct collapse_control *cc)
__releases(&khugepaged_mm_lock)
__acquires(&khugepaged_mm_lock)
@@ -2427,7 +2427,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
goto breakouterloop_mmap_lock;
progress++;
- if (unlikely(hpage_collapse_test_exit_or_disable(mm)))
+ if (unlikely(collapse_test_exit_or_disable(mm)))
goto breakouterloop;
vma_iter_init(&vmi, mm, khugepaged_scan.address);
@@ -2435,7 +2435,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
unsigned long hstart, hend;
cond_resched();
- if (unlikely(hpage_collapse_test_exit_or_disable(mm))) {
+ if (unlikely(collapse_test_exit_or_disable(mm))) {
progress++;
break;
}
@@ -2456,7 +2456,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
bool mmap_locked = true;
cond_resched();
- if (unlikely(hpage_collapse_test_exit_or_disable(mm)))
+ if (unlikely(collapse_test_exit_or_disable(mm)))
goto breakouterloop;
VM_BUG_ON(khugepaged_scan.address < hstart ||
@@ -2469,12 +2469,12 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
mmap_read_unlock(mm);
mmap_locked = false;
- *result = hpage_collapse_scan_file(mm,
+ *result = collapse_scan_file(mm,
khugepaged_scan.address, file, pgoff, cc);
fput(file);
if (*result == SCAN_PTE_MAPPED_HUGEPAGE) {
mmap_read_lock(mm);
- if (hpage_collapse_test_exit_or_disable(mm))
+ if (collapse_test_exit_or_disable(mm))
goto breakouterloop;
*result = collapse_pte_mapped_thp(mm,
khugepaged_scan.address, false);
@@ -2483,7 +2483,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
mmap_read_unlock(mm);
}
} else {
- *result = hpage_collapse_scan_pmd(mm, vma,
+ *result = collapse_scan_pmd(mm, vma,
khugepaged_scan.address, &mmap_locked, cc);
}
@@ -2516,7 +2516,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
* Release the current mm_slot if this mm is about to die, or
* if we scanned all vmas of this mm.
*/
- if (hpage_collapse_test_exit(mm) || !vma) {
+ if (collapse_test_exit(mm) || !vma) {
/*
* Make sure that if mm_users is reaching zero while
* khugepaged runs here, khugepaged_exit will find
@@ -2567,8 +2567,8 @@ static void khugepaged_do_scan(struct collapse_control *cc)
pass_through_head++;
if (khugepaged_has_work() &&
pass_through_head < 2)
- progress += khugepaged_scan_mm_slot(pages - progress,
- &result, cc);
+ progress += collapse_scan_mm_slot(pages - progress,
+ &result, cc);
else
progress = pages;
spin_unlock(&khugepaged_mm_lock);
@@ -2807,12 +2807,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
mmap_read_unlock(mm);
mmap_locked = false;
- result = hpage_collapse_scan_file(mm, addr, file, pgoff,
- cc);
+ result = collapse_scan_file(mm, addr, file, pgoff, cc);
fput(file);
} else {
- result = hpage_collapse_scan_pmd(mm, vma, addr,
- &mmap_locked, cc);
+ result = collapse_scan_pmd(mm, vma, addr,
+ &mmap_locked, cc);
}
if (!mmap_locked)
*lock_dropped = true;
diff --git a/mm/mremap.c b/mm/mremap.c
index 672264807db6..db31d0231e05 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -244,7 +244,7 @@ static int move_ptes(struct pagetable_move_control *pmc,
goto out;
}
/*
- * Now new_pte is none, so hpage_collapse_scan_file() path can not find
+ * Now new_pte is none, so collapse_scan_file() path can not find
* this by traversing file->f_mapping, so there is no concurrency with
* retract_page_tables(). In addition, we already hold the exclusive
* mmap_lock, so this new_pte page is stable, so there is no need to get
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 02/16] introduce collapse_single_pmd to unify khugepaged and madvise_collapse
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
The khugepaged daemon and madvise_collapse have two different
implementations that do almost the same thing.
Create collapse_single_pmd to increase code reuse and create an entry
point to these two users.
Refactor madvise_collapse and collapse_scan_mm_slot to use the new
collapse_single_pmd function. This introduces a minor behavioral change
that is most likely an undiscovered bug. The current implementation of
khugepaged tests collapse_test_exit_or_disable before calling
collapse_pte_mapped_thp, but we weren't doing it in the madvise_collapse
case. By unifying these two callers madvise_collapse now also performs
this check. We also modify the return value to be SCAN_ANY_PROCESS which
properly indicates that this process is no longer valid to operate on.
We also guard the khugepaged_pages_collapsed variable to ensure its only
incremented for khugepaged.
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 97 ++++++++++++++++++++++++++-----------------------
1 file changed, 52 insertions(+), 45 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 959be77f2e65..433ea7283488 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -2392,6 +2392,53 @@ static int collapse_scan_file(struct mm_struct *mm, unsigned long addr,
return result;
}
+/*
+ * Try to collapse a single PMD starting at a PMD aligned addr, and return
+ * the results.
+ */
+static int collapse_single_pmd(unsigned long addr,
+ struct vm_area_struct *vma, bool *mmap_locked,
+ struct collapse_control *cc)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ int result;
+ struct file *file;
+ pgoff_t pgoff;
+
+ if (vma_is_anonymous(vma)) {
+ result = collapse_scan_pmd(mm, vma, addr, mmap_locked, cc);
+ goto end;
+ }
+
+ file = get_file(vma->vm_file);
+ pgoff = linear_page_index(vma, addr);
+
+ mmap_read_unlock(mm);
+ *mmap_locked = false;
+ result = collapse_scan_file(mm, addr, file, pgoff, cc);
+ fput(file);
+ if (result != SCAN_PTE_MAPPED_HUGEPAGE)
+ goto end;
+
+ mmap_read_lock(mm);
+ *mmap_locked = true;
+ if (collapse_test_exit_or_disable(mm)) {
+ mmap_read_unlock(mm);
+ *mmap_locked = false;
+ return SCAN_ANY_PROCESS;
+ }
+ result = collapse_pte_mapped_thp(mm, addr, !cc->is_khugepaged);
+ if (result == SCAN_PMD_MAPPED)
+ result = SCAN_SUCCEED;
+ mmap_read_unlock(mm);
+ *mmap_locked = false;
+
+end:
+ if (cc->is_khugepaged && result == SCAN_SUCCEED)
+ ++khugepaged_pages_collapsed;
+ return result;
+}
+
static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,
struct collapse_control *cc)
__releases(&khugepaged_mm_lock)
@@ -2462,34 +2509,9 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,
VM_BUG_ON(khugepaged_scan.address < hstart ||
khugepaged_scan.address + HPAGE_PMD_SIZE >
hend);
- if (!vma_is_anonymous(vma)) {
- struct file *file = get_file(vma->vm_file);
- pgoff_t pgoff = linear_page_index(vma,
- khugepaged_scan.address);
-
- mmap_read_unlock(mm);
- mmap_locked = false;
- *result = collapse_scan_file(mm,
- khugepaged_scan.address, file, pgoff, cc);
- fput(file);
- if (*result == SCAN_PTE_MAPPED_HUGEPAGE) {
- mmap_read_lock(mm);
- if (collapse_test_exit_or_disable(mm))
- goto breakouterloop;
- *result = collapse_pte_mapped_thp(mm,
- khugepaged_scan.address, false);
- if (*result == SCAN_PMD_MAPPED)
- *result = SCAN_SUCCEED;
- mmap_read_unlock(mm);
- }
- } else {
- *result = collapse_scan_pmd(mm, vma,
- khugepaged_scan.address, &mmap_locked, cc);
- }
-
- if (*result == SCAN_SUCCEED)
- ++khugepaged_pages_collapsed;
+ *result = collapse_single_pmd(khugepaged_scan.address,
+ vma, &mmap_locked, cc);
/* move to next address */
khugepaged_scan.address += HPAGE_PMD_SIZE;
progress += HPAGE_PMD_NR;
@@ -2801,35 +2823,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
}
mmap_assert_locked(mm);
- if (!vma_is_anonymous(vma)) {
- struct file *file = get_file(vma->vm_file);
- pgoff_t pgoff = linear_page_index(vma, addr);
- mmap_read_unlock(mm);
- mmap_locked = false;
- result = collapse_scan_file(mm, addr, file, pgoff, cc);
- fput(file);
- } else {
- result = collapse_scan_pmd(mm, vma, addr,
- &mmap_locked, cc);
- }
+ result = collapse_single_pmd(addr, vma, &mmap_locked, cc);
+
if (!mmap_locked)
*lock_dropped = true;
-handle_result:
switch (result) {
case SCAN_SUCCEED:
case SCAN_PMD_MAPPED:
++thps;
break;
- case SCAN_PTE_MAPPED_HUGEPAGE:
- BUG_ON(mmap_locked);
- mmap_read_lock(mm);
- result = collapse_pte_mapped_thp(mm, addr, true);
- mmap_read_unlock(mm);
- goto handle_result;
/* Whitelisted set of results where continuing OK */
case SCAN_NO_PTE_TABLE:
+ case SCAN_PTE_MAPPED_HUGEPAGE:
case SCAN_PTE_NON_PRESENT:
case SCAN_PTE_UFFD_WP:
case SCAN_LACK_REFERENCED_PAGE:
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 03/16] khugepaged: generalize hugepage_vma_revalidate for mTHP support
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
For khugepaged to support different mTHP orders, we must generalize this
to check if the PMD is not shared by another VMA and that the order is
enabled.
No functional change in this patch. Also correct a comment about the
functionality of the revalidation.
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Acked-by: David Hildenbrand <david@redhat.com>
Co-developed-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 433ea7283488..69fc6b41f010 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -892,14 +892,13 @@ static int collapse_find_target_node(struct collapse_control *cc)
/*
* If mmap_lock temporarily dropped, revalidate vma
- * before taking mmap_lock.
+ * after taking the mmap_lock again.
* Returns enum scan_result value.
*/
static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
- bool expect_anon,
- struct vm_area_struct **vmap,
- struct collapse_control *cc)
+ bool expect_anon, struct vm_area_struct **vmap,
+ struct collapse_control *cc, unsigned int order)
{
struct vm_area_struct *vma;
enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :
@@ -912,15 +911,16 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
if (!vma)
return SCAN_VMA_NULL;
+ /* Always check the PMD order to ensure its not shared by another VMA */
if (!thp_vma_suitable_order(vma, address, PMD_ORDER))
return SCAN_ADDRESS_RANGE;
- if (!thp_vma_allowable_order(vma, vma->vm_flags, type, PMD_ORDER))
+ if (!thp_vma_allowable_orders(vma, vma->vm_flags, type, BIT(order)))
return SCAN_VMA_CHECK;
/*
* Anon VMA expected, the address may be unmapped then
* remapped to file after khugepaged reaquired the mmap_lock.
*
- * thp_vma_allowable_order may return true for qualified file
+ * thp_vma_allowable_orders may return true for qualified file
* vmas.
*/
if (expect_anon && (!(*vmap)->anon_vma || !vma_is_anonymous(*vmap)))
@@ -1117,7 +1117,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
goto out_nolock;
mmap_read_lock(mm);
- result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
+ result = hugepage_vma_revalidate(mm, address, true, &vma, cc,
+ HPAGE_PMD_ORDER);
if (result != SCAN_SUCCEED) {
mmap_read_unlock(mm);
goto out_nolock;
@@ -1151,7 +1152,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
* mmap_lock.
*/
mmap_write_lock(mm);
- result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
+ result = hugepage_vma_revalidate(mm, address, true, &vma, cc,
+ HPAGE_PMD_ORDER);
if (result != SCAN_SUCCEED)
goto out_up_write;
/* check if the pmd is still valid */
@@ -2814,7 +2816,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
mmap_read_lock(mm);
mmap_locked = true;
result = hugepage_vma_revalidate(mm, addr, false, &vma,
- cc);
+ cc, HPAGE_PMD_ORDER);
if (result != SCAN_SUCCEED) {
last_fail = result;
goto out_nolock;
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 04/16] khugepaged: generalize alloc_charge_folio()
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
From: Dev Jain <dev.jain@arm.com>
Pass order to alloc_charge_folio() and update mTHP statistics.
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Acked-by: David Hildenbrand <david@redhat.com>
Co-developed-by: Nico Pache <npache@redhat.com>
Signed-off-by: Nico Pache <npache@redhat.com>
Signed-off-by: Dev Jain <dev.jain@arm.com>
---
Documentation/admin-guide/mm/transhuge.rst | 8 ++++++++
include/linux/huge_mm.h | 2 ++
mm/huge_memory.c | 4 ++++
mm/khugepaged.c | 17 +++++++++++------
4 files changed, 25 insertions(+), 6 deletions(-)
diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
index 5fbc3d89bb07..c51932e6275d 100644
--- a/Documentation/admin-guide/mm/transhuge.rst
+++ b/Documentation/admin-guide/mm/transhuge.rst
@@ -639,6 +639,14 @@ anon_fault_fallback_charge
instead falls back to using huge pages with lower orders or
small pages even though the allocation was successful.
+collapse_alloc
+ is incremented every time a huge page is successfully allocated for a
+ khugepaged collapse.
+
+collapse_alloc_failed
+ is incremented every time a huge page allocation fails during a
+ khugepaged collapse.
+
zswpout
is incremented every time a huge page is swapped out to zswap in one
piece without splitting.
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 21162493a0a0..f93365e182b4 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -128,6 +128,8 @@ enum mthp_stat_item {
MTHP_STAT_ANON_FAULT_ALLOC,
MTHP_STAT_ANON_FAULT_FALLBACK,
MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE,
+ MTHP_STAT_COLLAPSE_ALLOC,
+ MTHP_STAT_COLLAPSE_ALLOC_FAILED,
MTHP_STAT_ZSWPOUT,
MTHP_STAT_SWPIN,
MTHP_STAT_SWPIN_FALLBACK,
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 8db0d81fca40..c1e1e91b0e61 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -621,6 +621,8 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC);
DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK);
DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
+DEFINE_MTHP_STAT_ATTR(collapse_alloc, MTHP_STAT_COLLAPSE_ALLOC);
+DEFINE_MTHP_STAT_ATTR(collapse_alloc_failed, MTHP_STAT_COLLAPSE_ALLOC_FAILED);
DEFINE_MTHP_STAT_ATTR(zswpout, MTHP_STAT_ZSWPOUT);
DEFINE_MTHP_STAT_ATTR(swpin, MTHP_STAT_SWPIN);
DEFINE_MTHP_STAT_ATTR(swpin_fallback, MTHP_STAT_SWPIN_FALLBACK);
@@ -686,6 +688,8 @@ static struct attribute *any_stats_attrs[] = {
#endif
&split_attr.attr,
&split_failed_attr.attr,
+ &collapse_alloc_attr.attr,
+ &collapse_alloc_failed_attr.attr,
NULL,
};
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 69fc6b41f010..8599c7fa112e 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1061,21 +1061,26 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,
}
static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,
- struct collapse_control *cc)
+ struct collapse_control *cc, unsigned int order)
{
gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
GFP_TRANSHUGE);
int node = collapse_find_target_node(cc);
struct folio *folio;
- folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, &cc->alloc_nmask);
+ folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask);
if (!folio) {
*foliop = NULL;
- count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
+ if (order == HPAGE_PMD_ORDER)
+ count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
+ count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC_FAILED);
return SCAN_ALLOC_HUGE_PAGE_FAIL;
}
- count_vm_event(THP_COLLAPSE_ALLOC);
+ if (order == HPAGE_PMD_ORDER)
+ count_vm_event(THP_COLLAPSE_ALLOC);
+ count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC);
+
if (unlikely(mem_cgroup_charge(folio, mm, gfp))) {
folio_put(folio);
*foliop = NULL;
@@ -1112,7 +1117,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
*/
mmap_read_unlock(mm);
- result = alloc_charge_folio(&folio, mm, cc);
+ result = alloc_charge_folio(&folio, mm, cc, HPAGE_PMD_ORDER);
if (result != SCAN_SUCCEED)
goto out_nolock;
@@ -1872,7 +1877,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem);
VM_BUG_ON(start & (HPAGE_PMD_NR - 1));
- result = alloc_charge_folio(&new_folio, mm, cc);
+ result = alloc_charge_folio(&new_folio, mm, cc, HPAGE_PMD_ORDER);
if (result != SCAN_SUCCEED)
goto out;
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 05/16] khugepaged: introduce is_mthp_order helper
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
In order to add mTHP support, we will often be checking if a given order
is a mTHP or PMD order. Lets create a simple helper function to keep the
code clean and readable.
Suggested-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 8599c7fa112e..9c041141b2e3 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -338,6 +338,11 @@ static bool pte_none_or_zero(pte_t pte)
return pte_present(pte) && is_zero_pfn(pte_pfn(pte));
}
+static bool is_mthp_order(unsigned int order)
+{
+ return order != HPAGE_PMD_ORDER;
+}
+
int hugepage_madvise(struct vm_area_struct *vma,
vm_flags_t *vm_flags, int advice)
{
@@ -1071,13 +1076,13 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,
folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask);
if (!folio) {
*foliop = NULL;
- if (order == HPAGE_PMD_ORDER)
+ if (!is_mthp_order(order))
count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC_FAILED);
return SCAN_ALLOC_HUGE_PAGE_FAIL;
}
- if (order == HPAGE_PMD_ORDER)
+ if (!is_mthp_order(order))
count_vm_event(THP_COLLAPSE_ALLOC);
count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC);
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 06/16] khugepaged: generalize __collapse_huge_page_* for mTHP support
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
generalize the order of the __collapse_huge_page_* functions
to support future mTHP collapse.
mTHP collapse will not honor the khugepaged_max_ptes_shared or
khugepaged_max_ptes_swap parameters, and will fail if it encounters a
shared or swapped entry.
No functional changes in this patch.
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: David Hildenbrand <david@redhat.com>
Co-developed-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 78 ++++++++++++++++++++++++++++++-------------------
1 file changed, 48 insertions(+), 30 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 9c041141b2e3..8dab49c53128 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -541,25 +541,25 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte,
}
static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
- unsigned long start_addr,
- pte_t *pte,
- struct collapse_control *cc,
- struct list_head *compound_pagelist)
+ unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
+ unsigned int order, struct list_head *compound_pagelist)
{
struct page *page = NULL;
struct folio *folio = NULL;
unsigned long addr = start_addr;
pte_t *_pte;
int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0;
+ const unsigned long nr_pages = 1UL << order;
+ int max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order);
- for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
+ for (_pte = pte; _pte < pte + nr_pages;
_pte++, addr += PAGE_SIZE) {
pte_t pteval = ptep_get(_pte);
if (pte_none_or_zero(pteval)) {
++none_or_zero;
if (!userfaultfd_armed(vma) &&
(!cc->is_khugepaged ||
- none_or_zero <= khugepaged_max_ptes_none)) {
+ none_or_zero <= max_ptes_none)) {
continue;
} else {
result = SCAN_EXCEED_NONE_PTE;
@@ -587,8 +587,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
/* See collapse_scan_pmd(). */
if (folio_maybe_mapped_shared(folio)) {
++shared;
- if (cc->is_khugepaged &&
- shared > khugepaged_max_ptes_shared) {
+ /*
+ * TODO: Support shared pages without leading to further
+ * mTHP collapses. Currently bringing in new pages via
+ * shared may cause a future higher order collapse on a
+ * rescan of the same range.
+ */
+ if (is_mthp_order(order) || (cc->is_khugepaged &&
+ shared > khugepaged_max_ptes_shared)) {
result = SCAN_EXCEED_SHARED_PTE;
count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
goto out;
@@ -681,18 +687,18 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
}
static void __collapse_huge_page_copy_succeeded(pte_t *pte,
- struct vm_area_struct *vma,
- unsigned long address,
- spinlock_t *ptl,
- struct list_head *compound_pagelist)
+ struct vm_area_struct *vma, unsigned long address,
+ spinlock_t *ptl, unsigned int order,
+ struct list_head *compound_pagelist)
{
- unsigned long end = address + HPAGE_PMD_SIZE;
+ unsigned long end = address + (PAGE_SIZE << order);
struct folio *src, *tmp;
pte_t pteval;
pte_t *_pte;
unsigned int nr_ptes;
+ const unsigned long nr_pages = 1UL << order;
- for (_pte = pte; _pte < pte + HPAGE_PMD_NR; _pte += nr_ptes,
+ for (_pte = pte; _pte < pte + nr_pages; _pte += nr_ptes,
address += nr_ptes * PAGE_SIZE) {
nr_ptes = 1;
pteval = ptep_get(_pte);
@@ -745,13 +751,11 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
}
static void __collapse_huge_page_copy_failed(pte_t *pte,
- pmd_t *pmd,
- pmd_t orig_pmd,
- struct vm_area_struct *vma,
- struct list_head *compound_pagelist)
+ pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma,
+ unsigned int order, struct list_head *compound_pagelist)
{
spinlock_t *pmd_ptl;
-
+ const unsigned long nr_pages = 1UL << order;
/*
* Re-establish the PMD to point to the original page table
* entry. Restoring PMD needs to be done prior to releasing
@@ -765,7 +769,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,
* Release both raw and compound pages isolated
* in __collapse_huge_page_isolate.
*/
- release_pte_pages(pte, pte + HPAGE_PMD_NR, compound_pagelist);
+ release_pte_pages(pte, pte + nr_pages, compound_pagelist);
}
/*
@@ -785,16 +789,16 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,
*/
static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio,
pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma,
- unsigned long address, spinlock_t *ptl,
+ unsigned long address, spinlock_t *ptl, unsigned int order,
struct list_head *compound_pagelist)
{
unsigned int i;
int result = SCAN_SUCCEED;
-
+ const unsigned long nr_pages = 1UL << order;
/*
* Copying pages' contents is subject to memory poison at any iteration.
*/
- for (i = 0; i < HPAGE_PMD_NR; i++) {
+ for (i = 0; i < nr_pages; i++) {
pte_t pteval = ptep_get(pte + i);
struct page *page = folio_page(folio, i);
unsigned long src_addr = address + i * PAGE_SIZE;
@@ -813,10 +817,10 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio,
if (likely(result == SCAN_SUCCEED))
__collapse_huge_page_copy_succeeded(pte, vma, address, ptl,
- compound_pagelist);
+ order, compound_pagelist);
else
__collapse_huge_page_copy_failed(pte, pmd, orig_pmd, vma,
- compound_pagelist);
+ order, compound_pagelist);
return result;
}
@@ -989,13 +993,12 @@ static int check_pmd_still_valid(struct mm_struct *mm,
* Returns result: if not SCAN_SUCCEED, mmap_lock has been released.
*/
static int __collapse_huge_page_swapin(struct mm_struct *mm,
- struct vm_area_struct *vma,
- unsigned long start_addr, pmd_t *pmd,
- int referenced)
+ struct vm_area_struct *vma, unsigned long start_addr,
+ pmd_t *pmd, int referenced, unsigned int order)
{
int swapped_in = 0;
vm_fault_t ret = 0;
- unsigned long addr, end = start_addr + (HPAGE_PMD_NR * PAGE_SIZE);
+ unsigned long addr, end = start_addr + (PAGE_SIZE << order);
int result;
pte_t *pte = NULL;
spinlock_t *ptl;
@@ -1027,6 +1030,19 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,
pte_present(vmf.orig_pte))
continue;
+ /*
+ * TODO: Support swapin without leading to further mTHP
+ * collapses. Currently bringing in new pages via swapin may
+ * cause a future higher order collapse on a rescan of the same
+ * range.
+ */
+ if (is_mthp_order(order)) {
+ pte_unmap(pte);
+ mmap_read_unlock(mm);
+ result = SCAN_EXCEED_SWAP_PTE;
+ goto out;
+ }
+
vmf.pte = pte;
vmf.ptl = ptl;
ret = do_swap_page(&vmf);
@@ -1147,7 +1163,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
* that case. Continuing to collapse causes inconsistency.
*/
result = __collapse_huge_page_swapin(mm, vma, address, pmd,
- referenced);
+ referenced, HPAGE_PMD_ORDER);
if (result != SCAN_SUCCEED)
goto out_nolock;
}
@@ -1195,6 +1211,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl);
if (pte) {
result = __collapse_huge_page_isolate(vma, address, pte, cc,
+ HPAGE_PMD_ORDER,
&compound_pagelist);
spin_unlock(pte_ptl);
} else {
@@ -1225,6 +1242,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
result = __collapse_huge_page_copy(pte, folio, pmd, _pmd,
vma, address, pte_ptl,
+ HPAGE_PMD_ORDER,
&compound_pagelist);
pte_unmap(pte);
if (unlikely(result != SCAN_SUCCEED))
--
2.51.1
^ permalink raw reply related
* [PATCH v13 mm-new 07/16] khugepaged: introduce collapse_max_ptes_none helper function
From: Nico Pache @ 2025-12-01 17:46 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel, linux-mm, linux-doc
Cc: david, ziy, baolin.wang, lorenzo.stoakes, Liam.Howlett,
ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <20251201174627.23295-1-npache@redhat.com>
The current mechanism for determining mTHP collapse scales the
khugepaged_max_ptes_none value based on the target order. This
introduces an undesirable feedback loop, or "creep", when max_ptes_none
is set to a value greater than HPAGE_PMD_NR / 2.
With this configuration, a successful collapse to order N will populate
enough pages to satisfy the collapse condition on order N+1 on the next
scan. This leads to unnecessary work and memory churn.
To fix this issue introduce a helper function that will limit mTHP
collapse support to two max_ptes_none values, 0 and HPAGE_PMD_NR - 1.
This effectively supports two modes:
- max_ptes_none=0: never introduce new none-pages for mTHP collapse.
- max_ptes_none=511 (on 4k pagesz): Always collapse to the highest
available mTHP order.
This removes the possiblilty of "creep", while not modifying any uAPI
expectations. A warning will be emitted if any non-supported
max_ptes_none value is configured with mTHP enabled.
The limits can be ignored by passing full_scan=true, this is useful for
madvise_collapse (which ignores limits), or in the case of
collapse_scan_pmd(), allows the full PMD to be scanned when mTHP
collapse is available.
Signed-off-by: Nico Pache <npache@redhat.com>
---
mm/khugepaged.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 42 insertions(+), 1 deletion(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 8dab49c53128..f425238d5d4f 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -463,6 +463,44 @@ void __khugepaged_enter(struct mm_struct *mm)
wake_up_interruptible(&khugepaged_wait);
}
+/**
+ * collapse_max_ptes_none - Calculate maximum allowed empty PTEs for collapse
+ * @order: The folio order being collapsed to
+ * @full_scan: Whether this is a full scan (ignore limits)
+ *
+ * For madvise-triggered collapses (full_scan=true), all limits are bypassed
+ * and allow up to HPAGE_PMD_NR - 1 empty PTEs.
+ *
+ * For PMD-sized collapses (order == HPAGE_PMD_ORDER), use the configured
+ * khugepaged_max_ptes_none value.
+ *
+ * For mTHP collapses, we currently only support khugepaged_max_pte_none values
+ * of 0 or (HPAGE_PMD_NR - 1). Any other value will emit a warning and no mTHP
+ * collapse will be attempted
+ *
+ * Return: Maximum number of empty PTEs allowed for the collapse operation
+ */
+static unsigned int collapse_max_ptes_none(unsigned int order, bool full_scan)
+{
+ /* ignore max_ptes_none limits */
+ if (full_scan)
+ return HPAGE_PMD_NR - 1;
+
+ if (!is_mthp_order(order))
+ return khugepaged_max_ptes_none;
+
+ /* Zero/non-present collapse disabled. */
+ if (!khugepaged_max_ptes_none)
+ return 0;
+
+ if (khugepaged_max_ptes_none == HPAGE_PMD_NR - 1)
+ return (1 << order) - 1;
+
+ pr_warn_once("mTHP collapse only supports max_ptes_none values of 0 or %d\n",
+ HPAGE_PMD_NR - 1);
+ return -EINVAL;
+}
+
void khugepaged_enter_vma(struct vm_area_struct *vma,
vm_flags_t vm_flags)
{
@@ -550,7 +588,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
pte_t *_pte;
int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0;
const unsigned long nr_pages = 1UL << order;
- int max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order);
+ int max_ptes_none = collapse_max_ptes_none(order, !cc->is_khugepaged);
+
+ if (max_ptes_none == -EINVAL)
+ goto out;
for (_pte = pte; _pte < pte + nr_pages;
_pte++, addr += PAGE_SIZE) {
--
2.51.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox