* Re: [PATCH v8 20/28] KVM: arm64: Add clock support for the pKVM hyp
From: Vincent Donnefort @ 2025-11-20 11:36 UTC (permalink / raw)
To: Marc Zyngier, g
Cc: rostedt, mhiramat, mathieu.desnoyers, linux-trace-kernel,
oliver.upton, joey.gouly, suzuki.poulose, yuzenghui, kvmarm,
linux-arm-kernel, jstultz, qperret, will, aneesh.kumar,
kernel-team, linux-kernel
In-Reply-To: <86cy5erprc.wl-maz@kernel.org>
[...]
> > +/* Using host provided data. Do not use for anything else than debugging. */
> > +u64 trace_clock(void)
> > +{
> > + struct clock_data *clock = &trace_clock_data;
> > + u64 bank = smp_load_acquire(&clock->cur);
> > + u64 cyc, ns;
> > +
> > + cyc = __arch_counter_get_cntpct() - clock->data[bank].epoch_cyc;
>
> I can only imagine that you don't care about the broken systems that
> do not have a monotonic counter, and that will happily have their
> counter swing back and forth?
I haven't used the _stable() variant for the affected devices... Hum this will
not be trivial from the hypervisor. I'll see what I can do.
>
> Also, you may want to document why you are using the physical counter
> instead of the virtual one. I guess that you don't want CNTVOFF_EL2 to
> apply when HCR_EL2.E2H==0, but given that you never allow anything
> else for pKVM, this is slightly odd.
You mean I might as well use __arch_counter_get_cntvct() as CNTVOFF_EL2 will
always be ignored in the pKVM case?
>
> Thanks,
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply
* Re: [RFC PATCH 0/8] xfs: single block atomic writes for buffered IO
From: Ojaswin Mujoo @ 2025-11-20 10:37 UTC (permalink / raw)
To: Dave Chinner
Cc: John Garry, Ritesh Harjani, Christoph Hellwig, Christian Brauner,
djwong, tytso, willy, dchinner, linux-xfs, linux-kernel,
linux-ext4, linux-fsdevel, linux-mm, jack, nilay, martin.petersen,
rostedt, axboe, linux-block, linux-trace-kernel
In-Reply-To: <aRuKz4F3xATf8IUp@dread.disaster.area>
On Tue, Nov 18, 2025 at 07:51:27AM +1100, Dave Chinner wrote:
> On Mon, Nov 17, 2025 at 10:59:55AM +0000, John Garry wrote:
> > On 16/11/2025 08:11, Dave Chinner wrote:
> > > > This patch set focuses on HW accelerated single block atomic writes with
> > > > buffered IO, to get some early reviews on the core design.
> > > What hardware acceleration? Hardware atomic writes are do not make
> > > IO faster; they only change IO failure semantics in certain corner
> > > cases.
> >
> > I think that he references using REQ_ATOMIC-based bio vs xfs software-based
> > atomic writes (which reuse the CoW infrastructure). And the former is
> > considerably faster from my testing (for DIO, obvs). But the latter has not
> > been optimized.
>
Hi Dave,
Thanks for the review and insights.
Going through the discussions in previous emails and this email, I
understand that there are 2 main points/approaches that you've
mentioned:
1. Using COW extents to track atomic ranges
- Discussed inline below.
2. Using write-through for RWF_ATOMIC buffered-IO (Suggested in [1])
- [1] https://lore.kernel.org/linux-ext4/aRmHRk7FGD4nCT0s@dread.disaster.area/
- I will respond inline in the above thread.
> For DIO, REQ_ATOMIC IO will generally be faster than the software
> fallback because no page cache interactions or data copy is required
> by the DIO REQ_ATOMIC fast path.
>
> But we are considering buffered writes, which *must* do a data copy,
> and so the behaviour and performance differential of doing a COW vs
> trying to force writeback to do REQ_ATOMIC IO is going to be much
> different.
>
> Consider that the way atomic buffered writes have been implemented
> in writeback - turning off all folio and IO merging. This means
> writeback efficiency of atomic writes is going to be horrendous
> compared to COW writes that don't use REQ_ATOMIC.
Yes, I agree that it is a bit of an overkill.
>
> Further, REQ_ATOMIC buffered writes need to turn off delayed
> allocation because if you can't allocate aligned extents then the
> atomic write can *never* be performed. Hence we have to allocate up
> front where we can return errors to userspace immediately, rather
> than just reserve space and punt allocation to writeback. i.e. we
> have to avoid the situation where we have dirty "atomic" data in the
> page cache that cannot be written because physical allocation fails.
>
> The likely outcome of turning off delalloc is that it further
> degrades buffered atomic write writeback efficiency because it
> removes the ability for the filesystem to optimise physical locality
> of writeback IO. e.g. adjacent allocation across multiple small
> files or packing of random writes in a single file to allow them to
> merge at the block layer into one big IO...
>
> REQ_ATOMIC is a natural fit for DIO because DIO is largely a "one
> write syscall, one physical IO" style interface. Buffered writes,
> OTOH, completely decouples application IO from physical IO, and so
> there is no real "atomic" connection between the data being written
> into the page caceh and the physical IO that is performed at some
> time later.
>
> This decoupling of physical IO is what brings all the problems and
> inefficiencies. The filesystem being able to mark the RWF_ATOMIC
> write range as a COW range at submission time creates a natural
> "atomic IO" behaviour without requiring the page cache or writeback
> to even care that the data needs to be written atomically.
>
> From there, we optimise the COW IO path to record that
> the new COW extent was created for the purpose of an atomic write.
> Then when we go to write back data over that extent, the filesystem
> can chose to do a REQ_ATOMIC write to do an atomic overwrite instead
> of allocating a new extent and swapping the BMBT extent pointers at
> IO completion time.
>
> We really don't care if 4x16kB adjacent RWF_ATOMIC writes are
> submitted as 1x64kB REQ_ATOMIC IO or 4 individual 16kB REQ_ATOMIC
> IOs. The former is much more efficient from an IO perspective, and
> the COW path can actually optimise for this because it can track the
> atomic write ranges in cache exactly. If the range is larger (or
> unaligned) than what REQ_ATOMIC can handle, we use COW writeback to
> optimise for maximum writeback bandwidth, otherwise we use
> REQ_ATOMIC to optimise for minimum writeback submission and
> completion overhead...
Okay IIUC, you are suggesting that, instead of tracking the atomic
ranges in page cache and ifs, lets move that to the filesystem, for
example in XFS we can:
1. In write iomap_begin path, for RWF_ATOMIC, create a COW extent and
mark it as atomic.
2. Carry on with the memcpy to folio and finish the write path.
3. During writeback, at XFS can detect that there is a COW atomic
extent. It can then:
3.1 See that it is an overlap that can be done with REQ_ATOMIC
directly
3.2 Else, finish the atomic IO in software emulated way just like we
do for direct IO currently.
I believe the above example with XFS can also be extended to a FS like
ext4 without needing COW range, as long as we can ensure that we always
meet the conditions for REQ_ATOMIC during writeback (example by using
bigalloc for aligned extents and being careful not to cross the atomic
write limits)
>
> IOWs, I think that for XFS (and other COW-capable filesystems) we
> should be looking at optimising the COW IO path to use REQ_ATOMIC
> where appropriate to create a direct overwrite fast path for
> RWF_ATOMIC buffered writes. This seems a more natural and a lot less
> intrusive than trying to blast through the page caceh abstractions
> to directly couple userspace IO boundaries to physical writeback IO
> boundaries...
I agree that this approach avoids bloating the page cache and ifs layers
with RWF_ATOMIC implementation details. That being said, the task of
managing the atomic ranges is now pushed down to the FS and is no longer
generic which might introduce friction in onboarding of new FSes in the
future. Regardless, from the discussion, I believe at this point we are
okay to make that trade-off.
Let me take some time to look into the XFS COW paths and try to implement
this approach. Thanks for the suggestion!
Regards,
ojaswin
>
> -Dave.
> --
> Dave Chinner
> david@fromorbit.com
^ permalink raw reply
* Re: [PATCH V2 2/2] mm/khugepaged: map dirty/writeback pages failures to EAGAIN
From: Dev Jain @ 2025-11-20 9:55 UTC (permalink / raw)
To: Garg, Shivank, Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Barry Song, Lance Yang, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Zach O'Keefe, linux-mm, linux-kernel,
linux-trace-kernel
In-Reply-To: <33476cb4-318b-49db-9cc1-a354eca9e883@amd.com>
On 20/11/25 1:47 pm, Garg, Shivank wrote:
>
> On 11/20/2025 1:33 PM, Dev Jain wrote:
>> On 20/11/25 12:20 pm, Shivank Garg wrote:
>> SCAN_PAGE_NOT_CLEAN is confusing - NOT_CLEAN literally means dirty, so why not SCAN_PAGE_DIRTY?
>> Or SCAN_PAGE_DIRTY_OR_UNDER_WRITEBACK? Since folio_test_writeback() is true as a result of
>> the folio being dirty, maybe just SCAN_PAGE_DIRTY can do.
>>
>> Reviewed-by: Dev Jain <dev.jain@arm.com>
>>
> Thanks for the review.
>
> I chose not to use SCAN_PAGE_DIRTY because dirty and writeback have different meanings[1]:
>
> Dirty: Memory that is waiting to be written back to disk
> Writeback: Memory that is actively being written back to disk
>
> [1] https://www.kernel.org/doc/Documentation/filesystems/proc.txt
>
> IIUC, a page under writeback is no longer dirty, so using SCAN_PAGE_DIRTY would be misleading
> for pages in the writeback state.
>
> I considered SCAN_PAGE_DIRTY_OR_WRITEBACK initially but felt it was too long.
>
> SCAN_PAGE_NOT_CLEAN covers both states that indicate the page is not in a clean/stable
> state suitable for collapse.
>
> [1] https://www.kernel.org/doc/Documentation/filesystems/proc.txt
Alright, makes sense.
>
> Thanks,
> Shivank
^ permalink raw reply
* Re: [PATCH v8 19/28] KVM: arm64: Support unaligned fixmap in the pKVM hyp
From: Vincent Donnefort @ 2025-11-20 9:55 UTC (permalink / raw)
To: Marc Zyngier
Cc: rostedt, mhiramat, mathieu.desnoyers, linux-trace-kernel,
oliver.upton, joey.gouly, suzuki.poulose, yuzenghui, kvmarm,
linux-arm-kernel, jstultz, qperret, will, aneesh.kumar,
kernel-team, linux-kernel
In-Reply-To: <86ecpurq13.wl-maz@kernel.org>
Hi Marc,
Thanks for having a look at the series.
On Wed, Nov 19, 2025 at 03:38:32PM +0000, Marc Zyngier wrote:
> On Fri, 07 Nov 2025 09:38:31 +0000,
> Vincent Donnefort <vdonnefort@google.com> wrote:
> >
> > Return the fixmap VA with the page offset, instead of the page base
> > address. This allows to use hyp_fixmap_map() seamlessly regardless of
> > the address alignment.
> >
> > Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
> >
> > diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
> > index ae8391baebc3..75014dc7d82e 100644
> > --- a/arch/arm64/kvm/hyp/nvhe/mm.c
> > +++ b/arch/arm64/kvm/hyp/nvhe/mm.c
> > @@ -239,7 +239,7 @@ static void *fixmap_map_slot(struct hyp_fixmap_slot *slot, phys_addr_t phys)
> > WRITE_ONCE(*ptep, pte);
> > dsb(ishst);
> >
> > - return (void *)slot->addr;
> > + return (void *)slot->addr + offset_in_page(phys);
> > }
> >
> > void *hyp_fixmap_map(phys_addr_t phys)
>
> This looks slightly odd. fixmap_map_slot() should return *a slot*, not
> something in it. Can't hyp_fixmap_map() do that instead?
That was fixing hyp_fixblock_map() at the same time. I will move the
offset_in_page() in both callers.
>
> Thanks,
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply
* Re: [PATCH 00/44] Change a lot of min_t() that might mask high bits
From: Lorenzo Stoakes @ 2025-11-20 9:38 UTC (permalink / raw)
To: david.laight.linux
Cc: linux-kernel, Alan Stern, Alexander Viro, Alexei Starovoitov,
Andi Shyti, Andreas Dilger, Andrew Lunn, Andrew Morton,
Andrii Nakryiko, Andy Shevchenko, Ard Biesheuvel,
Arnaldo Carvalho de Melo, Bjorn Helgaas, Borislav Petkov,
Christian Brauner, Christian König, Christoph Hellwig,
Daniel Borkmann, Dan Williams, Dave Hansen, Dave Jiang,
David Ahern, David Hildenbrand, Davidlohr Bueso, David S. Miller,
Dennis Zhou, Eric Dumazet, Greg Kroah-Hartman, Herbert Xu,
Ingo Molnar, Jakub Kicinski, Jakub Sitnicki, James E.J. Bottomley,
Jarkko Sakkinen, Jason A. Donenfeld, Jens Axboe, Jiri Slaby,
Johannes Weiner, John Allen, Jonathan Cameron, Juergen Gross,
Kees Cook, KP Singh, Linus Walleij, Martin K. Petersen,
Matthew Wilcox (Oracle), Mika Westerberg, Mike Rapoport,
Miklos Szeredi, Namhyung Kim, Neal Cardwell, nic_swsd,
OGAWA Hirofumi, Olivia Mackall, Paolo Abeni, Paolo Bonzini,
Peter Huewe, Peter Zijlstra, Rafael J. Wysocki,
Sean Christopherson, Srinivas Kandagatla, Stefano Stabellini,
Steven Rostedt, Tejun Heo, Theodore Ts'o, Thomas Gleixner,
Tom Lendacky, Willem de Bruijn, x86, Yury Norov, amd-gfx, bpf,
cgroups, dri-devel, io-uring, kvm, linux-acpi, linux-block,
linux-crypto, linux-cxl, linux-efi, linux-ext4, linux-fsdevel,
linux-gpio, linux-i2c, linux-integrity, linux-mm, linux-nvme,
linux-pci, linux-perf-users, linux-scsi, linux-serial,
linux-trace-kernel, linux-usb, mptcp, netdev, usb-storage
In-Reply-To: <20251119224140.8616-1-david.laight.linux@gmail.com>
On Wed, Nov 19, 2025 at 10:40:56PM +0000, david.laight.linux@gmail.com wrote:
> From: David Laight <david.laight.linux@gmail.com>
>
> It in not uncommon for code to use min_t(uint, a, b) when one of a or b
> is 64bit and can have a value that is larger than 2^32;
> This is particularly prevelant with:
> uint_var = min_t(uint, uint_var, uint64_expression);
>
> Casts to u8 and u16 are very likely to discard significant bits.
>
> These can be detected at compile time by changing min_t(), for example:
> #define CHECK_SIZE(fn, type, val) \
> BUILD_BUG_ON_MSG(sizeof (val) > sizeof (type) && \
> !statically_true(((val) >> 8 * (sizeof (type) - 1)) < 256), \
> fn "() significant bits of '" #val "' may be discarded")
>
> #define min_t(type, x, y) ({ \
> CHECK_SIZE("min_t", type, x); \
> CHECK_SIZE("min_t", type, y); \
> __cmp_once(min, type, x, y); })
>
> (and similar changes to max_t() and clamp_t().)
Have we made sure that the introduction of these don't cause a combinatorial
explosion like previous min()/max() changes did?
>
> This shows up some real bugs, some unlikely bugs and some false positives.
> In most cases both arguments are unsigned type (just different ones)
> and min_t() can just be replaced by min().
>
> The patches are all independant and are most of the ones needed to
> get the x86-64 kernel I build to compile.
> I've not tried building an allyesconfig or allmodconfig kernel.
Well I have a beefy box at my disposal so tried thiese for you :)
Both allyesconfig & allmodconfig works fine for x86-64 (I tried both for good
measure)
> I've also not included the patch to minmax.h itself.
>
> I've tried to put the patches that actually fix things first.
> The last one is 0009.
>
> I gave up on fixing sched/fair.c - it is too broken for a single patch!
> The patch for net/ipv4/tcp.c is also absent because do_tcp_getsockopt()
> needs multiple/larger changes to make it 'sane'.
I guess this isn't broken per se there just retain min_t()/max_t() right?
>
> I've had to trim the 124 maintainers/lists that get_maintainer.pl finds
> from 124 to under 100 to be able to send the cover letter.
> The individual patches only go to the addresses found for the associated files.
> That reduces the number of emails to a less unsane number.
>
> David Laight (44):
> x86/asm/bitops: Change the return type of variable__ffs() to unsigned
> int
> ext4: Fix saturation of 64bit inode times for old filesystems
> perf: Fix branch stack callchain limit
> io_uring/net: Change some dubious min_t()
> ipc/msg: Fix saturation of percpu counts in msgctl_info()
> bpf: Verifier, remove some unusual uses of min_t() and max_t()
> net/core/flow_dissector: Fix cap of __skb_flow_dissect() return value.
> net: ethtool: Use min3() instead of nested min_t(u16,...)
> ipv6: __ip6_append_data() don't abuse max_t() casts
> x86/crypto: ctr_crypt() use min() instead of min_t()
> arch/x96/kvm: use min() instead of min_t()
> block: use min() instead of min_t()
> drivers/acpi: use min() instead of min_t()
> drivers/char/hw_random: use min3() instead of nested min_t()
> drivers/char/tpm: use min() instead of min_t()
> drivers/crypto/ccp: use min() instead of min_t()
> drivers/cxl: use min() instead of min_t()
> drivers/gpio: use min() instead of min_t()
> drivers/gpu/drm/amd: use min() instead of min_t()
> drivers/i2c/busses: use min() instead of min_t()
> drivers/net/ethernet/realtek: use min() instead of min_t()
> drivers/nvme: use min() instead of min_t()
> arch/x86/mm: use min() instead of min_t()
> drivers/nvmem: use min() instead of min_t()
> drivers/pci: use min() instead of min_t()
> drivers/scsi: use min() instead of min_t()
> drivers/tty/vt: use umin() instead of min_t(u16, ...) for row/col
> limits
> drivers/usb/storage: use min() instead of min_t()
> drivers/xen: use min() instead of min_t()
> fs: use min() or umin() instead of min_t()
> block: bvec.h: use min() instead of min_t()
> nodemask: use min() instead of min_t()
> ipc: use min() instead of min_t()
> bpf: use min() instead of min_t()
> bpf_trace: use min() instead of min_t()
> lib/bucket_locks: use min() instead of min_t()
> lib/crypto/mpi: use min() instead of min_t()
> lib/dynamic_queue_limits: use max() instead of max_t()
> mm: use min() instead of min_t()
> net: Don't pass bitfields to max_t()
> net/core: Change loop conditions so min() can be used
> net: use min() instead of min_t()
> net/netlink: Use umin() to avoid min_t(int, ...) discarding high bits
> net/mptcp: Change some dubious min_t(int, ...) to min()
>
> arch/x86/crypto/aesni-intel_glue.c | 3 +-
> arch/x86/include/asm/bitops.h | 18 +++++-------
> arch/x86/kvm/emulate.c | 3 +-
> arch/x86/kvm/lapic.c | 2 +-
> arch/x86/kvm/mmu/mmu.c | 2 +-
> arch/x86/mm/pat/set_memory.c | 12 ++++----
> block/blk-iocost.c | 6 ++--
> block/blk-settings.c | 2 +-
> block/partitions/efi.c | 3 +-
> drivers/acpi/property.c | 2 +-
> drivers/char/hw_random/core.c | 2 +-
> drivers/char/tpm/tpm1-cmd.c | 2 +-
> drivers/char/tpm/tpm_tis_core.c | 4 +--
> drivers/crypto/ccp/ccp-dev.c | 2 +-
> drivers/cxl/core/mbox.c | 2 +-
> drivers/gpio/gpiolib-acpi-core.c | 2 +-
> .../gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c | 4 +--
> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +-
> .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 2 +-
> drivers/i2c/busses/i2c-designware-master.c | 2 +-
> drivers/net/ethernet/realtek/r8169_main.c | 3 +-
> drivers/nvme/host/pci.c | 3 +-
> drivers/nvme/host/zns.c | 3 +-
> drivers/nvmem/core.c | 2 +-
> drivers/pci/probe.c | 3 +-
> drivers/scsi/hosts.c | 2 +-
> drivers/tty/vt/selection.c | 9 +++---
> drivers/usb/storage/protocol.c | 3 +-
> drivers/xen/grant-table.c | 2 +-
> fs/buffer.c | 2 +-
> fs/exec.c | 2 +-
> fs/ext4/ext4.h | 2 +-
> fs/ext4/mballoc.c | 3 +-
> fs/ext4/resize.c | 2 +-
> fs/ext4/super.c | 2 +-
> fs/fat/dir.c | 4 +--
> fs/fat/file.c | 3 +-
> fs/fuse/dev.c | 2 +-
> fs/fuse/file.c | 8 ++---
> fs/splice.c | 2 +-
> include/linux/bvec.h | 3 +-
> include/linux/nodemask.h | 9 +++---
> include/linux/perf_event.h | 2 +-
> include/net/tcp_ecn.h | 5 ++--
> io_uring/net.c | 6 ++--
> ipc/mqueue.c | 4 +--
> ipc/msg.c | 6 ++--
> kernel/bpf/core.c | 4 +--
> kernel/bpf/log.c | 2 +-
> kernel/bpf/verifier.c | 29 +++++++------------
> kernel/trace/bpf_trace.c | 2 +-
> lib/bucket_locks.c | 2 +-
> lib/crypto/mpi/mpicoder.c | 2 +-
> lib/dynamic_queue_limits.c | 2 +-
> mm/gup.c | 4 +--
> mm/memblock.c | 2 +-
> mm/memory.c | 2 +-
> mm/percpu.c | 2 +-
> mm/truncate.c | 3 +-
> mm/vmscan.c | 2 +-
> net/core/datagram.c | 6 ++--
> net/core/flow_dissector.c | 7 ++---
> net/core/net-sysfs.c | 3 +-
> net/core/skmsg.c | 4 +--
> net/ethtool/cmis_cdb.c | 7 ++---
> net/ipv4/fib_trie.c | 2 +-
> net/ipv4/tcp_input.c | 4 +--
> net/ipv4/tcp_output.c | 5 ++--
> net/ipv4/tcp_timer.c | 4 +--
> net/ipv6/addrconf.c | 8 ++---
> net/ipv6/ip6_output.c | 7 +++--
> net/ipv6/ndisc.c | 5 ++--
> net/mptcp/protocol.c | 8 ++---
> net/netlink/genetlink.c | 9 +++---
> net/packet/af_packet.c | 2 +-
> net/unix/af_unix.c | 4 +--
> 76 files changed, 141 insertions(+), 176 deletions(-)
>
> --
> 2.39.5
>
^ permalink raw reply
* Re: [PATCH V2 2/2] mm/khugepaged: map dirty/writeback pages failures to EAGAIN
From: Garg, Shivank @ 2025-11-20 8:17 UTC (permalink / raw)
To: Dev Jain, Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Barry Song, Lance Yang, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Zach O'Keefe, linux-mm, linux-kernel,
linux-trace-kernel
In-Reply-To: <6c1d6b80-d290-4110-9a49-53e7404136bc@arm.com>
On 11/20/2025 1:33 PM, Dev Jain wrote:
>
> On 20/11/25 12:20 pm, Shivank Garg wrote:
> SCAN_PAGE_NOT_CLEAN is confusing - NOT_CLEAN literally means dirty, so why not SCAN_PAGE_DIRTY?
> Or SCAN_PAGE_DIRTY_OR_UNDER_WRITEBACK? Since folio_test_writeback() is true as a result of
> the folio being dirty, maybe just SCAN_PAGE_DIRTY can do.
>
> Reviewed-by: Dev Jain <dev.jain@arm.com>
>
Thanks for the review.
I chose not to use SCAN_PAGE_DIRTY because dirty and writeback have different meanings[1]:
Dirty: Memory that is waiting to be written back to disk
Writeback: Memory that is actively being written back to disk
[1] https://www.kernel.org/doc/Documentation/filesystems/proc.txt
IIUC, a page under writeback is no longer dirty, so using SCAN_PAGE_DIRTY would be misleading
for pages in the writeback state.
I considered SCAN_PAGE_DIRTY_OR_WRITEBACK initially but felt it was too long.
SCAN_PAGE_NOT_CLEAN covers both states that indicate the page is not in a clean/stable
state suitable for collapse.
[1] https://www.kernel.org/doc/Documentation/filesystems/proc.txt
Thanks,
Shivank
^ permalink raw reply
* Re: [PATCH V2 2/2] mm/khugepaged: map dirty/writeback pages failures to EAGAIN
From: Dev Jain @ 2025-11-20 8:03 UTC (permalink / raw)
To: Shivank Garg, Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Barry Song, Lance Yang, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Zach O'Keefe, linux-mm, linux-kernel,
linux-trace-kernel
In-Reply-To: <20251120065043.41738-10-shivankg@amd.com>
On 20/11/25 12:20 pm, Shivank Garg wrote:
> When collapse_file encounters dirty or writeback pages in file-backed
> mappings, it currently SCAN_FAIL which maps to -EINVAL. This is
> misleading as EINVAL suggests invalid arguments, whereas dirty/writeback
> pages represent transient conditions that may resolve on retry.
>
> Introduce SCAN_PAGE_NOT_CLEAN to cover both dirty and writeback states,
> mapping it to -EAGAIN. For MADV_COLLAPSE, this provides userspace with
> a clear signal that retry may succeed after writeback completes, making
> -EAGAIN semantically correct. For khugepaged, this is harmless as it
> will naturally revisit the range during periodic scans after async
> writeback completes.
>
> Signed-off-by: Shivank Garg <shivankg@amd.com>
> ---
> include/trace/events/huge_memory.h | 3 ++-
> mm/khugepaged.c | 8 +++++---
> 2 files changed, 7 insertions(+), 4 deletions(-)
>
> diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
> index 4cde53b45a85..1caf24b951e1 100644
> --- a/include/trace/events/huge_memory.h
> +++ b/include/trace/events/huge_memory.h
> @@ -37,7 +37,8 @@
> EM( SCAN_PAGE_HAS_PRIVATE, "page_has_private") \
> EM( SCAN_STORE_FAILED, "store_failed") \
> EM( SCAN_COPY_MC, "copy_poisoned_page") \
> - EMe(SCAN_PAGE_FILLED, "page_filled")
> + EM( SCAN_PAGE_FILLED, "page_filled") \
> + EMe(SCAN_PAGE_NOT_CLEAN, "page_not_clean")
>
> #undef EM
> #undef EMe
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 066a332c76ad..282b413d17e8 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -59,6 +59,7 @@ enum scan_result {
> SCAN_STORE_FAILED,
> SCAN_COPY_MC,
> SCAN_PAGE_FILLED,
> + SCAN_PAGE_NOT_CLEAN,
> };
>
> #define CREATE_TRACE_POINTS
> @@ -1968,11 +1969,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
> */
> xas_unlock_irq(&xas);
> filemap_flush(mapping);
> - result = SCAN_FAIL;
> + result = SCAN_PAGE_NOT_CLEAN;
> goto xa_unlocked;
> } else if (folio_test_writeback(folio)) {
> xas_unlock_irq(&xas);
> - result = SCAN_FAIL;
> + result = SCAN_PAGE_NOT_CLEAN;
> goto xa_unlocked;
> } else if (folio_trylock(folio)) {
> folio_get(folio);
> @@ -2019,7 +2020,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
> * folio is dirty because it hasn't been flushed
> * since first write.
> */
> - result = SCAN_FAIL;
> + result = SCAN_PAGE_NOT_CLEAN;
> goto out_unlock;
> }
>
> @@ -2748,6 +2749,7 @@ static int madvise_collapse_errno(enum scan_result r)
> case SCAN_PAGE_LRU:
> case SCAN_DEL_PAGE_LRU:
> case SCAN_PAGE_FILLED:
> + case SCAN_PAGE_NOT_CLEAN:
> return -EAGAIN;
> /*
> * Other: Trying again likely not to succeed / error intrinsic to
SCAN_PAGE_NOT_CLEAN is confusing - NOT_CLEAN literally means dirty, so why not SCAN_PAGE_DIRTY?
Or SCAN_PAGE_DIRTY_OR_UNDER_WRITEBACK? Since folio_test_writeback() is true as a result of
the folio being dirty, maybe just SCAN_PAGE_DIRTY can do.
Reviewed-by: Dev Jain <dev.jain@arm.com>
^ permalink raw reply
* Re: [PATCH v5 6/6] perf tools: Flush remaining samples w/o deferred callchains
From: Ian Rogers @ 2025-11-20 7:21 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, James Clark, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users,
Steven Rostedt, Josh Poimboeuf, Indu Bhagat, Jens Remus,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <CAP-5=fVZcmYgyRJVNbet9ZLjihAiXNy7D8cX+QqHe8EKqJ1qOQ@mail.gmail.com>
On Wed, Nov 19, 2025 at 9:29 PM Ian Rogers <irogers@google.com> wrote:
>
> On Wed, Nov 19, 2025 at 6:11 PM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > It's possible that some kernel samples don't have matching deferred
> > callchain records when the profiling session was ended before the
> > threads came back to userspace. Let's flush the samples before
> > finish the session.
> >
> > Also 32-bit systems can see partial mmap for the data. In that case,
> > deferred samples won't point to the correct data once the mapping moves
> > to the next portion of the file. Copy the original sample before it
> > unmaps the current data.
>
> I think it is simpler to always copy. We may have events from
> synthesis, inject, .. and not the reader. Relying on callers to know
> that someone made a copy of the event and to make a defensive copy on
> their behalf just feels error prone.
>
> In the python session API I need to deal with the lifetime of events.
> Currently the events are copied:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/python.c?h=perf-tools-next#n507
> and I'm doing this for session tool callbacks:
> https://lore.kernel.org/lkml/20251029053413.355154-12-irogers@google.com/
> I think it can be made lazier by knowing the tool callback can assume
> the event and sample are valid. We can delay the copying of the
> event/sample for if the pyevent has a reference count >1 and we're
> returning out of the tool callback. Doing some kind of global
> knowledge in the reader for maintaining the correctness of memory, I'm
> just not clear on how to make it always work.
I believe we always reuse the memory for the event, per event, in pipe mode:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/session.c?h=perf-tools-next#n1868
so a lazy copy will be broken for the pipe mode case.
Thanks,
Ian
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> > tools/perf/util/session.c | 98 +++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 98 insertions(+)
> >
> > diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> > index 2e777fd1bcf6707b..b781e01ddcb4876b 100644
> > --- a/tools/perf/util/session.c
> > +++ b/tools/perf/util/session.c
> > @@ -1288,8 +1288,13 @@ static int evlist__deliver_sample(struct evlist *evlist, const struct perf_tool
> > struct deferred_event {
> > struct list_head list;
> > union perf_event *event;
> > + bool allocated;
> > };
> >
> > +/*
> > + * This is called when a deferred callchain record comes up. Find all matching
> > + * samples, merge the callchains and process them.
> > + */
> > static int evlist__deliver_deferred_samples(struct evlist *evlist,
> > const struct perf_tool *tool,
> > union perf_event *event,
> > @@ -1331,6 +1336,86 @@ static int evlist__deliver_deferred_samples(struct evlist *evlist,
> > free(orig_sample.callchain);
> >
> > list_del(&de->list);
> > + if (de->allocated)
> > + free(de->event);
> > + free(de);
> > +
> > + if (ret)
> > + break;
> > + }
> > + return ret;
> > +}
> > +
> > +/*
> > + * This is called when the backing mmap is about to go away. It needs to save
> > + * the original sample data until it finds the matching deferred callchains.
> > + */
> > +static void evlist__copy_deferred_samples(struct evlist *evlist,
> > + const struct perf_tool *tool,
> > + struct machine *machine)
> > +{
> > + struct deferred_event *de, *tmp;
> > + struct evsel *evsel;
> > + int ret = 0;
> > +
> > + list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) {
> > + struct perf_sample sample;
> > + size_t sz = de->event->header.size;
> > + void *buf;
> > +
> > + if (de->allocated)
> > + continue;
> > +
> > + buf = malloc(sz);
> > + if (buf) {
> > + memcpy(buf, de->event, sz);
> > + de->event = buf;
> > + de->allocated = true;
> > + continue;
> > + }
> > +
> > + /* The allocation failed, flush the sample now */
> > + ret = evlist__parse_sample(evlist, de->event, &sample);
> > + if (ret == 0) {
> > + evsel = evlist__id2evsel(evlist, sample.id);
> > + evlist__deliver_sample(evlist, tool, de->event,
> > + &sample, evsel, machine);
> > + }
> > +
> > + list_del(&de->list);
> > + BUG_ON(de->allocated);
> > + free(de);
> > + }
> > +}
> > +
> > +/*
> > + * This is called at the end of the data processing for the session. Flush the
> > + * remaining samples as there's no hope for matching deferred callchains.
> > + */
> > +static int evlist__flush_deferred_samples(struct evlist *evlist,
> > + const struct perf_tool *tool,
> > + struct machine *machine)
> > +{
> > + struct deferred_event *de, *tmp;
> > + struct evsel *evsel;
> > + int ret = 0;
> > +
> > + list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) {
> > + struct perf_sample sample;
> > +
> > + ret = evlist__parse_sample(evlist, de->event, &sample);
> > + if (ret < 0) {
> > + pr_err("failed to parse original sample\n");
> > + break;
> > + }
> > +
> > + evsel = evlist__id2evsel(evlist, sample.id);
> > + ret = evlist__deliver_sample(evlist, tool, de->event,
> > + &sample, evsel, machine);
> > +
> > + list_del(&de->list);
> > + if (de->allocated)
> > + free(de->event);
> > free(de);
> >
> > if (ret)
> > @@ -1374,6 +1459,7 @@ static int machines__deliver_event(struct machines *machines,
> > return -ENOMEM;
> >
> > de->event = event;
> > + de->allocated = false;
> > list_add_tail(&de->list, &evlist->deferred_samples);
> > return 0;
> > }
> > @@ -2218,6 +2304,8 @@ reader__mmap(struct reader *rd, struct perf_session *session)
> > }
> >
> > if (mmaps[rd->mmap_idx]) {
> > + evlist__copy_deferred_samples(session->evlist, session->tool,
> > + &session->machines.host);
> > munmap(mmaps[rd->mmap_idx], rd->mmap_size);
> > mmaps[rd->mmap_idx] = NULL;
> > }
> > @@ -2372,6 +2460,11 @@ static int __perf_session__process_events(struct perf_session *session)
> > if (err)
> > goto out_err;
> > err = auxtrace__flush_events(session, tool);
> > + if (err)
> > + goto out_err;
> > + err = evlist__flush_deferred_samples(session->evlist,
> > + session->tool,
> > + &session->machines.host);
> > if (err)
> > goto out_err;
> > err = perf_session__flush_thread_stacks(session);
> > @@ -2494,6 +2587,11 @@ static int __perf_session__process_dir_events(struct perf_session *session)
> > if (ret)
> > goto out_err;
> >
> > + ret = evlist__flush_deferred_samples(session->evlist, tool,
> > + &session->machines.host);
> > + if (ret)
> > + goto out_err;
> > +
> > ret = perf_session__flush_thread_stacks(session);
> > out_err:
> > ui_progress__finish();
> > --
> > 2.52.0.rc1.455.g30608eb744-goog
> >
^ permalink raw reply
* Re: [PATCH 2/2] mm/khugepaged: return EAGAIN for transient dirty pages in MADV_COLLAPSE
From: Garg, Shivank @ 2025-11-20 7:03 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang,
Liam R . Howlett, Nico Pache, Ryan Roberts, Dev Jain, Barry Song,
Lance Yang, Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Zach O'Keefe, linux-mm, linux-kernel, linux-trace-kernel
In-Reply-To: <0efa89a3-2080-4daa-a190-2ced8e14ed9d@lucifer.local>
On 11/19/2025 11:46 PM, Lorenzo Stoakes wrote:
> On Wed, Nov 19, 2025 at 03:55:29PM +0530, Garg, Shivank wrote:
>>>
>>> With comments below addressed, LGTM so:
>>>
>>> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
>>
>> Thank you for the review.
>>
>> Looking at the code, the dirty/writeback checks where I'm making changes
>> are all in the !is_shmem branch, so it only affects regular files, not
>> shmem.
>
> Yeah sorry I think I was being somehow blind to the fact that each
> dirty/writeback test has !is_shmem, esp. given I literally quote it there :)
>
> So err ignore me I think here haha
>
>>
>> Should I mention in the commit message that these changes are limited
>> to regular files and don't affect shmem?
>
> No that's fine.
>
>>
>> I'm not sure I fully understood your concern on shmem. Could you please elaborate?
>
> Yeah I think I just misread the code after a long day :P
>
>>
Thanks for clarifying!
I held off adding your Reviewed-by tag to v2 since there were some more changes
from v1. Let me know if you're still happy with it and I can add it for v3.
Best Regards,
Shivank
^ permalink raw reply
* [PATCH V2 2/2] mm/khugepaged: map dirty/writeback pages failures to EAGAIN
From: Shivank Garg @ 2025-11-20 6:50 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, Shivank Garg
In-Reply-To: <20251120065043.41738-6-shivankg@amd.com>
When collapse_file encounters dirty or writeback pages in file-backed
mappings, it currently SCAN_FAIL which maps to -EINVAL. This is
misleading as EINVAL suggests invalid arguments, whereas dirty/writeback
pages represent transient conditions that may resolve on retry.
Introduce SCAN_PAGE_NOT_CLEAN to cover both dirty and writeback states,
mapping it to -EAGAIN. For MADV_COLLAPSE, this provides userspace with
a clear signal that retry may succeed after writeback completes, making
-EAGAIN semantically correct. For khugepaged, this is harmless as it
will naturally revisit the range during periodic scans after async
writeback completes.
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
include/trace/events/huge_memory.h | 3 ++-
mm/khugepaged.c | 8 +++++---
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
index 4cde53b45a85..1caf24b951e1 100644
--- a/include/trace/events/huge_memory.h
+++ b/include/trace/events/huge_memory.h
@@ -37,7 +37,8 @@
EM( SCAN_PAGE_HAS_PRIVATE, "page_has_private") \
EM( SCAN_STORE_FAILED, "store_failed") \
EM( SCAN_COPY_MC, "copy_poisoned_page") \
- EMe(SCAN_PAGE_FILLED, "page_filled")
+ EM( SCAN_PAGE_FILLED, "page_filled") \
+ EMe(SCAN_PAGE_NOT_CLEAN, "page_not_clean")
#undef EM
#undef EMe
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 066a332c76ad..282b413d17e8 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -59,6 +59,7 @@ enum scan_result {
SCAN_STORE_FAILED,
SCAN_COPY_MC,
SCAN_PAGE_FILLED,
+ SCAN_PAGE_NOT_CLEAN,
};
#define CREATE_TRACE_POINTS
@@ -1968,11 +1969,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
*/
xas_unlock_irq(&xas);
filemap_flush(mapping);
- result = SCAN_FAIL;
+ result = SCAN_PAGE_NOT_CLEAN;
goto xa_unlocked;
} else if (folio_test_writeback(folio)) {
xas_unlock_irq(&xas);
- result = SCAN_FAIL;
+ result = SCAN_PAGE_NOT_CLEAN;
goto xa_unlocked;
} else if (folio_trylock(folio)) {
folio_get(folio);
@@ -2019,7 +2020,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
* folio is dirty because it hasn't been flushed
* since first write.
*/
- result = SCAN_FAIL;
+ result = SCAN_PAGE_NOT_CLEAN;
goto out_unlock;
}
@@ -2748,6 +2749,7 @@ static int madvise_collapse_errno(enum scan_result r)
case SCAN_PAGE_LRU:
case SCAN_DEL_PAGE_LRU:
case SCAN_PAGE_FILLED:
+ case SCAN_PAGE_NOT_CLEAN:
return -EAGAIN;
/*
* Other: Trying again likely not to succeed / error intrinsic to
--
2.43.0
^ permalink raw reply related
* [PATCH V2 1/2] mm/khugepaged: do synchronous writeback for MADV_COLLAPSE
From: Shivank Garg @ 2025-11-20 6:50 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, Shivank Garg, Branden Moore
In-Reply-To: <20251120065043.41738-6-shivankg@amd.com>
When MADV_COLLAPSE is called on file-backed mappings (e.g., executable
text sections), the pages may still be dirty from recent writes and
cause collapse to fail with -EINVAL. This is particularly problematic
for freshly copied executables on filesystems, where page cache folios
remain dirty until background writeback completes.
The current code in collapse_file() triggers async writeback via
filemap_flush() and expects khugepaged to revisit the page later.
However, MADV_COLLAPSE is a synchronous operation where userspace
expects immediate results.
Perform synchronous writeback in madvise_collapse() before attempting
collapse to avoid failing on first attempt.
Reported-by: Branden Moore <Branden.Moore@amd.com>
Closes: https://lore.kernel.org/all/4e26fe5e-7374-467c-a333-9dd48f85d7cc@amd.com
Fixes: 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
mm/khugepaged.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 97d1b2824386..066a332c76ad 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -22,6 +22,7 @@
#include <linux/dax.h>
#include <linux/ksm.h>
#include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
#include <asm/tlb.h>
#include "internal.h"
@@ -2784,6 +2785,31 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
hstart = (start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK;
hend = end & HPAGE_PMD_MASK;
+ /*
+ * For file-backed VMAs, perform synchronous writeback to ensure
+ * dirty folios are flushed before attempting collapse. This avoids
+ * failing on the first attempt when freshly-written executable text
+ * is still dirty in the page cache.
+ */
+ if (!vma_is_anonymous(vma) && vma->vm_file) {
+ struct address_space *mapping = vma->vm_file->f_mapping;
+
+ if (mapping_can_writeback(mapping)) {
+ pgoff_t pgoff_start = linear_page_index(vma, hstart);
+ pgoff_t pgoff_end = linear_page_index(vma, hend);
+ loff_t lstart = (loff_t)pgoff_start << PAGE_SHIFT;
+ loff_t lend = ((loff_t)pgoff_end << PAGE_SHIFT) - 1;
+
+ mmap_read_unlock(mm);
+ mmap_locked = false;
+
+ if (filemap_write_and_wait_range(mapping, lstart, lend)) {
+ last_fail = SCAN_FAIL;
+ goto out_maybelock;
+ }
+ }
+ }
+
for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
int result = SCAN_FAIL;
--
2.43.0
^ permalink raw reply related
* [PATCH V2 0/2] mm/khugepaged: fix dirty page handling for MADV_COLLAPSE
From: Shivank Garg @ 2025-11-20 6:50 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, Shivank Garg
MADV_COLLAPSE on file-backed mappings fails with -EINVAL when TEXT pages
are dirty. This may affect real scenarios: package/container updates,
executing binaries immediately after writing them, etc.
The issue is that collapse_file() triggers async writeback and returns
SCAN_FAIL (maps to -EINVAL), expecting khugepaged to revisit later. But
MADV_COLLAPSE is synchronous and userspace expects immediate success or a
clear retry signal.
Reproduction:
- Copy 2MB-aligned executable to freshly mounted XFS/ext4
- Call MADV_COLLAPSE on .text section
- First call fails with -EINVAL (text pages dirty from copy)
- Second call succeeds (async writeback completed)
Issue Report:
https://lore.kernel.org/all/4e26fe5e-7374-467c-a333-9dd48f85d7cc@amd.com
v2:
- Move writeback to madvise_collapse() (better abstraction, proper
mmap_lock handling and does VMA revalidation after I/O) (Lorenzo)
- Rename to SCAN_PAGE_DIRTY to SCAN_PAGE_NOT_CLEAN and extend its use
for all dirty/writeback folio cases that previously returned incorrect
results (Dev)
v1: https://lore.kernel.org/all/20251110113254.77822-1-shivankg@amd.com
Shivank Garg (2):
mm/khugepaged: do synchronous writeback for MADV_COLLAPSE
mm/khugepaged: map dirty/writeback pages failures to EAGAIN
include/trace/events/huge_memory.h | 3 ++-
mm/khugepaged.c | 34 +++++++++++++++++++++++++++---
2 files changed, 33 insertions(+), 4 deletions(-)
base-commit: 4a3f8fc3adb7046e44bd1feb2f5c5fe95296894f
--
2.43.0
^ permalink raw reply
* Re: [PATCH v12 mm-new 14/15] khugepaged: run khugepaged for all orders
From: Baolin Wang @ 2025-11-20 6:37 UTC (permalink / raw)
To: Lorenzo Stoakes, Nico Pache
Cc: linux-kernel, linux-trace-kernel, linux-mm, linux-doc, david, ziy,
Liam.Howlett, ryan.roberts, dev.jain, corbet, rostedt, mhiramat,
mathieu.desnoyers, akpm, baohua, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, surenb, zokeefe, hannes,
rientjes, mhocko, rdunlap, hughd, richard.weiyang, lance.yang,
vbabka, rppt, jannh, pfalcato
In-Reply-To: <2f1cdad3-6ac4-4502-a599-5bef9dbe0847@lucifer.local>
On 2025/11/19 20:13, Lorenzo Stoakes wrote:
> On Wed, Oct 22, 2025 at 12:37:16PM -0600, Nico Pache wrote:
>> From: Baolin Wang <baolin.wang@linux.alibaba.com>
>>
>> If any order (m)THP is enabled we should allow running khugepaged to
>> attempt scanning and collapsing mTHPs. In order for khugepaged to operate
>> when only mTHP sizes are specified in sysfs, we must modify the predicate
>> function that determines whether it ought to run to do so.
>>
>> This function is currently called hugepage_pmd_enabled(), this patch
>> renames it to hugepage_enabled() and updates the logic to check to
>> determine whether any valid orders may exist which would justify
>> khugepaged running.
>>
>> We must also update collapse_allowable_orders() to check all orders if
>> the vma is anonymous and the collapse is khugepaged.
>>
>> After this patch khugepaged mTHP collapse is fully enabled.
>>
>> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>> Signed-off-by: Nico Pache <npache@redhat.com>
>> ---
>> mm/khugepaged.c | 25 +++++++++++++------------
>> 1 file changed, 13 insertions(+), 12 deletions(-)
>>
>> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
>> index 54f5c7888e46..8ed9f8e2d376 100644
>> --- a/mm/khugepaged.c
>> +++ b/mm/khugepaged.c
>> @@ -418,23 +418,23 @@ static inline int collapse_test_exit_or_disable(struct mm_struct *mm)
>> mm_flags_test(MMF_DISABLE_THP_COMPLETELY, mm);
>> }
>>
>> -static bool hugepage_pmd_enabled(void)
>> +static bool hugepage_enabled(void)
>> {
>> /*
>> * We cover the anon, shmem and the file-backed case here; file-backed
>> * hugepages, when configured in, are determined by the global control.
>> - * Anon pmd-sized hugepages are determined by the pmd-size control.
>> + * Anon hugepages are determined by its per-size mTHP control.
>> * Shmem pmd-sized hugepages are also determined by its pmd-size control,
>> * except when the global shmem_huge is set to SHMEM_HUGE_DENY.
>> */
>> if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) &&
>> hugepage_global_enabled())
>> return true;
>> - if (test_bit(PMD_ORDER, &huge_anon_orders_always))
>> + if (READ_ONCE(huge_anon_orders_always))
>> return true;
>> - if (test_bit(PMD_ORDER, &huge_anon_orders_madvise))
>> + if (READ_ONCE(huge_anon_orders_madvise))
>> return true;
>> - if (test_bit(PMD_ORDER, &huge_anon_orders_inherit) &&
>> + if (READ_ONCE(huge_anon_orders_inherit) &&
>> hugepage_global_enabled())
>> return true;
>> if (IS_ENABLED(CONFIG_SHMEM) && shmem_hpage_pmd_enabled())
>> @@ -508,7 +508,8 @@ static unsigned long collapse_allowable_orders(struct vm_area_struct *vma,
>> vm_flags_t vm_flags, bool is_khugepaged)
>> {
>> enum tva_type tva_flags = is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE;
>> - unsigned long orders = BIT(HPAGE_PMD_ORDER);
>> + unsigned long orders = is_khugepaged && vma_is_anonymous(vma) ?
>> + THP_ORDERS_ALL_ANON : BIT(HPAGE_PMD_ORDER);
>
> Why are we doing this? If this is explicitly enabling mTHP for anon, which it
> seems to be, can we please make this a little more explicit :)
>
> I'd prefer this not to be a horribly squashed ternary, rather:
>
> unsigned long orders;
>
> /* We explicitly allow mTHP collapse for anonymous khugepaged ONLY. */
> if (is_khugepaged && vma_is_anonymous(vma))
> orders = THP_ORDERS_ALL_ANON;
> else
> orders = BIT(HPAGE_PMD_ORDER);
Yes, LGTM.
> Also, does THP_ORDERS_ALL_ANON account for KHUGEPAGED_MIN_MTHP_ORDER? It's weird
> to say that an order is allowed that isn't permitted by mTHP (e.g. order-0).
The THP_ORDERS_ALL_ANON has already filtered out order 0 and order 1, so
it matches the definition of KHUGEPAGED_MIN_MTHP_ORDER.
/*
* Mask of all large folio orders supported for anonymous THP; all
orders up to
* and including PMD_ORDER, except order-0 (which is not "huge") and
order-1
* (which is a limitation of the THP implementation).
*/
#define THP_ORDERS_ALL_ANON ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
^ permalink raw reply
* Re: [PATCH v4 2/9] mm: add atomic VMA flags and set VM_MAYBE_GUARD as such
From: Lance Yang @ 2025-11-20 6:11 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: Jonathan Corbet, David Hildenbrand, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers, Jann Horn,
Pedro Falcato, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, linux-kernel, linux-fsdevel, linux-doc,
linux-mm, linux-trace-kernel, linux-kselftest, Andrei Vagin,
Andrew Morton
In-Reply-To: <97e57abed09f2663077ed7a36fb8206e243171a9.1763460113.git.lorenzo.stoakes@oracle.com>
On 2025/11/18 18:17, Lorenzo Stoakes wrote:
> This patch adds the ability to atomically set VMA flags with only the mmap
> read/VMA read lock held.
>
> As this could be hugely problematic for VMA flags in general given that
> all other accesses are non-atomic and serialised by the mmap/VMA locks, we
> implement this with a strict allow-list - that is, only designated flags
> are allowed to do this.
>
> We make VM_MAYBE_GUARD one of these flags.
>
> Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> Reviewed-by: Pedro Falcato <pfalcato@suse.de>
> Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
> ---
LGTM! Feel free to add:
Reviewed-by: Lance Yang <lance.yang@linux.dev>
^ permalink raw reply
* Re: [PATCH v4 1/9] mm: introduce VM_MAYBE_GUARD and make visible in /proc/$pid/smaps
From: Lance Yang @ 2025-11-20 5:48 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: Jonathan Corbet, David Hildenbrand, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers, Jann Horn,
Pedro Falcato, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, linux-kernel, linux-fsdevel, linux-doc,
linux-mm, linux-trace-kernel, linux-kselftest, Andrei Vagin,
Andrew Morton
In-Reply-To: <cf8ef821eba29b6c5b5e138fffe95d6dcabdedb9.1763460113.git.lorenzo.stoakes@oracle.com>
On 2025/11/18 18:17, Lorenzo Stoakes wrote:
> Currently, if a user needs to determine if guard regions are present in a
> range, they have to scan all VMAs (or have knowledge of which ones might
> have guard regions).
>
> Since commit 8e2f2aeb8b48 ("fs/proc/task_mmu: add guard region bit to
> pagemap") and the related commit a516403787e0 ("fs/proc: extend the
> PAGEMAP_SCAN ioctl to report guard regions"), users can use either
> /proc/$pid/pagemap or the PAGEMAP_SCAN functionality to perform this
> operation at a virtual address level.
>
> This is not ideal, and it gives no visibility at a /proc/$pid/smaps level
> that guard regions exist in ranges.
>
> This patch remedies the situation by establishing a new VMA flag,
> VM_MAYBE_GUARD, to indicate that a VMA may contain guard regions (it is
> uncertain because we cannot reasonably determine whether a
> MADV_GUARD_REMOVE call has removed all of the guard regions in a VMA, and
> additionally VMAs may change across merge/split).
>
> We utilise 0x800 for this flag which makes it available to 32-bit
> architectures also, a flag that was previously used by VM_DENYWRITE, which
> was removed in commit 8d0920bde5eb ("mm: remove VM_DENYWRITE") and hasn't
> bee reused yet.
>
> We also update the smaps logic and documentation to identify these VMAs.
>
> Another major use of this functionality is that we can use it to identify
> that we ought to copy page tables on fork.
>
> We do not actually implement usage of this flag in mm/madvise.c yet as we
> need to allow some VMA flags to be applied atomically under mmap/VMA read
> lock in order to avoid the need to acquire a write lock for this purpose.
>
> Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> Reviewed-by: Pedro Falcato <pfalcato@suse.de>
> Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
> Acked-by: David Hildenbrand (Red Hat) <david@kernel.org>
> ---
LGTM! Feel free to add:
Reviewed-by: Lance Yang <lance.yang@linux.dev>
^ permalink raw reply
* Re: [PATCH v5 6/6] perf tools: Flush remaining samples w/o deferred callchains
From: Ian Rogers @ 2025-11-20 5:29 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, James Clark, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users,
Steven Rostedt, Josh Poimboeuf, Indu Bhagat, Jens Remus,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120021046.94490-7-namhyung@kernel.org>
On Wed, Nov 19, 2025 at 6:11 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> It's possible that some kernel samples don't have matching deferred
> callchain records when the profiling session was ended before the
> threads came back to userspace. Let's flush the samples before
> finish the session.
>
> Also 32-bit systems can see partial mmap for the data. In that case,
> deferred samples won't point to the correct data once the mapping moves
> to the next portion of the file. Copy the original sample before it
> unmaps the current data.
I think it is simpler to always copy. We may have events from
synthesis, inject, .. and not the reader. Relying on callers to know
that someone made a copy of the event and to make a defensive copy on
their behalf just feels error prone.
In the python session API I need to deal with the lifetime of events.
Currently the events are copied:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/python.c?h=perf-tools-next#n507
and I'm doing this for session tool callbacks:
https://lore.kernel.org/lkml/20251029053413.355154-12-irogers@google.com/
I think it can be made lazier by knowing the tool callback can assume
the event and sample are valid. We can delay the copying of the
event/sample for if the pyevent has a reference count >1 and we're
returning out of the tool callback. Doing some kind of global
knowledge in the reader for maintaining the correctness of memory, I'm
just not clear on how to make it always work.
Thanks,
Ian
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
> tools/perf/util/session.c | 98 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 98 insertions(+)
>
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index 2e777fd1bcf6707b..b781e01ddcb4876b 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -1288,8 +1288,13 @@ static int evlist__deliver_sample(struct evlist *evlist, const struct perf_tool
> struct deferred_event {
> struct list_head list;
> union perf_event *event;
> + bool allocated;
> };
>
> +/*
> + * This is called when a deferred callchain record comes up. Find all matching
> + * samples, merge the callchains and process them.
> + */
> static int evlist__deliver_deferred_samples(struct evlist *evlist,
> const struct perf_tool *tool,
> union perf_event *event,
> @@ -1331,6 +1336,86 @@ static int evlist__deliver_deferred_samples(struct evlist *evlist,
> free(orig_sample.callchain);
>
> list_del(&de->list);
> + if (de->allocated)
> + free(de->event);
> + free(de);
> +
> + if (ret)
> + break;
> + }
> + return ret;
> +}
> +
> +/*
> + * This is called when the backing mmap is about to go away. It needs to save
> + * the original sample data until it finds the matching deferred callchains.
> + */
> +static void evlist__copy_deferred_samples(struct evlist *evlist,
> + const struct perf_tool *tool,
> + struct machine *machine)
> +{
> + struct deferred_event *de, *tmp;
> + struct evsel *evsel;
> + int ret = 0;
> +
> + list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) {
> + struct perf_sample sample;
> + size_t sz = de->event->header.size;
> + void *buf;
> +
> + if (de->allocated)
> + continue;
> +
> + buf = malloc(sz);
> + if (buf) {
> + memcpy(buf, de->event, sz);
> + de->event = buf;
> + de->allocated = true;
> + continue;
> + }
> +
> + /* The allocation failed, flush the sample now */
> + ret = evlist__parse_sample(evlist, de->event, &sample);
> + if (ret == 0) {
> + evsel = evlist__id2evsel(evlist, sample.id);
> + evlist__deliver_sample(evlist, tool, de->event,
> + &sample, evsel, machine);
> + }
> +
> + list_del(&de->list);
> + BUG_ON(de->allocated);
> + free(de);
> + }
> +}
> +
> +/*
> + * This is called at the end of the data processing for the session. Flush the
> + * remaining samples as there's no hope for matching deferred callchains.
> + */
> +static int evlist__flush_deferred_samples(struct evlist *evlist,
> + const struct perf_tool *tool,
> + struct machine *machine)
> +{
> + struct deferred_event *de, *tmp;
> + struct evsel *evsel;
> + int ret = 0;
> +
> + list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) {
> + struct perf_sample sample;
> +
> + ret = evlist__parse_sample(evlist, de->event, &sample);
> + if (ret < 0) {
> + pr_err("failed to parse original sample\n");
> + break;
> + }
> +
> + evsel = evlist__id2evsel(evlist, sample.id);
> + ret = evlist__deliver_sample(evlist, tool, de->event,
> + &sample, evsel, machine);
> +
> + list_del(&de->list);
> + if (de->allocated)
> + free(de->event);
> free(de);
>
> if (ret)
> @@ -1374,6 +1459,7 @@ static int machines__deliver_event(struct machines *machines,
> return -ENOMEM;
>
> de->event = event;
> + de->allocated = false;
> list_add_tail(&de->list, &evlist->deferred_samples);
> return 0;
> }
> @@ -2218,6 +2304,8 @@ reader__mmap(struct reader *rd, struct perf_session *session)
> }
>
> if (mmaps[rd->mmap_idx]) {
> + evlist__copy_deferred_samples(session->evlist, session->tool,
> + &session->machines.host);
> munmap(mmaps[rd->mmap_idx], rd->mmap_size);
> mmaps[rd->mmap_idx] = NULL;
> }
> @@ -2372,6 +2460,11 @@ static int __perf_session__process_events(struct perf_session *session)
> if (err)
> goto out_err;
> err = auxtrace__flush_events(session, tool);
> + if (err)
> + goto out_err;
> + err = evlist__flush_deferred_samples(session->evlist,
> + session->tool,
> + &session->machines.host);
> if (err)
> goto out_err;
> err = perf_session__flush_thread_stacks(session);
> @@ -2494,6 +2587,11 @@ static int __perf_session__process_dir_events(struct perf_session *session)
> if (ret)
> goto out_err;
>
> + ret = evlist__flush_deferred_samples(session->evlist, tool,
> + &session->machines.host);
> + if (ret)
> + goto out_err;
> +
> ret = perf_session__flush_thread_stacks(session);
> out_err:
> ui_progress__finish();
> --
> 2.52.0.rc1.455.g30608eb744-goog
>
^ permalink raw reply
* Re: [PATCH v5 5/6] perf tools: Merge deferred user callchains
From: Ian Rogers @ 2025-11-20 5:13 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, James Clark, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users,
Steven Rostedt, Josh Poimboeuf, Indu Bhagat, Jens Remus,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120021046.94490-6-namhyung@kernel.org>
On Wed, Nov 19, 2025 at 6:11 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Save samples with deferred callchains in a separate list and deliver
> them after merging the user callchains. If users don't want to merge
> they can set tool->merge_deferred_callchains to false to prevent the
> behavior.
>
> With previous result, now perf script will show the merged callchains.
>
> $ perf script
> ...
> pwd 2312 121.163435: 249113 cpu/cycles/P:
> ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms])
> ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms])
> ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms])
> ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms])
> ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms])
> ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms])
> ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kallsyms])
> 7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
> 7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
> 7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
> ...
>
> The old output can be get using --no-merge-callchain option.
> Also perf report can get the user callchain entry at the end.
>
> $ perf report --no-children --stdio -q -S __build_id_parse.isra.0
> # symbol: __build_id_parse.isra.0
> 8.40% pwd [kernel.kallsyms]
> |
> ---__build_id_parse.isra.0
> perf_event_mmap
> mprotect_fixup
> do_mprotect_pkey
> __x64_sys_mprotect
> do_syscall_64
> entry_SYSCALL_64_after_hwframe
> mprotect
> _dl_sysdep_start
> _dl_start_user
>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
> tools/perf/Documentation/perf-script.txt | 5 ++
> tools/perf/builtin-inject.c | 1 +
> tools/perf/builtin-report.c | 1 +
> tools/perf/builtin-script.c | 4 ++
> tools/perf/util/callchain.c | 29 ++++++++++
> tools/perf/util/callchain.h | 3 ++
> tools/perf/util/evlist.c | 1 +
> tools/perf/util/evlist.h | 2 +
> tools/perf/util/session.c | 67 +++++++++++++++++++++++-
> tools/perf/util/tool.c | 2 +
> tools/perf/util/tool.h | 1 +
> 11 files changed, 115 insertions(+), 1 deletion(-)
>
> diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
> index 28bec7e78bc858ba..03d1129606328d6d 100644
> --- a/tools/perf/Documentation/perf-script.txt
> +++ b/tools/perf/Documentation/perf-script.txt
> @@ -527,6 +527,11 @@ include::itrace.txt[]
> The known limitations include exception handing such as
> setjmp/longjmp will have calls/returns not match.
>
> +--merge-callchains::
> + Enable merging deferred user callchains if available. This is the
> + default behavior. If you want to see separate CALLCHAIN_DEFERRED
> + records for some reason, use --no-merge-callchains explicitly.
> +
> :GMEXAMPLECMD: script
> :GMEXAMPLESUBCMD:
> include::guest-files.txt[]
> diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
> index bd9245d2dd41aa48..51d2721b6db9dccb 100644
> --- a/tools/perf/builtin-inject.c
> +++ b/tools/perf/builtin-inject.c
> @@ -2527,6 +2527,7 @@ int cmd_inject(int argc, const char **argv)
> inject.tool.auxtrace = perf_event__repipe_auxtrace;
> inject.tool.bpf_metadata = perf_event__repipe_op2_synth;
> inject.tool.dont_split_sample_group = true;
> + inject.tool.merge_deferred_callchains = false;
> inject.session = __perf_session__new(&data, &inject.tool,
> /*trace_event_repipe=*/inject.output.is_pipe,
> /*host_env=*/NULL);
> diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
> index 2bc269f5fcef8023..add6b1c2aaf04270 100644
> --- a/tools/perf/builtin-report.c
> +++ b/tools/perf/builtin-report.c
> @@ -1614,6 +1614,7 @@ int cmd_report(int argc, const char **argv)
> report.tool.event_update = perf_event__process_event_update;
> report.tool.feature = process_feature_event;
> report.tool.ordering_requires_timestamps = true;
> + report.tool.merge_deferred_callchains = !dump_trace;
>
> session = perf_session__new(&data, &report.tool);
> if (IS_ERR(session)) {
> diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
> index 85b42205a71b3993..62e43d3c5ad731a0 100644
> --- a/tools/perf/builtin-script.c
> +++ b/tools/perf/builtin-script.c
> @@ -4009,6 +4009,7 @@ int cmd_script(int argc, const char **argv)
> bool header_only = false;
> bool script_started = false;
> bool unsorted_dump = false;
> + bool merge_deferred_callchains = true;
> char *rec_script_path = NULL;
> char *rep_script_path = NULL;
> struct perf_session *session;
> @@ -4162,6 +4163,8 @@ int cmd_script(int argc, const char **argv)
> "Guest code can be found in hypervisor process"),
> OPT_BOOLEAN('\0', "stitch-lbr", &script.stitch_lbr,
> "Enable LBR callgraph stitching approach"),
> + OPT_BOOLEAN('\0', "merge-callchains", &merge_deferred_callchains,
> + "Enable merge deferred user callchains"),
> OPTS_EVSWITCH(&script.evswitch),
> OPT_END()
> };
> @@ -4418,6 +4421,7 @@ int cmd_script(int argc, const char **argv)
> script.tool.throttle = process_throttle_event;
> script.tool.unthrottle = process_throttle_event;
> script.tool.ordering_requires_timestamps = true;
> + script.tool.merge_deferred_callchains = merge_deferred_callchains;
> session = perf_session__new(&data, &script.tool);
> if (IS_ERR(session))
> return PTR_ERR(session);
> diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c
> index 2884187ccbbecfdc..71dc5a070065dd2a 100644
> --- a/tools/perf/util/callchain.c
> +++ b/tools/perf/util/callchain.c
> @@ -1838,3 +1838,32 @@ int sample__for_each_callchain_node(struct thread *thread, struct evsel *evsel,
> }
> return 0;
> }
> +
> +int sample__merge_deferred_callchain(struct perf_sample *sample_orig,
> + struct perf_sample *sample_callchain)
> +{
> + u64 nr_orig = sample_orig->callchain->nr - 1;
> + u64 nr_deferred = sample_callchain->callchain->nr;
> + struct ip_callchain *callchain;
> +
> + if (sample_orig->callchain->nr < 2) {
> + sample_orig->deferred_callchain = false;
> + return -EINVAL;
> + }
> +
> + callchain = calloc(1 + nr_orig + nr_deferred, sizeof(u64));
> + if (callchain == NULL) {
> + sample_orig->deferred_callchain = false;
> + return -ENOMEM;
> + }
> +
> + callchain->nr = nr_orig + nr_deferred;
> + /* copy original including PERF_CONTEXT_USER_DEFERRED (but the cookie) */
> + memcpy(callchain->ips, sample_orig->callchain->ips, nr_orig * sizeof(u64));
> + /* copy deferred user callchains */
> + memcpy(&callchain->ips[nr_orig], sample_callchain->callchain->ips,
> + nr_deferred * sizeof(u64));
> +
> + sample_orig->callchain = callchain;
> + return 0;
> +}
> diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h
> index d5ae4fbb7ce5fa44..2a52af8c80ace33c 100644
> --- a/tools/perf/util/callchain.h
> +++ b/tools/perf/util/callchain.h
> @@ -318,4 +318,7 @@ int sample__for_each_callchain_node(struct thread *thread, struct evsel *evsel,
> struct perf_sample *sample, int max_stack,
> bool symbols, callchain_iter_fn cb, void *data);
>
> +int sample__merge_deferred_callchain(struct perf_sample *sample_orig,
> + struct perf_sample *sample_callchain);
> +
> #endif /* __PERF_CALLCHAIN_H */
> diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
> index e8217efdda5323c6..03674d2cbd015e4f 100644
> --- a/tools/perf/util/evlist.c
> +++ b/tools/perf/util/evlist.c
> @@ -85,6 +85,7 @@ void evlist__init(struct evlist *evlist, struct perf_cpu_map *cpus,
> evlist->ctl_fd.pos = -1;
> evlist->nr_br_cntr = -1;
> metricgroup__rblist_init(&evlist->metric_events);
> + INIT_LIST_HEAD(&evlist->deferred_samples);
> }
>
> struct evlist *evlist__new(void)
> diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
> index 5e71e3dc60423079..911834ae7c2a6f76 100644
> --- a/tools/perf/util/evlist.h
> +++ b/tools/perf/util/evlist.h
> @@ -92,6 +92,8 @@ struct evlist {
> * of struct metric_expr.
> */
> struct rblist metric_events;
> + /* samples with deferred_callchain would wait here. */
> + struct list_head deferred_samples;
> };
>
> struct evsel_str_handler {
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index 361e15c1f26a96d0..2e777fd1bcf6707b 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -1285,6 +1285,60 @@ static int evlist__deliver_sample(struct evlist *evlist, const struct perf_tool
> per_thread);
> }
>
> +struct deferred_event {
> + struct list_head list;
> + union perf_event *event;
Is this the old version of the patch? No comment and it seems the
event's memory isn't copied. I'm worried as we have events in stack
allocated memory such as:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/session.c?h=perf-tools-next#n1618
or copies:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/builtin-inject.c?h=perf-tools-next#n336
or just because the ring buffer overwrites itself. My belief is that
we don't hold events, and the associated parsed sample, longer than
the tool callback because it'll be wrong/corrupt after that. Here the
deferred callchain events are all being held longer than a single tool
event callback.
> +};
> +
> +static int evlist__deliver_deferred_samples(struct evlist *evlist,
> + const struct perf_tool *tool,
> + union perf_event *event,
> + struct perf_sample *sample,
> + struct machine *machine)
> +{
> + struct deferred_event *de, *tmp;
> + struct evsel *evsel;
> + int ret = 0;
> +
> + if (!tool->merge_deferred_callchains) {
> + evsel = evlist__id2evsel(evlist, sample->id);
> + return tool->callchain_deferred(tool, event, sample,
> + evsel, machine);
> + }
> +
> + list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) {
> + struct perf_sample orig_sample;
> +
> + ret = evlist__parse_sample(evlist, de->event, &orig_sample);
> + if (ret < 0) {
> + pr_err("failed to parse original sample\n");
> + break;
> + }
> +
> + if (sample->tid != orig_sample.tid)
> + continue;
> +
> + if (event->callchain_deferred.cookie == orig_sample.deferred_cookie)
> + sample__merge_deferred_callchain(&orig_sample, sample);
> + else
> + orig_sample.deferred_callchain = false;
> +
> + evsel = evlist__id2evsel(evlist, orig_sample.id);
> + ret = evlist__deliver_sample(evlist, tool, de->event,
> + &orig_sample, evsel, machine);
> +
> + if (orig_sample.deferred_callchain)
> + free(orig_sample.callchain);
> +
> + list_del(&de->list);
There's no free of de->event.
> + free(de);
> +
> + if (ret)
> + break;
> + }
> + return ret;
> +}
> +
> static int machines__deliver_event(struct machines *machines,
> struct evlist *evlist,
> union perf_event *event,
> @@ -1313,6 +1367,16 @@ static int machines__deliver_event(struct machines *machines,
> return 0;
> }
> dump_sample(evsel, event, sample, perf_env__arch(machine->env));
> + if (sample->deferred_callchain && tool->merge_deferred_callchains) {
> + struct deferred_event *de = malloc(sizeof(*de));
> +
> + if (de == NULL)
> + return -ENOMEM;
> +
> + de->event = event;
Here the event is assigned but not copied.
Thanks,
Ian
> + list_add_tail(&de->list, &evlist->deferred_samples);
> + return 0;
> + }
> return evlist__deliver_sample(evlist, tool, event, sample, evsel, machine);
> case PERF_RECORD_MMAP:
> return tool->mmap(tool, event, sample, machine);
> @@ -1372,7 +1436,8 @@ static int machines__deliver_event(struct machines *machines,
> return tool->aux_output_hw_id(tool, event, sample, machine);
> case PERF_RECORD_CALLCHAIN_DEFERRED:
> dump_deferred_callchain(evsel, event, sample);
> - return tool->callchain_deferred(tool, event, sample, evsel, machine);
> + return evlist__deliver_deferred_samples(evlist, tool, event,
> + sample, machine);
> default:
> ++evlist->stats.nr_unknown_events;
> return -1;
> diff --git a/tools/perf/util/tool.c b/tools/perf/util/tool.c
> index e77f0e2ecc1f79db..27ba5849c74a2e7d 100644
> --- a/tools/perf/util/tool.c
> +++ b/tools/perf/util/tool.c
> @@ -266,6 +266,7 @@ void perf_tool__init(struct perf_tool *tool, bool ordered_events)
> tool->cgroup_events = false;
> tool->no_warn = false;
> tool->show_feat_hdr = SHOW_FEAT_NO_HEADER;
> + tool->merge_deferred_callchains = true;
>
> tool->sample = process_event_sample_stub;
> tool->mmap = process_event_stub;
> @@ -448,6 +449,7 @@ void delegate_tool__init(struct delegate_tool *tool, struct perf_tool *delegate)
> tool->tool.cgroup_events = delegate->cgroup_events;
> tool->tool.no_warn = delegate->no_warn;
> tool->tool.show_feat_hdr = delegate->show_feat_hdr;
> + tool->tool.merge_deferred_callchains = delegate->merge_deferred_callchains;
>
> tool->tool.sample = delegate_sample;
> tool->tool.read = delegate_read;
> diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
> index 9b9f0a8cbf3de4b5..e96b69d25a5b737d 100644
> --- a/tools/perf/util/tool.h
> +++ b/tools/perf/util/tool.h
> @@ -90,6 +90,7 @@ struct perf_tool {
> bool cgroup_events;
> bool no_warn;
> bool dont_split_sample_group;
> + bool merge_deferred_callchains;
> enum show_feature_header show_feat_hdr;
> };
>
> --
> 2.52.0.rc1.455.g30608eb744-goog
>
^ permalink raw reply
* Re: [PATCH bpf-next v3 0/6] bpf trampoline support "jmp" mode
From: Xu Kuohai @ 2025-11-20 3:24 UTC (permalink / raw)
To: Leon Hwang, Menglong Dong, Menglong Dong, Alexei Starovoitov
Cc: Alexei Starovoitov, Steven Rostedt, Daniel Borkmann,
John Fastabend, Andrii Nakryiko, Martin KaFai Lau, Eduard,
Song Liu, Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo,
Jiri Olsa, Masami Hiramatsu, Mark Rutland, Mathieu Desnoyers,
jiang.biao, bpf, LKML, linux-trace-kernel
In-Reply-To: <e3c8daef-5267-4dda-9009-209a94224374@linux.dev>
On 11/20/2025 10:07 AM, Leon Hwang wrote:
>
>
> On 19/11/25 20:36, Xu Kuohai wrote:
>> On 11/19/2025 10:55 AM, Leon Hwang wrote:
>>>
>>>
>>> On 19/11/25 10:47, Menglong Dong wrote:
>>>> On 2025/11/19 08:28, Alexei Starovoitov wrote:
>>>>> On Tue, Nov 18, 2025 at 4:36 AM Menglong Dong
>>>>> <menglong8.dong@gmail.com> wrote:
>>>>>>
>>>>>> As we can see above, the performance of fexit increase from
>>>>>> 80.544M/s to
>>>>>> 136.540M/s, and the "fmodret" increase from 78.301M/s to 159.248M/s.
>>>>>
>>>>> Nice! Now we're talking.
>>>>>
>>>>> I think arm64 CPUs have a similar RSB-like return address predictor.
>>>>> Do we need to do something similar there?
>>>>> The question is not targeted to you, Menglong,
>>>>> just wondering.
>>>>
>>>> I did some research before, and I find that most arch
>>>> have such RSB-like stuff. I'll have a look at the loongarch
>>>> later(maybe after the LPC, as I'm forcing on the English practice),
>>>> and Leon is following the arm64.
>>>
>>> Yep, happy to take this on.
>>>
>>> I'm reviewing the arm64 JIT code now and will experiment with possible
>>> approaches to handle this as well.
>>>
>>
>> Unfortunately, the arm64 trampoline uses a tricky approach to bypass BTI
>> by using ret instruction to invoke the patched function. This conflicts
>> with the current approach, and seems there is no straightforward solution.
>>
> Hi Kuohai,
>
> Thanks for the explanation.
>
> Do you recall the original reason for using a ret instruction to bypass
> BTI in the arm64 trampoline? I'm trying to understand whether that
> constraint is fundamental or historical.
arm64 direct jump instructions (b and bl) support only a ±128 MB range.
But the distance between the trampoline and the patched function may
exceed this range. So an indirect jump is required.
With BTI enabled, indirect jump instructions (br and blr) require a landing
pad at the jump target. The target is the instruction immediately after
the call site in the patched function. It may be any instruction, including
non-landing-pad instructions. If it is ot a landing pad, a BTI exception
occurs when trampline jump back using BR/BLR.
Since the RET instruction does not require landing pad, it is chosen to
return to the patched function.
See [1] for reference.
[1] https://lore.kernel.org/bpf/20230401234144.3719742-1-xukuohai@huaweicloud.com/
> I'm wondering if we could structure the control flow like this:
>
> foo "bl" bar -> bar:
> bar "br" trampoline -> trampoline:
> trampoline "bl" -> bar func body:
As mentioned above, the problem is that the bl may be out of range.
If blr instruction is used instead, the target instruction must be a landing
pad when BTI is enabled. One approach is to reserve an extra nop at the call
site and patch it into a bti instruction at runtime when needed.
> bar func body "ret" -> trampoline
> trampoline "ret" -> foo
>
> This would introduce two "bl"s and two "ret"s, keeping the RAS balanced
> in a way similar to the x86 approach.
>
> With this structure, we could also shrink the frame layout:
>
> * SP + retaddr_off [ self ip ]
> * [ FP ]
>
> And then store the "self" return address elsewhere on the stack.
>
> Do you think something along these lines could work?
>
> Thanks,
> Leon
^ permalink raw reply
* [PATCH v5 6/6] perf tools: Flush remaining samples w/o deferred callchains
From: Namhyung Kim @ 2025-11-20 2:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Jens Remus, Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120021046.94490-1-namhyung@kernel.org>
It's possible that some kernel samples don't have matching deferred
callchain records when the profiling session was ended before the
threads came back to userspace. Let's flush the samples before
finish the session.
Also 32-bit systems can see partial mmap for the data. In that case,
deferred samples won't point to the correct data once the mapping moves
to the next portion of the file. Copy the original sample before it
unmaps the current data.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/util/session.c | 98 +++++++++++++++++++++++++++++++++++++++
1 file changed, 98 insertions(+)
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 2e777fd1bcf6707b..b781e01ddcb4876b 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1288,8 +1288,13 @@ static int evlist__deliver_sample(struct evlist *evlist, const struct perf_tool
struct deferred_event {
struct list_head list;
union perf_event *event;
+ bool allocated;
};
+/*
+ * This is called when a deferred callchain record comes up. Find all matching
+ * samples, merge the callchains and process them.
+ */
static int evlist__deliver_deferred_samples(struct evlist *evlist,
const struct perf_tool *tool,
union perf_event *event,
@@ -1331,6 +1336,86 @@ static int evlist__deliver_deferred_samples(struct evlist *evlist,
free(orig_sample.callchain);
list_del(&de->list);
+ if (de->allocated)
+ free(de->event);
+ free(de);
+
+ if (ret)
+ break;
+ }
+ return ret;
+}
+
+/*
+ * This is called when the backing mmap is about to go away. It needs to save
+ * the original sample data until it finds the matching deferred callchains.
+ */
+static void evlist__copy_deferred_samples(struct evlist *evlist,
+ const struct perf_tool *tool,
+ struct machine *machine)
+{
+ struct deferred_event *de, *tmp;
+ struct evsel *evsel;
+ int ret = 0;
+
+ list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) {
+ struct perf_sample sample;
+ size_t sz = de->event->header.size;
+ void *buf;
+
+ if (de->allocated)
+ continue;
+
+ buf = malloc(sz);
+ if (buf) {
+ memcpy(buf, de->event, sz);
+ de->event = buf;
+ de->allocated = true;
+ continue;
+ }
+
+ /* The allocation failed, flush the sample now */
+ ret = evlist__parse_sample(evlist, de->event, &sample);
+ if (ret == 0) {
+ evsel = evlist__id2evsel(evlist, sample.id);
+ evlist__deliver_sample(evlist, tool, de->event,
+ &sample, evsel, machine);
+ }
+
+ list_del(&de->list);
+ BUG_ON(de->allocated);
+ free(de);
+ }
+}
+
+/*
+ * This is called at the end of the data processing for the session. Flush the
+ * remaining samples as there's no hope for matching deferred callchains.
+ */
+static int evlist__flush_deferred_samples(struct evlist *evlist,
+ const struct perf_tool *tool,
+ struct machine *machine)
+{
+ struct deferred_event *de, *tmp;
+ struct evsel *evsel;
+ int ret = 0;
+
+ list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) {
+ struct perf_sample sample;
+
+ ret = evlist__parse_sample(evlist, de->event, &sample);
+ if (ret < 0) {
+ pr_err("failed to parse original sample\n");
+ break;
+ }
+
+ evsel = evlist__id2evsel(evlist, sample.id);
+ ret = evlist__deliver_sample(evlist, tool, de->event,
+ &sample, evsel, machine);
+
+ list_del(&de->list);
+ if (de->allocated)
+ free(de->event);
free(de);
if (ret)
@@ -1374,6 +1459,7 @@ static int machines__deliver_event(struct machines *machines,
return -ENOMEM;
de->event = event;
+ de->allocated = false;
list_add_tail(&de->list, &evlist->deferred_samples);
return 0;
}
@@ -2218,6 +2304,8 @@ reader__mmap(struct reader *rd, struct perf_session *session)
}
if (mmaps[rd->mmap_idx]) {
+ evlist__copy_deferred_samples(session->evlist, session->tool,
+ &session->machines.host);
munmap(mmaps[rd->mmap_idx], rd->mmap_size);
mmaps[rd->mmap_idx] = NULL;
}
@@ -2372,6 +2460,11 @@ static int __perf_session__process_events(struct perf_session *session)
if (err)
goto out_err;
err = auxtrace__flush_events(session, tool);
+ if (err)
+ goto out_err;
+ err = evlist__flush_deferred_samples(session->evlist,
+ session->tool,
+ &session->machines.host);
if (err)
goto out_err;
err = perf_session__flush_thread_stacks(session);
@@ -2494,6 +2587,11 @@ static int __perf_session__process_dir_events(struct perf_session *session)
if (ret)
goto out_err;
+ ret = evlist__flush_deferred_samples(session->evlist, tool,
+ &session->machines.host);
+ if (ret)
+ goto out_err;
+
ret = perf_session__flush_thread_stacks(session);
out_err:
ui_progress__finish();
--
2.52.0.rc1.455.g30608eb744-goog
^ permalink raw reply related
* [PATCH v5 5/6] perf tools: Merge deferred user callchains
From: Namhyung Kim @ 2025-11-20 2:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Jens Remus, Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120021046.94490-1-namhyung@kernel.org>
Save samples with deferred callchains in a separate list and deliver
them after merging the user callchains. If users don't want to merge
they can set tool->merge_deferred_callchains to false to prevent the
behavior.
With previous result, now perf script will show the merged callchains.
$ perf script
...
pwd 2312 121.163435: 249113 cpu/cycles/P:
ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms])
ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms])
ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms])
ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms])
ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms])
ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms])
ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kallsyms])
7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
...
The old output can be get using --no-merge-callchain option.
Also perf report can get the user callchain entry at the end.
$ perf report --no-children --stdio -q -S __build_id_parse.isra.0
# symbol: __build_id_parse.isra.0
8.40% pwd [kernel.kallsyms]
|
---__build_id_parse.isra.0
perf_event_mmap
mprotect_fixup
do_mprotect_pkey
__x64_sys_mprotect
do_syscall_64
entry_SYSCALL_64_after_hwframe
mprotect
_dl_sysdep_start
_dl_start_user
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/Documentation/perf-script.txt | 5 ++
tools/perf/builtin-inject.c | 1 +
tools/perf/builtin-report.c | 1 +
tools/perf/builtin-script.c | 4 ++
tools/perf/util/callchain.c | 29 ++++++++++
tools/perf/util/callchain.h | 3 ++
tools/perf/util/evlist.c | 1 +
tools/perf/util/evlist.h | 2 +
tools/perf/util/session.c | 67 +++++++++++++++++++++++-
tools/perf/util/tool.c | 2 +
tools/perf/util/tool.h | 1 +
11 files changed, 115 insertions(+), 1 deletion(-)
diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
index 28bec7e78bc858ba..03d1129606328d6d 100644
--- a/tools/perf/Documentation/perf-script.txt
+++ b/tools/perf/Documentation/perf-script.txt
@@ -527,6 +527,11 @@ include::itrace.txt[]
The known limitations include exception handing such as
setjmp/longjmp will have calls/returns not match.
+--merge-callchains::
+ Enable merging deferred user callchains if available. This is the
+ default behavior. If you want to see separate CALLCHAIN_DEFERRED
+ records for some reason, use --no-merge-callchains explicitly.
+
:GMEXAMPLECMD: script
:GMEXAMPLESUBCMD:
include::guest-files.txt[]
diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index bd9245d2dd41aa48..51d2721b6db9dccb 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -2527,6 +2527,7 @@ int cmd_inject(int argc, const char **argv)
inject.tool.auxtrace = perf_event__repipe_auxtrace;
inject.tool.bpf_metadata = perf_event__repipe_op2_synth;
inject.tool.dont_split_sample_group = true;
+ inject.tool.merge_deferred_callchains = false;
inject.session = __perf_session__new(&data, &inject.tool,
/*trace_event_repipe=*/inject.output.is_pipe,
/*host_env=*/NULL);
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 2bc269f5fcef8023..add6b1c2aaf04270 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -1614,6 +1614,7 @@ int cmd_report(int argc, const char **argv)
report.tool.event_update = perf_event__process_event_update;
report.tool.feature = process_feature_event;
report.tool.ordering_requires_timestamps = true;
+ report.tool.merge_deferred_callchains = !dump_trace;
session = perf_session__new(&data, &report.tool);
if (IS_ERR(session)) {
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 85b42205a71b3993..62e43d3c5ad731a0 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -4009,6 +4009,7 @@ int cmd_script(int argc, const char **argv)
bool header_only = false;
bool script_started = false;
bool unsorted_dump = false;
+ bool merge_deferred_callchains = true;
char *rec_script_path = NULL;
char *rep_script_path = NULL;
struct perf_session *session;
@@ -4162,6 +4163,8 @@ int cmd_script(int argc, const char **argv)
"Guest code can be found in hypervisor process"),
OPT_BOOLEAN('\0', "stitch-lbr", &script.stitch_lbr,
"Enable LBR callgraph stitching approach"),
+ OPT_BOOLEAN('\0', "merge-callchains", &merge_deferred_callchains,
+ "Enable merge deferred user callchains"),
OPTS_EVSWITCH(&script.evswitch),
OPT_END()
};
@@ -4418,6 +4421,7 @@ int cmd_script(int argc, const char **argv)
script.tool.throttle = process_throttle_event;
script.tool.unthrottle = process_throttle_event;
script.tool.ordering_requires_timestamps = true;
+ script.tool.merge_deferred_callchains = merge_deferred_callchains;
session = perf_session__new(&data, &script.tool);
if (IS_ERR(session))
return PTR_ERR(session);
diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c
index 2884187ccbbecfdc..71dc5a070065dd2a 100644
--- a/tools/perf/util/callchain.c
+++ b/tools/perf/util/callchain.c
@@ -1838,3 +1838,32 @@ int sample__for_each_callchain_node(struct thread *thread, struct evsel *evsel,
}
return 0;
}
+
+int sample__merge_deferred_callchain(struct perf_sample *sample_orig,
+ struct perf_sample *sample_callchain)
+{
+ u64 nr_orig = sample_orig->callchain->nr - 1;
+ u64 nr_deferred = sample_callchain->callchain->nr;
+ struct ip_callchain *callchain;
+
+ if (sample_orig->callchain->nr < 2) {
+ sample_orig->deferred_callchain = false;
+ return -EINVAL;
+ }
+
+ callchain = calloc(1 + nr_orig + nr_deferred, sizeof(u64));
+ if (callchain == NULL) {
+ sample_orig->deferred_callchain = false;
+ return -ENOMEM;
+ }
+
+ callchain->nr = nr_orig + nr_deferred;
+ /* copy original including PERF_CONTEXT_USER_DEFERRED (but the cookie) */
+ memcpy(callchain->ips, sample_orig->callchain->ips, nr_orig * sizeof(u64));
+ /* copy deferred user callchains */
+ memcpy(&callchain->ips[nr_orig], sample_callchain->callchain->ips,
+ nr_deferred * sizeof(u64));
+
+ sample_orig->callchain = callchain;
+ return 0;
+}
diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h
index d5ae4fbb7ce5fa44..2a52af8c80ace33c 100644
--- a/tools/perf/util/callchain.h
+++ b/tools/perf/util/callchain.h
@@ -318,4 +318,7 @@ int sample__for_each_callchain_node(struct thread *thread, struct evsel *evsel,
struct perf_sample *sample, int max_stack,
bool symbols, callchain_iter_fn cb, void *data);
+int sample__merge_deferred_callchain(struct perf_sample *sample_orig,
+ struct perf_sample *sample_callchain);
+
#endif /* __PERF_CALLCHAIN_H */
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index e8217efdda5323c6..03674d2cbd015e4f 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -85,6 +85,7 @@ void evlist__init(struct evlist *evlist, struct perf_cpu_map *cpus,
evlist->ctl_fd.pos = -1;
evlist->nr_br_cntr = -1;
metricgroup__rblist_init(&evlist->metric_events);
+ INIT_LIST_HEAD(&evlist->deferred_samples);
}
struct evlist *evlist__new(void)
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 5e71e3dc60423079..911834ae7c2a6f76 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -92,6 +92,8 @@ struct evlist {
* of struct metric_expr.
*/
struct rblist metric_events;
+ /* samples with deferred_callchain would wait here. */
+ struct list_head deferred_samples;
};
struct evsel_str_handler {
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 361e15c1f26a96d0..2e777fd1bcf6707b 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1285,6 +1285,60 @@ static int evlist__deliver_sample(struct evlist *evlist, const struct perf_tool
per_thread);
}
+struct deferred_event {
+ struct list_head list;
+ union perf_event *event;
+};
+
+static int evlist__deliver_deferred_samples(struct evlist *evlist,
+ const struct perf_tool *tool,
+ union perf_event *event,
+ struct perf_sample *sample,
+ struct machine *machine)
+{
+ struct deferred_event *de, *tmp;
+ struct evsel *evsel;
+ int ret = 0;
+
+ if (!tool->merge_deferred_callchains) {
+ evsel = evlist__id2evsel(evlist, sample->id);
+ return tool->callchain_deferred(tool, event, sample,
+ evsel, machine);
+ }
+
+ list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) {
+ struct perf_sample orig_sample;
+
+ ret = evlist__parse_sample(evlist, de->event, &orig_sample);
+ if (ret < 0) {
+ pr_err("failed to parse original sample\n");
+ break;
+ }
+
+ if (sample->tid != orig_sample.tid)
+ continue;
+
+ if (event->callchain_deferred.cookie == orig_sample.deferred_cookie)
+ sample__merge_deferred_callchain(&orig_sample, sample);
+ else
+ orig_sample.deferred_callchain = false;
+
+ evsel = evlist__id2evsel(evlist, orig_sample.id);
+ ret = evlist__deliver_sample(evlist, tool, de->event,
+ &orig_sample, evsel, machine);
+
+ if (orig_sample.deferred_callchain)
+ free(orig_sample.callchain);
+
+ list_del(&de->list);
+ free(de);
+
+ if (ret)
+ break;
+ }
+ return ret;
+}
+
static int machines__deliver_event(struct machines *machines,
struct evlist *evlist,
union perf_event *event,
@@ -1313,6 +1367,16 @@ static int machines__deliver_event(struct machines *machines,
return 0;
}
dump_sample(evsel, event, sample, perf_env__arch(machine->env));
+ if (sample->deferred_callchain && tool->merge_deferred_callchains) {
+ struct deferred_event *de = malloc(sizeof(*de));
+
+ if (de == NULL)
+ return -ENOMEM;
+
+ de->event = event;
+ list_add_tail(&de->list, &evlist->deferred_samples);
+ return 0;
+ }
return evlist__deliver_sample(evlist, tool, event, sample, evsel, machine);
case PERF_RECORD_MMAP:
return tool->mmap(tool, event, sample, machine);
@@ -1372,7 +1436,8 @@ static int machines__deliver_event(struct machines *machines,
return tool->aux_output_hw_id(tool, event, sample, machine);
case PERF_RECORD_CALLCHAIN_DEFERRED:
dump_deferred_callchain(evsel, event, sample);
- return tool->callchain_deferred(tool, event, sample, evsel, machine);
+ return evlist__deliver_deferred_samples(evlist, tool, event,
+ sample, machine);
default:
++evlist->stats.nr_unknown_events;
return -1;
diff --git a/tools/perf/util/tool.c b/tools/perf/util/tool.c
index e77f0e2ecc1f79db..27ba5849c74a2e7d 100644
--- a/tools/perf/util/tool.c
+++ b/tools/perf/util/tool.c
@@ -266,6 +266,7 @@ void perf_tool__init(struct perf_tool *tool, bool ordered_events)
tool->cgroup_events = false;
tool->no_warn = false;
tool->show_feat_hdr = SHOW_FEAT_NO_HEADER;
+ tool->merge_deferred_callchains = true;
tool->sample = process_event_sample_stub;
tool->mmap = process_event_stub;
@@ -448,6 +449,7 @@ void delegate_tool__init(struct delegate_tool *tool, struct perf_tool *delegate)
tool->tool.cgroup_events = delegate->cgroup_events;
tool->tool.no_warn = delegate->no_warn;
tool->tool.show_feat_hdr = delegate->show_feat_hdr;
+ tool->tool.merge_deferred_callchains = delegate->merge_deferred_callchains;
tool->tool.sample = delegate_sample;
tool->tool.read = delegate_read;
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index 9b9f0a8cbf3de4b5..e96b69d25a5b737d 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -90,6 +90,7 @@ struct perf_tool {
bool cgroup_events;
bool no_warn;
bool dont_split_sample_group;
+ bool merge_deferred_callchains;
enum show_feature_header show_feat_hdr;
};
--
2.52.0.rc1.455.g30608eb744-goog
^ permalink raw reply related
* [PATCH v5 4/6] perf script: Display PERF_RECORD_CALLCHAIN_DEFERRED
From: Namhyung Kim @ 2025-11-20 2:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Jens Remus, Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120021046.94490-1-namhyung@kernel.org>
Handle the deferred callchains in the script output.
$ perf script
...
pwd 2312 121.163435: 249113 cpu/cycles/P:
ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms])
ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms])
ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms])
ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms])
ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms])
ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms])
ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kallsyms])
b00000006 (cookie) ([unknown])
pwd 2312 121.163447: DEFERRED CALLCHAIN [cookie: b00000006]
7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/builtin-script.c | 89 +++++++++++++++++++++++++++++++++
tools/perf/util/evsel_fprintf.c | 5 +-
2 files changed, 93 insertions(+), 1 deletion(-)
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 011962e1ee0f6898..85b42205a71b3993 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -2706,6 +2706,94 @@ static int process_sample_event(const struct perf_tool *tool,
return ret;
}
+static int process_deferred_sample_event(const struct perf_tool *tool,
+ union perf_event *event,
+ struct perf_sample *sample,
+ struct evsel *evsel,
+ struct machine *machine)
+{
+ struct perf_script *scr = container_of(tool, struct perf_script, tool);
+ struct perf_event_attr *attr = &evsel->core.attr;
+ struct evsel_script *es = evsel->priv;
+ unsigned int type = output_type(attr->type);
+ struct addr_location al;
+ FILE *fp = es->fp;
+ int ret = 0;
+
+ if (output[type].fields == 0)
+ return 0;
+
+ /* Set thread to NULL to indicate addr_al and al are not initialized */
+ addr_location__init(&al);
+
+ if (perf_time__ranges_skip_sample(scr->ptime_range, scr->range_num,
+ sample->time)) {
+ goto out_put;
+ }
+
+ if (debug_mode) {
+ if (sample->time < last_timestamp) {
+ pr_err("Samples misordered, previous: %" PRIu64
+ " this: %" PRIu64 "\n", last_timestamp,
+ sample->time);
+ nr_unordered++;
+ }
+ last_timestamp = sample->time;
+ goto out_put;
+ }
+
+ if (filter_cpu(sample))
+ goto out_put;
+
+ if (machine__resolve(machine, &al, sample) < 0) {
+ pr_err("problem processing %d event, skipping it.\n",
+ event->header.type);
+ ret = -1;
+ goto out_put;
+ }
+
+ if (al.filtered)
+ goto out_put;
+
+ if (!show_event(sample, evsel, al.thread, &al, NULL))
+ goto out_put;
+
+ if (evswitch__discard(&scr->evswitch, evsel))
+ goto out_put;
+
+ perf_sample__fprintf_start(scr, sample, al.thread, evsel,
+ PERF_RECORD_CALLCHAIN_DEFERRED, fp);
+ fprintf(fp, "DEFERRED CALLCHAIN [cookie: %llx]",
+ (unsigned long long)event->callchain_deferred.cookie);
+
+ if (PRINT_FIELD(IP)) {
+ struct callchain_cursor *cursor = NULL;
+
+ if (symbol_conf.use_callchain && sample->callchain) {
+ cursor = get_tls_callchain_cursor();
+ if (thread__resolve_callchain(al.thread, cursor, evsel,
+ sample, NULL, NULL,
+ scripting_max_stack)) {
+ pr_info("cannot resolve deferred callchains\n");
+ cursor = NULL;
+ }
+ }
+
+ fputc(cursor ? '\n' : ' ', fp);
+ sample__fprintf_sym(sample, &al, 0, output[type].print_ip_opts,
+ cursor, symbol_conf.bt_stop_list, fp);
+ }
+
+ fprintf(fp, "\n");
+
+ if (verbose > 0)
+ fflush(fp);
+
+out_put:
+ addr_location__exit(&al);
+ return ret;
+}
+
// Used when scr->per_event_dump is not set
static struct evsel_script es_stdout;
@@ -4303,6 +4391,7 @@ int cmd_script(int argc, const char **argv)
perf_tool__init(&script.tool, !unsorted_dump);
script.tool.sample = process_sample_event;
+ script.tool.callchain_deferred = process_deferred_sample_event;
script.tool.mmap = perf_event__process_mmap;
script.tool.mmap2 = perf_event__process_mmap2;
script.tool.comm = perf_event__process_comm;
diff --git a/tools/perf/util/evsel_fprintf.c b/tools/perf/util/evsel_fprintf.c
index 103984b29b1e10ae..10f1a03c28601e36 100644
--- a/tools/perf/util/evsel_fprintf.c
+++ b/tools/perf/util/evsel_fprintf.c
@@ -168,7 +168,10 @@ int sample__fprintf_callchain(struct perf_sample *sample, int left_alignment,
node_al.addr = addr;
node_al.map = map__get(map);
- if (print_symoffset) {
+ if (sample->deferred_callchain &&
+ sample->deferred_cookie == node->ip) {
+ printed += fprintf(fp, "(cookie)");
+ } else if (print_symoffset) {
printed += __symbol__fprintf_symname_offs(sym, &node_al,
print_unknown_as_addr,
true, fp);
--
2.52.0.rc1.455.g30608eb744-goog
^ permalink raw reply related
* [PATCH v5 3/6] perf record: Add --call-graph fp,defer option for deferred callchains
From: Namhyung Kim @ 2025-11-20 2:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Jens Remus, Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120021046.94490-1-namhyung@kernel.org>
Add a new callchain record mode option for deferred callchains. For now
it only works with FP (frame-pointer) mode.
And add the missing feature detection logic to clear the flag on old
kernels.
$ perf record --call-graph fp,defer -vv true
...
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 136
config 0 (PERF_COUNT_HW_CPU_CYCLES)
{ sample_period, sample_freq } 4000
sample_type IP|TID|TIME|CALLCHAIN|PERIOD
read_format ID|LOST
disabled 1
inherit 1
mmap 1
comm 1
freq 1
enable_on_exec 1
task 1
sample_id_all 1
mmap2 1
comm_exec 1
ksymbol 1
bpf_event 1
defer_callchain 1
defer_output 1
------------------------------------------------------------
sys_perf_event_open: pid 162755 cpu 0 group_fd -1 flags 0x8
sys_perf_event_open failed, error -22
switching off deferred callchain support
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/Documentation/perf-config.txt | 3 +++
tools/perf/Documentation/perf-record.txt | 4 ++++
tools/perf/util/callchain.c | 16 +++++++++++++---
tools/perf/util/callchain.h | 1 +
tools/perf/util/evsel.c | 19 +++++++++++++++++++
tools/perf/util/evsel.h | 1 +
6 files changed, 41 insertions(+), 3 deletions(-)
diff --git a/tools/perf/Documentation/perf-config.txt b/tools/perf/Documentation/perf-config.txt
index c6f33565966735fe..642d1c490d9e3bcd 100644
--- a/tools/perf/Documentation/perf-config.txt
+++ b/tools/perf/Documentation/perf-config.txt
@@ -452,6 +452,9 @@ Variables
kernel space is controlled not by this option but by the
kernel config (CONFIG_UNWINDER_*).
+ The 'defer' mode can be used with 'fp' mode to enable deferred
+ user callchains (like 'fp,defer').
+
call-graph.dump-size::
The size of stack to dump in order to do post-unwinding. Default is 8192 (byte).
When using dwarf into record-mode, the default size will be used if omitted.
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 067891bd7da6edc8..e8b9aadbbfa50574 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -325,6 +325,10 @@ OPTIONS
by default. User can change the number by passing it after comma
like "--call-graph fp,32".
+ Also "defer" can be used with "fp" (like "--call-graph fp,defer") to
+ enable deferred user callchain which will collect user-space callchains
+ when the thread returns to the user space.
+
-q::
--quiet::
Don't print any warnings or messages, useful for scripting.
diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c
index d7b7eef740b9d6ed..2884187ccbbecfdc 100644
--- a/tools/perf/util/callchain.c
+++ b/tools/perf/util/callchain.c
@@ -275,9 +275,13 @@ int parse_callchain_record(const char *arg, struct callchain_param *param)
if (tok) {
unsigned long size;
- size = strtoul(tok, &name, 0);
- if (size < (unsigned) sysctl__max_stack())
- param->max_stack = size;
+ if (!strncmp(tok, "defer", sizeof("defer"))) {
+ param->defer = true;
+ } else {
+ size = strtoul(tok, &name, 0);
+ if (size < (unsigned) sysctl__max_stack())
+ param->max_stack = size;
+ }
}
break;
@@ -314,6 +318,12 @@ int parse_callchain_record(const char *arg, struct callchain_param *param)
} while (0);
free(buf);
+
+ if (param->defer && param->record_mode != CALLCHAIN_FP) {
+ pr_err("callchain: deferred callchain only works with FP\n");
+ return -EINVAL;
+ }
+
return ret;
}
diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h
index 86ed9e4d04f9ee7b..d5ae4fbb7ce5fa44 100644
--- a/tools/perf/util/callchain.h
+++ b/tools/perf/util/callchain.h
@@ -98,6 +98,7 @@ extern bool dwarf_callchain_users;
struct callchain_param {
bool enabled;
+ bool defer;
enum perf_call_graph_mode record_mode;
u32 dump_size;
enum chain_mode mode;
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 5ee3e7dee93fbbcb..7772ee9cfe3ac1c7 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -1065,6 +1065,9 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
pr_info("Disabling user space callchains for function trace event.\n");
attr->exclude_callchain_user = 1;
}
+
+ if (param->defer && !attr->exclude_callchain_user)
+ attr->defer_callchain = 1;
}
void evsel__config_callchain(struct evsel *evsel, struct record_opts *opts,
@@ -1511,6 +1514,7 @@ void evsel__config(struct evsel *evsel, struct record_opts *opts,
attr->mmap2 = track && !perf_missing_features.mmap2;
attr->comm = track;
attr->build_id = track && opts->build_id;
+ attr->defer_output = track && callchain->defer;
/*
* ksymbol is tracked separately with text poke because it needs to be
@@ -2199,6 +2203,10 @@ static int __evsel__prepare_open(struct evsel *evsel, struct perf_cpu_map *cpus,
static void evsel__disable_missing_features(struct evsel *evsel)
{
+ if (perf_missing_features.defer_callchain && evsel->core.attr.defer_callchain)
+ evsel->core.attr.defer_callchain = 0;
+ if (perf_missing_features.defer_callchain && evsel->core.attr.defer_output)
+ evsel->core.attr.defer_output = 0;
if (perf_missing_features.inherit_sample_read && evsel->core.attr.inherit &&
(evsel->core.attr.sample_type & PERF_SAMPLE_READ))
evsel->core.attr.inherit = 0;
@@ -2473,6 +2481,13 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu
/* Please add new feature detection here. */
+ attr.defer_callchain = true;
+ if (has_attr_feature(&attr, /*flags=*/0))
+ goto found;
+ perf_missing_features.defer_callchain = true;
+ pr_debug2("switching off deferred callchain support\n");
+ attr.defer_callchain = false;
+
attr.inherit = true;
attr.sample_type = PERF_SAMPLE_READ | PERF_SAMPLE_TID;
if (has_attr_feature(&attr, /*flags=*/0))
@@ -2584,6 +2599,10 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu
errno = old_errno;
check:
+ if ((evsel->core.attr.defer_callchain || evsel->core.attr.defer_output) &&
+ perf_missing_features.defer_callchain)
+ return true;
+
if (evsel->core.attr.inherit &&
(evsel->core.attr.sample_type & PERF_SAMPLE_READ) &&
perf_missing_features.inherit_sample_read)
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 3ae4ac8f9a37e009..a08130ff2e47a887 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -221,6 +221,7 @@ struct perf_missing_features {
bool branch_counters;
bool aux_action;
bool inherit_sample_read;
+ bool defer_callchain;
};
extern struct perf_missing_features perf_missing_features;
--
2.52.0.rc1.455.g30608eb744-goog
^ permalink raw reply related
* [PATCH v5 2/6] perf tools: Minimal DEFERRED_CALLCHAIN support
From: Namhyung Kim @ 2025-11-20 2:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Jens Remus, Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120021046.94490-1-namhyung@kernel.org>
Add a new event type for deferred callchains and a new callback for the
struct perf_tool. For now it doesn't actually handle the deferred
callchains but it just marks the sample if it has the PERF_CONTEXT_
USER_DEFFERED in the callchain array.
At least, perf report can dump the raw data with this change. Actually
this requires the next commit to enable attr.defer_callchain, but if you
already have a data file, it'll show the following result.
$ perf report -D
...
0x2158@perf.data [0x40]: event: 22
.
. ... raw event: size 64 bytes
. 0000: 16 00 00 00 02 00 40 00 06 00 00 00 0b 00 00 00 ......@.........
. 0010: 03 00 00 00 00 00 00 00 a7 7f 33 fe 18 7f 00 00 ..........3.....
. 0020: 0f 0e 33 fe 18 7f 00 00 48 14 33 fe 18 7f 00 00 ..3.....H.3.....
. 0030: 08 09 00 00 08 09 00 00 e6 7a e7 35 1c 00 00 00 .........z.5....
121163447014 0x2158 [0x40]: PERF_RECORD_CALLCHAIN_DEFERRED(IP, 0x2): 2312/2312: 0xb00000006
... FP chain: nr:3
..... 0: 00007f18fe337fa7
..... 1: 00007f18fe330e0f
..... 2: 00007f18fe331448
: unhandled!
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/lib/perf/include/perf/event.h | 13 ++++++++++
tools/perf/util/event.c | 1 +
tools/perf/util/evsel.c | 31 +++++++++++++++++++++--
tools/perf/util/machine.c | 1 +
tools/perf/util/perf_event_attr_fprintf.c | 2 ++
tools/perf/util/sample.h | 2 ++
tools/perf/util/session.c | 20 +++++++++++++++
tools/perf/util/tool.c | 3 +++
tools/perf/util/tool.h | 3 ++-
9 files changed, 73 insertions(+), 3 deletions(-)
diff --git a/tools/lib/perf/include/perf/event.h b/tools/lib/perf/include/perf/event.h
index aa1e91c97a226e1a..43a8cb04994fa033 100644
--- a/tools/lib/perf/include/perf/event.h
+++ b/tools/lib/perf/include/perf/event.h
@@ -151,6 +151,18 @@ struct perf_record_switch {
__u32 next_prev_tid;
};
+struct perf_record_callchain_deferred {
+ struct perf_event_header header;
+ /*
+ * This is to match kernel and (deferred) user stacks together.
+ * The kernel part will be in the sample callchain array after
+ * the PERF_CONTEXT_USER_DEFERRED entry.
+ */
+ __u64 cookie;
+ __u64 nr;
+ __u64 ips[];
+};
+
struct perf_record_header_attr {
struct perf_event_header header;
struct perf_event_attr attr;
@@ -523,6 +535,7 @@ union perf_event {
struct perf_record_read read;
struct perf_record_throttle throttle;
struct perf_record_sample sample;
+ struct perf_record_callchain_deferred callchain_deferred;
struct perf_record_bpf_event bpf;
struct perf_record_ksymbol ksymbol;
struct perf_record_text_poke_event text_poke;
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index fcf44149feb20c35..4c92cc1a952c1d9f 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -61,6 +61,7 @@ static const char *perf_event__names[] = {
[PERF_RECORD_CGROUP] = "CGROUP",
[PERF_RECORD_TEXT_POKE] = "TEXT_POKE",
[PERF_RECORD_AUX_OUTPUT_HW_ID] = "AUX_OUTPUT_HW_ID",
+ [PERF_RECORD_CALLCHAIN_DEFERRED] = "CALLCHAIN_DEFERRED",
[PERF_RECORD_HEADER_ATTR] = "ATTR",
[PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE",
[PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA",
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 989c56d4a23f74f4..5ee3e7dee93fbbcb 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -3089,6 +3089,20 @@ int evsel__parse_sample(struct evsel *evsel, union perf_event *event,
data->data_src = PERF_MEM_DATA_SRC_NONE;
data->vcpu = -1;
+ if (event->header.type == PERF_RECORD_CALLCHAIN_DEFERRED) {
+ const u64 max_callchain_nr = UINT64_MAX / sizeof(u64);
+
+ data->callchain = (struct ip_callchain *)&event->callchain_deferred.nr;
+ if (data->callchain->nr > max_callchain_nr)
+ return -EFAULT;
+
+ data->deferred_cookie = event->callchain_deferred.cookie;
+
+ if (evsel->core.attr.sample_id_all)
+ perf_evsel__parse_id_sample(evsel, event, data);
+ return 0;
+ }
+
if (event->header.type != PERF_RECORD_SAMPLE) {
if (!evsel->core.attr.sample_id_all)
return 0;
@@ -3213,12 +3227,25 @@ int evsel__parse_sample(struct evsel *evsel, union perf_event *event,
if (type & PERF_SAMPLE_CALLCHAIN) {
const u64 max_callchain_nr = UINT64_MAX / sizeof(u64);
+ u64 callchain_nr;
OVERFLOW_CHECK_u64(array);
data->callchain = (struct ip_callchain *)array++;
- if (data->callchain->nr > max_callchain_nr)
+ callchain_nr = data->callchain->nr;
+ if (callchain_nr > max_callchain_nr)
return -EFAULT;
- sz = data->callchain->nr * sizeof(u64);
+ sz = callchain_nr * sizeof(u64);
+ /*
+ * Save the cookie for the deferred user callchain. The last 2
+ * entries in the callchain should be the context marker and the
+ * cookie. The cookie will be used to match PERF_RECORD_
+ * CALLCHAIN_DEFERRED later.
+ */
+ if (evsel->core.attr.defer_callchain && callchain_nr >= 2 &&
+ data->callchain->ips[callchain_nr - 2] == PERF_CONTEXT_USER_DEFERRED) {
+ data->deferred_cookie = data->callchain->ips[callchain_nr - 1];
+ data->deferred_callchain = true;
+ }
OVERFLOW_CHECK(array, sz, max_size);
array = (void *)array + sz;
}
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index b5dd42588c916d91..841b711d970e9457 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -2124,6 +2124,7 @@ static int add_callchain_ip(struct thread *thread,
*cpumode = PERF_RECORD_MISC_KERNEL;
break;
case PERF_CONTEXT_USER:
+ case PERF_CONTEXT_USER_DEFERRED:
*cpumode = PERF_RECORD_MISC_USER;
break;
default:
diff --git a/tools/perf/util/perf_event_attr_fprintf.c b/tools/perf/util/perf_event_attr_fprintf.c
index 66b666d9ce649dd7..741c3d657a8b6ae7 100644
--- a/tools/perf/util/perf_event_attr_fprintf.c
+++ b/tools/perf/util/perf_event_attr_fprintf.c
@@ -343,6 +343,8 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_event_attr *attr,
PRINT_ATTRf(inherit_thread, p_unsigned);
PRINT_ATTRf(remove_on_exec, p_unsigned);
PRINT_ATTRf(sigtrap, p_unsigned);
+ PRINT_ATTRf(defer_callchain, p_unsigned);
+ PRINT_ATTRf(defer_output, p_unsigned);
PRINT_ATTRn("{ wakeup_events, wakeup_watermark }", wakeup_events, p_unsigned, false);
PRINT_ATTRf(bp_type, p_unsigned);
diff --git a/tools/perf/util/sample.h b/tools/perf/util/sample.h
index fae834144ef42105..a8307b20a9ea8066 100644
--- a/tools/perf/util/sample.h
+++ b/tools/perf/util/sample.h
@@ -107,6 +107,8 @@ struct perf_sample {
/** @weight3: On x86 holds retire_lat, on powerpc holds p_stage_cyc. */
u16 weight3;
bool no_hw_idx; /* No hw_idx collected in branch_stack */
+ bool deferred_callchain; /* Has deferred user callchains */
+ u64 deferred_cookie;
char insn[MAX_INSN];
void *raw_data;
struct ip_callchain *callchain;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 4b0236b2df2913e1..361e15c1f26a96d0 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -720,6 +720,7 @@ static perf_event__swap_op perf_event__swap_ops[] = {
[PERF_RECORD_CGROUP] = perf_event__cgroup_swap,
[PERF_RECORD_TEXT_POKE] = perf_event__text_poke_swap,
[PERF_RECORD_AUX_OUTPUT_HW_ID] = perf_event__all64_swap,
+ [PERF_RECORD_CALLCHAIN_DEFERRED] = perf_event__all64_swap,
[PERF_RECORD_HEADER_ATTR] = perf_event__hdr_attr_swap,
[PERF_RECORD_HEADER_EVENT_TYPE] = perf_event__event_type_swap,
[PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
@@ -854,6 +855,9 @@ static void callchain__printf(struct evsel *evsel,
for (i = 0; i < callchain->nr; i++)
printf("..... %2d: %016" PRIx64 "\n",
i, callchain->ips[i]);
+
+ if (sample->deferred_callchain)
+ printf("...... (deferred)\n");
}
static void branch_stack__printf(struct perf_sample *sample,
@@ -1123,6 +1127,19 @@ static void dump_sample(struct evsel *evsel, union perf_event *event,
sample_read__printf(sample, evsel->core.attr.read_format);
}
+static void dump_deferred_callchain(struct evsel *evsel, union perf_event *event,
+ struct perf_sample *sample)
+{
+ if (!dump_trace)
+ return;
+
+ printf("(IP, 0x%x): %d/%d: %#" PRIx64 "\n",
+ event->header.misc, sample->pid, sample->tid, sample->deferred_cookie);
+
+ if (evsel__has_callchain(evsel))
+ callchain__printf(evsel, sample);
+}
+
static void dump_read(struct evsel *evsel, union perf_event *event)
{
struct perf_record_read *read_event = &event->read;
@@ -1353,6 +1370,9 @@ static int machines__deliver_event(struct machines *machines,
return tool->text_poke(tool, event, sample, machine);
case PERF_RECORD_AUX_OUTPUT_HW_ID:
return tool->aux_output_hw_id(tool, event, sample, machine);
+ case PERF_RECORD_CALLCHAIN_DEFERRED:
+ dump_deferred_callchain(evsel, event, sample);
+ return tool->callchain_deferred(tool, event, sample, evsel, machine);
default:
++evlist->stats.nr_unknown_events;
return -1;
diff --git a/tools/perf/util/tool.c b/tools/perf/util/tool.c
index 22a8a4ffe05f778e..e77f0e2ecc1f79db 100644
--- a/tools/perf/util/tool.c
+++ b/tools/perf/util/tool.c
@@ -287,6 +287,7 @@ void perf_tool__init(struct perf_tool *tool, bool ordered_events)
tool->read = process_event_sample_stub;
tool->throttle = process_event_stub;
tool->unthrottle = process_event_stub;
+ tool->callchain_deferred = process_event_sample_stub;
tool->attr = process_event_synth_attr_stub;
tool->event_update = process_event_synth_event_update_stub;
tool->tracing_data = process_event_synth_tracing_data_stub;
@@ -335,6 +336,7 @@ bool perf_tool__compressed_is_stub(const struct perf_tool *tool)
}
CREATE_DELEGATE_SAMPLE(read);
CREATE_DELEGATE_SAMPLE(sample);
+CREATE_DELEGATE_SAMPLE(callchain_deferred);
#define CREATE_DELEGATE_ATTR(name) \
static int delegate_ ## name(const struct perf_tool *tool, \
@@ -468,6 +470,7 @@ void delegate_tool__init(struct delegate_tool *tool, struct perf_tool *delegate)
tool->tool.ksymbol = delegate_ksymbol;
tool->tool.bpf = delegate_bpf;
tool->tool.text_poke = delegate_text_poke;
+ tool->tool.callchain_deferred = delegate_callchain_deferred;
tool->tool.attr = delegate_attr;
tool->tool.event_update = delegate_event_update;
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index 88337cee1e3e2be3..9b9f0a8cbf3de4b5 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -44,7 +44,8 @@ enum show_feature_header {
struct perf_tool {
event_sample sample,
- read;
+ read,
+ callchain_deferred;
event_op mmap,
mmap2,
comm,
--
2.52.0.rc1.455.g30608eb744-goog
^ permalink raw reply related
* [PATCH v5 1/6] tools headers UAPI: Sync linux/perf_event.h for deferred callchains
From: Namhyung Kim @ 2025-11-20 2:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Jens Remus, Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251120021046.94490-1-namhyung@kernel.org>
It needs to sync with the kernel to support user space changes for the
deferred callchains.
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/include/uapi/linux/perf_event.h | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/tools/include/uapi/linux/perf_event.h b/tools/include/uapi/linux/perf_event.h
index 78a362b8002776e5..d292f96bc06f86bc 100644
--- a/tools/include/uapi/linux/perf_event.h
+++ b/tools/include/uapi/linux/perf_event.h
@@ -463,7 +463,9 @@ struct perf_event_attr {
inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */
remove_on_exec : 1, /* event is removed from task on exec */
sigtrap : 1, /* send synchronous SIGTRAP on event */
- __reserved_1 : 26;
+ defer_callchain: 1, /* request PERF_RECORD_CALLCHAIN_DEFERRED records */
+ defer_output : 1, /* output PERF_RECORD_CALLCHAIN_DEFERRED records */
+ __reserved_1 : 24;
union {
__u32 wakeup_events; /* wake up every n events */
@@ -1239,6 +1241,22 @@ enum perf_event_type {
*/
PERF_RECORD_AUX_OUTPUT_HW_ID = 21,
+ /*
+ * This user callchain capture was deferred until shortly before
+ * returning to user space. Previous samples would have kernel
+ * callchains only and they need to be stitched with this to make full
+ * callchains.
+ *
+ * struct {
+ * struct perf_event_header header;
+ * u64 cookie;
+ * u64 nr;
+ * u64 ips[nr];
+ * struct sample_id sample_id;
+ * };
+ */
+ PERF_RECORD_CALLCHAIN_DEFERRED = 22,
+
PERF_RECORD_MAX, /* non-ABI */
};
@@ -1269,6 +1287,7 @@ enum perf_callchain_context {
PERF_CONTEXT_HV = (__u64)-32,
PERF_CONTEXT_KERNEL = (__u64)-128,
PERF_CONTEXT_USER = (__u64)-512,
+ PERF_CONTEXT_USER_DEFERRED = (__u64)-640,
PERF_CONTEXT_GUEST = (__u64)-2048,
PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
--
2.52.0.rc1.455.g30608eb744-goog
^ permalink raw reply related
* [PATCHSET v5 0/6] perf tools: Add deferred callchain support
From: Namhyung Kim @ 2025-11-20 2:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, James Clark
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users, Steven Rostedt, Josh Poimboeuf, Indu Bhagat,
Jens Remus, Mathieu Desnoyers, linux-trace-kernel, bpf
Hello,
This is a new version of deferred callchain support as the kernel part
is merged to the tip tree. Actually this is based on Steve's work (v16).
https://lore.kernel.org/r/20250908175319.841517121@kernel.org
v5 changes)
* update delegate tools (Ian)
* copy and flush remaining samples (Ian)
* add Ian's Reviewed-by tags
v4: https://lore.kernel.org/r/20251115234106.348571-1-namhyung@kernel.org
* add --call-graph fp,defer option (Ian, Steve)
* add more comment on the cookie (Ian)
* display cookie part in the deferred callchain (Ian)
v3: https://lore.kernel.org/r/20251114070018.160330-1-namhyung@kernel.org
* handle new attr.defer_output to generate deferred callchains
* fix crash when cookies don't match (Steven)
* disable merging for perf inject
* fix missing feature detection bug
* symbolize merged callchains properly
Here's an example session.
$ perf record --call-graph fp,defer pwd
/home/namhyung/project/linux
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.010 MB perf.data (29 samples) ]
$ perf evlist -v
cpu/cycles/P: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0 (PERF_COUNT_HW_CPU_CYCLES),
{ sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CALLCHAIN|PERIOD,
read_format: ID|LOST, disabled: 1, inherit: 1, mmap: 1, comm: 1, freq: 1, enable_on_exec: 1,
task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1, build_id: 1,
defer_callchain: 1, defer_output: 1
$ perf script
...
pwd 2312 121.163435: 249113 cpu/cycles/P:
ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms])
ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms])
ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms])
ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms])
ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms])
ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms])
ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kallsyms])
7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
...
$ perf script --no-merge-callchains
...
pwd 2312 121.163435: 249113 cpu/cycles/P:
ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms])
ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms])
ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms])
ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms])
ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms])
ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms])
ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kallsyms])
b00000006 (cookie) ([unknown])
pwd 2312 121.163447: DEFERRED CALLCHAIN [cookie: b00000006]
7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
...
The code is available at 'perf/defer-callchain-v5' branch in
git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
Thanks,
Namhyung
Namhyung Kim (6):
tools headers UAPI: Sync linux/perf_event.h for deferred callchains
perf tools: Minimal DEFERRED_CALLCHAIN support
perf record: Add --call-graph fp,defer option for deferred callchains
perf script: Display PERF_RECORD_CALLCHAIN_DEFERRED
perf tools: Merge deferred user callchains
perf tools: Flush remaining samples w/o deferred callchains
tools/include/uapi/linux/perf_event.h | 21 ++-
tools/lib/perf/include/perf/event.h | 13 ++
tools/perf/Documentation/perf-config.txt | 3 +
tools/perf/Documentation/perf-record.txt | 4 +
tools/perf/Documentation/perf-script.txt | 5 +
tools/perf/builtin-inject.c | 1 +
tools/perf/builtin-report.c | 1 +
tools/perf/builtin-script.c | 93 +++++++++++
tools/perf/util/callchain.c | 45 +++++-
tools/perf/util/callchain.h | 4 +
tools/perf/util/event.c | 1 +
tools/perf/util/evlist.c | 1 +
tools/perf/util/evlist.h | 2 +
tools/perf/util/evsel.c | 50 +++++-
tools/perf/util/evsel.h | 1 +
tools/perf/util/evsel_fprintf.c | 5 +-
tools/perf/util/machine.c | 1 +
tools/perf/util/perf_event_attr_fprintf.c | 2 +
tools/perf/util/sample.h | 2 +
tools/perf/util/session.c | 183 ++++++++++++++++++++++
tools/perf/util/tool.c | 5 +
tools/perf/util/tool.h | 4 +-
22 files changed, 439 insertions(+), 8 deletions(-)
--
2.52.0.rc1.455.g30608eb744-goog
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox