Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Andrii Nakryiko]

On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > /proc/<pid>/maps file is extremely useful in practice for various tasks
> > involving figuring out process memory layout, what files are backing any
> > given memory range, etc. One important class of applications that
> > absolutely rely on this are profilers/stack symbolizers. They would
> > normally capture stack trace containing absolute memory addresses of
> > some functions, and would then use /proc/<pid>/maps file to file
> > corresponding backing ELF files, file offsets within them, and then
> > continue from there to get yet more information (ELF symbols, DWARF
> > information) to get human-readable symbolic information.
> >
> > As such, there are both performance and correctness requirement
> > involved. This address to VMA information translation has to be done as
> > efficiently as possible, but also not miss any VMA (especially in the
> > case of loading/unloading shared libraries).
> >
> > Unfortunately, for all the /proc/<pid>/maps file universality and
> > usefulness, it doesn't fit the above 100%.
>
> Is this a new change or has it always been this way?
>

Probably always has been this way. My first exposure to profiling and
stack symbolization was about 7 years ago, and already then
/proc/<pid>/maps was the only way to do this, and not a 100% fit even
then.

> > First, it's text based, which makes its programmatic use from
> > applications and libraries unnecessarily cumbersome and slow due to the
> > need to do text parsing to get necessary pieces of information.
>
> slow in what way?  How has it never been noticed before as a problem?

It's just inherently slower to parse text to fish out a bunch of
integers (vma_start address, offset, inode+dev and file paths are
typical pieces needed to "normalize" captured stack trace addresses).
It's not too bad in terms of programming and performance for
scanf-like APIs, but without scanf, you are dealing with splitting by
whitespaces and tons of unnecessary string allocations.

It was noticed, I think people using this for profiling/symbolization
are not necessarily well versed in kernel development and they just
get by with what kernel provides.

>
> And exact numbers are appreciated please, yes open/read/close seems
> slower than open/ioctl/close, but is it really overall an issue in the
> real world for anything?
>
> Text apis are good as everyone can handle them, ioctls are harder for
> obvious reasons.

Yes, and acknowledged the usefulness of text-based interface. But it's
my (and other people I've talked with that had to deal with these
textual interfaces) opinion that using binary interfaces are far
superior when it comes to *programmatic* usage (i.e., from
C/C++/Rust/whatever languages directly). Textual is great for bash
scripts and human debugging, of course.

>
> > Second, it's main purpose is to emit all VMAs sequentially, but in
> > practice captured addresses would fall only into a small subset of all
> > process' VMAs, mainly containing executable text. Yet, library would
> > need to parse most or all of the contents to find needed VMAs, as there
> > is no way to skip VMAs that are of no use. Efficient library can do the
> > linear pass and it is still relatively efficient, but it's definitely an
> > overhead that can be avoided, if there was a way to do more targeted
> > querying of the relevant VMA information.
>
> I don't understand, is this a bug in the current files?  If so, why not
> just fix that up?
>

It's not a bug, I think /proc/<pid>/maps was targeted to describe
*entire* address space, but for profiling and symbolization needs we
need to find only a small subset of relevant VMAs. There is nothing
wrong with existing implementation, it's just not a 100% fit for the
more specialized "let's find relevant VMAs for this set of addresses"
problem.

> And again "efficient" need to be quantified.

You probably saw patch #5 where I solve exactly the same problem in
two different ways. And the problem is typical for symbolization: you
are given a bunch of addresses within some process, we need to find
files they belong to and what file offset they are mapped to. This is
then used to, for example, match them to ELF symbols representing
functions.

>
> > Another problem when writing generic stack trace symbolization library
> > is an unfortunate performance-vs-correctness tradeoff that needs to be
> > made.
>
> What requirement has caused a "generic stack trace symbolization
> library" to be needed at all?  What is the problem you are trying to
> solve that is not already solved by existing tools?

Capturing stack trace is a very common part, especially for BPF-based
tools and applications. E.g., bpftrace allows one to capture stack
traces for some "interesting events" (whatever that is, some kernel
function call, user function call, perf event, there is tons of
flexibility). Stack traces answer "how did we get here", but it's just
an array of addresses, which need to be translated to something that
humans can make sense of.

That's what the symbolization library is helping with. This process is
multi-step, quite involved, hard to get right with a good balance of
efficiency, correctness and fullness of information (there is always a
choice of doing simplistic symbolization using just ELF symbols, or
much more expensive but also fuller symbolization using DWARF
information, which gives also file name + line number information, can
symbolize inlined functions, etc).

One such library is blazesym ([0], cc'ed Daniel, who's working on it),
which is developed by Meta for both internal use in our fleet-wide
profiler, and is also in the process of being integrated into bpftrace
(to improve bpftrace's current somewhat limited symbolization approach
based on BCC). There is also a non-Meta project (I believe Datadog)
that is using it for its own needs.

Symbolization is quite a common task, that's highly non-trivial.

  [0] https://github.com/libbpf/blazesym

>
> > Library has to make a decision to either cache parsed contents of
> > /proc/<pid>/maps for service future requests (if application requests to
> > symbolize another set of addresses, captured at some later time, which
> > is typical for periodic/continuous profiling cases) to avoid higher
> > costs of needed to re-parse this file or caching the contents in memory
> > to speed up future requests. In the former case, more memory is used for
> > the cache and there is a risk of getting stale data if application
> > loaded/unloaded shared libraries, or otherwise changed its set of VMAs
> > through additiona mmap() calls (and other means of altering memory
> > address space). In the latter case, it's the performance hit that comes
> > from re-opening the file and re-reading/re-parsing its contents all over
> > again.
>
> Again, "performance hit" needs to be justified, it shouldn't be much
> overall.

I'm not sure how to answer whether it's much or not. Can you be a bit
more specific on what you'd like to see?

But I want to say that sensitivity to any overhead differs a lot
depending on specifics. As general rule, we try to minimize any
resource usage of the profiler/symbolizer itself on the host that is
being profiled, to minimize the disruption of the production workload.
So anything that can be done to optimize any part of the overall
profiling process is a benefit.

But while for big servers tolerance might be higher in terms of
re-opening and re-parsing a bunch of text files, we also have use
cases on much less powerful and very performance sensitive Oculus VR
devices, for example. There, any extra piece of work is scrutinized,
so having to parse text on those relatively weak devices does add up.
Enough to spend effort to optimize text parsing in blazesym's Rust
code (see [1] for recent improvements).

  [1] https://github.com/libbpf/blazesym/pull/643/commits/b89b91b42b994b135a0079bf04b2319c0054f745

>
> > This patch aims to solve this problem by providing a new API built on
> > top of /proc/<pid>/maps. It is ioctl()-based and built as a binary
> > interface, avoiding the cost and awkwardness of textual representation
> > for programmatic use.
>
> Some people find text easier to handle for programmatic use :)

I don't disagree, but pretty much everyone I discussed having to deal
with text-based kernel APIs are pretty uniformly in favor of
binary-based interfaces, if they are available.

But note, I'm not proposing to deprecate or remove text-based
/proc/<pid>/maps. And the main point of this work is not so much
binary vs text, as more selecting "point-based" querying capability as
opposed to the "iterate everything" approach of /proc/<pid>/maps.

>
> > It's designed to be extensible and
> > forward/backward compatible by including user-specified field size and
> > using copy_struct_from_user() approach. But, most importantly, it allows
> > to do point queries for specific single address, specified by user. And
> > this is done efficiently using VMA iterator.
>
> Ok, maybe this is the main issue, you only want one at a time?

Yes. More or less, I need "a few" that cover a captured set of addresses.

>
> > User has a choice to pick either getting VMA that covers provided
> > address or -ENOENT if none is found (exact, least surprising, case). Or,
> > with an extra query flag (PROCFS_PROCMAP_EXACT_OR_NEXT_VMA), they can
> > get either VMA that covers the address (if there is one), or the closest
> > next VMA (i.e., VMA with the smallest vm_start > addr). The later allows
> > more efficient use, but, given it could be a surprising behavior,
> > requires an explicit opt-in.
> >
> > Basing this ioctl()-based API on top of /proc/<pid>/maps's FD makes
> > sense given it's querying the same set of VMA data. All the permissions
> > checks performed on /proc/<pid>/maps opening fit here as well.
> > ioctl-based implementation is fetching remembered mm_struct reference,
> > but otherwise doesn't interfere with seq_file-based implementation of
> > /proc/<pid>/maps textual interface, and so could be used together or
> > independently without paying any price for that.
> >
> > There is one extra thing that /proc/<pid>/maps doesn't currently
> > provide, and that's an ability to fetch ELF build ID, if present. User
> > has control over whether this piece of information is requested or not
> > by either setting build_id_size field to zero or non-zero maximum buffer
> > size they provided through build_id_addr field (which encodes user
> > pointer as __u64 field).
> >
> > The need to get ELF build ID reliably is an important aspect when
> > dealing with profiling and stack trace symbolization, and
> > /proc/<pid>/maps textual representation doesn't help with this,
> > requiring applications to open underlying ELF binary through
> > /proc/<pid>/map_files/<start>-<end> symlink, which adds an extra
> > permissions implications due giving a full access to the binary from
> > (potentially) another process, while all application is interested in is
> > build ID. Giving an ability to request just build ID doesn't introduce
> > any additional security concerns, on top of what /proc/<pid>/maps is
> > already concerned with, simplifying the overall logic.
> >
> > Kernel already implements build ID fetching, which is used from BPF
> > subsystem. We are reusing this code here, but plan a follow up changes
> > to make it work better under more relaxed assumption (compared to what
> > existing code assumes) of being called from user process context, in
> > which page faults are allowed. BPF-specific implementation currently
> > bails out if necessary part of ELF file is not paged in, all due to
> > extra BPF-specific restrictions (like the need to fetch build ID in
> > restrictive contexts such as NMI handler).
> >
> > Note also, that fetching VMA name (e.g., backing file path, or special
> > hard-coded or user-provided names) is optional just like build ID. If
> > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > it, saving resources.
> >
> > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
>
> Where is the userspace code that uses this new api you have created?

So I added a faithful comparison of existing /proc/<pid>/maps vs new
ioctl() API to solve a common problem (as described above) in patch
#5. The plan is to put it in mentioned blazesym library at the very
least.

I'm sure perf would benefit from this as well (cc'ed Arnaldo and
linux-perf-user), as they need to do stack symbolization as well.

It will be up to other similar projects to adopt this, but we'll
definitely get this into blazesym as it is actually a problem for the
abovementioned Oculus use case. We already had to make a tradeoff (see
[2], this wasn't done just because we could, but it was requested by
Oculus customers) to cache the contents of /proc/<pid>/maps and run
the risk of missing some shared libraries that can be loaded later. It
would be great to not have to do this tradeoff, which this new API
would enable.

  [2] https://github.com/libbpf/blazesym/commit/6b521314126b3ae6f2add43e93234b59fed48ccf

>
> > ---
> >  fs/proc/task_mmu.c      | 165 ++++++++++++++++++++++++++++++++++++++++
> >  include/uapi/linux/fs.h |  32 ++++++++
> >  2 files changed, 197 insertions(+)
> >
> > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > index 8e503a1635b7..cb7b1ff1a144 100644
> > --- a/fs/proc/task_mmu.c
> > +++ b/fs/proc/task_mmu.c
> > @@ -22,6 +22,7 @@
> >  #include <linux/pkeys.h>
> >  #include <linux/minmax.h>
> >  #include <linux/overflow.h>
> > +#include <linux/buildid.h>
> >
> >  #include <asm/elf.h>
> >  #include <asm/tlb.h>
> > @@ -375,11 +376,175 @@ static int pid_maps_open(struct inode *inode, struct file *file)
> >       return do_maps_open(inode, file, &proc_pid_maps_op);
> >  }
> >
> > +static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg)
> > +{
> > +     struct procfs_procmap_query karg;
> > +     struct vma_iterator iter;
> > +     struct vm_area_struct *vma;
> > +     struct mm_struct *mm;
> > +     const char *name = NULL;
> > +     char build_id_buf[BUILD_ID_SIZE_MAX], *name_buf = NULL;
> > +     __u64 usize;
> > +     int err;
> > +
> > +     if (copy_from_user(&usize, (void __user *)uarg, sizeof(usize)))
> > +             return -EFAULT;
> > +     if (usize > PAGE_SIZE)
>
> Nice, where did you document that?  And how is that portable given that
> PAGE_SIZE can be different on different systems?

I'm happy to document everything, can you please help by pointing
where this documentation has to live?

This is mostly fool-proofing, though, because the user has to pass
sizeof(struct procfs_procmap_query), which I don't see ever getting
close to even 4KB (not even saying about 64KB). This is just to
prevent copy_struct_from_user() below to do too much zero-checking.

>
> and why aren't you checking the actual structure size instead?  You can
> easily run off the end here without knowing it.

See copy_struct_from_user(), it does more checks. This is a helper
designed specifically to deal with use cases like this where kernel
struct size can change and user space might be newer or older.
copy_struct_from_user() has a nice documentation describing all these
nuances.

>
> > +             return -E2BIG;
> > +     if (usize < offsetofend(struct procfs_procmap_query, query_addr))
> > +             return -EINVAL;
>
> Ok, so you have two checks?  How can the first one ever fail?

Hmm.. If usize = 8, copy_from_user() won't fail, usize > PAGE_SIZE
won't fail, but this one will fail.

The point of this check is that user has to specify at least first
three fields of procfs_procmap_query (size, query_flags, and
query_addr), because without those the query is meaningless.
>
>
> > +     err = copy_struct_from_user(&karg, sizeof(karg), uarg, usize);

and this helper does more checks validating that the user either has a
shorter struct (and then zero-fills the rest of kernel-side struct) or
has longer (and then the longer part has to be zero filled). Do check
copy_struct_from_user() documentation, it's great.

> > +     if (err)
> > +             return err;
> > +
> > +     if (karg.query_flags & ~PROCFS_PROCMAP_EXACT_OR_NEXT_VMA)
> > +             return -EINVAL;
> > +     if (!!karg.vma_name_size != !!karg.vma_name_addr)
> > +             return -EINVAL;
> > +     if (!!karg.build_id_size != !!karg.build_id_addr)
> > +             return -EINVAL;
>
> So you want values to be set, right?

Either both should be set, or neither. It's ok for both size/addr
fields to be zero, in which case it indicates that the user doesn't
want this part of information (which is usually a bit more expensive
to get and might not be necessary for all the cases).

>
> > +
> > +     mm = priv->mm;
> > +     if (!mm || !mmget_not_zero(mm))
> > +             return -ESRCH;
>
> What is this error for?  Where is this documentned?

I copied it from existing /proc/<pid>/maps checks. I presume it's
guarding the case when mm might be already put. So if the process is
gone, but we have /proc/<pid>/maps file open?

>
> > +     if (mmap_read_lock_killable(mm)) {
> > +             mmput(mm);
> > +             return -EINTR;
> > +     }
> > +
> > +     vma_iter_init(&iter, mm, karg.query_addr);
> > +     vma = vma_next(&iter);
> > +     if (!vma) {
> > +             err = -ENOENT;
> > +             goto out;
> > +     }
> > +     /* user wants covering VMA, not the closest next one */
> > +     if (!(karg.query_flags & PROCFS_PROCMAP_EXACT_OR_NEXT_VMA) &&
> > +         vma->vm_start > karg.query_addr) {
> > +             err = -ENOENT;
> > +             goto out;
> > +     }
> > +
> > +     karg.vma_start = vma->vm_start;
> > +     karg.vma_end = vma->vm_end;
> > +
> > +     if (vma->vm_file) {
> > +             const struct inode *inode = file_user_inode(vma->vm_file);
> > +
> > +             karg.vma_offset = ((__u64)vma->vm_pgoff) << PAGE_SHIFT;
> > +             karg.dev_major = MAJOR(inode->i_sb->s_dev);
> > +             karg.dev_minor = MINOR(inode->i_sb->s_dev);
>
> So the major/minor is that of the file superblock?  Why?

Because inode number is unique only within given super block (and even
then it's more complicated, e.g., btrfs subvolumes add more headaches,
I believe). inode + dev maj/min is sometimes used for cache/reuse of
per-binary information (e.g., pre-processed DWARF information, which
is *very* expensive, so anything that allows to avoid doing this is
helpful).

>
> > +             karg.inode = inode->i_ino;
>
> What is userspace going to do with this?
>

See above.

> > +     } else {
> > +             karg.vma_offset = 0;
> > +             karg.dev_major = 0;
> > +             karg.dev_minor = 0;
> > +             karg.inode = 0;
>
> Why not set everything to 0 up above at the beginning so you never miss
> anything, and you don't miss any holes accidentally in the future.
>

Stylistic preference, I find this more explicit, but I don't care much
one way or another.

> > +     }
> > +
> > +     karg.vma_flags = 0;
> > +     if (vma->vm_flags & VM_READ)
> > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_READABLE;
> > +     if (vma->vm_flags & VM_WRITE)
> > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_WRITABLE;
> > +     if (vma->vm_flags & VM_EXEC)
> > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_EXECUTABLE;
> > +     if (vma->vm_flags & VM_MAYSHARE)
> > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_SHARED;
> > +

[...]

> > diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> > index 45e4e64fd664..fe8924a8d916 100644
> > --- a/include/uapi/linux/fs.h
> > +++ b/include/uapi/linux/fs.h
> > @@ -393,4 +393,36 @@ struct pm_scan_arg {
> >       __u64 return_mask;
> >  };
> >
> > +/* /proc/<pid>/maps ioctl */
> > +#define PROCFS_IOCTL_MAGIC 0x9f
>
> Don't you need to document this in the proper place?

I probably do, but I'm asking for help in knowing where. procfs is not
a typical area of kernel I'm working with, so any pointers are highly
appreciated.

>
> > +#define PROCFS_PROCMAP_QUERY _IOWR(PROCFS_IOCTL_MAGIC, 1, struct procfs_procmap_query)
> > +
> > +enum procmap_query_flags {
> > +     PROCFS_PROCMAP_EXACT_OR_NEXT_VMA = 0x01,
> > +};
> > +
> > +enum procmap_vma_flags {
> > +     PROCFS_PROCMAP_VMA_READABLE = 0x01,
> > +     PROCFS_PROCMAP_VMA_WRITABLE = 0x02,
> > +     PROCFS_PROCMAP_VMA_EXECUTABLE = 0x04,
> > +     PROCFS_PROCMAP_VMA_SHARED = 0x08,
>
> Are these bits?  If so, please use the bit macro for it to make it
> obvious.
>

Yes, they are. When I tried BIT(1), it didn't compile. I chose not to
add any extra #includes to this UAPI header, but I can figure out the
necessary dependency and do BIT(), I just didn't feel like BIT() adds
much here, tbh.

> > +};
> > +
> > +struct procfs_procmap_query {
> > +     __u64 size;
> > +     __u64 query_flags;              /* in */
>
> Does this map to the procmap_vma_flags enum?  if so, please say so.

no, procmap_query_flags, and yes, I will

>
> > +     __u64 query_addr;               /* in */
> > +     __u64 vma_start;                /* out */
> > +     __u64 vma_end;                  /* out */
> > +     __u64 vma_flags;                /* out */
> > +     __u64 vma_offset;               /* out */
> > +     __u64 inode;                    /* out */
>
> What is the inode for, you have an inode for the file already, why give
> it another one?

This is inode of vma's backing file, same as /proc/<pid>/maps' file
column. What inode of file do I already have here? You mean of
/proc/<pid>/maps itself? It's useless for the intended purposes.

>
> > +     __u32 dev_major;                /* out */
> > +     __u32 dev_minor;                /* out */
>
> What is major/minor for?

This is the same information as emitted by /proc/<pid>/maps,
identifies superblock of vma's backing file. As I mentioned above, it
can be used for caching per-file (i.e., per-ELF binary) information
(for example).

>
> > +     __u32 vma_name_size;            /* in/out */
> > +     __u32 build_id_size;            /* in/out */
> > +     __u64 vma_name_addr;            /* in */
> > +     __u64 build_id_addr;            /* in */
>
> Why not document this all using kerneldoc above the structure?

Yes, sorry, I slacked a bit on adding this upfront. I knew we'll be
figuring out the best place and approach, and so wanted to avoid
documentation churn.

Would something like what we have for pm_scan_arg and pagemap APIs
work? I see it added a few simple descriptions for pm_scan_arg struct,
and there is Documentation/admin-guide/mm/pagemap.rst. Should I add
Documentation/admin-guide/mm/procmap.rst (admin-guide part feels off,
though)? Anyways, I'm hoping for pointers where all this should be
documented. Thank you!

>
> anyway, I don't like ioctls, but there is a place for them, you just
> have to actually justify the use for them and not say "not efficient
> enough" as that normally isn't an issue overall.

I've written a demo tool in patch #5 which performs real-world task:
mapping addresses to their VMAs (specifically calculating file offset,
finding vma_start + vma_end range to further access files from
/proc/<pid>/map_files/<start>-<end>). I did the implementation
faithfully, doing it in the most optimal way for both APIs. I showed
that for "typical" (it's hard to specify what typical is, of course,
too many variables) scenario (it was data collected on a real server
running real service, 30 seconds of process-specific stack traces were
captured, if I remember correctly). I showed that doing exactly the
same amount of work is ~35x times slower with /proc/<pid>/maps.

Take another process, another set of addresses, another anything, and
the numbers will be different, but I think it gives the right idea.

But I think we are overpivoting on text vs binary distinction here.
It's the more targeted querying of VMAs that's beneficial here. This
allows applications to not cache anything and just re-query when doing
periodic or continuous profiling (where addresses are coming in not as
one batch, as a sequence of batches extended in time).

/proc/<pid>/maps, for all its usefulness, just can't provide this sort
of ability, as it wasn't designed to do that and is targeting
different use cases.

And then, a new ability to request reliable (it's not 100% reliable
today, I'm going to address that as a follow up) build ID is *crucial*
for some scenarios. The mentioned Oculus use case, the need to fully
access underlying ELF binary just to get build ID is frowned upon. And
for a good reason. Profiler only needs build ID, which is no secret
and not sensitive information. This new (and binary, yes) API allows
to add this into an API without breaking any backwards compatibility.

>
> thanks,
>
> greg k-h

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Arnaldo Carvalho de Melo]

On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > hard-coded or user-provided names) is optional just like build ID. If
> > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > it, saving resources.

> > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>

> > Where is the userspace code that uses this new api you have created?
 
> So I added a faithful comparison of existing /proc/<pid>/maps vs new
> ioctl() API to solve a common problem (as described above) in patch
> #5. The plan is to put it in mentioned blazesym library at the very
> least.
> 
> I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> linux-perf-user), as they need to do stack symbolization as well.

At some point, when BPF iterators became a thing we thought about, IIRC
Jiri did some experimentation, but I lost track, of using BPF to
synthesize PERF_RECORD_MMAP2 records for pre-existing maps, the layout
as in uapi/linux/perf_event.h:

        /*
         * The MMAP2 records are an augmented version of MMAP, they add
         * maj, min, ino numbers to be used to uniquely identify each mapping
         *
         * struct {
         *      struct perf_event_header        header;
         *
         *      u32                             pid, tid;
         *      u64                             addr;
         *      u64                             len;
         *      u64                             pgoff;
         *      union {
         *              struct {
         *                      u32             maj;
         *                      u32             min;
         *                      u64             ino;
         *                      u64             ino_generation;
         *              };
         *              struct {
         *                      u8              build_id_size;
         *                      u8              __reserved_1;
         *                      u16             __reserved_2;
         *                      u8              build_id[20];
         *              };
         *      };
         *      u32                             prot, flags;
         *      char                            filename[];
         *      struct sample_id                sample_id;
         * };
         */
        PERF_RECORD_MMAP2                       = 10,

 *   PERF_RECORD_MISC_MMAP_BUILD_ID      - PERF_RECORD_MMAP2 event

As perf.data files can be used for many purposes we want them all, so we
setup a meta data perf file descriptor to go on receiving the new mmaps
while we read /proc/<pid>/maps, to reduce the chance of missing maps, do
it in parallel, etc:

⬢[acme@toolbox perf-tools-next]$ perf record -h 'event synthesis'

 Usage: perf record [<options>] [<command>]
    or: perf record [<options>] -- <command> [<options>]

        --num-thread-synthesize <n>
                          number of threads to run for event synthesis
        --synth <no|all|task|mmap|cgroup>
                          Fine-tune event synthesis: default=all

⬢[acme@toolbox perf-tools-next]$

For this specific initial synthesis of everything the plan, as mentioned
about Jiri's experiments, was to use a BPF iterator to just feed the
perf ring buffer with those events, that way userspace would just
receive the usual records it gets when a new mmap is put in place, the
BPF iterator would just feed the preexisting mmaps, as instructed via
the perf_event_attr for the perf_event_open syscall.

For people not wanting BPF, i.e. disabling it altogether in perf or
disabling just BPF skels, then we would fallback to the current method,
or to the one being discussed here when it becomes available.

One thing to have in mind is for this iterator not to generate duplicate
records for non-pre-existing mmaps, i.e. we would need some generation
number that would be bumped when asking for such pre-existing maps
PERF_RECORD_MMAP2 dumps.
 
> It will be up to other similar projects to adopt this, but we'll
> definitely get this into blazesym as it is actually a problem for the

At some point looking at plugging blazesym somehow with perf may be
something to consider, indeed.

- Arnaldo

> abovementioned Oculus use case. We already had to make a tradeoff (see
> [2], this wasn't done just because we could, but it was requested by
> Oculus customers) to cache the contents of /proc/<pid>/maps and run
> the risk of missing some shared libraries that can be loaded later. It
> would be great to not have to do this tradeoff, which this new API
> would enable.
> 
>   [2] https://github.com/libbpf/blazesym/commit/6b521314126b3ae6f2add43e93234b59fed48ccf
> 
> >
> > > ---
> > >  fs/proc/task_mmu.c      | 165 ++++++++++++++++++++++++++++++++++++++++
> > >  include/uapi/linux/fs.h |  32 ++++++++
> > >  2 files changed, 197 insertions(+)
> > >
> > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > index 8e503a1635b7..cb7b1ff1a144 100644
> > > --- a/fs/proc/task_mmu.c
> > > +++ b/fs/proc/task_mmu.c
> > > @@ -22,6 +22,7 @@
> > >  #include <linux/pkeys.h>
> > >  #include <linux/minmax.h>
> > >  #include <linux/overflow.h>
> > > +#include <linux/buildid.h>
> > >
> > >  #include <asm/elf.h>
> > >  #include <asm/tlb.h>
> > > @@ -375,11 +376,175 @@ static int pid_maps_open(struct inode *inode, struct file *file)
> > >       return do_maps_open(inode, file, &proc_pid_maps_op);
> > >  }
> > >
> > > +static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg)
> > > +{
> > > +     struct procfs_procmap_query karg;
> > > +     struct vma_iterator iter;
> > > +     struct vm_area_struct *vma;
> > > +     struct mm_struct *mm;
> > > +     const char *name = NULL;
> > > +     char build_id_buf[BUILD_ID_SIZE_MAX], *name_buf = NULL;
> > > +     __u64 usize;
> > > +     int err;
> > > +
> > > +     if (copy_from_user(&usize, (void __user *)uarg, sizeof(usize)))
> > > +             return -EFAULT;
> > > +     if (usize > PAGE_SIZE)
> >
> > Nice, where did you document that?  And how is that portable given that
> > PAGE_SIZE can be different on different systems?
> 
> I'm happy to document everything, can you please help by pointing
> where this documentation has to live?
> 
> This is mostly fool-proofing, though, because the user has to pass
> sizeof(struct procfs_procmap_query), which I don't see ever getting
> close to even 4KB (not even saying about 64KB). This is just to
> prevent copy_struct_from_user() below to do too much zero-checking.
> 
> >
> > and why aren't you checking the actual structure size instead?  You can
> > easily run off the end here without knowing it.
> 
> See copy_struct_from_user(), it does more checks. This is a helper
> designed specifically to deal with use cases like this where kernel
> struct size can change and user space might be newer or older.
> copy_struct_from_user() has a nice documentation describing all these
> nuances.
> 
> >
> > > +             return -E2BIG;
> > > +     if (usize < offsetofend(struct procfs_procmap_query, query_addr))
> > > +             return -EINVAL;
> >
> > Ok, so you have two checks?  How can the first one ever fail?
> 
> Hmm.. If usize = 8, copy_from_user() won't fail, usize > PAGE_SIZE
> won't fail, but this one will fail.
> 
> The point of this check is that user has to specify at least first
> three fields of procfs_procmap_query (size, query_flags, and
> query_addr), because without those the query is meaningless.
> >
> >
> > > +     err = copy_struct_from_user(&karg, sizeof(karg), uarg, usize);
> 
> and this helper does more checks validating that the user either has a
> shorter struct (and then zero-fills the rest of kernel-side struct) or
> has longer (and then the longer part has to be zero filled). Do check
> copy_struct_from_user() documentation, it's great.
> 
> > > +     if (err)
> > > +             return err;
> > > +
> > > +     if (karg.query_flags & ~PROCFS_PROCMAP_EXACT_OR_NEXT_VMA)
> > > +             return -EINVAL;
> > > +     if (!!karg.vma_name_size != !!karg.vma_name_addr)
> > > +             return -EINVAL;
> > > +     if (!!karg.build_id_size != !!karg.build_id_addr)
> > > +             return -EINVAL;
> >
> > So you want values to be set, right?
> 
> Either both should be set, or neither. It's ok for both size/addr
> fields to be zero, in which case it indicates that the user doesn't
> want this part of information (which is usually a bit more expensive
> to get and might not be necessary for all the cases).
> 
> >
> > > +
> > > +     mm = priv->mm;
> > > +     if (!mm || !mmget_not_zero(mm))
> > > +             return -ESRCH;
> >
> > What is this error for?  Where is this documentned?
> 
> I copied it from existing /proc/<pid>/maps checks. I presume it's
> guarding the case when mm might be already put. So if the process is
> gone, but we have /proc/<pid>/maps file open?
> 
> >
> > > +     if (mmap_read_lock_killable(mm)) {
> > > +             mmput(mm);
> > > +             return -EINTR;
> > > +     }
> > > +
> > > +     vma_iter_init(&iter, mm, karg.query_addr);
> > > +     vma = vma_next(&iter);
> > > +     if (!vma) {
> > > +             err = -ENOENT;
> > > +             goto out;
> > > +     }
> > > +     /* user wants covering VMA, not the closest next one */
> > > +     if (!(karg.query_flags & PROCFS_PROCMAP_EXACT_OR_NEXT_VMA) &&
> > > +         vma->vm_start > karg.query_addr) {
> > > +             err = -ENOENT;
> > > +             goto out;
> > > +     }
> > > +
> > > +     karg.vma_start = vma->vm_start;
> > > +     karg.vma_end = vma->vm_end;
> > > +
> > > +     if (vma->vm_file) {
> > > +             const struct inode *inode = file_user_inode(vma->vm_file);
> > > +
> > > +             karg.vma_offset = ((__u64)vma->vm_pgoff) << PAGE_SHIFT;
> > > +             karg.dev_major = MAJOR(inode->i_sb->s_dev);
> > > +             karg.dev_minor = MINOR(inode->i_sb->s_dev);
> >
> > So the major/minor is that of the file superblock?  Why?
> 
> Because inode number is unique only within given super block (and even
> then it's more complicated, e.g., btrfs subvolumes add more headaches,
> I believe). inode + dev maj/min is sometimes used for cache/reuse of
> per-binary information (e.g., pre-processed DWARF information, which
> is *very* expensive, so anything that allows to avoid doing this is
> helpful).
> 
> >
> > > +             karg.inode = inode->i_ino;
> >
> > What is userspace going to do with this?
> >
> 
> See above.
> 
> > > +     } else {
> > > +             karg.vma_offset = 0;
> > > +             karg.dev_major = 0;
> > > +             karg.dev_minor = 0;
> > > +             karg.inode = 0;
> >
> > Why not set everything to 0 up above at the beginning so you never miss
> > anything, and you don't miss any holes accidentally in the future.
> >
> 
> Stylistic preference, I find this more explicit, but I don't care much
> one way or another.
> 
> > > +     }
> > > +
> > > +     karg.vma_flags = 0;
> > > +     if (vma->vm_flags & VM_READ)
> > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_READABLE;
> > > +     if (vma->vm_flags & VM_WRITE)
> > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_WRITABLE;
> > > +     if (vma->vm_flags & VM_EXEC)
> > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_EXECUTABLE;
> > > +     if (vma->vm_flags & VM_MAYSHARE)
> > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_SHARED;
> > > +
> 
> [...]
> 
> > > diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> > > index 45e4e64fd664..fe8924a8d916 100644
> > > --- a/include/uapi/linux/fs.h
> > > +++ b/include/uapi/linux/fs.h
> > > @@ -393,4 +393,36 @@ struct pm_scan_arg {
> > >       __u64 return_mask;
> > >  };
> > >
> > > +/* /proc/<pid>/maps ioctl */
> > > +#define PROCFS_IOCTL_MAGIC 0x9f
> >
> > Don't you need to document this in the proper place?
> 
> I probably do, but I'm asking for help in knowing where. procfs is not
> a typical area of kernel I'm working with, so any pointers are highly
> appreciated.
> 
> >
> > > +#define PROCFS_PROCMAP_QUERY _IOWR(PROCFS_IOCTL_MAGIC, 1, struct procfs_procmap_query)
> > > +
> > > +enum procmap_query_flags {
> > > +     PROCFS_PROCMAP_EXACT_OR_NEXT_VMA = 0x01,
> > > +};
> > > +
> > > +enum procmap_vma_flags {
> > > +     PROCFS_PROCMAP_VMA_READABLE = 0x01,
> > > +     PROCFS_PROCMAP_VMA_WRITABLE = 0x02,
> > > +     PROCFS_PROCMAP_VMA_EXECUTABLE = 0x04,
> > > +     PROCFS_PROCMAP_VMA_SHARED = 0x08,
> >
> > Are these bits?  If so, please use the bit macro for it to make it
> > obvious.
> >
> 
> Yes, they are. When I tried BIT(1), it didn't compile. I chose not to
> add any extra #includes to this UAPI header, but I can figure out the
> necessary dependency and do BIT(), I just didn't feel like BIT() adds
> much here, tbh.
> 
> > > +};
> > > +
> > > +struct procfs_procmap_query {
> > > +     __u64 size;
> > > +     __u64 query_flags;              /* in */
> >
> > Does this map to the procmap_vma_flags enum?  if so, please say so.
> 
> no, procmap_query_flags, and yes, I will
> 
> >
> > > +     __u64 query_addr;               /* in */
> > > +     __u64 vma_start;                /* out */
> > > +     __u64 vma_end;                  /* out */
> > > +     __u64 vma_flags;                /* out */
> > > +     __u64 vma_offset;               /* out */
> > > +     __u64 inode;                    /* out */
> >
> > What is the inode for, you have an inode for the file already, why give
> > it another one?
> 
> This is inode of vma's backing file, same as /proc/<pid>/maps' file
> column. What inode of file do I already have here? You mean of
> /proc/<pid>/maps itself? It's useless for the intended purposes.
> 
> >
> > > +     __u32 dev_major;                /* out */
> > > +     __u32 dev_minor;                /* out */
> >
> > What is major/minor for?
> 
> This is the same information as emitted by /proc/<pid>/maps,
> identifies superblock of vma's backing file. As I mentioned above, it
> can be used for caching per-file (i.e., per-ELF binary) information
> (for example).
> 
> >
> > > +     __u32 vma_name_size;            /* in/out */
> > > +     __u32 build_id_size;            /* in/out */
> > > +     __u64 vma_name_addr;            /* in */
> > > +     __u64 build_id_addr;            /* in */
> >
> > Why not document this all using kerneldoc above the structure?
> 
> Yes, sorry, I slacked a bit on adding this upfront. I knew we'll be
> figuring out the best place and approach, and so wanted to avoid
> documentation churn.
> 
> Would something like what we have for pm_scan_arg and pagemap APIs
> work? I see it added a few simple descriptions for pm_scan_arg struct,
> and there is Documentation/admin-guide/mm/pagemap.rst. Should I add
> Documentation/admin-guide/mm/procmap.rst (admin-guide part feels off,
> though)? Anyways, I'm hoping for pointers where all this should be
> documented. Thank you!
> 
> >
> > anyway, I don't like ioctls, but there is a place for them, you just
> > have to actually justify the use for them and not say "not efficient
> > enough" as that normally isn't an issue overall.
> 
> I've written a demo tool in patch #5 which performs real-world task:
> mapping addresses to their VMAs (specifically calculating file offset,
> finding vma_start + vma_end range to further access files from
> /proc/<pid>/map_files/<start>-<end>). I did the implementation
> faithfully, doing it in the most optimal way for both APIs. I showed
> that for "typical" (it's hard to specify what typical is, of course,
> too many variables) scenario (it was data collected on a real server
> running real service, 30 seconds of process-specific stack traces were
> captured, if I remember correctly). I showed that doing exactly the
> same amount of work is ~35x times slower with /proc/<pid>/maps.
> 
> Take another process, another set of addresses, another anything, and
> the numbers will be different, but I think it gives the right idea.
> 
> But I think we are overpivoting on text vs binary distinction here.
> It's the more targeted querying of VMAs that's beneficial here. This
> allows applications to not cache anything and just re-query when doing
> periodic or continuous profiling (where addresses are coming in not as
> one batch, as a sequence of batches extended in time).
> 
> /proc/<pid>/maps, for all its usefulness, just can't provide this sort
> of ability, as it wasn't designed to do that and is targeting
> different use cases.
> 
> And then, a new ability to request reliable (it's not 100% reliable
> today, I'm going to address that as a follow up) build ID is *crucial*
> for some scenarios. The mentioned Oculus use case, the need to fully
> access underlying ELF binary just to get build ID is frowned upon. And
> for a good reason. Profiler only needs build ID, which is no secret
> and not sensitive information. This new (and binary, yes) API allows
> to add this into an API without breaking any backwards compatibility.
> 
> >
> > thanks,
> >
> > greg k-h

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Namhyung Kim]

Hello,

On Mon, May 6, 2024 at 6:58 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>
> On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > > hard-coded or user-provided names) is optional just like build ID. If
> > > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > > it, saving resources.
>
> > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
>
> > > Where is the userspace code that uses this new api you have created?
>
> > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > ioctl() API to solve a common problem (as described above) in patch
> > #5. The plan is to put it in mentioned blazesym library at the very
> > least.
> >
> > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > linux-perf-user), as they need to do stack symbolization as well.

I think the general use case in perf is different.  This ioctl API is great
for live tracing of a single (or a small number of) process(es).  And
yes, perf tools have those tracing use cases too.  But I think the
major use case of perf tools is system-wide profiling.

For system-wide profiling, you need to process samples of many
different processes at a high frequency.  Now perf record doesn't
process them and just save it for offline processing (well, it does
at the end to find out build-ID but it can be omitted).

Doing it online is possible (like perf top) but it would add more
overhead during the profiling.  And we cannot move processing
or symbolization to the end of profiling because some (short-
lived) tasks can go away.

Also it should support perf report (offline) on data from a
different kernel or even a different machine.

So it saves the memory map of processes and symbolizes
the stack trace with it later.  Of course it needs to be updated
as the memory map changes and that's why it tracks mmap
or similar syscalls with PERF_RECORD_MMAP[2] records.

A problem with this approach is to get the initial state of all
(or a target for non-system-wide mode) existing processes.
We call it synthesizing, and read /proc/PID/maps to generate
the mmap records.

I think the below comment from Arnaldo talked about how
we can improve the synthesizing (which is sequential access
to proc maps) using BPF.

Thanks,
Namhyung


>
> At some point, when BPF iterators became a thing we thought about, IIRC
> Jiri did some experimentation, but I lost track, of using BPF to
> synthesize PERF_RECORD_MMAP2 records for pre-existing maps, the layout
> as in uapi/linux/perf_event.h:
>
>         /*
>          * The MMAP2 records are an augmented version of MMAP, they add
>          * maj, min, ino numbers to be used to uniquely identify each mapping
>          *
>          * struct {
>          *      struct perf_event_header        header;
>          *
>          *      u32                             pid, tid;
>          *      u64                             addr;
>          *      u64                             len;
>          *      u64                             pgoff;
>          *      union {
>          *              struct {
>          *                      u32             maj;
>          *                      u32             min;
>          *                      u64             ino;
>          *                      u64             ino_generation;
>          *              };
>          *              struct {
>          *                      u8              build_id_size;
>          *                      u8              __reserved_1;
>          *                      u16             __reserved_2;
>          *                      u8              build_id[20];
>          *              };
>          *      };
>          *      u32                             prot, flags;
>          *      char                            filename[];
>          *      struct sample_id                sample_id;
>          * };
>          */
>         PERF_RECORD_MMAP2                       = 10,
>
>  *   PERF_RECORD_MISC_MMAP_BUILD_ID      - PERF_RECORD_MMAP2 event
>
> As perf.data files can be used for many purposes we want them all, so we
> setup a meta data perf file descriptor to go on receiving the new mmaps
> while we read /proc/<pid>/maps, to reduce the chance of missing maps, do
> it in parallel, etc:
>
> ⬢[acme@toolbox perf-tools-next]$ perf record -h 'event synthesis'
>
>  Usage: perf record [<options>] [<command>]
>     or: perf record [<options>] -- <command> [<options>]
>
>         --num-thread-synthesize <n>
>                           number of threads to run for event synthesis
>         --synth <no|all|task|mmap|cgroup>
>                           Fine-tune event synthesis: default=all
>
> ⬢[acme@toolbox perf-tools-next]$
>
> For this specific initial synthesis of everything the plan, as mentioned
> about Jiri's experiments, was to use a BPF iterator to just feed the
> perf ring buffer with those events, that way userspace would just
> receive the usual records it gets when a new mmap is put in place, the
> BPF iterator would just feed the preexisting mmaps, as instructed via
> the perf_event_attr for the perf_event_open syscall.
>
> For people not wanting BPF, i.e. disabling it altogether in perf or
> disabling just BPF skels, then we would fallback to the current method,
> or to the one being discussed here when it becomes available.
>
> One thing to have in mind is for this iterator not to generate duplicate
> records for non-pre-existing mmaps, i.e. we would need some generation
> number that would be bumped when asking for such pre-existing maps
> PERF_RECORD_MMAP2 dumps.
>
> > It will be up to other similar projects to adopt this, but we'll
> > definitely get this into blazesym as it is actually a problem for the
>
> At some point looking at plugging blazesym somehow with perf may be
> something to consider, indeed.
>
> - Arnaldo
>
> > abovementioned Oculus use case. We already had to make a tradeoff (see
> > [2], this wasn't done just because we could, but it was requested by
> > Oculus customers) to cache the contents of /proc/<pid>/maps and run
> > the risk of missing some shared libraries that can be loaded later. It
> > would be great to not have to do this tradeoff, which this new API
> > would enable.
> >
> >   [2] https://github.com/libbpf/blazesym/commit/6b521314126b3ae6f2add43e93234b59fed48ccf
> >
> > >
> > > > ---
> > > >  fs/proc/task_mmu.c      | 165 ++++++++++++++++++++++++++++++++++++++++
> > > >  include/uapi/linux/fs.h |  32 ++++++++
> > > >  2 files changed, 197 insertions(+)
> > > >
> > > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > > index 8e503a1635b7..cb7b1ff1a144 100644
> > > > --- a/fs/proc/task_mmu.c
> > > > +++ b/fs/proc/task_mmu.c
> > > > @@ -22,6 +22,7 @@
> > > >  #include <linux/pkeys.h>
> > > >  #include <linux/minmax.h>
> > > >  #include <linux/overflow.h>
> > > > +#include <linux/buildid.h>
> > > >
> > > >  #include <asm/elf.h>
> > > >  #include <asm/tlb.h>
> > > > @@ -375,11 +376,175 @@ static int pid_maps_open(struct inode *inode, struct file *file)
> > > >       return do_maps_open(inode, file, &proc_pid_maps_op);
> > > >  }
> > > >
> > > > +static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg)
> > > > +{
> > > > +     struct procfs_procmap_query karg;
> > > > +     struct vma_iterator iter;
> > > > +     struct vm_area_struct *vma;
> > > > +     struct mm_struct *mm;
> > > > +     const char *name = NULL;
> > > > +     char build_id_buf[BUILD_ID_SIZE_MAX], *name_buf = NULL;
> > > > +     __u64 usize;
> > > > +     int err;
> > > > +
> > > > +     if (copy_from_user(&usize, (void __user *)uarg, sizeof(usize)))
> > > > +             return -EFAULT;
> > > > +     if (usize > PAGE_SIZE)
> > >
> > > Nice, where did you document that?  And how is that portable given that
> > > PAGE_SIZE can be different on different systems?
> >
> > I'm happy to document everything, can you please help by pointing
> > where this documentation has to live?
> >
> > This is mostly fool-proofing, though, because the user has to pass
> > sizeof(struct procfs_procmap_query), which I don't see ever getting
> > close to even 4KB (not even saying about 64KB). This is just to
> > prevent copy_struct_from_user() below to do too much zero-checking.
> >
> > >
> > > and why aren't you checking the actual structure size instead?  You can
> > > easily run off the end here without knowing it.
> >
> > See copy_struct_from_user(), it does more checks. This is a helper
> > designed specifically to deal with use cases like this where kernel
> > struct size can change and user space might be newer or older.
> > copy_struct_from_user() has a nice documentation describing all these
> > nuances.
> >
> > >
> > > > +             return -E2BIG;
> > > > +     if (usize < offsetofend(struct procfs_procmap_query, query_addr))
> > > > +             return -EINVAL;
> > >
> > > Ok, so you have two checks?  How can the first one ever fail?
> >
> > Hmm.. If usize = 8, copy_from_user() won't fail, usize > PAGE_SIZE
> > won't fail, but this one will fail.
> >
> > The point of this check is that user has to specify at least first
> > three fields of procfs_procmap_query (size, query_flags, and
> > query_addr), because without those the query is meaningless.
> > >
> > >
> > > > +     err = copy_struct_from_user(&karg, sizeof(karg), uarg, usize);
> >
> > and this helper does more checks validating that the user either has a
> > shorter struct (and then zero-fills the rest of kernel-side struct) or
> > has longer (and then the longer part has to be zero filled). Do check
> > copy_struct_from_user() documentation, it's great.
> >
> > > > +     if (err)
> > > > +             return err;
> > > > +
> > > > +     if (karg.query_flags & ~PROCFS_PROCMAP_EXACT_OR_NEXT_VMA)
> > > > +             return -EINVAL;
> > > > +     if (!!karg.vma_name_size != !!karg.vma_name_addr)
> > > > +             return -EINVAL;
> > > > +     if (!!karg.build_id_size != !!karg.build_id_addr)
> > > > +             return -EINVAL;
> > >
> > > So you want values to be set, right?
> >
> > Either both should be set, or neither. It's ok for both size/addr
> > fields to be zero, in which case it indicates that the user doesn't
> > want this part of information (which is usually a bit more expensive
> > to get and might not be necessary for all the cases).
> >
> > >
> > > > +
> > > > +     mm = priv->mm;
> > > > +     if (!mm || !mmget_not_zero(mm))
> > > > +             return -ESRCH;
> > >
> > > What is this error for?  Where is this documentned?
> >
> > I copied it from existing /proc/<pid>/maps checks. I presume it's
> > guarding the case when mm might be already put. So if the process is
> > gone, but we have /proc/<pid>/maps file open?
> >
> > >
> > > > +     if (mmap_read_lock_killable(mm)) {
> > > > +             mmput(mm);
> > > > +             return -EINTR;
> > > > +     }
> > > > +
> > > > +     vma_iter_init(&iter, mm, karg.query_addr);
> > > > +     vma = vma_next(&iter);
> > > > +     if (!vma) {
> > > > +             err = -ENOENT;
> > > > +             goto out;
> > > > +     }
> > > > +     /* user wants covering VMA, not the closest next one */
> > > > +     if (!(karg.query_flags & PROCFS_PROCMAP_EXACT_OR_NEXT_VMA) &&
> > > > +         vma->vm_start > karg.query_addr) {
> > > > +             err = -ENOENT;
> > > > +             goto out;
> > > > +     }
> > > > +
> > > > +     karg.vma_start = vma->vm_start;
> > > > +     karg.vma_end = vma->vm_end;
> > > > +
> > > > +     if (vma->vm_file) {
> > > > +             const struct inode *inode = file_user_inode(vma->vm_file);
> > > > +
> > > > +             karg.vma_offset = ((__u64)vma->vm_pgoff) << PAGE_SHIFT;
> > > > +             karg.dev_major = MAJOR(inode->i_sb->s_dev);
> > > > +             karg.dev_minor = MINOR(inode->i_sb->s_dev);
> > >
> > > So the major/minor is that of the file superblock?  Why?
> >
> > Because inode number is unique only within given super block (and even
> > then it's more complicated, e.g., btrfs subvolumes add more headaches,
> > I believe). inode + dev maj/min is sometimes used for cache/reuse of
> > per-binary information (e.g., pre-processed DWARF information, which
> > is *very* expensive, so anything that allows to avoid doing this is
> > helpful).
> >
> > >
> > > > +             karg.inode = inode->i_ino;
> > >
> > > What is userspace going to do with this?
> > >
> >
> > See above.
> >
> > > > +     } else {
> > > > +             karg.vma_offset = 0;
> > > > +             karg.dev_major = 0;
> > > > +             karg.dev_minor = 0;
> > > > +             karg.inode = 0;
> > >
> > > Why not set everything to 0 up above at the beginning so you never miss
> > > anything, and you don't miss any holes accidentally in the future.
> > >
> >
> > Stylistic preference, I find this more explicit, but I don't care much
> > one way or another.
> >
> > > > +     }
> > > > +
> > > > +     karg.vma_flags = 0;
> > > > +     if (vma->vm_flags & VM_READ)
> > > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_READABLE;
> > > > +     if (vma->vm_flags & VM_WRITE)
> > > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_WRITABLE;
> > > > +     if (vma->vm_flags & VM_EXEC)
> > > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_EXECUTABLE;
> > > > +     if (vma->vm_flags & VM_MAYSHARE)
> > > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_SHARED;
> > > > +
> >
> > [...]
> >
> > > > diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> > > > index 45e4e64fd664..fe8924a8d916 100644
> > > > --- a/include/uapi/linux/fs.h
> > > > +++ b/include/uapi/linux/fs.h
> > > > @@ -393,4 +393,36 @@ struct pm_scan_arg {
> > > >       __u64 return_mask;
> > > >  };
> > > >
> > > > +/* /proc/<pid>/maps ioctl */
> > > > +#define PROCFS_IOCTL_MAGIC 0x9f
> > >
> > > Don't you need to document this in the proper place?
> >
> > I probably do, but I'm asking for help in knowing where. procfs is not
> > a typical area of kernel I'm working with, so any pointers are highly
> > appreciated.
> >
> > >
> > > > +#define PROCFS_PROCMAP_QUERY _IOWR(PROCFS_IOCTL_MAGIC, 1, struct procfs_procmap_query)
> > > > +
> > > > +enum procmap_query_flags {
> > > > +     PROCFS_PROCMAP_EXACT_OR_NEXT_VMA = 0x01,
> > > > +};
> > > > +
> > > > +enum procmap_vma_flags {
> > > > +     PROCFS_PROCMAP_VMA_READABLE = 0x01,
> > > > +     PROCFS_PROCMAP_VMA_WRITABLE = 0x02,
> > > > +     PROCFS_PROCMAP_VMA_EXECUTABLE = 0x04,
> > > > +     PROCFS_PROCMAP_VMA_SHARED = 0x08,
> > >
> > > Are these bits?  If so, please use the bit macro for it to make it
> > > obvious.
> > >
> >
> > Yes, they are. When I tried BIT(1), it didn't compile. I chose not to
> > add any extra #includes to this UAPI header, but I can figure out the
> > necessary dependency and do BIT(), I just didn't feel like BIT() adds
> > much here, tbh.
> >
> > > > +};
> > > > +
> > > > +struct procfs_procmap_query {
> > > > +     __u64 size;
> > > > +     __u64 query_flags;              /* in */
> > >
> > > Does this map to the procmap_vma_flags enum?  if so, please say so.
> >
> > no, procmap_query_flags, and yes, I will
> >
> > >
> > > > +     __u64 query_addr;               /* in */
> > > > +     __u64 vma_start;                /* out */
> > > > +     __u64 vma_end;                  /* out */
> > > > +     __u64 vma_flags;                /* out */
> > > > +     __u64 vma_offset;               /* out */
> > > > +     __u64 inode;                    /* out */
> > >
> > > What is the inode for, you have an inode for the file already, why give
> > > it another one?
> >
> > This is inode of vma's backing file, same as /proc/<pid>/maps' file
> > column. What inode of file do I already have here? You mean of
> > /proc/<pid>/maps itself? It's useless for the intended purposes.
> >
> > >
> > > > +     __u32 dev_major;                /* out */
> > > > +     __u32 dev_minor;                /* out */
> > >
> > > What is major/minor for?
> >
> > This is the same information as emitted by /proc/<pid>/maps,
> > identifies superblock of vma's backing file. As I mentioned above, it
> > can be used for caching per-file (i.e., per-ELF binary) information
> > (for example).
> >
> > >
> > > > +     __u32 vma_name_size;            /* in/out */
> > > > +     __u32 build_id_size;            /* in/out */
> > > > +     __u64 vma_name_addr;            /* in */
> > > > +     __u64 build_id_addr;            /* in */
> > >
> > > Why not document this all using kerneldoc above the structure?
> >
> > Yes, sorry, I slacked a bit on adding this upfront. I knew we'll be
> > figuring out the best place and approach, and so wanted to avoid
> > documentation churn.
> >
> > Would something like what we have for pm_scan_arg and pagemap APIs
> > work? I see it added a few simple descriptions for pm_scan_arg struct,
> > and there is Documentation/admin-guide/mm/pagemap.rst. Should I add
> > Documentation/admin-guide/mm/procmap.rst (admin-guide part feels off,
> > though)? Anyways, I'm hoping for pointers where all this should be
> > documented. Thank you!
> >
> > >
> > > anyway, I don't like ioctls, but there is a place for them, you just
> > > have to actually justify the use for them and not say "not efficient
> > > enough" as that normally isn't an issue overall.
> >
> > I've written a demo tool in patch #5 which performs real-world task:
> > mapping addresses to their VMAs (specifically calculating file offset,
> > finding vma_start + vma_end range to further access files from
> > /proc/<pid>/map_files/<start>-<end>). I did the implementation
> > faithfully, doing it in the most optimal way for both APIs. I showed
> > that for "typical" (it's hard to specify what typical is, of course,
> > too many variables) scenario (it was data collected on a real server
> > running real service, 30 seconds of process-specific stack traces were
> > captured, if I remember correctly). I showed that doing exactly the
> > same amount of work is ~35x times slower with /proc/<pid>/maps.
> >
> > Take another process, another set of addresses, another anything, and
> > the numbers will be different, but I think it gives the right idea.
> >
> > But I think we are overpivoting on text vs binary distinction here.
> > It's the more targeted querying of VMAs that's beneficial here. This
> > allows applications to not cache anything and just re-query when doing
> > periodic or continuous profiling (where addresses are coming in not as
> > one batch, as a sequence of batches extended in time).
> >
> > /proc/<pid>/maps, for all its usefulness, just can't provide this sort
> > of ability, as it wasn't designed to do that and is targeting
> > different use cases.
> >
> > And then, a new ability to request reliable (it's not 100% reliable
> > today, I'm going to address that as a follow up) build ID is *crucial*
> > for some scenarios. The mentioned Oculus use case, the need to fully
> > access underlying ELF binary just to get build ID is frowned upon. And
> > for a good reason. Profiler only needs build ID, which is no secret
> > and not sensitive information. This new (and binary, yes) API allows
> > to add this into an API without breaking any backwards compatibility.
> >
> > >
> > > thanks,
> > >
> > > greg k-h
>

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Andrii Nakryiko]

On Mon, May 6, 2024 at 11:05 AM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Hello,
>
> On Mon, May 6, 2024 at 6:58 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >
> > On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > > On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > > > hard-coded or user-provided names) is optional just like build ID. If
> > > > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > > > it, saving resources.
> >
> > > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> >
> > > > Where is the userspace code that uses this new api you have created?
> >
> > > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > > ioctl() API to solve a common problem (as described above) in patch
> > > #5. The plan is to put it in mentioned blazesym library at the very
> > > least.
> > >
> > > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > > linux-perf-user), as they need to do stack symbolization as well.
>
> I think the general use case in perf is different.  This ioctl API is great
> for live tracing of a single (or a small number of) process(es).  And
> yes, perf tools have those tracing use cases too.  But I think the
> major use case of perf tools is system-wide profiling.

The intended use case is also a system-wide profiling, but I haven't
heard that opening a file per process is a big bottleneck or a
limitation, tbh.

>
> For system-wide profiling, you need to process samples of many
> different processes at a high frequency.  Now perf record doesn't
> process them and just save it for offline processing (well, it does
> at the end to find out build-ID but it can be omitted).
>
> Doing it online is possible (like perf top) but it would add more
> overhead during the profiling.  And we cannot move processing
> or symbolization to the end of profiling because some (short-
> lived) tasks can go away.

We do have some setups where we install a BPF program that monitors
process exit and mmap() events and emits (proactively) VMA
information. It's not applicable everywhere, and in some setups (like
Oculus case) we just accept that short-lived processes will be missed
at the expense of less interruption, simpler and less privileged
"agents" doing profiling and address resolution logic.

So the problem space, as can be seen, is pretty vast and varied, and
there is no single API that would serve all the needs perfectly.

>
> Also it should support perf report (offline) on data from a
> different kernel or even a different machine.

We fetch build ID (and resolve file offset) and offload actual
symbolization to a dedicated fleet of servers, whenever possible. We
don't yet do it for kernel stack traces, but we are moving in this
direction (and there are their own problems with /proc/kallsyms being
text-based, listing everything, and pretty big all in itself; but
that's a separate topic).

>
> So it saves the memory map of processes and symbolizes
> the stack trace with it later.  Of course it needs to be updated
> as the memory map changes and that's why it tracks mmap
> or similar syscalls with PERF_RECORD_MMAP[2] records.
>
> A problem with this approach is to get the initial state of all
> (or a target for non-system-wide mode) existing processes.
> We call it synthesizing, and read /proc/PID/maps to generate
> the mmap records.
>
> I think the below comment from Arnaldo talked about how
> we can improve the synthesizing (which is sequential access
> to proc maps) using BPF.

Yep. We can also benchmark using this new ioctl() to fetch a full set
of VMAs, it might still be good enough.

>
> Thanks,
> Namhyung
>

[...]

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Arnaldo Carvalho de Melo]

On Mon, May 06, 2024 at 11:05:17AM -0700, Namhyung Kim wrote:
> On Mon, May 6, 2024 at 6:58 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > > On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > > > hard-coded or user-provided names) is optional just like build ID. If
> > > > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > > > it, saving resources.

> > > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>

> > > > Where is the userspace code that uses this new api you have created?

> > > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > > ioctl() API to solve a common problem (as described above) in patch
> > > #5. The plan is to put it in mentioned blazesym library at the very
> > > least.
> > >
> > > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > > linux-perf-user), as they need to do stack symbolization as well.
 
> I think the general use case in perf is different.  This ioctl API is great
> for live tracing of a single (or a small number of) process(es).  And
> yes, perf tools have those tracing use cases too.  But I think the
> major use case of perf tools is system-wide profiling.
 
> For system-wide profiling, you need to process samples of many
> different processes at a high frequency.  Now perf record doesn't
> process them and just save it for offline processing (well, it does
> at the end to find out build-ID but it can be omitted).

Since:

  Author: Jiri Olsa <jolsa@kernel.org>
  Date:   Mon Dec 14 11:54:49 2020 +0100
  1ca6e80254141d26 ("perf tools: Store build id when available in PERF_RECORD_MMAP2 metadata events")

We don't need to to process the events to find the build ids. I haven't
checked if we still do it to find out which DSOs had hits, but we
shouldn't need to do it for build-ids (unless they were not in memory
when the kernel tried to stash them in the PERF_RECORD_MMAP2, which I
haven't checked but IIRC is a possibility if that ELF part isn't in
memory at the time we want to copy it).

If we're still traversing it like that I guess we can have a knob and
make it the default to not do that and instead create the perf.data
build ID header table with all the build-ids we got from
PERF_RECORD_MMAP2, a (slightly) bigger perf.data file but no event
processing at the end of a 'perf record' session.

> Doing it online is possible (like perf top) but it would add more
> overhead during the profiling.  And we cannot move processing

It comes in the PERF_RECORD_MMAP2, filled by the kernel.

> or symbolization to the end of profiling because some (short-
> lived) tasks can go away.

right
 
> Also it should support perf report (offline) on data from a
> different kernel or even a different machine.

right
 
> So it saves the memory map of processes and symbolizes
> the stack trace with it later.  Of course it needs to be updated
> as the memory map changes and that's why it tracks mmap
> or similar syscalls with PERF_RECORD_MMAP[2] records.
 
> A problem with this approach is to get the initial state of all
> (or a target for non-system-wide mode) existing processes.
> We call it synthesizing, and read /proc/PID/maps to generate
> the mmap records.
 
> I think the below comment from Arnaldo talked about how
> we can improve the synthesizing (which is sequential access
> to proc maps) using BPF.

Yes, I wonder how far Jiri went, Jiri?

- Arnaldo
 
> Thanks,
> Namhyung
> 
> 
> >
> > At some point, when BPF iterators became a thing we thought about, IIRC
> > Jiri did some experimentation, but I lost track, of using BPF to
> > synthesize PERF_RECORD_MMAP2 records for pre-existing maps, the layout
> > as in uapi/linux/perf_event.h:
> >
> >         /*
> >          * The MMAP2 records are an augmented version of MMAP, they add
> >          * maj, min, ino numbers to be used to uniquely identify each mapping
> >          *
> >          * struct {
> >          *      struct perf_event_header        header;
> >          *
> >          *      u32                             pid, tid;
> >          *      u64                             addr;
> >          *      u64                             len;
> >          *      u64                             pgoff;
> >          *      union {
> >          *              struct {
> >          *                      u32             maj;
> >          *                      u32             min;
> >          *                      u64             ino;
> >          *                      u64             ino_generation;
> >          *              };
> >          *              struct {
> >          *                      u8              build_id_size;
> >          *                      u8              __reserved_1;
> >          *                      u16             __reserved_2;
> >          *                      u8              build_id[20];
> >          *              };
> >          *      };
> >          *      u32                             prot, flags;
> >          *      char                            filename[];
> >          *      struct sample_id                sample_id;
> >          * };
> >          */
> >         PERF_RECORD_MMAP2                       = 10,
> >
> >  *   PERF_RECORD_MISC_MMAP_BUILD_ID      - PERF_RECORD_MMAP2 event
> >
> > As perf.data files can be used for many purposes we want them all, so we
> > setup a meta data perf file descriptor to go on receiving the new mmaps
> > while we read /proc/<pid>/maps, to reduce the chance of missing maps, do
> > it in parallel, etc:
> >
> > ⬢[acme@toolbox perf-tools-next]$ perf record -h 'event synthesis'
> >
> >  Usage: perf record [<options>] [<command>]
> >     or: perf record [<options>] -- <command> [<options>]
> >
> >         --num-thread-synthesize <n>
> >                           number of threads to run for event synthesis
> >         --synth <no|all|task|mmap|cgroup>
> >                           Fine-tune event synthesis: default=all
> >
> > ⬢[acme@toolbox perf-tools-next]$
> >
> > For this specific initial synthesis of everything the plan, as mentioned
> > about Jiri's experiments, was to use a BPF iterator to just feed the
> > perf ring buffer with those events, that way userspace would just
> > receive the usual records it gets when a new mmap is put in place, the
> > BPF iterator would just feed the preexisting mmaps, as instructed via
> > the perf_event_attr for the perf_event_open syscall.
> >
> > For people not wanting BPF, i.e. disabling it altogether in perf or
> > disabling just BPF skels, then we would fallback to the current method,
> > or to the one being discussed here when it becomes available.
> >
> > One thing to have in mind is for this iterator not to generate duplicate
> > records for non-pre-existing mmaps, i.e. we would need some generation
> > number that would be bumped when asking for such pre-existing maps
> > PERF_RECORD_MMAP2 dumps.
> >
> > > It will be up to other similar projects to adopt this, but we'll
> > > definitely get this into blazesym as it is actually a problem for the
> >
> > At some point looking at plugging blazesym somehow with perf may be
> > something to consider, indeed.
> >
> > - Arnaldo
> >
> > > abovementioned Oculus use case. We already had to make a tradeoff (see
> > > [2], this wasn't done just because we could, but it was requested by
> > > Oculus customers) to cache the contents of /proc/<pid>/maps and run
> > > the risk of missing some shared libraries that can be loaded later. It
> > > would be great to not have to do this tradeoff, which this new API
> > > would enable.
> > >
> > >   [2] https://github.com/libbpf/blazesym/commit/6b521314126b3ae6f2add43e93234b59fed48ccf
> > >
> > > >
> > > > > ---
> > > > >  fs/proc/task_mmu.c      | 165 ++++++++++++++++++++++++++++++++++++++++
> > > > >  include/uapi/linux/fs.h |  32 ++++++++
> > > > >  2 files changed, 197 insertions(+)
> > > > >
> > > > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > > > index 8e503a1635b7..cb7b1ff1a144 100644
> > > > > --- a/fs/proc/task_mmu.c
> > > > > +++ b/fs/proc/task_mmu.c
> > > > > @@ -22,6 +22,7 @@
> > > > >  #include <linux/pkeys.h>
> > > > >  #include <linux/minmax.h>
> > > > >  #include <linux/overflow.h>
> > > > > +#include <linux/buildid.h>
> > > > >
> > > > >  #include <asm/elf.h>
> > > > >  #include <asm/tlb.h>
> > > > > @@ -375,11 +376,175 @@ static int pid_maps_open(struct inode *inode, struct file *file)
> > > > >       return do_maps_open(inode, file, &proc_pid_maps_op);
> > > > >  }
> > > > >
> > > > > +static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg)
> > > > > +{
> > > > > +     struct procfs_procmap_query karg;
> > > > > +     struct vma_iterator iter;
> > > > > +     struct vm_area_struct *vma;
> > > > > +     struct mm_struct *mm;
> > > > > +     const char *name = NULL;
> > > > > +     char build_id_buf[BUILD_ID_SIZE_MAX], *name_buf = NULL;
> > > > > +     __u64 usize;
> > > > > +     int err;
> > > > > +
> > > > > +     if (copy_from_user(&usize, (void __user *)uarg, sizeof(usize)))
> > > > > +             return -EFAULT;
> > > > > +     if (usize > PAGE_SIZE)
> > > >
> > > > Nice, where did you document that?  And how is that portable given that
> > > > PAGE_SIZE can be different on different systems?
> > >
> > > I'm happy to document everything, can you please help by pointing
> > > where this documentation has to live?
> > >
> > > This is mostly fool-proofing, though, because the user has to pass
> > > sizeof(struct procfs_procmap_query), which I don't see ever getting
> > > close to even 4KB (not even saying about 64KB). This is just to
> > > prevent copy_struct_from_user() below to do too much zero-checking.
> > >
> > > >
> > > > and why aren't you checking the actual structure size instead?  You can
> > > > easily run off the end here without knowing it.
> > >
> > > See copy_struct_from_user(), it does more checks. This is a helper
> > > designed specifically to deal with use cases like this where kernel
> > > struct size can change and user space might be newer or older.
> > > copy_struct_from_user() has a nice documentation describing all these
> > > nuances.
> > >
> > > >
> > > > > +             return -E2BIG;
> > > > > +     if (usize < offsetofend(struct procfs_procmap_query, query_addr))
> > > > > +             return -EINVAL;
> > > >
> > > > Ok, so you have two checks?  How can the first one ever fail?
> > >
> > > Hmm.. If usize = 8, copy_from_user() won't fail, usize > PAGE_SIZE
> > > won't fail, but this one will fail.
> > >
> > > The point of this check is that user has to specify at least first
> > > three fields of procfs_procmap_query (size, query_flags, and
> > > query_addr), because without those the query is meaningless.
> > > >
> > > >
> > > > > +     err = copy_struct_from_user(&karg, sizeof(karg), uarg, usize);
> > >
> > > and this helper does more checks validating that the user either has a
> > > shorter struct (and then zero-fills the rest of kernel-side struct) or
> > > has longer (and then the longer part has to be zero filled). Do check
> > > copy_struct_from_user() documentation, it's great.
> > >
> > > > > +     if (err)
> > > > > +             return err;
> > > > > +
> > > > > +     if (karg.query_flags & ~PROCFS_PROCMAP_EXACT_OR_NEXT_VMA)
> > > > > +             return -EINVAL;
> > > > > +     if (!!karg.vma_name_size != !!karg.vma_name_addr)
> > > > > +             return -EINVAL;
> > > > > +     if (!!karg.build_id_size != !!karg.build_id_addr)
> > > > > +             return -EINVAL;
> > > >
> > > > So you want values to be set, right?
> > >
> > > Either both should be set, or neither. It's ok for both size/addr
> > > fields to be zero, in which case it indicates that the user doesn't
> > > want this part of information (which is usually a bit more expensive
> > > to get and might not be necessary for all the cases).
> > >
> > > >
> > > > > +
> > > > > +     mm = priv->mm;
> > > > > +     if (!mm || !mmget_not_zero(mm))
> > > > > +             return -ESRCH;
> > > >
> > > > What is this error for?  Where is this documentned?
> > >
> > > I copied it from existing /proc/<pid>/maps checks. I presume it's
> > > guarding the case when mm might be already put. So if the process is
> > > gone, but we have /proc/<pid>/maps file open?
> > >
> > > >
> > > > > +     if (mmap_read_lock_killable(mm)) {
> > > > > +             mmput(mm);
> > > > > +             return -EINTR;
> > > > > +     }
> > > > > +
> > > > > +     vma_iter_init(&iter, mm, karg.query_addr);
> > > > > +     vma = vma_next(&iter);
> > > > > +     if (!vma) {
> > > > > +             err = -ENOENT;
> > > > > +             goto out;
> > > > > +     }
> > > > > +     /* user wants covering VMA, not the closest next one */
> > > > > +     if (!(karg.query_flags & PROCFS_PROCMAP_EXACT_OR_NEXT_VMA) &&
> > > > > +         vma->vm_start > karg.query_addr) {
> > > > > +             err = -ENOENT;
> > > > > +             goto out;
> > > > > +     }
> > > > > +
> > > > > +     karg.vma_start = vma->vm_start;
> > > > > +     karg.vma_end = vma->vm_end;
> > > > > +
> > > > > +     if (vma->vm_file) {
> > > > > +             const struct inode *inode = file_user_inode(vma->vm_file);
> > > > > +
> > > > > +             karg.vma_offset = ((__u64)vma->vm_pgoff) << PAGE_SHIFT;
> > > > > +             karg.dev_major = MAJOR(inode->i_sb->s_dev);
> > > > > +             karg.dev_minor = MINOR(inode->i_sb->s_dev);
> > > >
> > > > So the major/minor is that of the file superblock?  Why?
> > >
> > > Because inode number is unique only within given super block (and even
> > > then it's more complicated, e.g., btrfs subvolumes add more headaches,
> > > I believe). inode + dev maj/min is sometimes used for cache/reuse of
> > > per-binary information (e.g., pre-processed DWARF information, which
> > > is *very* expensive, so anything that allows to avoid doing this is
> > > helpful).
> > >
> > > >
> > > > > +             karg.inode = inode->i_ino;
> > > >
> > > > What is userspace going to do with this?
> > > >
> > >
> > > See above.
> > >
> > > > > +     } else {
> > > > > +             karg.vma_offset = 0;
> > > > > +             karg.dev_major = 0;
> > > > > +             karg.dev_minor = 0;
> > > > > +             karg.inode = 0;
> > > >
> > > > Why not set everything to 0 up above at the beginning so you never miss
> > > > anything, and you don't miss any holes accidentally in the future.
> > > >
> > >
> > > Stylistic preference, I find this more explicit, but I don't care much
> > > one way or another.
> > >
> > > > > +     }
> > > > > +
> > > > > +     karg.vma_flags = 0;
> > > > > +     if (vma->vm_flags & VM_READ)
> > > > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_READABLE;
> > > > > +     if (vma->vm_flags & VM_WRITE)
> > > > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_WRITABLE;
> > > > > +     if (vma->vm_flags & VM_EXEC)
> > > > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_EXECUTABLE;
> > > > > +     if (vma->vm_flags & VM_MAYSHARE)
> > > > > +             karg.vma_flags |= PROCFS_PROCMAP_VMA_SHARED;
> > > > > +
> > >
> > > [...]
> > >
> > > > > diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> > > > > index 45e4e64fd664..fe8924a8d916 100644
> > > > > --- a/include/uapi/linux/fs.h
> > > > > +++ b/include/uapi/linux/fs.h
> > > > > @@ -393,4 +393,36 @@ struct pm_scan_arg {
> > > > >       __u64 return_mask;
> > > > >  };
> > > > >
> > > > > +/* /proc/<pid>/maps ioctl */
> > > > > +#define PROCFS_IOCTL_MAGIC 0x9f
> > > >
> > > > Don't you need to document this in the proper place?
> > >
> > > I probably do, but I'm asking for help in knowing where. procfs is not
> > > a typical area of kernel I'm working with, so any pointers are highly
> > > appreciated.
> > >
> > > >
> > > > > +#define PROCFS_PROCMAP_QUERY _IOWR(PROCFS_IOCTL_MAGIC, 1, struct procfs_procmap_query)
> > > > > +
> > > > > +enum procmap_query_flags {
> > > > > +     PROCFS_PROCMAP_EXACT_OR_NEXT_VMA = 0x01,
> > > > > +};
> > > > > +
> > > > > +enum procmap_vma_flags {
> > > > > +     PROCFS_PROCMAP_VMA_READABLE = 0x01,
> > > > > +     PROCFS_PROCMAP_VMA_WRITABLE = 0x02,
> > > > > +     PROCFS_PROCMAP_VMA_EXECUTABLE = 0x04,
> > > > > +     PROCFS_PROCMAP_VMA_SHARED = 0x08,
> > > >
> > > > Are these bits?  If so, please use the bit macro for it to make it
> > > > obvious.
> > > >
> > >
> > > Yes, they are. When I tried BIT(1), it didn't compile. I chose not to
> > > add any extra #includes to this UAPI header, but I can figure out the
> > > necessary dependency and do BIT(), I just didn't feel like BIT() adds
> > > much here, tbh.
> > >
> > > > > +};
> > > > > +
> > > > > +struct procfs_procmap_query {
> > > > > +     __u64 size;
> > > > > +     __u64 query_flags;              /* in */
> > > >
> > > > Does this map to the procmap_vma_flags enum?  if so, please say so.
> > >
> > > no, procmap_query_flags, and yes, I will
> > >
> > > >
> > > > > +     __u64 query_addr;               /* in */
> > > > > +     __u64 vma_start;                /* out */
> > > > > +     __u64 vma_end;                  /* out */
> > > > > +     __u64 vma_flags;                /* out */
> > > > > +     __u64 vma_offset;               /* out */
> > > > > +     __u64 inode;                    /* out */
> > > >
> > > > What is the inode for, you have an inode for the file already, why give
> > > > it another one?
> > >
> > > This is inode of vma's backing file, same as /proc/<pid>/maps' file
> > > column. What inode of file do I already have here? You mean of
> > > /proc/<pid>/maps itself? It's useless for the intended purposes.
> > >
> > > >
> > > > > +     __u32 dev_major;                /* out */
> > > > > +     __u32 dev_minor;                /* out */
> > > >
> > > > What is major/minor for?
> > >
> > > This is the same information as emitted by /proc/<pid>/maps,
> > > identifies superblock of vma's backing file. As I mentioned above, it
> > > can be used for caching per-file (i.e., per-ELF binary) information
> > > (for example).
> > >
> > > >
> > > > > +     __u32 vma_name_size;            /* in/out */
> > > > > +     __u32 build_id_size;            /* in/out */
> > > > > +     __u64 vma_name_addr;            /* in */
> > > > > +     __u64 build_id_addr;            /* in */
> > > >
> > > > Why not document this all using kerneldoc above the structure?
> > >
> > > Yes, sorry, I slacked a bit on adding this upfront. I knew we'll be
> > > figuring out the best place and approach, and so wanted to avoid
> > > documentation churn.
> > >
> > > Would something like what we have for pm_scan_arg and pagemap APIs
> > > work? I see it added a few simple descriptions for pm_scan_arg struct,
> > > and there is Documentation/admin-guide/mm/pagemap.rst. Should I add
> > > Documentation/admin-guide/mm/procmap.rst (admin-guide part feels off,
> > > though)? Anyways, I'm hoping for pointers where all this should be
> > > documented. Thank you!
> > >
> > > >
> > > > anyway, I don't like ioctls, but there is a place for them, you just
> > > > have to actually justify the use for them and not say "not efficient
> > > > enough" as that normally isn't an issue overall.
> > >
> > > I've written a demo tool in patch #5 which performs real-world task:
> > > mapping addresses to their VMAs (specifically calculating file offset,
> > > finding vma_start + vma_end range to further access files from
> > > /proc/<pid>/map_files/<start>-<end>). I did the implementation
> > > faithfully, doing it in the most optimal way for both APIs. I showed
> > > that for "typical" (it's hard to specify what typical is, of course,
> > > too many variables) scenario (it was data collected on a real server
> > > running real service, 30 seconds of process-specific stack traces were
> > > captured, if I remember correctly). I showed that doing exactly the
> > > same amount of work is ~35x times slower with /proc/<pid>/maps.
> > >
> > > Take another process, another set of addresses, another anything, and
> > > the numbers will be different, but I think it gives the right idea.
> > >
> > > But I think we are overpivoting on text vs binary distinction here.
> > > It's the more targeted querying of VMAs that's beneficial here. This
> > > allows applications to not cache anything and just re-query when doing
> > > periodic or continuous profiling (where addresses are coming in not as
> > > one batch, as a sequence of batches extended in time).
> > >
> > > /proc/<pid>/maps, for all its usefulness, just can't provide this sort
> > > of ability, as it wasn't designed to do that and is targeting
> > > different use cases.
> > >
> > > And then, a new ability to request reliable (it's not 100% reliable
> > > today, I'm going to address that as a follow up) build ID is *crucial*
> > > for some scenarios. The mentioned Oculus use case, the need to fully
> > > access underlying ELF binary just to get build ID is frowned upon. And
> > > for a good reason. Profiler only needs build ID, which is no secret
> > > and not sensitive information. This new (and binary, yes) API allows
> > > to add this into an API without breaking any backwards compatibility.
> > >
> > > >
> > > > thanks,
> > > >
> > > > greg k-h
> >

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Arnaldo Carvalho de Melo]

On Mon, May 06, 2024 at 03:53:40PM -0300, Arnaldo Carvalho de Melo wrote:
> On Mon, May 06, 2024 at 11:05:17AM -0700, Namhyung Kim wrote:
> > On Mon, May 6, 2024 at 6:58 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > > On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > > > On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > > > > hard-coded or user-provided names) is optional just like build ID. If
> > > > > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > > > > it, saving resources.
> 
> > > > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> 
> > > > > Where is the userspace code that uses this new api you have created?
> 
> > > > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > > > ioctl() API to solve a common problem (as described above) in patch
> > > > #5. The plan is to put it in mentioned blazesym library at the very
> > > > least.
> > > >
> > > > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > > > linux-perf-user), as they need to do stack symbolization as well.
>  
> > I think the general use case in perf is different.  This ioctl API is great
> > for live tracing of a single (or a small number of) process(es).  And
> > yes, perf tools have those tracing use cases too.  But I think the
> > major use case of perf tools is system-wide profiling.
>  
> > For system-wide profiling, you need to process samples of many
> > different processes at a high frequency.  Now perf record doesn't
> > process them and just save it for offline processing (well, it does
> > at the end to find out build-ID but it can be omitted).
> 
> Since:
> 
>   Author: Jiri Olsa <jolsa@kernel.org>
>   Date:   Mon Dec 14 11:54:49 2020 +0100
>   1ca6e80254141d26 ("perf tools: Store build id when available in PERF_RECORD_MMAP2 metadata events")
> 
> We don't need to to process the events to find the build ids. I haven't
> checked if we still do it to find out which DSOs had hits, but we
> shouldn't need to do it for build-ids (unless they were not in memory
> when the kernel tried to stash them in the PERF_RECORD_MMAP2, which I
> haven't checked but IIRC is a possibility if that ELF part isn't in
> memory at the time we want to copy it).

> If we're still traversing it like that I guess we can have a knob and
> make it the default to not do that and instead create the perf.data
> build ID header table with all the build-ids we got from
> PERF_RECORD_MMAP2, a (slightly) bigger perf.data file but no event
> processing at the end of a 'perf record' session.

But then we don't process the PERF_RECORD_MMAP2 in 'perf record', it
just goes on directly to the perf.data file :-\

Humm, perhaps the sideband thread...

- Arnaldo

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Namhyung Kim]

On Mon, May 6, 2024 at 12:16 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> On Mon, May 06, 2024 at 03:53:40PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Mon, May 06, 2024 at 11:05:17AM -0700, Namhyung Kim wrote:
> > > On Mon, May 6, 2024 at 6:58 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > > > On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > > > > On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > > > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > > > > > hard-coded or user-provided names) is optional just like build ID. If
> > > > > > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > > > > > it, saving resources.
> >
> > > > > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> >
> > > > > > Where is the userspace code that uses this new api you have created?
> >
> > > > > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > > > > ioctl() API to solve a common problem (as described above) in patch
> > > > > #5. The plan is to put it in mentioned blazesym library at the very
> > > > > least.
> > > > >
> > > > > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > > > > linux-perf-user), as they need to do stack symbolization as well.
> >
> > > I think the general use case in perf is different.  This ioctl API is great
> > > for live tracing of a single (or a small number of) process(es).  And
> > > yes, perf tools have those tracing use cases too.  But I think the
> > > major use case of perf tools is system-wide profiling.
> >
> > > For system-wide profiling, you need to process samples of many
> > > different processes at a high frequency.  Now perf record doesn't
> > > process them and just save it for offline processing (well, it does
> > > at the end to find out build-ID but it can be omitted).
> >
> > Since:
> >
> >   Author: Jiri Olsa <jolsa@kernel.org>
> >   Date:   Mon Dec 14 11:54:49 2020 +0100
> >   1ca6e80254141d26 ("perf tools: Store build id when available in PERF_RECORD_MMAP2 metadata events")
> >
> > We don't need to to process the events to find the build ids. I haven't
> > checked if we still do it to find out which DSOs had hits, but we
> > shouldn't need to do it for build-ids (unless they were not in memory
> > when the kernel tried to stash them in the PERF_RECORD_MMAP2, which I
> > haven't checked but IIRC is a possibility if that ELF part isn't in
> > memory at the time we want to copy it).
>
> > If we're still traversing it like that I guess we can have a knob and
> > make it the default to not do that and instead create the perf.data
> > build ID header table with all the build-ids we got from
> > PERF_RECORD_MMAP2, a (slightly) bigger perf.data file but no event
> > processing at the end of a 'perf record' session.
>
> But then we don't process the PERF_RECORD_MMAP2 in 'perf record', it
> just goes on directly to the perf.data file :-\

Yep, we don't process build-IDs at the end if --buildid-mmap
option is given.  It won't have build-ID header table but it's
not needed anymore and perf report can know build-ID from
MMAP2 directly.

Thanks,
Namhyung

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Andrii Nakryiko]

On Mon, May 6, 2024 at 6:58 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>
> On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > > hard-coded or user-provided names) is optional just like build ID. If
> > > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > > it, saving resources.
>
> > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
>
> > > Where is the userspace code that uses this new api you have created?
>
> > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > ioctl() API to solve a common problem (as described above) in patch
> > #5. The plan is to put it in mentioned blazesym library at the very
> > least.
> >
> > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > linux-perf-user), as they need to do stack symbolization as well.
>
> At some point, when BPF iterators became a thing we thought about, IIRC
> Jiri did some experimentation, but I lost track, of using BPF to
> synthesize PERF_RECORD_MMAP2 records for pre-existing maps, the layout
> as in uapi/linux/perf_event.h:
>
>         /*
>          * The MMAP2 records are an augmented version of MMAP, they add
>          * maj, min, ino numbers to be used to uniquely identify each mapping
>          *
>          * struct {
>          *      struct perf_event_header        header;
>          *
>          *      u32                             pid, tid;
>          *      u64                             addr;
>          *      u64                             len;
>          *      u64                             pgoff;
>          *      union {
>          *              struct {
>          *                      u32             maj;
>          *                      u32             min;
>          *                      u64             ino;
>          *                      u64             ino_generation;
>          *              };
>          *              struct {
>          *                      u8              build_id_size;
>          *                      u8              __reserved_1;
>          *                      u16             __reserved_2;
>          *                      u8              build_id[20];
>          *              };
>          *      };
>          *      u32                             prot, flags;
>          *      char                            filename[];
>          *      struct sample_id                sample_id;
>          * };
>          */
>         PERF_RECORD_MMAP2                       = 10,
>
>  *   PERF_RECORD_MISC_MMAP_BUILD_ID      - PERF_RECORD_MMAP2 event
>
> As perf.data files can be used for many purposes we want them all, so we

ok, so because you want them all and you don't know which VMAs will be
useful or not, it's a different problem. BPF iterators will be faster
purely due to avoiding binary -> text -> binary conversion path, but
other than that you'll still retrieve all VMAs.

You can still do the same full VMA iteration with this new API, of
course, but advantages are probably smaller as you'll be retrieving a
full set of VMAs regardless (though it would be interesting to compare
anyways).

> setup a meta data perf file descriptor to go on receiving the new mmaps
> while we read /proc/<pid>/maps, to reduce the chance of missing maps, do
> it in parallel, etc:
>
> ⬢[acme@toolbox perf-tools-next]$ perf record -h 'event synthesis'
>
>  Usage: perf record [<options>] [<command>]
>     or: perf record [<options>] -- <command> [<options>]
>
>         --num-thread-synthesize <n>
>                           number of threads to run for event synthesis
>         --synth <no|all|task|mmap|cgroup>
>                           Fine-tune event synthesis: default=all
>
> ⬢[acme@toolbox perf-tools-next]$
>
> For this specific initial synthesis of everything the plan, as mentioned
> about Jiri's experiments, was to use a BPF iterator to just feed the
> perf ring buffer with those events, that way userspace would just
> receive the usual records it gets when a new mmap is put in place, the
> BPF iterator would just feed the preexisting mmaps, as instructed via
> the perf_event_attr for the perf_event_open syscall.
>
> For people not wanting BPF, i.e. disabling it altogether in perf or
> disabling just BPF skels, then we would fallback to the current method,
> or to the one being discussed here when it becomes available.
>
> One thing to have in mind is for this iterator not to generate duplicate
> records for non-pre-existing mmaps, i.e. we would need some generation
> number that would be bumped when asking for such pre-existing maps
> PERF_RECORD_MMAP2 dumps.

Looking briefly at struct vm_area_struct, it doesn't seems like the
kernel maintains any sort of generation (at least not at
vm_area_struct level), so this would be nice to have, I'm sure, but
isn't really related to adding this API. Once the kernel does have
this "VMA generation" counter, it can be trivially added to this
binary interface (which can't be said about /proc/<pid>/maps,
unfortunately).

>
> > It will be up to other similar projects to adopt this, but we'll
> > definitely get this into blazesym as it is actually a problem for the
>
> At some point looking at plugging blazesym somehow with perf may be
> something to consider, indeed.

In the above I meant direct use of this new API in perf code itself,
but yes, blazesym is a generic library for symbolization that handles
ELF/DWARF/GSYM (and I believe more formats), so it indeed might make
sense to use it.

>
> - Arnaldo
>
> > abovementioned Oculus use case. We already had to make a tradeoff (see
> > [2], this wasn't done just because we could, but it was requested by
> > Oculus customers) to cache the contents of /proc/<pid>/maps and run
> > the risk of missing some shared libraries that can be loaded later. It
> > would be great to not have to do this tradeoff, which this new API
> > would enable.
> >
> >   [2] https://github.com/libbpf/blazesym/commit/6b521314126b3ae6f2add43e93234b59fed48ccf
> >

[...]

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Arnaldo Carvalho de Melo]

On Mon, May 06, 2024 at 11:41:43AM -0700, Andrii Nakryiko wrote:
> On Mon, May 6, 2024 at 6:58 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >
> > On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > > On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > > > hard-coded or user-provided names) is optional just like build ID. If
> > > > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > > > it, saving resources.
> >
> > > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> >
> > > > Where is the userspace code that uses this new api you have created?
> >
> > > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > > ioctl() API to solve a common problem (as described above) in patch
> > > #5. The plan is to put it in mentioned blazesym library at the very
> > > least.
> > >
> > > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > > linux-perf-user), as they need to do stack symbolization as well.
> >
> > At some point, when BPF iterators became a thing we thought about, IIRC
> > Jiri did some experimentation, but I lost track, of using BPF to
> > synthesize PERF_RECORD_MMAP2 records for pre-existing maps, the layout
> > as in uapi/linux/perf_event.h:
> >
> >         /*
> >          * The MMAP2 records are an augmented version of MMAP, they add
> >          * maj, min, ino numbers to be used to uniquely identify each mapping
> >          *
> >          * struct {
> >          *      struct perf_event_header        header;
> >          *
> >          *      u32                             pid, tid;
> >          *      u64                             addr;
> >          *      u64                             len;
> >          *      u64                             pgoff;
> >          *      union {
> >          *              struct {
> >          *                      u32             maj;
> >          *                      u32             min;
> >          *                      u64             ino;
> >          *                      u64             ino_generation;
> >          *              };
> >          *              struct {
> >          *                      u8              build_id_size;
> >          *                      u8              __reserved_1;
> >          *                      u16             __reserved_2;
> >          *                      u8              build_id[20];
> >          *              };
> >          *      };
> >          *      u32                             prot, flags;
> >          *      char                            filename[];
> >          *      struct sample_id                sample_id;
> >          * };
> >          */
> >         PERF_RECORD_MMAP2                       = 10,
> >
> >  *   PERF_RECORD_MISC_MMAP_BUILD_ID      - PERF_RECORD_MMAP2 event
> >
> > As perf.data files can be used for many purposes we want them all, so we
> 
> ok, so because you want them all and you don't know which VMAs will be
> useful or not, it's a different problem. BPF iterators will be faster
> purely due to avoiding binary -> text -> binary conversion path, but
> other than that you'll still retrieve all VMAs.

But not using tons of syscalls to parse text data from /proc.
 
> You can still do the same full VMA iteration with this new API, of
> course, but advantages are probably smaller as you'll be retrieving a
> full set of VMAs regardless (though it would be interesting to compare
> anyways).

sure, I can't see how it would be faster, but yeah, interesting to see
what is the difference.
 
> > setup a meta data perf file descriptor to go on receiving the new mmaps
> > while we read /proc/<pid>/maps, to reduce the chance of missing maps, do
> > it in parallel, etc:
> >
> > ⬢[acme@toolbox perf-tools-next]$ perf record -h 'event synthesis'
> >
> >  Usage: perf record [<options>] [<command>]
> >     or: perf record [<options>] -- <command> [<options>]
> >
> >         --num-thread-synthesize <n>
> >                           number of threads to run for event synthesis
> >         --synth <no|all|task|mmap|cgroup>
> >                           Fine-tune event synthesis: default=all
> >
> > ⬢[acme@toolbox perf-tools-next]$
> >
> > For this specific initial synthesis of everything the plan, as mentioned
> > about Jiri's experiments, was to use a BPF iterator to just feed the
> > perf ring buffer with those events, that way userspace would just
> > receive the usual records it gets when a new mmap is put in place, the
> > BPF iterator would just feed the preexisting mmaps, as instructed via
> > the perf_event_attr for the perf_event_open syscall.
> >
> > For people not wanting BPF, i.e. disabling it altogether in perf or
> > disabling just BPF skels, then we would fallback to the current method,
> > or to the one being discussed here when it becomes available.
> >
> > One thing to have in mind is for this iterator not to generate duplicate
> > records for non-pre-existing mmaps, i.e. we would need some generation
> > number that would be bumped when asking for such pre-existing maps
> > PERF_RECORD_MMAP2 dumps.
> 
> Looking briefly at struct vm_area_struct, it doesn't seems like the
> kernel maintains any sort of generation (at least not at
> vm_area_struct level), so this would be nice to have, I'm sure, but

Yeah, this would be something specific to the "retrieve me the list of
VMAs" bulky thing, i.e. the kernel perf code (or the BPF that would
generate the PERF_RECORD_MMAP2 records by using a BPF vma iterator)
would bump the generation number and store it to the VMA in
perf_event_mmap() so that the iterator doesn't consider it, as it is a
new mmap that is being just sent to whoever is listening, and the perf
tool that put in place the BPF program to iterate is listening.

> isn't really related to adding this API. Once the kernel does have

Well, perf wants to enumerate pre-existing mmaps _and_ after that
finishes to know about new mmaps, so we need to know a way to avoid
having the BPF program enumerating pre-existing maps sending
PERF_RECORD_MMAP2 for maps perf already knows about via a regular
PERF_RECORD_MMAP2 sent when a new mmap is put in place.

So there is an overlap where perf (or any other tool wanting to
enumerate all pre-existing maps and new ones) can receive info for the
same map from the enumerator and from the existing mechanism generating
PERF_RECORD_MMAP2 records.

- Arnaldo

> this "VMA generation" counter, it can be trivially added to this
> binary interface (which can't be said about /proc/<pid>/maps,
> unfortunately).
> 
> >
> > > It will be up to other similar projects to adopt this, but we'll
> > > definitely get this into blazesym as it is actually a problem for the
> >
> > At some point looking at plugging blazesym somehow with perf may be
> > something to consider, indeed.
> 
> In the above I meant direct use of this new API in perf code itself,
> but yes, blazesym is a generic library for symbolization that handles
> ELF/DWARF/GSYM (and I believe more formats), so it indeed might make
> sense to use it.
> 
> >
> > - Arnaldo
> >
> > > abovementioned Oculus use case. We already had to make a tradeoff (see
> > > [2], this wasn't done just because we could, but it was requested by
> > > Oculus customers) to cache the contents of /proc/<pid>/maps and run
> > > the risk of missing some shared libraries that can be loaded later. It
> > > would be great to not have to do this tradeoff, which this new API
> > > would enable.
> > >
> > >   [2] https://github.com/libbpf/blazesym/commit/6b521314126b3ae6f2add43e93234b59fed48ccf
> > >
> 
> [...]

Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps

[Andrii Nakryiko]

On Mon, May 6, 2024 at 1:35 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>
> On Mon, May 06, 2024 at 11:41:43AM -0700, Andrii Nakryiko wrote:
> > On Mon, May 6, 2024 at 6:58 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > >
> > > On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > > > On Sat, May 4, 2024 at 8:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > > > Note also, that fetching VMA name (e.g., backing file path, or special
> > > > > > hard-coded or user-provided names) is optional just like build ID. If
> > > > > > user sets vma_name_size to zero, kernel code won't attempt to retrieve
> > > > > > it, saving resources.
> > >
> > > > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> > >
> > > > > Where is the userspace code that uses this new api you have created?
> > >
> > > > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > > > ioctl() API to solve a common problem (as described above) in patch
> > > > #5. The plan is to put it in mentioned blazesym library at the very
> > > > least.
> > > >
> > > > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > > > linux-perf-user), as they need to do stack symbolization as well.
> > >
> > > At some point, when BPF iterators became a thing we thought about, IIRC
> > > Jiri did some experimentation, but I lost track, of using BPF to
> > > synthesize PERF_RECORD_MMAP2 records for pre-existing maps, the layout
> > > as in uapi/linux/perf_event.h:
> > >
> > >         /*
> > >          * The MMAP2 records are an augmented version of MMAP, they add
> > >          * maj, min, ino numbers to be used to uniquely identify each mapping
> > >          *
> > >          * struct {
> > >          *      struct perf_event_header        header;
> > >          *
> > >          *      u32                             pid, tid;
> > >          *      u64                             addr;
> > >          *      u64                             len;
> > >          *      u64                             pgoff;
> > >          *      union {
> > >          *              struct {
> > >          *                      u32             maj;
> > >          *                      u32             min;
> > >          *                      u64             ino;
> > >          *                      u64             ino_generation;
> > >          *              };
> > >          *              struct {
> > >          *                      u8              build_id_size;
> > >          *                      u8              __reserved_1;
> > >          *                      u16             __reserved_2;
> > >          *                      u8              build_id[20];
> > >          *              };
> > >          *      };
> > >          *      u32                             prot, flags;
> > >          *      char                            filename[];
> > >          *      struct sample_id                sample_id;
> > >          * };
> > >          */
> > >         PERF_RECORD_MMAP2                       = 10,
> > >
> > >  *   PERF_RECORD_MISC_MMAP_BUILD_ID      - PERF_RECORD_MMAP2 event
> > >
> > > As perf.data files can be used for many purposes we want them all, so we
> >
> > ok, so because you want them all and you don't know which VMAs will be
> > useful or not, it's a different problem. BPF iterators will be faster
> > purely due to avoiding binary -> text -> binary conversion path, but
> > other than that you'll still retrieve all VMAs.
>
> But not using tons of syscalls to parse text data from /proc.

In terms of syscall *count* you win with 4KB text reads, there are
fewer syscalls because of this 4KB-based batching. But the cost of
syscall + amount of user-space processing is a different matter. My
benchmark in perf (see patch #5 discussion) suggests that even with
more ioctl() syscalls, perf would win here.

But I also realized that what you really need (I think, correct me if
I'm wrong) is only file-backed VMAs, because all the other ones are
not that useful for symbolization. So I'm adding a minimal change to
my code to allow the user to specify another query flag to only return
file-backed VMAs. I'm going to try it with perf code and see how that
helps. I'll post results in patch #5 thread, once I have them.

>
> > You can still do the same full VMA iteration with this new API, of
> > course, but advantages are probably smaller as you'll be retrieving a
> > full set of VMAs regardless (though it would be interesting to compare
> > anyways).
>
> sure, I can't see how it would be faster, but yeah, interesting to see
> what is the difference.

see patch #5 thread, seems like it's still a bit faster

>
> > > setup a meta data perf file descriptor to go on receiving the new mmaps
> > > while we read /proc/<pid>/maps, to reduce the chance of missing maps, do
> > > it in parallel, etc:
> > >
> > > ⬢[acme@toolbox perf-tools-next]$ perf record -h 'event synthesis'
> > >
> > >  Usage: perf record [<options>] [<command>]
> > >     or: perf record [<options>] -- <command> [<options>]
> > >
> > >         --num-thread-synthesize <n>
> > >                           number of threads to run for event synthesis
> > >         --synth <no|all|task|mmap|cgroup>
> > >                           Fine-tune event synthesis: default=all
> > >
> > > ⬢[acme@toolbox perf-tools-next]$
> > >
> > > For this specific initial synthesis of everything the plan, as mentioned
> > > about Jiri's experiments, was to use a BPF iterator to just feed the
> > > perf ring buffer with those events, that way userspace would just
> > > receive the usual records it gets when a new mmap is put in place, the
> > > BPF iterator would just feed the preexisting mmaps, as instructed via
> > > the perf_event_attr for the perf_event_open syscall.
> > >
> > > For people not wanting BPF, i.e. disabling it altogether in perf or
> > > disabling just BPF skels, then we would fallback to the current method,
> > > or to the one being discussed here when it becomes available.
> > >
> > > One thing to have in mind is for this iterator not to generate duplicate
> > > records for non-pre-existing mmaps, i.e. we would need some generation
> > > number that would be bumped when asking for such pre-existing maps
> > > PERF_RECORD_MMAP2 dumps.
> >
> > Looking briefly at struct vm_area_struct, it doesn't seems like the
> > kernel maintains any sort of generation (at least not at
> > vm_area_struct level), so this would be nice to have, I'm sure, but
>
> Yeah, this would be something specific to the "retrieve me the list of
> VMAs" bulky thing, i.e. the kernel perf code (or the BPF that would
> generate the PERF_RECORD_MMAP2 records by using a BPF vma iterator)
> would bump the generation number and store it to the VMA in
> perf_event_mmap() so that the iterator doesn't consider it, as it is a
> new mmap that is being just sent to whoever is listening, and the perf
> tool that put in place the BPF program to iterate is listening.

Ok, we went on *so many* tangents in emails on this patch set :) Seems
like there are a bunch of perf-specific improvements possible which
are completely irrelevant to the API I'm proposing. Let's please keep
them separate (and you, perf folks, should propose them upstream),
it's getting hard to see what this patch set is actually about with
all the tangential emails.

>
> > isn't really related to adding this API. Once the kernel does have
>
> Well, perf wants to enumerate pre-existing mmaps _and_ after that
> finishes to know about new mmaps, so we need to know a way to avoid
> having the BPF program enumerating pre-existing maps sending
> PERF_RECORD_MMAP2 for maps perf already knows about via a regular
> PERF_RECORD_MMAP2 sent when a new mmap is put in place.
>
> So there is an overlap where perf (or any other tool wanting to
> enumerate all pre-existing maps and new ones) can receive info for the
> same map from the enumerator and from the existing mechanism generating
> PERF_RECORD_MMAP2 records.
>
> - Arnaldo
>
> > this "VMA generation" counter, it can be trivially added to this
> > binary interface (which can't be said about /proc/<pid>/maps,
> > unfortunately).
> >
> > >
> > > > It will be up to other similar projects to adopt this, but we'll
> > > > definitely get this into blazesym as it is actually a problem for the
> > >
> > > At some point looking at plugging blazesym somehow with perf may be
> > > something to consider, indeed.
> >
> > In the above I meant direct use of this new API in perf code itself,
> > but yes, blazesym is a generic library for symbolization that handles
> > ELF/DWARF/GSYM (and I believe more formats), so it indeed might make
> > sense to use it.
> >
> > >
> > > - Arnaldo
> > >
> > > > abovementioned Oculus use case. We already had to make a tradeoff (see
> > > > [2], this wasn't done just because we could, but it was requested by
> > > > Oculus customers) to cache the contents of /proc/<pid>/maps and run
> > > > the risk of missing some shared libraries that can be loaded later. It
> > > > would be great to not have to do this tradeoff, which this new API
> > > > would enable.
> > > >
> > > >   [2] https://github.com/libbpf/blazesym/commit/6b521314126b3ae6f2add43e93234b59fed48ccf
> > > >
> >
> > [...]

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Andrii Nakryiko]

On Sat, May 4, 2024 at 8:29 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > Implement a simple tool/benchmark for comparing address "resolution"
> > logic based on textual /proc/<pid>/maps interface and new binary
> > ioctl-based PROCFS_PROCMAP_QUERY command.
>
> Of course an artificial benchmark of "read a whole file" vs. "a tiny
> ioctl" is going to be different, but step back and show how this is
> going to be used in the real world overall.  Pounding on this file is
> not a normal operation, right?
>

It's not artificial at all. It's *exactly* what, say, blazesym library
is doing (see [0], it's Rust and part of the overall library API, I
think C code in this patch is way easier to follow for someone not
familiar with implementation of blazesym, but both implementations are
doing exactly the same sequence of steps). You can do it even less
efficiently by parsing the whole file, building an in-memory lookup
table, then looking up addresses one by one. But that's even slower
and more memory-hungry. So I didn't even bother implementing that, it
would put /proc/<pid>/maps at even more disadvantage.

Other applications that deal with stack traces (including perf) would
be doing one of those two approaches, depending on circumstances and
level of sophistication of code (and sensitivity to performance).

  [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1e8e5d901cddb2b33ab1/src/normalize/user.rs#L193

> thanks,
>
> greg k-h

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Ian Rogers]

On Sat, May 4, 2024 at 2:57 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Sat, May 4, 2024 at 8:29 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > > Implement a simple tool/benchmark for comparing address "resolution"
> > > logic based on textual /proc/<pid>/maps interface and new binary
> > > ioctl-based PROCFS_PROCMAP_QUERY command.
> >
> > Of course an artificial benchmark of "read a whole file" vs. "a tiny
> > ioctl" is going to be different, but step back and show how this is
> > going to be used in the real world overall.  Pounding on this file is
> > not a normal operation, right?
> >
>
> It's not artificial at all. It's *exactly* what, say, blazesym library
> is doing (see [0], it's Rust and part of the overall library API, I
> think C code in this patch is way easier to follow for someone not
> familiar with implementation of blazesym, but both implementations are
> doing exactly the same sequence of steps). You can do it even less
> efficiently by parsing the whole file, building an in-memory lookup
> table, then looking up addresses one by one. But that's even slower
> and more memory-hungry. So I didn't even bother implementing that, it
> would put /proc/<pid>/maps at even more disadvantage.
>
> Other applications that deal with stack traces (including perf) would
> be doing one of those two approaches, depending on circumstances and
> level of sophistication of code (and sensitivity to performance).

The code in perf doing this is here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/synthetic-events.c#n440
The code is using the api/io.h code:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/api/io.h
Using perf to profile perf it was observed time was spent allocating
buffers and locale related activities when using stdio, so io is a
lighter weight alternative, albeit with more verbose code than fscanf.
You could add this as an alternate /proc/<pid>/maps reader, we have a
similar benchmark in `perf bench internals synthesize`.

Thanks,
Ian

>   [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1e8e5d901cddb2b33ab1/src/normalize/user.rs#L193
>
> > thanks,
> >
> > greg k-h
>

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Andrii Nakryiko]

On Sat, May 4, 2024 at 10:09 PM Ian Rogers <irogers@google.com> wrote:
>
> On Sat, May 4, 2024 at 2:57 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Sat, May 4, 2024 at 8:29 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > > > Implement a simple tool/benchmark for comparing address "resolution"
> > > > logic based on textual /proc/<pid>/maps interface and new binary
> > > > ioctl-based PROCFS_PROCMAP_QUERY command.
> > >
> > > Of course an artificial benchmark of "read a whole file" vs. "a tiny
> > > ioctl" is going to be different, but step back and show how this is
> > > going to be used in the real world overall.  Pounding on this file is
> > > not a normal operation, right?
> > >
> >
> > It's not artificial at all. It's *exactly* what, say, blazesym library
> > is doing (see [0], it's Rust and part of the overall library API, I
> > think C code in this patch is way easier to follow for someone not
> > familiar with implementation of blazesym, but both implementations are
> > doing exactly the same sequence of steps). You can do it even less
> > efficiently by parsing the whole file, building an in-memory lookup
> > table, then looking up addresses one by one. But that's even slower
> > and more memory-hungry. So I didn't even bother implementing that, it
> > would put /proc/<pid>/maps at even more disadvantage.
> >
> > Other applications that deal with stack traces (including perf) would
> > be doing one of those two approaches, depending on circumstances and
> > level of sophistication of code (and sensitivity to performance).
>
> The code in perf doing this is here:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/synthetic-events.c#n440
> The code is using the api/io.h code:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/api/io.h
> Using perf to profile perf it was observed time was spent allocating
> buffers and locale related activities when using stdio, so io is a
> lighter weight alternative, albeit with more verbose code than fscanf.
> You could add this as an alternate /proc/<pid>/maps reader, we have a
> similar benchmark in `perf bench internals synthesize`.
>

If I add a new implementation using this ioctl() into
perf_event__synthesize_mmap_events(), will it be tested from this
`perf bench internals synthesize`? I'm not too familiar with perf code
organization, sorry if it's a stupid question. If not, where exactly
is the code that would be triggered from benchmark?

> Thanks,
> Ian
>
> >   [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1e8e5d901cddb2b33ab1/src/normalize/user.rs#L193
> >
> > > thanks,
> > >
> > > greg k-h
> >

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Ian Rogers]

On Mon, May 6, 2024 at 11:32 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Sat, May 4, 2024 at 10:09 PM Ian Rogers <irogers@google.com> wrote:
> >
> > On Sat, May 4, 2024 at 2:57 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Sat, May 4, 2024 at 8:29 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > >
> > > > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > > > > Implement a simple tool/benchmark for comparing address "resolution"
> > > > > logic based on textual /proc/<pid>/maps interface and new binary
> > > > > ioctl-based PROCFS_PROCMAP_QUERY command.
> > > >
> > > > Of course an artificial benchmark of "read a whole file" vs. "a tiny
> > > > ioctl" is going to be different, but step back and show how this is
> > > > going to be used in the real world overall.  Pounding on this file is
> > > > not a normal operation, right?
> > > >
> > >
> > > It's not artificial at all. It's *exactly* what, say, blazesym library
> > > is doing (see [0], it's Rust and part of the overall library API, I
> > > think C code in this patch is way easier to follow for someone not
> > > familiar with implementation of blazesym, but both implementations are
> > > doing exactly the same sequence of steps). You can do it even less
> > > efficiently by parsing the whole file, building an in-memory lookup
> > > table, then looking up addresses one by one. But that's even slower
> > > and more memory-hungry. So I didn't even bother implementing that, it
> > > would put /proc/<pid>/maps at even more disadvantage.
> > >
> > > Other applications that deal with stack traces (including perf) would
> > > be doing one of those two approaches, depending on circumstances and
> > > level of sophistication of code (and sensitivity to performance).
> >
> > The code in perf doing this is here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/synthetic-events.c#n440
> > The code is using the api/io.h code:
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/api/io.h
> > Using perf to profile perf it was observed time was spent allocating
> > buffers and locale related activities when using stdio, so io is a
> > lighter weight alternative, albeit with more verbose code than fscanf.
> > You could add this as an alternate /proc/<pid>/maps reader, we have a
> > similar benchmark in `perf bench internals synthesize`.
> >
>
> If I add a new implementation using this ioctl() into
> perf_event__synthesize_mmap_events(), will it be tested from this
> `perf bench internals synthesize`? I'm not too familiar with perf code
> organization, sorry if it's a stupid question. If not, where exactly
> is the code that would be triggered from benchmark?

Yes it would be triggered :-)

Thanks,
Ian

> > Thanks,
> > Ian
> >
> > >   [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1e8e5d901cddb2b33ab1/src/normalize/user.rs#L193
> > >
> > > > thanks,
> > > >
> > > > greg k-h
> > >

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Andrii Nakryiko]

On Mon, May 6, 2024 at 11:43 AM Ian Rogers <irogers@google.com> wrote:
>
> On Mon, May 6, 2024 at 11:32 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Sat, May 4, 2024 at 10:09 PM Ian Rogers <irogers@google.com> wrote:
> > >
> > > On Sat, May 4, 2024 at 2:57 PM Andrii Nakryiko
> > > <andrii.nakryiko@gmail.com> wrote:
> > > >
> > > > On Sat, May 4, 2024 at 8:29 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > >
> > > > > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > > > > > Implement a simple tool/benchmark for comparing address "resolution"
> > > > > > logic based on textual /proc/<pid>/maps interface and new binary
> > > > > > ioctl-based PROCFS_PROCMAP_QUERY command.
> > > > >
> > > > > Of course an artificial benchmark of "read a whole file" vs. "a tiny
> > > > > ioctl" is going to be different, but step back and show how this is
> > > > > going to be used in the real world overall.  Pounding on this file is
> > > > > not a normal operation, right?
> > > > >
> > > >
> > > > It's not artificial at all. It's *exactly* what, say, blazesym library
> > > > is doing (see [0], it's Rust and part of the overall library API, I
> > > > think C code in this patch is way easier to follow for someone not
> > > > familiar with implementation of blazesym, but both implementations are
> > > > doing exactly the same sequence of steps). You can do it even less
> > > > efficiently by parsing the whole file, building an in-memory lookup
> > > > table, then looking up addresses one by one. But that's even slower
> > > > and more memory-hungry. So I didn't even bother implementing that, it
> > > > would put /proc/<pid>/maps at even more disadvantage.
> > > >
> > > > Other applications that deal with stack traces (including perf) would
> > > > be doing one of those two approaches, depending on circumstances and
> > > > level of sophistication of code (and sensitivity to performance).
> > >
> > > The code in perf doing this is here:
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/synthetic-events.c#n440
> > > The code is using the api/io.h code:
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/api/io.h
> > > Using perf to profile perf it was observed time was spent allocating
> > > buffers and locale related activities when using stdio, so io is a
> > > lighter weight alternative, albeit with more verbose code than fscanf.
> > > You could add this as an alternate /proc/<pid>/maps reader, we have a
> > > similar benchmark in `perf bench internals synthesize`.
> > >
> >
> > If I add a new implementation using this ioctl() into
> > perf_event__synthesize_mmap_events(), will it be tested from this
> > `perf bench internals synthesize`? I'm not too familiar with perf code
> > organization, sorry if it's a stupid question. If not, where exactly
> > is the code that would be triggered from benchmark?
>
> Yes it would be triggered :-)

Ok, I don't exactly know how to interpret the results (and what the
benchmark is doing), but numbers don't seem to be worse. They actually
seem to be a bit better.

I pushed my code that adds perf integration to [0]. That commit has
results, but I'll post them here (with invocation parameters).
perf-ioctl is the version with ioctl()-based implementation,
perf-parse is, logically, text-parsing version. Here are the results
(and see my notes below the results as well):

TEXT-BASED
==========

# ./perf-parse bench internals synthesize
# Running 'internals/synthesize' benchmark:
Computing performance of single threaded perf event synthesis by
synthesizing events on the perf process itself:
  Average synthesis took: 80.311 usec (+- 0.077 usec)
  Average num. events: 32.000 (+- 0.000)
  Average time per event 2.510 usec
  Average data synthesis took: 84.429 usec (+- 0.066 usec)
  Average num. events: 179.000 (+- 0.000)
  Average time per event 0.472 usec

# ./perf-parse bench internals synthesize
# Running 'internals/synthesize' benchmark:
Computing performance of single threaded perf event synthesis by
synthesizing events on the perf process itself:
  Average synthesis took: 79.900 usec (+- 0.077 usec)
  Average num. events: 32.000 (+- 0.000)
  Average time per event 2.497 usec
  Average data synthesis took: 84.832 usec (+- 0.074 usec)
  Average num. events: 180.000 (+- 0.000)
  Average time per event 0.471 usec

# ./perf-parse bench internals synthesize --mt -M 8
# Running 'internals/synthesize' benchmark:
Computing performance of multi threaded perf event synthesis by
synthesizing events on CPU 0:
  Number of synthesis threads: 1
    Average synthesis took: 36338.100 usec (+- 406.091 usec)
    Average num. events: 14091.300 (+- 7.433)
    Average time per event 2.579 usec
  Number of synthesis threads: 2
    Average synthesis took: 37071.200 usec (+- 746.498 usec)
    Average num. events: 14085.900 (+- 1.900)
    Average time per event 2.632 usec
  Number of synthesis threads: 3
    Average synthesis took: 33932.300 usec (+- 626.861 usec)
    Average num. events: 14085.900 (+- 1.900)
    Average time per event 2.409 usec
  Number of synthesis threads: 4
    Average synthesis took: 33822.700 usec (+- 506.290 usec)
    Average num. events: 14099.200 (+- 8.761)
    Average time per event 2.399 usec
  Number of synthesis threads: 5
    Average synthesis took: 33348.200 usec (+- 389.771 usec)
    Average num. events: 14085.900 (+- 1.900)
    Average time per event 2.367 usec
  Number of synthesis threads: 6
    Average synthesis took: 33269.600 usec (+- 350.341 usec)
    Average num. events: 14084.000 (+- 0.000)
    Average time per event 2.362 usec
  Number of synthesis threads: 7
    Average synthesis took: 32663.900 usec (+- 338.870 usec)
    Average num. events: 14085.900 (+- 1.900)
    Average time per event 2.319 usec
  Number of synthesis threads: 8
    Average synthesis took: 32748.400 usec (+- 285.450 usec)
    Average num. events: 14085.900 (+- 1.900)
    Average time per event 2.325 usec

IOCTL-BASED
===========
# ./perf-ioctl bench internals synthesize
# Running 'internals/synthesize' benchmark:
Computing performance of single threaded perf event synthesis by
synthesizing events on the perf process itself:
  Average synthesis took: 72.996 usec (+- 0.076 usec)
  Average num. events: 31.000 (+- 0.000)
  Average time per event 2.355 usec
  Average data synthesis took: 79.067 usec (+- 0.074 usec)
  Average num. events: 178.000 (+- 0.000)
  Average time per event 0.444 usec

# ./perf-ioctl bench internals synthesize
# Running 'internals/synthesize' benchmark:
Computing performance of single threaded perf event synthesis by
synthesizing events on the perf process itself:
  Average synthesis took: 73.921 usec (+- 0.073 usec)
  Average num. events: 31.000 (+- 0.000)
  Average time per event 2.385 usec
  Average data synthesis took: 80.545 usec (+- 0.070 usec)
  Average num. events: 178.000 (+- 0.000)
  Average time per event 0.453 usec

# ./perf-ioctl bench internals synthesize --mt -M 8
# Running 'internals/synthesize' benchmark:
Computing performance of multi threaded perf event synthesis by
synthesizing events on CPU 0:
  Number of synthesis threads: 1
    Average synthesis took: 35609.500 usec (+- 428.576 usec)
    Average num. events: 14040.700 (+- 1.700)
    Average time per event 2.536 usec
  Number of synthesis threads: 2
    Average synthesis took: 34293.800 usec (+- 453.811 usec)
    Average num. events: 14040.700 (+- 1.700)
    Average time per event 2.442 usec
  Number of synthesis threads: 3
    Average synthesis took: 32385.200 usec (+- 363.106 usec)
    Average num. events: 14040.700 (+- 1.700)
    Average time per event 2.307 usec
  Number of synthesis threads: 4
    Average synthesis took: 33113.100 usec (+- 553.931 usec)
    Average num. events: 14054.500 (+- 11.469)
    Average time per event 2.356 usec
  Number of synthesis threads: 5
    Average synthesis took: 31600.600 usec (+- 297.349 usec)
    Average num. events: 14012.500 (+- 4.590)
    Average time per event 2.255 usec
  Number of synthesis threads: 6
    Average synthesis took: 32309.900 usec (+- 472.225 usec)
    Average num. events: 14004.000 (+- 0.000)
    Average time per event 2.307 usec
  Number of synthesis threads: 7
    Average synthesis took: 31400.100 usec (+- 206.261 usec)
    Average num. events: 14004.800 (+- 0.800)
    Average time per event 2.242 usec
  Number of synthesis threads: 8
    Average synthesis took: 31601.400 usec (+- 303.350 usec)
    Average num. events: 14005.700 (+- 1.700)
    Average time per event 2.256 usec

I also double-checked (using strace) that it does what it is supposed
to do, and it seems like everything checks out. Here's text-based
strace log:

openat(AT_FDCWD, "/proc/35876/task/35876/maps", O_RDONLY) = 3
read(3, "00400000-0040c000 r--p 00000000 "..., 8192) = 3997
read(3, "7f519d4d3000-7f519d516000 r--p 0"..., 8192) = 4025
read(3, "7f519dc3d000-7f519dc44000 r-xp 0"..., 8192) = 4048
read(3, "7f519dd2d000-7f519dd2f000 r--p 0"..., 8192) = 4017
read(3, "7f519dff6000-7f519dff8000 r--p 0"..., 8192) = 2744
read(3, "", 8192)                       = 0
close(3)                                = 0


BTW, note how the kernel doesn't serve more than 4KB of data, even
though perf provides 8KB buffer (that's to Greg's question about
optimizing using bigger buffers, I suspect without seq_file changes,
it won't work).

And here's an abbreviated log for ioctl version, it has lots more (but
much faster) ioctl() syscalls, given it dumps everything:

openat(AT_FDCWD, "/proc/36380/task/36380/maps", O_RDONLY) = 3
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0

 ... 195 ioctl() calls in total ...

ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50)
= -1 ENOENT (No such file or directory)
close(3)                                = 0


So, it's not the optimal usage of this API, and yet it's still better
(or at least not worse) than text-based API.

  [0] https://github.com/anakryiko/linux/commit/0841fe675ed30f5605c5b228e18f5612ea253b35

>
> Thanks,
> Ian
>
> > > Thanks,
> > > Ian
> > >
> > > >   [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1e8e5d901cddb2b33ab1/src/normalize/user.rs#L193
> > > >
> > > > > thanks,
> > > > >
> > > > > greg k-h
> > > >

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Andrii Nakryiko]

On Mon, May 6, 2024 at 10:06 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Mon, May 6, 2024 at 11:43 AM Ian Rogers <irogers@google.com> wrote:
> >
> > On Mon, May 6, 2024 at 11:32 AM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Sat, May 4, 2024 at 10:09 PM Ian Rogers <irogers@google.com> wrote:
> > > >
> > > > On Sat, May 4, 2024 at 2:57 PM Andrii Nakryiko
> > > > <andrii.nakryiko@gmail.com> wrote:
> > > > >
> > > > > On Sat, May 4, 2024 at 8:29 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > >
> > > > > > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > > > > > > Implement a simple tool/benchmark for comparing address "resolution"
> > > > > > > logic based on textual /proc/<pid>/maps interface and new binary
> > > > > > > ioctl-based PROCFS_PROCMAP_QUERY command.
> > > > > >
> > > > > > Of course an artificial benchmark of "read a whole file" vs. "a tiny
> > > > > > ioctl" is going to be different, but step back and show how this is
> > > > > > going to be used in the real world overall.  Pounding on this file is
> > > > > > not a normal operation, right?
> > > > > >
> > > > >
> > > > > It's not artificial at all. It's *exactly* what, say, blazesym library
> > > > > is doing (see [0], it's Rust and part of the overall library API, I
> > > > > think C code in this patch is way easier to follow for someone not
> > > > > familiar with implementation of blazesym, but both implementations are
> > > > > doing exactly the same sequence of steps). You can do it even less
> > > > > efficiently by parsing the whole file, building an in-memory lookup
> > > > > table, then looking up addresses one by one. But that's even slower
> > > > > and more memory-hungry. So I didn't even bother implementing that, it
> > > > > would put /proc/<pid>/maps at even more disadvantage.
> > > > >
> > > > > Other applications that deal with stack traces (including perf) would
> > > > > be doing one of those two approaches, depending on circumstances and
> > > > > level of sophistication of code (and sensitivity to performance).
> > > >
> > > > The code in perf doing this is here:
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/synthetic-events.c#n440
> > > > The code is using the api/io.h code:
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/api/io.h
> > > > Using perf to profile perf it was observed time was spent allocating
> > > > buffers and locale related activities when using stdio, so io is a
> > > > lighter weight alternative, albeit with more verbose code than fscanf.
> > > > You could add this as an alternate /proc/<pid>/maps reader, we have a
> > > > similar benchmark in `perf bench internals synthesize`.
> > > >
> > >
> > > If I add a new implementation using this ioctl() into
> > > perf_event__synthesize_mmap_events(), will it be tested from this
> > > `perf bench internals synthesize`? I'm not too familiar with perf code
> > > organization, sorry if it's a stupid question. If not, where exactly
> > > is the code that would be triggered from benchmark?
> >
> > Yes it would be triggered :-)
>
> Ok, I don't exactly know how to interpret the results (and what the
> benchmark is doing), but numbers don't seem to be worse. They actually
> seem to be a bit better.
>
> I pushed my code that adds perf integration to [0]. That commit has
> results, but I'll post them here (with invocation parameters).
> perf-ioctl is the version with ioctl()-based implementation,
> perf-parse is, logically, text-parsing version. Here are the results
> (and see my notes below the results as well):
>
> TEXT-BASED
> ==========
>
> # ./perf-parse bench internals synthesize
> # Running 'internals/synthesize' benchmark:
> Computing performance of single threaded perf event synthesis by
> synthesizing events on the perf process itself:
>   Average synthesis took: 80.311 usec (+- 0.077 usec)
>   Average num. events: 32.000 (+- 0.000)
>   Average time per event 2.510 usec
>   Average data synthesis took: 84.429 usec (+- 0.066 usec)
>   Average num. events: 179.000 (+- 0.000)
>   Average time per event 0.472 usec
>
> # ./perf-parse bench internals synthesize
> # Running 'internals/synthesize' benchmark:
> Computing performance of single threaded perf event synthesis by
> synthesizing events on the perf process itself:
>   Average synthesis took: 79.900 usec (+- 0.077 usec)
>   Average num. events: 32.000 (+- 0.000)
>   Average time per event 2.497 usec
>   Average data synthesis took: 84.832 usec (+- 0.074 usec)
>   Average num. events: 180.000 (+- 0.000)
>   Average time per event 0.471 usec
>
> # ./perf-parse bench internals synthesize --mt -M 8
> # Running 'internals/synthesize' benchmark:
> Computing performance of multi threaded perf event synthesis by
> synthesizing events on CPU 0:
>   Number of synthesis threads: 1
>     Average synthesis took: 36338.100 usec (+- 406.091 usec)
>     Average num. events: 14091.300 (+- 7.433)
>     Average time per event 2.579 usec
>   Number of synthesis threads: 2
>     Average synthesis took: 37071.200 usec (+- 746.498 usec)
>     Average num. events: 14085.900 (+- 1.900)
>     Average time per event 2.632 usec
>   Number of synthesis threads: 3
>     Average synthesis took: 33932.300 usec (+- 626.861 usec)
>     Average num. events: 14085.900 (+- 1.900)
>     Average time per event 2.409 usec
>   Number of synthesis threads: 4
>     Average synthesis took: 33822.700 usec (+- 506.290 usec)
>     Average num. events: 14099.200 (+- 8.761)
>     Average time per event 2.399 usec
>   Number of synthesis threads: 5
>     Average synthesis took: 33348.200 usec (+- 389.771 usec)
>     Average num. events: 14085.900 (+- 1.900)
>     Average time per event 2.367 usec
>   Number of synthesis threads: 6
>     Average synthesis took: 33269.600 usec (+- 350.341 usec)
>     Average num. events: 14084.000 (+- 0.000)
>     Average time per event 2.362 usec
>   Number of synthesis threads: 7
>     Average synthesis took: 32663.900 usec (+- 338.870 usec)
>     Average num. events: 14085.900 (+- 1.900)
>     Average time per event 2.319 usec
>   Number of synthesis threads: 8
>     Average synthesis took: 32748.400 usec (+- 285.450 usec)
>     Average num. events: 14085.900 (+- 1.900)
>     Average time per event 2.325 usec
>
> IOCTL-BASED
> ===========
> # ./perf-ioctl bench internals synthesize
> # Running 'internals/synthesize' benchmark:
> Computing performance of single threaded perf event synthesis by
> synthesizing events on the perf process itself:
>   Average synthesis took: 72.996 usec (+- 0.076 usec)
>   Average num. events: 31.000 (+- 0.000)
>   Average time per event 2.355 usec
>   Average data synthesis took: 79.067 usec (+- 0.074 usec)
>   Average num. events: 178.000 (+- 0.000)
>   Average time per event 0.444 usec
>
> # ./perf-ioctl bench internals synthesize
> # Running 'internals/synthesize' benchmark:
> Computing performance of single threaded perf event synthesis by
> synthesizing events on the perf process itself:
>   Average synthesis took: 73.921 usec (+- 0.073 usec)
>   Average num. events: 31.000 (+- 0.000)
>   Average time per event 2.385 usec
>   Average data synthesis took: 80.545 usec (+- 0.070 usec)
>   Average num. events: 178.000 (+- 0.000)
>   Average time per event 0.453 usec
>
> # ./perf-ioctl bench internals synthesize --mt -M 8
> # Running 'internals/synthesize' benchmark:
> Computing performance of multi threaded perf event synthesis by
> synthesizing events on CPU 0:
>   Number of synthesis threads: 1
>     Average synthesis took: 35609.500 usec (+- 428.576 usec)
>     Average num. events: 14040.700 (+- 1.700)
>     Average time per event 2.536 usec
>   Number of synthesis threads: 2
>     Average synthesis took: 34293.800 usec (+- 453.811 usec)
>     Average num. events: 14040.700 (+- 1.700)
>     Average time per event 2.442 usec
>   Number of synthesis threads: 3
>     Average synthesis took: 32385.200 usec (+- 363.106 usec)
>     Average num. events: 14040.700 (+- 1.700)
>     Average time per event 2.307 usec
>   Number of synthesis threads: 4
>     Average synthesis took: 33113.100 usec (+- 553.931 usec)
>     Average num. events: 14054.500 (+- 11.469)
>     Average time per event 2.356 usec
>   Number of synthesis threads: 5
>     Average synthesis took: 31600.600 usec (+- 297.349 usec)
>     Average num. events: 14012.500 (+- 4.590)
>     Average time per event 2.255 usec
>   Number of synthesis threads: 6
>     Average synthesis took: 32309.900 usec (+- 472.225 usec)
>     Average num. events: 14004.000 (+- 0.000)
>     Average time per event 2.307 usec
>   Number of synthesis threads: 7
>     Average synthesis took: 31400.100 usec (+- 206.261 usec)
>     Average num. events: 14004.800 (+- 0.800)
>     Average time per event 2.242 usec
>   Number of synthesis threads: 8
>     Average synthesis took: 31601.400 usec (+- 303.350 usec)
>     Average num. events: 14005.700 (+- 1.700)
>     Average time per event 2.256 usec
>
> I also double-checked (using strace) that it does what it is supposed
> to do, and it seems like everything checks out. Here's text-based
> strace log:
>
> openat(AT_FDCWD, "/proc/35876/task/35876/maps", O_RDONLY) = 3
> read(3, "00400000-0040c000 r--p 00000000 "..., 8192) = 3997
> read(3, "7f519d4d3000-7f519d516000 r--p 0"..., 8192) = 4025
> read(3, "7f519dc3d000-7f519dc44000 r-xp 0"..., 8192) = 4048
> read(3, "7f519dd2d000-7f519dd2f000 r--p 0"..., 8192) = 4017
> read(3, "7f519dff6000-7f519dff8000 r--p 0"..., 8192) = 2744
> read(3, "", 8192)                       = 0
> close(3)                                = 0
>
>
> BTW, note how the kernel doesn't serve more than 4KB of data, even
> though perf provides 8KB buffer (that's to Greg's question about
> optimizing using bigger buffers, I suspect without seq_file changes,
> it won't work).
>
> And here's an abbreviated log for ioctl version, it has lots more (but
> much faster) ioctl() syscalls, given it dumps everything:
>
> openat(AT_FDCWD, "/proc/36380/task/36380/maps", O_RDONLY) = 3
> ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
>
>  ... 195 ioctl() calls in total ...
>
> ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50)
> = -1 ENOENT (No such file or directory)
> close(3)                                = 0
>
>
> So, it's not the optimal usage of this API, and yet it's still better
> (or at least not worse) than text-based API.
>

In another reply to Arnaldo on patch #2 I mentioned the idea of
allowing to iterate only file-backed VMAs (as it seems like what
symbolizers would only care about, but I might be wrong here). So I
tried that quickly, given it's a trivial addition to my code. See
results below (it is slightly faster, but not much, because most of
VMAs in that benchmark seem to be indeed file-backed anyways), just
for completeness. I'm not sure if that would be useful/used by perf,
so please let me know.

As I mentioned above, it's not radically faster in this perf
benchmark, because we still request about 170 VMAs (vs ~195 if we
iterate *all* of them), so not a big change. The ratio will vary
depending on what the process is doing, of course. Anyways, just for
completeness, I'm not sure if I have to add this "filter" to the
actual implementation.

# ./perf-filebacked bench internals synthesize
# Running 'internals/synthesize' benchmark:
Computing performance of single threaded perf event synthesis by
synthesizing events on the perf process itself:
  Average synthesis took: 65.759 usec (+- 0.063 usec)
  Average num. events: 30.000 (+- 0.000)
  Average time per event 2.192 usec
  Average data synthesis took: 73.840 usec (+- 0.080 usec)
  Average num. events: 153.000 (+- 0.000)
  Average time per event 0.483 usec

# ./perf-filebacked bench internals synthesize
# Running 'internals/synthesize' benchmark:
Computing performance of single threaded perf event synthesis by
synthesizing events on the perf process itself:
  Average synthesis took: 66.245 usec (+- 0.059 usec)
  Average num. events: 30.000 (+- 0.000)
  Average time per event 2.208 usec
  Average data synthesis took: 70.627 usec (+- 0.074 usec)
  Average num. events: 153.000 (+- 0.000)
  Average time per event 0.462 usec

# ./perf-filebacked bench internals synthesize --mt -M 8
# Running 'internals/synthesize' benchmark:
Computing performance of multi threaded perf event synthesis by
synthesizing events on CPU 0:
  Number of synthesis threads: 1
    Average synthesis took: 33477.500 usec (+- 556.102 usec)
    Average num. events: 10125.700 (+- 1.620)
    Average time per event 3.306 usec
  Number of synthesis threads: 2
    Average synthesis took: 30473.700 usec (+- 221.933 usec)
    Average num. events: 10127.000 (+- 0.000)
    Average time per event 3.009 usec
  Number of synthesis threads: 3
    Average synthesis took: 29775.200 usec (+- 315.212 usec)
    Average num. events: 10128.700 (+- 0.667)
    Average time per event 2.940 usec
  Number of synthesis threads: 4
    Average synthesis took: 29477.100 usec (+- 621.258 usec)
    Average num. events: 10129.000 (+- 0.000)
    Average time per event 2.910 usec
  Number of synthesis threads: 5
    Average synthesis took: 29777.900 usec (+- 294.710 usec)
    Average num. events: 10144.700 (+- 11.597)
    Average time per event 2.935 usec
  Number of synthesis threads: 6
    Average synthesis took: 27774.700 usec (+- 357.569 usec)
    Average num. events: 10158.500 (+- 14.710)
    Average time per event 2.734 usec
  Number of synthesis threads: 7
    Average synthesis took: 27437.200 usec (+- 233.626 usec)
    Average num. events: 10135.700 (+- 2.700)
    Average time per event 2.707 usec
  Number of synthesis threads: 8
    Average synthesis took: 28784.600 usec (+- 477.630 usec)
    Average num. events: 10133.000 (+- 0.000)
    Average time per event 2.841 usec

>   [0] https://github.com/anakryiko/linux/commit/0841fe675ed30f5605c5b228e18f5612ea253b35
>
> >
> > Thanks,
> > Ian
> >
> > > > Thanks,
> > > > Ian
> > > >
> > > > >   [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1e8e5d901cddb2b33ab1/src/normalize/user.rs#L193
> > > > >
> > > > > > thanks,
> > > > > >
> > > > > > greg k-h
> > > > >

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Namhyung Kim]

On Tue, May 7, 2024 at 10:29 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Mon, May 6, 2024 at 10:06 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Mon, May 6, 2024 at 11:43 AM Ian Rogers <irogers@google.com> wrote:
> > >
> > > On Mon, May 6, 2024 at 11:32 AM Andrii Nakryiko
> > > <andrii.nakryiko@gmail.com> wrote:
> > > >
> > > > On Sat, May 4, 2024 at 10:09 PM Ian Rogers <irogers@google.com> wrote:
> > > > >
> > > > > On Sat, May 4, 2024 at 2:57 PM Andrii Nakryiko
> > > > > <andrii.nakryiko@gmail.com> wrote:
> > > > > >
> > > > > > On Sat, May 4, 2024 at 8:29 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > > >
> > > > > > > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > > > > > > > Implement a simple tool/benchmark for comparing address "resolution"
> > > > > > > > logic based on textual /proc/<pid>/maps interface and new binary
> > > > > > > > ioctl-based PROCFS_PROCMAP_QUERY command.
> > > > > > >
> > > > > > > Of course an artificial benchmark of "read a whole file" vs. "a tiny
> > > > > > > ioctl" is going to be different, but step back and show how this is
> > > > > > > going to be used in the real world overall.  Pounding on this file is
> > > > > > > not a normal operation, right?
> > > > > > >
> > > > > >
> > > > > > It's not artificial at all. It's *exactly* what, say, blazesym library
> > > > > > is doing (see [0], it's Rust and part of the overall library API, I
> > > > > > think C code in this patch is way easier to follow for someone not
> > > > > > familiar with implementation of blazesym, but both implementations are
> > > > > > doing exactly the same sequence of steps). You can do it even less
> > > > > > efficiently by parsing the whole file, building an in-memory lookup
> > > > > > table, then looking up addresses one by one. But that's even slower
> > > > > > and more memory-hungry. So I didn't even bother implementing that, it
> > > > > > would put /proc/<pid>/maps at even more disadvantage.
> > > > > >
> > > > > > Other applications that deal with stack traces (including perf) would
> > > > > > be doing one of those two approaches, depending on circumstances and
> > > > > > level of sophistication of code (and sensitivity to performance).
> > > > >
> > > > > The code in perf doing this is here:
> > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/synthetic-events.c#n440
> > > > > The code is using the api/io.h code:
> > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/api/io.h
> > > > > Using perf to profile perf it was observed time was spent allocating
> > > > > buffers and locale related activities when using stdio, so io is a
> > > > > lighter weight alternative, albeit with more verbose code than fscanf.
> > > > > You could add this as an alternate /proc/<pid>/maps reader, we have a
> > > > > similar benchmark in `perf bench internals synthesize`.
> > > > >
> > > >
> > > > If I add a new implementation using this ioctl() into
> > > > perf_event__synthesize_mmap_events(), will it be tested from this
> > > > `perf bench internals synthesize`? I'm not too familiar with perf code
> > > > organization, sorry if it's a stupid question. If not, where exactly
> > > > is the code that would be triggered from benchmark?
> > >
> > > Yes it would be triggered :-)
> >
> > Ok, I don't exactly know how to interpret the results (and what the
> > benchmark is doing), but numbers don't seem to be worse. They actually
> > seem to be a bit better.
> >
> > I pushed my code that adds perf integration to [0]. That commit has
> > results, but I'll post them here (with invocation parameters).
> > perf-ioctl is the version with ioctl()-based implementation,
> > perf-parse is, logically, text-parsing version. Here are the results
> > (and see my notes below the results as well):
> >
> > TEXT-BASED
> > ==========
> >
> > # ./perf-parse bench internals synthesize
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of single threaded perf event synthesis by
> > synthesizing events on the perf process itself:
> >   Average synthesis took: 80.311 usec (+- 0.077 usec)
> >   Average num. events: 32.000 (+- 0.000)
> >   Average time per event 2.510 usec
> >   Average data synthesis took: 84.429 usec (+- 0.066 usec)
> >   Average num. events: 179.000 (+- 0.000)
> >   Average time per event 0.472 usec
> >
> > # ./perf-parse bench internals synthesize
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of single threaded perf event synthesis by
> > synthesizing events on the perf process itself:
> >   Average synthesis took: 79.900 usec (+- 0.077 usec)
> >   Average num. events: 32.000 (+- 0.000)
> >   Average time per event 2.497 usec
> >   Average data synthesis took: 84.832 usec (+- 0.074 usec)
> >   Average num. events: 180.000 (+- 0.000)
> >   Average time per event 0.471 usec
> >
> > # ./perf-parse bench internals synthesize --mt -M 8
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of multi threaded perf event synthesis by
> > synthesizing events on CPU 0:
> >   Number of synthesis threads: 1
> >     Average synthesis took: 36338.100 usec (+- 406.091 usec)
> >     Average num. events: 14091.300 (+- 7.433)
> >     Average time per event 2.579 usec
> >   Number of synthesis threads: 2
> >     Average synthesis took: 37071.200 usec (+- 746.498 usec)
> >     Average num. events: 14085.900 (+- 1.900)
> >     Average time per event 2.632 usec
> >   Number of synthesis threads: 3
> >     Average synthesis took: 33932.300 usec (+- 626.861 usec)
> >     Average num. events: 14085.900 (+- 1.900)
> >     Average time per event 2.409 usec
> >   Number of synthesis threads: 4
> >     Average synthesis took: 33822.700 usec (+- 506.290 usec)
> >     Average num. events: 14099.200 (+- 8.761)
> >     Average time per event 2.399 usec
> >   Number of synthesis threads: 5
> >     Average synthesis took: 33348.200 usec (+- 389.771 usec)
> >     Average num. events: 14085.900 (+- 1.900)
> >     Average time per event 2.367 usec
> >   Number of synthesis threads: 6
> >     Average synthesis took: 33269.600 usec (+- 350.341 usec)
> >     Average num. events: 14084.000 (+- 0.000)
> >     Average time per event 2.362 usec
> >   Number of synthesis threads: 7
> >     Average synthesis took: 32663.900 usec (+- 338.870 usec)
> >     Average num. events: 14085.900 (+- 1.900)
> >     Average time per event 2.319 usec
> >   Number of synthesis threads: 8
> >     Average synthesis took: 32748.400 usec (+- 285.450 usec)
> >     Average num. events: 14085.900 (+- 1.900)
> >     Average time per event 2.325 usec
> >
> > IOCTL-BASED
> > ===========
> > # ./perf-ioctl bench internals synthesize
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of single threaded perf event synthesis by
> > synthesizing events on the perf process itself:
> >   Average synthesis took: 72.996 usec (+- 0.076 usec)
> >   Average num. events: 31.000 (+- 0.000)
> >   Average time per event 2.355 usec
> >   Average data synthesis took: 79.067 usec (+- 0.074 usec)
> >   Average num. events: 178.000 (+- 0.000)
> >   Average time per event 0.444 usec
> >
> > # ./perf-ioctl bench internals synthesize
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of single threaded perf event synthesis by
> > synthesizing events on the perf process itself:
> >   Average synthesis took: 73.921 usec (+- 0.073 usec)
> >   Average num. events: 31.000 (+- 0.000)
> >   Average time per event 2.385 usec
> >   Average data synthesis took: 80.545 usec (+- 0.070 usec)
> >   Average num. events: 178.000 (+- 0.000)
> >   Average time per event 0.453 usec
> >
> > # ./perf-ioctl bench internals synthesize --mt -M 8
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of multi threaded perf event synthesis by
> > synthesizing events on CPU 0:
> >   Number of synthesis threads: 1
> >     Average synthesis took: 35609.500 usec (+- 428.576 usec)
> >     Average num. events: 14040.700 (+- 1.700)
> >     Average time per event 2.536 usec
> >   Number of synthesis threads: 2
> >     Average synthesis took: 34293.800 usec (+- 453.811 usec)
> >     Average num. events: 14040.700 (+- 1.700)
> >     Average time per event 2.442 usec
> >   Number of synthesis threads: 3
> >     Average synthesis took: 32385.200 usec (+- 363.106 usec)
> >     Average num. events: 14040.700 (+- 1.700)
> >     Average time per event 2.307 usec
> >   Number of synthesis threads: 4
> >     Average synthesis took: 33113.100 usec (+- 553.931 usec)
> >     Average num. events: 14054.500 (+- 11.469)
> >     Average time per event 2.356 usec
> >   Number of synthesis threads: 5
> >     Average synthesis took: 31600.600 usec (+- 297.349 usec)
> >     Average num. events: 14012.500 (+- 4.590)
> >     Average time per event 2.255 usec
> >   Number of synthesis threads: 6
> >     Average synthesis took: 32309.900 usec (+- 472.225 usec)
> >     Average num. events: 14004.000 (+- 0.000)
> >     Average time per event 2.307 usec
> >   Number of synthesis threads: 7
> >     Average synthesis took: 31400.100 usec (+- 206.261 usec)
> >     Average num. events: 14004.800 (+- 0.800)
> >     Average time per event 2.242 usec
> >   Number of synthesis threads: 8
> >     Average synthesis took: 31601.400 usec (+- 303.350 usec)
> >     Average num. events: 14005.700 (+- 1.700)
> >     Average time per event 2.256 usec
> >
> > I also double-checked (using strace) that it does what it is supposed
> > to do, and it seems like everything checks out. Here's text-based
> > strace log:
> >
> > openat(AT_FDCWD, "/proc/35876/task/35876/maps", O_RDONLY) = 3
> > read(3, "00400000-0040c000 r--p 00000000 "..., 8192) = 3997
> > read(3, "7f519d4d3000-7f519d516000 r--p 0"..., 8192) = 4025
> > read(3, "7f519dc3d000-7f519dc44000 r-xp 0"..., 8192) = 4048
> > read(3, "7f519dd2d000-7f519dd2f000 r--p 0"..., 8192) = 4017
> > read(3, "7f519dff6000-7f519dff8000 r--p 0"..., 8192) = 2744
> > read(3, "", 8192)                       = 0
> > close(3)                                = 0
> >
> >
> > BTW, note how the kernel doesn't serve more than 4KB of data, even
> > though perf provides 8KB buffer (that's to Greg's question about
> > optimizing using bigger buffers, I suspect without seq_file changes,
> > it won't work).
> >
> > And here's an abbreviated log for ioctl version, it has lots more (but
> > much faster) ioctl() syscalls, given it dumps everything:
> >
> > openat(AT_FDCWD, "/proc/36380/task/36380/maps", O_RDONLY) = 3
> > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> >
> >  ... 195 ioctl() calls in total ...
> >
> > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50)
> > = -1 ENOENT (No such file or directory)
> > close(3)                                = 0
> >
> >
> > So, it's not the optimal usage of this API, and yet it's still better
> > (or at least not worse) than text-based API.

It's surprising that more ioctl is cheaper than less read and parse.

> >
>
> In another reply to Arnaldo on patch #2 I mentioned the idea of
> allowing to iterate only file-backed VMAs (as it seems like what
> symbolizers would only care about, but I might be wrong here). So I

Yep, I think it's enough to get file-backed VMAs only.


> tried that quickly, given it's a trivial addition to my code. See
> results below (it is slightly faster, but not much, because most of
> VMAs in that benchmark seem to be indeed file-backed anyways), just
> for completeness. I'm not sure if that would be useful/used by perf,
> so please let me know.

Thanks for doing this.  It'd be useful as it provides better synthesizing
performance.  The startup latency of perf record is a problem, I need
to take a look if it can be improved more.

Thanks,
Namhyung


>
> As I mentioned above, it's not radically faster in this perf
> benchmark, because we still request about 170 VMAs (vs ~195 if we
> iterate *all* of them), so not a big change. The ratio will vary
> depending on what the process is doing, of course. Anyways, just for
> completeness, I'm not sure if I have to add this "filter" to the
> actual implementation.
>
> # ./perf-filebacked bench internals synthesize
> # Running 'internals/synthesize' benchmark:
> Computing performance of single threaded perf event synthesis by
> synthesizing events on the perf process itself:
>   Average synthesis took: 65.759 usec (+- 0.063 usec)
>   Average num. events: 30.000 (+- 0.000)
>   Average time per event 2.192 usec
>   Average data synthesis took: 73.840 usec (+- 0.080 usec)
>   Average num. events: 153.000 (+- 0.000)
>   Average time per event 0.483 usec
>
> # ./perf-filebacked bench internals synthesize
> # Running 'internals/synthesize' benchmark:
> Computing performance of single threaded perf event synthesis by
> synthesizing events on the perf process itself:
>   Average synthesis took: 66.245 usec (+- 0.059 usec)
>   Average num. events: 30.000 (+- 0.000)
>   Average time per event 2.208 usec
>   Average data synthesis took: 70.627 usec (+- 0.074 usec)
>   Average num. events: 153.000 (+- 0.000)
>   Average time per event 0.462 usec
>
> # ./perf-filebacked bench internals synthesize --mt -M 8
> # Running 'internals/synthesize' benchmark:
> Computing performance of multi threaded perf event synthesis by
> synthesizing events on CPU 0:
>   Number of synthesis threads: 1
>     Average synthesis took: 33477.500 usec (+- 556.102 usec)
>     Average num. events: 10125.700 (+- 1.620)
>     Average time per event 3.306 usec
>   Number of synthesis threads: 2
>     Average synthesis took: 30473.700 usec (+- 221.933 usec)
>     Average num. events: 10127.000 (+- 0.000)
>     Average time per event 3.009 usec
>   Number of synthesis threads: 3
>     Average synthesis took: 29775.200 usec (+- 315.212 usec)
>     Average num. events: 10128.700 (+- 0.667)
>     Average time per event 2.940 usec
>   Number of synthesis threads: 4
>     Average synthesis took: 29477.100 usec (+- 621.258 usec)
>     Average num. events: 10129.000 (+- 0.000)
>     Average time per event 2.910 usec
>   Number of synthesis threads: 5
>     Average synthesis took: 29777.900 usec (+- 294.710 usec)
>     Average num. events: 10144.700 (+- 11.597)
>     Average time per event 2.935 usec
>   Number of synthesis threads: 6
>     Average synthesis took: 27774.700 usec (+- 357.569 usec)
>     Average num. events: 10158.500 (+- 14.710)
>     Average time per event 2.734 usec
>   Number of synthesis threads: 7
>     Average synthesis took: 27437.200 usec (+- 233.626 usec)
>     Average num. events: 10135.700 (+- 2.700)
>     Average time per event 2.707 usec
>   Number of synthesis threads: 8
>     Average synthesis took: 28784.600 usec (+- 477.630 usec)
>     Average num. events: 10133.000 (+- 0.000)
>     Average time per event 2.841 usec
>
> >   [0] https://github.com/anakryiko/linux/commit/0841fe675ed30f5605c5b228e18f5612ea253b35
> >
> > >
> > > Thanks,
> > > Ian
> > >
> > > > > Thanks,
> > > > > Ian
> > > > >
> > > > > >   [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1e8e5d901cddb2b33ab1/src/normalize/user.rs#L193
> > > > > >
> > > > > > > thanks,
> > > > > > >
> > > > > > > greg k-h
> > > > > >
>

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Andrii Nakryiko]

On Tue, May 7, 2024 at 3:27 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Tue, May 7, 2024 at 10:29 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Mon, May 6, 2024 at 10:06 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Mon, May 6, 2024 at 11:43 AM Ian Rogers <irogers@google.com> wrote:
> > > >
> > > > On Mon, May 6, 2024 at 11:32 AM Andrii Nakryiko
> > > > <andrii.nakryiko@gmail.com> wrote:
> > > > >
> > > > > On Sat, May 4, 2024 at 10:09 PM Ian Rogers <irogers@google.com> wrote:
> > > > > >
> > > > > > On Sat, May 4, 2024 at 2:57 PM Andrii Nakryiko
> > > > > > <andrii.nakryiko@gmail.com> wrote:
> > > > > > >
> > > > > > > On Sat, May 4, 2024 at 8:29 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > > > >
> > > > > > > > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > > > > > > > > Implement a simple tool/benchmark for comparing address "resolution"
> > > > > > > > > logic based on textual /proc/<pid>/maps interface and new binary
> > > > > > > > > ioctl-based PROCFS_PROCMAP_QUERY command.
> > > > > > > >
> > > > > > > > Of course an artificial benchmark of "read a whole file" vs. "a tiny
> > > > > > > > ioctl" is going to be different, but step back and show how this is
> > > > > > > > going to be used in the real world overall.  Pounding on this file is
> > > > > > > > not a normal operation, right?
> > > > > > > >
> > > > > > >
> > > > > > > It's not artificial at all. It's *exactly* what, say, blazesym library
> > > > > > > is doing (see [0], it's Rust and part of the overall library API, I
> > > > > > > think C code in this patch is way easier to follow for someone not
> > > > > > > familiar with implementation of blazesym, but both implementations are
> > > > > > > doing exactly the same sequence of steps). You can do it even less
> > > > > > > efficiently by parsing the whole file, building an in-memory lookup
> > > > > > > table, then looking up addresses one by one. But that's even slower
> > > > > > > and more memory-hungry. So I didn't even bother implementing that, it
> > > > > > > would put /proc/<pid>/maps at even more disadvantage.
> > > > > > >
> > > > > > > Other applications that deal with stack traces (including perf) would
> > > > > > > be doing one of those two approaches, depending on circumstances and
> > > > > > > level of sophistication of code (and sensitivity to performance).
> > > > > >
> > > > > > The code in perf doing this is here:
> > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/synthetic-events.c#n440
> > > > > > The code is using the api/io.h code:
> > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/api/io.h
> > > > > > Using perf to profile perf it was observed time was spent allocating
> > > > > > buffers and locale related activities when using stdio, so io is a
> > > > > > lighter weight alternative, albeit with more verbose code than fscanf.
> > > > > > You could add this as an alternate /proc/<pid>/maps reader, we have a
> > > > > > similar benchmark in `perf bench internals synthesize`.
> > > > > >
> > > > >
> > > > > If I add a new implementation using this ioctl() into
> > > > > perf_event__synthesize_mmap_events(), will it be tested from this
> > > > > `perf bench internals synthesize`? I'm not too familiar with perf code
> > > > > organization, sorry if it's a stupid question. If not, where exactly
> > > > > is the code that would be triggered from benchmark?
> > > >
> > > > Yes it would be triggered :-)
> > >
> > > Ok, I don't exactly know how to interpret the results (and what the
> > > benchmark is doing), but numbers don't seem to be worse. They actually
> > > seem to be a bit better.
> > >
> > > I pushed my code that adds perf integration to [0]. That commit has
> > > results, but I'll post them here (with invocation parameters).
> > > perf-ioctl is the version with ioctl()-based implementation,
> > > perf-parse is, logically, text-parsing version. Here are the results
> > > (and see my notes below the results as well):
> > >
> > > TEXT-BASED
> > > ==========
> > >
> > > # ./perf-parse bench internals synthesize
> > > # Running 'internals/synthesize' benchmark:
> > > Computing performance of single threaded perf event synthesis by
> > > synthesizing events on the perf process itself:
> > >   Average synthesis took: 80.311 usec (+- 0.077 usec)
> > >   Average num. events: 32.000 (+- 0.000)
> > >   Average time per event 2.510 usec
> > >   Average data synthesis took: 84.429 usec (+- 0.066 usec)
> > >   Average num. events: 179.000 (+- 0.000)
> > >   Average time per event 0.472 usec
> > >
> > > # ./perf-parse bench internals synthesize
> > > # Running 'internals/synthesize' benchmark:
> > > Computing performance of single threaded perf event synthesis by
> > > synthesizing events on the perf process itself:
> > >   Average synthesis took: 79.900 usec (+- 0.077 usec)
> > >   Average num. events: 32.000 (+- 0.000)
> > >   Average time per event 2.497 usec
> > >   Average data synthesis took: 84.832 usec (+- 0.074 usec)
> > >   Average num. events: 180.000 (+- 0.000)
> > >   Average time per event 0.471 usec
> > >
> > > # ./perf-parse bench internals synthesize --mt -M 8
> > > # Running 'internals/synthesize' benchmark:
> > > Computing performance of multi threaded perf event synthesis by
> > > synthesizing events on CPU 0:
> > >   Number of synthesis threads: 1
> > >     Average synthesis took: 36338.100 usec (+- 406.091 usec)
> > >     Average num. events: 14091.300 (+- 7.433)
> > >     Average time per event 2.579 usec
> > >   Number of synthesis threads: 2
> > >     Average synthesis took: 37071.200 usec (+- 746.498 usec)
> > >     Average num. events: 14085.900 (+- 1.900)
> > >     Average time per event 2.632 usec
> > >   Number of synthesis threads: 3
> > >     Average synthesis took: 33932.300 usec (+- 626.861 usec)
> > >     Average num. events: 14085.900 (+- 1.900)
> > >     Average time per event 2.409 usec
> > >   Number of synthesis threads: 4
> > >     Average synthesis took: 33822.700 usec (+- 506.290 usec)
> > >     Average num. events: 14099.200 (+- 8.761)
> > >     Average time per event 2.399 usec
> > >   Number of synthesis threads: 5
> > >     Average synthesis took: 33348.200 usec (+- 389.771 usec)
> > >     Average num. events: 14085.900 (+- 1.900)
> > >     Average time per event 2.367 usec
> > >   Number of synthesis threads: 6
> > >     Average synthesis took: 33269.600 usec (+- 350.341 usec)
> > >     Average num. events: 14084.000 (+- 0.000)
> > >     Average time per event 2.362 usec
> > >   Number of synthesis threads: 7
> > >     Average synthesis took: 32663.900 usec (+- 338.870 usec)
> > >     Average num. events: 14085.900 (+- 1.900)
> > >     Average time per event 2.319 usec
> > >   Number of synthesis threads: 8
> > >     Average synthesis took: 32748.400 usec (+- 285.450 usec)
> > >     Average num. events: 14085.900 (+- 1.900)
> > >     Average time per event 2.325 usec
> > >
> > > IOCTL-BASED
> > > ===========
> > > # ./perf-ioctl bench internals synthesize
> > > # Running 'internals/synthesize' benchmark:
> > > Computing performance of single threaded perf event synthesis by
> > > synthesizing events on the perf process itself:
> > >   Average synthesis took: 72.996 usec (+- 0.076 usec)
> > >   Average num. events: 31.000 (+- 0.000)
> > >   Average time per event 2.355 usec
> > >   Average data synthesis took: 79.067 usec (+- 0.074 usec)
> > >   Average num. events: 178.000 (+- 0.000)
> > >   Average time per event 0.444 usec
> > >
> > > # ./perf-ioctl bench internals synthesize
> > > # Running 'internals/synthesize' benchmark:
> > > Computing performance of single threaded perf event synthesis by
> > > synthesizing events on the perf process itself:
> > >   Average synthesis took: 73.921 usec (+- 0.073 usec)
> > >   Average num. events: 31.000 (+- 0.000)
> > >   Average time per event 2.385 usec
> > >   Average data synthesis took: 80.545 usec (+- 0.070 usec)
> > >   Average num. events: 178.000 (+- 0.000)
> > >   Average time per event 0.453 usec
> > >
> > > # ./perf-ioctl bench internals synthesize --mt -M 8
> > > # Running 'internals/synthesize' benchmark:
> > > Computing performance of multi threaded perf event synthesis by
> > > synthesizing events on CPU 0:
> > >   Number of synthesis threads: 1
> > >     Average synthesis took: 35609.500 usec (+- 428.576 usec)
> > >     Average num. events: 14040.700 (+- 1.700)
> > >     Average time per event 2.536 usec
> > >   Number of synthesis threads: 2
> > >     Average synthesis took: 34293.800 usec (+- 453.811 usec)
> > >     Average num. events: 14040.700 (+- 1.700)
> > >     Average time per event 2.442 usec
> > >   Number of synthesis threads: 3
> > >     Average synthesis took: 32385.200 usec (+- 363.106 usec)
> > >     Average num. events: 14040.700 (+- 1.700)
> > >     Average time per event 2.307 usec
> > >   Number of synthesis threads: 4
> > >     Average synthesis took: 33113.100 usec (+- 553.931 usec)
> > >     Average num. events: 14054.500 (+- 11.469)
> > >     Average time per event 2.356 usec
> > >   Number of synthesis threads: 5
> > >     Average synthesis took: 31600.600 usec (+- 297.349 usec)
> > >     Average num. events: 14012.500 (+- 4.590)
> > >     Average time per event 2.255 usec
> > >   Number of synthesis threads: 6
> > >     Average synthesis took: 32309.900 usec (+- 472.225 usec)
> > >     Average num. events: 14004.000 (+- 0.000)
> > >     Average time per event 2.307 usec
> > >   Number of synthesis threads: 7
> > >     Average synthesis took: 31400.100 usec (+- 206.261 usec)
> > >     Average num. events: 14004.800 (+- 0.800)
> > >     Average time per event 2.242 usec
> > >   Number of synthesis threads: 8
> > >     Average synthesis took: 31601.400 usec (+- 303.350 usec)
> > >     Average num. events: 14005.700 (+- 1.700)
> > >     Average time per event 2.256 usec
> > >
> > > I also double-checked (using strace) that it does what it is supposed
> > > to do, and it seems like everything checks out. Here's text-based
> > > strace log:
> > >
> > > openat(AT_FDCWD, "/proc/35876/task/35876/maps", O_RDONLY) = 3
> > > read(3, "00400000-0040c000 r--p 00000000 "..., 8192) = 3997
> > > read(3, "7f519d4d3000-7f519d516000 r--p 0"..., 8192) = 4025
> > > read(3, "7f519dc3d000-7f519dc44000 r-xp 0"..., 8192) = 4048
> > > read(3, "7f519dd2d000-7f519dd2f000 r--p 0"..., 8192) = 4017
> > > read(3, "7f519dff6000-7f519dff8000 r--p 0"..., 8192) = 2744
> > > read(3, "", 8192)                       = 0
> > > close(3)                                = 0
> > >
> > >
> > > BTW, note how the kernel doesn't serve more than 4KB of data, even
> > > though perf provides 8KB buffer (that's to Greg's question about
> > > optimizing using bigger buffers, I suspect without seq_file changes,
> > > it won't work).
> > >
> > > And here's an abbreviated log for ioctl version, it has lots more (but
> > > much faster) ioctl() syscalls, given it dumps everything:
> > >
> > > openat(AT_FDCWD, "/proc/36380/task/36380/maps", O_RDONLY) = 3
> > > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > >
> > >  ... 195 ioctl() calls in total ...
> > >
> > > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) = 0
> > > ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50)
> > > = -1 ENOENT (No such file or directory)
> > > close(3)                                = 0
> > >
> > >
> > > So, it's not the optimal usage of this API, and yet it's still better
> > > (or at least not worse) than text-based API.
>
> It's surprising that more ioctl is cheaper than less read and parse.

I encourage you to try this locally, just in case I missed something
([0]). But it does seem this way. I have mitigations and retpoline
off, so syscall switch is pretty fast (under 0.5 microsecond).

  [0] https://github.com/anakryiko/linux/tree/procfs-proc-maps-ioctl
>
> > >
> >
> > In another reply to Arnaldo on patch #2 I mentioned the idea of
> > allowing to iterate only file-backed VMAs (as it seems like what
> > symbolizers would only care about, but I might be wrong here). So I
>
> Yep, I think it's enough to get file-backed VMAs only.
>

Ok, I guess I'll keep this functionality for v2 then, it's a pretty
trivial extension to existing logic.

>
> > tried that quickly, given it's a trivial addition to my code. See
> > results below (it is slightly faster, but not much, because most of
> > VMAs in that benchmark seem to be indeed file-backed anyways), just
> > for completeness. I'm not sure if that would be useful/used by perf,
> > so please let me know.
>
> Thanks for doing this.  It'd be useful as it provides better synthesizing
> performance.  The startup latency of perf record is a problem, I need
> to take a look if it can be improved more.
>
> Thanks,
> Namhyung
>
>
> >
> > As I mentioned above, it's not radically faster in this perf
> > benchmark, because we still request about 170 VMAs (vs ~195 if we
> > iterate *all* of them), so not a big change. The ratio will vary
> > depending on what the process is doing, of course. Anyways, just for
> > completeness, I'm not sure if I have to add this "filter" to the
> > actual implementation.
> >
> > # ./perf-filebacked bench internals synthesize
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of single threaded perf event synthesis by
> > synthesizing events on the perf process itself:
> >   Average synthesis took: 65.759 usec (+- 0.063 usec)
> >   Average num. events: 30.000 (+- 0.000)
> >   Average time per event 2.192 usec
> >   Average data synthesis took: 73.840 usec (+- 0.080 usec)
> >   Average num. events: 153.000 (+- 0.000)
> >   Average time per event 0.483 usec
> >
> > # ./perf-filebacked bench internals synthesize
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of single threaded perf event synthesis by
> > synthesizing events on the perf process itself:
> >   Average synthesis took: 66.245 usec (+- 0.059 usec)
> >   Average num. events: 30.000 (+- 0.000)
> >   Average time per event 2.208 usec
> >   Average data synthesis took: 70.627 usec (+- 0.074 usec)
> >   Average num. events: 153.000 (+- 0.000)
> >   Average time per event 0.462 usec
> >
> > # ./perf-filebacked bench internals synthesize --mt -M 8
> > # Running 'internals/synthesize' benchmark:
> > Computing performance of multi threaded perf event synthesis by
> > synthesizing events on CPU 0:
> >   Number of synthesis threads: 1
> >     Average synthesis took: 33477.500 usec (+- 556.102 usec)
> >     Average num. events: 10125.700 (+- 1.620)
> >     Average time per event 3.306 usec
> >   Number of synthesis threads: 2
> >     Average synthesis took: 30473.700 usec (+- 221.933 usec)
> >     Average num. events: 10127.000 (+- 0.000)
> >     Average time per event 3.009 usec
> >   Number of synthesis threads: 3
> >     Average synthesis took: 29775.200 usec (+- 315.212 usec)
> >     Average num. events: 10128.700 (+- 0.667)
> >     Average time per event 2.940 usec
> >   Number of synthesis threads: 4
> >     Average synthesis took: 29477.100 usec (+- 621.258 usec)
> >     Average num. events: 10129.000 (+- 0.000)
> >     Average time per event 2.910 usec
> >   Number of synthesis threads: 5
> >     Average synthesis took: 29777.900 usec (+- 294.710 usec)
> >     Average num. events: 10144.700 (+- 11.597)
> >     Average time per event 2.935 usec
> >   Number of synthesis threads: 6
> >     Average synthesis took: 27774.700 usec (+- 357.569 usec)
> >     Average num. events: 10158.500 (+- 14.710)
> >     Average time per event 2.734 usec
> >   Number of synthesis threads: 7
> >     Average synthesis took: 27437.200 usec (+- 233.626 usec)
> >     Average num. events: 10135.700 (+- 2.700)
> >     Average time per event 2.707 usec
> >   Number of synthesis threads: 8
> >     Average synthesis took: 28784.600 usec (+- 477.630 usec)
> >     Average num. events: 10133.000 (+- 0.000)
> >     Average time per event 2.841 usec
> >
> > >   [0] https://github.com/anakryiko/linux/commit/0841fe675ed30f5605c5b228e18f5612ea253b35
> > >
> > > >
> > > > Thanks,
> > > > Ian
> > > >
> > > > > > Thanks,
> > > > > > Ian
> > > > > >
> > > > > > >   [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1e8e5d901cddb2b33ab1/src/normalize/user.rs#L193
> > > > > > >
> > > > > > > > thanks,
> > > > > > > >
> > > > > > > > greg k-h
> > > > > > >
> >

Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc/<pid>/maps APIs

[Arnaldo Carvalho de Melo]

On Tue, May 07, 2024 at 03:56:40PM -0700, Andrii Nakryiko wrote:
> On Tue, May 7, 2024 at 3:27 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > On Tue, May 7, 2024 at 10:29 AM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> > > In another reply to Arnaldo on patch #2 I mentioned the idea of
> > > allowing to iterate only file-backed VMAs (as it seems like what
> > > symbolizers would only care about, but I might be wrong here). So I

> > Yep, I think it's enough to get file-backed VMAs only.
 
> Ok, I guess I'll keep this functionality for v2 then, it's a pretty
> trivial extension to existing logic.

Maps for JITed code, for isntance, aren't backed by files:

commit 578c03c86fadcc6fd7319ddf41dd4d1d88aab77a
Author: Namhyung Kim <namhyung@kernel.org>
Date:   Thu Jan 16 10:49:31 2014 +0900

    perf symbols: Fix JIT symbol resolution on heap
    
    Gaurav reported that perf cannot profile JIT program if it executes the
    code on heap.  This was because current map__new() only handle JIT on
    anon mappings - extends it to handle no_dso (heap, stack) case too.
    
    This patch assumes JIT profiling only provides dynamic function symbols
    so check the mapping type to distinguish the case.  It'd provide no
    symbols for data mapping - if we need to support symbols on data
    mappings later it should be changed.
    
    Reported-by: Gaurav Jain <gjain@fb.com>
    Signed-off-by: Namhyung Kim <namhyung@kernel.org>
    Tested-by: Gaurav Jain <gjain@fb.com>

⬢[acme@toolbox perf-tools-next]$ git show 89365e6c9ad4c0e090e4c6a4b67a3ce319381d89
commit 89365e6c9ad4c0e090e4c6a4b67a3ce319381d89
Author: Andi Kleen <ak@linux.intel.com>
Date:   Wed Apr 24 17:03:02 2013 -0700

    perf tools: Handle JITed code in shared memory
    
    Need to check for /dev/zero.
    
    Most likely more strings are missing too.
    
    Signed-off-by: Andi Kleen <ak@linux.intel.com>
    Link: http://lkml.kernel.org/r/1366848182-30449-1-git-send-email-andi@firstfloor.org
    Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index 6fcb9de623401b8a..8bcdf9e54089acaf 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -21,6 +21,7 @@ const char *map_type__name[MAP__NR_TYPES] = {
 static inline int is_anon_memory(const char *filename)
 {
        return !strcmp(filename, "//anon") ||
+              !strcmp(filename, "/dev/zero (deleted)") ||
               !strcmp(filename, "/anon_hugepage (deleted)");
 }

etc.

- Arnaldo