Sampling of non-C-like stacks with eBPF and perf_events?

Sampling of non-C-like stacks with eBPF and perf_events?

[Ben Gamari]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

Hi all,

I have recently been exploring the possibility of using a
BPF_PROG_TYPE_PERF_EVENT program to implement stack sampling for
languages which do not use the platform's %sp for their stack pointer
(in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack
pointer). Specifically, the idea is to use a sampling perf_events
session with an eBPF overflow handler which locates the
currently-running thread's stack and records it in the sample ringbuffer
(see [2] for my current attempt). At this point I only care about
user-space samples.

However, I quickly ran up against the fact that perf_event's stack
sampling logic (namely perf_output_sample_ustack) is called from an IRQ
context. This appears to preclude use of a sleepable BPF program, which
would be necessary to use bpf_copy_from_user. Indeed, the fact that the
usual stack sampling logic uses copy_from_user_inatomic rather than
copy_from_user suggests that this isn't a safe context for sleeping.

So, I'm at this point a bit unclear on how to proceed. I can see a few
possible directions forward, although none are particularly enticing:

* Add a bpf_copy_from_user_atomic helper, which can be called from a
  non-sleepable context like a perf_events overflow handler. This would
  take the same set_fs() and pagefault_disable() precautions as
  perf_output_sample_ustack to ensure that the access is safe and aborts
  on fault.

* Introduce a new BPF program type,
  BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked by
  perf_output_sample_ustack to locate the stack to be sampled.

Do either of these ideas sound upstreamable? Perhaps there are other
ideas on how to attack this general problem? I do not believe Haskell is
alone in its struggle with the current inflexibility of stack sampling;
the JVM introduced framepointer support specifically to allow callgraph
sampling; however, dedicating a register and code to this seems like an
unfortunate compromise, especially on x86-64 where registers are already
fairly precious.

Any thoughts or suggestions would be greatly appreciated.

Cheers,

- Ben


[1] https://www.haskell.org/ghc/
[2] https://gitlab.haskell.org/bgamari/hs-bpf-prof/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Sampling of non-C-like stacks with eBPF and perf_events?

[Alexei Starovoitov]

On Fri, Dec 17, 2021 at 1:53 PM Ben Gamari <ben@smart-cactus.org> wrote:
>
> Hi all,
>
> I have recently been exploring the possibility of using a
> BPF_PROG_TYPE_PERF_EVENT program to implement stack sampling for
> languages which do not use the platform's %sp for their stack pointer
> (in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack
> pointer). Specifically, the idea is to use a sampling perf_events
> session with an eBPF overflow handler which locates the
> currently-running thread's stack and records it in the sample ringbuffer
> (see [2] for my current attempt). At this point I only care about
> user-space samples.
>
> However, I quickly ran up against the fact that perf_event's stack
> sampling logic (namely perf_output_sample_ustack) is called from an IRQ
> context. This appears to preclude use of a sleepable BPF program, which
> would be necessary to use bpf_copy_from_user. Indeed, the fact that the
> usual stack sampling logic uses copy_from_user_inatomic rather than
> copy_from_user suggests that this isn't a safe context for sleeping.
>
> So, I'm at this point a bit unclear on how to proceed. I can see a few
> possible directions forward, although none are particularly enticing:
>
> * Add a bpf_copy_from_user_atomic helper, which can be called from a
>   non-sleepable context like a perf_events overflow handler. This would
>   take the same set_fs() and pagefault_disable() precautions as
>   perf_output_sample_ustack to ensure that the access is safe and aborts
>   on fault.
>
> * Introduce a new BPF program type,
>   BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked by
>   perf_output_sample_ustack to locate the stack to be sampled.
>
> Do either of these ideas sound upstreamable? Perhaps there are other
> ideas on how to attack this general problem? I do not believe Haskell is
> alone in its struggle with the current inflexibility of stack sampling;
> the JVM introduced framepointer support specifically to allow callgraph
> sampling; however, dedicating a register and code to this seems like an
> unfortunate compromise, especially on x86-64 where registers are already
> fairly precious.
>
> Any thoughts or suggestions would be greatly appreciated.

Hi Ben,

if you're sampling the stack trace of the current process
there is no need for copy_from_user and sleepable.
The memory with the stack trace unlikely was paged out.
So normal bpf_probe_read_user() will work fine.

This approach was used to implement 'pyperf'.
It walks python stack traces:
https://github.com/iovisor/bcc/tree/master/examples/cpp/pyperf
What you're trying to do for haskel sounds very similar.

Re: Sampling of non-C-like stacks with eBPF and perf_events?

[Ian Rogers]

Hi Alexei and Ben,

This sounds awesome! Somewhat off-topic, I wonder if we could include
the pyperf and ghc support in regular perf? I think there is an
assumption that these languages are a minority concern, but I think
everyone would benefit from being packaged with perf, being kept in
sync with how the APIs evolve, code reuse, etc.

Thanks,
Ian

On Fri, Jan 21, 2022 at 4:05 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Fri, Dec 17, 2021 at 1:53 PM Ben Gamari <ben@smart-cactus.org> wrote:
> >
> > Hi all,
> >
> > I have recently been exploring the possibility of using a
> > BPF_PROG_TYPE_PERF_EVENT program to implement stack sampling for
> > languages which do not use the platform's %sp for their stack pointer
> > (in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack
> > pointer). Specifically, the idea is to use a sampling perf_events
> > session with an eBPF overflow handler which locates the
> > currently-running thread's stack and records it in the sample ringbuffer
> > (see [2] for my current attempt). At this point I only care about
> > user-space samples.
> >
> > However, I quickly ran up against the fact that perf_event's stack
> > sampling logic (namely perf_output_sample_ustack) is called from an IRQ
> > context. This appears to preclude use of a sleepable BPF program, which
> > would be necessary to use bpf_copy_from_user. Indeed, the fact that the
> > usual stack sampling logic uses copy_from_user_inatomic rather than
> > copy_from_user suggests that this isn't a safe context for sleeping.
> >
> > So, I'm at this point a bit unclear on how to proceed. I can see a few
> > possible directions forward, although none are particularly enticing:
> >
> > * Add a bpf_copy_from_user_atomic helper, which can be called from a
> >   non-sleepable context like a perf_events overflow handler. This would
> >   take the same set_fs() and pagefault_disable() precautions as
> >   perf_output_sample_ustack to ensure that the access is safe and aborts
> >   on fault.
> >
> > * Introduce a new BPF program type,
> >   BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked by
> >   perf_output_sample_ustack to locate the stack to be sampled.
> >
> > Do either of these ideas sound upstreamable? Perhaps there are other
> > ideas on how to attack this general problem? I do not believe Haskell is
> > alone in its struggle with the current inflexibility of stack sampling;
> > the JVM introduced framepointer support specifically to allow callgraph
> > sampling; however, dedicating a register and code to this seems like an
> > unfortunate compromise, especially on x86-64 where registers are already
> > fairly precious.
> >
> > Any thoughts or suggestions would be greatly appreciated.
>
> Hi Ben,
>
> if you're sampling the stack trace of the current process
> there is no need for copy_from_user and sleepable.
> The memory with the stack trace unlikely was paged out.
> So normal bpf_probe_read_user() will work fine.
>
> This approach was used to implement 'pyperf'.
> It walks python stack traces:
> https://github.com/iovisor/bcc/tree/master/examples/cpp/pyperf
> What you're trying to do for haskel sounds very similar.

Re: Sampling of non-C-like stacks with eBPF and perf_events?

[Gabriele]

Just FYI I have plans to make a BPF variant of Austin
(https://github.com/P403n1x87/austin) as an alternative to the
ptrace-based austinp variant
(https://github.com/P403n1x87/austin#native-frame-stack). This will
then do what pyperf does, plus all of Austin's other features, like GC
sampling etc...

On Mon, 24 Jan 2022 at 12:50, Ian Rogers <irogers@google.com> wrote:
>
> Hi Alexei and Ben,
>
> This sounds awesome! Somewhat off-topic, I wonder if we could include
> the pyperf and ghc support in regular perf? I think there is an
> assumption that these languages are a minority concern, but I think
> everyone would benefit from being packaged with perf, being kept in
> sync with how the APIs evolve, code reuse, etc.
>
> Thanks,
> Ian
>
> On Fri, Jan 21, 2022 at 4:05 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Fri, Dec 17, 2021 at 1:53 PM Ben Gamari <ben@smart-cactus.org> wrote:
> > >
> > > Hi all,
> > >
> > > I have recently been exploring the possibility of using a
> > > BPF_PROG_TYPE_PERF_EVENT program to implement stack sampling for
> > > languages which do not use the platform's %sp for their stack pointer
> > > (in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack
> > > pointer). Specifically, the idea is to use a sampling perf_events
> > > session with an eBPF overflow handler which locates the
> > > currently-running thread's stack and records it in the sample ringbuffer
> > > (see [2] for my current attempt). At this point I only care about
> > > user-space samples.
> > >
> > > However, I quickly ran up against the fact that perf_event's stack
> > > sampling logic (namely perf_output_sample_ustack) is called from an IRQ
> > > context. This appears to preclude use of a sleepable BPF program, which
> > > would be necessary to use bpf_copy_from_user. Indeed, the fact that the
> > > usual stack sampling logic uses copy_from_user_inatomic rather than
> > > copy_from_user suggests that this isn't a safe context for sleeping.
> > >
> > > So, I'm at this point a bit unclear on how to proceed. I can see a few
> > > possible directions forward, although none are particularly enticing:
> > >
> > > * Add a bpf_copy_from_user_atomic helper, which can be called from a
> > >   non-sleepable context like a perf_events overflow handler. This would
> > >   take the same set_fs() and pagefault_disable() precautions as
> > >   perf_output_sample_ustack to ensure that the access is safe and aborts
> > >   on fault.
> > >
> > > * Introduce a new BPF program type,
> > >   BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked by
> > >   perf_output_sample_ustack to locate the stack to be sampled.
> > >
> > > Do either of these ideas sound upstreamable? Perhaps there are other
> > > ideas on how to attack this general problem? I do not believe Haskell is
> > > alone in its struggle with the current inflexibility of stack sampling;
> > > the JVM introduced framepointer support specifically to allow callgraph
> > > sampling; however, dedicating a register and code to this seems like an
> > > unfortunate compromise, especially on x86-64 where registers are already
> > > fairly precious.
> > >
> > > Any thoughts or suggestions would be greatly appreciated.
> >
> > Hi Ben,
> >
> > if you're sampling the stack trace of the current process
> > there is no need for copy_from_user and sleepable.
> > The memory with the stack trace unlikely was paged out.
> > So normal bpf_probe_read_user() will work fine.
> >
> > This approach was used to implement 'pyperf'.
> > It walks python stack traces:
> > https://github.com/iovisor/bcc/tree/master/examples/cpp/pyperf
> > What you're trying to do for haskel sounds very similar.



-- 
"Egli è scritto in lingua matematica, e i caratteri son triangoli,
cerchi, ed altre figure
geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola;
senza questi è un aggirarsi vanamente per un oscuro laberinto."

-- G. Galilei, Il saggiatore.