Sampling of non-C-like stacks with eBPF and perf_events?

[Ben Gamari <ben@smart-cactus.org>]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

Hi all,

I have recently been exploring the possibility of using a
BPF_PROG_TYPE_PERF_EVENT program to implement stack sampling for
languages which do not use the platform's %sp for their stack pointer
(in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack
pointer). Specifically, the idea is to use a sampling perf_events
session with an eBPF overflow handler which locates the
currently-running thread's stack and records it in the sample ringbuffer
(see [2] for my current attempt). At this point I only care about
user-space samples.

However, I quickly ran up against the fact that perf_event's stack
sampling logic (namely perf_output_sample_ustack) is called from an IRQ
context. This appears to preclude use of a sleepable BPF program, which
would be necessary to use bpf_copy_from_user. Indeed, the fact that the
usual stack sampling logic uses copy_from_user_inatomic rather than
copy_from_user suggests that this isn't a safe context for sleeping.

So, I'm at this point a bit unclear on how to proceed. I can see a few
possible directions forward, although none are particularly enticing:

* Add a bpf_copy_from_user_atomic helper, which can be called from a
  non-sleepable context like a perf_events overflow handler. This would
  take the same set_fs() and pagefault_disable() precautions as
  perf_output_sample_ustack to ensure that the access is safe and aborts
  on fault.

* Introduce a new BPF program type,
  BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked by
  perf_output_sample_ustack to locate the stack to be sampled.

Do either of these ideas sound upstreamable? Perhaps there are other
ideas on how to attack this general problem? I do not believe Haskell is
alone in its struggle with the current inflexibility of stack sampling;
the JVM introduced framepointer support specifically to allow callgraph
sampling; however, dedicating a register and code to this seems like an
unfortunate compromise, especially on x86-64 where registers are already
fairly precious.

Any thoughts or suggestions would be greatly appreciated.

Cheers,

- Ben


[1] https://www.haskell.org/ghc/
[2] https://gitlab.haskell.org/bgamari/hs-bpf-prof/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]