Re: long BPF stack traces Re: Broken stack traces with --call-graph=fp and a multi-threaded app due to page faults?

[Maksymilian Graczyk <maksymilian.graczyk@cern.ch>]

Hi all,

Thank you for your responses.

As I said in my first e-mail, perf_event_max_stack is already set to 
1024 and it doesn't help unfortunately.

Also, as Namhyung has spotted, my problem isn't limited only to BPF, I 
get broken stacks for both on-CPU and off-CPU stuff. The page faults I 
caught in my debugging sessions actually come from on-CPU stack 
traversal and they cannot be currently (if at all) handled by perf 
because a sample is taken inside an interrupt handler.

In the meantime, I have managed to find the exact cause of my page 
faults, which is narrower than what I posted in my last update: it's 
NUMA automatic balancing, which unmaps program's pages to migrate them 
closer to a CPU running that program on the next page fault. Disabling 
it with "sysctl kernel.numa_balancing=0" seems to fix the broken stack 
issue and is a smaller overkill than turning off NUMA awareness 
completely in the kernel, but it's still only a workaround with a 
potential performance penalty (unless there is a manual way of setting 
up NUMA properly so that the performance penalty is negligible and 
callchains are not broken).

The ideal solution for me would be fixing perf so that stack unwinding 
can always be fully done without disabling or tinkering with any 
performance-enhancing mechanisms enabled by default. By "the proposed 
sframe unwinder", do you mean the set of patches suggested in this 
thread? 
https://lore.kernel.org/all/cover.1699487758.git.jpoimboe@kernel.org/

Best regards,
Maks Graczyk