Re: [RFC] bpf: lbr: enable reading LBR from tracing bpf programs

[Song Liu <songliubraving@fb.com>]

Hi Peter,

Thanks for these helpful information and insights!

> On Aug 19, 2021, at 4:57 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> 
> On Wed, Aug 18, 2021 at 04:46:32PM +0000, Song Liu wrote:
> 
>>> Urgghhh.. I so really hate BPF specials like this.
>> 
>> I don't really like this design either. But it does show that LBR can be
>> very useful in non-PMI scenario. 
>> 
>>> Also, the PMI race
>>> you describe is because you're doing abysmal layer violations. If you'd
>>> have used perf_pmu_disable() that wouldn't have been a problem.
>> 
>> Do you mean instead of disable/enable lbr, we disable/enable the whole 
>> pmu? 
> 
> Yep, that way you're serialized against PMIs. It's what all of the perf
> core does.
> 
>>> I'd much rather see a generic 'fake/inject' PMI facility, something that
>>> works across the board and isn't tied to x86/intel.
>> 
>> How would that work? Do we have a function to trigger PMI from software, 
>> and then gather the LBR data after the PMI? This does sound like a much
>> cleaner solution. Where can I find code examples that fake/inject PMI?
> 
> We don't yet have anything like it; but it would look a little like:
> 
> void perf_inject_event(struct perf_event *event, struct pt_regs *regs)
> {
> 	struct perf_sample_data data;
> 	struct pmu *pmu = event->pmu;
> 	unsigned long flags;
> 
> 	local_irq_save(flags);
> 	perf_pmu_disable(pmu);
> 
> 	perf_sample_data_init(&data, 0, 0);
> 	/*
> 	 * XXX or a variant with more _ that starts at the overflow
> 	 * handler...
> 	 */
> 	__perf_event_overflow(event, 0, &data, regs);
> 
> 	perf_pmu_enable(pmu);
> 	local_irq_restore(flags);
> }
> 
> But please consider carefully, I haven't...

Hmm... This is a little weird to me. 
IIUC, we need to call perf_inject_event() after the software event, say
a kretprobe, triggers. So it gonna look like:

  1. kretprobe trigger;
  2. handler calls perf_inject_event();
  3. PMI kicks in, and saves LBR;
  4. after the PMI, consumer of LBR uses the saved data;

However, given perf_inject_event() disables PMU, we can just save the LBR
right there? And it should be a lot easier? Something like:

  1. kretprobe triggers;
  2. handler calls perf_snapshot_lbr();
     2.1 perf_pmu_disable(pmu);
     2.2 saves LBR 
     2.3 perf_pmu_enable(pmu);
  3. consumer of LBR uses the saved data;

What is the downside of this approach? 

> 
>> There is another limitation right now: we need to enable LBR with a 
>> hardware perf event (cycles, etc.). However, unless we use the event for 
>> something else, it wastes a hardware counter. So I was thinking to allow
>> software event, i.e. dummy event, to enable LBR. Does this idea sound 
>> sane to you?
> 
> We have a VLBR dummy event, but I'm not sure it does exactly as you
> want. However, we should also consider Power, which also has the branch
> stack feature.

VLBR event does look similar to the use case we have. I will take a closer
look. Thanks for the pointer!

> 
> You can't really make a software event with LBR on, because then it
> wouldn't be a software event anymore. You'll need some hybrid like
> thing, which will be yuck and I suspect it needs arch support one way or
> the other :/

Yeah, I guess it could be a "LBR only hardware event". All it needs to do 
is to keep LBR enabled (lbr_users++). I will try to keep the arch support
clean. 

Thanks,
Song