Re: [HELP] failed to resolve CO-RE relocation

[Namhyung Kim <namhyung@gmail.com>]

On Wed, Apr 26, 2023 at 09:26:59PM -0700, Andrii Nakryiko wrote:
> On Wed, Apr 26, 2023 at 7:21 PM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > Hello Andrii,
> >
> > On Wed, Apr 26, 2023 at 6:19 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Wed, Apr 26, 2023 at 5:14 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I'm having a problem of loading perf lock contention BPF program [1]
> > > > on old kernels.  It has collect_lock_syms() to get the address of each
> > > > CPU's run-queue lock.  The kernel 5.14 changed the name of the field
> > > > so there's bpf_core_field_exists to check the name like below.
> > > >
> > > >         if (bpf_core_field_exists(rq_new->__lock))
> > > >                 lock_addr = (__u64)&rq_new->__lock;
> > > >         else
> > > >                 lock_addr = (__u64)&rq_old->lock;
> > >
> > > I suspect compiler rewrites it to something like
> > >
> > >    lock_addr = (__u64)&rq_old->lock;
> > >    if (bpf_core_field_exists(rq_new->__lock))
> > >         lock_addr = (__u64)&rq_new->__lock;
> > >
> > > so rq_old relocation always happens and ends up being not guarded
> > > properly. You can try adding barrier_var(rq_new) and
> > > barrier_var(rq_old) around if and inside branches, that should
> > > pessimize compiler
> > >
> > > alternatively if you do
> > >
> > > if (bpf_core_field_exists(rq_new->__lock))
> > >     lock_addr = (__u64)&rq_new->__lock;
> > > else if (bpf_core_field_exists(rq_old->lock))
> > >     lock_addr = (__u64)&rq_old->lock;
> > > else
> > >     lock_addr = 0; /* or signal error somehow */
> > >
> > > It might work as well.
> >
> > Thanks a lot for your comment!
> >
> > I've tried the below code but no luck. :(
> 
> Can you post an output of llvm-objdump -d <your.bpf.o> of the program
> (collect_lock_syms?) containing above code (or at least relevant
> portions with some buffer before/after to get a sense of what's going
> on)

Sure.

Here's the full source code:

  SEC("raw_tp/bpf_test_finish")
  int BPF_PROG(collect_lock_syms)
  {
	__u64 lock_addr;
	__u32 lock_flag;

	for (int i = 0; i < MAX_CPUS; i++) {
		struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
		struct rq___new *rq_new = (void *)rq;
		struct rq___old *rq_old = (void *)rq;

		if (rq == NULL)
			break;

		barrier_var(rq_old);
		barrier_var(rq_new);

		if (bpf_core_field_exists(rq_old->lock)) {
			barrier_var(rq_old);
			lock_addr = (__u64)&rq_old->lock;
		} else if (bpf_core_field_exists(rq_new->__lock)) {
			barrier_var(rq_new);
			lock_addr = (__u64)&rq_new->__lock;
		} else
			continue;

		lock_flag = LOCK_CLASS_RQLOCK;
		bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
	}
	return 0;
  }

And the disassembly is:

  $ llvm-objdump -d util/bpf_skel/.tmp/lock_contention.bpf.o
  ...
  Disassembly of section raw_tp/bpf_test_finish:
  
  0000000000000000 <collect_lock_syms>:
       0:	b7 06 00 00 00 00 00 00	r6 = 0
       1:	b7 07 00 00 01 00 00 00	r7 = 1
       2:	b7 09 00 00 01 00 00 00	r9 = 1
       3:	b7 08 00 00 00 00 00 00	r8 = 0

  0000000000000020 <LBB3_1>:
       4:	18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00	r1 = 0 ll
       6:	bf 62 00 00 00 00 00 00	r2 = r6
       7:	85 00 00 00 99 00 00 00	call 153
       8:	15 00 12 00 00 00 00 00	if r0 == 0 goto +18 <LBB3_8>
       9:	bf 01 00 00 00 00 00 00	r1 = r0
      10:	15 07 12 00 00 00 00 00	if r7 == 0 goto +18 <LBB3_4>
      11:	0f 81 00 00 00 00 00 00	r1 += r8

  0000000000000060 <LBB3_6>:
      12:	63 9a f4 ff 00 00 00 00	*(u32 *)(r10 - 12) = r9
      13:	7b 1a f8 ff 00 00 00 00	*(u64 *)(r10 - 8) = r1
      14:	bf a2 00 00 00 00 00 00	r2 = r10
      15:	07 02 00 00 f8 ff ff ff	r2 += -8
      16:	bf a3 00 00 00 00 00 00	r3 = r10
      17:	07 03 00 00 f4 ff ff ff	r3 += -12
      18:	18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00	r1 = 0 ll
      20:	b7 04 00 00 00 00 00 00	r4 = 0
      21:	85 00 00 00 02 00 00 00	call 2

  00000000000000b0 <LBB3_7>:
      22:	07 06 00 00 01 00 00 00	r6 += 1
      23:	bf 61 00 00 00 00 00 00	r1 = r6
      24:	67 01 00 00 20 00 00 00	r1 <<= 32
      25:	77 01 00 00 20 00 00 00	r1 >>= 32
      26:	55 01 e9 ff 00 04 00 00	if r1 != 1024 goto -23 <LBB3_1>

  00000000000000d8 <LBB3_8>:
      27:	b7 00 00 00 00 00 00 00	r0 = 0
      28:	95 00 00 00 00 00 00 00	exit

  00000000000000e8 <LBB3_4>:
      29:	b7 01 00 00 01 00 00 00	r1 = 1
      30:	15 01 f7 ff 00 00 00 00	if r1 == 0 goto -9 <LBB3_7>
      31:	b7 01 00 00 00 00 00 00	r1 = 0
      32:	0f 10 00 00 00 00 00 00	r0 += r1
      33:	bf 01 00 00 00 00 00 00	r1 = r0
      34:	05 00 e9 ff 00 00 00 00	goto -23 <LBB3_6>

The error message is like this:

  libbpf: prog 'collect_lock_syms': BPF program load failed: Invalid argument
  libbpf: prog 'collect_lock_syms': -- BEGIN PROG LOAD LOG --
  reg type unsupported for arg#0 function collect_lock_syms#380
  0: R1=ctx(off=0,imm=0) R10=fp0
  ; int BPF_PROG(collect_lock_syms)
  0: (b7) r6 = 0                        ; R6_w=0
  1: (b7) r7 = 0                        ; R7_w=0
  2: (b7) r9 = 1                        ; R9_w=1
  3: <invalid CO-RE relocation>
  failed to resolve CO-RE relocation <byte_off> [381] struct rq___old.lock (0:0 @ offset 0)
  processed 4 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
  -- END PROG LOAD LOG --
  libbpf: prog 'collect_lock_syms': failed to load: -22
  libbpf: failed to load object 'lock_contention_bpf'
  libbpf: failed to load BPF skeleton 'lock_contention_bpf': -22
  Failed to load lock-contention BPF skeleton
  lock contention BPF setup failed


Thanks,
Namhyung