Re: [PATCH bpf-next] tools/resolve_btfids: fix cross-compilation to non-host endianness

[Viktor Malik <vmalik@redhat.com>]

On 1/28/24 21:17, Jiri Olsa wrote:
> On Fri, Jan 26, 2024 at 03:40:11PM -0800, Andrii Nakryiko wrote:
>> On Tue, Jan 23, 2024 at 4:08 AM Viktor Malik <vmalik@redhat.com> wrote:
>>>
>>> The .BTF_ids section is pre-filled with zeroed BTF ID entries during the
>>> build and afterwards patched by resolve_btfids with correct values.
>>> Since resolve_btfids always writes in host-native endianness, it relies
>>> on libelf to do the translation when the target ELF is cross-compiled to
>>> a different endianness (this was introduced in commit 61e8aeda9398
>>> ("bpf: Fix libelf endian handling in resolv_btfids")).
>>>
>>> Unfortunately, the translation will corrupt the flags fields of SET8
>>> entries because these were written during vmlinux compilation and are in
>>> the correct endianness already. This will lead to numerous selftests
>>> failures such as:
>>>
>>>     $ sudo ./test_verifier 502 502
>>>     #502/p sleepable fentry accept FAIL
>>>     Failed to load prog 'Invalid argument'!
>>>     bpf_fentry_test1 is not sleepable
>>>     verification time 34 usec
>>>     stack depth 0
>>>     processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
>>>     Summary: 0 PASSED, 0 SKIPPED, 1 FAILED
> 
> hum, I'd think we should have hit such bug long time ago.. set8 is
> there for some time already.. nice ;-)
> 
>>>
>>> Since it's not possible to instruct libelf to translate just certain
>>> values, let's manually bswap the flags in resolve_btfids when needed, so
>>> that libelf then translates everything correctly.
>>>
>>> Fixes: ef2c6f370a63 ("tools/resolve_btfids: Add support for 8-byte BTF sets")
>>> Signed-off-by: Viktor Malik <vmalik@redhat.com>
>>> ---
>>>  tools/bpf/resolve_btfids/main.c | 35 +++++++++++++++++++++++++++++++--
>>>  1 file changed, 33 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c
>>> index 27a23196d58e..440d3d066ce4 100644
>>> --- a/tools/bpf/resolve_btfids/main.c
>>> +++ b/tools/bpf/resolve_btfids/main.c
>>> @@ -646,18 +646,31 @@ static int cmp_id(const void *pa, const void *pb)
>>>         return *a - *b;
>>>  }
>>>
>>> +static int need_bswap(int elf_byte_order)
>>> +{
>>> +       return __BYTE_ORDER == __LITTLE_ENDIAN && elf_byte_order != ELFDATA2LSB ||
>>> +              __BYTE_ORDER == __BIG_ENDIAN && elf_byte_order != ELFDATA2MSB;
>>
>> return (__BYTE_ORDER == __LITTLE_ENDIAN) != (elf_byte_order == ELFDATA2LSB);
>>
>> ?
>>

It seemed to me a bit less readable this way, but sure, no problem with
this form either.

>>> +}
>>> +
>>>  static int sets_patch(struct object *obj)
>>>  {
>>>         Elf_Data *data = obj->efile.idlist;
>>>         int *ptr = data->d_buf;
>>>         struct rb_node *next;
>>> +       GElf_Ehdr ehdr;
>>> +
>>> +       if (gelf_getehdr(obj->efile.elf, &ehdr) == NULL) {
>>> +               pr_err("FAILED cannot get ELF header: %s\n",
>>> +                       elf_errmsg(-1));
>>> +               return -1;
>>> +       }
>>
>> calculate needs_bswap() once here?

Good idea, will do.

>>>
>>>         next = rb_first(&obj->sets);
>>>         while (next) {
>>> -               unsigned long addr, idx;
>>> +               unsigned long addr, idx, flags;
>>>                 struct btf_id *id;
>>>                 int *base;
>>> -               int cnt;
>>> +               int cnt, i;
>>>
>>>                 id   = rb_entry(next, struct btf_id, rb_node);
>>>                 addr = id->addr[0];
>>> @@ -679,6 +692,24 @@ static int sets_patch(struct object *obj)
>>>
>>>                 qsort(base, cnt, id->is_set8 ? sizeof(uint64_t) : sizeof(int), cmp_id);
>>>
>>> +               /*
>>> +                * When ELF endianness does not match endianness of the host,
>>> +                * libelf will do the translation when updating the ELF. This,
>>> +                * however, corrupts SET8 flags which are already in the target
>>> +                * endianness. So, let's bswap them to the host endianness and
>>> +                * libelf will then correctly translate everything.
>>> +                */
>>> +               if (id->is_set8 && need_bswap(ehdr.e_ident[EI_DATA])) {
>>> +                       for (i = 0; i < cnt; i++) {
>>> +                               /*
>>> +                                * header and entries are 8-byte, flags is the
>>> +                                * second half of an entry
>>> +                                */
>>> +                               flags = idx + (i + 1) * 2 + 1;
>>> +                               ptr[flags] = bswap_32(ptr[flags]);
>>
>> we are dealing with struct btf_id_set8, right? Can't we #include
>> include/linux/btf_ids.h and use that type for all these offset
>> calculations?..
> 
> we could, there's tools/include/linux/btf_ids.h, which we could include
> in here, we do that in selftests.. but it needs to be updated with latest
> kernel updates (at least with set8 struct)
> 
>>
>> I have the same question for existing code, tbh, so maybe there was
>> some good reason, not sure...
> 
> I think the test came later and I did not think of it for the resolve_btfids
> itself, I guess it might make the code more readable

Agreed, let's use that. I'll also refactor the existing code using types
from btf_ids.h for v2 of this patchset.

Viktor

> 
> thanks,
> jirka
> 
>>
>>> +                       }
>>> +               }
>>> +
>>>                 next = rb_next(next);
>>>         }
>>>         return 0;
>>> --
>>> 2.43.0
>>>
>>>
>