Re: [PATCH bpf-next 6/7] bpftool: Add LLVM as default library for disassembling JIT-ed programs

[Quentin Monnet <quentin@isovalent.com>]

On 07/09/2022 17:10, Alexei Starovoitov wrote:
> On Wed, Sep 7, 2022 at 7:20 AM Quentin Monnet <quentin@isovalent.com> wrote:
>>
>> On 07/09/2022 00:46, Alexei Starovoitov wrote:
>>> On Tue, Sep 6, 2022 at 6:36 AM Quentin Monnet <quentin@isovalent.com> wrote:
>>>>
>>>> Naturally, the display of disassembled instructions comes with a few
>>>> minor differences. Here is a sample output with libbfd (already
>>>> supported before this patch):
>>>>
>>>>     # bpftool prog dump jited id 56
>>>>     bpf_prog_6deef7357e7b4530:
>>>>        0:   nopl   0x0(%rax,%rax,1)
>>>>        5:   xchg   %ax,%ax
>>>>        7:   push   %rbp
>>>>        8:   mov    %rsp,%rbp
>>>>        b:   push   %rbx
>>>>        c:   push   %r13
>>>>        e:   push   %r14
>>>>       10:   mov    %rdi,%rbx
>>>>       13:   movzwq 0xb0(%rbx),%r13
>>>>       1b:   xor    %r14d,%r14d
>>>>       1e:   or     $0x2,%r14d
>>>>       22:   mov    $0x1,%eax
>>>>       27:   cmp    $0x2,%r14
>>>>       2b:   jne    0x000000000000002f
>>>>       2d:   xor    %eax,%eax
>>>>       2f:   pop    %r14
>>>>       31:   pop    %r13
>>>>       33:   pop    %rbx
>>>>       34:   leave
>>>>       35:   ret
>>>>       36:   int3
>>>>
>>>> LLVM supports several variants that we could set when initialising the
>>>> disassembler, for example with:
>>>>
>>>>     LLVMSetDisasmOptions(*ctx,
>>>>                          LLVMDisassembler_Option_AsmPrinterVariant);
>>>>
>>>> but the default printer is kept for now. Here is the output with LLVM:
>>>>
>>>>     # bpftool prog dump jited id 56
>>>>     bpf_prog_6deef7357e7b4530:
>>>>        0:   nopl    (%rax,%rax)
>>>>        5:   nop
>>>>        7:   pushq   %rbp
>>>>        8:   movq    %rsp, %rbp
>>>>        b:   pushq   %rbx
>>>>        c:   pushq   %r13
>>>>        e:   pushq   %r14
>>>>       10:   movq    %rdi, %rbx
>>>>       13:   movzwq  176(%rbx), %r13
>>>>       1b:   xorl    %r14d, %r14d
>>>>       1e:   orl     $2, %r14d
>>>>       22:   movl    $1, %eax
>>>>       27:   cmpq    $2, %r14
>>>>       2b:   jne     2
>>>>       2d:   xorl    %eax, %eax
>>>>       2f:   popq    %r14
>>>
>>> If I'm reading the asm correctly the difference is significant.
>>> jne 0x2f was an absolute address and jmps were easy
>>> to follow.
>>> While in llvm disasm it's 'jne 2' ?! What is 2 ?
>>> 2 bytes from the next insn of 0x2d ?
>>
>> Yes, that's it. Apparently, this is how the operand is encoded, and
>> libbfd does the translation to the absolute address:
>>
>>     # bpftool prog dump jited id 7868 opcodes
>>     [...]
>>       2b:   jne    0x000000000000002f
>>             75 02
>>     [...]
>>
>> The same difference is observable between objdump and llvm-objdump on an
>> x86-64 binary for example, although they usually have labels to refer to
>> ("jne     -22 <_obstack_memory_used+0x7d0>"), making the navigation
>> easier. The only mention I could find of that difference is a report
>> from 2013 [0].
>>
>> [0] https://discourse.llvm.org/t/llvm-objdump-disassembling-jmp/29584/2
>>
>>> That is super hard to read.
>>> Is there a way to tune/configure llvm disasm?
>>
>> There's a function and some options to tune it, but I tried them and
>> none applies to converting the jump operands.
>>
>>     int LLVMSetDisasmOptions(LLVMDisasmContextRef DC, uint64_t Options);
>>
>>     /* The option to produce marked up assembly. */
>>     #define LLVMDisassembler_Option_UseMarkup 1
>>     /* The option to print immediates as hex. */
>>     #define LLVMDisassembler_Option_PrintImmHex 2
>>     /* The option use the other assembler printer variant */
>>     #define LLVMDisassembler_Option_AsmPrinterVariant 4
>>     /* The option to set comment on instructions */
>>     #define LLVMDisassembler_Option_SetInstrComments 8
>>     /* The option to print latency information alongside instructions */
>>     #define LLVMDisassembler_Option_PrintLatency 16
>>
>> I found that LLVMDisassembler_Option_AsmPrinterVariant read better,
>> although in my patch I kept the default output which looked closer to
>> the existing from libbfd. Here's what the option produces:
>>
>>     bpf_prog_6deef7357e7b4530:
>>        0:   nop     dword ptr [rax + rax]
>>        5:   nop
>>        7:   push    rbp
>>        8:   mov     rbp, rsp
>>        b:   push    rbx
>>        c:   push    r13
>>        e:   push    r14
>>       10:   mov     rbx, rdi
>>       13:   movzx   r13, word ptr [rbx + 180]
>>       1b:   xor     r14d, r14d
>>       1e:   or      r14d, 2
>>       22:   mov     eax, 1
>>       27:   cmp     r14, 2
>>       2b:   jne     2
>>       2d:   xor     eax, eax
>>       2f:   pop     r14
>>       31:   pop     r13
>>       33:   pop     rbx
>>       34:   leave
>>       35:   re
>>
>> But the jne operand remains a '2'. I'm not aware of any option to change
>> it in LLVM's disassembler :(.
> 
> Hmm. llvm-objdump -d test_maps
> looks fine:
>   41bfcb: e8 6f f7 ff ff                   callq    0x41b73f
> <find_extern_btf_id>
> 
> the must be something llvm disasm is missing when you feed raw bytes
> into it.
> Please keep investigating. In this form I'm afraid it's no go.

OK, I'll keep looking