Re: GCC and binutils support for BPF V4 instructions

["Jose E. Marchesi" <jose.marchesi@oracle.com>]

> On 7/29/23 9:54 PM, Jose E. Marchesi wrote:
>> 
>>> On Sat, Jul 29, 2023 at 1:29 AM Jose E. Marchesi
>>> <jose.marchesi@oracle.com> wrote:
>>>>
>>>>
>>>>> On Fri, Jul 28, 2023 at 11:01 AM Jose E. Marchesi
>>>>> <jose.marchesi@oracle.com> wrote:
>>>>>>
>>>>>>
>>>>>>>> On 7/28/23 9:41 AM, Jose E. Marchesi wrote:
>>>>>>>>> Hello.
>>>>>>>>> Just a heads up regarding the new BPF V4 instructions and their
>>>>>>>>> support
>>>>>>>>> in the GNU Toolchain.
>>>>>>>>> V4 sdiv/smod instructions
>>>>>>>>>     Binutils has been updated to use the V4 encoding of these
>>>>>>>>>     instructions, which used to be part of the xbpf testing dialect used
>>>>>>>>>     in GCC.  GCC generates these instructions for signed division when
>>>>>>>>>     -mcpu=v4 or higher.
>>>>>>>>> V4 sign-extending register move instructions
>>>>>>>>> V4 signed load instructions
>>>>>>>>> V4 byte swap instructions
>>>>>>>>>     Supported in assembler, disassembler and linker.  GCC generates
>>>>>>>>> these
>>>>>>>>>     instructions when -mcpu=v4 or higher.
>>>>>>>>> V4 32-bit unconditional jump instruction
>>>>>>>>>     Supported in assembler and disassembler.  GCC doesn't generate
>>>>>>>>> that
>>>>>>>>>     instruction.
>>>>>>>>>     However, the assembler has been expanded in order to perform the
>>>>>>>>>     following relaxations when the disp16 field of a jump instruction is
>>>>>>>>>     known at assembly time, and is overflown, unless -mno-relax is
>>>>>>>>>     specified:
>>>>>>>>>       JA disp16  -> JAL disp32
>>>>>>>>>       Jxx disp16 -> Jxx +1; JA +1; JAL disp32
>>>>>>>>>     Where Jxx is one of the conditional jump instructions such as
>>>>>>>>> jeq,
>>>>>>>>>     jlt, etc.
>>>>>>>>
>>>>>>>> Sounds great. The above 'JA/Jxx disp16' transformation matches
>>>>>>>> what llvm did as well.
>>>>>>>
>>>>>>> Not by chance ;)
>>>>>>>
>>>>>>> Now what is pending in binutils is to relax these jumps in the linker as
>>>>>>> well.  But it is very low priority, compared to get these kernel
>>>>>>> selftests building and running.  So it will happen, but probably not
>>>>>>> anytime soon.
>>>>>>
>>>>>> By the way, for doing things like that (further object transformations
>>>>>> by linkers and the like) we will need to have the ELF files annotated
>>>>>> with:
>>>>>>
>>>>>> - The BPF cpu version the object was compiled for: v1, v2, v3, v4, and
>>>>>>
>>>>>> - Individual flags specifying the BPF cpu capabilities (alu32, bswap,
>>>>>>    jmp32, etc) required/expected by the code in the object.
>>>>>>
>>>>>> Note it is interesting to being able to denote both, for flexibility.
>>>>>>
>>>>>> There are 32 bits available for machine-specific flags in e_flags, which
>>>>>> are commonly used for this purpose by other arches.  For BPF I would
>>>>>> suggest something like:
>>>>>>
>>>>>> #define EF_BPF_ALU32  0x00000001
>>>>>> #define EF_BPF_JMP32  0x00000002
>>>>>> #define EF_BPF_BSWAP  0x00000004
>>>>>> #define EF_BPF_SDIV   0x00000008
>>>>>> #define EF_BPF_CPUVER 0x00FF0000
>>>>>
>>>>> Interesting idea. I don't mind, but what are we going to do with this info?
>>>>> I cannot think of anything useful libbpf could do with it.
>>>>> For other archs such flags make sense, since disasm of everything
>>>>> to discover properties is hard. For BPF we will parse all insns anyway,
>>>>> so additional info in ELF doesn't give any additional insight.
>>>>
>>>> I mainly had link-time relaxation in mind.  The linker needs to know
>>>> what instructions are available (JMP32 or not) in order to decide what
>>>> to relax, and to what.
>>>
>>> But the assembler has little choice when the jump target is >16bits.
>>> It can use jmp32 or error.
>> When the assembler sees a jump instruction:
>>     goto EXPR
>> there are several possibilities:
>> 1. EXPR consists on a literal number like 1, -10 or 0xff, or an
>>     expression that can be resolved during the first assembler pass (like
>>     8 * 64).  The numerical result is interpreted as number of 64-bit
>>     words minus one.  In this case, the assembler can immediately decide
>>     whether the operand is >16 bits, relaxing to the jmp32 jump if cpu >=
>>     v4 and unless -mno-relax is passed in the command line.
>> 2. EXPR is a symbolic expression involving a symbol that can be
>> resolved
>>     during the second assembler pass.  For example, `foo + 10'.  In this
>>     case, there are two possibilities:
>>     2.1. The symbol is an absolute symbol.  In this case the value
>> is
>>          interpreted as-such and no conversion is done by the assembler.
>>          So if for example the user invokes the assembler passing
>>          `--defsym foo=10', the assembled instruction is `ja 20'.
>>     2.2. The symbol is a PC-relative or section-relative symbol.  In
>> this
>>          case the value is interpreted as a byte offset (the assembler
>>          takes care to transform offsets relative to the current section
>>          into PC-relative offsets whenever necessary).  This is the case
>>          of labels.  For these symbols, the BPF assembler converts the
>>          value from bytes to number of 64-bit words minus one.  So for
>>          example for `ja done' where `done' has the value 256 bytes, the
>>          assembled instruction is `ja 31'.
>> 3. EXPR is a symbolic expression involving a symbol that cannot be
>>     resolved during the second assembler pass.  In this case, a
>>     relocation for the 16-bit immediate field in the instruction is
>>     generated in the assembled object.  There is no R_BPF_64_16
>>     relocation defined by BPF as of yet, so we are using
>>     R_BPF_GNU_64_16=256, which as we agreed uses a high relocation number
>>     to avoid collisions.  Since gas is a standalone assembler, it seems
>>     sensible to emit a relocation rather than erroing out in these
>>     situations.  ld knows how to handle these relocs when linking BPF
>>     objects together.
>> 
>>> I guess you're proposing to encode this e_flags in the text of asm ?
>>> Special asm directive that will force asm to error or use jmp32?
>> GAS uses command-line options for that.
>> When GCC is invoked with -mcpu=v3, for example, it passes the
>> corresponding option to the assembler so it expects a BPF V3 assembly
>> program. In that scenario, if the user does a jump to an address that is
>>> 16bit in an inline asm, the assembler will error out,
>> because relaxing to jmp32 is not a possibility in V3.  Ditto for
>> compiler options like -msdiv or -mjmp32, that both clang and GCC
>> support.
>> I don't know how clang configures its integrated assembler... I
>> guess by
>> calling some function.  But it is the same principle: if you tell clang
>> to generate v3 bpf and you include a header that uses a v4 instruction
>> (or overflown jump that would require relaxation) in inline asm, you
>> want an error.
>
> If -mcpu=<version> is specified in the clang command line,
> then the cpu <version> will be encoded in IR and will be
> passed to the integrated assembler. And if you specify
> -mcpu=v3 in the command line and your code has
> cpu v4 inline assembly code, the compiler will error out.

Perfect :)
Thanks for the confirmation.

>> 
>>>> Also as you mention the disassembler can look in the object to determine
>>>> which instructions shall be recognized and with insructions shall be
>>>> reported as <unknown>.  Right now it is necessary to pass an explicit
>>>> option to the assembler, and the default is v4.
>>>
>>> Disambiguating between unknown and exact insn kinda makes sense for disasm.
>>> For assembler it's kinda weird. If text says 'sdiv' the asm should emit
>>> binary code for it regardless of asm directive.
>> Unless configured to not do so?  See above.
>> 
>>> It seems e_flags can only be emitted by assembler.
>>> Like if it needs to use jmp32 it will add EF_BPF_JMP32.
>> Yep.
>> 
>>> Still feels that we can live without these flags, but not a bad
>>> addition.
>> The individual flags... I am not sure, other arches have them, but
>> maybe
>> having them in BPF doesn't make much sense and it is not worth the extra
>> complication and wasted bits in e_flags.  How realistic is to expect
>> that some kernel may support a particular version of the BPF ISA, and
>> also have support for some particular instruction from a later ISA as
>> the result of a backport or something?  Not for me to judge... I was
>> already bitten by my utter ignorance on kernel business when I added
>> that silly useless -mkernel=VERSION option to GCC 8-)
>> What I am pretty sure is that we will need something like
>> EF_BPF_CPUVER
>> if we are ever gonna support relaxation in any linker external to
>> libbpf, and also to detect (and error/warn) when several objects with
>> different BPF versions are linked together.
>> 
>>> As far as flag names, let's use EF_ prefix. I think it's more canonical.
>>> And single 0xF is probably enough for cpu ver.
>> Agreed.