[PATCH bpf-next 0/4] bpf: arm64: Indirect jumps

[PATCH bpf-next 0/4] bpf: arm64: Indirect jumps

[Puranjay Mohan]

This set adds the support of indirect jumps to the arm64 JIT. It
involves calling bpf_prog_update_insn_ptrs() to support instructions
array map. The second piece is supporting BPF_JMP|BPF_X|BPF_JA, SRC=0,
DST=Rx, off=0, imm=0 instruction that is trivial to implement on arm64.

When running the selftests after doing the above changes, I found that
on arm64 builds of llvm, a relocation section was being generated for
.jumptables sections and it was making libbpf fail like:

libbpf: relocation against STT_SECTION in non-exec section is not supported!
Error: failed to link 'tools/testing/selftests/bpf/cpuv4/bpf_gotox.bpf.o': Invalid argument (22)

Which is due to:

Relocation section '.rel.jumptables' at offset 0x5b50 contains 263 entries:
    Offset             Info             Type               Symbol's Value  Symbol's Name
0000000000000000  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall
0000000000000008  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall
0000000000000010  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall

This rel section is not generated by x86 builds of LLVM. The third patch
of this set makes libbpf ignore relocation sections for .jumptables.

The final patch enables selftests on arm64:

 [root@localhost bpf]# ./test_progs-cpuv4 -a "*gotox*"
 #20/1    bpf_gotox/one-switch:OK
 #20/2    bpf_gotox/one-switch-non-zero-sec-offset:OK
 #20/3    bpf_gotox/two-switches:OK
 #20/4    bpf_gotox/big-jump-table:OK
 #20/5    bpf_gotox/static-global:OK
 #20/6    bpf_gotox/nonstatic-global:OK
 #20/7    bpf_gotox/other-sec:OK
 #20/8    bpf_gotox/static-global-other-sec:OK
 #20/9    bpf_gotox/nonstatic-global-other-sec:OK
 #20/10   bpf_gotox/one-jump-two-maps:OK
 #20/11   bpf_gotox/one-map-two-jumps:OK
 #20      bpf_gotox:OK
 #537/1   verifier_gotox/jump_table_ok:OK
 #537/2   verifier_gotox/jump_table_reserved_field_src_reg:OK
 #537/3   verifier_gotox/jump_table_reserved_field_non_zero_off:OK
 #537/4   verifier_gotox/jump_table_reserved_field_non_zero_imm:OK
 #537/5   verifier_gotox/jump_table_no_jump_table:OK
 #537/6   verifier_gotox/jump_table_incorrect_dst_reg_type:OK
 #537/7   verifier_gotox/jump_table_invalid_read_size_u32:OK
 #537/8   verifier_gotox/jump_table_invalid_read_size_u16:OK
 #537/9   verifier_gotox/jump_table_invalid_read_size_u8:OK
 #537/10  verifier_gotox/jump_table_misaligned_access:OK
 #537/11  verifier_gotox/jump_table_invalid_mem_acceess_pos:OK
 #537/12  verifier_gotox/jump_table_invalid_mem_acceess_neg:OK
 #537/13  verifier_gotox/jump_table_add_sub_ok:OK
 #537/14  verifier_gotox/jump_table_no_writes:OK
 #537/15  verifier_gotox/jump_table_use_reg_r0:OK
 #537/16  verifier_gotox/jump_table_use_reg_r1:OK
 #537/17  verifier_gotox/jump_table_use_reg_r2:OK
 #537/18  verifier_gotox/jump_table_use_reg_r3:OK
 #537/19  verifier_gotox/jump_table_use_reg_r4:OK
 #537/20  verifier_gotox/jump_table_use_reg_r5:OK
 #537/21  verifier_gotox/jump_table_use_reg_r6:OK
 #537/22  verifier_gotox/jump_table_use_reg_r7:OK
 #537/23  verifier_gotox/jump_table_use_reg_r8:OK
 #537/24  verifier_gotox/jump_table_use_reg_r9:OK
 #537/25  verifier_gotox/jump_table_outside_subprog:OK
 #537/26  verifier_gotox/jump_table_contains_non_unique_values:OK
 #537     verifier_gotox:OK
 Summary: 2/37 PASSED, 0 SKIPPED, 0 FAILED

Puranjay Mohan (4):
  bpf: arm64: Add support for instructions array
  bpf: arm64: Add support for indirect jumps
  libbpf: Ignore relocations for .jumptables sections
  selftests: bpf: Enable gotox tests from arm64

 arch/arm64/net/bpf_jit_comp.c                      | 11 +++++++++++
 tools/lib/bpf/linker.c                             |  4 ++++
 tools/testing/selftests/bpf/progs/verifier_gotox.c |  4 ++--
 3 files changed, 17 insertions(+), 2 deletions(-)

-- 
2.47.3

[PATCH bpf-next 1/4] bpf: arm64: Add support for instructions array

[Puranjay Mohan]

Add support for the instructions array map type in the arm64 JIT by
calling bpf_prog_update_insn_ptrs() with the offsets that map
xlated_offset to the jited_offset in the final image. arm64 JIT already
has this offset array which was being used for
bpf_prog_fill_jited_linfo() and can be used directly for
bpf_prog_update_insn_ptrs.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 arch/arm64/net/bpf_jit_comp.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 0c9a50a1e73e..4a2afc0cefc4 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2231,6 +2231,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		for (i = 0; i <= prog->len; i++)
 			ctx.offset[i] *= AARCH64_INSN_SIZE;
 		bpf_prog_fill_jited_linfo(prog, ctx.offset + 1);
+		/*
+		 * The bpf_prog_update_insn_ptrs function expects offsets to
+		 * point to the first byte of the jitted instruction (unlike
+		 * the bpf_prog_fill_jited_linfo above, which, for historical
+		 * reasons, expects to point to the next instruction)
+		 */
+		bpf_prog_update_insn_ptrs(prog, ctx.offset, ctx.ro_image);
 out_off:
 		if (!ro_header && priv_stack_ptr) {
 			free_percpu(priv_stack_ptr);
-- 
2.47.3

[PATCH bpf-next 2/4] bpf: arm64: Add support for indirect jumps

[Puranjay Mohan]

Add support for a new instruction

	BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0

which does an indirect jump to a location stored in Rx.  The register
Rx should have type PTR_TO_INSN. This new type assures that the Rx
register contains a value (or a range of values) loaded from a
correct jump table – map of type instruction array.

ARM64 JIT supports indirect jumps to all registers through the A64_BR()
macro, use it to implement this new instruction.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 arch/arm64/net/bpf_jit_comp.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 4a2afc0cefc4..4cfb549f2b43 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -1452,6 +1452,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
 		emit(A64_ASR(is64, dst, dst, imm), ctx);
 		break;
 
+	/* JUMP reg */
+	case BPF_JMP | BPF_JA | BPF_X:
+		emit(A64_BR(dst), ctx);
+		break;
 	/* JUMP off */
 	case BPF_JMP | BPF_JA:
 	case BPF_JMP32 | BPF_JA:
-- 
2.47.3

[PATCH bpf-next 3/4] libbpf: Ignore relocations for .jumptables sections

[Puranjay Mohan]

.jumptables sections generated by LLVM store byte offsets from the start
of the section to the jump's target instruction. On arm64 builds of
LLVM, it generates a rel section to add the section address to each
entry in the jump table, therefore storing the absolute address of the
jump target into the table. For BPF usage of .jumptables, we only care
about the instruction offsets and calculate the final address much later
after the JIT has finished, and called bpf_prog_update_insn_ptrs()

Without the change in this commit, compilation fails with:

libbpf: relocation against STT_SECTION in non-exec section is not
supported!  Error: failed to link
'tools/testing/selftests/bpf/cpuv4/bpf_gotox.bpf.o': Invalid argument (22)

This is due to the presence of a relocation section for .jumptables
(non-exec) against syscall (STT_SECTION):

Relocation section '.rel.jumptables' at offset 0x5b50 contains 263 entries:
  Offset        Info             Type         Symbol's Value  Symbol's Name
000000000000  0000000300000002 R_BPF_64_ABS64  0000000000000000 syscall
000000000008  0000000300000002 R_BPF_64_ABS64  0000000000000000 syscall
[...]

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 tools/lib/bpf/linker.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
index f4403e3cf994..f4295962983b 100644
--- a/tools/lib/bpf/linker.c
+++ b/tools/lib/bpf/linker.c
@@ -634,6 +634,10 @@ static bool is_ignored_sec(struct src_sec *sec)
 		if (strcmp(name, BTF_ELF_SEC) == 0 ||
 		    strcmp(name, BTF_EXT_ELF_SEC) == 0)
 			return true;
+
+		/* .jumptables don't need relocations */
+		if (strcmp(name, JUMPTABLES_SEC) == 0)
+			return true;
 	}
 
 	return false;
-- 
2.47.3

[PATCH bpf-next 4/4] selftests: bpf: Enable gotox tests from arm64

[Puranjay Mohan]

arm64 JIT now supports gotox instruction and jumptables, so run tests in
verifier_gotox.c for arm64.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 tools/testing/selftests/bpf/progs/verifier_gotox.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c
index b6710f134a1d..536c9f3e2170 100644
--- a/tools/testing/selftests/bpf/progs/verifier_gotox.c
+++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c
@@ -6,7 +6,7 @@
 #include "bpf_misc.h"
 #include "../../../include/linux/filter.h"
 
-#ifdef __TARGET_ARCH_x86
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
 
 #define DEFINE_SIMPLE_JUMP_TABLE_PROG(NAME, SRC_REG, OFF, IMM, OUTCOME)	\
 									\
@@ -384,6 +384,6 @@ jt0_%=:								\
 	: __clobber_all);
 }
 
-#endif /* __TARGET_ARCH_x86 */
+#endif /* __TARGET_ARCH_x86 || __TARGET_ARCH_arm64 */
 
 char _license[] SEC("license") = "GPL";
-- 
2.47.3

Re: [PATCH bpf-next 0/4] bpf: arm64: Indirect jumps

[Anton Protopopov]

On 25/11/17 12:46AM, Puranjay Mohan wrote:
> This set adds the support of indirect jumps to the arm64 JIT. It
> involves calling bpf_prog_update_insn_ptrs() to support instructions
> array map. The second piece is supporting BPF_JMP|BPF_X|BPF_JA, SRC=0,
> DST=Rx, off=0, imm=0 instruction that is trivial to implement on arm64.
>
> When running the selftests after doing the above changes, I found that
> on arm64 builds of llvm, a relocation section was being generated for
> .jumptables sections and it was making libbpf fail like:
> 
> libbpf: relocation against STT_SECTION in non-exec section is not supported!
> Error: failed to link 'tools/testing/selftests/bpf/cpuv4/bpf_gotox.bpf.o': Invalid argument (22)
> 
> Which is due to:
> 
> Relocation section '.rel.jumptables' at offset 0x5b50 contains 263 entries:
>     Offset             Info             Type               Symbol's Value  Symbol's Name
> 0000000000000000  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall
> 0000000000000008  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall
> 0000000000000010  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall
> 
> This rel section is not generated by x86 builds of LLVM. The third patch
> of this set makes libbpf ignore relocation sections for .jumptables.

I added Yonghong to this thread. He had fixed this problem in
https://github.com/llvm/llvm-project/pull/166301 changes doesn't seem to be
x86-specific...

> 
> The final patch enables selftests on arm64:
> 
>  [root@localhost bpf]# ./test_progs-cpuv4 -a "*gotox*"
>  #20/1    bpf_gotox/one-switch:OK
>  #20/2    bpf_gotox/one-switch-non-zero-sec-offset:OK
>  #20/3    bpf_gotox/two-switches:OK
>  #20/4    bpf_gotox/big-jump-table:OK
>  #20/5    bpf_gotox/static-global:OK
>  #20/6    bpf_gotox/nonstatic-global:OK
>  #20/7    bpf_gotox/other-sec:OK
>  #20/8    bpf_gotox/static-global-other-sec:OK
>  #20/9    bpf_gotox/nonstatic-global-other-sec:OK
>  #20/10   bpf_gotox/one-jump-two-maps:OK
>  #20/11   bpf_gotox/one-map-two-jumps:OK
>  #20      bpf_gotox:OK
>  #537/1   verifier_gotox/jump_table_ok:OK
>  #537/2   verifier_gotox/jump_table_reserved_field_src_reg:OK
>  #537/3   verifier_gotox/jump_table_reserved_field_non_zero_off:OK
>  #537/4   verifier_gotox/jump_table_reserved_field_non_zero_imm:OK
>  #537/5   verifier_gotox/jump_table_no_jump_table:OK
>  #537/6   verifier_gotox/jump_table_incorrect_dst_reg_type:OK
>  #537/7   verifier_gotox/jump_table_invalid_read_size_u32:OK
>  #537/8   verifier_gotox/jump_table_invalid_read_size_u16:OK
>  #537/9   verifier_gotox/jump_table_invalid_read_size_u8:OK
>  #537/10  verifier_gotox/jump_table_misaligned_access:OK
>  #537/11  verifier_gotox/jump_table_invalid_mem_acceess_pos:OK
>  #537/12  verifier_gotox/jump_table_invalid_mem_acceess_neg:OK
>  #537/13  verifier_gotox/jump_table_add_sub_ok:OK
>  #537/14  verifier_gotox/jump_table_no_writes:OK
>  #537/15  verifier_gotox/jump_table_use_reg_r0:OK
>  #537/16  verifier_gotox/jump_table_use_reg_r1:OK
>  #537/17  verifier_gotox/jump_table_use_reg_r2:OK
>  #537/18  verifier_gotox/jump_table_use_reg_r3:OK
>  #537/19  verifier_gotox/jump_table_use_reg_r4:OK
>  #537/20  verifier_gotox/jump_table_use_reg_r5:OK
>  #537/21  verifier_gotox/jump_table_use_reg_r6:OK
>  #537/22  verifier_gotox/jump_table_use_reg_r7:OK
>  #537/23  verifier_gotox/jump_table_use_reg_r8:OK
>  #537/24  verifier_gotox/jump_table_use_reg_r9:OK
>  #537/25  verifier_gotox/jump_table_outside_subprog:OK
>  #537/26  verifier_gotox/jump_table_contains_non_unique_values:OK
>  #537     verifier_gotox:OK
>  Summary: 2/37 PASSED, 0 SKIPPED, 0 FAILED

Cool!

> Puranjay Mohan (4):
>   bpf: arm64: Add support for instructions array
>   bpf: arm64: Add support for indirect jumps
>   libbpf: Ignore relocations for .jumptables sections
>   selftests: bpf: Enable gotox tests from arm64
> 
>  arch/arm64/net/bpf_jit_comp.c                      | 11 +++++++++++
>  tools/lib/bpf/linker.c                             |  4 ++++
>  tools/testing/selftests/bpf/progs/verifier_gotox.c |  4 ++--
>  3 files changed, 17 insertions(+), 2 deletions(-)
> 
> -- 
> 2.47.3
> 

Re: [PATCH bpf-next 1/4] bpf: arm64: Add support for instructions array

[Anton Protopopov]

On 25/11/17 12:46AM, Puranjay Mohan wrote:
> Add support for the instructions array map type in the arm64 JIT by
> calling bpf_prog_update_insn_ptrs() with the offsets that map
> xlated_offset to the jited_offset in the final image. arm64 JIT already
> has this offset array which was being used for
> bpf_prog_fill_jited_linfo() and can be used directly for
> bpf_prog_update_insn_ptrs.
> 
> Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
> ---
>  arch/arm64/net/bpf_jit_comp.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index 0c9a50a1e73e..4a2afc0cefc4 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
> @@ -2231,6 +2231,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		for (i = 0; i <= prog->len; i++)
>  			ctx.offset[i] *= AARCH64_INSN_SIZE;
>  		bpf_prog_fill_jited_linfo(prog, ctx.offset + 1);
> +		/*
> +		 * The bpf_prog_update_insn_ptrs function expects offsets to
> +		 * point to the first byte of the jitted instruction (unlike
> +		 * the bpf_prog_fill_jited_linfo above, which, for historical
> +		 * reasons, expects to point to the next instruction)
> +		 */
> +		bpf_prog_update_insn_ptrs(prog, ctx.offset, ctx.ro_image);
>  out_off:
>  		if (!ro_header && priv_stack_ptr) {
>  			free_percpu(priv_stack_ptr);
> -- 
> 2.47.3
> 

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>

Re: [PATCH bpf-next 2/4] bpf: arm64: Add support for indirect jumps

[Anton Protopopov]

On 25/11/17 12:46AM, Puranjay Mohan wrote:
> Add support for a new instruction
> 
> 	BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0
> 
> which does an indirect jump to a location stored in Rx.  The register
> Rx should have type PTR_TO_INSN. This new type assures that the Rx
> register contains a value (or a range of values) loaded from a
> correct jump table – map of type instruction array.
> 
> ARM64 JIT supports indirect jumps to all registers through the A64_BR()
> macro, use it to implement this new instruction.
> 
> Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
> ---
>  arch/arm64/net/bpf_jit_comp.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index 4a2afc0cefc4..4cfb549f2b43 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
> @@ -1452,6 +1452,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
>  		emit(A64_ASR(is64, dst, dst, imm), ctx);
>  		break;
>  
> +	/* JUMP reg */
> +	case BPF_JMP | BPF_JA | BPF_X:
> +		emit(A64_BR(dst), ctx);
> +		break;
>  	/* JUMP off */
>  	case BPF_JMP | BPF_JA:
>  	case BPF_JMP32 | BPF_JA:
> -- 
> 2.47.3
> 

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>

Re: [PATCH bpf-next 4/4] selftests: bpf: Enable gotox tests from arm64

[Anton Protopopov]

On 25/11/17 12:46AM, Puranjay Mohan wrote:
> arm64 JIT now supports gotox instruction and jumptables, so run tests in
> verifier_gotox.c for arm64.
> 
> Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
> ---
>  tools/testing/selftests/bpf/progs/verifier_gotox.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c
> index b6710f134a1d..536c9f3e2170 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_gotox.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c
> @@ -6,7 +6,7 @@
>  #include "bpf_misc.h"
>  #include "../../../include/linux/filter.h"
>  
> -#ifdef __TARGET_ARCH_x86
> +#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)

Great to see this :)

>  #define DEFINE_SIMPLE_JUMP_TABLE_PROG(NAME, SRC_REG, OFF, IMM, OUTCOME)	\
>  									\
> @@ -384,6 +384,6 @@ jt0_%=:								\
>  	: __clobber_all);
>  }
>  
> -#endif /* __TARGET_ARCH_x86 */
> +#endif /* __TARGET_ARCH_x86 || __TARGET_ARCH_arm64 */
>  
>  char _license[] SEC("license") = "GPL";
> -- 
> 2.47.3
> 

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>

Re: [PATCH bpf-next 0/4] bpf: arm64: Indirect jumps

[Puranjay Mohan]

Anton Protopopov <a.s.protopopov@gmail.com> writes:

> On 25/11/17 12:46AM, Puranjay Mohan wrote:
>> This set adds the support of indirect jumps to the arm64 JIT. It
>> involves calling bpf_prog_update_insn_ptrs() to support instructions
>> array map. The second piece is supporting BPF_JMP|BPF_X|BPF_JA, SRC=0,
>> DST=Rx, off=0, imm=0 instruction that is trivial to implement on arm64.
>>
>> When running the selftests after doing the above changes, I found that
>> on arm64 builds of llvm, a relocation section was being generated for
>> .jumptables sections and it was making libbpf fail like:
>> 
>> libbpf: relocation against STT_SECTION in non-exec section is not supported!
>> Error: failed to link 'tools/testing/selftests/bpf/cpuv4/bpf_gotox.bpf.o': Invalid argument (22)
>> 
>> Which is due to:
>> 
>> Relocation section '.rel.jumptables' at offset 0x5b50 contains 263 entries:
>>     Offset             Info             Type               Symbol's Value  Symbol's Name
>> 0000000000000000  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall
>> 0000000000000008  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall
>> 0000000000000010  0000000300000002 R_BPF_64_ABS64         0000000000000000 syscall
>> 
>> This rel section is not generated by x86 builds of LLVM. The third patch
>> of this set makes libbpf ignore relocation sections for .jumptables.
>
> I added Yonghong to this thread. He had fixed this problem in
> https://github.com/llvm/llvm-project/pull/166301 changes doesn't seem to be
> x86-specific...

My arm64 build didn't have this change, that is why it was failing.
After pulling the latest changes and building again, it works without
the libbpf patch.

So, I will send v2 and drop the libbpf patch.

Thanks,
Puranjay