From: Ilya Leoshkevich <iii@linux.ibm.com>
To: Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>
Cc: bpf@vger.kernel.org, Heiko Carstens <hca@linux.ibm.com>,
Vasily Gorbik <gor@linux.ibm.com>,
Alexander Gordeev <agordeev@linux.ibm.com>,
Ilya Leoshkevich <iii@linux.ibm.com>
Subject: [PATCH bpf-next v2 30/31] s390/bpf: Implement bpf_jit_supports_subprog_tailcalls()
Date: Sat, 28 Jan 2023 01:06:49 +0100 [thread overview]
Message-ID: <20230128000650.1516334-31-iii@linux.ibm.com> (raw)
In-Reply-To: <20230128000650.1516334-1-iii@linux.ibm.com>
Allow mixing subprogs and tail calls by passing the current tail
call count to subprogs.
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
---
arch/s390/net/bpf_jit_comp.c | 37 ++++++++++++++++++++++++++----------
1 file changed, 27 insertions(+), 10 deletions(-)
diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
index 990d624006c4..e035dd24b430 100644
--- a/arch/s390/net/bpf_jit_comp.c
+++ b/arch/s390/net/bpf_jit_comp.c
@@ -58,7 +58,6 @@ struct bpf_jit {
#define SEEN_MEM BIT(0) /* use mem[] for temporary storage */
#define SEEN_LITERAL BIT(1) /* code uses literals */
#define SEEN_FUNC BIT(2) /* calls C functions */
-#define SEEN_TAIL_CALL BIT(3) /* code uses tail calls */
#define SEEN_STACK (SEEN_FUNC | SEEN_MEM)
/*
@@ -549,20 +548,23 @@ static void bpf_jit_plt(void *plt, void *ret, void *target)
* Save registers and create stack frame if necessary.
* See stack frame layout description in "bpf_jit.h"!
*/
-static void bpf_jit_prologue(struct bpf_jit *jit, u32 stack_depth)
+static void bpf_jit_prologue(struct bpf_jit *jit, struct bpf_prog *fp,
+ u32 stack_depth)
{
/* No-op for hotpatching */
/* brcl 0,prologue_plt */
EMIT6_PCREL_RILC(0xc0040000, 0, jit->prologue_plt);
jit->prologue_plt_ret = jit->prg;
- if (jit->seen & SEEN_TAIL_CALL) {
+ if (fp->aux->func_idx == 0) {
+ /* Initialize the tail call counter in the main program. */
/* xc STK_OFF_TCCNT(4,%r15),STK_OFF_TCCNT(%r15) */
_EMIT6(0xd703f000 | STK_OFF_TCCNT, 0xf000 | STK_OFF_TCCNT);
} else {
/*
- * There are no tail calls. Insert nops in order to have
- * tail_call_start at a predictable offset.
+ * Skip the tail call counter initialization in subprograms.
+ * Insert nops in order to have tail_call_start at a
+ * predictable offset.
*/
bpf_skip(jit, 6);
}
@@ -1410,6 +1412,19 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
REG_SET_SEEN(BPF_REG_5);
jit->seen |= SEEN_FUNC;
+ /*
+ * Copy the tail call counter to where the callee expects it.
+ *
+ * Note 1: The callee can increment the tail call counter, but
+ * we do not load it back, since the x86 JIT does not do this
+ * either.
+ *
+ * Note 2: We assume that the verifier does not let us call the
+ * main program, which clears the tail call counter on entry.
+ */
+ /* mvc STK_OFF_TCCNT(4,%r15),N(%r15) */
+ _EMIT6(0xd203f000 | STK_OFF_TCCNT,
+ 0xf000 | (STK_OFF_TCCNT + STK_OFF + stack_depth));
/* lgrl %w1,func */
EMIT6_PCREL_RILB(0xc4080000, REG_W1, _EMIT_CONST_U64(func));
/* %r1() */
@@ -1426,10 +1441,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
* B1: pointer to ctx
* B2: pointer to bpf_array
* B3: index in bpf_array
- */
- jit->seen |= SEEN_TAIL_CALL;
-
- /*
+ *
* if (index >= array->map.max_entries)
* goto out;
*/
@@ -1793,7 +1805,7 @@ static int bpf_jit_prog(struct bpf_jit *jit, struct bpf_prog *fp,
jit->prg = 0;
jit->excnt = 0;
- bpf_jit_prologue(jit, stack_depth);
+ bpf_jit_prologue(jit, fp, stack_depth);
if (bpf_set_addr(jit, 0) < 0)
return -1;
for (i = 0; i < fp->len; i += insn_count) {
@@ -2461,3 +2473,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
return ret;
}
+
+bool bpf_jit_supports_subprog_tailcalls(void)
+{
+ return true;
+}
--
2.39.1
next prev parent reply other threads:[~2023-01-28 0:08 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-28 0:06 [PATCH bpf-next v2 00/31] Support bpf trampoline for s390x Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 01/31] bpf: Use ARG_CONST_SIZE_OR_ZERO for 3rd argument of bpf_tcp_raw_gen_syncookie_ipv{4,6}() Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 02/31] bpf: Change BPF_MAX_TRAMP_LINKS to enum Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 03/31] selftests/bpf: Query BPF_MAX_TRAMP_LINKS using BTF Ilya Leoshkevich
2023-01-28 20:49 ` Alexei Starovoitov
2023-01-29 1:50 ` Alexei Starovoitov
2023-01-28 0:06 ` [PATCH bpf-next v2 04/31] selftests/bpf: Fix liburandom_read.so linker error Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 05/31] selftests/bpf: Fix symlink creation error Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 06/31] selftests/bpf: Fix kfree_skb on s390x Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 07/31] selftests/bpf: Set errno when urand_spawn() fails Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 08/31] selftests/bpf: Fix decap_sanity_ns cleanup Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 09/31] selftests/bpf: Fix verify_pkcs7_sig on s390x Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 10/31] selftests/bpf: Fix xdp_do_redirect " Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 11/31] selftests/bpf: Fix cgrp_local_storage " Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 12/31] selftests/bpf: Check stack_mprotect() return value Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 13/31] selftests/bpf: Increase SIZEOF_BPF_LOCAL_STORAGE_ELEM on s390x Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 14/31] selftests/bpf: Add a sign-extension test for kfuncs Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 15/31] selftests/bpf: Fix test_lsm on s390x Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 16/31] selftests/bpf: Fix test_xdp_adjust_tail_grow2 " Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 17/31] selftests/bpf: Fix vmlinux test " Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 18/31] selftests/bpf: Fix sk_assign " Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 19/31] selftests/bpf: Fix xdp_synproxy/tc " Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 20/31] selftests/bpf: Fix profiler " Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 21/31] libbpf: Simplify barrier_var() Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 22/31] libbpf: Fix unbounded memory access in bpf_usdt_arg() Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 23/31] libbpf: Fix BPF_PROBE_READ{_STR}_INTO() on s390x Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 24/31] bpf: iterators: Split iterators.lskel.h into little- and big- endian versions Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 25/31] bpf: btf: Add BTF_FMODEL_SIGNED_ARG flag Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 26/31] s390/bpf: Fix a typo in a comment Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 27/31] s390/bpf: Add expoline to tail calls Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 28/31] s390/bpf: Implement bpf_arch_text_poke() Ilya Leoshkevich
2023-01-28 0:06 ` [PATCH bpf-next v2 29/31] s390/bpf: Implement arch_prepare_bpf_trampoline() Ilya Leoshkevich
2023-01-28 8:37 ` kernel test robot
2023-01-28 0:06 ` Ilya Leoshkevich [this message]
2023-01-28 0:06 ` [PATCH bpf-next v2 31/31] s390/bpf: Implement bpf_jit_supports_kfunc_call() Ilya Leoshkevich
2023-01-28 20:50 ` [PATCH bpf-next v2 00/31] Support bpf trampoline for s390x patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230128000650.1516334-31-iii@linux.ibm.com \
--to=iii@linux.ibm.com \
--cc=agordeev@linux.ibm.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=gor@linux.ibm.com \
--cc=hca@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox