From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by smtp.lore.kernel.org (Postfix) with ESMTP id E12E0CD6E79 for ; Mon, 8 Jun 2026 20:33:48 +0000 (UTC) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 225BF4065A; Mon, 8 Jun 2026 22:33:32 +0200 (CEST) Received: from mail-dl1-f50.google.com (mail-dl1-f50.google.com [74.125.82.50]) by mails.dpdk.org (Postfix) with ESMTP id 5E1414069F for ; Mon, 8 Jun 2026 22:33:30 +0200 (CEST) Received: by mail-dl1-f50.google.com with SMTP id a92af1059eb24-137ec563a95so5852799c88.0 for ; Mon, 08 Jun 2026 13:33:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20251104.gappssmtp.com; s=20251104; t=1780950809; x=1781555609; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BO4cjNqWJ+dPmXlwsbMJcJUBpk+xMgUUMtBUc+KKKSw=; b=RcfOKNf7gxwAjIk2bQZbYwUeImu9lt53jNDbgKKPSIZMh2fAfWbFLz39zCOFRH0zAy UdkYU6XceqL7PyWgAkYzmbL75yENehcyXNnfFq6O2yJ0u3q7tHbJYlUt5FeGw4xJEdgH eR82cB+x2VqLh8iDisVP4D8xVzerL9njImGBszk2xx0xlfkQawQzcjnhwyM9FO//zMAN mQXLsNIPR1r3wtHxbbPdbw/whtZoD4aMsRsqw3JmBPeAxpdYGCuon6TjAxxMib4Ynv7V /TZG8qISZmsKYYleuPPEQgfPWQ/1lA4fI6YyGPzBj9bHk2hKyVYxQRMUjzc/zcOiRe+0 8dFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780950809; x=1781555609; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=BO4cjNqWJ+dPmXlwsbMJcJUBpk+xMgUUMtBUc+KKKSw=; b=NXbHiX0x67yze/UwX6OkuvYGkOQKhcmnquySpGpBaEF1vvg9I/K1UtvViILRKF1g6f JD5IcrFSCn9KmSxHCoQ0wLRL9Aj92THSR6tuvsecPYUwXWTlPKLhQxlArQFkhV5a3su9 IAhGXW7zH7Pdh860Tmwf8OFMYA0XOVxzEMX2Fv296bOkDlxX+xsdZquB3Hj5rJYKSyIm V2A76QDkL+XUJZrn4aiq9/ujrbx766E512TEw581jiserlMagbDjNN0z0Q+RA1UCCoZD VwkufsYiI14H2NZQB+hQjlb7o80rzsonAyWnQL2wYvwKCEhz9+kXEwUkcys05fEhK1Bm UpOw== X-Gm-Message-State: AOJu0YwWfd238RWnNGEqt5+e1RyTkacZcNumjA5ce4ceCf9ErcCz/DiX hs11xNWoySu9AiM78X24tWNmvKbG1YA6DF3krpXFL6OOaVbJpdA1OIY4Ks5sjn5y5BG6NkQBHNV JVqj1 X-Gm-Gg: Acq92OEnyZdURvV0nKZy20ZUwAQaXdrxWxa5Pe465jHP4AGwT/CulNDp1/gxdd1d+Ff zjGmQHCBWNY8+G+Maij5YJxxJaoHPkx5mEoO9MWgp4o6dFI7uF6MZbwScxnSduPBEQ3nzPNNDtH 0BpWTN2mcUL4StwbXC600kvzp2bBD7VCfRDUyBXjg8VZezxvEwWnolbqm8ewB1pMhMnJmvsDBEX AvR5aId5uK/zqesN/rskjvsaoB1AyJsqXEHjaaaB8qY4dJ2RZ073rh24593OFrg6b6Ez4B6D42U dsBw39mCbkBJvJOVIx9fL0FUHDEZri+Nn+h4xDTvw29BBGjTjisZJa9VviJHYslQIjN/0SrqsNM 3i7TCGwjRpxd5j/3W9PDGXGtzSUBcWenZD6hOQlqE2TdsTSUwwNORTVRb2LStyCkiMnGMhv0RNf G5q9wKWGn30VKtIFaLoOGqkFpMMCcEgFvPfS4w3fDWCI03OueFvE98mvUyTO9Tby9IwttA8B3s X-Received: by 2002:a05:7300:72c9:b0:2ed:935:aa33 with SMTP id 5a478bee46e88-3077af330fdmr9901316eec.5.1780950809196; Mon, 08 Jun 2026 13:33:29 -0700 (PDT) Received: from phoenix.lan (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3074db560d7sm25804178eec.5.2026.06.08.13.33.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Jun 2026 13:33:28 -0700 (PDT) From: Stephen Hemminger To: dev@dpdk.org Cc: Stephen Hemminger , Wathsala Vithanage , Konstantin Ananyev , Marat Khalili Subject: [PATCH 4/4] bpf/arm64: add BPF_ABS/BPF_IND packet load support Date: Mon, 8 Jun 2026 13:28:50 -0700 Message-ID: <20260608203322.1116296-5-stephen@networkplumber.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260608203322.1116296-1-stephen@networkplumber.org> References: <20260608203322.1116296-1-stephen@networkplumber.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The arm64 JIT rejected BPF_LD | BPF_ABS and BPF_LD | BPF_IND with "invalid opcode", so cBPF programs converted by rte_bpf_convert() could not be JITed. Add these opcodes, mirroring the x86 JIT: a fast path for data held in the first mbuf segment and a __rte_pktmbuf_read() slow path for everything else. Programs using these opcodes now use the call register layout, since the slow path makes a function call. Bugzilla ID: 1427 Signed-off-by: Stephen Hemminger --- lib/bpf/bpf_jit_arm64.c | 147 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 146 insertions(+), 1 deletion(-) diff --git a/lib/bpf/bpf_jit_arm64.c b/lib/bpf/bpf_jit_arm64.c index 099822e9f1..6952c61806 100644 --- a/lib/bpf/bpf_jit_arm64.c +++ b/lib/bpf/bpf_jit_arm64.c @@ -1123,6 +1123,133 @@ emit_branch(struct a64_jit_ctx *ctx, uint8_t op, uint32_t i, int16_t off) emit_b_cond(ctx, ebpf_to_a64_cond(op), jump_offset_get(ctx, i, off)); } +/* LD_ABS/LD_IND code block offsets (in arm64 instructions) */ +enum { + LDMB_FAST_OFS, /* fast path */ + LDMB_SLOW_OFS, /* slow path */ + LDMB_FIN_OFS, /* common tail */ + LDMB_OFS_NUM +}; + +/* + * Helper for emit_ld_mbuf(): fast path. + * Compute the packet offset; if it lies inside the first segment leave the + * data pointer in R0, otherwise branch to the slow path. + */ +static void +emit_ldmb_fast_path(struct a64_jit_ctx *ctx, uint8_t src, uint8_t mode, + uint32_t sz, int32_t imm, const uint32_t ofs[LDMB_OFS_NUM]) +{ + uint8_t r0 = ebpf_to_a64_reg(ctx, EBPF_REG_0); + uint8_t r6 = ebpf_to_a64_reg(ctx, EBPF_REG_6); + uint8_t tmp1 = ebpf_to_a64_reg(ctx, TMP_REG_1); + uint8_t tmp2 = ebpf_to_a64_reg(ctx, TMP_REG_2); + uint8_t tmp3 = ebpf_to_a64_reg(ctx, TMP_REG_3); + + /* off = imm (+ src for BPF_IND) */ + emit_mov_imm(ctx, 1, tmp1, imm); + if (mode == BPF_IND) + emit_add(ctx, 1, tmp1, src); + + /* if ((int64_t)(mbuf->data_len - off) < sz) goto slow_path */ + emit_mov_imm(ctx, 1, tmp2, offsetof(struct rte_mbuf, data_len)); + emit_ldr(ctx, BPF_H, tmp2, r6, tmp2); + emit_sub(ctx, 1, tmp2, tmp1); + emit_mov_imm(ctx, 1, tmp3, sz); + emit_cmp(ctx, 1, tmp2, tmp3); + emit_b_cond(ctx, A64_LT, (int32_t)(ofs[LDMB_SLOW_OFS] - ctx->idx)); + + /* R0 = mbuf->buf_addr + mbuf->data_off + off */ + emit_mov_imm(ctx, 1, tmp2, offsetof(struct rte_mbuf, data_off)); + emit_ldr(ctx, BPF_H, tmp2, r6, tmp2); + emit_mov_imm(ctx, 1, r0, offsetof(struct rte_mbuf, buf_addr)); + emit_ldr(ctx, EBPF_DW, r0, r6, r0); + emit_add(ctx, 1, r0, tmp2); + emit_add(ctx, 1, r0, tmp1); + + emit_b(ctx, (int32_t)(ofs[LDMB_FIN_OFS] - ctx->idx)); +} + +/* + * Helper for emit_ld_mbuf(): slow path. + * R0 = __rte_pktmbuf_read(mbuf, off, sz, buf); return 0 if NULL. + * The scratch buffer is the space reserved by __rte_bpf_validate() at the + * bottom of the eBPF stack frame, i.e. (frame_pointer - stack_ofs). + */ +static void +emit_ldmb_slow_path(struct a64_jit_ctx *ctx, uint32_t sz, uint32_t stack_ofs) +{ + uint8_t r0 = ebpf_to_a64_reg(ctx, EBPF_REG_0); + uint8_t r6 = ebpf_to_a64_reg(ctx, EBPF_REG_6); + uint8_t fp = ebpf_to_a64_reg(ctx, EBPF_FP); + uint8_t tmp1 = ebpf_to_a64_reg(ctx, TMP_REG_1); + + /* arguments of __rte_pktmbuf_read(mbuf, off, len, buf) */ + emit_mov_64(ctx, A64_R(1), tmp1); /* off (held in tmp1) */ + emit_mov_64(ctx, A64_R(0), r6); /* mbuf */ + emit_mov_imm(ctx, 0, A64_R(2), sz); /* len */ + emit_sub_imm_64(ctx, A64_R(3), fp, stack_ofs); /* buf */ + + emit_call(ctx, tmp1, (void *)(uintptr_t)__rte_pktmbuf_read); + emit_return_zero_if_src_zero(ctx, 1, r0); +} + +/* + * Helper for emit_ld_mbuf(): common tail. + * Load the value pointed to by R0 and convert from network byte order. + */ +static void +emit_ldmb_fin(struct a64_jit_ctx *ctx, uint8_t opsz, uint32_t sz) +{ + uint8_t r0 = ebpf_to_a64_reg(ctx, EBPF_REG_0); + + emit_ldr(ctx, opsz, r0, r0, A64_ZR); + if (opsz != BPF_B) + emit_be(ctx, r0, sz * 8); +} + +/* + * Emit code for BPF_LD | BPF_ABS and BPF_LD | BPF_IND packet loads: + * + * off = imm (+ src for BPF_IND) + * if (mbuf->data_len - off >= sz) -- fast path + * ptr = mbuf->buf_addr + mbuf->data_off + off; + * else -- slow path + * ptr = __rte_pktmbuf_read(mbuf, off, sz, buf); + * if (ptr == NULL) + * return 0; + * R0 = ntoh(*(size *)ptr); -- common tail + * + * The three blocks are sized in a dry run so the forward branches can be + * resolved, then emitted for real (arm64 instructions are fixed width, so + * the dry run reproduces the real instruction count exactly). + */ +static void +emit_ld_mbuf(struct a64_jit_ctx *ctx, uint8_t op, uint8_t src, int32_t imm, + uint32_t stack_ofs) +{ + uint8_t mode = BPF_MODE(op); + uint8_t opsz = BPF_SIZE(op); + uint32_t sz = bpf_size(opsz); + uint32_t ofs[LDMB_OFS_NUM]; + + /* seed offsets so the dry-run branches stay in range */ + ofs[LDMB_FAST_OFS] = ofs[LDMB_SLOW_OFS] = ofs[LDMB_FIN_OFS] = ctx->idx; + + /* dry run to record block offsets */ + emit_ldmb_fast_path(ctx, src, mode, sz, imm, ofs); + ofs[LDMB_SLOW_OFS] = ctx->idx; + emit_ldmb_slow_path(ctx, sz, stack_ofs); + ofs[LDMB_FIN_OFS] = ctx->idx; + emit_ldmb_fin(ctx, opsz, sz); + + /* rewind and emit for real with resolved offsets */ + ctx->idx = ofs[LDMB_FAST_OFS]; + emit_ldmb_fast_path(ctx, src, mode, sz, imm, ofs); + emit_ldmb_slow_path(ctx, sz, stack_ofs); + emit_ldmb_fin(ctx, opsz, sz); +} + static void check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf) { @@ -1135,8 +1262,17 @@ check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf) op = ins->code; switch (op) { - /* Call imm */ + /* + * BPF_ABS/BPF_IND can fall through to __rte_pktmbuf_read(), + * so they need the call-clobbered register layout as well. + */ case (BPF_JMP | EBPF_CALL): + case (BPF_LD | BPF_ABS | BPF_B): + case (BPF_LD | BPF_ABS | BPF_H): + case (BPF_LD | BPF_ABS | BPF_W): + case (BPF_LD | BPF_IND | BPF_B): + case (BPF_LD | BPF_IND | BPF_H): + case (BPF_LD | BPF_IND | BPF_W): ctx->foundcall = 1; return; } @@ -1338,6 +1474,15 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf) emit_mov_imm(ctx, 1, dst, u64); i++; break; + /* R0 = ntoh(*(size *)(mbuf data + (src) + imm)) */ + case (BPF_LD | BPF_ABS | BPF_B): + case (BPF_LD | BPF_ABS | BPF_H): + case (BPF_LD | BPF_ABS | BPF_W): + case (BPF_LD | BPF_IND | BPF_B): + case (BPF_LD | BPF_IND | BPF_H): + case (BPF_LD | BPF_IND | BPF_W): + emit_ld_mbuf(ctx, op, src, imm, bpf->stack_sz); + break; /* *(size *)(dst + off) = src */ case (BPF_STX | BPF_MEM | BPF_B): case (BPF_STX | BPF_MEM | BPF_H): -- 2.53.0