From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6B2D5679 for ; Thu, 13 Jul 2023 06:07:40 +0000 (UTC) Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85B5D1FDE for ; Wed, 12 Jul 2023 23:07:37 -0700 (PDT) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36CMvdcS003868 for ; Wed, 12 Jul 2023 23:07:37 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=OsbzCfO5ngHBR/YbaSHk/CHun0bFb/OGEGdwFs6JDgg=; b=HZdcgz9JlpJL4TwdTY77w1tfw4dZ2uLr49EhSI0pTKp4B6nzMJOKLKubtvKjKvFYNudS VfzFqgGiTq2cc73n4XM53c5qX8W0sam97eboZXTpNs7pqgPEeallr+bj8rJBg5aq2Huw xSuETFNRQCLtACBoyZtLuKsCSZTnoAmHUTY= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3rsgc92vc6-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 12 Jul 2023 23:07:37 -0700 Received: from twshared35445.38.frc1.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 12 Jul 2023 23:07:36 -0700 Received: by devbig309.ftw3.facebook.com (Postfix, from userid 128203) id 7119C22EFA27A; Wed, 12 Jul 2023 23:07:34 -0700 (PDT) From: Yonghong Song To: CC: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Fangrui Song , Subject: [PATCH bpf-next v2 03/15] bpf: Support new sign-extension mov insns Date: Wed, 12 Jul 2023 23:07:34 -0700 Message-ID: <20230713060734.390551-1-yhs@fb.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713060718.388258-1-yhs@fb.com> References: <20230713060718.388258-1-yhs@fb.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe Content-Type: text/plain X-Proofpoint-GUID: 4yY0PE-PTGfwv1VvnlX4YzXgfBdZFS9W X-Proofpoint-ORIG-GUID: 4yY0PE-PTGfwv1VvnlX4YzXgfBdZFS9W X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-13_03,2023-07-11_01,2023-05-22_02 X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Add interpreter/jit support for new sign-extension mov insns. The original 'MOV' insn is extended to support signed version for both ALU and ALU64 operations. For ALU mode, the insn->off value of 8 or 16 indicates sign-extension from 8- or 16-bit value to 32-bit value. For ALU64 mode, the insn->off value of 8/16/32 indicates sign-extension from 8-, 16- or 32-bit value to 64-bit value. Signed-off-by: Yonghong Song --- arch/x86/net/bpf_jit_comp.c | 43 ++++++++++- kernel/bpf/core.c | 28 ++++++- kernel/bpf/verifier.c | 150 +++++++++++++++++++++++++++++------- 3 files changed, 190 insertions(+), 31 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index addeea95f397..a740a1a6e71d 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -701,6 +701,38 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 = dst_reg, u32 src_reg) *pprog =3D prog; } =20 +static void emit_movsx_reg(u8 **pprog, int num_bits, bool is64, u32 dst_= reg, + u32 src_reg) +{ + u8 *prog =3D *pprog; + + if (is64) { + /* movs[b,w,l]q dst, src */ + if (num_bits =3D=3D 8) + EMIT4(add_2mod(0x48, src_reg, dst_reg), 0x0f, 0xbe, + add_2reg(0xC0, src_reg, dst_reg)); + else if (num_bits =3D=3D 16) + EMIT4(add_2mod(0x48, src_reg, dst_reg), 0x0f, 0xbf, + add_2reg(0xC0, src_reg, dst_reg)); + else if (num_bits =3D=3D 32) + EMIT3(add_2mod(0x48, src_reg, dst_reg), 0x63, + add_2reg(0xC0, src_reg, dst_reg)); + } else { + /* movs[b,w]l dst, src */ + if (num_bits =3D=3D 8) { + EMIT4(add_2mod(0x40, src_reg, dst_reg), 0x0f, 0xbe, + add_2reg(0xC0, src_reg, dst_reg)); + } else if (num_bits =3D=3D 16) { + if (is_ereg(dst_reg) || is_ereg(src_reg)) + EMIT1(add_2mod(0x40, src_reg, dst_reg)); + EMIT3(add_2mod(0x0f, src_reg, dst_reg), 0xbf, + add_2reg(0xC0, src_reg, dst_reg)); + } + } + + *pprog =3D prog; +} + /* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_= reg */ static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int o= ff) { @@ -1051,9 +1083,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *= addrs, u8 *image, u8 *rw_image =20 case BPF_ALU64 | BPF_MOV | BPF_X: case BPF_ALU | BPF_MOV | BPF_X: - emit_mov_reg(&prog, - BPF_CLASS(insn->code) =3D=3D BPF_ALU64, - dst_reg, src_reg); + if (insn->off =3D=3D 0) + emit_mov_reg(&prog, + BPF_CLASS(insn->code) =3D=3D BPF_ALU64, + dst_reg, src_reg); + else + emit_movsx_reg(&prog, insn->off, + BPF_CLASS(insn->code) =3D=3D BPF_ALU64, + dst_reg, src_reg); break; =20 /* neg dst */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 8a1cc658789e..fe648a158c9e 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -61,6 +61,7 @@ #define AX regs[BPF_REG_AX] #define ARG1 regs[BPF_REG_ARG1] #define CTX regs[BPF_REG_CTX] +#define OFF insn->off #define IMM insn->imm =20 struct bpf_mem_alloc bpf_global_ma; @@ -1736,13 +1737,36 @@ static u64 ___bpf_prog_run(u64 *regs, const struc= t bpf_insn *insn) DST =3D -DST; CONT; ALU_MOV_X: - DST =3D (u32) SRC; + switch (OFF) { + case 0: + DST =3D (u32) SRC; + break; + case 8: + DST =3D (u32)(s8) SRC; + break; + case 16: + DST =3D (u32)(s16) SRC; + break; + } CONT; ALU_MOV_K: DST =3D (u32) IMM; CONT; ALU64_MOV_X: - DST =3D SRC; + switch (OFF) { + case 0: + DST =3D SRC; + break; + case 8: + DST =3D (s8) SRC; + break; + case 16: + DST =3D (s16) SRC; + break; + case 32: + DST =3D (s32) SRC; + break; + } CONT; ALU64_MOV_K: DST =3D IMM; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index fbe4ca72d4c1..5fee9f24cb5e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3421,7 +3421,7 @@ static int backtrack_insn(struct bpf_verifier_env *= env, int idx, int subseq_idx, return 0; if (opcode =3D=3D BPF_MOV) { if (BPF_SRC(insn->code) =3D=3D BPF_X) { - /* dreg =3D sreg + /* dreg =3D sreg or dreg =3D (s8, s16, s32)sreg * dreg needs precision after this insn * sreg needs precision before this insn */ @@ -5866,6 +5866,64 @@ static void coerce_reg_to_size_sx(struct bpf_reg_s= tate *reg, int size) set_sext64_default_val(reg, size); } =20 +static void set_sext32_default_val(struct bpf_reg_state *reg, int size) +{ + if (size =3D=3D 1) { + reg->s32_min_value =3D S8_MIN; + reg->s32_max_value =3D S8_MAX; + } else { + /* size =3D=3D 2 */ + reg->s32_min_value =3D S16_MIN; + reg->s32_max_value =3D S16_MAX; + } + reg->u32_min_value =3D 0; + reg->u32_max_value =3D U32_MAX; +} + +static void coerce_subreg_to_size_sx(struct bpf_reg_state *reg, int size= ) +{ + s32 init_s32_max, init_s32_min, s32_max, s32_min; + u32 top_smax_value, top_smin_value; + u32 num_bits =3D size * 8; + + top_smax_value =3D ((u32)reg->s32_max_value >> num_bits) << num_bits; + top_smin_value =3D ((u32)reg->s32_min_value >> num_bits) << num_bits; + + if (top_smax_value !=3D top_smin_value) + goto out; + + /* find the s32_min and s32_min after sign extension */ + if (size =3D=3D 1) { + init_s32_max =3D (s8)reg->s32_max_value; + init_s32_min =3D (s8)reg->s32_min_value; + } else { + /* size =3D=3D 2 */ + init_s32_max =3D (s16)reg->s32_max_value; + init_s32_min =3D (s16)reg->s32_min_value; + } + s32_max =3D max(init_s32_max, init_s32_min); + s32_min =3D min(init_s32_max, init_s32_min); + + if (s32_min >=3D 0 && s32_max >=3D 0) { + reg->s32_min_value =3D s32_min; + reg->s32_max_value =3D s32_max; + reg->u32_min_value =3D 0; + reg->u32_max_value =3D U32_MAX; + return; + } + + if (s32_min < 0 && s32_max < 0) { + reg->s32_min_value =3D s32_min; + reg->s32_max_value =3D s32_max; + reg->u32_min_value =3D (u32)s32_max; + reg->u32_max_value =3D (u32)s32_min; + return; + } + +out: + set_sext32_default_val(reg, size); +} + static bool bpf_map_is_rdonly(const struct bpf_map *map) { /* A map is considered read-only if the following condition are true: @@ -13003,11 +13061,23 @@ static int check_alu_op(struct bpf_verifier_env= *env, struct bpf_insn *insn) } else if (opcode =3D=3D BPF_MOV) { =20 if (BPF_SRC(insn->code) =3D=3D BPF_X) { - if (insn->imm !=3D 0 || insn->off !=3D 0) { + if (insn->imm !=3D 0) { verbose(env, "BPF_MOV uses reserved fields\n"); return -EINVAL; } =20 + if (BPF_CLASS(insn->code) =3D=3D BPF_ALU) { + if (insn->off !=3D 0 && insn->off !=3D 8 && insn->off !=3D 16) { + verbose(env, "BPF_MOV uses reserved fields\n"); + return -EINVAL; + } + } else { + if (insn->off !=3D 0 && insn->off !=3D 8 && insn->off !=3D 16 && ins= n->off !=3D 32) { + verbose(env, "BPF_MOV uses reserved fields\n"); + return -EINVAL; + } + } + /* check src operand */ err =3D check_reg_arg(env, insn->src_reg, SRC_OP); if (err) @@ -13031,18 +13101,32 @@ static int check_alu_op(struct bpf_verifier_env= *env, struct bpf_insn *insn) !tnum_is_const(src_reg->var_off); =20 if (BPF_CLASS(insn->code) =3D=3D BPF_ALU64) { - /* case: R1 =3D R2 - * copy register state to dest reg - */ - if (need_id) - /* Assign src and dst registers the same ID - * that will be used by find_equal_scalars() - * to propagate min/max range. + if (insn->off =3D=3D 0) { + /* case: R1 =3D R2 + * copy register state to dest reg */ - src_reg->id =3D ++env->id_gen; - copy_register_state(dst_reg, src_reg); - dst_reg->live |=3D REG_LIVE_WRITTEN; - dst_reg->subreg_def =3D DEF_NOT_SUBREG; + if (need_id) + /* Assign src and dst registers the same ID + * that will be used by find_equal_scalars() + * to propagate min/max range. + */ + src_reg->id =3D ++env->id_gen; + copy_register_state(dst_reg, src_reg); + dst_reg->live |=3D REG_LIVE_WRITTEN; + dst_reg->subreg_def =3D DEF_NOT_SUBREG; + } else { + /* case: R1 =3D (s8, s16 s32)R2 */ + bool no_sext =3D src_reg->umax_value < (1ULL << (insn->off - 1)); + + if (no_sext && need_id) + src_reg->id =3D ++env->id_gen; + copy_register_state(dst_reg, src_reg); + if (!no_sext) + dst_reg->id =3D 0; + coerce_reg_to_size_sx(dst_reg, insn->off >> 3); + dst_reg->live |=3D REG_LIVE_WRITTEN; + dst_reg->subreg_def =3D DEF_NOT_SUBREG; + } } else { /* R1 =3D (u32) R2 */ if (is_pointer_value(env, insn->src_reg)) { @@ -13051,19 +13135,33 @@ static int check_alu_op(struct bpf_verifier_env= *env, struct bpf_insn *insn) insn->src_reg); return -EACCES; } else if (src_reg->type =3D=3D SCALAR_VALUE) { - bool is_src_reg_u32 =3D src_reg->umax_value <=3D U32_MAX; - - if (is_src_reg_u32 && need_id) - src_reg->id =3D ++env->id_gen; - copy_register_state(dst_reg, src_reg); - /* Make sure ID is cleared if src_reg is not in u32 range otherwise - * dst_reg min/max could be incorrectly - * propagated into src_reg by find_equal_scalars() - */ - if (!is_src_reg_u32) - dst_reg->id =3D 0; - dst_reg->live |=3D REG_LIVE_WRITTEN; - dst_reg->subreg_def =3D env->insn_idx + 1; + if (insn->off =3D=3D 0) { + bool is_src_reg_u32 =3D src_reg->umax_value <=3D U32_MAX; + + if (is_src_reg_u32 && need_id) + src_reg->id =3D ++env->id_gen; + copy_register_state(dst_reg, src_reg); + /* Make sure ID is cleared if src_reg is not in u32 range otherwis= e + * dst_reg min/max could be incorrectly + * propagated into src_reg by find_equal_scalars() + */ + if (!is_src_reg_u32) + dst_reg->id =3D 0; + dst_reg->live |=3D REG_LIVE_WRITTEN; + dst_reg->subreg_def =3D env->insn_idx + 1; + } else { + /* case: W1 =3D (s8, s16)W2 */ + bool no_sext =3D src_reg->umax_value < (1ULL << (insn->off - 1)); + + if (no_sext && need_id) + src_reg->id =3D ++env->id_gen; + copy_register_state(dst_reg, src_reg); + if (!no_sext) + dst_reg->id =3D 0; + dst_reg->live |=3D REG_LIVE_WRITTEN; + dst_reg->subreg_def =3D env->insn_idx + 1; + coerce_subreg_to_size_sx(dst_reg, insn->off >> 3); + } } else { mark_reg_unknown(env, regs, insn->dst_reg); --=20 2.34.1