From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1069C282C0 for ; Wed, 23 Jan 2019 06:46:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6F47B21726 for ; Wed, 23 Jan 2019 06:46:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="U1kpqxWZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726934AbfAWGqL (ORCPT ); Wed, 23 Jan 2019 01:46:11 -0500 Received: from mail-qt1-f196.google.com ([209.85.160.196]:37711 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726296AbfAWGpz (ORCPT ); Wed, 23 Jan 2019 01:45:55 -0500 Received: by mail-qt1-f196.google.com with SMTP id t33so1230454qtt.4 for ; Tue, 22 Jan 2019 22:45:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sZ13AbHnuvLZVKbeynrGIqewmS2+tKKDMbUUW5WomSc=; b=U1kpqxWZL4o4C1+ZefDanfnPDEYB5OSmxOwKKwMxg3mNTE7bF6ceV6/0OuTdwzcz9M JMgPyBmMwqegkVijvSryccgxFs83yzw0VtNFNvFdBPg+dPqR7avzCu1DuMvN+tCpHLHp jbhe01XrOdtlSYBh0V65jCYLDv1iMuutZ2fumuqdS1sAfOB9Y0eCLY7oVkGWe0RSXLzS nwGRKTkrF+GKItu3oypdSI0X2KxDL3AL9AgZ2Nz1qE3LoY/8U30ueWhRkC4STDj1tB+p lQDo+sA1keJ+8cxMjUXpE7RpxdqvswOOjsCukOUuf7mUmiNa//jpAM4OJjNhJFPVhT0J ey5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sZ13AbHnuvLZVKbeynrGIqewmS2+tKKDMbUUW5WomSc=; b=LBn6V7b+TRUaorP5qeuwQm0BnIUIytg24efW25KgxhBmOILu6za88f9j9/qyPUs9qN GEoA1QHDtgRujhdgaPqzNjw3oRqv+4OnI2EiHs1ACOnRcBb2uaoA7uHs6+3rDxt0e8Qi p6DquMOIhN+mMYT19aJEW04LnV+2dbzD+NuOr2pglpceHgrUZ+vNVKWQIOYluudF0Yal mXsbIc76suiYhWU73CFKDdMZcqZULFOvKxXpCvBzX7h7/w0xAvvZq4sHkQBgxIP4ft9c CK5L2MI5VB9JZ/MvMhg/LzmH6C1W4BabYmDRFcK6MWM63iEM7d6byFJTGp1wAQChHJiS iqmQ== X-Gm-Message-State: AJcUukcuyA3SgItIiwhWnn9Xw9Xlxll4kebsl3wTfeagjkzoWGHd2he+ 1a2WEhNGYpS2XAzZ6wzw04fuzw== X-Google-Smtp-Source: ALg8bN7T34Wl/ixBW0bCSOdjk+Eh5B036gDTNCCYCx0bwkJB5mSNRTNVzT3tjBha/nKzgLUYQh1MwA== X-Received: by 2002:aed:3a22:: with SMTP id n31mr1013633qte.29.1548225953728; Tue, 22 Jan 2019 22:45:53 -0800 (PST) Received: from jkicinski-Precision-T1700.netronome.com ([66.60.152.14]) by smtp.gmail.com with ESMTPSA id k6sm80589046qtk.41.2019.01.22.22.45.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Jan 2019 22:45:53 -0800 (PST) From: Jakub Kicinski To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: yhs@fb.com, netdev@vger.kernel.org, oss-drivers@netronome.com, kafai@fb.com, Jakub Kicinski Subject: [PATCH bpf-next v5 07/12] bpf: notify offload JITs about optimizations Date: Tue, 22 Jan 2019 22:45:24 -0800 Message-Id: <20190123064529.13518-8-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190123064529.13518-1-jakub.kicinski@netronome.com> References: <20190123064529.13518-1-jakub.kicinski@netronome.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Let offload JITs know when instructions are replaced and optimized out, so they can update their state appropriately. The optimizations are best effort, if JIT returns an error from any callback verifier will stop notifying it as state may now be out of sync, but the verifier continues making progress. Signed-off-by: Jakub Kicinski Reviewed-by: Quentin Monnet --- include/linux/bpf.h | 7 +++++++ include/linux/bpf_verifier.h | 5 +++++ kernel/bpf/offload.c | 35 +++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++++++ 4 files changed, 53 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e734f163bd0b..3851529062ec 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -268,9 +268,15 @@ struct bpf_verifier_ops { }; struct bpf_prog_offload_ops { + /* verifier basic callbacks */ int (*insn_hook)(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx); int (*finalize)(struct bpf_verifier_env *env); + /* verifier optimization callbacks (called after .finalize) */ + int (*replace_insn)(struct bpf_verifier_env *env, u32 off, + struct bpf_insn *insn); + int (*remove_insns)(struct bpf_verifier_env *env, u32 off, u32 cnt); + /* program management callbacks */ int (*prepare)(struct bpf_prog *prog); int (*translate)(struct bpf_prog *prog); void (*destroy)(struct bpf_prog *prog); @@ -283,6 +289,7 @@ struct bpf_prog_offload { void *dev_priv; struct list_head offloads; bool dev_state; + bool opt_failed; void *jited_image; u32 jited_len; }; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index f3ae00ee5516..0620e418dde5 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -266,5 +266,10 @@ int bpf_prog_offload_verifier_prep(struct bpf_prog *prog); int bpf_prog_offload_verify_insn(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx); int bpf_prog_offload_finalize(struct bpf_verifier_env *env); +void +bpf_prog_offload_replace_insn(struct bpf_verifier_env *env, u32 off, + struct bpf_insn *insn); +void +bpf_prog_offload_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt); #endif /* _LINUX_BPF_VERIFIER_H */ diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 54cf2b9c44a4..39dba8c90331 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -173,6 +173,41 @@ int bpf_prog_offload_finalize(struct bpf_verifier_env *env) return ret; } +void +bpf_prog_offload_replace_insn(struct bpf_verifier_env *env, u32 off, + struct bpf_insn *insn) +{ + const struct bpf_prog_offload_ops *ops; + struct bpf_prog_offload *offload; + int ret = -EOPNOTSUPP; + + down_read(&bpf_devs_lock); + offload = env->prog->aux->offload; + if (offload) { + ops = offload->offdev->ops; + if (!offload->opt_failed && ops->replace_insn) + ret = ops->replace_insn(env, off, insn); + offload->opt_failed |= ret; + } + up_read(&bpf_devs_lock); +} + +void +bpf_prog_offload_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) +{ + struct bpf_prog_offload *offload; + int ret = -EOPNOTSUPP; + + down_read(&bpf_devs_lock); + offload = env->prog->aux->offload; + if (offload) { + if (!offload->opt_failed && offload->offdev->ops->remove_insns) + ret = offload->offdev->ops->remove_insns(env, off, cnt); + offload->opt_failed |= ret; + } + up_read(&bpf_devs_lock); +} + static void __bpf_prog_offload_destroy(struct bpf_prog *prog) { struct bpf_prog_offload *offload = prog->aux->offload; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index f2c49b4235df..8cfe39ef770f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6558,6 +6558,9 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) unsigned int orig_prog_len = env->prog->len; int err; + if (bpf_prog_is_dev_bound(env->prog->aux)) + bpf_prog_offload_remove_insns(env, off, cnt); + err = bpf_remove_insns(env->prog, off, cnt); if (err) return err; @@ -6632,6 +6635,9 @@ static void opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env) else continue; + if (bpf_prog_is_dev_bound(env->prog->aux)) + bpf_prog_offload_replace_insn(env, i, &ja); + memcpy(insn, &ja, sizeof(ja)); } } -- 2.19.2