From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67523C61CE4 for ; Sat, 19 Jan 2019 18:07:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 367A92087E for ; Sat, 19 Jan 2019 18:07:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="wJVZhe/5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729056AbfASSH3 (ORCPT ); Sat, 19 Jan 2019 13:07:29 -0500 Received: from mail-qk1-f194.google.com ([209.85.222.194]:33114 "EHLO mail-qk1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728996AbfASSHX (ORCPT ); Sat, 19 Jan 2019 13:07:23 -0500 Received: by mail-qk1-f194.google.com with SMTP id d15so9931154qkj.0 for ; Sat, 19 Jan 2019 10:07:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dXEbksO9uhoZ15d0RFoladERzOkceAvEeAQC4y5ItSk=; b=wJVZhe/5EGoRdvxaTvYksETBEZGaZcg66PmcK27C5aWuxkYjkY7vmdr+ECFJu2s6bL 0WKx9P0PVxZN7MoVWt3dzf14f/HOo0/Qosv74afUFoyoGj0kkEs09sqp1G4Z2MVb3RYE ie5dDiy9cT6t4vhTA0oldcvzEkNJCvE5PH6KkPxnCm2112ynPihQMOmAD92O422Mswpk 6NEoxzlOszOi93rQS1L5CHacILOjdExP35jZV3WsiKHZH6uTAXGqXRK59ZHJcQEFrSzA /WN2FqOxeSMRgLFJb43HIi6EgVvR5CLZMEWBdYGiV+XQp72i75xuBOp1Mu09DUktDJMR 7Krw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dXEbksO9uhoZ15d0RFoladERzOkceAvEeAQC4y5ItSk=; b=j4OULhK7LI+auZLIixoVza2zwY8ccamH11lZ+QOTZDOHQJdNkChUvVsLDqxM2bh3/c x4DC41IyndDgPpwVs2A4z5sGJ7liV9brfrr6BwmoU+h1fDyfTKIxI5LNx0p5stN7A5Eq PIo7oeCWf427x1uiMZyDrgHz0LrrfwUxOINeXAL4NieCfg6mahFljnqsSJgq9K6Qh9h0 FSy6utlZXAWVXVIiZPzUaImicRmrSat+0l9kBxWh37DVcGEyazI//s91qLAK9S+pPrJ5 e8QrsyLY17a0fG9NsfZbdn/W04g/h1lF4MRxFxuDqcVSgd+YBTcwbAci6RQEgxpQJDWh DVGw== X-Gm-Message-State: AJcUukceXzqP02nf/N9XvCWcm5JVt0J/4s+Qv48uFKLgVF2M2W+cvZDM K8k9C6QwAITk2geNLdVr/nTR8w== X-Google-Smtp-Source: ALg8bN6sHsFg2nuTloalu9961NsjTRjnkbDVmWbIcGowl45Oz6xbOPmNQAA2Od73BaSAQQGAcZ+Ezw== X-Received: by 2002:a37:9046:: with SMTP id s67mr18247736qkd.289.1547921242556; Sat, 19 Jan 2019 10:07:22 -0800 (PST) Received: from jkicinski-Precision-T1700.netronome.com ([66.60.152.14]) by smtp.gmail.com with ESMTPSA id v2sm46521518qte.75.2019.01.19.10.07.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 Jan 2019 10:07:22 -0800 (PST) From: Jakub Kicinski To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: netdev@vger.kernel.org, oss-drivers@netronome.com, yhs@fb.com, Jakub Kicinski Subject: [PATCH bpf-next v5 07/12] bpf: notify offload JITs about optimizations Date: Sat, 19 Jan 2019 10:06:51 -0800 Message-Id: <20190119180656.24635-8-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190119180656.24635-1-jakub.kicinski@netronome.com> References: <20190119180656.24635-1-jakub.kicinski@netronome.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Let offload JITs know when instructions are replaced and optimized out, so they can update their state appropriately. The optimizations are best effort, if JIT returns an error from any callback verifier will stop notifying it as state may now be out of sync, but the verifier continues making progress. Signed-off-by: Jakub Kicinski Reviewed-by: Quentin Monnet --- include/linux/bpf.h | 7 +++++++ include/linux/bpf_verifier.h | 5 +++++ kernel/bpf/offload.c | 35 +++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++++++ 4 files changed, 53 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e734f163bd0b..3851529062ec 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -268,9 +268,15 @@ struct bpf_verifier_ops { }; struct bpf_prog_offload_ops { + /* verifier basic callbacks */ int (*insn_hook)(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx); int (*finalize)(struct bpf_verifier_env *env); + /* verifier optimization callbacks (called after .finalize) */ + int (*replace_insn)(struct bpf_verifier_env *env, u32 off, + struct bpf_insn *insn); + int (*remove_insns)(struct bpf_verifier_env *env, u32 off, u32 cnt); + /* program management callbacks */ int (*prepare)(struct bpf_prog *prog); int (*translate)(struct bpf_prog *prog); void (*destroy)(struct bpf_prog *prog); @@ -283,6 +289,7 @@ struct bpf_prog_offload { void *dev_priv; struct list_head offloads; bool dev_state; + bool opt_failed; void *jited_image; u32 jited_len; }; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index f3ae00ee5516..0620e418dde5 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -266,5 +266,10 @@ int bpf_prog_offload_verifier_prep(struct bpf_prog *prog); int bpf_prog_offload_verify_insn(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx); int bpf_prog_offload_finalize(struct bpf_verifier_env *env); +void +bpf_prog_offload_replace_insn(struct bpf_verifier_env *env, u32 off, + struct bpf_insn *insn); +void +bpf_prog_offload_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt); #endif /* _LINUX_BPF_VERIFIER_H */ diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 54cf2b9c44a4..39dba8c90331 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -173,6 +173,41 @@ int bpf_prog_offload_finalize(struct bpf_verifier_env *env) return ret; } +void +bpf_prog_offload_replace_insn(struct bpf_verifier_env *env, u32 off, + struct bpf_insn *insn) +{ + const struct bpf_prog_offload_ops *ops; + struct bpf_prog_offload *offload; + int ret = -EOPNOTSUPP; + + down_read(&bpf_devs_lock); + offload = env->prog->aux->offload; + if (offload) { + ops = offload->offdev->ops; + if (!offload->opt_failed && ops->replace_insn) + ret = ops->replace_insn(env, off, insn); + offload->opt_failed |= ret; + } + up_read(&bpf_devs_lock); +} + +void +bpf_prog_offload_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) +{ + struct bpf_prog_offload *offload; + int ret = -EOPNOTSUPP; + + down_read(&bpf_devs_lock); + offload = env->prog->aux->offload; + if (offload) { + if (!offload->opt_failed && offload->offdev->ops->remove_insns) + ret = offload->offdev->ops->remove_insns(env, off, cnt); + offload->opt_failed |= ret; + } + up_read(&bpf_devs_lock); +} + static void __bpf_prog_offload_destroy(struct bpf_prog *prog) { struct bpf_prog_offload *offload = prog->aux->offload; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index fdb5b9aa5fa0..7ce8322c330c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6551,6 +6551,9 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) unsigned int orig_prog_len = env->prog->len; int err; + if (bpf_prog_is_dev_bound(env->prog->aux)) + bpf_prog_offload_remove_insns(env, off, cnt); + err = bpf_remove_insns(env->prog, off, cnt); if (err) return err; @@ -6625,6 +6628,9 @@ static void opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env) else continue; + if (bpf_prog_is_dev_bound(env->prog->aux)) + bpf_prog_offload_replace_insn(env, i, &ja); + memcpy(insn, &ja, sizeof(ja)); } } -- 2.19.2