[PATCH bpf-next 02/13] bpf: fetch the vmlinux BTF where kernel types enter a program

[Jay Wang <wanjay@amazon.com>]

bpf_check() fetches the vmlinux BTF up front for every program, whether
the program uses kernel types or not.  That is free today because the
BTF is parsed at boot anyway.  Once it can be parsed on first use
(CONFIG_DEBUG_INFO_BTF_LAZY, later in this series), it would make every
program load pay for parsing 5-7 MB of BTF, including the socket filters
systemd loads at boot.

Drop the up-front fetch and fetch where kernel types enter the verifier
state instead:

 - bpf_add_kfunc_call(), for the first kfunc call of a program;
 - check_ptr_to_map_access(), for map_ptr field access;
 - check_pseudo_btf_id(), for ldimm64 of a kernel variable (ksym);
 - check_helper_call(), for helpers whose prototype takes
   ARG_PTR_TO_BTF_ID or returns RET_PTR_TO_BTF_ID, and for
   bpf_snprintf_btf() and bpf_seq_printf_btf(), which take a kernel
   type id inside struct btf_ptr (helper_uses_vmlinux_btf()).

The other ways in already fetch the BTF themselves: attach_btf in
bpf_prog_load(), struct_ops map creation, CO-RE candidate lookup,
bpf_find_btf_id(), btf_prepare_func_args() for kernel-typed arguments
and the kernel context type lookups.  The remaining reads of the
btf_vmlinux global in the verifier are on paths dominated by one of
these fetches.  A program that uses none of them, such as a socket
filter, does not need the vmlinux BTF any more, and verifies the same
whether or not it has been parsed.

Four related changes keep it that way:

 - The "in-kernel BTF is malformed" check in bpf_check() read the
   global right after the up-front fetch.  Without the fetch it would
   pass or fail depending on whether something else had parsed the BTF
   before, so drop it; every fetch point reports a missing BTF itself
   and passes on an error of the parse, as that check did.

 - check_ptr_to_map_access() checks allow_ptr_leaks before it fetches
   the BTF, and bpf_prog_load() falls back to the vmlinux BTF for an
   attach_btf_id only with CAP_BPF, which all program types that take
   one need, so that an unprivileged load cannot make the kernel parse
   it only to be refused.  The map_ptr permission error can then not
   name the map's struct any more and says "map_ptr access is allowed
   only to CAP_PERFMON and CAP_SYS_ADMIN".  A ksym, which unprivileged
   programs may use, still parses it.

 - Add bpf_peek_btf_vmlinux(), which returns the vmlinux BTF only if it
   is parsed already, and use it in bpf_btf_printf_prepare().  The BTF
   printf helpers run in program context, which may be atomic, and the
   verifier has fetched the BTF for every program that calls them.

 - is_kfunc_dynptr_write(), reached from the argument checks that
   helpers and kfuncs share, compared meta->btf with the global, which
   a helper's NULL matches while the global is NULL.  Check that the
   call is a kfunc first, as the other kfunc predicates there do.

Helpers taking ARG_PTR_TO_BTF_ID_SOCK_COMMON are left out on purpose:
they take sockets without BTF types too, the shared argument check reads
the global only for a PTR_TO_BTF_ID register, and every source of such
a register fetched the BTF.

No functional change with CONFIG_DEBUG_INFO_BTF=y, where kfunc
registrations at boot parse the BTF, other than that error message and
that, if the vmlinux BTF fails to parse (a broken toolchain), programs
that use no kernel types now load instead of failing.  Without BTF the
new helper check is compiled out and the existing error paths are
unchanged.

Co-developed-by: Ihor Solodrai <ihor.solodrai@linux.dev>
TODO-NEEDS-IHOR: Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 include/linux/bpf.h      |  1 +
 kernel/bpf/syscall.c     |  8 +++-
 kernel/bpf/verifier.c    | 98 ++++++++++++++++++++++++++++++----------
 kernel/trace/bpf_trace.c |  3 +-
 4 files changed, 83 insertions(+), 27 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 580f89c450b5..1fd6ef2ef5c6 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -3257,6 +3257,7 @@ static inline s32 bpf_call_args_imm(s16 idx)
 #endif
 
 struct btf *bpf_get_btf_vmlinux(void);
+struct btf *bpf_peek_btf_vmlinux(void);
 
 /* Map specifics */
 struct xdp_frame;
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index ca72b35cdcc9..9273b7dee0aa 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -3052,8 +3052,12 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
 				goto put_token;
 			}
 		}
-	} else if (attr->attach_btf_id || multi_func) {
-		/* fall back to vmlinux BTF, if BTF type ID is specified */
+	} else if ((attr->attach_btf_id || multi_func) && bpf_cap) {
+		/*
+		 * fall back to vmlinux BTF, if BTF type ID is specified; only
+		 * program types that need CAP_BPF take one, and
+		 * bpf_prog_load_check_attach() refuses it for the others
+		 */
 		attach_btf = bpf_get_btf_vmlinux();
 		if (IS_ERR(attach_btf)) {
 			err = PTR_ERR(attach_btf);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5601d2056218..1b901e3cb3c2 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3060,6 +3060,18 @@ static int fetch_kfunc_meta(struct bpf_verifier_env *env,
 static int gen_kfunc_arg_proto(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 			       const struct btf_func_model *fm, struct bpf_func_proto *proto);
 
+/*
+ * The vmlinux BTF failed to parse where a program first needs it: pass the
+ * error on, as bpf_check() did when it fetched the BTF up front.  It need
+ * not be for good: a parse on first use that runs out of memory is retried
+ * by the next caller.
+ */
+static int vmlinux_btf_err(struct bpf_verifier_env *env, const struct btf *btf)
+{
+	verbose(env, "in-kernel BTF could not be parsed: %ld\n", PTR_ERR(btf));
+	return PTR_ERR(btf);
+}
+
 int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 {
 	struct bpf_call_arg_meta meta;
@@ -3076,7 +3088,11 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	tab = prog_aux->kfunc_tab;
 	btf_tab = prog_aux->kfunc_btf_tab;
 	if (!tab) {
-		if (!btf_vmlinux) {
+		struct btf *btf = bpf_get_btf_vmlinux();
+
+		if (IS_ERR(btf))
+			return vmlinux_btf_err(env, btf);
+		if (!btf) {
 			verbose(env, "calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF\n");
 			return -ENOTSUPP;
 		}
@@ -6867,30 +6883,33 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,
 	enum bpf_type_flag flag = 0;
 	const struct btf_type *t;
 	const char *tname;
+	struct btf *btf;
 	u32 btf_id;
 	int ret;
 
-	if (!btf_vmlinux) {
-		verbose(env, "map_ptr access not supported without CONFIG_DEBUG_INFO_BTF\n");
-		return -ENOTSUPP;
-	}
-
 	if (!map->ops->map_btf_id || !*map->ops->map_btf_id) {
 		verbose(env, "map_ptr access not supported for map type %d\n",
 			map->map_type);
 		return -ENOTSUPP;
 	}
 
-	t = btf_type_by_id(btf_vmlinux, *map->ops->map_btf_id);
-	tname = btf_name_by_offset(btf_vmlinux, t->name_off);
-
+	/* before the BTF fetch: unprivileged loads must not parse it */
 	if (!env->allow_ptr_leaks) {
-		verbose(env,
-			"'struct %s' access is allowed only to CAP_PERFMON and CAP_SYS_ADMIN\n",
-			tname);
+		verbose(env, "map_ptr access is allowed only to CAP_PERFMON and CAP_SYS_ADMIN\n");
 		return -EPERM;
 	}
 
+	btf = bpf_get_btf_vmlinux();
+	if (IS_ERR(btf))
+		return vmlinux_btf_err(env, btf);
+	if (!btf) {
+		verbose(env, "map_ptr access not supported without CONFIG_DEBUG_INFO_BTF\n");
+		return -ENOTSUPP;
+	}
+
+	t = btf_type_by_id(btf, *map->ops->map_btf_id);
+	tname = btf_name_by_offset(btf, t->name_off);
+
 	if (off < 0) {
 		verbose(env, "%s is %s invalid negative access: off=%d\n",
 			reg_arg_name(env, argno), tname, off);
@@ -12606,6 +12625,22 @@ static int release_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 	return err;
 }
 
+/* Does helper @fn bring kernel BTF types into the program? */
+static bool helper_uses_vmlinux_btf(enum bpf_func_id func_id, const struct bpf_func_proto *fn)
+{
+	int i;
+
+	/* these take a kernel type id in struct btf_ptr */
+	if (func_id == BPF_FUNC_snprintf_btf || func_id == BPF_FUNC_seq_printf_btf)
+		return true;
+	if (base_type(fn->ret_type) == RET_PTR_TO_BTF_ID)
+		return true;
+	for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++)
+		if (base_type(fn->arg_type[i]) == ARG_PTR_TO_BTF_ID)
+			return true;
+	return false;
+}
+
 static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			     int *insn_idx_p)
 {
@@ -12675,6 +12710,14 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 		return err;
 	}
 
+	if (IS_ENABLED(CONFIG_DEBUG_INFO_BTF) && helper_uses_vmlinux_btf(func_id, fn)) {
+		/* never NULL with CONFIG_DEBUG_INFO_BTF */
+		struct btf *btf = bpf_get_btf_vmlinux();
+
+		if (IS_ERR(btf))
+			return vmlinux_btf_err(env, btf);
+	}
+
 	if (fn->might_sleep && !in_sleepable_context(env)) {
 		const char *suggestion;
 
@@ -13900,7 +13943,7 @@ BTF_SET_END(dynptr_write_kfuncs)
 
 static bool is_kfunc_dynptr_write(struct bpf_call_arg_meta *meta)
 {
-	return meta->btf == btf_vmlinux &&
+	return is_kfunc(meta) && meta->btf == btf_vmlinux &&
 	       btf_id_set_contains(&dynptr_write_kfuncs, meta->func_id);
 }
 
@@ -20898,12 +20941,14 @@ static int check_pseudo_btf_id(struct bpf_verifier_env *env,
 			return -EINVAL;
 		}
 	} else {
-		if (!btf_vmlinux) {
+		btf = bpf_get_btf_vmlinux();
+		if (IS_ERR(btf))
+			return vmlinux_btf_err(env, btf);
+		if (!btf) {
 			verbose(env, "kernel is missing BTF, make sure CONFIG_DEBUG_INFO_BTF=y is specified in Kconfig.\n");
 			return -EINVAL;
 		}
-		btf_get(btf_vmlinux);
-		btf = btf_vmlinux;
+		btf_get(btf);
 	}
 
 	err = __check_pseudo_btf_id(env, insn, aux, btf);
@@ -23122,6 +23167,13 @@ struct btf *bpf_get_btf_vmlinux(void)
 	return btf;
 }
 
+/* The vmlinux BTF if it is parsed already: for callers that cannot sleep */
+struct btf *bpf_peek_btf_vmlinux(void)
+{
+	/* Pairs with the smp_store_release() in bpf_get_btf_vmlinux() */
+	return smp_load_acquire(&btf_vmlinux);
+}
+
 /*
  * The add_fd_from_fd_array() is executed only if fd_array_cnt is non-zero. In
  * this case expect that every file descriptor in the array is either a map or
@@ -23845,7 +23897,12 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret)
 		goto err_prep;
 
-	bpf_get_btf_vmlinux();
+	/*
+	 * The vmlinux BTF is not fetched here but where kernel types enter
+	 * the program (attach_btf, kfuncs, ksyms, map_ptr, BTF-typed helpers,
+	 * CO-RE, kernel ctx types), so that programs which use none of them
+	 * do not need it (CONFIG_DEBUG_INFO_BTF_LAZY).
+	 */
 
 	/* Serialize verification of unprivileged programs. */
 	if (!is_priv)
@@ -23866,13 +23923,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 
 	mark_verifier_state_clean(env);
 
-	if (IS_ERR(btf_vmlinux)) {
-		/* Either gcc or pahole or kernel are broken. */
-		verbose(env, "in-kernel BTF is malformed\n");
-		ret = PTR_ERR(btf_vmlinux);
-		goto skip_full_check;
-	}
-
 	env->strict_alignment = !!(attr->prog_flags & BPF_F_STRICT_ALIGNMENT);
 	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS))
 		env->strict_alignment = true;
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index dfac5e1b8f40..84335e594388 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1018,7 +1018,8 @@ static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
 	if (btf_ptr_size != sizeof(struct btf_ptr))
 		return -EINVAL;
 
-	*btf = bpf_get_btf_vmlinux();
+	/* fetched by the verifier, see helper_uses_vmlinux_btf() */
+	*btf = bpf_peek_btf_vmlinux();
 
 	if (IS_ERR_OR_NULL(*btf))
 		return IS_ERR(*btf) ? PTR_ERR(*btf) : -EINVAL;
-- 
2.47.3