[PATCH bpf-next 0/2] Fix libbpf BPF skeleton forward/backward compat

[PATCH bpf-next 0/2] Fix libbpf BPF skeleton forward/backward compat

[Andrii Nakryiko]

Fix recently identified (but long standing) bug with handling BPF skeleton
forward and backward compatibility. On libbpf side, even though BPF skeleton
was always designed to be forward and backwards compatible through recording
actual size of constrituents of BPF skeleton itself (map/prog/var skeleton
definitions), libbpf implementation did implicitly hard-code those sizes by
virtue of using a trivial array access syntax.

This issue will only affect libbpf used as a shared library. Statically
compiled libbpfs will always be in sync with BPF skeleton, bypassing this
problem altogether.

This patch set fixes libbpf, but also mitigates the problem for old libbpf
versions by teaching bpftool to generate more conservative BPF skeleton,
if possible (i.e., if there are no struct_ops maps defined).

Andrii Nakryiko (2):
  bpftool: improve skeleton backwards compat with old buggy libbpfs
  libbpf: fix BPF skeleton forward/backward compat handling

 tools/bpf/bpftool/gen.c | 46 ++++++++++++++++++++++++++++------------
 tools/lib/bpf/libbpf.c  | 47 +++++++++++++++++++++++------------------
 2 files changed, 59 insertions(+), 34 deletions(-)

-- 
2.43.0

[PATCH bpf-next 1/2] bpftool: improve skeleton backwards compat with old buggy libbpfs

[Andrii Nakryiko]

Old versions of libbpf don't handle varying sizes of bpf_map_skeleton
struct correctly. As such, BPF skeleton generated by newest bpftool
might not be compatible with older libbpf (though only when libbpf is
used as a shared library), even though it, by design, should.

Going forward libbpf will be fixed, plus we'll release bug fixed
versions of relevant old libbpfs, but meanwhile try to mitigate from
bpftool side by conservatively assuming older and smaller definition of
bpf_map_skeleton, if possible. Meaning, if there are no struct_ops maps.

If there are struct_ops, then presumably user would like to have
auto-attaching logic and struct_ops map link placeholders, so use the
full bpf_map_skeleton definition in that case.

Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
---
 tools/bpf/bpftool/gen.c | 46 ++++++++++++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 14 deletions(-)

diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c
index 51eaed76db97..70aaa32ddcc9 100644
--- a/tools/bpf/bpftool/gen.c
+++ b/tools/bpf/bpftool/gen.c
@@ -852,24 +852,41 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
 {
 	struct bpf_map *map;
 	char ident[256];
-	size_t i;
+	size_t i, map_sz;
 
 	if (!map_cnt)
 		return;
 
+	/* for backward compatibility with old libbpf versions that don't
+	 * handle new BPF skeleton with new struct bpf_map_skeleton definition
+	 * that includes link field, avoid specifying new increased size,
+	 * unless we absolutely have to (i.e., if there are struct_ops maps
+	 * present)
+	 */
+	map_sz = offsetof(struct bpf_map_skeleton, link);
+	if (populate_links) {
+		bpf_object__for_each_map(map, obj) {
+			if (bpf_map__type(map) == BPF_MAP_TYPE_STRUCT_OPS) {
+				map_sz = sizeof(struct bpf_map_skeleton);
+				break;
+			}
+		}
+	}
+
 	codegen("\
 		\n\
-									\n\
+								    \n\
 			/* maps */				    \n\
 			s->map_cnt = %zu;			    \n\
-			s->map_skel_sz = sizeof(*s->maps);	    \n\
-			s->maps = (struct bpf_map_skeleton *)calloc(s->map_cnt, s->map_skel_sz);\n\
+			s->map_skel_sz = %zu;			    \n\
+			s->maps = (struct bpf_map_skeleton *)calloc(s->map_cnt,\n\
+					sizeof(*s->maps) > %zu ? sizeof(*s->maps) : %zu);\n\
 			if (!s->maps) {				    \n\
 				err = -ENOMEM;			    \n\
 				goto err;			    \n\
 			}					    \n\
 		",
-		map_cnt
+		map_cnt, map_sz, map_sz, map_sz
 	);
 	i = 0;
 	bpf_object__for_each_map(map, obj) {
@@ -878,23 +895,22 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
 
 		codegen("\
 			\n\
-									\n\
-				s->maps[%zu].name = \"%s\";	    \n\
-				s->maps[%zu].map = &obj->maps.%s;   \n\
+								    \n\
+				map = (struct bpf_map_skeleton *)((char *)s->maps + %zu * s->map_skel_sz);\n\
+				map->name = \"%s\";		    \n\
+				map->map = &obj->maps.%s;	    \n\
 			",
-			i, bpf_map__name(map), i, ident);
+			i, bpf_map__name(map), ident);
 		/* memory-mapped internal maps */
 		if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
-			printf("\ts->maps[%zu].mmaped = (void **)&obj->%s;\n",
-				i, ident);
+			printf("\tmap->mmaped = (void **)&obj->%s;  \n", ident);
 		}
 
 		if (populate_links && bpf_map__type(map) == BPF_MAP_TYPE_STRUCT_OPS) {
 			codegen("\
 				\n\
-					s->maps[%zu].link = &obj->links.%s;\n\
-				",
-				i, ident);
+					map->link = &obj->links.%s; \n\
+				", ident);
 		}
 		i++;
 	}
@@ -1463,6 +1479,7 @@ static int do_skeleton(int argc, char **argv)
 		%1$s__create_skeleton(struct %1$s *obj)			    \n\
 		{							    \n\
 			struct bpf_object_skeleton *s;			    \n\
+			struct bpf_map_skeleton *map __attribute__((unused));\n\
 			int err;					    \n\
 									    \n\
 			s = (struct bpf_object_skeleton *)calloc(1, sizeof(*s));\n\
@@ -1753,6 +1770,7 @@ static int do_subskeleton(int argc, char **argv)
 		{							    \n\
 			struct %1$s *obj;				    \n\
 			struct bpf_object_subskeleton *s;		    \n\
+			struct bpf_map_skeleton *map __attribute__((unused));\n\
 			int err;					    \n\
 									    \n\
 			obj = (struct %1$s *)calloc(1, sizeof(*obj));	    \n\
-- 
2.43.0

[PATCH bpf-next 2/2] libbpf: fix BPF skeleton forward/backward compat handling

[Andrii Nakryiko]

BPF skeleton was designed from day one to be extensible. Generated BPF
skeleton code specifies actual sizes of map/prog/variable skeletons for
that reason and libbpf is supposed to work with newer/older versions
correctly.

Unfortunately, it was missed that we implicitly embed hard-coded most
up-to-date (according to libbpf's version of libbpf.h header used to
compile BPF skeleton header) sizes of those strucs, which can differ
from the actual sizes at runtime when libbpf is used as a shared
library.

We have a few places were we just index array of maps/progs/vars, which
implicitly uses these potentially invalid sizes of structs.

This patch aims to fix this problem going forward. Once this lands,
we'll backport these changes in Github repo to create patched releases
for older libbpfs.

Fixes: d66562fba1ce ("libbpf: Add BPF object skeleton support")
Fixes: 430025e5dca5 ("libbpf: Add subskeleton scaffolding")
Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
---
 tools/lib/bpf/libbpf.c | 47 ++++++++++++++++++++++++------------------
 1 file changed, 27 insertions(+), 20 deletions(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 4a28fac4908a..e92f933267d1 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -13712,14 +13712,15 @@ int libbpf_num_possible_cpus(void)
 
 static int populate_skeleton_maps(const struct bpf_object *obj,
 				  struct bpf_map_skeleton *maps,
-				  size_t map_cnt)
+				  size_t map_cnt, size_t map_skel_sz)
 {
 	int i;
 
 	for (i = 0; i < map_cnt; i++) {
-		struct bpf_map **map = maps[i].map;
-		const char *name = maps[i].name;
-		void **mmaped = maps[i].mmaped;
+		struct bpf_map_skeleton *map_skel = (void *)maps + i * map_skel_sz;
+		struct bpf_map **map = map_skel->map;
+		const char *name = map_skel->name;
+		void **mmaped = map_skel->mmaped;
 
 		*map = bpf_object__find_map_by_name(obj, name);
 		if (!*map) {
@@ -13736,13 +13737,14 @@ static int populate_skeleton_maps(const struct bpf_object *obj,
 
 static int populate_skeleton_progs(const struct bpf_object *obj,
 				   struct bpf_prog_skeleton *progs,
-				   size_t prog_cnt)
+				   size_t prog_cnt, size_t prog_skel_sz)
 {
 	int i;
 
 	for (i = 0; i < prog_cnt; i++) {
-		struct bpf_program **prog = progs[i].prog;
-		const char *name = progs[i].name;
+		struct bpf_prog_skeleton *prog_skel = (void *)progs + i * prog_skel_sz;
+		struct bpf_program **prog = prog_skel->prog;
+		const char *name = prog_skel->name;
 
 		*prog = bpf_object__find_program_by_name(obj, name);
 		if (!*prog) {
@@ -13783,13 +13785,13 @@ int bpf_object__open_skeleton(struct bpf_object_skeleton *s,
 	}
 
 	*s->obj = obj;
-	err = populate_skeleton_maps(obj, s->maps, s->map_cnt);
+	err = populate_skeleton_maps(obj, s->maps, s->map_cnt, s->map_skel_sz);
 	if (err) {
 		pr_warn("failed to populate skeleton maps for '%s': %d\n", s->name, err);
 		return libbpf_err(err);
 	}
 
-	err = populate_skeleton_progs(obj, s->progs, s->prog_cnt);
+	err = populate_skeleton_progs(obj, s->progs, s->prog_cnt, s->prog_skel_sz);
 	if (err) {
 		pr_warn("failed to populate skeleton progs for '%s': %d\n", s->name, err);
 		return libbpf_err(err);
@@ -13819,20 +13821,20 @@ int bpf_object__open_subskeleton(struct bpf_object_subskeleton *s)
 		return libbpf_err(-errno);
 	}
 
-	err = populate_skeleton_maps(s->obj, s->maps, s->map_cnt);
+	err = populate_skeleton_maps(s->obj, s->maps, s->map_cnt, s->map_skel_sz);
 	if (err) {
 		pr_warn("failed to populate subskeleton maps: %d\n", err);
 		return libbpf_err(err);
 	}
 
-	err = populate_skeleton_progs(s->obj, s->progs, s->prog_cnt);
+	err = populate_skeleton_progs(s->obj, s->progs, s->prog_cnt, s->prog_skel_sz);
 	if (err) {
 		pr_warn("failed to populate subskeleton maps: %d\n", err);
 		return libbpf_err(err);
 	}
 
 	for (var_idx = 0; var_idx < s->var_cnt; var_idx++) {
-		var_skel = &s->vars[var_idx];
+		var_skel = (void *)s->vars + var_idx * s->var_skel_sz;
 		map = *var_skel->map;
 		map_type_id = bpf_map__btf_value_type_id(map);
 		map_type = btf__type_by_id(btf, map_type_id);
@@ -13879,10 +13881,11 @@ int bpf_object__load_skeleton(struct bpf_object_skeleton *s)
 	}
 
 	for (i = 0; i < s->map_cnt; i++) {
-		struct bpf_map *map = *s->maps[i].map;
+		struct bpf_map_skeleton *map_skel = (void *)s->maps + i * s->map_skel_sz;
+		struct bpf_map *map = *map_skel->map;
 		size_t mmap_sz = bpf_map_mmap_sz(map);
 		int prot, map_fd = map->fd;
-		void **mmaped = s->maps[i].mmaped;
+		void **mmaped = map_skel->mmaped;
 
 		if (!mmaped)
 			continue;
@@ -13930,8 +13933,9 @@ int bpf_object__attach_skeleton(struct bpf_object_skeleton *s)
 	int i, err;
 
 	for (i = 0; i < s->prog_cnt; i++) {
-		struct bpf_program *prog = *s->progs[i].prog;
-		struct bpf_link **link = s->progs[i].link;
+		struct bpf_prog_skeleton *prog_skel = (void *)s->progs + i * s->prog_skel_sz;
+		struct bpf_program *prog = *prog_skel->prog;
+		struct bpf_link **link = prog_skel->link;
 
 		if (!prog->autoload || !prog->autoattach)
 			continue;
@@ -13970,8 +13974,9 @@ int bpf_object__attach_skeleton(struct bpf_object_skeleton *s)
 		return 0;
 
 	for (i = 0; i < s->map_cnt; i++) {
-		struct bpf_map *map = *s->maps[i].map;
-		struct bpf_link **link = s->maps[i].link;
+		struct bpf_map_skeleton *map_skel = (void *)s->maps + i * s->map_skel_sz;
+		struct bpf_map *map = *map_skel->map;
+		struct bpf_link **link = map_skel->link;
 
 		if (!map->autocreate || !map->autoattach)
 			continue;
@@ -14000,7 +14005,8 @@ void bpf_object__detach_skeleton(struct bpf_object_skeleton *s)
 	int i;
 
 	for (i = 0; i < s->prog_cnt; i++) {
-		struct bpf_link **link = s->progs[i].link;
+		struct bpf_prog_skeleton *prog_skel = (void *)s->progs + i * s->prog_skel_sz;
+		struct bpf_link **link = prog_skel->link;
 
 		bpf_link__destroy(*link);
 		*link = NULL;
@@ -14010,7 +14016,8 @@ void bpf_object__detach_skeleton(struct bpf_object_skeleton *s)
 		return;
 
 	for (i = 0; i < s->map_cnt; i++) {
-		struct bpf_link **link = s->maps[i].link;
+		struct bpf_map_skeleton *map_skel = (void *)s->maps + i * s->map_skel_sz;
+		struct bpf_link **link = map_skel->link;
 
 		if (link) {
 			bpf_link__destroy(*link);
-- 
2.43.0

Re: [PATCH bpf-next 2/2] libbpf: fix BPF skeleton forward/backward compat handling

[Alan Maguire]

On 04/07/2024 01:15, Andrii Nakryiko wrote:
> BPF skeleton was designed from day one to be extensible. Generated BPF
> skeleton code specifies actual sizes of map/prog/variable skeletons for
> that reason and libbpf is supposed to work with newer/older versions
> correctly.
> 
> Unfortunately, it was missed that we implicitly embed hard-coded most
> up-to-date (according to libbpf's version of libbpf.h header used to
> compile BPF skeleton header) sizes of those strucs, which can differ
> from the actual sizes at runtime when libbpf is used as a shared
> library.
> 
> We have a few places were we just index array of maps/progs/vars, which
> implicitly uses these potentially invalid sizes of structs.
> 
> This patch aims to fix this problem going forward. Once this lands,
> we'll backport these changes in Github repo to create patched releases
> for older libbpfs.
> 
> Fixes: d66562fba1ce ("libbpf: Add BPF object skeleton support")
> Fixes: 430025e5dca5 ("libbpf: Add subskeleton scaffolding")

Great catch! I suppose it also sort of

Fixes: 08ac454e258 ("libbpf: Auto-attach struct_ops BPF maps in BPF
skeleton")

...since that introduced the new bpf_map_skeleton field. Not a big deal
since it's new and not a stable backport candidate.

> Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
> Signed-off-by: Andrii Nakryiko <andrii@kernel.org>

I'm guessing that you found this when auto-attach silently didn't happen?

Nit: would it be worth dropping a debug logging message here


        /* Skeleton is created with earlier version of bpftool
         * which does not support auto-attachment
         */
-        if (s->map_skel_sz < sizeof(struct bpf_map_skeleton))
+        if (s->map_skel_sz < sizeof(struct bpf_map_skeleton)) {
+		pr_debug("libbpf version mismatch, cannot auto-attach\n");
		return 0;
+	}

...as it's a hard issue to spot?

For the series:

Reviewed-by: Alan Maguire <alan.maguire@oracle.com>

Re: [PATCH bpf-next 1/2] bpftool: improve skeleton backwards compat with old buggy libbpfs

[Quentin Monnet]

On 04/07/2024 01:15, Andrii Nakryiko wrote:
> Old versions of libbpf don't handle varying sizes of bpf_map_skeleton
> struct correctly. As such, BPF skeleton generated by newest bpftool
> might not be compatible with older libbpf (though only when libbpf is
> used as a shared library), even though it, by design, should.
> 
> Going forward libbpf will be fixed, plus we'll release bug fixed
> versions of relevant old libbpfs, but meanwhile try to mitigate from
> bpftool side by conservatively assuming older and smaller definition of
> bpf_map_skeleton, if possible. Meaning, if there are no struct_ops maps.
> 
> If there are struct_ops, then presumably user would like to have
> auto-attaching logic and struct_ops map link placeholders, so use the
> full bpf_map_skeleton definition in that case.
> 
> Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
> Signed-off-by: Andrii Nakryiko <andrii@kernel.org>

Note: I don't know to what extent we enforce this, but kernel docs state
that "Since Co-developed-by: denotes authorship, every Co-developed-by:
must be immediately followed by a Signed-off-by: of the associated
co-author". Mykyta's sign-off is missing from both patches.

Other than that, the patch looks good, thanks for fixing bpftool!

Acked-by: Quentin Monnet <qmo@kernel.org>

Re: [PATCH bpf-next 1/2] bpftool: improve skeleton backwards compat with old buggy libbpfs

[Eduard Zingerman]

On Wed, 2024-07-03 at 17:15 -0700, Andrii Nakryiko wrote:
> Old versions of libbpf don't handle varying sizes of bpf_map_skeleton
> struct correctly. As such, BPF skeleton generated by newest bpftool
> might not be compatible with older libbpf (though only when libbpf is
> used as a shared library), even though it, by design, should.
> 
> Going forward libbpf will be fixed, plus we'll release bug fixed
> versions of relevant old libbpfs, but meanwhile try to mitigate from
> bpftool side by conservatively assuming older and smaller definition of
> bpf_map_skeleton, if possible. Meaning, if there are no struct_ops maps.
> 
> If there are struct_ops, then presumably user would like to have
> auto-attaching logic and struct_ops map link placeholders, so use the
> full bpf_map_skeleton definition in that case.
> 
> Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
> Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> ---

Silly question, here is a fragment of the profiler.skel.h generated
with bpftool build (tools/bpf/bpftool/profiler.skel.h):

static inline int
profiler_bpf__create_skeleton(struct profiler_bpf *obj)
{
	/* ... */

	map = (struct bpf_map_skeleton *)((char *)s->maps + 0 * s->map_skel_sz);
	map->name = "events";
	map->map = &obj->maps.events;

	/* ... 4 more like this ... */

	/* ... */

	s->progs[0].name = "fentry_XXX";
	s->progs[0].prog = &obj->progs.fentry_XXX;
	s->progs[0].link = &obj->links.fentry_XXX;

	s->progs[1].name = "fexit_XXX";
	s->progs[1].prog = &obj->progs.fexit_XXX;
	s->progs[1].link = &obj->links.fexit_XXX;

	/* ... */
}

Do we need to handle 'progs' array access in a same way as maps?

[...]

> @@ -878,23 +895,22 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
>  
>  		codegen("\
>  			\n\
> -									\n\
> -				s->maps[%zu].name = \"%s\";	    \n\
> -				s->maps[%zu].map = &obj->maps.%s;   \n\
> +								    \n\
> +				map = (struct bpf_map_skeleton *)((char *)s->maps + %zu * s->map_skel_sz);\n\
> +				map->name = \"%s\";		    \n\
> +				map->map = &obj->maps.%s;	    \n\
>  			",
> -			i, bpf_map__name(map), i, ident);
> +			i, bpf_map__name(map), ident);
>  		/* memory-mapped internal maps */
>  		if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
> -			printf("\ts->maps[%zu].mmaped = (void **)&obj->%s;\n",
> -				i, ident);
> +			printf("\tmap->mmaped = (void **)&obj->%s;  \n", ident);
                                                                  ^^
                                              nit: this generates extra white space
>  		}
>  
>  		if (populate_links && bpf_map__type(map) == BPF_MAP_TYPE_STRUCT_OPS) {
>  			codegen("\
>  				\n\
> -					s->maps[%zu].link = &obj->links.%s;\n\
> -				",
> -				i, ident);
> +					map->link = &obj->links.%s; \n\
> +				", ident);
>  		}
>  		i++;
>  	}

[...]

Re: [PATCH bpf-next 2/2] libbpf: fix BPF skeleton forward/backward compat handling

[Eduard Zingerman]

On Wed, 2024-07-03 at 17:15 -0700, Andrii Nakryiko wrote:
> BPF skeleton was designed from day one to be extensible. Generated BPF
> skeleton code specifies actual sizes of map/prog/variable skeletons for
> that reason and libbpf is supposed to work with newer/older versions
> correctly.
> 
> Unfortunately, it was missed that we implicitly embed hard-coded most
> up-to-date (according to libbpf's version of libbpf.h header used to
> compile BPF skeleton header) sizes of those strucs, which can differ
                                                  ^^
                                          nit: "struct"
> from the actual sizes at runtime when libbpf is used as a shared
> library.
> 
> We have a few places were we just index array of maps/progs/vars, which
> implicitly uses these potentially invalid sizes of structs.
> 
> This patch aims to fix this problem going forward. Once this lands,
> we'll backport these changes in Github repo to create patched releases
> for older libbpfs.
> 
> Fixes: d66562fba1ce ("libbpf: Add BPF object skeleton support")
> Fixes: 430025e5dca5 ("libbpf: Add subskeleton scaffolding")
> Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
> Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> ---

I double-checked all uses of bpf_object_skeleton->{maps,progs},
all seems in order.

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

Re: [PATCH bpf-next 2/2] libbpf: fix BPF skeleton forward/backward compat handling

[Eduard Zingerman]

On Thu, 2024-07-04 at 16:16 +0100, Alan Maguire wrote:

[...]

> Nit: would it be worth dropping a debug logging message here
> 
> 
>         /* Skeleton is created with earlier version of bpftool
>          * which does not support auto-attachment
>          */
> -        if (s->map_skel_sz < sizeof(struct bpf_map_skeleton))
> +        if (s->map_skel_sz < sizeof(struct bpf_map_skeleton)) {
> +		pr_debug("libbpf version mismatch, cannot auto-attach\n");
> 		return 0;
> +	}
> 
> ...as it's a hard issue to spot?

+1 for debug message, but this is not an error condition,
so I'd say something like "..., skipping auto-attach for struct_ops".

Re: [PATCH bpf-next 1/2] bpftool: improve skeleton backwards compat with old buggy libbpfs

[Andrii Nakryiko]

On Thu, Jul 4, 2024 at 12:21 PM Quentin Monnet <qmo@kernel.org> wrote:
>
> On 04/07/2024 01:15, Andrii Nakryiko wrote:
> > Old versions of libbpf don't handle varying sizes of bpf_map_skeleton
> > struct correctly. As such, BPF skeleton generated by newest bpftool
> > might not be compatible with older libbpf (though only when libbpf is
> > used as a shared library), even though it, by design, should.
> >
> > Going forward libbpf will be fixed, plus we'll release bug fixed
> > versions of relevant old libbpfs, but meanwhile try to mitigate from
> > bpftool side by conservatively assuming older and smaller definition of
> > bpf_map_skeleton, if possible. Meaning, if there are no struct_ops maps.
> >
> > If there are struct_ops, then presumably user would like to have
> > auto-attaching logic and struct_ops map link placeholders, so use the
> > full bpf_map_skeleton definition in that case.
> >
> > Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
> > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
>
> Note: I don't know to what extent we enforce this, but kernel docs state
> that "Since Co-developed-by: denotes authorship, every Co-developed-by:
> must be immediately followed by a Signed-off-by: of the associated
> co-author". Mykyta's sign-off is missing from both patches.
>

Oh, sorry about that. I don't do co-developed-by often, so didn't
realize. I'll add Mykyta's Signed-off-by in v2, thanks

> Other than that, the patch looks good, thanks for fixing bpftool!
>
> Acked-by: Quentin Monnet <qmo@kernel.org>

Re: [PATCH bpf-next 1/2] bpftool: improve skeleton backwards compat with old buggy libbpfs

[Andrii Nakryiko]

On Thu, Jul 4, 2024 at 1:31 PM Eduard Zingerman <eddyz87@gmail.com> wrote:
>
> On Wed, 2024-07-03 at 17:15 -0700, Andrii Nakryiko wrote:
> > Old versions of libbpf don't handle varying sizes of bpf_map_skeleton
> > struct correctly. As such, BPF skeleton generated by newest bpftool
> > might not be compatible with older libbpf (though only when libbpf is
> > used as a shared library), even though it, by design, should.
> >
> > Going forward libbpf will be fixed, plus we'll release bug fixed
> > versions of relevant old libbpfs, but meanwhile try to mitigate from
> > bpftool side by conservatively assuming older and smaller definition of
> > bpf_map_skeleton, if possible. Meaning, if there are no struct_ops maps.
> >
> > If there are struct_ops, then presumably user would like to have
> > auto-attaching logic and struct_ops map link placeholders, so use the
> > full bpf_map_skeleton definition in that case.
> >
> > Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
> > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> > ---
>
> Silly question, here is a fragment of the profiler.skel.h generated
> with bpftool build (tools/bpf/bpftool/profiler.skel.h):
>
> static inline int
> profiler_bpf__create_skeleton(struct profiler_bpf *obj)
> {
>         /* ... */
>
>         map = (struct bpf_map_skeleton *)((char *)s->maps + 0 * s->map_skel_sz);
>         map->name = "events";
>         map->map = &obj->maps.events;
>
>         /* ... 4 more like this ... */
>
>         /* ... */
>
>         s->progs[0].name = "fentry_XXX";
>         s->progs[0].prog = &obj->progs.fentry_XXX;
>         s->progs[0].link = &obj->links.fentry_XXX;
>
>         s->progs[1].name = "fexit_XXX";
>         s->progs[1].prog = &obj->progs.fexit_XXX;
>         s->progs[1].link = &obj->links.fexit_XXX;
>
>         /* ... */
> }
>
> Do we need to handle 'progs' array access in a same way as maps?

Given bpf_prog_skeleton has never been extended yet (and maybe never
will be), I chose not to uglify this unnecessarily. My thinking/hope
is that by the time we get to extending prog_skeleton struct, all
actively used libbpf versions will be patched up and will handle this
correctly without the hacks we have to do for map_skeleton.


>
> [...]
>
> > @@ -878,23 +895,22 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
> >
> >               codegen("\
> >                       \n\
> > -                                                                     \n\
> > -                             s->maps[%zu].name = \"%s\";         \n\
> > -                             s->maps[%zu].map = &obj->maps.%s;   \n\
> > +                                                                 \n\
> > +                             map = (struct bpf_map_skeleton *)((char *)s->maps + %zu * s->map_skel_sz);\n\
> > +                             map->name = \"%s\";                 \n\
> > +                             map->map = &obj->maps.%s;           \n\
> >                       ",
> > -                     i, bpf_map__name(map), i, ident);
> > +                     i, bpf_map__name(map), ident);
> >               /* memory-mapped internal maps */
> >               if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
> > -                     printf("\ts->maps[%zu].mmaped = (void **)&obj->%s;\n",
> > -                             i, ident);
> > +                     printf("\tmap->mmaped = (void **)&obj->%s;  \n", ident);
>                                                                   ^^
>                                               nit: this generates extra white space
> >               }
> >
> >               if (populate_links && bpf_map__type(map) == BPF_MAP_TYPE_STRUCT_OPS) {
> >                       codegen("\
> >                               \n\
> > -                                     s->maps[%zu].link = &obj->links.%s;\n\
> > -                             ",
> > -                             i, ident);
> > +                                     map->link = &obj->links.%s; \n\
> > +                             ", ident);
> >               }
> >               i++;
> >       }
>
> [...]

Re: [PATCH bpf-next 2/2] libbpf: fix BPF skeleton forward/backward compat handling

[Andrii Nakryiko]

On Thu, Jul 4, 2024 at 8:17 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On 04/07/2024 01:15, Andrii Nakryiko wrote:
> > BPF skeleton was designed from day one to be extensible. Generated BPF
> > skeleton code specifies actual sizes of map/prog/variable skeletons for
> > that reason and libbpf is supposed to work with newer/older versions
> > correctly.
> >
> > Unfortunately, it was missed that we implicitly embed hard-coded most
> > up-to-date (according to libbpf's version of libbpf.h header used to
> > compile BPF skeleton header) sizes of those strucs, which can differ
> > from the actual sizes at runtime when libbpf is used as a shared
> > library.
> >
> > We have a few places were we just index array of maps/progs/vars, which
> > implicitly uses these potentially invalid sizes of structs.
> >
> > This patch aims to fix this problem going forward. Once this lands,
> > we'll backport these changes in Github repo to create patched releases
> > for older libbpfs.
> >
> > Fixes: d66562fba1ce ("libbpf: Add BPF object skeleton support")
> > Fixes: 430025e5dca5 ("libbpf: Add subskeleton scaffolding")
>
> Great catch! I suppose it also sort of
>
> Fixes: 08ac454e258 ("libbpf: Auto-attach struct_ops BPF maps in BPF
> skeleton")
>
> ...since that introduced the new bpf_map_skeleton field. Not a big deal
> since it's new and not a stable backport candidate.
>

yeah, I put the original changes that introduced this
bug/inflexibility in the first place. Either way Github releases and
backporting will be done separate from the kernel repo.

> > Co-developed-by: Mykyta Yatsenko <yatsenko@meta.com>
> > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
>
> I'm guessing that you found this when auto-attach silently didn't happen?

nope, we actually got crashes due to memory corruption in our internal
production testing

>
> Nit: would it be worth dropping a debug logging message here
>
>
>         /* Skeleton is created with earlier version of bpftool
>          * which does not support auto-attachment
>          */
> -        if (s->map_skel_sz < sizeof(struct bpf_map_skeleton))
> +        if (s->map_skel_sz < sizeof(struct bpf_map_skeleton)) {
> +               pr_debug("libbpf version mismatch, cannot auto-attach\n");

ok, sure. But as Eduard pointed out, it's not really a bug or anything
like that, it's an expected backwards compat mechanism. I can add
debug-level (or probably info-level?) message, though.

>                 return 0;
> +       }
>
> ...as it's a hard issue to spot?
>
> For the series:
>
> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>

Re: [PATCH bpf-next 2/2] libbpf: fix BPF skeleton forward/backward compat handling

[Andrii Nakryiko]

On Thu, Jul 4, 2024 at 1:56 PM Eduard Zingerman <eddyz87@gmail.com> wrote:
>
> On Thu, 2024-07-04 at 16:16 +0100, Alan Maguire wrote:
>
> [...]
>
> > Nit: would it be worth dropping a debug logging message here
> >
> >
> >         /* Skeleton is created with earlier version of bpftool
> >          * which does not support auto-attachment
> >          */
> > -        if (s->map_skel_sz < sizeof(struct bpf_map_skeleton))
> > +        if (s->map_skel_sz < sizeof(struct bpf_map_skeleton)) {
> > +             pr_debug("libbpf version mismatch, cannot auto-attach\n");
> >               return 0;
> > +     }
> >
> > ...as it's a hard issue to spot?
>
> +1 for debug message, but this is not an error condition,
> so I'd say something like "..., skipping auto-attach for struct_ops".

agreed, thinking to make it info-level and only output if we actually
have struct_ops, so not to spam people unnecessarily

Re: [PATCH bpf-next 1/2] bpftool: improve skeleton backwards compat with old buggy libbpfs

[Eduard Zingerman]

On Mon, 2024-07-08 at 10:15 -0700, Andrii Nakryiko wrote:

[...]

> > static inline int
> > profiler_bpf__create_skeleton(struct profiler_bpf *obj)
> > {
> >         /* ... */
> > 
> >         map = (struct bpf_map_skeleton *)((char *)s->maps + 0 * s->map_skel_sz);
> >         map->name = "events";
> >         map->map = &obj->maps.events;
> > 
> >         /* ... 4 more like this ... */
> > 
> >         /* ... */
> > 
> >         s->progs[0].name = "fentry_XXX";
> >         s->progs[0].prog = &obj->progs.fentry_XXX;
> >         s->progs[0].link = &obj->links.fentry_XXX;
> > 
> >         s->progs[1].name = "fexit_XXX";
> >         s->progs[1].prog = &obj->progs.fexit_XXX;
> >         s->progs[1].link = &obj->links.fexit_XXX;
> > 
> >         /* ... */
> > }
> > 
> > Do we need to handle 'progs' array access in a same way as maps?
> 
> Given bpf_prog_skeleton has never been extended yet (and maybe never
> will be), I chose not to uglify this unnecessarily. My thinking/hope
> is that by the time we get to extending prog_skeleton struct, all
> actively used libbpf versions will be patched up and will handle this
> correctly without the hacks we have to do for map_skeleton.

Understood, fair enough.

[...]