Re: [PATCH bpf-next v2 0/5] Subskeleton support for BPF libraries

[Yonghong Song <yhs@fb.com>]

On 3/10/22 4:11 PM, Delyan Kratunov wrote:
> In the quest for ever more modularity, a new need has arisen - the ability to
> access data associated with a BPF library from a corresponding userspace library.
> The catch is that we don't want the userspace library to know about the structure of the
> final BPF object that the BPF library is linked into.
> 
> In pursuit of this modularity, this patch series introduces *subskeletons.*
> Subskeletons are similar in use and design to skeletons with a couple of differences:
> 
> 1. The generated storage types do not rely on contiguous storage for the library's
> variables because they may be interspersed randomly throughout the final BPF object's sections.
> 
> 2. Subskeletons do not own objects and instead require a loaded bpf_object* to
> be passed at runtime in order to be initialized. By extension, symbols are resolved at
> runtime by parsing the final object's BTF. This has the interesting effect that the same
> userspace code can interoperate with the library BPF code *linked into different final objects.*
> 
> 3. Subskeletons allow access to all global variables, programs, and custom maps. They also expose
> the internal maps *of the final object*. This allows bpf_var_skeleton objects to contain a bpf_map**
> instead of a section name.
> 
> Changes since v1:
>   - Introduced new strict mode knob for single-routine-in-.text compatibility behavior, which
>     disproportionately affects library objects. bpftool works in 1.0 mode so subskeleton generation
>     doesn't have to worry about this now.
>   - Made bpf_map_btf_value_type_id available earlier and used it wherever applicable.
>   - Refactoring in bpftool gen.c per review comments.
>   - Subskels now use typeof() for array and func proto globals to avoid the need for runtime split btf.
>   - Expanded the subskeleton test to include arrays, custom maps, extern maps, weak symbols, and kconfigs.
>   - selftests/bpf/Makefile now generates a subskel.h for every skel.h it would make.
> 
> For reference, here is a shortened subskeleton header:
> 
> #ifndef __TEST_SUBSKELETON_LIB_SUBSKEL_H__
> #define __TEST_SUBSKELETON_LIB_SUBSKEL_H__
> 
> struct test_subskeleton_lib {
> 	struct bpf_object *obj;
> 	struct bpf_object_subskeleton *subskel;
> 	struct {
> 		struct bpf_map *map2;
> 		struct bpf_map *map1;
> 		struct bpf_map *data;
> 		struct bpf_map *rodata;
> 		struct bpf_map *bss;
> 		struct bpf_map *kconfig;
> 	} maps;
> 	struct {
> 		struct bpf_program *lib_perf_handler;
> 	} progs;
> 	struct test_subskeleton_lib__data {
> 		int *var6;
> 		int *var2;
> 		int *var5;
> 	} data;
> 	struct test_subskeleton_lib__rodata {
> 		int *var1;
> 	} rodata;
> 	struct test_subskeleton_lib__bss {
> 		struct {
> 			int var3_1;
> 			__s64 var3_2;
> 		} *var3;
> 		int *libout1;
> 		typeof(int[4]) *var4;
> 		typeof(int (*)()) *fn_ptr;
> 	} bss;
> 	struct test_subskeleton_lib__kconfig {
> 		_Bool *CONFIG_BPF_SYSCALL;
> 	} kconfig;
> 
> static inline struct test_subskeleton_lib *
> test_subskeleton_lib__open(const struct bpf_object *src)
> {
> 	struct test_subskeleton_lib *obj;
> 	struct bpf_object_subskeleton *s;
> 	int err;
> 
> 	...
> 	s = (struct bpf_object_subskeleton *)calloc(1, sizeof(*s));
> 	...
> 
> 	s->var_cnt = 9;
> 	...
> 
> 	s->vars[0].name = "var6";
> 	s->vars[0].map = &obj->maps.data;
> 	s->vars[0].addr = (void**) &obj->data.var6;
>    ...
> 
> 	/* maps */
> 	...
> 
> 	/* programs */
> 	s->prog_cnt = 1;
> 	...
> 
> 	err = bpf_object__open_subskeleton(s);
>    ...
> 	return obj;
> }
> #endif /* __TEST_SUBSKELETON_LIB_SUBSKEL_H__ */

When I tried to build the patch set with parallel mode (-j),
    make -C tools/testing/selftests/bpf -j
I hit the following errors:

/bin/sh: line 1: 3484984 Bus error               (core dumped) 
/home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/sbin/bpftool 
gen skeleton 
/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.linked3.o 
name test_ksyms_weak > 
/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h
make: *** [Makefile:496: 
/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h] 
Error 135
make: *** Deleting file 
'/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h'
make: *** Waiting for unfinished jobs....
make: Leaving directory 
'/home/yhs/work/bpf-next/tools/testing/selftests/bpf'

Probably some make file related issues.
I didn't hit this issue before without this patch set.

> 
> Delyan Kratunov (5):
>    libbpf: add new strict flag for .text subprograms
>    libbpf: init btf_{key,value}_type_id on internal map open
>    libbpf: add subskeleton scaffolding
>    bpftool: add support for subskeletons
>    selftests/bpf: test subskeleton functionality
> 
>   .../bpf/bpftool/Documentation/bpftool-gen.rst |  25 +
>   tools/bpf/bpftool/bash-completion/bpftool     |  14 +-
>   tools/bpf/bpftool/gen.c                       | 589 ++++++++++++++++--
>   tools/lib/bpf/libbpf.c                        | 151 ++++-
>   tools/lib/bpf/libbpf.h                        |  29 +
>   tools/lib/bpf/libbpf.map                      |   2 +
>   tools/lib/bpf/libbpf_legacy.h                 |   6 +
>   tools/testing/selftests/bpf/.gitignore        |   1 +
>   tools/testing/selftests/bpf/Makefile          |  10 +-
>   .../selftests/bpf/prog_tests/subskeleton.c    |  83 +++
>   .../selftests/bpf/progs/test_subskeleton.c    |  23 +
>   .../bpf/progs/test_subskeleton_lib.c          |  56 ++
>   .../bpf/progs/test_subskeleton_lib2.c         |  16 +
>   13 files changed, 919 insertions(+), 86 deletions(-)
>   create mode 100644 tools/testing/selftests/bpf/prog_tests/subskeleton.c
>   create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton.c
>   create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton_lib.c
>   create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton_lib2.c
> 
> --
> 2.34.1