[PATCH virtme-docker v5] tests: add bpftests support

[PATCH virtme-docker v5] tests: add bpftests support

[Geliang Tang]

This patch builds bpf tests after building other mptcp selftests in btf
mode:

	cd tools/testing/selftests/bpf && make

And add run_bpftest_all to run bpftests:

	./test_progs -t mptcp
	./test_progs-no_alu32 -t mptcp

Now mptcp bpf selftests can be tested in this way:

	docker run ... auto-btf
	run_loop run_bpftest_all in .virtme-exec-run

Disable IA32_EMULATION config to fix this error:

	./include/linux/if.h:28:10: fatal
		error: sys/socket.h: no such file or directory

Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/406
Signed-off-by: Geliang Tang <geliang@kernel.org>
---
v5:
 - drop mptcp_progs.sh script, use test_progs* directly.

v4:
 - test mode in run_bpftest_all

RESEND:
 - Update email only.

v3:
 - depends on "bpf, btf: Add DEBUG_INFO_BTF checks for
   __register_bpf_struct_ops", which I sent to BPF ML.
 - cleanup.

v2:
 - drop links for 'test_progs, bpf_testmod.ko'.
 - rename bpf_selftest.sh to mptcp_progs.sh.
 - fix bugs in run_loop.
---
 entrypoint.sh | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/entrypoint.sh b/entrypoint.sh
index 95ff0f9..3a2a87c 100755
--- a/entrypoint.sh
+++ b/entrypoint.sh
@@ -87,6 +87,8 @@ VIRTME_RUN_EXPECT="${VIRTME_SCRIPTS_DIR}/virtme.expect"
 
 SELFTESTS_DIR="${INPUT_SELFTESTS_DIR:-tools/testing/selftests/net/mptcp}"
 SELFTESTS_CONFIG="${SELFTESTS_DIR}/config"
+BPFTESTS_DIR="${INPUT_BPFTESTS_DIR:-tools/testing/selftests/bpf}"
+BPFTESTS_CONFIG="${BPFTESTS_DIR}/config"
 
 export CCACHE_MAXSIZE="${INPUT_CCACHE_MAXSIZE}"
 export CCACHE_DIR="${VIRTME_WORKDIR}/ccache"
@@ -316,6 +318,12 @@ gen_kconfig() { local mode kconfig=()
 	# We need more debug info but it is slow to generate
 	if [ "${mode}" = "btf" ]; then
 		kconfig+=(-e DEBUG_INFO_BTF)
+		# Extra options are needed for bpftests
+		./scripts/kconfig/merge_config.sh -m "${VIRTME_KCONFIG}" "${BPFTESTS_CONFIG}"
+		kconfig+=(-e DEBUG_INFO_BTF_MODULES -e MODULE_ALLOW_BTF_MISMATCH)
+		# Fix ./include/linux/if.h:28:10: fatal error:
+		#		sys/socket.h: no such file or directory
+		kconfig+=(-d IA32_EMULATION)
 	elif is_ci || [ "${mode}" != "debsym" ]; then
 		kconfig+=(-e DEBUG_INFO_REDUCED -e DEBUG_INFO_SPLIT)
 	fi
@@ -428,6 +436,15 @@ build_selftests() {
 	_make_o KHDR_INCLUDES="-I${VIRTME_BUILD_DIR}/include" -C "${SELFTESTS_DIR}"
 }
 
+build_bpftests() {
+	if [ "${INPUT_BUILD_SKIP_BPFTESTS}" = 1 ]; then
+		printinfo "Skip bpftests build"
+		return 0
+	fi
+
+	_make_o KHDR_INCLUDES="-I${VIRTME_BUILD_DIR}/include" -C "${BPFTESTS_DIR}"
+}
+
 build_packetdrill() { local old_pwd kversion kver_maj kver_min branch
 	if [ "${INPUT_BUILD_SKIP_PACKETDRILL}" = 1 ]; then
 		printinfo "Skip Packetdrill build"
@@ -515,6 +532,9 @@ prepare() { local mode no_tap=1
 	printinfo "Prepare the environment"
 
 	build_selftests
+	if [ "${mode}" = "btf" ]; then
+		build_bpftests
+	fi
 	build_packetdrill
 	prepare_hosts_file
 
@@ -772,11 +792,45 @@ run_packetdrill_all() { local pktd_dir rc=0
 	return \${rc}
 }
 
+# \$1: output tap file; rest: command to launch
+_run_bpftest_one_tap() {
+	cd "${KERNEL_SRC}/${BPFTESTS_DIR}"
+	_tap "\${@}"
+}
+
+# \$1: script file; rest: command to launch
+run_bpftest_one() { local sf tap
+	sf=\$(basename \${1})
+	tap=bpftest_\${sf}
+	shift
+
+	_can_run "\${tap}" || return 0
+
+	_run_bpftest_one_tap "${RESULTS_DIR}/\${tap}" "./\${sf}" "\${@}"
+}
+
+run_bpftest_all() {
+	if [ "${mode}" = "btf" ]; then
+		local sf rc=0
+
+		for sf in "${KERNEL_SRC}/${BPFTESTS_DIR}/"test_progs*; do
+			if [ -x "\${sf}" ]; then
+				run_bpftest_one "\${sf}" "-t" "mptcp" || rc=\${?}
+			fi
+		done
+
+		return \${rc}
+	else
+		echo "skip bpftest, only run it in btf mode"
+	fi
+}
+
 run_all() {
 	run_kunit_all
 	run_selftest_all
 	run_mptcp_connect_mmap
 	run_packetdrill_all
+	run_bpftest_all
 }
 
 has_call_trace() {
-- 
2.40.1

Re: [PATCH virtme-docker v5] tests: add bpftests support

[Matthieu Baerts]

Hi Geliang,

On 12/02/2024 10:24, Geliang Tang wrote:
> This patch builds bpf tests after building other mptcp selftests in btf
> mode:
> 
> 	cd tools/testing/selftests/bpf && make
> 
> And add run_bpftest_all to run bpftests:
> 
> 	./test_progs -t mptcp
> 	./test_progs-no_alu32 -t mptcp
> 
> Now mptcp bpf selftests can be tested in this way:
> 
> 	docker run ... auto-btf
> 	run_loop run_bpftest_all in .virtme-exec-run
> 
> Disable IA32_EMULATION config to fix this error:
> 
> 	./include/linux/if.h:28:10: fatal
> 		error: sys/socket.h: no such file or directory

Thank you for this patch! It's good we finally have an easy way to
validate these tests! v5 looks better without the mptcp_progs.sh script!

I just managed to find some time to test it and include it in the repo.
I had to modify a bit where the test_progs are located:


https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/commit/2051f42

I guess on your side, you had an "old" test_progs in the source file,
instead of using the one from .virtme/build.


What's a bit annoying, is that all built files from BPF selftests are in
the root dir of the build dir, in the middle of all kernel built files.
I tried to change that by changing "O=" this in build_bpftests():

  mkdir -p "${VIRTME_BPFTESTS_DIR}"
  _make O="${VIRTME_BPFTESTS_DIR}" prepare
  _make O="${VIRTME_BPFTESTS_DIR}" -C "${BPFTESTS_DIR}"

where:

  VIRTME_BPFTESTS_DIR="${VIRTME_BUILD_DIR}/${BPFTESTS_DIR}"

(I don't think KHDR_INCLUDES is needed, I didn't check)

But I had issues with bpf_testmod we don't need :-/
I tried by setting KDIR="${VIRTME_BUILD_DIR}" but it didn't help.

If you see a way to use another build dir that is specific to the bpf
tests, that would be cleaner, I think.

> Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/406

Before really closing it, I will look at modifying the .yml files used
by the CI to have a new job validating only the BPF tests.


Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.