Re: [RFC] dwarves/pahole: Add test scripts

[Nathan Chancellor <nathan@kernel.org>]

Hi Jiri,

Just some drive by comments. I think the idea is neat, I am always a fan
of automating more things :)

On Tue, Feb 23, 2021 at 02:23:21PM +0100, Jiri Olsa wrote:
> hi,
> I cleaned up a bit my testing scripts, that I'm using for testing
> btf encoding changes. It's far from ideal and convoluted, but let's
> have discussion if this could be kicked into something useful for
> everybody.
> 
> There are 2 scripts:
>   kernel-objects-build.sh - compiles kernel for several archs and
>                             stores vmlinux and kernel modules
> 
>   kernel-objects-test.sh  - goes through objects stored by ^^^
>                             and runs tests on each of them
> 
> The general idea is that all objects are compiled already with
> BTF debuginfo with available pahole. The test script then:
>   - takes each objects and dumps its current BTF data
>   - then create new BTF data with given pahole binary
>   - dumps the new BTF data and makes the comparison
> 
> I was thinking about support for comparing 2 pahole binaries,
> but so far that did not fit into my workflow. Normally I have
> latest globally available pahole, which is used to build the
> kernel binaries and then I'm playing with new pahole binary,
> which I'm putting to the test.
> 
> Example.. prepare vmlinux and modules for all archs:
> 
>         $ ./kernel-objects-build.sh
>         output:  /tmp/pahole.test.nsQ
>         kdir:    /home/jolsa/linux
>         pahole:  /opt/dwarves/bin/pahole
>         objects: /home/jolsa/.pahole_test_objects
> 
>         cleanup /home/jolsa/linux
>         ...
> 
> All objects are stored under ~/pahole_test_objects/ directories:
> 
>         $ ls ~/.pahole_test_objects/
>         aarch64-clang
>         aarch64-gcc
>         powerpc-gcc
>         powerpcle-gcc
>         s390x-gcc
>         x86-clang
>         x86-gcc
> 
> Each containing vmlinux and modules:
> 
> 	$ ls ~/.pahole_test_objects/x86-gcc/
> 	efivarfs.ko  iptable_nat.ko  nf_log_arp.ko  nf_log_common.ko  nf_log_ipv4.ko  nf_log_ipv6.ko
> 	vmlinux  x86_pkg_temp_thermal.ko  xt_addrtype.ko  xt_LOG.ko  xt_mark.ko  xt_MASQUERADE.ko  xt_nat.ko
> 
> Run test on all of them with new './pahole' binary:
> 
>         $ ./kernel-objects-test.sh -B ~/linux/tools/bpf/bpftool/bpftool -P ./pahole
>         pahole:  /home/jolsa/pahole/build/pahole
>         bpftool: /home/jolsa/linux/tools/bpf/bpftool/bpftool
>         base:    /tmp/pahole.test.oxv
>         objects: /home/jolsa/.pahole_test_objects
>         fail:    no
>         cleanup: yes
> 
>         test_funcs      on /home/jolsa/.pahole_test_objects/aarch64-clang/vmlinux ... OK
>         test_format_c   on /home/jolsa/.pahole_test_objects/aarch64-clang/vmlinux ... OK
>         test_btfdiff    on /home/jolsa/.pahole_test_objects/aarch64-clang/vmlinux ... FAIL
>         test_funcs      on /home/jolsa/.pahole_test_objects/aarch64-clang/8021q.ko ... OK
>         test_format_c   on /home/jolsa/.pahole_test_objects/aarch64-clang/8021q.ko ... OK
>         test_funcs      on /home/jolsa/.pahole_test_objects/aarch64-clang/act_gact.ko ... OK
>         test_format_c   on /home/jolsa/.pahole_test_objects/aarch64-clang/act_gact.ko ... OK
>         ...
> 
> There are several options that helps to set other binaries/dirs
> or stop and debug issues.
> 
> thoughts?
> 
> thanks,
> jirka
> 
> 
> ---
>  kernel-objects-build.sh | 132 +++++++++++++++++++
>  kernel-objects-test.sh  | 282 ++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 414 insertions(+)
>  create mode 100755 kernel-objects-build.sh
>  create mode 100755 kernel-objects-test.sh
> 
> diff --git a/kernel-objects-build.sh b/kernel-objects-build.sh
> new file mode 100755
> index 000000000000..b92729994ded
> --- /dev/null
> +++ b/kernel-objects-build.sh
> @@ -0,0 +1,132 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +
> +set -u
> +set -e
> +
> +exec 2>&1
> +
> +OBJECTS="${HOME}/.pahole_test_objects"
> +KDIR=${HOME}/linux
> +PAHOLE=$(which pahole)
> +OUTPUT=
> +
> +usage()
> +{
> +	cat <<EOF
> +Usage: $0 [-k KERNEL] [-O OUTPUT] [-o OBJECTS]
> +
> +The script prepares vmlinux and kernel modules for different archs/C:
> +
> +  - x86 gcc/clang
> +  - arm64 gcc/clang
> +  - powerpc gcc
> +  - s390x gcc
> +
> +Options:
> +  -k) Update kernel tree directory (default HOME/linux)
> +  -O) Update temp output directory (default mktemp /tmp/pahole.test.XXX)
> +  -o) Update final objects directory (default HOME/.pahole.test.XXX)
> +
> +Make images under '/tmp/build', and place it under 'objects':
> +
> +  $ $0 -o objects -O /tmp/build/
> +
> +EOF
> +}
> +
> +build()
> +{
> +	local name=$1
> +	local opts=$2

A more robust way to handle this might be

shift
local opts=$@

> +
> +	echo "build ${name} (${OUTPUT}/output)"
> +
> +	mkdir -p ${OBJECTS}/${name}
> +	mkdir -p ${OUTPUT}
> +
> +	pushd ${KDIR}
> +	make ${opts} -j"$(nproc)" O=${OUTPUT} olddefconfig > ${OUTPUT}/output 2>&1

Then change this to

make "${opts[@]}"

shellcheck complains about implicit word splitting (and finds some other
things in the other script).

> +	scripts/config \
> +		--file ${OUTPUT}/.config \
> +		-e BPF_SYSCALL \
> +		-e DEBUG_INFO \
> +		-e DEBUG_INFO_BTF \
> +		-e FTRACE \
> +		-e FUNCTION_TRACER \
> +		>> ${OUTPUT}/output 2>&1
> +	make ${opts} -j"$(nproc)" O=${OUTPUT} PAHOLE=${PAHOLE} olddefconfig all >> ${OUTPUT}/output 2>&1
> +
> +	cp ${OUTPUT}/vmlinux ${OBJECTS}/${name}
> +	find ${OUTPUT} -name '*.ko' | xargs cp -t ${OBJECTS}/${name}
> +
> +	rm -rf ${OUTPUT}
> +	popd
> +}
> +
> +main()
> +{
> +	while getopts 'k:o:O:' opt; do
> +		case ${opt} in
> +		k)
> +			KDIR="$OPTARG"
> +			;;
> +		O)
> +			OUTPUT="$OPTARG"
> +			;;
> +		o)
> +			OBJECTS="$OPTARG"
> +			;;
> +		esac
> +	done
> +	shift $((OPTIND -1))
> +
> +	if [[ $# -ne 0 ]]; then
> +		usage
> +		exit 1
> +	fi
> +
> +        if [[ "${OUTPUT}" == "" ]]; then
> +                OUTPUT=$(mktemp -d /tmp/pahole.test.XXX)
> +        fi
> +
> +	PAHOLE=$(realpath ${PAHOLE})
> +	OBJECTS=$(realpath ${OBJECTS})
> +
> +	echo "output:  ${OUTPUT}"
> +	echo "kdir:    ${KDIR}"
> +	echo "pahole:  ${PAHOLE}"
> +	echo "objects: ${OBJECTS}"
> +	echo
> +
> +	mkdir -p ${OBJECTS}
> +
> +	echo "cleanup ${KDIR}"
> +	make -C ${KDIR} mrproper
> +
> +
> +	build x86-clang     "LLVM=1"

With that change above, you could unquote these options and just pass
them in as regular parameters.

> +	build x86-gcc       ""
> +
> +	build aarch64-clang "ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- LLVM=1"
> +	build aarch64-gcc   "ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-"
> +
> +#	build powerpc-clang "ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- LLVM=1"
> +	build powerpc-gcc   "ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu-"
> +
> +#	build powerpcle-clang "ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- LLVM=1"
> +	build powerpcle-gcc   "ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu-"
> +
> +#	build s390x-clang   "ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- LLVM=1"

powerpc64le and s390 can build with CC=clang, instead of LLVM=1.

I will see if I can give this a run locally over the next week or so.

Cheers,
Nathan

> +	build s390x-gcc     "ARCH=s390 CROSS_COMPILE=s390x-linux-gnu-"
> +}
> +
> +catch()
> +{
> +	local exit_code=$1
> +	exit ${exit_code}
> +}
> +
> +trap 'catch "$?"' EXIT
> +
> +main "$@"
> diff --git a/kernel-objects-test.sh b/kernel-objects-test.sh
> new file mode 100755
> index 000000000000..a34c22c2eb09
> --- /dev/null
> +++ b/kernel-objects-test.sh
> @@ -0,0 +1,282 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +
> +set -u
> +
> +exec 2>&1
> +
> +PAHOLE=$(which pahole)
> +BPFTOOL=$(which bpftool)
> +BTFDIFF=$(which btfdiff)
> +
> +OBJECTS="$HOME/.pahole_test_objects"
> +CLEANUP="yes"
> +BASE=
> +FAIL="no"
> +
> +function test_funcs()
> +{
> +	local vmlinux=$1
> +	local obj=$2
> +	local err=0
> +
> +	cp ${obj} ${BASE}/object
> +
> +	if [[ ${obj} == *.ko ]]; then
> +		${BPFTOOL} --base ${vmlinux} btf dump file ${BASE}/object > ${BASE}/btf.old
> +		${PAHOLE} -V -J --btf_base ${vmlinux} ${BASE}/object > ${BASE}/output
> +		${BPFTOOL} --base ${vmlinux} btf dump file ${BASE}/object > ${BASE}/btf.new
> +	else
> +		${BPFTOOL} btf dump file ${BASE}/object > ${BASE}/btf.old
> +		${PAHOLE} -V -J ${BASE}/object > ${BASE}/output
> +		${BPFTOOL} btf dump file ${BASE}/object > ${BASE}/btf.new
> +	fi
> +
> +	diff -puw ${BASE}/btf.old ${BASE}/btf.new > ${BASE}/diff.all
> +	if [ $? -ne 0 ]; then
> +		funcs_old=${BASE}/funcs.old
> +		funcs_new=${BASE}/funcs.new
> +
> +		cat ${BASE}/btf.old | grep 'FUNC ' | awk '{ print $3 }' | sort | uniq > ${funcs_old}
> +		cat ${BASE}/btf.new | grep 'FUNC ' | awk '{ print $3 }' | sort | uniq > ${funcs_new}
> +
> +		diff -puw ${funcs_old} ${funcs_new} > ${BASE}/diff.funcs
> +	fi
> +
> +	if [[ $? -ne 0 ]]; then
> +		err=1
> +	fi
> +
> +	return ${err};
> +}
> +
> +function test_format_c()
> +{
> +	local vmlinux=$1
> +	local obj=$2
> +	local err=0
> +
> +	cp ${obj} ${BASE}/object
> +
> +	if [[ ${obj} == *.ko ]]; then
> +		${BPFTOOL} --base ${vmlinux} btf dump file ${BASE}/object format c > ${BASE}/c.old
> +		${PAHOLE} -V -J --btf_base ${vmlinux} ${BASE}/object > ${BASE}/output
> +		${BPFTOOL} --base ${vmlinux} btf dump file ${BASE}/object format c > ${BASE}/c.new
> +	else
> +		${BPFTOOL} btf dump file ${BASE}/object format c > ${BASE}/c.old
> +		${PAHOLE} -V -J ${BASE}/object > ${BASE}/output
> +		${BPFTOOL} btf dump file ${BASE}/object format c > ${BASE}/c.new
> +	fi
> +
> +	diff -puw ${BASE}/c.old ${BASE}/c.new > ${BASE}/diff.all
> +	if [[ $? -ne 0 ]]; then
> +		err=1
> +	fi
> +
> +	return ${err};
> +}
> +
> +function test_btfdiff()
> +{
> +	local vmlinux=$1
> +	local obj=$2
> +	local err=0
> +
> +	if [[ -x ${BTFDIFF} ]]; then
> +		${BTFDIFF} ${obj} > ${BASE}/output
> +		if [[ -s "${BASE}/output" ]]; then
> +			err=1
> +		fi
> +	else
> +		err=2
> +	fi
> +
> +	return ${err}
> +}
> +
> +usage()
> +{
> +	cat <<EOF
> +Usage: $0 [-f] [-o object] [-O objects] [-b BASE] [-P PAHOLE] [-B BPFTOOL] -- [test]
> +
> +The script runs tests on objects with BTF data.
> +
> +Options:
> +  -f) Stop on failure
> +  -o) Run tests on specific objects
> +  -O) Update the root objects directory (default HOME/.pahole_test_objects)
> +  -b) Update work base/temporary directory (default mktemp -d /tmp/pahole.test.XXX)
> +  -P) Update pahole path (default which pahole)
> +  -B) Update bpftool path (default which bpftool)
> +
> +Test image under 'objects':
> +
> +  $ $0 -O objects/
> +
> +Test specific image (objects/aarch64-clang) and stop on failure:
> +
> +  $ $0 -o o objects/aarch64-clang -f
> +
> +Run specific test (test_format_c):
> +
> +  $ $0 -o o objects/aarch64-clang -f test_format_c
> +EOF
> +}
> +
> +do_test()
> +{
> +	local test_name=$1
> +	local vmlinux=$2
> +	local obj=$3
> +
> +	printf "%-15s on %s ... " "${test_name}"  "${obj}"
> +
> +	eval ${test_name} ${vmlinux} ${obj}
> +	local err=$?
> +
> +	case ${err} in
> +	0)
> +		echo "OK"
> +		;;
> +	1)
> +		echo "FAIL"
> +		;;
> +	2)
> +		echo "SKIP"
> +		;;
> +	esac
> +
> +	if [[ ${err} -eq 1 && "${FAIL}" == "yes" ]]; then
> +		exit 1
> +	fi
> +
> +	return ${err}
> +}
> +
> +run_tests()
> +{
> +	local vmlinux=$1
> +	local obj=$2
> +	local test_name=$3
> +
> +	if [[ "${test_name}" != "all" ]]; then
> +		do_test ${test_name} ${vmlinux} ${obj}
> +	else
> +		do_test test_funcs ${vmlinux} ${obj}
> +		do_test test_format_c ${vmlinux} ${obj}
> +
> +		# btfdiff is only for vmlinux
> +		if [[ ${obj} != *.ko ]]; then
> +			do_test test_btfdiff ${vmlinux} ${obj}
> +		fi
> +	fi
> +}
> +
> +do_obj()
> +{
> +	local obj=$1
> +	local test_name=$2
> +	local vmlinux=${obj}/vmlinux
> +
> +	run_tests ${vmlinux} ${vmlinux} ${test_name}
> +
> +	for kmod in $(ls ${obj}/*.ko); do
> +		run_tests ${vmlinux} ${kmod} ${test_name}
> +	done
> +}
> +
> +main()
> +{
> +	local test_name="all"
> +
> +	while getopts 'b:o:dhP:B:fO:' opt; do
> +		case ${opt} in
> +		f)
> +			FAIL="yes"
> +			CLEANUP="no"
> +			;;
> +		o)
> +			obj="$OPTARG"
> +			;;
> +		O)
> +			OBJECTS="$OPTARG"
> +			;;
> +		b)
> +			BASE="$OPTARG"
> +			;;
> +		P)
> +			PAHOLE="$OPTARG"
> +			;;
> +		B)
> +			BPFTOOL="$OPTARG"
> +			;;
> +		h)
> +			usage
> +			exit 0
> +			;;
> +		\? )
> +			echo "Invalid Option: -$OPTARG"
> +			usage
> +			exit 1
> +			;;
> +		: )
> +			echo "Invalid Option: -$OPTARG requires an argument"
> +			usage
> +			exit 1
> +			;;
> +		esac
> +	done
> +	shift $((OPTIND -1))
> +
> +	if [[ $# -gt 1 ]]; then
> +		echo "Invalid test: $@"
> +		usage
> +		exit 1
> +	fi
> +
> +	if [[ $# -eq 1 ]]; then
> +		test_name="$@"
> +	fi
> +
> +	if [[ "${BASE}" == "" ]]; then
> +		BASE=$(mktemp -d /tmp/pahole.test.XXX)
> +	else
> +		mkdir -p ${BASE}
> +	fi
> +
> +	PAHOLE=$(realpath ${PAHOLE})
> +	BPFTOOL=$(realpath ${BPFTOOL})
> +	OBJECTS=$(realpath ${OBJECTS})
> +
> +	echo "pahole:  ${PAHOLE}"
> +	echo "bpftool: ${BPFTOOL}"
> +	echo "base:    ${BASE}"
> +	echo "objects: ${obj:-${OBJECTS}}"
> +	echo "fail:    ${FAIL}"
> +	echo "cleanup: ${CLEANUP}"
> +	echo
> +
> +	if [[ "${obj:=""}" != "" ]]; then
> +		do_obj ${obj} ${test_name}
> +	else
> +		for obj in $(ls ${OBJECTS}); do
> +			do_obj ${OBJECTS}/${obj} ${test_name}
> +		done
> +	fi
> +}
> +
> +catch()
> +{
> +	local exit_code=$1
> +	if [[ "${BASE:=""}" != "" && "${CLEANUP}" == "yes" ]]; then
> +		rm -rf ${BASE}
> +	else
> +		echo
> +		echo "Keeping test data in: ${BASE}"
> +	fi
> +	exit ${exit_code}
> +}
> +
> +trap 'catch "$?"' EXIT
> +
> +main "$@"
> -- 
> 2.29.2
>