From: Joseph Kogut <joseph.kogut@gmail.com>
To: buildroot@buildroot.org
Cc: Joseph Kogut <joseph.kogut@gmail.com>
Subject: [Buildroot] [PATCH v3] utils/test-pkg: add concurrency parameter
Date: Tue, 28 Oct 2025 12:26:10 -0700 [thread overview]
Message-ID: <20251028-concurrent-test-pkg-v3-1-ae014d18e5f5@gmail.com> (raw)
Builds run by test-pkg are often not CPU limited, and so running
multiple builds concurrently has the potential to speed things up quite
a lot.
Add a concurrency parameter to the script that allows for multiple
builds to run concurrently. The default is the current behavior, running
a single build at a time. If -C|--concurrent=0 is specified, the number of
concurrent builds is set to the output of $(nproc) to match the number
of logical CPUs.
$ time bash utils/test-pkg -c sdl2.config -p sdl2 -C1
bootlin-armv5-uclibc [1/6]: OK
bootlin-armv7-glibc [2/6]: OK
bootlin-armv7m-uclibc [3/6]: SKIPPED
bootlin-x86-64-musl [4/6]: OK
br-arm-full-static [5/6]: SKIPPED
arm-aarch64 [6/6]: OK
6 builds, 2 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 2m32.979s
user 6m15.356s
sys 0m30.878s
$ time bash utils/test-pkg -c sdl2.config -p sdl2 -C0
bootlin-armv5-uclibc [1/6]: OK
bootlin-armv7-glibc [2/6]: OK
bootlin-armv7m-uclibc [3/6]: SKIPPED
bootlin-x86-64-musl [4/6]: OK
br-arm-full-static [5/6]: SKIPPED
arm-aarch64 [6/6]: OK
6 builds, 2 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 0m41.704s
user 6m40.802s
sys 0m29.976s
Signed-off-by: Joseph Kogut <joseph.kogut@gmail.com>
---
The patch appears to behave as intended in my testing, and it speeds up
package testing quite a lot. The commit description shows the results
testing sdl2, and I've pasted another result below from building libpng,
which illustrates the time saved with a large number of builds.
$ time utils/test-pkg -c libpng.config -p libpng -a
arm-aarch64 [ 1/35]: OK
bootlin-aarch64-glibc [ 2/35]: OK
bootlin-arcle-hs38-uclibc [ 3/35]: OK
bootlin-armv5-uclibc [ 4/35]: OK
bootlin-armv7-glibc [ 5/35]: OK
bootlin-armv7-musl [ 6/35]: OK
bootlin-armv7m-uclibc [ 7/35]: OK
bootlin-m68k-5208-uclibc [ 8/35]: OK
bootlin-m68k-68040-uclibc [ 9/35]: OK
bootlin-microblazeel-uclibc [10/35]: OK
bootlin-mipsel-uclibc [11/35]: OK
bootlin-mipsel32r6-glibc [12/35]: OK
bootlin-openrisc-uclibc [13/35]: OK
bootlin-powerpc-e500mc-uclibc [14/35]: OK
bootlin-powerpc64le-power8-glibc [15/35]: OK
bootlin-riscv32-glibc [16/35]: OK
bootlin-riscv64-glibc [17/35]: OK
bootlin-riscv64-musl [18/35]: OK
bootlin-s390x-z13-glibc [19/35]: OK
bootlin-sh4-uclibc [20/35]: OK
bootlin-sparc-uclibc [21/35]: OK
bootlin-sparc64-glibc [22/35]: OK
bootlin-x86-64-glibc [23/35]: OK
bootlin-x86-64-musl [24/35]: OK
bootlin-x86-64-uclibc [25/35]: OK
bootlin-x86-i686-musl [26/35]: OK
bootlin-xtensa-uclibc [27/35]: OK
br-arm-basic [28/35]: OK
br-arm-full-nothread [29/35]: OK
br-arm-full-static [30/35]: OK
br-i386-pentium4-full [31/35]: OK
br-mips64-n64-full [32/35]: OK
br-mips64r6-el-hf-glibc [33/35]: OK
br-powerpc-603e-basic-cpp [34/35]: OK
br-powerpc64-power7-glibc [35/35]: OK
35 builds, 0 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 23m34.593s
user 25m30.113s
sys 2m10.823s
$ time utils/test-pkg -c libpng.config -p libpng -a -C0
arm-aarch64 [ 1/35]: OK
bootlin-aarch64-glibc [ 2/35]: OK
bootlin-arcle-hs38-uclibc [ 3/35]: OK
bootlin-armv5-uclibc [ 4/35]: OK
bootlin-armv7-glibc [ 5/35]: OK
bootlin-armv7-musl [ 6/35]: OK
bootlin-armv7m-uclibc [ 7/35]: OK
bootlin-m68k-5208-uclibc [ 8/35]: OK
bootlin-m68k-68040-uclibc [ 9/35]: OK
bootlin-microblazeel-uclibc [10/35]: OK
bootlin-mipsel-uclibc [11/35]: OK
bootlin-mipsel32r6-glibc [12/35]: OK
bootlin-openrisc-uclibc [13/35]: OK
bootlin-powerpc-e500mc-uclibc [14/35]: OK
bootlin-powerpc64le-power8-glibc [15/35]: OK
bootlin-riscv32-glibc [16/35]: OK
bootlin-riscv64-glibc [17/35]: OK
bootlin-riscv64-musl [18/35]: OK
bootlin-s390x-z13-glibc [19/35]: OK
bootlin-sh4-uclibc [20/35]: OK
bootlin-sparc-uclibc [21/35]: OK
bootlin-sparc64-glibc [22/35]: OK
bootlin-x86-64-glibc [23/35]: OK
bootlin-x86-64-musl [24/35]: OK
bootlin-x86-64-uclibc [25/35]: OK
bootlin-x86-i686-musl [26/35]: OK
bootlin-xtensa-uclibc [27/35]: OK
br-arm-basic [28/35]: OK
br-arm-full-nothread [29/35]: OK
br-arm-full-static [30/35]: OK
br-i386-pentium4-full [31/35]: OK
br-mips64-n64-full [32/35]: OK
br-mips64r6-el-hf-glibc [33/35]: OK
br-powerpc-603e-basic-cpp [34/35]: OK
br-powerpc64-power7-glibc [35/35]: OK
35 builds, 0 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 2m6.510s
user 41m24.460s
sys 3m31.484s
This test was performed on a 16-core Ryzen 9 9950X, and the
all-toolchain build was ~11x faster compared to the same serial run.
As always, feedback is very welcome.
---
Changes in v3:
- Fix improper removal of '-T' short option during rebase
- Minor tweak to remove newline between prompt and output
- Simplify make job termination
- Link to v2: https://lore.kernel.org/r/20251023-concurrent-test-pkg-v2-1-959ad443d49b@gmail.com
Changes in v2:
- Rebase on origin/master
- Properly restore cursor on interrupt
- Properly terminate running jobs on interrupt
- Add animated spinner for running builds
- Simplify status updates
- Link to v1: https://lore.kernel.org/r/20251022-concurrent-test-pkg-v1-1-1fe96df1102b@gmail.com
---
utils/test-pkg | 142 ++++++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 121 insertions(+), 21 deletions(-)
diff --git a/utils/test-pkg b/utils/test-pkg
index cea7ace7cb..f9dc86343d 100755
--- a/utils/test-pkg
+++ b/utils/test-pkg
@@ -3,27 +3,45 @@ set -e
TOOLCHAINS_CSV='support/config-fragments/autobuild/toolchain-configs.csv'
TEMP_CONF=""
-abort=0
+
+# associative array tracking running build jobs by PID
+declare -a running
+
+# offet for end of output
+end_offs=0
+
+restore_cursor() {
+ tput cnorm
+ tput rc
+ tput cud "$((end_offs + 1))"
+ printf '\n'
+}
do_abort() {
- abort=1
+ restore_cursor
+ do_clean
}
do_clean() {
if [ -n "${TEMP_CONF}" ]; then
rm -f "${TEMP_CONF}"
fi
+
+ # Terminate any running jobs
+ pkill -TERM --parent $$ 2>/dev/null || true
+ wait
}
main() {
local o O opts
- local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only
+ local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only \
+ concurrency
local ret nb nb_skip nb_fail nb_legal nb_show nb_tc build_dir keep
local -a toolchains
local pkg_br_name
- o='hakc:d:n:p:r:t:T:'
- O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:,toolchain-name:'
+ o='hakc:d:n:p:r:t:T:C:'
+ O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:,toolchain-name:,concurrency:'
opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")"
eval set -- "${opts}"
@@ -33,6 +51,7 @@ main() {
number=0
mode=0
prepare_only=0
+ concurrency=1
toolchains_csv="${TOOLCHAINS_CSV}"
while [ ${#} -gt 0 ]; do
case "${1}" in
@@ -69,6 +88,9 @@ main() {
(-T|--toolchain-name)
toolchain_name="${2}"; shift 2
;;
+ (-C|--concurrency)
+ concurrency="${2}"; shift 2
+ ;;
(--)
shift; break
;;
@@ -88,6 +110,13 @@ main() {
if [ ! -e "${cfg}" ]; then
printf "error: %s: no such file\n" "${cfg}" >&2; exit 1
fi
+ if [ "${concurrency}" -eq 0 ] 2>/dev/null; then
+ concurrency=$(nproc)
+ fi
+
+ if ! [ "${concurrency}" -gt 0 ] 2>/dev/null; then
+ printf "error: concurrency must be an integer\n" >&2; exit 1
+ fi
if [ -z "${dir}" ]; then
dir="${HOME}/br-test-pkg"
fi
@@ -137,31 +166,99 @@ main() {
nb_fail=0
nb_legal=0
nb_show=0
- for toolchainconfig in "${toolchains[@]}"; do
- : $((nb++))
- toolchain="$(basename "${toolchainconfig}" .config)"
- build_dir="${dir}/${toolchain}"
- printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} "${nb}" "${nb_tc}"
- build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" "${prepare_only}" && ret=0 || ret=${?}
- case ${ret} in
- (0) printf "OK\n";;
- (1) : $((nb_skip++)); printf "SKIPPED\n";;
- (2) : $((nb_fail++)); printf "FAILED\n";;
- (3) : $((nb_legal++)); printf "FAILED\n";;
- (4) : $((nb_show++)); printf "FAILED\n";;
- esac
- if [ "${abort}" -eq 1 ]; then
- return 1
- fi
+ tput civis
+
+ # Allocate lines for all toolchains up front to avoid scroll-invalidating
+ # the saved cursor
+ for ((i = 0; i < nb_tc; i++)); do printf '\n'; done
+ tput cuu "$((nb_tc + 1))"
+ tput sc
+
+ declare -A pid_to_idx
+ declare -a display_order
+
+ spinc='/-\|'
+ spini=0
+ while (( nb < nb_tc || ${#running[@]} > 0)); do
+ while (( nb < nb_tc && ${#running[@]} < concurrency )); do
+ toolchainconfig=${toolchains[$nb]}
+ toolchain="$(basename "${toolchainconfig}" .config)"
+ build_dir="${dir}/${toolchain}"
+
+ build_one \
+ "${build_dir}" \
+ "${toolchainconfig}" \
+ "${cfg}" \
+ "${pkg}" \
+ "${prepare_only}" &
+
+ pid=$!; pid_to_idx[${pid}]=${nb}
+ running+=( "$pid" )
+ slot=${#display_order[@]}
+ display_order+=( "$nb" )
+ end_offs=${#display_order[@]}
+ nb=$((nb + 1))
+ done
+
+ for i in "${!running[@]}"; do
+ pid="${running[${i}]}"
+ idx="${pid_to_idx[$pid]}"
+ toolchainconfig=${toolchains[$idx]}
+ toolchain="$(basename "${toolchainconfig}" .config)"
+
+ if ! kill -0 "${pid}" 2>/dev/null; then
+ wait "${pid}" && ret=0 || ret=${?}
+ case "${ret}" in
+ (0) stat="OK";;
+ (1) : $((nb_skip++)); stat="SKIPPED";;
+ (2) : $((nb_fail++)); stat="FAILED";;
+ (3) : $((nb_legal++)); stat="FAILED";;
+ (4) : $((nb_show++)); stat="FAILED";;
+ esac
+
+ unset 'running[i]'
+ else
+ stat="${spinc:$spini:1}"
+ fi
+
+ # Find the line to print the status on for this PID
+ for slot in "${!display_order[@]}"; do
+ if [[ ${display_order[$slot]} -eq ${idx} ]]; then
+ break
+ fi
+ done
+
+ update_line "$slot" "%40s [%*d/%d]: %s" \
+ "${toolchain}" \
+ "${#nb_tc}" \
+ "$((idx + 1))" \
+ "${nb_tc}" \
+ "${stat}"
+ done
+
+ running=( "${running[@]}" )
+ spini=$(((spini+1) % ${#spinc}))
+ sleep 0.1
done
+ restore_cursor
printf "%d builds, %d skipped, %d build failed, %d legal-info failed, %d show-info failed\n" \
"${nb}" "${nb_skip}" "${nb_fail}" "${nb_legal}" "${nb_show}"
return $((nb_fail + nb_legal))
}
+update_line() {
+ local slot=$1; shift
+ tput rc
+ tput cud "$((slot + 1))"
+ fmt=$1; shift
+
+ # shellcheck disable=SC2059
+ printf -- "\033[K${fmt}" "$@"
+}
+
build_one() {
local dir="${1}"
local toolchainconfig="${2}"
@@ -288,6 +385,9 @@ Options:
-r N, --random N
Limit the tests to the N randomly selected toolchains.
+ -C N, --concurrency N
+ Run N builds concurrently. If N is 0, match the number of logical CPUs.
+
-t CSVFILE, --toolchains-csv CSVFILE
CSV file containing the paths to config fragments of toolchains to
try. If not specified, the toolchains in ${TOOLCHAINS_CSV} will be
---
base-commit: 6144b0f4b73bea810809f09d23bbe76b4979bc13
change-id: 20250619-concurrent-test-pkg-f3ad6d3c01b4
Best regards,
--
Joseph Kogut <joseph.kogut@gmail.com>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
next reply other threads:[~2025-10-28 19:26 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-10-28 19:26 Joseph Kogut [this message]
2026-02-03 16:48 ` [Buildroot] [PATCH v3] utils/test-pkg: add concurrency parameter Thomas Petazzoni via buildroot
2026-02-03 17:24 ` Joseph Kogut
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20251028-concurrent-test-pkg-v3-1-ae014d18e5f5@gmail.com \
--to=joseph.kogut@gmail.com \
--cc=buildroot@buildroot.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.