* [Buildroot] [PATCH v3] utils/test-pkg: add concurrency parameter
@ 2025-10-28 19:26 Joseph Kogut
2026-02-03 16:48 ` Thomas Petazzoni via buildroot
0 siblings, 1 reply; 3+ messages in thread
From: Joseph Kogut @ 2025-10-28 19:26 UTC (permalink / raw)
To: buildroot; +Cc: Joseph Kogut
Builds run by test-pkg are often not CPU limited, and so running
multiple builds concurrently has the potential to speed things up quite
a lot.
Add a concurrency parameter to the script that allows for multiple
builds to run concurrently. The default is the current behavior, running
a single build at a time. If -C|--concurrent=0 is specified, the number of
concurrent builds is set to the output of $(nproc) to match the number
of logical CPUs.
$ time bash utils/test-pkg -c sdl2.config -p sdl2 -C1
bootlin-armv5-uclibc [1/6]: OK
bootlin-armv7-glibc [2/6]: OK
bootlin-armv7m-uclibc [3/6]: SKIPPED
bootlin-x86-64-musl [4/6]: OK
br-arm-full-static [5/6]: SKIPPED
arm-aarch64 [6/6]: OK
6 builds, 2 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 2m32.979s
user 6m15.356s
sys 0m30.878s
$ time bash utils/test-pkg -c sdl2.config -p sdl2 -C0
bootlin-armv5-uclibc [1/6]: OK
bootlin-armv7-glibc [2/6]: OK
bootlin-armv7m-uclibc [3/6]: SKIPPED
bootlin-x86-64-musl [4/6]: OK
br-arm-full-static [5/6]: SKIPPED
arm-aarch64 [6/6]: OK
6 builds, 2 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 0m41.704s
user 6m40.802s
sys 0m29.976s
Signed-off-by: Joseph Kogut <joseph.kogut@gmail.com>
---
The patch appears to behave as intended in my testing, and it speeds up
package testing quite a lot. The commit description shows the results
testing sdl2, and I've pasted another result below from building libpng,
which illustrates the time saved with a large number of builds.
$ time utils/test-pkg -c libpng.config -p libpng -a
arm-aarch64 [ 1/35]: OK
bootlin-aarch64-glibc [ 2/35]: OK
bootlin-arcle-hs38-uclibc [ 3/35]: OK
bootlin-armv5-uclibc [ 4/35]: OK
bootlin-armv7-glibc [ 5/35]: OK
bootlin-armv7-musl [ 6/35]: OK
bootlin-armv7m-uclibc [ 7/35]: OK
bootlin-m68k-5208-uclibc [ 8/35]: OK
bootlin-m68k-68040-uclibc [ 9/35]: OK
bootlin-microblazeel-uclibc [10/35]: OK
bootlin-mipsel-uclibc [11/35]: OK
bootlin-mipsel32r6-glibc [12/35]: OK
bootlin-openrisc-uclibc [13/35]: OK
bootlin-powerpc-e500mc-uclibc [14/35]: OK
bootlin-powerpc64le-power8-glibc [15/35]: OK
bootlin-riscv32-glibc [16/35]: OK
bootlin-riscv64-glibc [17/35]: OK
bootlin-riscv64-musl [18/35]: OK
bootlin-s390x-z13-glibc [19/35]: OK
bootlin-sh4-uclibc [20/35]: OK
bootlin-sparc-uclibc [21/35]: OK
bootlin-sparc64-glibc [22/35]: OK
bootlin-x86-64-glibc [23/35]: OK
bootlin-x86-64-musl [24/35]: OK
bootlin-x86-64-uclibc [25/35]: OK
bootlin-x86-i686-musl [26/35]: OK
bootlin-xtensa-uclibc [27/35]: OK
br-arm-basic [28/35]: OK
br-arm-full-nothread [29/35]: OK
br-arm-full-static [30/35]: OK
br-i386-pentium4-full [31/35]: OK
br-mips64-n64-full [32/35]: OK
br-mips64r6-el-hf-glibc [33/35]: OK
br-powerpc-603e-basic-cpp [34/35]: OK
br-powerpc64-power7-glibc [35/35]: OK
35 builds, 0 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 23m34.593s
user 25m30.113s
sys 2m10.823s
$ time utils/test-pkg -c libpng.config -p libpng -a -C0
arm-aarch64 [ 1/35]: OK
bootlin-aarch64-glibc [ 2/35]: OK
bootlin-arcle-hs38-uclibc [ 3/35]: OK
bootlin-armv5-uclibc [ 4/35]: OK
bootlin-armv7-glibc [ 5/35]: OK
bootlin-armv7-musl [ 6/35]: OK
bootlin-armv7m-uclibc [ 7/35]: OK
bootlin-m68k-5208-uclibc [ 8/35]: OK
bootlin-m68k-68040-uclibc [ 9/35]: OK
bootlin-microblazeel-uclibc [10/35]: OK
bootlin-mipsel-uclibc [11/35]: OK
bootlin-mipsel32r6-glibc [12/35]: OK
bootlin-openrisc-uclibc [13/35]: OK
bootlin-powerpc-e500mc-uclibc [14/35]: OK
bootlin-powerpc64le-power8-glibc [15/35]: OK
bootlin-riscv32-glibc [16/35]: OK
bootlin-riscv64-glibc [17/35]: OK
bootlin-riscv64-musl [18/35]: OK
bootlin-s390x-z13-glibc [19/35]: OK
bootlin-sh4-uclibc [20/35]: OK
bootlin-sparc-uclibc [21/35]: OK
bootlin-sparc64-glibc [22/35]: OK
bootlin-x86-64-glibc [23/35]: OK
bootlin-x86-64-musl [24/35]: OK
bootlin-x86-64-uclibc [25/35]: OK
bootlin-x86-i686-musl [26/35]: OK
bootlin-xtensa-uclibc [27/35]: OK
br-arm-basic [28/35]: OK
br-arm-full-nothread [29/35]: OK
br-arm-full-static [30/35]: OK
br-i386-pentium4-full [31/35]: OK
br-mips64-n64-full [32/35]: OK
br-mips64r6-el-hf-glibc [33/35]: OK
br-powerpc-603e-basic-cpp [34/35]: OK
br-powerpc64-power7-glibc [35/35]: OK
35 builds, 0 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 2m6.510s
user 41m24.460s
sys 3m31.484s
This test was performed on a 16-core Ryzen 9 9950X, and the
all-toolchain build was ~11x faster compared to the same serial run.
As always, feedback is very welcome.
---
Changes in v3:
- Fix improper removal of '-T' short option during rebase
- Minor tweak to remove newline between prompt and output
- Simplify make job termination
- Link to v2: https://lore.kernel.org/r/20251023-concurrent-test-pkg-v2-1-959ad443d49b@gmail.com
Changes in v2:
- Rebase on origin/master
- Properly restore cursor on interrupt
- Properly terminate running jobs on interrupt
- Add animated spinner for running builds
- Simplify status updates
- Link to v1: https://lore.kernel.org/r/20251022-concurrent-test-pkg-v1-1-1fe96df1102b@gmail.com
---
utils/test-pkg | 142 ++++++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 121 insertions(+), 21 deletions(-)
diff --git a/utils/test-pkg b/utils/test-pkg
index cea7ace7cb..f9dc86343d 100755
--- a/utils/test-pkg
+++ b/utils/test-pkg
@@ -3,27 +3,45 @@ set -e
TOOLCHAINS_CSV='support/config-fragments/autobuild/toolchain-configs.csv'
TEMP_CONF=""
-abort=0
+
+# associative array tracking running build jobs by PID
+declare -a running
+
+# offet for end of output
+end_offs=0
+
+restore_cursor() {
+ tput cnorm
+ tput rc
+ tput cud "$((end_offs + 1))"
+ printf '\n'
+}
do_abort() {
- abort=1
+ restore_cursor
+ do_clean
}
do_clean() {
if [ -n "${TEMP_CONF}" ]; then
rm -f "${TEMP_CONF}"
fi
+
+ # Terminate any running jobs
+ pkill -TERM --parent $$ 2>/dev/null || true
+ wait
}
main() {
local o O opts
- local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only
+ local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only \
+ concurrency
local ret nb nb_skip nb_fail nb_legal nb_show nb_tc build_dir keep
local -a toolchains
local pkg_br_name
- o='hakc:d:n:p:r:t:T:'
- O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:,toolchain-name:'
+ o='hakc:d:n:p:r:t:T:C:'
+ O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:,toolchain-name:,concurrency:'
opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")"
eval set -- "${opts}"
@@ -33,6 +51,7 @@ main() {
number=0
mode=0
prepare_only=0
+ concurrency=1
toolchains_csv="${TOOLCHAINS_CSV}"
while [ ${#} -gt 0 ]; do
case "${1}" in
@@ -69,6 +88,9 @@ main() {
(-T|--toolchain-name)
toolchain_name="${2}"; shift 2
;;
+ (-C|--concurrency)
+ concurrency="${2}"; shift 2
+ ;;
(--)
shift; break
;;
@@ -88,6 +110,13 @@ main() {
if [ ! -e "${cfg}" ]; then
printf "error: %s: no such file\n" "${cfg}" >&2; exit 1
fi
+ if [ "${concurrency}" -eq 0 ] 2>/dev/null; then
+ concurrency=$(nproc)
+ fi
+
+ if ! [ "${concurrency}" -gt 0 ] 2>/dev/null; then
+ printf "error: concurrency must be an integer\n" >&2; exit 1
+ fi
if [ -z "${dir}" ]; then
dir="${HOME}/br-test-pkg"
fi
@@ -137,31 +166,99 @@ main() {
nb_fail=0
nb_legal=0
nb_show=0
- for toolchainconfig in "${toolchains[@]}"; do
- : $((nb++))
- toolchain="$(basename "${toolchainconfig}" .config)"
- build_dir="${dir}/${toolchain}"
- printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} "${nb}" "${nb_tc}"
- build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" "${prepare_only}" && ret=0 || ret=${?}
- case ${ret} in
- (0) printf "OK\n";;
- (1) : $((nb_skip++)); printf "SKIPPED\n";;
- (2) : $((nb_fail++)); printf "FAILED\n";;
- (3) : $((nb_legal++)); printf "FAILED\n";;
- (4) : $((nb_show++)); printf "FAILED\n";;
- esac
- if [ "${abort}" -eq 1 ]; then
- return 1
- fi
+ tput civis
+
+ # Allocate lines for all toolchains up front to avoid scroll-invalidating
+ # the saved cursor
+ for ((i = 0; i < nb_tc; i++)); do printf '\n'; done
+ tput cuu "$((nb_tc + 1))"
+ tput sc
+
+ declare -A pid_to_idx
+ declare -a display_order
+
+ spinc='/-\|'
+ spini=0
+ while (( nb < nb_tc || ${#running[@]} > 0)); do
+ while (( nb < nb_tc && ${#running[@]} < concurrency )); do
+ toolchainconfig=${toolchains[$nb]}
+ toolchain="$(basename "${toolchainconfig}" .config)"
+ build_dir="${dir}/${toolchain}"
+
+ build_one \
+ "${build_dir}" \
+ "${toolchainconfig}" \
+ "${cfg}" \
+ "${pkg}" \
+ "${prepare_only}" &
+
+ pid=$!; pid_to_idx[${pid}]=${nb}
+ running+=( "$pid" )
+ slot=${#display_order[@]}
+ display_order+=( "$nb" )
+ end_offs=${#display_order[@]}
+ nb=$((nb + 1))
+ done
+
+ for i in "${!running[@]}"; do
+ pid="${running[${i}]}"
+ idx="${pid_to_idx[$pid]}"
+ toolchainconfig=${toolchains[$idx]}
+ toolchain="$(basename "${toolchainconfig}" .config)"
+
+ if ! kill -0 "${pid}" 2>/dev/null; then
+ wait "${pid}" && ret=0 || ret=${?}
+ case "${ret}" in
+ (0) stat="OK";;
+ (1) : $((nb_skip++)); stat="SKIPPED";;
+ (2) : $((nb_fail++)); stat="FAILED";;
+ (3) : $((nb_legal++)); stat="FAILED";;
+ (4) : $((nb_show++)); stat="FAILED";;
+ esac
+
+ unset 'running[i]'
+ else
+ stat="${spinc:$spini:1}"
+ fi
+
+ # Find the line to print the status on for this PID
+ for slot in "${!display_order[@]}"; do
+ if [[ ${display_order[$slot]} -eq ${idx} ]]; then
+ break
+ fi
+ done
+
+ update_line "$slot" "%40s [%*d/%d]: %s" \
+ "${toolchain}" \
+ "${#nb_tc}" \
+ "$((idx + 1))" \
+ "${nb_tc}" \
+ "${stat}"
+ done
+
+ running=( "${running[@]}" )
+ spini=$(((spini+1) % ${#spinc}))
+ sleep 0.1
done
+ restore_cursor
printf "%d builds, %d skipped, %d build failed, %d legal-info failed, %d show-info failed\n" \
"${nb}" "${nb_skip}" "${nb_fail}" "${nb_legal}" "${nb_show}"
return $((nb_fail + nb_legal))
}
+update_line() {
+ local slot=$1; shift
+ tput rc
+ tput cud "$((slot + 1))"
+ fmt=$1; shift
+
+ # shellcheck disable=SC2059
+ printf -- "\033[K${fmt}" "$@"
+}
+
build_one() {
local dir="${1}"
local toolchainconfig="${2}"
@@ -288,6 +385,9 @@ Options:
-r N, --random N
Limit the tests to the N randomly selected toolchains.
+ -C N, --concurrency N
+ Run N builds concurrently. If N is 0, match the number of logical CPUs.
+
-t CSVFILE, --toolchains-csv CSVFILE
CSV file containing the paths to config fragments of toolchains to
try. If not specified, the toolchains in ${TOOLCHAINS_CSV} will be
---
base-commit: 6144b0f4b73bea810809f09d23bbe76b4979bc13
change-id: 20250619-concurrent-test-pkg-f3ad6d3c01b4
Best regards,
--
Joseph Kogut <joseph.kogut@gmail.com>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [Buildroot] [PATCH v3] utils/test-pkg: add concurrency parameter
2025-10-28 19:26 [Buildroot] [PATCH v3] utils/test-pkg: add concurrency parameter Joseph Kogut
@ 2026-02-03 16:48 ` Thomas Petazzoni via buildroot
2026-02-03 17:24 ` Joseph Kogut
0 siblings, 1 reply; 3+ messages in thread
From: Thomas Petazzoni via buildroot @ 2026-02-03 16:48 UTC (permalink / raw)
To: Joseph Kogut; +Cc: buildroot
Hello Joseph,
On Tue, Oct 28, 2025 at 12:26:10PM -0700, Joseph Kogut wrote:
> Builds run by test-pkg are often not CPU limited, and so running
> multiple builds concurrently has the potential to speed things up quite
> a lot.
Thanks for your patch, and sorry for the delay in getting back to
you. We took advantage of the on-going Buildroot Developers Meeting to
(finally) have a discussion around your contribution. We believe the
feature is definitely useful, however the hand-written implementation
of the parallelization seems a bit tricky. Can you instead use
something like GNU Parallel which is designed for that?
Thanks a lot!
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Buildroot] [PATCH v3] utils/test-pkg: add concurrency parameter
2026-02-03 16:48 ` Thomas Petazzoni via buildroot
@ 2026-02-03 17:24 ` Joseph Kogut
0 siblings, 0 replies; 3+ messages in thread
From: Joseph Kogut @ 2026-02-03 17:24 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: buildroot
Hello Thomas,
On Tue, Feb 3, 2026 at 8:48 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Thanks for your patch, and sorry for the delay in getting back to
> you. We took advantage of the on-going Buildroot Developers Meeting to
> (finally) have a discussion around your contribution. We believe the
> feature is definitely useful, however the hand-written implementation
> of the parallelization seems a bit tricky. Can you instead use
> something like GNU Parallel which is designed for that?
Thanks for taking the time to discuss this! I originally sent it as an
RFC because I had similar concerns regarding the implementation, and
wanted to start a discussion about something more suitable.
I can spin a GNU parallel version and resubmit.
Thanks,
Joseph
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-02-03 17:25 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-10-28 19:26 [Buildroot] [PATCH v3] utils/test-pkg: add concurrency parameter Joseph Kogut
2026-02-03 16:48 ` Thomas Petazzoni via buildroot
2026-02-03 17:24 ` Joseph Kogut
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox