[PATCH] bitbakery: automate build testing across all MACHINEs/LIBCs

[PATCH] bitbakery: automate build testing across all MACHINEs/LIBCs

[Trevor Gamblin]

bitbakery simplifies the process of build testing an upgraded or
patched recipe. It cycles through all qemu targets for glibc and musl,
and then generates a log file in the format "<recipe>-bitbakery.log",
containing the results of each build along with some basic system info
(time, hostname, OS, kernel version). Additionally, it retains the
previous run's log for comparison, renamed to <recipe>-bitbakery.log.old
when a new set of builds is triggered.

Sample log output:

BITBAKERY LOG FOR aspell
START TIME: 2020-01-17_08:39:05
HOSTNAME: yow-tgamblin-fedora1
OS: Fedora 31 (Server Edition)
KERNEL: 5.4.7-200.fc31.x86_64
===============
BUILD RESULTS:
[glibc]
FAIL: qemuarm64
PASS: qemuarm
PASS: qemuarmv5
PASS: qemumips64
PASS: qemumips
PASS: qemuppc
PASS: qemuriscv64
FAIL: qemux86-64
FAIL: qemux86
[musl]
PASS: qemuarm64
PASS: qemuarm
PASS: qemuarmv5
PASS: qemumips64
PASS: qemumips
PASS: qemuppc
PASS: qemuriscv64
PASS: qemux86-64
PASS: qemux86
===============
PASSED: 15
FAILED: 3
---
 scripts/bitbakery | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 scripts/bitbakery

diff --git a/scripts/bitbakery b/scripts/bitbakery
new file mode 100755
index 0000000000..cb2957a7bd
--- /dev/null
+++ b/scripts/bitbakery
@@ -0,0 +1,47 @@
+#!/bin/sh
+#  Copyright (c) 2020 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+# Get path to oe-core directory
+OE_CORE_PATH=$( cd - || exit )
+
+# Get target list and host machine information
+TARGET_LIST=$(ls "$OE_CORE_PATH"/meta/conf/machine | grep qemu | sed -e 's/\.conf//')
+LIBC_LIST="glibc musl"
+START_TIME=$(date "+%Y-%m-%d_%H:%M:%S")
+LOG_FILE="$1-bitbakery.log"
+OS_INFO=$(cat /etc/os-release | grep "PRETTY_NAME=" | awk -F "=" '{print $2}' | sed -e 's/^"//' -e 's/"$//')
+
+# Append an existing log file for this build with .old if one exists
+if [ -f "${LOG_FILE}" ]; then
+   mv "${LOG_FILE}" "${LOG_FILE}.old"
+else
+       touch "${LOG_FILE}"
+fi
+
+# Fill the log file with build and host info
+echo "BITBAKERY LOG FOR $1" >> "${LOG_FILE}"
+echo "START TIME: ${START_TIME}" >> "${LOG_FILE}"
+echo "HOSTNAME: $(uname -n)" >> "${LOG_FILE}"
+echo "OS: ${OS_INFO}" >> "${LOG_FILE}"
+echo "KERNEL: $(uname -r)" >> "${LOG_FILE}"
+echo "===============" >> "${LOG_FILE}"
+echo "BUILD RESULTS:" >> "${LOG_FILE}"
+
+for j in ${LIBC_LIST}; do
+    echo "[$j]" >> "${LOG_FILE}"
+    for i in ${TARGET_LIST}; do
+    echo "$i" "$j"; \
+        TCLIBC=$j MACHINE=$i bitbake "$1" && echo "PASS: $i" >> "${LOG_FILE}" || echo "FAIL: $i" >> "${LOG_FILE}";
+    done;
+done
+
+# Get pass/fail totals and add them to the end of the log
+PASSED=$(grep "PASS:" "${LOG_FILE}" | wc -l)
+FAILED=$(grep "FAIL:" "${LOG_FILE}" | wc -l)
+
+echo "===============" >> "${LOG_FILE}"
+echo "PASSED: ${PASSED}" >> "${LOG_FILE}"
+echo "FAILED: ${FAILED}" >> "${LOG_FILE}"
-- 
2.24.1

Re: [PATCH] bitbakery: automate build testing across all MACHINEs/LIBCs

[Trevor Gamblin]

On 1/20/20 8:27 AM, Trevor Gamblin wrote:
> bitbakery simplifies the process of build testing an upgraded or
> patched recipe. It cycles through all qemu targets for glibc and musl,
> and then generates a log file in the format "<recipe>-bitbakery.log",
> containing the results of each build along with some basic system info
> (time, hostname, OS, kernel version). Additionally, it retains the
> previous run's log for comparison, renamed to <recipe>-bitbakery.log.old
> when a new set of builds is triggered.
>
> Sample log output:
>
> BITBAKERY LOG FOR aspell
> START TIME: 2020-01-17_08:39:05
> HOSTNAME: yow-tgamblin-fedora1
> OS: Fedora 31 (Server Edition)
> KERNEL: 5.4.7-200.fc31.x86_64
> ===============
> BUILD RESULTS:
> [glibc]
> FAIL: qemuarm64
> PASS: qemuarm
> PASS: qemuarmv5
> PASS: qemumips64
> PASS: qemumips
> PASS: qemuppc
> PASS: qemuriscv64
> FAIL: qemux86-64
> FAIL: qemux86
> [musl]
> PASS: qemuarm64
> PASS: qemuarm
> PASS: qemuarmv5
> PASS: qemumips64
> PASS: qemumips
> PASS: qemuppc
> PASS: qemuriscv64
> PASS: qemux86-64
> PASS: qemux86
> ===============
> PASSED: 15
> FAILED: 3
> ---
>   scripts/bitbakery | 47 +++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 47 insertions(+)
>   create mode 100755 scripts/bitbakery
>
> diff --git a/scripts/bitbakery b/scripts/bitbakery
> new file mode 100755
> index 0000000000..cb2957a7bd
> --- /dev/null
> +++ b/scripts/bitbakery
> @@ -0,0 +1,47 @@
> +#!/bin/sh
> +#  Copyright (c) 2020 Wind River Systems, Inc.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +# Get path to oe-core directory
> +OE_CORE_PATH=$( cd - || exit )
> +
> +# Get target list and host machine information
> +TARGET_LIST=$(ls "$OE_CORE_PATH"/meta/conf/machine | grep qemu | sed -e 's/\.conf//')
> +LIBC_LIST="glibc musl"
> +START_TIME=$(date "+%Y-%m-%d_%H:%M:%S")
> +LOG_FILE="$1-bitbakery.log"
> +OS_INFO=$(cat /etc/os-release | grep "PRETTY_NAME=" | awk -F "=" '{print $2}' | sed -e 's/^"//' -e 's/"$//')
> +
> +# Append an existing log file for this build with .old if one exists
> +if [ -f "${LOG_FILE}" ]; then
> +   mv "${LOG_FILE}" "${LOG_FILE}.old"
> +else
> +       touch "${LOG_FILE}"
> +fi
> +
> +# Fill the log file with build and host info
> +echo "BITBAKERY LOG FOR $1" >> "${LOG_FILE}"
> +echo "START TIME: ${START_TIME}" >> "${LOG_FILE}"
> +echo "HOSTNAME: $(uname -n)" >> "${LOG_FILE}"
> +echo "OS: ${OS_INFO}" >> "${LOG_FILE}"
> +echo "KERNEL: $(uname -r)" >> "${LOG_FILE}"
> +echo "===============" >> "${LOG_FILE}"
> +echo "BUILD RESULTS:" >> "${LOG_FILE}"
> +
> +for j in ${LIBC_LIST}; do
> +    echo "[$j]" >> "${LOG_FILE}"
> +    for i in ${TARGET_LIST}; do
> +    echo "$i" "$j"; \
> +        TCLIBC=$j MACHINE=$i bitbake "$1" && echo "PASS: $i" >> "${LOG_FILE}" || echo "FAIL: $i" >> "${LOG_FILE}";
> +    done;
> +done
> +
> +# Get pass/fail totals and add them to the end of the log
> +PASSED=$(grep "PASS:" "${LOG_FILE}" | wc -l)
> +FAILED=$(grep "FAIL:" "${LOG_FILE}" | wc -l)
> +
> +echo "===============" >> "${LOG_FILE}"
> +echo "PASSED: ${PASSED}" >> "${LOG_FILE}"
> +echo "FAILED: ${FAILED}" >> "${LOG_FILE}"
Sending a v2 - didn't sign off.

✗ patchtest: failure for bitbakery: automate build testing across all MACHINEs/LIBCs

[Patchwork]

== Series Details ==

Series: bitbakery: automate build testing across all MACHINEs/LIBCs
Revision: 1
URL   : https://patchwork.openembedded.org/series/22159/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            bitbakery: automate build testing across all MACHINEs/LIBCs
 Issue             Patch is missing Signed-off-by [test_signed_off_by_presence] 
  Suggested fix    Sign off the patch (either manually or with "git commit --amend -s")



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe