Hi! First... what I am currently using for testing: gitlab; I simply watch for green ticks there. It takes ~15 minutes to get results. Red cross in compile stage is pretty sure sign of build error. Red cross in testing means I need to look at the results, but it is pretty easy to see "network timeout" or "board unavailable" problems, and mostly I can just hit retry and get the green tick. What wories me about gitlab is that we have single board for 4.4-st, and just two for 4.4-cip. Now, kernelci. https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html I got this link in the irc meeting: https://linux.kernelci.org/job/cip/ Lets start with build results. linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3 I click on "3" and get build error summary: Errors Summary 2 arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m 2 arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’ ...but those are not kernel problems, that's buildsystem incompatible with config or something like that. Nothing for me to solve. Let me go through more: linux-4.19.y-cip v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7 111 8 1 linux-4.19.y-cip v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e 106 7 1 linux-4.19.y-cip-rt v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c 181 16 6 linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5 linux-5.10.y-cip-rt v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3 linux-5.10.y-cip v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3 linux-4.19.y-cip v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120 That's 19 errors total + outlier of v4.19.239-cip72 with 120 errors. None of the 19 errors is real. KernelCI needs to fix their configs/build environment. 120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as cm_x2xx_defconfig ‐ arm ‐ gcc-10 0 warnings 0 errors. When I click for details I get "No test results found". https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/ Nothing for me to debug, either, agreed? So lets look at test results. Branch Kernel Commit Build Status Test Results Date linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3 2457 256 21 2022-06-12 21 failures. If I click that, I can see that most regressions (11) are at baseline test plan. The machine that looks relevant to us is qemu_arm64-virt-gicv2 defconfig+arm64-chromebook -- https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/ New regression, last pass: v5.10.112-cip6 I get to https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/ which looks quite useful, it lists successful run and failed run, v5.10.121-cip9 fails while v5.10.112-cip6 works. Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . And sorry, no I don't see anything I could debug there. Did the docker fail to start? Hmm. Lets compare it with the older results. https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible. Lets try one more. rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10 https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/ It gets me to: https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html I see some errors... Like missing firmware (not a kernel problem). At the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging? 1368 12:21:39.212881 /usr/bin/tpm2_getcap 1369 12:21:39.251829 ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2 1370 12:21:39.261387 ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error 1371 12:21:39.272254 ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001) 1372 12:21:39.281854 ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified 1373 12:21:39.285316 ERROR: Unable to run tpm2_getcap Userland seems to say that it does not support this device. I guess I could compare it with older boot? https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html Aha, no, could not, 404. Now, let's go through Jan's example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/? . Again, errors look like missconfigured build system, with arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3 arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’ Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404. I spent quite some time trying to click around to find an error I could debug. I could not find one. All of above looks like kernelCI configuration problems. Can they be fixed and/or filtered out? Thanks and best regards, Pavel -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany