Re: [OE-core] [RFC 0/4] Oe-selftest for Clang, LLVM, LLD

["Mathieu Dubois-Briand" <mathieu.dubois-briand@bootlin.com>]

On Tue Feb 3, 2026 at 3:03 PM CET, Deepesh via lists.openembedded.org Varatharajan wrote:
> From: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com>
>
> This patch series introduces a test framework to run selftests for both
> native and target builds of Clang, LLVM, and LLD.
>
> Summary of changes: 
> -Patch 1: Disable clang-tools-extra tests because Clang's CMake cannot 
>           see the llvm-bcanalyzer target.
> -Patch 2: Enabled the required CMake flags for clang, llvm & lld tests
> -Patch 3: Selftest script for Clang, LLVM, and LLD including exclude list
> -Patch 4: Fix rust build failure by replacing target llvm-config with native
>
> Testing:
> 1. The test suite has been successfully validated on x86-64, arm64, riscv64,
> and arm32 for all three components.
> 2. Clang, LLVM & LLD all together ~80K tests are there and we currently 
> ~200 tests to exclude list which are failing (0.25%).
>
> WIP:
> LLVM target tests are currently failing on x86. This is under investigation
> and will be addressed
>
> Deepesh Varatharajan (4):
>   clang-tools-extra: disable tests
>   clang: Enable tests for llvm, clang, lld
>   oeqa/selftest/clang: Add selftest for Clang/LLVM/LLD
>   rust: Fix rust build failure
>
>  meta/lib/oeqa/selftest/cases/clang.py         | 301 ++++++++++++++++++
>  .../0042-guard-clang-tools-extra-test.patch   |  74 +++++
>  meta/recipes-devtools/clang/clang_git.bb      |   3 +-
>  meta/recipes-devtools/clang/common.inc        |   1 +
>  meta/recipes-devtools/clang/lld_git.bb        |   2 +-
>  meta/recipes-devtools/clang/llvm_git.bb       |  11 +-
>  meta/recipes-devtools/rust/rust_1.92.0.bb     |   5 +-
>  7 files changed, 389 insertions(+), 8 deletions(-)
>  create mode 100644 meta/lib/oeqa/selftest/cases/clang.py
>  create mode 100644 meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch

Hi Deepesh,

Thanks for this series. I know it's still RFC, but it went in my
week-end batch of RFC patches tested on the autobuilder.

So far, we had the following issues:

2026-02-14 17:23:02,046 - oe-selftest - INFO - clang.ClangFamilyNativeSelfTest.test_clang_family_native (subunit.RemotedTestCase)
2026-02-14 17:23:02,049 - oe-selftest - INFO -  ... FAIL
...
subprocess.CalledProcessError: Command '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1314344/tmp/work/aarch64-linux/clang-native/21.1.8/build/bin/llvm-lit --filter-out 'arm|i386|x86_64|x86|mips|riscv|riscv64|ppc|ppc64|hexagon|sparc|sparcv9|msp430|loongarch|Mac|macho|Darwin|OSX|wasm|Windows|Win|MinGW|COFF|zos|FreeBSD|aix|fuchsia|ve' '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1314344/tmp/work/aarch64-linux/clang-native/21.1.8/build/test' -o '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1314344/tmp/work/aarch64-linux/clang-native/21.1.8/temp/clang-native-results.log'' returned non-zero exit status 127.
...

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3316
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3214
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3090

A second one:

2026-02-14 18:07:00,151 - oe-selftest - INFO - 2: 1/1 8/21 (222.35s) (0 failed) (clang.LLDSelfTestSystemEmulated.test_lld)
2026-02-14 18:07:00,152 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/qemux86-tc/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/clang.py", line 298, in test_lld
    raise AssertionError(f"llvm-lit failed for LLD: {output}")
AssertionError: llvm-lit failed for LLD: sh: line 1: cd: /srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2874234/tmp/work/core2-32-poky-linux/lld/21.1.8/build/bin: No such file or directory

https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/3136
https://autobuilder.yoctoproject.org/valkyrie/#/builders/42/builds/3113

And a third one:

2026-02-14 21:12:51,873 - oe-selftest - INFO - 3: 1/1 21/21 (18519.15s) (0 failed) (clang.LLVMSelfTestSystemEmulated.test_llvm)
2026-02-14 21:12:51,873 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/qemuriscv64-tc/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/clang.py", line 217, in test_llvm
    raise AssertionError(f"llvm-lit failed for LLVM: {output}")
AssertionError: llvm-lit failed for LLVM: /srv/pokybuild/yocto-worker/qemuriscv64-tc/build/build-st-2197806/tmp/hosttools/ld.gold: line 2: /usr/bin/ld.gold: No such file or directory
-- Testing: 22642 of 57881 tests, 4 workers --
...
Process ForkServerPoolWorker-2:
Traceback (most recent call last):
  File "/usr/lib/python3.14/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    ~~~^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/multiprocessing/queues.py", line 397, in put
    self._writer.send_bytes(obj)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/usr/lib/python3.14/multiprocessing/connection.py", line 206, in send_bytes
    self._send_bytes(m[offset:offset + size])
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/multiprocessing/connection.py", line 444, in _send_bytes
    self._send(header + buf)
    ~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/multiprocessing/connection.py", line 400, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

https://autobuilder.yoctoproject.org/valkyrie/#/builders/58/builds/1041
https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/3210

Thanks,
Mathieu

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com