Re: [PATCH v11] Rust Oe-Selftest implementation

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2022-09-07 at 20:27 +0530, pgowda.cve@gmail.com wrote:
> From: pgowda <pgowda.cve@gmail.com>
> 
> The patch implements Rust testing framework similar to other selftest,
> specifically the gcc selftest in OE. It uses the client and server
> based method to test the binaries for cross-target on the image.
> The test framework is a wrapper around the Rust build system as ./x.py
> test. It tests many functionalities of Rust distribution like tools,
> documentation, libraries, packages, tools, Cargo, Crater etc.
> Please refer the following link for detailed description of Rust
> testing:-
> https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-tests
> 
> To support the rust tests in oe-core, the following functions were
> added:-
> setup_cargo_environment(): Build bootstrap and some early stage tools.
> do_rust_setup_snapshot(): Install the snapshot version of rust binaries.
> do_configure(): To generate config.toml
> do_compile(): To build "remote-test-server" for qemu target image.
> 
> Approximate Number of Tests Run in the Rust Testsuite :- 18000
> Approximate Number of Tests that FAIL in bitbake environment :- 100-150
> Normally majority of the testcases are present in major folder "test/"
> It contributes to more than 80% of the testcases present in Rust test
> framework. These tests pass as expected on any Rust versions without
> much fuss. The tests that fail are of less important and contribute to
> less than 2% of the total testcases. These minor tests are observed to
> work on some versions and fail on others. They have to be added, ignored
> or excluded for different versions as per the behavior.
> These tests have been ignored or excluded in the Rust selftest
> environment to generate success of completing the testsuite.
> 
> These tests work in parallel mode even in the skipped test mode as
> expected. Although the patch to disable tests is large, it is very simple
> in that it only disables tests. When updating to a newer version of Rust,
> the patch can usually be ported in a day.
> 
> Signed-off-by: pgowda <pgowda.cve@gmail.com>
> Signed-off-by: Vinay Kumar <vinay.m.engg@gmail.com>
> ---
>  meta/lib/oeqa/selftest/cases/rust.py          |  54 ++
>  meta/recipes-devtools/rust/rust-source.inc    |   3 +-
>  meta/recipes-devtools/rust/rust.inc           |   1 +
>  .../rust/rust/rust-oe-selftest.patch          | 508 ++++++++++++++++++
>  meta/recipes-devtools/rust/rust_1.63.0.bb     |   5 +
>  5 files changed, 570 insertions(+), 1 deletion(-)
>  create mode 100644 meta/lib/oeqa/selftest/cases/rust.py
>  create mode 100644 meta/recipes-devtools/rust/rust/rust-oe-selftest.patch

I ran this through some testing on the autobuilder. The first time was
ok, it only ran on qemux86 and qemux86-64 as it was only enabled for
toolchain-system but ran successfully. 

On the second run I added @OETestTag("toolchain-user") as well which
meant it ran for arm/mips/powerpc. It looks like there was one worrying
intermittent failure:

qemux86-64 intermittent issue:
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5788/steps/19/logs/stdio

and then some arch specific failures:

qemuppc failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/5790/steps/19/logs/stdio

qemumips64 failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5786/steps/20/logs/stdio

qemumips failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/5804/steps/20/logs/stdio

qemuarm64, qemuarm and qemux86 passed on this run.

We need to understand and fix that qemux86-64 intermittent issue before
this can merge. I wasn't sure what we want to do with mips and ppc.

Cheers,

Richard