* Re: [PATCH v10] Rust Oe-Selftest implementation
[not found] <20220904063331.82988-1-naveen.gowda@windriver.com>
@ 2022-09-04 13:34 ` Richard Purdie
2022-09-05 14:55 ` [OE-core] " pgowda cve
0 siblings, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2022-09-04 13:34 UTC (permalink / raw)
To: Naveen, openembedded-core; +Cc: Randy.MacLeod, vinay.m.engg
On Sat, 2022-09-03 at 23:33 -0700, Naveen wrote:
> The patch implements Rust testing framework similar to other selftest,
> specifically the gcc selftest in OE. It uses the client and server
> based method to test the binaries for cross-target on the image.
> The test framework is a wrapper around the Rust build system as ./x.py
> test.
> It tests many functionalities of Rust distribution like tools,
> documentation, libraries, packages, tools, Cargo, Crater etc.
> Please refer the following link for detailed description of Rust
> testing:-
> https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-tests
>
> To support the rust tests in oe-core, the following functions were
> added:-
> setup_cargo_environment(): Build bootstrap and some early stage tools.
> do_rust_setup_snapshot(): Install the snapshot version of rust binaries.
> do_configure(): To generate config.toml
> do_compile(): To build "remote-test-server" for qemu target image.
>
> Approximate Number of Tests Run in the Rust Testsuite :- 18000
> Approximate Number of Tests that FAIL in bitbake environment :- 100-150
> Normally majority of the testcases are present in major folder "test/"
> It contributes to more than 80% of the testcases present in Rust test
> framework. These tests pass as expected on any Rust versions without
> much fuss. The tests that fail are of less important and contribute to
> less than 2% of the total testcases. These minor tests are observed to
> work on some versions and fail on others. They have to be added, ignored
> or excluded for different versions as per the behavior.
> These tests have been ignored or excluded in the Rust selftest
> environment to generate success of completing the testsuite.
>
> These tests work in parallel mode even in the skipped test mode as
> expected. Although the patch to disable tests is large, it is very simple
> in that it only disables tests. When updating to a newer version of Rust,
> the patch can usually be ported in a day.
This is getting better. See comments inline below. Running it locally I
got:
Compiling lint-docs v0.1.0 (/media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/src/tools/lint-docs)
Finished release [optimized] target(s) in 6.32s
Building stage0 tool rustbook (x86_64-unknown-linux-gnu)
Compiling autocfg v1.1.0
Compiling proc-macro2 v1.0.37
Compiling unicode-xid v0.2.2
Compiling cfg-if v0.1.10
Compiling syn v1.0.91
Compiling ppv-lite86 v0.2.8
Compiling serde v1.0.125
Compiling memchr v2.4.1
Compiling siphasher v0.3.3
Compiling log v0.4.14
Compiling parking_lot_core v0.8.5
Compiling lazy_static v1.4.0
Compiling scopeguard v1.1.0
Compiling bitflags v1.2.1
Compiling smallvec v1.8.1
Compiling new_debug_unreachable v1.0.4
Compiling mac v0.1.1
Compiling matches v0.1.8
Compiling ucd-trie v0.1.3
Compiling utf-8 v0.7.5
Compiling precomputed-hash v0.1.1
Compiling maplit v1.0.2
Compiling tinyvec v0.3.4
Compiling version_check v0.9.3
Compiling serde_derive v1.0.125
Compiling once_cell v1.12.0
Compiling termcolor v1.1.2
Compiling hashbrown v0.12.3
Compiling unicode-segmentation v1.9.0
Compiling percent-encoding v2.1.0
Compiling regex-syntax v0.6.25
Compiling os_str_bytes v6.0.0
Compiling pulldown-cmark v0.9.1
Compiling strsim v0.10.0
Compiling quick-error v1.2.3
Compiling anyhow v1.0.51
Compiling regex-automata v0.1.10
Compiling textwrap v0.15.0
Compiling strum v0.18.0
Compiling unicode-width v0.1.8
Compiling quick-error v2.0.0
Compiling vec_map v0.8.2
Compiling strsim v0.8.0
Compiling shlex v1.0.0
Compiling topological-sort v0.1.0
Compiling ansi_term v0.12.1
Compiling libc v0.2.126
Compiling getrandom v0.2.0
Compiling instant v0.1.12
Compiling atty v0.2.14
Compiling time v0.1.43
Compiling rand v0.8.5
Compiling unicode-bidi v0.3.4
Compiling futf v0.1.5
Compiling phf_shared v0.10.0
Compiling textwrap v0.11.0
Compiling humantime v1.3.0
Compiling pest v2.1.3
Compiling form_urlencoded v1.0.1
Compiling tendril v0.4.3
Compiling clap_lex v0.2.2
Compiling lock_api v0.4.7
Compiling num-traits v0.2.12
Compiling indexmap v1.9.1
Compiling num-integer v0.1.43
Compiling phf v0.10.1
Compiling clap v2.34.0
Compiling unicode-normalization v0.1.13
Compiling unicase v2.6.0
Compiling heck v0.3.1
Compiling aho-corasick v0.7.18
Compiling bstr v0.2.13
Compiling parking_lot v0.11.2
Compiling quote v1.0.18
Compiling tempfile v3.2.0
Compiling rand_core v0.6.2
Compiling rand_chacha v0.3.0
Compiling idna v0.2.0
Compiling pest_meta v2.1.3
Compiling clap v3.2.5
Compiling opener v0.5.0
Compiling url v2.2.2
Compiling chrono v0.4.19
Compiling phf_generator v0.10.0
Compiling regex v1.5.5
Compiling phf_codegen v0.10.0
Compiling string_cache_codegen v0.5.2
Compiling markup5ever v0.11.0
Compiling env_logger v0.7.1
Compiling clap_complete v3.1.1
Compiling string_cache v0.8.3Compiling lint-docs v0.1.0 (/media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/src/tools/lint-docs)
Finished release [optimized] target(s) in 6.32s
Building stage0 tool rustbook (x86_64-unknown-linux-gnu)
Compiling autocfg v1.1.0
Compiling proc-macro2 v1.0.37
Compiling unicode-xid v0.2.2
Compiling cfg-if v0.1.10
Compiling syn v1.0.91
Compiling ppv-lite86 v0.2.8
Compiling serde v1.0.125
Compiling memchr v2.4.1
Compiling siphasher v0.3.3
Compiling log v0.4.14
Compiling parking_lot_core v0.8.5
Compiling lazy_static v1.4.0
Compiling scopeguard v1.1.0
Compiling bitflags v1.2.1
Compiling smallvec v1.8.1
Compiling new_debug_unreachable v1.0.4
Compiling mac v0.1.1
Compiling matches v0.1.8
Compiling ucd-trie v0.1.3
Compiling utf-8 v0.7.5
Compiling precomputed-hash v0.1.1
Compiling maplit v1.0.2
Compiling tinyvec v0.3.4
Compiling version_check v0.9.3
Compiling serde_derive v1.0.125
Compiling once_cell v1.12.0
Compiling termcolor v1.1.2
Compiling hashbrown v0.12.3
Compiling unicode-segmentation v1.9.0
Compiling percent-encoding v2.1.0
Compiling regex-syntax v0.6.25
Compiling os_str_bytes v6.0.0
Compiling pulldown-cmark v0.9.1
Compiling strsim v0.10.0
Compiling quick-error v1.2.3
Compiling anyhow v1.0.51
Compiling regex-automata v0.1.10
Compiling textwrap v0.15.0
Compiling strum v0.18.0
Compiling unicode-width v0.1.8
Compiling quick-error v2.0.0
Compiling vec_map v0.8.2
Compiling strsim v0.8.0
Compiling shlex v1.0.0
Compiling topological-sort v0.1.0
Compiling ansi_term v0.12.1
Compiling libc v0.2.126
Compiling getrandom v0.2.0
Compiling instant v0.1.12
Compiling atty v0.2.14
Compiling time v0.1.43
Compiling rand v0.8.5
Compiling unicode-bidi v0.3.4
Compiling futf v0.1.5
Compiling phf_shared v0.10.0
Compiling textwrap v0.11.0
Compiling humantime v1.3.0
Compiling pest v2.1.3
Compiling form_urlencoded v1.0.1
Compiling tendril v0.4.3
Compiling clap_lex v0.2.2
Compiling lock_api v0.4.7
Compiling num-traits v0.2.12
Compiling indexmap v1.9.1
Compiling num-integer v0.1.43
Compiling phf v0.10.1
Compiling clap v2.34.0
Compiling unicode-normalization v0.1.13
Compiling unicase v2.6.0
Compiling heck v0.3.1
Compiling aho-corasick v0.7.18
Compiling bstr v0.2.13
Compiling parking_lot v0.11.2
Compiling quote v1.0.18
Compiling tempfile v3.2.0
Compiling rand_core v0.6.2
Compiling rand_chacha v0.3.0
Compiling idna v0.2.0
Compiling pest_meta v2.1.3
Compiling clap v3.2.5
Compiling opener v0.5.0
Compiling url v2.2.2
Compiling chrono v0.4.19
Compiling phf_generator v0.10.0
Compiling regex v1.5.5
Compiling phf_codegen v0.10.0
Compiling string_cache_codegen v0.5.2
Compiling markup5ever v0.11.0
Compiling env_logger v0.7.1
Compiling clap_complete v3.1.1
Compiling string_cache v0.8.3
Compiling serde_json v1.0.59
Compiling toml v0.5.7
Compiling pest_generator v2.1.3
Compiling html5ever v0.26.0
Compiling pest_derive v2.1.0
Compiling strum_macros v0.18.0
Compiling handlebars v4.1.0
Compiling ammonia v3.2.0
Compiling elasticlunr-rs v2.3.9
Compiling mdbook v0.4.18
Compiling rustbook v0.1.0 (/media/build1/poky/build-
st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-
src/src/tools/rustbook)
Finished release [optimized] target(s) in 17.73s
Rustbook (x86_64-poky-linux-gnu) - rustc
Documenting stage1 std (x86_64-poky-linux-gnu)
Documenting core v0.0.0 (/media/build1/poky/build-st/tmp/work/core2-
64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/library/core)
thread 'main' panicked at 'RUSTDOC_LIBDIR was not set',
src/bootstrap/bin/rustdoc.rs:15:48
note: run with `RUST_BACKTRACE=1` environment variable to display a
backtrace
error: could not document `core`
Caused by:
process didn't exit successfully: `/media/build1/poky/build-
st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-
src/build/bootstrap/debug/rustdoc --edition=2021 --crate-type lib --
crate-name core library/core/src/lib.rs --target x86_64-poky-linux-gnu
-o /media/build1/poky/build-st/tmp/work/core2-64-poky-
linux/rust/1.63.0-r0/rustc-1.63.0-src/build/x86_64-unknown-linux-
gnu/stage1-std/x86_64-poky-linux-gnu/doc -Zunstable-options --check-cfg
'names()' --check-cfg 'values()' --error-format=json --json=diagnostic-
rendered-ansi,artifacts,future-incompat --markdown-css rust.css --
markdown-no-toc -Z unstable-options --resource-suffix 1.63.0 --index-
page /media/build1/poky/build-st/tmp/work/core2-64-poky-
linux/rust/1.63.0-r0/rustc-1.63.0-src/src/doc/index.md -C
metadata=af344c1b4dc223ee -L dependency=/media/build1/poky/build-
st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-
src/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-poky-linux-
gnu/release/deps -L dependency=/media/build1/poky/build-
st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-
src/build/x86_64-unknown-linux-gnu/stage1-std/release/deps -Csymbol-
mangling-version=legacy -Zunstable-options -Zunstable-options '--check-
cfg=values(bootstrap)' '--check-cfg=values(stdarch_intel_sde)' '--
check-cfg=values(no_fp_fmt_parse)' '--check-
cfg=values(no_global_oom_handling)' '--check-cfg=values(freebsd12)' '--
check-cfg=values(backtrace_in_libstd)' '--check-
cfg=values(target_env,"libnx")' '--check-
cfg=values(target_os,"watchos")' '--check-
cfg=values(target_arch,"asmjs","spirv","nvptx","nvptx64","le32","xtensa
")' '--check-cfg=values(dont_compile_me)' -Dwarnings '-
Wrustdoc::invalid_codeblock_attributes' --crate-version 1.63.0 '-
Zcrate-attr=doc(html_root_url="https://doc.rust-lang.org/1.63.0/")'`
(exit status: 101)
Compiling serde_json v1.0.59
Compiling toml v0.5.7
Compiling pest_generator v2.1.3
Compiling html5ever v0.26.0
Compiling pest_derive v2.1.0
Compiling strum_macros v0.18.0
Compiling handlebars v4.1.0
Compiling ammonia v3.2.0
Compiling elasticlunr-rs v2.3.9
Compiling mdbook v0.4.18
Compiling rustbook v0.1.0 (/media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/src/tools/rustbook)
Finished release [optimized] target(s) in 17.73s
Rustbook (x86_64-poky-linux-gnu) - rustc
Documenting stage1 std (x86_64-poky-linux-gnu)
Documenting core v0.0.0 (/media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/library/core)
thread 'main' panicked at 'RUSTDOC_LIBDIR was not set', src/bootstrap/bin/rustdoc.rs:15:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: could not document `core`
Caused by:
process didn't exit successfully: `/media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/build/bootstrap/debug/rustdoc --edition=2021 --crate-type lib --crate-name core library/core/src/lib.rs --target x86_64-poky-linux-gnu -o /media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-poky-linux-gnu/doc -Zunstable-options --check-cfg 'names()' --check-cfg 'values()' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --markdown-css rust.css --markdown-no-toc -Z unstable-options --resource-suffix 1.63.0 --index-page /media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/src/doc/index.md -C metadata=af344c1b4dc223ee -L dependency=/media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-poky-linux-gnu/release/deps -L dependency=/media/build1/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.63.0-r0/rustc-1.63.0-src/build/x86_64-unknown-linux-gnu/stage1-std/release/deps -Csymbol-mangling-version=legacy -Zunstable-options -Zunstable-options '--check-cfg=values(bootstrap)' '--check-cfg=values(stdarch_intel_sde)' '--check-cfg=values(no_fp_fmt_parse)' '--check-cfg=values(no_global_oom_handling)' '--check-cfg=values(freebsd12)' '--check-cfg=values(backtrace_in_libstd)' '--check-cfg=values(target_env,"libnx")' '--check-cfg=values(target_os,"watchos")' '--check-cfg=values(target_arch,"asmjs","spirv","nvptx","nvptx64","le32","xtensa")' '--check-cfg=values(dont_compile_me)' -Dwarnings '-Wrustdoc::invalid_codeblock_attributes' --crate-version 1.63.0 '-Zcrate-attr=doc(html_root_url="https://doc.rust-lang.org/1.63.0/")'` (exit status: 101)
> diff --git a/meta/lib/oeqa/selftest/cases/rust.py b/meta/lib/oeqa/selftest/cases/rust.py
> new file mode 100644
> index 0000000000..274bfc144a
> --- /dev/null
> +++ b/meta/lib/oeqa/selftest/cases/rust.py
> @@ -0,0 +1,63 @@
> +# SPDX-License-Identifier: MIT
> +import os
> +import subprocess
> +from oeqa.core.decorator import OETestTag
> +from oeqa.core.case import OEPTestResultTestCase
> +from oeqa.selftest.case import OESelftestTestCase
> +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, Command
> +from oeqa.utils.sshcontrol import SSHControl
> +
> +# Total time taken for testing is of about 2hr 20min, with PARALLEL_MAKE set to 40 number of jobs.
> +class RustSelfTestBase(OESelftestTestCase, OEPTestResultTestCase):
> +
> + def run_check_emulated(self, *args, **kwargs):
> + # build remote-test-server before image build
> + recipe = "rust"
> + bitbake("{} -c test_compile".format(recipe))
> + builddir = get_bb_var("RUSTSRC", "rust")
Something is mangling spaces in your patch. There shouldn't be tabs
here.
> + # build core-image-minimal with required packages
> + default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"]
> + features = []
> + features.append('IMAGE_FEATURES += "ssh-server-openssh"')
> + features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages)))
> + self.write_config("\n".join(features))
> + bitbake("core-image-minimal")
> + # wrap the execution with a qemu instance.
> + # Tests are run with 512 tasks in parallel to execute all tests very quickly
> + with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu:
> + # Copy remote-test-server to image through scp
> + ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root")
> + ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/")
> + # Execute remote-test-server on image through background ssh
> + command = '~/remote-test-server -v remote'
> + sshrun=subprocess.Popen(("ssh", '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command),
> + shell=False,
> + stdout=subprocess.PIPE,
> + stderr=subprocess.PIPE)
> + # Get the values of variables.
> + tcpath = get_bb_var("TARGET_SYS", "rust")
> + targetsys = get_bb_var("RUST_TARGET_SYS", "rust")
> + rustlibpath = get_bb_var("WORKDIR", "rust")
> + tmpdir = get_bb_var("TMPDIR", "rust")
> +
> + # Exclude the test folders that error out while building
> + # TODO: Fix the errors and include them for testing
> + # no-fail-fast: Run all tests regardless of failure.
> + # bless: First runs rustfmt to format the codebase,
> + # then runs tidy checks.
> + testargs = "--exclude src/test/rustdoc --exclude src/test/rustdoc-json --exclude src/test/run-make-fulldeps --exclude src/tools/tidy --exclude src/tools/rustdoc-themes --exclude src/rustdoc-json-types --exclude src/librustdoc --exclude src/doc/unstable-book --exclude src/doc/rustdoc --exclude src/doc/rustc --exclude compiler/rustc --exclude library/panic_abort --exclude library/panic_unwind --exclude src/test/rustdoc --no-doc --no-fail-fast --bless"
> +
> + # Set path for target-poky-linux-gcc, RUST_TARGET_PATH and hosttools.
> + cmd = " export PATH=%s/recipe-sysroot-native/usr/bin:$PATH;" % rustlibpath
> + cmd = cmd + " export TARGET_VENDOR=\"-poky\";"
> + cmd = cmd + " export PATH=%s/recipe-sysroot-native/usr/bin/%s:%s/hosttools:$PATH;" % (rustlibpath, tcpath, tmpdir)
> + cmd = cmd + " export RUST_TARGET_PATH=%s/rust-targets;" % rustlibpath
> + # Trigger testing.
> + cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip
> + cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py test %s --target %s ;" % (builddir, testargs, targetsys)
> + result = runCmd(cmd)
> +
> +@OETestTag("toolchain-system")
> +class RustSelfTestSystemEmulated(RustSelfTestBase):
> + def test_rust(self):
> + self.run_check_emulated("rust")
I think this can be merged with the RustSelfTestBase class as I don't
really see why it is separate?
> diff --git a/meta/recipes-devtools/rust/rust-source.inc b/meta/recipes-devtools/rust/rust-source.inc
> index ce6c983fc0..7386d7a3ea 100644
> --- a/meta/recipes-devtools/rust/rust-source.inc
> +++ b/meta/recipes-devtools/rust/rust-source.inc
> @@ -4,7 +4,8 @@ SRC_URI[rust.sha256sum] = "8f44af6dc44cc4146634a4dd5e4cc5470b3052a2337019b870c0e
> SRC_URI:append:class-target:pn-rust = " \
> file://hardcodepaths.patch \
> file://crossbeam_atomic.patch \
> - file://0001-Add-ENOTSUP-constant-for-riscv32-musl.patch"
> + file://0001-Add-ENOTSUP-constant-for-riscv32-musl.patch \
> + file://rust-oe-selftest.patch;striplevel=1"
> SRC_URI:append:class-nativesdk:pn-nativesdk-rust = " file://hardcodepaths.patch"
>
> RUSTSRC = "${WORKDIR}/rustc-${PV}-src"
> diff --git a/meta/recipes-devtools/rust/rust.inc b/meta/recipes-devtools/rust/rust.inc
> index 284347dedc..70f4d01808 100644
> --- a/meta/recipes-devtools/rust/rust.inc
> +++ b/meta/recipes-devtools/rust/rust.inc
> @@ -18,7 +18,7 @@ export FORCE_CRATE_HASH="${BB_TASKHASH}"
> RUST_ALTERNATE_EXE_PATH ?= "${STAGING_LIBDIR}/llvm-rust/bin/llvm-config"
> RUST_ALTERNATE_EXE_PATH_NATIVE = "${STAGING_LIBDIR_NATIVE}/llvm-rust/bin/llvm-config"
>
> -# We don't want to use bitbakes vendoring because the rust sources do their
> +# We don't want to use bitbake's vendoring because the rust sources do their
> # own vendoring.
> CARGO_DISABLE_BITBAKE_VENDORING = "1"
>
If we want to change grammar, it should be in a separate patch.
> @@ -55,6 +55,7 @@ do_rust_setup_snapshot () {
> addtask rust_setup_snapshot after do_unpack before do_configure
> do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot"
> do_rust_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER"
> +addtask do_test_compile after do_configure do_rust_gen_targets
>
Please put that alongside the task definition.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread