Re: [v3] [RFC] Merge meta-rust to oe-core

["Randy MacLeod" <randy.macleod@windriver.com>]

On 2021-08-10 8:22 p.m., Randy MacLeod wrote:
> 
> I started working on the merge of meta-rust to oe-core again.
> 
> CCing widely at the beginning this time.
> Please reply to me privately if you want to be dropped.
> 
> I really need other people to sign up as [co-]maintainers!
> 
> I pushed what I did today to:
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rmacleod/rust-redo-aug-10-2021-a 
> 
> 
> $ git log --stat below.
> 
> I've only started but so far, what I see is that, as expected, 
> rust-hello-world works for
> qemux86-64, qemuarm and likely other targets. I'm doing a buildall-qemu 
> overnight.
> 
> The oe-selftests that we had problems with before are still a problem:
> 
> oe-selftest still has at least three tests failing:
> 
> 1.
> oe-selftest -r distrodata.Distrodata.test_maintainers
>    This is due having two versions of cargo and rust.
> 
>    --> I'll remove 1.51 unless people object.
> 
> 2.
> oe-selftest -r sstatetests.SStateTests.test_sstate_noop_samesigs
> oe-selftest -r sstatetests.SStateTests.test_sstate_sametune_samesigs
> 
> I've just started to look at these test failures.
> $ bitbake-dump-sigs shows a diff like:
> 
>   List of dependencies for variable BASEDEPENDS is set()
>   List of dependencies for variable BUILD_AR is {'BUILD_PREFIX'}
> -List of dependencies for variable BUILD_AS is {'BUILD_PREFIX', 
> 'BUILD_AS_ARCH'}
> +List of dependencies for variable BUILD_AS is {'BUILD_AS_ARCH', 
> 'BUILD_PREFIX'}
>   List of dependencies for variable BUILD_AS_ARCH is set()
> -List of dependencies for variable BUILD_CC is {'BUILD_PREFIX', 
> 'BUILD_CC_ARCH'}
> -List of dependencies for variable BUILD_CCLD is {'BUILD_PREFIX', 
> 'BUILD_CC_ARCH'}
> +List of dependencies for variable BUILD_CC is {'BUILD_CC_ARCH', 
> 'BUILD_PREFIX'}
> +List of dependencies for variable BUILD_CCLD is {'BUILD_CC_ARCH', 
> 'BUILD_PREFIX'}
>   List of dependencies for variable BUILD_CC_ARCH is set()
> 
> Tips welcome!
> 
> 
> Btw, I know that the rust community is working on reproducibility:
>     https://github.com/rust-lang/rust/labels/A-reproducibility
> This diffsigs problem may be related but I haven't looked at the issue 
> long enough yet to tell.
> 
> 
> 4.
> I'll run a test on the autobuilder to see if the debian build from Feb 
> is still a problem:
>    rust-native doesn't build on debian8:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/116/builds/40/steps/13/logs/stdio 
> 
> 
> 
> I'll run buildall-qemu rust-hello-world overnight.

Results below.

Vinay or I will look into glibc/qemuppc64
if no one else is working on that.

The musl build failures are all alike with the error shown below.
Has anyone see that before?

../Randy


BUILDALL-QEMU LOG FOR rust-hello-world

START TIME: 2021-08-10_17:23:20

HOSTNAME: ala-lpggp3

HOST OS: Ubuntu 18.04.3 LTS

HOST KERNEL: 5.4.0-80-generic

===============

BUILD RESULTS:

[glibc]

PASS: qemuriscv32

PASS: qemuarmv5

PASS: qemumips

PASS: qemux86-64

PASS: qemuarm64

PASS: qemumips64

PASS: qemuarm

PASS: qemuppc

PASS: qemuriscv64

FAIL: qemuppc64

PASS: qemux86

[musl]

FAIL: qemuriscv32

FAIL: qemuarmv5

FAIL: qemumips

PASS: qemux86-64

PASS: qemuarm64

FAIL: qemumips64

FAIL: qemuarm

FAIL: qemuppc

FAIL: qemuriscv64

FAIL: qemuppc64

FAIL: qemux86

===============

PASSED: 12

FAILED: 10



---

Typical musl build failure log:

Sstate summary: Wanted 62 Local 0 Network 0 Missed 62 Current 142 (0% 
match, 69% complete) 


NOTE: Executing Tasks 
 


ERROR: rust-cross-arm-1.54.0-r0 do_prepare_recipe_sysroot: Error 
executing a python function in exec_python_func() autogenerated: 


 
 


The stack trace of python calls that resulted in this exception/failure 
was: 


File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 
 


      0001: 
 


  *** 0002:extend_recipe_sysroot(d) 
 


      0003: 


File: 
'/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/classes/staging.bbclass', 
lineno: 582, function: extend_recipe_sysroot 


      0578:                    if "/bin/" in l or "/sbin/" in l: 


      0579:                        # defer /*bin/* files until last in 
case they need libs 


      0580:                        binfiles[l] = (targetdir, dest) 
 


      0581:                    else: 
 


  *** 0582:                        staging_copyfile(l, targetdir, dest, 
postinsts, seendirs) 


      0583: 


      0584:    # Handle deferred binfiles 
 


      0585:    for l in binfiles: 


      0586:        (targetdir, dest) = binfiles[l] 
 


File: 
'/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/classes/staging.bbclass', 
lineno: 149, function: staging_copyfile 


      0145:    if os.path.islink(c): 
 


      0146:        linkto = os.readlink(c) 
 


      0147:        if os.path.lexists(dest): 


      0148:            if not os.path.islink(dest): 
 


  *** 0149:                raise OSError(errno.EEXIST, "Link %s already 
exists as a file" % dest, dest) 


      0150:            if os.readlink(dest) == linkto: 
 


      0151:                return dest 
 


      0152:            raise OSError(errno.EEXIST, "Link %s already 
exists to a different location? (%s vs %s)" % (dest, os.readlink(dest), 
linkto), dest)

      0153:        os.symlink(linkto, dest) 
 


Exception: FileExistsError: [Errno 17] Link 
/ala-lpggp31/rmacleod/src/distro/yocto/b/rust-aug-10-master-redo-a/tmp/work/x86_64-linux/rust-cross-arm/1.54.0-r0/recipe-sysroot/usr/include/c++/11.2.0/arm-poky-linux'

 
 


ERROR: Logfile of failure stored in: 
/ala-lpggp31/rmacleod/src/distro/yocto/b/rust-aug-10-master-redo-a/tmp/work/x86_64-linux/rust-cross-arm/1.54.0-r0/temp/log.do_prepare_recipe_sysroot.73161 


ERROR: Task 
(/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/recipes-devtools/rust/rust-cross_1.54.0.bb:do_prepare_recipe_sysroot) 
failed with exit code '1'

NOTE: Tasks Summary: Attempted 755 tasks of which 645 didn't need to be 
rerun and 1 failed. 


 
 


Summary: 1 task failed: 
 


 
/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/recipes-devtools/rust/rust-cross_1.54.0.bb:do_prepare_recipe_sysroot 
 


Summary: There was 1 ERROR message shown, returning a non-zero exit 
code.

> 
> Thanks,
> 
> ../Randy
> 
> 
> $ git log --stat origin/master..HEAD
> commit 725c11999a38244cc28e1d1fe17a7e05c5bdb7a3
> Author: Randy MacLeod <randy.macleod@windriver.com>
> Date:   Wed Feb 24 20:48:23 2021
> 
>      maintainers: Add myself as maintainer for rust pkgs
>      I'm willing to do the rust package maintenance but
>      if anyone else wants to sign up, that would be welcome.
>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
> 
>   meta/conf/distro/include/maintainers.inc | 10 ++++++++++
>   1 file changed, 10 insertions(+)
> 
> commit a658c5c7fac6160f7720bbf30ae35ccc143fcb57
> Author: Randy MacLeod <randy.macleod@windriver.com>
> Date:   Wed Feb 24 20:48:22 2021
> 
>      cargo/rust/rustfmt: exclude from world
>      cargo, rust, and rustfmt can't be built for the targets yet
>      so exclude them from world builds.
>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
> 
>   meta/recipes-devtools/cargo/cargo.inc         | 1 +
>   meta/recipes-example/rustfmt/rustfmt_1.4.2.bb | 2 +-
>   2 files changed, 2 insertions(+), 1 deletion(-)
> 
> commit 8860562733d7cce3cede27fc611f984a2d9351d6
> Author: Randy MacLeod <randy.macleod@windriver.com>
> Date:   Wed Feb 24 20:48:20 2021
> 
>      rust: update the README to conform to being in oe-core
>      Provide the full link to the meta-rust issue since
>      the README is no longer part of meta-rust.
>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
> 
>   meta/recipes-devtools/rust/README-rust.md | 25 +++++++------------------
>   1 file changed, 7 insertions(+), 18 deletions(-)
> 
> commit 7398219dcfffe746038d36bbbaacbd52265a3fc7
> Author: Randy MacLeod <randy.macleod@windriver.com>
> Date:   Wed Feb 24 20:48:18 2021
> 
>      rust: mv README.md to recipes-devtools/rust/README-rust.md
>      Import the meta-rust/README.md but relocate and rename it.
>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
> 
>   meta/recipes-devtools/rust/README-rust.md | 69 
> +++++++++++++++++++++++++++++++
>   1 file changed, 69 insertions(+)
> 
> commit 80586a849e1c725310f58ed9705febf97bc00bda
> Author: Randy MacLeod <Randy.MacLeod@windriver.com>
> Date:   Tue Aug 10 13:52:19 2021
> 
>      rust: initial merge of most of meta-rust
>      In the meta-rust repo at commit:
>         448047c Upgrade to 1.54.0 (#359)
>      Make the required directories:
>        mkdir ../oe-core/meta/recipes-devtools/rust
>        mkdir ../oe-core/meta/recipes-devtools/cargo
>        mkdir ../oe-core/meta/recipes-example
>      and then:
>        cp recipes-devtools/rust/* ../oe-core/meta/recipes-devtools/rust
>        cp recipes-devtools/cargo/* ../oe-core/meta/recipes-devtools/cargo
>        cp lib/crate.py ../oe-core/meta/lib
>        cp recipes-example/* ../oe-core/meta/recipes-example
>        cp conf/distro/include/rust_* ../oe-core/meta/conf/distro/include/
>        cp classes/* ../oe-core/meta/classes/
>        cp recipes-core/packagegroups/packagegroup-rust-cross-canadian.bb 
> ../oe-core/meta/recipes-core/packagegroups
>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
> 
>   meta/classes/cargo.bbclass                         |  89 ++
>   meta/classes/cargo_common.bbclass                  | 129 +++
>   meta/classes/crate-fetch.bbclass                   |  13 +
>   meta/classes/rust-bin.bbclass                      | 149 ++++
>   meta/classes/rust-common.bbclass                   | 174 ++++
>   meta/classes/rust.bbclass                          |  45 +
>   meta/conf/distro/include/rust_security_flags.inc   |   7 +
>   meta/conf/distro/include/rust_versions.inc         |  13 +
>   meta/lib/crate.py                                  | 149 ++++
>   .../packagegroup-rust-cross-canadian.bb            |  18 +
>   .../cargo/cargo-1.51.0/0001-Disable-http2.patch    |  28 +
>   .../cargo/cargo-1.51.0/riscv-march.patch           |  73 ++
>   .../cargo/cargo-1.51.0/rv64gc.patch                |  37 +
>   .../cargo/cargo-cross-canadian.inc                 |  74 ++
>   .../cargo/cargo-cross-canadian_1.51.0.bb           |   6 +
>   .../cargo/cargo-cross-canadian_1.54.0.bb           |   6 +
>   meta/recipes-devtools/cargo/cargo.inc              |  50 ++
>   meta/recipes-devtools/cargo/cargo_1.51.0.bb        |   8 +
>   meta/recipes-devtools/cargo/cargo_1.54.0.bb        |   4 +
>   meta/recipes-devtools/rust/files/riscv-march.patch |  73 ++
>   meta/recipes-devtools/rust/files/rv64gc.patch      |  37 +
>   ...001-Add-base-definitions-for-riscv64-musl.patch | 904 
> ++++++++++++++++++++
>   ...-linux-musl-mod.rs-add-riscv64-to-b64-set.patch |  29 +
>   ...3-FIXUP-Correct-definitions-to-match-musl.patch | 740 
> +++++++++++++++++
>   ...pdate-1.51.0-checksums-for-modified-files.patch |  21 +
>   .../0004-Update-checksums-for-modified-files.patch |  21 +
>   ...-definitions-for-riscv64-musl-libc-0.2.93.patch | 905 
> +++++++++++++++++++++
>   ...-musl-mod.rs-add-riscv64-to-b64-set-libc-.patch |  31 +
>   ...ect-definitions-to-match-musl-libc-0.2.93.patch | 741 
> +++++++++++++++++
>   ...ksums-for-modified-files-for-rust-1.54.0-.patch |  23 +
>   meta/recipes-devtools/rust/libstd-rs.inc           |  40 +
>   meta/recipes-devtools/rust/libstd-rs_1.51.0.bb     |  12 +
>   meta/recipes-devtools/rust/libstd-rs_1.54.0.bb     |  11 +
>   meta/recipes-devtools/rust/rust-common.inc         | 346 ++++++++
>   .../rust/rust-cross-canadian-common.inc            |  53 ++
>   meta/recipes-devtools/rust/rust-cross-canadian.inc |  78 ++
>   .../rust/rust-cross-canadian_1.51.0.bb             |   6 +
>   .../rust/rust-cross-canadian_1.54.0.bb             |   6 +
>   meta/recipes-devtools/rust/rust-cross.inc          |  71 ++
>   meta/recipes-devtools/rust/rust-cross_1.51.0.bb    |   2 +
>   meta/recipes-devtools/rust/rust-cross_1.54.0.bb    |   2 +
>   meta/recipes-devtools/rust/rust-llvm.inc           |  63 ++
>   .../rust-llvm/0001-nfc-Fix-missing-include.patch   |  26 +
>   .../0002-llvm-allow-env-override-of-exe-path.patch |  32 +
>   meta/recipes-devtools/rust/rust-llvm_1.51.0.bb     |   7 +
>   meta/recipes-devtools/rust/rust-llvm_1.54.0.bb     |   5 +
>   .../recipes-devtools/rust/rust-snapshot-1.51.0.inc |  18 +
>   .../recipes-devtools/rust/rust-snapshot-1.54.0.inc |  18 +
>   meta/recipes-devtools/rust/rust-snapshot.inc       |   9 +
>   meta/recipes-devtools/rust/rust-source-1.51.0.inc  |   3 +
>   meta/recipes-devtools/rust/rust-source-1.54.0.inc  |   3 +
>   meta/recipes-devtools/rust/rust-source.inc         |   3 +
>   meta/recipes-devtools/rust/rust-target.inc         |  10 +
>   .../rust/rust-tools-cross-canadian.inc             |  38 +
>   .../rust/rust-tools-cross-canadian_1.51.0.bb       |   6 +
>   .../rust/rust-tools-cross-canadian_1.54.0.bb       |   6 +
>   meta/recipes-devtools/rust/rust.inc                | 193 +++++
>   ...t-Fix-dash-vs-underscore-mismatches-in-op.patch |  75 ++
>   meta/recipes-devtools/rust/rust_1.51.0.bb          |  16 +
>   meta/recipes-devtools/rust/rust_1.54.0.bb          |  11 +
>   .../rust-hello-world/0001-enable-LTO.patch         |  23 +
>   .../rust-hello-world/rust-hello-world_git.bb       |  17 +
>   meta/recipes-example/rustfmt/rustfmt_1.4.2.bb      | 171 ++++
>   63 files changed, 5977 insertions(+)
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux