Re: [v3] [RFC] Merge meta-rust to oe-core

["Randy MacLeod" <randy.macleod@windriver.com>]

On 2021-08-11 9:14 a.m., Randy MacLeod wrote:
> On 2021-08-10 8:22 p.m., Randy MacLeod wrote:
>>
>> I started working on the merge of meta-rust to oe-core again.
>>
>> CCing widely at the beginning this time.
>> Please reply to me privately if you want to be dropped.
>>
>> I really need other people to sign up as [co-]maintainers!
>>
>> I pushed what I did today to:
>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rmacleod/rust-redo-aug-10-2021-a 
>>
>>
>> $ git log --stat below.
>>
>> I've only started but so far, what I see is that, as expected, 
>> rust-hello-world works for
>> qemux86-64, qemuarm and likely other targets. I'm doing a 
>> buildall-qemu overnight.
>>
>> The oe-selftests that we had problems with before are still a problem:
>>
>> oe-selftest still has at least three tests failing:
>>
>> 1.
>> oe-selftest -r distrodata.Distrodata.test_maintainers
>>    This is due having two versions of cargo and rust.
>>
>>    --> I'll remove 1.51 unless people object.
>>
>> 2.
>> oe-selftest -r sstatetests.SStateTests.test_sstate_noop_samesigs
>> oe-selftest -r sstatetests.SStateTests.test_sstate_sametune_samesigs
>>
>> I've just started to look at these test failures.
>> $ bitbake-dump-sigs shows a diff like:
>>
>>   List of dependencies for variable BASEDEPENDS is set()
>>   List of dependencies for variable BUILD_AR is {'BUILD_PREFIX'}
>> -List of dependencies for variable BUILD_AS is {'BUILD_PREFIX', 
>> 'BUILD_AS_ARCH'}
>> +List of dependencies for variable BUILD_AS is {'BUILD_AS_ARCH', 
>> 'BUILD_PREFIX'}
>>   List of dependencies for variable BUILD_AS_ARCH is set()
>> -List of dependencies for variable BUILD_CC is {'BUILD_PREFIX', 
>> 'BUILD_CC_ARCH'}
>> -List of dependencies for variable BUILD_CCLD is {'BUILD_PREFIX', 
>> 'BUILD_CC_ARCH'}
>> +List of dependencies for variable BUILD_CC is {'BUILD_CC_ARCH', 
>> 'BUILD_PREFIX'}
>> +List of dependencies for variable BUILD_CCLD is {'BUILD_CC_ARCH', 
>> 'BUILD_PREFIX'}
>>   List of dependencies for variable BUILD_CC_ARCH is set()
>>
>> Tips welcome!
>>
>>
>> Btw, I know that the rust community is working on reproducibility:
>>     https://github.com/rust-lang/rust/labels/A-reproducibility
>> This diffsigs problem may be related but I haven't looked at the 
>> issue long enough yet to tell.
>>
>>
>> 4.
>> I'll run a test on the autobuilder to see if the debian build from 
>> Feb is still a problem:
>>    rust-native doesn't build on debian8:
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/116/builds/40/steps/13/logs/stdio 
>>
>>
>>
>> I'll run buildall-qemu rust-hello-world overnight.
>
> Results below.
>
> Vinay or I will look into glibc/qemuppc64
> if no one else is working on that.
>
> The musl build failures are all alike with the error shown below.
> Has anyone see that before?
>
> ../Randy
>
>
> BUILDALL-QEMU LOG FOR rust-hello-world
>
> START TIME: 2021-08-10_17:23:20
>
> HOSTNAME: ala-lpggp3
>
> HOST OS: Ubuntu 18.04.3 LTS
>
> HOST KERNEL: 5.4.0-80-generic
>
> ===============
>
> BUILD RESULTS:
>
> [glibc]
>
> PASS: qemuriscv32
>
> PASS: qemuarmv5
>
> PASS: qemumips
>
> PASS: qemux86-64
>
> PASS: qemuarm64
>
> PASS: qemumips64
>
> PASS: qemuarm
>
> PASS: qemuppc
>
> PASS: qemuriscv64
>
> FAIL: qemuppc64
>
> PASS: qemux86
>
> [musl]
>
> FAIL: qemuriscv32
>
> FAIL: qemuarmv5
>
> FAIL: qemumips
>
> PASS: qemux86-64
>
> PASS: qemuarm64
>
> FAIL: qemumips64
>
> FAIL: qemuarm
>
> FAIL: qemuppc
>
> FAIL: qemuriscv64
>
> FAIL: qemuppc64
>
> FAIL: qemux86
>
> ===============
>
> PASSED: 12
>
> FAILED: 10


The musl failures seemed odd so I did a build without any
sstate-cache and indeed things are not as bad as last night's runs show.

I get the same results with a second run with the newly generated 
sstate-cache.

$ cat rust-hello-world-buildall.log
BUILDALL-QEMU LOG FOR rust-hello-world
START TIME: 2021-08-11_06:53:10
HOSTNAME: ala-lpggp3
HOST OS: Ubuntu 18.04.3 LTS
HOST KERNEL: 5.4.0-80-generic
===============
BUILD RESULTS:
[musl]
FAIL: qemuriscv32
PASS: qemuarmv5
PASS: qemumips
PASS: qemux86-64
PASS: qemuarm64
PASS: qemumips64
PASS: qemuarm
FAIL: qemuppc
PASS: qemuriscv64
FAIL: qemuppc64
PASS: qemux86
===============
PASSED: 8
FAILED: 3

../Randy


>
>
>
> ---
>
> Typical musl build failure log:
>
> Sstate summary: Wanted 62 Local 0 Network 0 Missed 62 Current 142 (0% 
> match, 69% complete)
>
> NOTE: Executing Tasks
>
>
> ERROR: rust-cross-arm-1.54.0-r0 do_prepare_recipe_sysroot: Error 
> executing a python function in exec_python_func() autogenerated:
>
>
>
>
>
> The stack trace of python calls that resulted in this 
> exception/failure was:
>
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>
>
>      0001:
>
>
>  *** 0002:extend_recipe_sysroot(d)
>
>
>      0003:
>
> File: 
> '/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/classes/staging.bbclass', 
> lineno: 582, function: extend_recipe_sysroot
>
>      0578:                    if "/bin/" in l or "/sbin/" in l:
>
>      0579:                        # defer /*bin/* files until last in 
> case they need libs
>
>      0580:                        binfiles[l] = (targetdir, dest)
>
>
>      0581:                    else:
>
>
>  *** 0582:                        staging_copyfile(l, targetdir, dest, 
> postinsts, seendirs)
>
>      0583:
>
>      0584:    # Handle deferred binfiles
>
>
>      0585:    for l in binfiles:
>
>      0586:        (targetdir, dest) = binfiles[l]
>
>
> File: 
> '/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/classes/staging.bbclass', 
> lineno: 149, function: staging_copyfile
>
>      0145:    if os.path.islink(c):
>
>
>      0146:        linkto = os.readlink(c)
>
>
>      0147:        if os.path.lexists(dest):
>
>      0148:            if not os.path.islink(dest):
>
>
>  *** 0149:                raise OSError(errno.EEXIST, "Link %s already 
> exists as a file" % dest, dest)
>
>      0150:            if os.readlink(dest) == linkto:
>
>
>      0151:                return dest
>
>
>      0152:            raise OSError(errno.EEXIST, "Link %s already 
> exists to a different location? (%s vs %s)" % (dest, 
> os.readlink(dest), linkto), dest)
>
>      0153:        os.symlink(linkto, dest)
>
>
> Exception: FileExistsError: [Errno 17] Link 
> /ala-lpggp31/rmacleod/src/distro/yocto/b/rust-aug-10-master-redo-a/tmp/work/x86_64-linux/rust-cross-arm/1.54.0-r0/recipe-sysroot/usr/include/c++/11.2.0/arm-poky-linux'
>
>
>
>
>
> ERROR: Logfile of failure stored in: 
> /ala-lpggp31/rmacleod/src/distro/yocto/b/rust-aug-10-master-redo-a/tmp/work/x86_64-linux/rust-cross-arm/1.54.0-r0/temp/log.do_prepare_recipe_sysroot.73161 
>
>
> ERROR: Task 
> (/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/recipes-devtools/rust/rust-cross_1.54.0.bb:do_prepare_recipe_sysroot) 
> failed with exit code '1'
>
> NOTE: Tasks Summary: Attempted 755 tasks of which 645 didn't need to 
> be rerun and 1 failed.
>
>
>
>
>
> Summary: 1 task failed:
>
>
>
> /ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/recipes-devtools/rust/rust-cross_1.54.0.bb:do_prepare_recipe_sysroot 
>
>
>
> Summary: There was 1 ERROR message shown, returning a non-zero exit code.
>
>>
>> Thanks,
>>
>> ../Randy
>>
>>
>> $ git log --stat origin/master..HEAD
>> commit 725c11999a38244cc28e1d1fe17a7e05c5bdb7a3
>> Author: Randy MacLeod <randy.macleod@windriver.com>
>> Date:   Wed Feb 24 20:48:23 2021
>>
>>      maintainers: Add myself as maintainer for rust pkgs
>>      I'm willing to do the rust package maintenance but
>>      if anyone else wants to sign up, that would be welcome.
>>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
>>
>>   meta/conf/distro/include/maintainers.inc | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> commit a658c5c7fac6160f7720bbf30ae35ccc143fcb57
>> Author: Randy MacLeod <randy.macleod@windriver.com>
>> Date:   Wed Feb 24 20:48:22 2021
>>
>>      cargo/rust/rustfmt: exclude from world
>>      cargo, rust, and rustfmt can't be built for the targets yet
>>      so exclude them from world builds.
>>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
>>
>>   meta/recipes-devtools/cargo/cargo.inc         | 1 +
>>   meta/recipes-example/rustfmt/rustfmt_1.4.2.bb | 2 +-
>>   2 files changed, 2 insertions(+), 1 deletion(-)
>>
>> commit 8860562733d7cce3cede27fc611f984a2d9351d6
>> Author: Randy MacLeod <randy.macleod@windriver.com>
>> Date:   Wed Feb 24 20:48:20 2021
>>
>>      rust: update the README to conform to being in oe-core
>>      Provide the full link to the meta-rust issue since
>>      the README is no longer part of meta-rust.
>>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
>>
>>   meta/recipes-devtools/rust/README-rust.md | 25 
>> +++++++------------------
>>   1 file changed, 7 insertions(+), 18 deletions(-)
>>
>> commit 7398219dcfffe746038d36bbbaacbd52265a3fc7
>> Author: Randy MacLeod <randy.macleod@windriver.com>
>> Date:   Wed Feb 24 20:48:18 2021
>>
>>      rust: mv README.md to recipes-devtools/rust/README-rust.md
>>      Import the meta-rust/README.md but relocate and rename it.
>>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
>>
>>   meta/recipes-devtools/rust/README-rust.md | 69 
>> +++++++++++++++++++++++++++++++
>>   1 file changed, 69 insertions(+)
>>
>> commit 80586a849e1c725310f58ed9705febf97bc00bda
>> Author: Randy MacLeod <Randy.MacLeod@windriver.com>
>> Date:   Tue Aug 10 13:52:19 2021
>>
>>      rust: initial merge of most of meta-rust
>>      In the meta-rust repo at commit:
>>         448047c Upgrade to 1.54.0 (#359)
>>      Make the required directories:
>>        mkdir ../oe-core/meta/recipes-devtools/rust
>>        mkdir ../oe-core/meta/recipes-devtools/cargo
>>        mkdir ../oe-core/meta/recipes-example
>>      and then:
>>        cp recipes-devtools/rust/* ../oe-core/meta/recipes-devtools/rust
>>        cp recipes-devtools/cargo/* 
>> ../oe-core/meta/recipes-devtools/cargo
>>        cp lib/crate.py ../oe-core/meta/lib
>>        cp recipes-example/* ../oe-core/meta/recipes-example
>>        cp conf/distro/include/rust_* 
>> ../oe-core/meta/conf/distro/include/
>>        cp classes/* ../oe-core/meta/classes/
>>        cp 
>> recipes-core/packagegroups/packagegroup-rust-cross-canadian.bb 
>> ../oe-core/meta/recipes-core/packagegroups
>>      Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
>>
>>   meta/classes/cargo.bbclass                         |  89 ++
>>   meta/classes/cargo_common.bbclass                  | 129 +++
>>   meta/classes/crate-fetch.bbclass                   |  13 +
>>   meta/classes/rust-bin.bbclass                      | 149 ++++
>>   meta/classes/rust-common.bbclass                   | 174 ++++
>>   meta/classes/rust.bbclass                          |  45 +
>>   meta/conf/distro/include/rust_security_flags.inc   |   7 +
>>   meta/conf/distro/include/rust_versions.inc         |  13 +
>>   meta/lib/crate.py                                  | 149 ++++
>>   .../packagegroup-rust-cross-canadian.bb            |  18 +
>>   .../cargo/cargo-1.51.0/0001-Disable-http2.patch    |  28 +
>>   .../cargo/cargo-1.51.0/riscv-march.patch           |  73 ++
>>   .../cargo/cargo-1.51.0/rv64gc.patch                |  37 +
>>   .../cargo/cargo-cross-canadian.inc                 |  74 ++
>>   .../cargo/cargo-cross-canadian_1.51.0.bb           |   6 +
>>   .../cargo/cargo-cross-canadian_1.54.0.bb           |   6 +
>>   meta/recipes-devtools/cargo/cargo.inc              |  50 ++
>>   meta/recipes-devtools/cargo/cargo_1.51.0.bb        |   8 +
>>   meta/recipes-devtools/cargo/cargo_1.54.0.bb        |   4 +
>>   meta/recipes-devtools/rust/files/riscv-march.patch |  73 ++
>>   meta/recipes-devtools/rust/files/rv64gc.patch      |  37 +
>>   ...001-Add-base-definitions-for-riscv64-musl.patch | 904 
>> ++++++++++++++++++++
>>   ...-linux-musl-mod.rs-add-riscv64-to-b64-set.patch |  29 +
>>   ...3-FIXUP-Correct-definitions-to-match-musl.patch | 740 
>> +++++++++++++++++
>>   ...pdate-1.51.0-checksums-for-modified-files.patch |  21 +
>>   .../0004-Update-checksums-for-modified-files.patch |  21 +
>>   ...-definitions-for-riscv64-musl-libc-0.2.93.patch | 905 
>> +++++++++++++++++++++
>>   ...-musl-mod.rs-add-riscv64-to-b64-set-libc-.patch |  31 +
>>   ...ect-definitions-to-match-musl-libc-0.2.93.patch | 741 
>> +++++++++++++++++
>>   ...ksums-for-modified-files-for-rust-1.54.0-.patch |  23 +
>>   meta/recipes-devtools/rust/libstd-rs.inc           |  40 +
>>   meta/recipes-devtools/rust/libstd-rs_1.51.0.bb     |  12 +
>>   meta/recipes-devtools/rust/libstd-rs_1.54.0.bb     |  11 +
>>   meta/recipes-devtools/rust/rust-common.inc         | 346 ++++++++
>>   .../rust/rust-cross-canadian-common.inc            |  53 ++
>>   meta/recipes-devtools/rust/rust-cross-canadian.inc |  78 ++
>>   .../rust/rust-cross-canadian_1.51.0.bb             |   6 +
>>   .../rust/rust-cross-canadian_1.54.0.bb             |   6 +
>>   meta/recipes-devtools/rust/rust-cross.inc          |  71 ++
>>   meta/recipes-devtools/rust/rust-cross_1.51.0.bb    |   2 +
>>   meta/recipes-devtools/rust/rust-cross_1.54.0.bb    |   2 +
>>   meta/recipes-devtools/rust/rust-llvm.inc           |  63 ++
>>   .../rust-llvm/0001-nfc-Fix-missing-include.patch   |  26 +
>>   .../0002-llvm-allow-env-override-of-exe-path.patch |  32 +
>>   meta/recipes-devtools/rust/rust-llvm_1.51.0.bb     |   7 +
>>   meta/recipes-devtools/rust/rust-llvm_1.54.0.bb     |   5 +
>>   .../recipes-devtools/rust/rust-snapshot-1.51.0.inc |  18 +
>>   .../recipes-devtools/rust/rust-snapshot-1.54.0.inc |  18 +
>>   meta/recipes-devtools/rust/rust-snapshot.inc       |   9 +
>>   meta/recipes-devtools/rust/rust-source-1.51.0.inc  |   3 +
>>   meta/recipes-devtools/rust/rust-source-1.54.0.inc  |   3 +
>>   meta/recipes-devtools/rust/rust-source.inc         |   3 +
>>   meta/recipes-devtools/rust/rust-target.inc         |  10 +
>>   .../rust/rust-tools-cross-canadian.inc             |  38 +
>>   .../rust/rust-tools-cross-canadian_1.51.0.bb       |   6 +
>>   .../rust/rust-tools-cross-canadian_1.54.0.bb       |   6 +
>>   meta/recipes-devtools/rust/rust.inc                | 193 +++++
>>   ...t-Fix-dash-vs-underscore-mismatches-in-op.patch |  75 ++
>>   meta/recipes-devtools/rust/rust_1.51.0.bb          |  16 +
>>   meta/recipes-devtools/rust/rust_1.54.0.bb          |  11 +
>>   .../rust-hello-world/0001-enable-LTO.patch         |  23 +
>>   .../rust-hello-world/rust-hello-world_git.bb       |  17 +
>>   meta/recipes-example/rustfmt/rustfmt_1.4.2.bb      | 171 ++++
>>   63 files changed, 5977 insertions(+)
>>
>>
>>
>
>

-- 
# Randy MacLeod
# Wind River Linux