Re: [OE-core] [PATCH V4] meta: rust - Bug fix for target definitions returning 'NoneType' for arm

[richard.purdie@linuxfoundation.org]

On Thu, 2022-05-12 at 22:59 -0700, Sundeep KOKKONDA wrote:
> [Yocto bug #14742]
> The build shows below error while building for arm machines.
> Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
> Detailed error info :
> 
> Steps to reproduce:
> 1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"'
> 2. bitbake core-image-minimal -cpopulate_sdk
> 
> Complete Error:
> ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated:
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:do_rust_gen_targets(d)
>      0003:
> File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets
>      0027:
>      0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}"
>      0029:python do_rust_gen_targets () {
>      0030:    wd = d.getVar('WORKDIR') + '/targets/'
>  *** 0031:    rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH'))
>      0032:    rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH'))
>      0033:    rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH'))
>      0034:}
>      0035:
> File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target
>      0326:    # build tspec
>      0327:    tspec = {}
>      0328:    tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi)
>      0329:    tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi)
>  *** 0330:    tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi))
>      0331:    tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi)
>      0332:    tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi)
>      0333:    tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi)
>      0334:    tspec['arch'] = arch_to_rust_target_arch(rust_arch)
> Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
> 
> Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType.
> (a) Locals at rust_gen_target for arm::
> tspec['data-layout'] =  None, Type of tspec['data-layout'] =  <class 'NoneType'>
> tspec['data-layout'] =  None, Type of tspec['data-layout'] =  <class 'NoneType'>
> DEBUG: Python function do_rust_gen_targets finished
> (b) Locals at rust_gen_target  for aarch64::
> tspec['data-layout'] =  aarch64-unknown-linux-gnu, Type of tspec['data-layout'] =  <class 'str'>
> tspec['max-atomic-width'] =  128, Type of tspec['max-atomic-width'] =  <class 'int'>
> 
> Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue.
> 
> The 'rust_gen_targets' Task added with its dependent variable list.
> 
> Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@gmail.com>
> ---
>  meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++-------
>  1 file changed, 9 insertions(+), 7 deletions(-)
> 
> diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc
> index 310aecef22..984fe9099e 100644
> --- a/meta/recipes-devtools/rust/rust-common.inc
> +++ b/meta/recipes-devtools/rust/rust-common.inc
> @@ -119,13 +119,13 @@ def llvm_features(d):
>  
>  
>  ## arm-unknown-linux-gnueabihf
> -DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
> -LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}"
> -TARGET_ENDIAN[arm-eabi] = "little"
> -TARGET_POINTER_WIDTH[arm-eabi] = "32"
> -TARGET_C_INT_WIDTH[arm-eabi] = "32"
> -MAX_ATOMIC_WIDTH[arm-eabi] = "64"
> -FEATURES[arm-eabi] = "+v6,+vfp2"
> +DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
> +LLVM_TARGET[arm] = "${RUST_TARGET_SYS}"
> +TARGET_ENDIAN[arm] = "little"
> +TARGET_POINTER_WIDTH[arm] = "32"
> +TARGET_C_INT_WIDTH[arm] = "32"
> +MAX_ATOMIC_WIDTH[arm] = "64"
> +FEATURES[arm] = "+v6,+vfp2"
>  
>  ## armv7-unknown-linux-gnueabihf
>  DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
> @@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""):
>      with open(wd + sys + '.json', 'w') as f:
>          json.dump(tspec, f, indent=4)
>  
> +do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES"
> +
>  python do_rust_gen_targets () {
>      wd = d.getVar('WORKDIR') + '/targets/'
>      build_arch = d.getVar('BUILD_ARCH')

We need to split this into two. One patch to fix the determinism issue,
the other to fix the arm bug.

The "good" news is that this patch now fails consistently on the
autobuilder, for example:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3522

sstatetests.SStateTests.test_sstate_allarch_samesigs is failing which
you can rerun with:

oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs

Now it fails reliably, we need to work out why the arm change breaks
that.

If you split the patches into two we can likely merge the
reproducibility one, then onwards with the arm issue.

In case it isn't clear, the failing signatures are for adwaita-icon-
theme which depends on librsvg-native which depends on rust-native.
What this means is that the signature of rust-native is changing when
MACHINE is changed in that test. This shouldn't happen. We already knew
that from our other discussion but now we can prove it. It would be
nice if a test showed us the native hash was changing and that is the
other issue we discussed but one step at a time!

To be clear, rust-native (and it's signature) should be independent of
MACHINE so we need to fix that. This means that currently rust-native
varies and rebuilds when you change MACHINE (after the dependency fix
in this patch).

Cheers,

Richard