Re: [OE-core] [PATCH V4] meta: rust - Bug fix for target definitions returning 'NoneType' for arm

["Sundeep KOKKONDA" <sundeep.kokkonda@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 9190 bytes --]

Hello,
On 14-05-2022 12:46, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2022-05-12 at 22:59 -0700, Sundeep KOKKONDA wrote:
>> [Yocto bug #14742]
>> The build shows below error while building for arm machines.
>> Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
>> Detailed error info :
>>
>> Steps to reproduce:
>> 1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"'
>> 2. bitbake core-image-minimal -cpopulate_sdk
>>
>> Complete Error:
>> ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated:
>> The stack trace of python calls that resulted in this exception/failure was:
>> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>>       0001:
>>   *** 0002:do_rust_gen_targets(d)
>>       0003:
>> File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets
>>       0027:
>>       0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}"
>>       0029:python do_rust_gen_targets () {
>>       0030:    wd = d.getVar('WORKDIR') + '/targets/'
>>   *** 0031:    rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH'))
>>       0032:    rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH'))
>>       0033:    rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH'))
>>       0034:}
>>       0035:
>> File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target
>>       0326:    # build tspec
>>       0327:    tspec = {}
>>       0328:    tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi)
>>       0329:    tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi)
>>   *** 0330:    tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi))
>>       0331:    tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi)
>>       0332:    tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi)
>>       0333:    tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi)
>>       0334:    tspec['arch'] = arch_to_rust_target_arch(rust_arch)
>> Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
>>
>> Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType.
>> (a) Locals at rust_gen_target for arm::
>> tspec['data-layout'] =  None, Type of tspec['data-layout'] =  <class 'NoneType'>
>> tspec['data-layout'] =  None, Type of tspec['data-layout'] =  <class 'NoneType'>
>> DEBUG: Python function do_rust_gen_targets finished
>> (b) Locals at rust_gen_target  for aarch64::
>> tspec['data-layout'] =  aarch64-unknown-linux-gnu, Type of tspec['data-layout'] =  <class 'str'>
>> tspec['max-atomic-width'] =  128, Type of tspec['max-atomic-width'] =  <class 'int'>
>>
>> Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue.
>>
>> The 'rust_gen_targets' Task added with its dependent variable list.
>>
>> Signed-off-by: Sundeep KOKKONDA<sundeep.kokkonda@gmail.com>
>> ---
>>   meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++-------
>>   1 file changed, 9 insertions(+), 7 deletions(-)
>>
>> diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc
>> index 310aecef22..984fe9099e 100644
>> --- a/meta/recipes-devtools/rust/rust-common.inc
>> +++ b/meta/recipes-devtools/rust/rust-common.inc
>> @@ -119,13 +119,13 @@ def llvm_features(d):
>>   
>>   
>>   ## arm-unknown-linux-gnueabihf
>> -DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
>> -LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}"
>> -TARGET_ENDIAN[arm-eabi] = "little"
>> -TARGET_POINTER_WIDTH[arm-eabi] = "32"
>> -TARGET_C_INT_WIDTH[arm-eabi] = "32"
>> -MAX_ATOMIC_WIDTH[arm-eabi] = "64"
>> -FEATURES[arm-eabi] = "+v6,+vfp2"
>> +DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
>> +LLVM_TARGET[arm] = "${RUST_TARGET_SYS}"
>> +TARGET_ENDIAN[arm] = "little"
>> +TARGET_POINTER_WIDTH[arm] = "32"
>> +TARGET_C_INT_WIDTH[arm] = "32"
>> +MAX_ATOMIC_WIDTH[arm] = "64"
>> +FEATURES[arm] = "+v6,+vfp2"
>>   
>>   ## armv7-unknown-linux-gnueabihf
>>   DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
>> @@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""):
>>       with open(wd + sys + '.json', 'w') as f:
>>           json.dump(tspec, f, indent=4)
>>   
>> +do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES"
>> +
>>   python do_rust_gen_targets () {
>>       wd = d.getVar('WORKDIR') + '/targets/'
>>       build_arch = d.getVar('BUILD_ARCH')
> We need to split this into two. One patch to fix the determinism issue,
> the other to fix the arm bug.
>
> The "good" news is that this patch now fails consistently on the
> autobuilder, for example:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3522
>
> sstatetests.SStateTests.test_sstate_allarch_samesigs is failing which
> you can rerun with:
>
> oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs
>
> Now it fails reliably, we need to work out why the arm change breaks
> that.
>
> If you split the patches into two we can likely merge the
> reproducibility one, then onwards with the arm issue.
>
> In case it isn't clear, the failing signatures are for adwaita-icon-
> theme which depends on librsvg-native which depends on rust-native.
> What this means is that the signature of rust-native is changing when
> MACHINE is changed in that test. This shouldn't happen. We already knew
> that from our other discussion but now we can prove it. It would be
> nice if a test showed us the native hash was changing and that is the
> other issue we discussed but one step at a time!
>
> To be clear, rust-native (and it's signature) should be independent of
> MACHINE so we need to fix that. This means that currently rust-native
> varies and rebuilds when you change MACHINE (after the dependency fix
> in this patch).

Hello Richard,

I analyzed the test failure and it is because of the newly added 
dependency on variable LLVM_TARGET. In think the patch split will not 
help here. We need to understand the reason for the introduced change in 
first place from 'arm' to 'arm-eabi' & why the rust_gen_target is not 
added dependency on these variables.

Hello Alex,

We need a few inputs from you regarding a 'arm' bug which is in 
discussion in this patch.

The changes in file /meta/recipes-devtools/rust/rust-common.inc/ with 
commit ID '/d6b563710e6cc0857843433d85023d47f9f2037d/' are causing the 
build error with 'arm' architecture (the /tspec /variables returning 
'/NoneType/') when  '/TOOLCHAIN_HOST_TASK:append = " 
packagegroup-rust-cross-canadian-${MACHINE}/" added in /local.conf/ (See 
the detailed build error at beginning of this email).

And to fix that error the changes for '/arm/' architecture are reverted 
(#arm-unknown-linux-gnueabihf variables reverted to '/arm/' from 
'/arm-eabi/') and the build error is solved. But, because no Task is 
depending on these variables, /bitbake /is unable to detect these 
changes while rebuilding (In other words, the Sstate-cache sigdata is 
not changed). So, the rust_gen_target task dependency to these variables 
are added and then the Sstate sigdata is updated with rebuild. However, 
the dependency (on variable LLVM_TARGET) is causing the failure in 
/oe-selftest test_sstate_allarch_samesigs/ testcase.

Can you let us know is it the right thing to add these variable 
dependencies for rust_gen_target task? And, to fix the 'arm NoneType' 
error is it the correct fix to revert the '/arm-eabi/' to '/arm/'?

Let me know if anything is not clear.

Below is the dependency list taken from /bitbake-diffsigs/ from the 
/test_sstate_allarch_samesigs/ test sigdata for x86_64 & qemuarm 
machines. The change in sigdata caused by differed values for variable 
TUNE_FEATURES{callconvention-hard}.
=================
...
List of dependencies for variable LLVM_TARGET[aarch64] is 
['RUST_TARGET_SYS']
List of dependencies for variable LLVM_TARGET[arm] is ['RUST_TARGET_SYS']
...
List of dependencies for variable RUST_TARGET_SYS is ['rust_base_triple']
...
List of dependencies for variable rust_base_triple is ['determine_libc', 
'target_is_armv7']
...
List of dependencies for variable determine_libc is ['BUILD_SYS', 
'HOST_SYS', 'RUST_LIBC', 'RUST_LIBC:class-native']    # The 
determine_libc is using the TUNE_FEATURES
List of dependencies for variable target_is_armv7 is []
...
TUNE_FEATURES{callconvention-hard} = *Unset *#in x86_64
TUNE_FEATURES{callconvention-hard} = *Set *#in qemuarm
...
====================


Thanks,
Sundeep K.


>
> Cheers,
>
> Richard
>

[-- Attachment #2: Type: text/html, Size: 10556 bytes --]