[PATCH 1/3] populate_sdk_ext.bbclass: set METADATA_REVISION with an DISTRO override

[PATCH 1/3] populate_sdk_ext.bbclass: set METADATA_REVISION with an DISTRO override

[Martin Jansa]

* otherwise it ends '<unknown>' inside esdk, because of parsing order:
  # $METADATA_REVISION [3 operations]
  #   set /OE/build/test-D/conf/local.conf:43
  #     "f2da54ef432eac89b0f18eaad68e602b6990b5de"
  #   immediate /OE/build/test-D/layers/poky/meta/classes/metadata_scm.bbclass:9
  #     "${@oe.buildcfg.detect_revision(d)}"
  #   set /OE/build/test-D/layers/poky/meta/classes/metadata_scm.bbclass:10
  #     [vardepvalue] "${METADATA_REVISION}"
  # pre-expansion value:
  #   "<unknown>"
  METADATA_REVISION="<unknown>"

* This causes base-files.do_install and following tasks to have different
  signatures between esdk and the build directory where this esdk was created:

  bitbake-diffsigs {test-D,poky/build-uninative-disabled}/tmp/stamps/qemux86_64-poky-linux/base-files/*do_install*sigdata*
  NOTE: Starting bitbake server...
  basehash changed from 5b6981cf58bfd57d416b0e31611b73a26baae635dd1ac31c08d46f95064c3ffc to dbdce042da4d7813d632b6d1cc87a16f728ad20e55fecbc392830e6acf72babd
  Variable METADATA_REVISION value changed from '<unknown>' to 'f2da54ef432eac89b0f18eaad68e602b6990b5de'

  and an warning from "python3 /OE/build/test-D/ext-sdk-prepare.py" when eSDK is being prepared for use:
  WARNING: The base-files:do_install sig is computed to be 83b9c9a6ef1145baac5a1e0d08814b9156af239c58fc42df95c25a9cd8a7f201,
    but the sig is locked to 3dc22233059075978e5503691e98e79e7cc60db94259dfcd886bca2291c0add7 in SIGGEN_LOCKEDSIGS_t-qemux86-64

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes-recipe/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
index f619fc9011..7f55c83ee2 100644
--- a/meta/classes-recipe/populate_sdk_ext.bbclass
+++ b/meta/classes-recipe/populate_sdk_ext.bbclass
@@ -376,7 +376,7 @@ python copy_buildsystem () {
             f.write('BUILDCFG_HEADER = ""\n\n')
 
             # Write METADATA_REVISION
-            f.write('METADATA_REVISION = "%s"\n\n' % d.getVar('METADATA_REVISION'))
+            f.write('METADATA_REVISION:%s = "%s"\n\n' % (d.getVar('DISTRO'), d.getVar('METADATA_REVISION')))
 
             f.write('# Provide a flag to indicate we are in the EXT_SDK Context\n')
             f.write('WITHIN_EXT_SDK = "1"\n\n')
-- 
2.40.0

[PATCH 2/3] populate_sdk_ext.bbclass: redirect stderr to stdout so that both end in LOGFILE

[Martin Jansa]

* this in the end doesn't help much, I was debugging warning (about base-files.do_install
  signature being different than expected) from:

  python3 $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'

  this shows the warning on console, but it doesn't end in $LOGFILE, because it
  writes only contents of cooker log into the $LOGFILE with:

    with open(logfile, 'a') as logf:
        logf.write('Preparing SDK for %s...\n' % ', '.join(sdk_targets))

        ret = run_command_interruptible('BB_SETSCENE_ENFORCE=1 bitbake --quiet %s' % ' '.join(sdk_targets))
        if not ret:
            ret = run_command_interruptible('bitbake --quiet build-sysroots')
        lastlog = get_last_consolelog()
        if lastlog:
            with open(lastlog, 'r') as f:
                for line in f:
                    logf.write(line)
        if ret:
            print('ERROR: SDK preparation failed: error log written to %s' % logfile)
            return ret

  maybe we could remove whole support for $LOGFILE parameter and just redirect
  the output like other commands on this line

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes-recipe/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
index 7f55c83ee2..8146036fa9 100644
--- a/meta/classes-recipe/populate_sdk_ext.bbclass
+++ b/meta/classes-recipe/populate_sdk_ext.bbclass
@@ -739,7 +739,7 @@ sdk_ext_postinst() {
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python3 $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+		sh -c ". buildtools/environment-setup* > $LOGFILE 2>&1 && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE 2>&1 && python3 $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
 	fi
 	if [ -e $target_sdk_dir/ext-sdk-prepare.py ]; then
 		rm $target_sdk_dir/ext-sdk-prepare.py
-- 
2.40.0

[PATCH 3/3] oeqa: print stderr in the AssertionError as well

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/lib/oeqa/utils/commands.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index c1f533802e..7b2f6ec124 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -209,7 +209,7 @@ def runCmd(command, ignore_status=False, timeout=None, assert_error=True, sync=T
                 exc_output = "\n... (last %d lines of output)\n" % limit_exc_output + \
                              '\n'.join(split[-limit_exc_output:])
         if assert_error:
-            raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
+            raise AssertionError("Command '%s' returned non-zero exit status %d:\nstderr: %s\nstdout:\n%s" % (command, result.status, result.error, exc_output))
         else:
             raise CommandError(result.status, command, exc_output)
 
-- 
2.40.0

Re: [OE-core] [PATCH 3/3] oeqa: print stderr in the AssertionError as well

[Luca Ceresoli]

Hello Martin,

On Tue, 11 Apr 2023 11:03:48 +0200
"Martin Jansa" <Martin.Jansa@gmail.com> wrote:

> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/lib/oeqa/utils/commands.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
> index c1f533802e..7b2f6ec124 100644
> --- a/meta/lib/oeqa/utils/commands.py
> +++ b/meta/lib/oeqa/utils/commands.py
> @@ -209,7 +209,7 @@ def runCmd(command, ignore_status=False, timeout=None, assert_error=True, sync=T
>                  exc_output = "\n... (last %d lines of output)\n" % limit_exc_output + \
>                               '\n'.join(split[-limit_exc_output:])
>          if assert_error:
> -            raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
> +            raise AssertionError("Command '%s' returned non-zero exit status %d:\nstderr: %s\nstdout:\n%s" % (command, result.status, result.error, exc_output))

Testing with this series produces errors on the selftest builds:

---------------------------------8<---------------------------------

AssertionError: Command 'echo foobar >&2; false' returned non-zero exit status 1:
stderr: 
stdout:
foobar

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/runcmd.py", line 61, in test_result_assertion
    self.assertRaisesRegexp(AssertionError, "Command 'echo .* false' returned non-zero exit status 1:\nfoobar",
  File "/usr/lib/python3.9/unittest/case.py", line 1329, in deprecated_func
    return original_func(*args, **kwargs)
  File "/usr/lib/python3.9/unittest/case.py", line 1276, in assertRaisesRegex
    return context.handle('assertRaisesRegex', args, kwargs)
  File "/usr/lib/python3.9/unittest/case.py", line 201, in handle
    callable_obj(*args, **kwargs)
  File "/usr/lib/python3.9/unittest/case.py", line 239, in __exit__
    self._raiseFailure('"{}" does not match "{}"'.format(
  File "/usr/lib/python3.9/unittest/case.py", line 163, in _raiseFailure
    raise self.test_case.failureException(msg)
AssertionError: "Command 'echo .* false' returned non-zero exit status 1:
foobar" does not match "Command 'echo foobar >&2; false' returned non-zero exit status 1:
stderr: 
stdout:
foobar"

---------------------------------8<---------------------------------

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5002/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5044/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5053/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5083/steps/15/logs/stdio


-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH 3/3] oeqa: print stderr in the AssertionError as well

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4599 bytes --]

Hello Luca,

I'm sorry, I've updated the runcmd.RunCmdTests.test_result_assertion test
case to pass with this change just to realize that:
"stderr": subprocess.STDOUT
in self.defaultopts does work and even the stderr output is ending in
stdout, so this change isn't very useful (as I don't see any runcmd users
passing different options than defaultopts).

Please drop 3/3 other 2 changes are independent on it.

I have one more change in this area which is the real fix for esdk tests
with uninative disabled, but that has some weird side-effect which is
triggering:

NOTE: Running task 3807 of 3809
(/OE/build/poky/meta/recipes-core/images/core-image-minimal.bb:
do_populate_lic_deploy)
ERROR: When reparsing
/OE/build/poky/meta/recipes-core/images/core-image-minimal.bb:do_populate_sdk_ext,
the basehash value changed from
8249821d6529e111a2f7f1101632fe140639df54a700e2c604a99ea150541a6d to
86086c51c409799a0d4d2a4fcac9e7e245cb36d86f236dcf7093e428a7c37ac9. The
metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake core-image-minimal -cdo_populate_sdk_ext -Snone
ERROR: Then:
ERROR: $ bitbake core-image-minimal -cdo_populate_sdk_ext -Sprintdiff

which I haven't resolved yet, so it might be a while before I send this 4th
change.

Regards,

On Tue, Apr 11, 2023 at 2:28 PM Luca Ceresoli <luca.ceresoli@bootlin.com>
wrote:

> Hello Martin,
>
> On Tue, 11 Apr 2023 11:03:48 +0200
> "Martin Jansa" <Martin.Jansa@gmail.com> wrote:
>
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >  meta/lib/oeqa/utils/commands.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/lib/oeqa/utils/commands.py
> b/meta/lib/oeqa/utils/commands.py
> > index c1f533802e..7b2f6ec124 100644
> > --- a/meta/lib/oeqa/utils/commands.py
> > +++ b/meta/lib/oeqa/utils/commands.py
> > @@ -209,7 +209,7 @@ def runCmd(command, ignore_status=False,
> timeout=None, assert_error=True, sync=T
> >                  exc_output = "\n... (last %d lines of output)\n" %
> limit_exc_output + \
> >                               '\n'.join(split[-limit_exc_output:])
> >          if assert_error:
> > -            raise AssertionError("Command '%s' returned non-zero exit
> status %d:\n%s" % (command, result.status, exc_output))
> > +            raise AssertionError("Command '%s' returned non-zero exit
> status %d:\nstderr: %s\nstdout:\n%s" % (command, result.status,
> result.error, exc_output))
>
> Testing with this series produces errors on the selftest builds:
>
> ---------------------------------8<---------------------------------
>
> AssertionError: Command 'echo foobar >&2; false' returned non-zero exit
> status 1:
> stderr:
> stdout:
> foobar
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File
> "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/runcmd.py",
> line 61, in test_result_assertion
>     self.assertRaisesRegexp(AssertionError, "Command 'echo .* false'
> returned non-zero exit status 1:\nfoobar",
>   File "/usr/lib/python3.9/unittest/case.py", line 1329, in deprecated_func
>     return original_func(*args, **kwargs)
>   File "/usr/lib/python3.9/unittest/case.py", line 1276, in
> assertRaisesRegex
>     return context.handle('assertRaisesRegex', args, kwargs)
>   File "/usr/lib/python3.9/unittest/case.py", line 201, in handle
>     callable_obj(*args, **kwargs)
>   File "/usr/lib/python3.9/unittest/case.py", line 239, in __exit__
>     self._raiseFailure('"{}" does not match "{}"'.format(
>   File "/usr/lib/python3.9/unittest/case.py", line 163, in _raiseFailure
>     raise self.test_case.failureException(msg)
> AssertionError: "Command 'echo .* false' returned non-zero exit status 1:
> foobar" does not match "Command 'echo foobar >&2; false' returned non-zero
> exit status 1:
> stderr:
> stdout:
> foobar"
>
> ---------------------------------8<---------------------------------
>
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5002/steps/14/logs/stdio
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5044/steps/14/logs/stdio
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5053/steps/14/logs/stdio
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5083/steps/15/logs/stdio
>
>
> --
> Luca Ceresoli, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>

[-- Attachment #2: Type: text/html, Size: 6256 bytes --]