[PATCH 00/14] selftest improvements to respect IMAGE_LINK_NAME

[Martin Jansa <martin.jansa@gmail.com>]

These changes are preparation for the actual changes from [YOCTO #12937],
but without changing anything in current artifacts naming schema.

Only change outside selftest was moving of INITRAMFS_IMAGE_NAME variable
and adding IMAGE_MACHINE_SUFFIX variable to make it easier to change both.

I've run complete selftest with my other changes for [YOCTO #12937] which
highlighted where the issues are and then again with only these changes
to make sure I didn't break it for current mickledore.

The most dangerous is probably:
git-submodule-test: disable upstream version check
because I cannot explain why distrodata.Distrodata.test_checkpkg isn't failing on AB

but maybe it's just because many (maybe all AB runs disable this check?), e.g.
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4948/steps/14/logs/stdio
shows it skipped:
ERROR: Command . ./oe-init-build-env; ${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=/srv/autobuilder/autobuilder.yocto.io/pub/repro-fail/ DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15 failed with exit code 1, see errors above. (1678628531.8: 4253.0)

There were some tests which were skipped (even without --skip-tests) or failed in my oe-selftest -a run:

== SKIPPED ==
A) INFO -  ... skipped 'Test needs meta-oe for nodejs'
  due to testing without meta-oe
  2023-03-12 12:08:44,526 - oe-selftest - INFO - RESULTS - devtool.DevtoolAddTests.test_devtool_add_npm: SKIPPED (2.22s)
  2023-03-12 12:08:44,527 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolCreateTests.test_recipetool_create_npm: SKIPPED (2.83s)

B) INFO -  ... skipped 'No "dash" found on test system.'
  no dash on gentoo
  2023-03-12 12:08:44,527 - oe-selftest - INFO - RESULTS - oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash: SKIPPED (0.00s)

C) INFO -  ... skipped 'Target does not use libssp'
  not testing on mingw32 with libssp:
  2023-03-12 12:08:44,527 - oe-selftest - INFO - RESULTS - gcc.GccLibSspSelfTest.test_libssp: SKIPPED (3.47s)
  2023-03-12 12:08:44,527 - oe-selftest - INFO - RESULTS - gcc.GccLibSspSelfTestSystemEmulated.test_libssp: SKIPPED (4.23s)

D) INFO -  ... skipped 'You must set up tap devices with scripts/runqemu-gen-tapdevs before running this test'
  even when the tapdevs were correctly configured with runqemu-gen-tapdevs before oe-selftest was started
  later resolved by "touch /etc/runqemu-nosudo" and it PASSED,
  https://git.openembedded.org/openembedded-core/commit/?id=2258345e19efff7717fe19a5026ec55f1b6f90b6
  looks suspicious and will try to improve it (to also ignore
  skipped taps like /tmp/qemu-tap-locks/tap0.skip which I'm using
  because tap0 is already used by openvpn running on my builder)
  2023-03-12 12:08:44,526 - oe-selftest - INFO - RESULTS - devtool.DevtoolExtractTests.test_devtool_deploy_target: SKIPPED (2.33s)

== FAILED ==
1) 2023-03-12 12:08:44,527 - oe-selftest - INFO - RESULTS - distrodata.Distrodata.test_checkpkg: FAILED (146.68s)

  2023-03-13 08:57:10,817 - oe-selftest - INFO - Traceback (most recent call last):
    File "/OE/build/poky/meta/lib/oeqa/selftest/cases/distrodata.py", line 40, in test_checkpkg
      self.assertTrue(len(regressed_failures) == 0 and len(regressed_successes) == 0, msg)
  AssertionError: False is not true :
  The following packages failed upstream version checks. Please fix them using UPSTREAM_CHECK_URI/UPSTREAM_CHECK_REGEX
  (when using tarballs) or UPSTREAM_CHECK_GITTAGREGEX (when using git). If an upstream version check cannot be performed
  (for example, if upstream does not use git tags), you can set UPSTREAM_VERSION_UNKNOWN to '1' in the recipe to acknowledge
  that the check cannot be performed.
  binutils

  used to show git-submodule-test as well (without the first patch here)
  I have no explanation why this isn't triggered on AB, when I've added
  more debug in this text, the binutils disappeared as well (so I was
  assuming it was just temporary network glitch, but today it returned
  again - with debug already removed).

2) 2023-03-12 12:08:44,527 - oe-selftest - INFO - RESULTS - fetch.Fetch.test_git_mirrors: FAILED (24.34s)

  This is a limitation of current selftest as discussed with RP on IRC.

  easy to explain, but more difficult to fix, it sets (PRE)MIRRORS:forcevariable = ""
  and expects to trigger fetch failure from dbus-wait, but PREMIRRORS:prepend in my 
  local.conf defeats it and dbus-wait is fetched successfully.

  I can switch from prepend to =+ in my local.conf but that will solve it only in my environment.

  RP suggested to change selftest to ignore user's local.conf
  or pick only some specific key things from it, but current
  state is what it is and after removing PREMIRRORS:prepend
  this test PASSED as well.

3) 2023-03-12 12:08:44,527 - oe-selftest - INFO - RESULTS - incompatible_lic.NoGPL3InImagesTests.test_core_image_full_cmdline_weston: FAILED (2043.13s)

  2023-03-13 09:01:25,976 - oe-selftest - INFO -  ... FAIL
  2023-03-13 09:01:25,977 - oe-selftest - INFO - Traceback (most recent call last):
    File "/OE/build/poky/meta/lib/oeqa/selftest/cases/incompatible_lic.py", line 161, in test_core_image_full_cmdline_weston
      bitbake('-c testimage core-image-full-cmdline core-image-weston')
    File "/OE/build/poky/meta/lib/oeqa/utils/commands.py", line 236, in bitbake
      return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/OE/build/poky/meta/lib/oeqa/utils/commands.py", line 214, in runCmd
      raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
  AssertionError: Command 'bitbake  -c testimage core-image-full-cmdline core-image-weston' returned non-zero exit status 1:

  seems to run many tests succesfully with the test image:

  core-image-weston () - Ran 71 tests in 34.038s
  core-image-weston - OK - All required tests passed (successes=14, skipped=57, failures=0, errors=0)
  NOTE: recipe core-image-weston-1.0-r0: task do_testimage: Succeeded
  NOTE: Tasks Summary: Attempted 1141 tasks of which 1139 didn't need to be rerun and 1 failed.

  Summary: 1 task failed:
    /OE/build/poky/meta/recipes-extended/images/core-image-full-cmdline.bb:do_testimage
  Summary: There were 2 WARNING messages.
  Summary: There were 3 ERROR messages, returning a non-zero exit code.

  but there is also:
  runqemu - INFO - Found /tmp/qemu-tap-locks/tap0.skip, skipping tap0
  runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap1.lock failed: [Errno 11] Resource temporarily unavailable

  I need to debug it a bit more, probably outside selftest as well, will try
  to run it when I don't need tap0 for openvpn.
  But it fails the same with current master without any of my changes as well.

4) runqemu.RunqemuTests still fail with my other [YOCTO #12937] changes
  but work fine with this pull request, I have some WIP changes locally
  but the testimage+runqemu interaction is far from simple and will need
  more time to debug to cover all possible use-cases for runqemu

runqemu.RunqemuTests.test_boot_machine_ext4.log:2023-03-13 09:03:12,008 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_machine_ext4: FAILED (63.37s)
runqemu.RunqemuTests.test_boot_recipe_image.log:2023-03-13 08:31:06,400 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_recipe_image: ERROR (47.56s)
runqemu.RunqemuTests.test_boot_recipe_image_vdi.log:2023-03-13 08:32:06,489 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_recipe_image_vdi: ERROR (46.82s)
runqemu.RunqemuTests.test_boot_recipe_image_vmdk.log:2023-03-13 08:33:07,020 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_recipe_image_vmdk: ERROR (47.30s)

The following changes since commit 65ca2c4b7349a4f7dcfcc580d926a99c673deb60:

  dhcpcd: Fix install conflict when enable multilib. (2023-03-11 00:08:38 +0000)

are available in the Git repository at:

  https://git.openembedded.org/openembedded-core-contrib jansa/pull
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=jansa/pull

Martin Jansa (14):
  git-submodule-test: disable upstream version check
  selftest: devtool: set BB_HASHSERVE_UPSTREAM when setting
    SSTATE_MIRROR
  selftest: wic: respect IMAGE_LINK_NAME
  selftest: wic: respect IMAGE_LINK_NAME also in
    test_rawcopy_plugin_qemu
  selftest: runqemu: respect IMAGE_LINK_NAME
  selftest: multiconfig-image-packager: try to respect IMAGE_LINK_NAME
  image-artifact-names.bbclass: add INITRAMFS_IMAGE_NAME from
    kernel.bbclass
  selftest: fitimage.py: respect INITRAMFS_IMAGE_NAME and
    KERNEL_FIT_LINK_NAME
  image-artifact-names: add IMAGE_MACHINE_SUFFIX variable
  selftest: gdbserver.py: respect IMAGE_LINK_NAME
  selftest: minidebuginfo.py respect IMAGE_LINK_NAME
  selftest: efibootpartition.py: fix QEMU_USE_KVM usage
  runqemu: get_first_file() rename cmd* to glob*
  selftest: imagefeatures.py: respect IMAGE_LINK_NAME for debugfs and
    manifest as well

 .../git-submodule-test/git-submodule-test.bb  |  2 +
 .../multiconfig-image-packager_0.1.bb         | 18 +++-
 meta-selftest/wic/test_rawcopy_plugin.wks.in  |  2 +-
 .../image-artifact-names.bbclass              | 16 +++-
 .../kernel-artifact-names.bbclass             |  2 +-
 meta/classes-recipe/kernel.bbclass            |  1 -
 meta/lib/oeqa/selftest/cases/devtool.py       |  1 +
 .../oeqa/selftest/cases/efibootpartition.py   |  4 +-
 meta/lib/oeqa/selftest/cases/fitimage.py      | 34 ++++----
 meta/lib/oeqa/selftest/cases/gdbserver.py     |  9 +-
 meta/lib/oeqa/selftest/cases/imagefeatures.py | 82 +++++++++----------
 meta/lib/oeqa/selftest/cases/minidebuginfo.py | 15 ++--
 meta/lib/oeqa/selftest/cases/runqemu.py       | 10 ++-
 meta/lib/oeqa/selftest/cases/wic.py           | 44 +++++-----
 scripts/runqemu                               | 36 ++++----
 15 files changed, 155 insertions(+), 121 deletions(-)

-- 
2.39.2