[PATCH v2 1/9] core-image-minimal: increase extra space to pass df.py oeqa runtime test

[PATCH v2 1/9] core-image-minimal: increase extra space to pass df.py oeqa runtime test

[Mikko Rapeli]

testimage.bbclass requires that ssh communication with target works
and then it runs oeqa runtime tests with that. Tests like df.py
check that there is more than 5Mb free space on the rootfs on target. Sadly
latest core-image-minimal only has 1.5 Mb free space if
ssh-server-dropbear is added to it. Thus by default, core-image-minimal
is now failing oeqa rutime df.py test.

Fix this by increasing core-image-minimal rootfs size by 5Mb if
testimage.bbclass is used which implies adding either
ssh-server-dropbear or ssh-server-openssh to the target rootfs.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/recipes-core/images/core-image-minimal.bb | 3 +++
 1 file changed, 3 insertions(+)

v2: no changes

v1: https://lists.openembedded.org/g/openembedded-core/message/186423

diff --git a/meta/recipes-core/images/core-image-minimal.bb b/meta/recipes-core/images/core-image-minimal.bb
index 84343adcd8..c04612bc15 100644
--- a/meta/recipes-core/images/core-image-minimal.bb
+++ b/meta/recipes-core/images/core-image-minimal.bb
@@ -10,3 +10,6 @@ inherit core-image
 
 IMAGE_ROOTFS_SIZE ?= "8192"
 IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("DISTRO_FEATURES", "systemd", " + 4096", "", d)}"
+
+# a bit more space needed for sshd to actually run the tests
+IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("IMAGE_CLASSES", "testimage", " + 5120", "", d)}"
-- 
2.34.1

[PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Mikko Rapeli]

Also set target and server IP addresses correctly to 127.0.0.1
so that TEST_TARGET_IP and TEST_SERVER_IP don't
need to be set manually with slirp.

Users run qemu images with slirp networking like this:

$ bitbake core-image-minimal
$ ../poky/scripts/runqemu slirp

slirp networking means that only one TCP port is forwarded from host
system to the qemu machine so that ssh into the machine works. ping
and other low level networking from host to target machine do not work,
but networking from target via host does work. This is much more easy to
setup than the bridge networking used by default, which I've never
managed to get running on various host machines.

To use slirp networking wita qemu and core-image-minimal to run testimage.bbclass tests, add
these to local.conf:

IMAGE_CLASSES += "testimage"
IMAGE_FEATURES += "ssh-server-dropbear"
TEST_RUNQEMUPARAMS += "slirp"

Then image can be compiled and tested with:

$ bitbake core-image-minimal
$ bitbake -c testimage core-image-minimal

Note that only few of the current oeqa runtime tests work against
core-image-minimal, but test results look like:

RESULTS:
RESULTS - date.DateTest.test_date: PASSED (3.64s)
RESULTS - df.DfTest.test_df: PASSED (0.55s)
RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s)
RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s)
RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s)
RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s)
RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s)
RESULTS - ping.PingTest.test_ping: PASSED (0.05s)
RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s)
RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s)
RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s)
RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s)
RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s)
RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s)
RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s)
RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s)
RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s)
RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s)
RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s)
RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s)
RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s)
RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s)
RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s)
RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s)
RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s)
RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s)
RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s)
RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s)
RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s)
RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s)
RESULTS - oe_syslog.SyslogTestConfig.test_syslog_startup_config: SKIPPED (0.00s)
RESULTS - opkg.OpkgRepoTest.test_opkg_install_from_repo: SKIPPED (0.00s)
RESULTS - perl.PerlTest.test_perl_works: SKIPPED (0.00s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: SKIPPED (0.00s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
RESULTS - python.PythonTest.test_python3: SKIPPED (0.00s)
RESULTS - rpm.RpmBasicTest.test_rpm_help: SKIPPED (0.00s)
RESULTS - rpm.RpmBasicTest.test_rpm_query: SKIPPED (0.00s)
RESULTS - rpm.RpmBasicTest.test_rpm_query_nonroot: SKIPPED (0.00s)
RESULTS - rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size: SKIPPED (0.00s)
RESULTS - rpm.RpmInstallRemoveTest.test_rpm_install: SKIPPED (0.00s)
RESULTS - rpm.RpmInstallRemoveTest.test_rpm_remove: SKIPPED (0.00s)
RESULTS - rust.RustCompileTest.test_cargo_compile: SKIPPED (0.00s)
RESULTS - rust.RustCompileTest.test_rust_compile: SKIPPED (0.00s)
RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s)
RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s)
RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s)
SUMMARY:
core-image-minimal () - Ran 70 tests in 22.173s
core-image-minimal - OK - All required tests passed (successes=14, skipped=56, failures=0, errors=0)
NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded
NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun and all succeeded.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/classes-recipe/testimage.bbclass | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

v2: set TEST_SERVER_IP correctly to default qemu dhcp server IP 10.0.2.2
so that testimage selftests pass

v1: https://lists.openembedded.org/g/openembedded-core/message/186424

diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass
index e3068348ff..7340996788 100644
--- a/meta/classes-recipe/testimage.bbclass
+++ b/meta/classes-recipe/testimage.bbclass
@@ -322,7 +322,7 @@ def testimage_main(d):
     ovmf = d.getVar("QEMU_USE_OVMF")
 
     slirp = False
-    if d.getVar("QEMU_USE_SLIRP"):
+    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
         slirp = True
 
     # TODO: We use the current implementation of qemu runner because of
@@ -369,10 +369,18 @@ def testimage_main(d):
     # runtime use network for download projects for build
     export_proxies(d)
 
+    if slirp:
+        target_ip = "127.0.0.1"
+        # from qemu target to host with default DHCP server
+        server_ip = "10.0.2.2"
+    else:
+        target_ip = d.getVar("TEST_TARGET_IP")
+        server_ip = d.getVar("TEST_SERVER_IP")
+
     # the robot dance
     target = OERuntimeTestContextExecutor.getTarget(
-        d.getVar("TEST_TARGET"), logger, d.getVar("TEST_TARGET_IP"),
-        d.getVar("TEST_SERVER_IP"), **target_kwargs)
+        d.getVar("TEST_TARGET"), logger, target_ip,
+        server_ip, **target_kwargs)
 
     # test context
     tc = OERuntimeTestContext(td, logger, target, image_packages, extract_dir)
-- 
2.34.1

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Khem Raj]

On Tue, Aug 22, 2023 at 11:10 PM Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
>
> Also set target and server IP addresses correctly to 127.0.0.1
> so that TEST_TARGET_IP and TEST_SERVER_IP don't
> need to be set manually with slirp.
>
> Users run qemu images with slirp networking like this:
>
> $ bitbake core-image-minimal
> $ ../poky/scripts/runqemu slirp
>
> slirp networking means that only one TCP port is forwarded from host
> system to the qemu machine so that ssh into the machine works. ping
> and other low level networking from host to target machine do not work,
> but networking from target via host does work. This is much more easy to
> setup than the bridge networking used by default, which I've never
> managed to get running on various host machines.
>
> To use slirp networking wita qemu and core-image-minimal to run testimage.bbclass tests, add
> these to local.conf:
>
> IMAGE_CLASSES += "testimage"
> IMAGE_FEATURES += "ssh-server-dropbear"
> TEST_RUNQEMUPARAMS += "slirp"
>
> Then image can be compiled and tested with:
>
> $ bitbake core-image-minimal
> $ bitbake -c testimage core-image-minimal
>
> Note that only few of the current oeqa runtime tests work against
> core-image-minimal, but test results look like:
>
> RESULTS:
> RESULTS - date.DateTest.test_date: PASSED (3.64s)
> RESULTS - df.DfTest.test_df: PASSED (0.55s)
> RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s)
> RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s)
> RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s)
> RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s)
> RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s)
> RESULTS - ping.PingTest.test_ping: PASSED (0.05s)
> RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s)
> RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s)
> RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s)
> RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s)
> RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s)
> RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s)
> RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s)
> RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s)
> RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s)
> RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s)
> RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s)
> RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s)
> RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s)
> RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s)
> RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s)
> RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s)
> RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s)
> RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s)
> RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s)
> RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s)
> RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s)
> RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s)
> RESULTS - oe_syslog.SyslogTestConfig.test_syslog_startup_config: SKIPPED (0.00s)
> RESULTS - opkg.OpkgRepoTest.test_opkg_install_from_repo: SKIPPED (0.00s)
> RESULTS - perl.PerlTest.test_perl_works: SKIPPED (0.00s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: SKIPPED (0.00s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> RESULTS - python.PythonTest.test_python3: SKIPPED (0.00s)
> RESULTS - rpm.RpmBasicTest.test_rpm_help: SKIPPED (0.00s)
> RESULTS - rpm.RpmBasicTest.test_rpm_query: SKIPPED (0.00s)
> RESULTS - rpm.RpmBasicTest.test_rpm_query_nonroot: SKIPPED (0.00s)
> RESULTS - rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size: SKIPPED (0.00s)
> RESULTS - rpm.RpmInstallRemoveTest.test_rpm_install: SKIPPED (0.00s)
> RESULTS - rpm.RpmInstallRemoveTest.test_rpm_remove: SKIPPED (0.00s)
> RESULTS - rust.RustCompileTest.test_cargo_compile: SKIPPED (0.00s)
> RESULTS - rust.RustCompileTest.test_rust_compile: SKIPPED (0.00s)
> RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s)
> RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s)
> RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED (0.00s)
> RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED (0.00s)
> RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s)
> RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s)
> RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s)
> RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor: SKIPPED (0.00s)
> RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s)
> RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s)
> RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s)
> SUMMARY:
> core-image-minimal () - Ran 70 tests in 22.173s
> core-image-minimal - OK - All required tests passed (successes=14, skipped=56, failures=0, errors=0)
> NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded
> NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun and all succeeded.
>
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/classes-recipe/testimage.bbclass | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)
>
> v2: set TEST_SERVER_IP correctly to default qemu dhcp server IP 10.0.2.2
> so that testimage selftests pass
>
> v1: https://lists.openembedded.org/g/openembedded-core/message/186424
>
> diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass
> index e3068348ff..7340996788 100644
> --- a/meta/classes-recipe/testimage.bbclass
> +++ b/meta/classes-recipe/testimage.bbclass
> @@ -322,7 +322,7 @@ def testimage_main(d):
>      ovmf = d.getVar("QEMU_USE_OVMF")
>
>      slirp = False
> -    if d.getVar("QEMU_USE_SLIRP"):
> +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
>          slirp = True
>
>      # TODO: We use the current implementation of qemu runner because of
> @@ -369,10 +369,18 @@ def testimage_main(d):
>      # runtime use network for download projects for build
>      export_proxies(d)
>
> +    if slirp:
> +        target_ip = "127.0.0.1"

will this work when running multiple instances of qemu ?
e.g.  try bitbake core-image-ptest-all

> +        # from qemu target to host with default DHCP server
> +        server_ip = "10.0.2.2"
> +    else:
> +        target_ip = d.getVar("TEST_TARGET_IP")
> +        server_ip = d.getVar("TEST_SERVER_IP")
> +
>      # the robot dance
>      target = OERuntimeTestContextExecutor.getTarget(
> -        d.getVar("TEST_TARGET"), logger, d.getVar("TEST_TARGET_IP"),
> -        d.getVar("TEST_SERVER_IP"), **target_kwargs)
> +        d.getVar("TEST_TARGET"), logger, target_ip,
> +        server_ip, **target_kwargs)
>
>      # test context
>      tc = OERuntimeTestContext(td, logger, target, image_packages, extract_dir)
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186547): https://lists.openembedded.org/g/openembedded-core/message/186547
> Mute This Topic: https://lists.openembedded.org/mt/100910036/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Mikko Rapeli]

Hi,

On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> On Tue, Aug 22, 2023 at 11:10 PM Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> >
> > Also set target and server IP addresses correctly to 127.0.0.1
> > so that TEST_TARGET_IP and TEST_SERVER_IP don't
> > need to be set manually with slirp.
> >
> > Users run qemu images with slirp networking like this:
> >
> > $ bitbake core-image-minimal
> > $ ../poky/scripts/runqemu slirp
> >
> > slirp networking means that only one TCP port is forwarded from host
> > system to the qemu machine so that ssh into the machine works. ping
> > and other low level networking from host to target machine do not work,
> > but networking from target via host does work. This is much more easy to
> > setup than the bridge networking used by default, which I've never
> > managed to get running on various host machines.
> >
> > To use slirp networking wita qemu and core-image-minimal to run testimage.bbclass tests, add
> > these to local.conf:
> >
> > IMAGE_CLASSES += "testimage"
> > IMAGE_FEATURES += "ssh-server-dropbear"
> > TEST_RUNQEMUPARAMS += "slirp"
> >
> > Then image can be compiled and tested with:
> >
> > $ bitbake core-image-minimal
> > $ bitbake -c testimage core-image-minimal
> >
> > Note that only few of the current oeqa runtime tests work against
> > core-image-minimal, but test results look like:
> >
> > RESULTS:
> > RESULTS - date.DateTest.test_date: PASSED (3.64s)
> > RESULTS - df.DfTest.test_df: PASSED (0.55s)
> > RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s)
> > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s)
> > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s)
> > RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s)
> > RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s)
> > RESULTS - ping.PingTest.test_ping: PASSED (0.05s)
> > RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s)
> > RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s)
> > RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s)
> > RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s)
> > RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s)
> > RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s)
> > RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s)
> > RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s)
> > RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s)
> > RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s)
> > RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s)
> > RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s)
> > RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s)
> > RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s)
> > RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s)
> > RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s)
> > RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s)
> > RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s)
> > RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s)
> > RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s)
> > RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s)
> > RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s)
> > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_startup_config: SKIPPED (0.00s)
> > RESULTS - opkg.OpkgRepoTest.test_opkg_install_from_repo: SKIPPED (0.00s)
> > RESULTS - perl.PerlTest.test_perl_works: SKIPPED (0.00s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: SKIPPED (0.00s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> > RESULTS - python.PythonTest.test_python3: SKIPPED (0.00s)
> > RESULTS - rpm.RpmBasicTest.test_rpm_help: SKIPPED (0.00s)
> > RESULTS - rpm.RpmBasicTest.test_rpm_query: SKIPPED (0.00s)
> > RESULTS - rpm.RpmBasicTest.test_rpm_query_nonroot: SKIPPED (0.00s)
> > RESULTS - rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size: SKIPPED (0.00s)
> > RESULTS - rpm.RpmInstallRemoveTest.test_rpm_install: SKIPPED (0.00s)
> > RESULTS - rpm.RpmInstallRemoveTest.test_rpm_remove: SKIPPED (0.00s)
> > RESULTS - rust.RustCompileTest.test_cargo_compile: SKIPPED (0.00s)
> > RESULTS - rust.RustCompileTest.test_rust_compile: SKIPPED (0.00s)
> > RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s)
> > RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s)
> > RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED (0.00s)
> > RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED (0.00s)
> > RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s)
> > RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s)
> > RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s)
> > RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor: SKIPPED (0.00s)
> > RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s)
> > RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s)
> > RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s)
> > SUMMARY:
> > core-image-minimal () - Ran 70 tests in 22.173s
> > core-image-minimal - OK - All required tests passed (successes=14, skipped=56, failures=0, errors=0)
> > NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded
> > NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun and all succeeded.
> >
> > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > ---
> >  meta/classes-recipe/testimage.bbclass | 14 +++++++++++---
> >  1 file changed, 11 insertions(+), 3 deletions(-)
> >
> > v2: set TEST_SERVER_IP correctly to default qemu dhcp server IP 10.0.2.2
> > so that testimage selftests pass
> >
> > v1: https://lists.openembedded.org/g/openembedded-core/message/186424
> >
> > diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass
> > index e3068348ff..7340996788 100644
> > --- a/meta/classes-recipe/testimage.bbclass
> > +++ b/meta/classes-recipe/testimage.bbclass
> > @@ -322,7 +322,7 @@ def testimage_main(d):
> >      ovmf = d.getVar("QEMU_USE_OVMF")
> >
> >      slirp = False
> > -    if d.getVar("QEMU_USE_SLIRP"):
> > +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
> >          slirp = True
> >
> >      # TODO: We use the current implementation of qemu runner because of
> > @@ -369,10 +369,18 @@ def testimage_main(d):
> >      # runtime use network for download projects for build
> >      export_proxies(d)
> >
> > +    if slirp:
> > +        target_ip = "127.0.0.1"
> 
> will this work when running multiple instances of qemu ?
> e.g.  try bitbake core-image-ptest-all

The port forwarding settings are same for each of the qemu virtual machine instances
currently.

So one instance reserves the 127.0.0.1 and TCP port 2222, by default.
Another instance will fail to reserve the port 2222 and qemu startup fails.

So I don't think "slirp" works with multiple qemu instances on a single machine.

This could be changed if the hostfwd port number selection was changed to be dynamic.
runqemu currently hard codes this as:

ostfwd = ",hostfwd=tcp:127.0.0.1:2222-:22,hostfwd=tcp:127.0.0.1:2323-:23"

A retry with different, increasing port numbers could be added.

Then qemurunner.py could be changed to detect the effective port from qemu startup logs,
as it already now partially does as a fallback, or maybe the current code would even work
with dynamic port numbers.

Cheers,

-Mikko

> 
> > +        # from qemu target to host with default DHCP server
> > +        server_ip = "10.0.2.2"
> > +    else:
> > +        target_ip = d.getVar("TEST_TARGET_IP")
> > +        server_ip = d.getVar("TEST_SERVER_IP")
> > +
> >      # the robot dance
> >      target = OERuntimeTestContextExecutor.getTarget(
> > -        d.getVar("TEST_TARGET"), logger, d.getVar("TEST_TARGET_IP"),
> > -        d.getVar("TEST_SERVER_IP"), **target_kwargs)
> > +        d.getVar("TEST_TARGET"), logger, target_ip,
> > +        server_ip, **target_kwargs)
> >
> >      # test context
> >      tc = OERuntimeTestContext(td, logger, target, image_packages, extract_dir)
> > --
> > 2.34.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#186547): https://lists.openembedded.org/g/openembedded-core/message/186547
> > Mute This Topic: https://lists.openembedded.org/mt/100910036/1997914
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Mikko Rapeli]

Hi,

On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> will this work when running multiple instances of qemu ?
> e.g.  try bitbake core-image-ptest-all

I was not aware of core-image-ptest-all. Tried to build it but it doesn't
seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
needed to test core-image-minimal:

Error: 
 Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
  - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
  - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
  - conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)

oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
just be skipped.

Cheers,

-Mikko

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Richard Purdie]

On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > will this work when running multiple instances of qemu ?
> > e.g.  try bitbake core-image-ptest-all
> 
> I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> needed to test core-image-minimal:
> 
> Error: 
>  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
>   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
>   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
>   - conflicting requests
> (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> 
> oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> just be skipped.

The autobuilder actually does that, the minimal image is just tested
with the small number of non-network tests. The main thing was to test
it does actually boot to a login prompt. We have other tests which test
the other areas with other images.

The reason for the above is that there will be ptest openssh images
which conflict with the dropbear ones. You can likely avoid that by
using:

IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"

The ptest images are designed to only include the ptest in question so
in theory are otherwise as minimal as the dependencies allow.

Cheers,

Richard

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Mikko Rapeli]

Hi,

On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > Hi,
> > 
> > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > will this work when running multiple instances of qemu ?
> > > e.g.  try bitbake core-image-ptest-all
> > 
> > I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> > needed to test core-image-minimal:
> > 
> > Error: 
> >  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
> >   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> >   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> >   - conflicting requests
> > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> > 
> > oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> > just be skipped.
> 
> The autobuilder actually does that, the minimal image is just tested
> with the small number of non-network tests. The main thing was to test
> it does actually boot to a login prompt. We have other tests which test
> the other areas with other images.

Yes, granted it's enough to test that boot to serial console login works.

> The reason for the above is that there will be ptest openssh images
> which conflict with the dropbear ones. You can likely avoid that by
> using:
> 
> IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"
> 
> The ptest images are designed to only include the ptest in question so
> in theory are otherwise as minimal as the dependencies allow.

Alright, this I could try. But I fear there is a log more missing from my
plain poky and default machine target to get the selftests and tests running.
This magic is somewhere in the autobuilder related git repositories, but from plain
poky checkout with a specific commit from master branch I don't know which versions
and repos to use so that the tests would be passing.

With these modifications in local.conf:

IMAGE_CLASSES += "testimage"
TEST_RUNQEMUPARAMS += "slirp"
IMAGE_FEATURES += "ssh-server-dropbear"
# update kernel to latest available in poky
PREFERRED_VERSION_linux-yocto = ""
SANITY_TESTED_DISTROS = ""

at least runtime_test.TestImage are passing with slirp now.

Without MAGE_FEATURES += "ssh-server-dropbear", "bitbake core-image-ptest-all" now succeeds
and "bitbake -c testimage core-image-ptest-all" is running the tests, seeminly in series.
At least there are no multiple qemu instances running in parallel and no failures related to
slirp ssh port being reserved by a single qemu instance. But the tests are reporting only skips
so maybe the autobuilder scripts have some settings which I don't have correctly set:

Cannot run ptests without @expectedFailure as ptests are expected to fail
QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
Cannot run ptests without @expectedFailure as ptests are expected to fail
QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
Cannot run ptests without @expectedFailure as ptests are expected to fail
RESULTS:
RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
SUMMARY:
core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)

The ptest execution seems to be skipped for all images.

Cheers,

-Mikko

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Alexander Kanavin]

On Wed, 23 Aug 2023 at 11:48, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:

> Cannot run ptests without @expectedFailure as ptests are expected to fail
> QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
> QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> RESULTS:
> RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
> RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
> RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> SUMMARY:
> core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
> core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)
>
> The ptest execution seems to be skipped for all images.

This is actually correct and ptest did run. We have a soft and hard
fail options for ptests, and it defaults to soft fail
(PTEST_EXPECT_FAILURE = "1" in core-image-ptest), i.e. warnings on the
autobuilder.

Alex

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Mikko Rapeli]

Hi,

On Wed, Aug 23, 2023 at 11:57:50AM +0200, Alexander Kanavin wrote:
> On Wed, 23 Aug 2023 at 11:48, Mikko Rapeli <builder@linaro.org> wrote:
> 
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
> > QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > RESULTS:
> > RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
> > RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
> > RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> > SUMMARY:
> > core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
> > core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)
> >
> > The ptest execution seems to be skipped for all images.
> 
> This is actually correct and ptest did run. We have a soft and hard
> fail options for ptests, and it defaults to soft fail
> (PTEST_EXPECT_FAILURE = "1" in core-image-ptest), i.e. warnings on the
> autobuilder.

OK, that is good to know. In this case with local.conf additions for "slirp":

IMAGE_CLASSES += "testimage"
#IMAGE_FEATURES += "ssh-server-dropbear"
TEST_RUNQEMUPARAMS += "slirp"
# update kernel to latest available in poky
PREFERRED_VERSION_linux-yocto = ""
SANITY_TESTED_DISTROS = ""

The tests passed with one warning, which I'll try to check:

WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
  File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
    self.fail(failmsg)
AssertionError: 
Failed ptests:
{'openssh': ['banner']}
...
SUMMARY:
core-image-ptest-lttng-tools () - Ran 5 tests in 2289.072s
core-image-ptest-lttng-tools - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)
NOTE: recipe core-image-ptest-lttng-tools-1.0-r0: task do_testimage: Succeeded
NOTE: Running noexec task 1405 of 1405 (/home/builder/src/poky/meta/recipes-core/images/core-image-ptest-all.bb:do_testimage)
NOTE: Tasks Summary: Attempted 1405 tasks of which 1304 didn't need to be rerun and all succeeded.

Summary: There was 1 WARNING message.

Cheers,

-Mikko

openssh ptest banner failure (was Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS )

[Mikko Rapeli]

Hi,

On Wed, Aug 23, 2023 at 01:49:39PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> The tests passed with one warning, which I'll try to check:
> 
> WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing ptests.
> Traceback (most recent call last):
>   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
>     self.do_ptestrunner()
>   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
>     self.fail(failmsg)
> AssertionError: 
> Failed ptests:
> {'openssh': ['banner']}

Has this openssh banner test on x86_64 been failing on yocto autobuilders?

Log tmp/work/qemux86_64-poky-linux/core-image-ptest-openssh/1.0/temp/log.do_testimage
shows:

PASS: environment passing
run test transfer.sh ...
PASS: transfer data
run test banner.sh ...
test banner: missing banner file
test banner: size 0
test banner: size 10
test banner: size 100
test banner: size 1000
cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
banner size 1000 mismatch
test banner: size 10000
test banner: size 100000
test banner: suppress banner (-q)
FAIL:  banner
make: *** [Makefile:223: t-exec] Error 1

On manual execution this passes:

$ runqemu slirp nographic novga core-image-ptest-openssh
...
root@qemux86-64:~# ptest-runner
START: ptest-runner
2023-08-23T12:04
BEGIN: /usr/lib/openssh/ptest
...
PASS: transfer data
run test banner.sh ...
test banner: missing banner file
test banner: size 0
test banner: size 10
test banner: size 100
test banner: size 1000
test banner: size 10000
test banner: size 100000
test banner: suppress banner (-q)
PASS: banner
...

client rekey tests are still running.

Cheers,

-Mikko

Re: openssh ptest banner failure (was Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS )

[Alexander Kanavin]

I've just seen this too. I think to trigger this you need to run the
test via bitbake -c testimage rather than directly over console - the
output is then piped over ssh, and something goes wrong due to the
large volume of banner data perhaps? Would *really* appreciate if you
can get to the bottom of it.

Alex

On Wed, 23 Aug 2023 at 15:01, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
>
> Hi,
>
> On Wed, Aug 23, 2023 at 01:49:39PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > The tests passed with one warning, which I'll try to check:
> >
> > WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing ptests.
> > Traceback (most recent call last):
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
> >     self.do_ptestrunner()
> >   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
> >     self.fail(failmsg)
> > AssertionError:
> > Failed ptests:
> > {'openssh': ['banner']}
>
> Has this openssh banner test on x86_64 been failing on yocto autobuilders?
>
> Log tmp/work/qemux86_64-poky-linux/core-image-ptest-openssh/1.0/temp/log.do_testimage
> shows:
>
> PASS: environment passing
> run test transfer.sh ...
> PASS: transfer data
> run test banner.sh ...
> test banner: missing banner file
> test banner: size 0
> test banner: size 10
> test banner: size 100
> test banner: size 1000
> cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
> banner size 1000 mismatch
> test banner: size 10000
> test banner: size 100000
> test banner: suppress banner (-q)
> FAIL:  banner
> make: *** [Makefile:223: t-exec] Error 1
>
> On manual execution this passes:
>
> $ runqemu slirp nographic novga core-image-ptest-openssh
> ...
> root@qemux86-64:~# ptest-runner
> START: ptest-runner
> 2023-08-23T12:04
> BEGIN: /usr/lib/openssh/ptest
> ...
> PASS: transfer data
> run test banner.sh ...
> test banner: missing banner file
> test banner: size 0
> test banner: size 10
> test banner: size 100
> test banner: size 1000
> test banner: size 10000
> test banner: size 100000
> test banner: suppress banner (-q)
> PASS: banner
> ...
>
> client rekey tests are still running.
>
> Cheers,
>
> -Mikko

Re: openssh ptest banner failure (was Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS )

[Mikko Rapeli]

Hi,

On Wed, Aug 23, 2023 at 03:24:43PM +0200, Alexander Kanavin wrote:
> I've just seen this too. I think to trigger this you need to run the
> test via bitbake -c testimage rather than directly over console - the
> output is then piped over ssh, and something goes wrong due to the
> large volume of banner data perhaps? Would *really* appreciate if you
> can get to the bottom of it.

Ok, I'll try to have a look.

I also just found out that:

https://wiki.yoctoproject.org/wiki/Weekly_Status

-> https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT

-> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
   Reported: 	2023-08-02 14:32 UTC by Alexandre Belloni 

So it's a known issue.

Cheers,

-Mikko

Re: openssh ptest banner failure (was Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS )

[Richard Purdie]

On Wed, 2023-08-23 at 16:01 +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Wed, Aug 23, 2023 at 01:49:39PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > The tests passed with one warning, which I'll try to check:
> > 
> > WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing ptests.
> > Traceback (most recent call last):
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
> >     self.do_ptestrunner()
> >   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
> >     self.fail(failmsg)
> > AssertionError: 
> > Failed ptests:
> > {'openssh': ['banner']}
> 
> Has this openssh banner test on x86_64 been failing on yocto autobuilders?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178

so yes, it has. Not always but sometimes. We do try and ensure there
are open bugs for any issues although I'm aware the current qemuppc one
does not have a bug as yet.

Cheers,

Richard

> 
> Log tmp/work/qemux86_64-poky-linux/core-image-ptest-openssh/1.0/temp/log.do_testimage
> shows:
> 
> PASS: environment passing
> run test transfer.sh ...
> PASS: transfer data
> run test banner.sh ...
> test banner: missing banner file
> test banner: size 0
> test banner: size 10
> test banner: size 100
> test banner: size 1000
> cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
> banner size 1000 mismatch
> test banner: size 10000
> test banner: size 100000
> test banner: suppress banner (-q)
> FAIL:  banner
> make: *** [Makefile:223: t-exec] Error 1
> 
> On manual execution this passes:
> 
> $ runqemu slirp nographic novga core-image-ptest-openssh
> ...
> root@qemux86-64:~# ptest-runner
> START: ptest-runner
> 2023-08-23T12:04
> BEGIN: /usr/lib/openssh/ptest
> ...
> PASS: transfer data
> run test banner.sh ...
> test banner: missing banner file
> test banner: size 0
> test banner: size 10
> test banner: size 100
> test banner: size 1000
> test banner: size 10000
> test banner: size 100000
> test banner: suppress banner (-q)
> PASS: banner
> ...
> 
> client rekey tests are still running.
> 
> Cheers,
> 
> -Mikko
> 

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Richard Purdie]

On Wed, 2023-08-23 at 12:47 +0300, Mikko Rapeli wrote:
> On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> > On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > > Hi,
> > > 
> > > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > > will this work when running multiple instances of qemu ?
> > > > e.g.  try bitbake core-image-ptest-all
> > > 
> > > I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> > > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> > > needed to test core-image-minimal:
> > > 
> > > Error: 
> > >  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
> > >   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> > >   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> > >   - conflicting requests
> > > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> > > 
> > > oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> > > just be skipped.
> > 
> > The autobuilder actually does that, the minimal image is just tested
> > with the small number of non-network tests. The main thing was to test
> > it does actually boot to a login prompt. We have other tests which test
> > the other areas with other images.
> 
> Yes, granted it's enough to test that boot to serial console login works.
> 
> > The reason for the above is that there will be ptest openssh images
> > which conflict with the dropbear ones. You can likely avoid that by
> > using:
> > 
> > IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"
> > 
> > The ptest images are designed to only include the ptest in question so
> > in theory are otherwise as minimal as the dependencies allow.
> 
> Alright, this I could try. But I fear there is a log more missing from my
> plain poky and default machine target to get the selftests and tests running.

There is no secret magic config the autobuilder uses. You keep asking
me for this and there isn't anything. It is actually starting to annoy
me a bit as there isn't anything "hidden".

The configurations used are all from this file:

https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json

Yes, there is a block of high level config around numbers of threads,
disk space monitoring, pressure regulation values and so on but we
purposefully keep the config to be as close to standard poky as we can.

When we run selftest we do a couple of things. Firstly we split the
machine and toolchain targets into separate areas.  We also split
reproducibility to it's own target and test mirroring elsewhere too.
This results in a slightly more complex selftest invocation:

OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

The only test which I don't think we run anywhere any more is the
test_checkpkg target.

You can see all this from the logs buildbot shows from it's UI on the
autobuilder too.

> This magic is somewhere in the autobuilder related git repositories, but from plain
> poky checkout with a specific commit from master branch I don't know which versions
> and repos to use so that the tests would be passing.
> 
> With these modifications in local.conf:
> 
> IMAGE_CLASSES += "testimage"
> TEST_RUNQEMUPARAMS += "slirp"

We do not use slirp on the autobuilder. We never have and we're
unlikely ever to do so and it is not something we officially support
for this. This is likely the biggest source of problems.

I appreciate that gives some networking challenges for people in
constrained environments but we did that primarily to allow for
simplifications in the rest of the setup.

> IMAGE_FEATURES += "ssh-server-dropbear"

I've already explained that this one does likely cause problems. We
simply don't run many tests against minimal images. 

> # update kernel to latest available in poky
> PREFERRED_VERSION_linux-yocto = ""

Not sure why this is needed?

> SANITY_TESTED_DISTROS = ""

This one we've discussed. It really should be fixed in a better way but
isn't anywhere near the top of the priority list.

> at least runtime_test.TestImage are passing with slirp now.
> 
> Without MAGE_FEATURES += "ssh-server-dropbear", "bitbake core-image-ptest-all" now succeeds
> and "bitbake -c testimage core-image-ptest-all" is running the tests, seeminly in series.
> At least there are no multiple qemu instances running in parallel and no failures related to
> slirp ssh port being reserved by a single qemu instance. But the tests are reporting only skips
> so maybe the autobuilder scripts have some settings which I don't have correctly set:
> 
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
> QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> RESULTS:
> RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
> RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
> RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> SUMMARY:
> core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
> core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)
> 
> The ptest execution seems to be skipped for all images.

I think Alex covers this.

You can compare it to what is shown on the autobuilder output. You can
also compare the testresults.json file too using "resulttool report" to
compare results with what the autobuilder runs.

Cheers,

Richard

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Mikko Rapeli]

Hi,

On Wed, Aug 23, 2023 at 11:49:39AM +0100, Richard Purdie wrote:
> On Wed, 2023-08-23 at 12:47 +0300, Mikko Rapeli wrote:
> > On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> > > On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > > > Hi,
> > > > 
> > > > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > > > will this work when running multiple instances of qemu ?
> > > > > e.g.  try bitbake core-image-ptest-all
> > > > 
> > > > I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> > > > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> > > > needed to test core-image-minimal:
> > > > 
> > > > Error: 
> > > >  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
> > > >   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> > > >   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> > > >   - conflicting requests
> > > > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> > > > 
> > > > oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> > > > just be skipped.
> > > 
> > > The autobuilder actually does that, the minimal image is just tested
> > > with the small number of non-network tests. The main thing was to test
> > > it does actually boot to a login prompt. We have other tests which test
> > > the other areas with other images.
> > 
> > Yes, granted it's enough to test that boot to serial console login works.
> > 
> > > The reason for the above is that there will be ptest openssh images
> > > which conflict with the dropbear ones. You can likely avoid that by
> > > using:
> > > 
> > > IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"
> > > 
> > > The ptest images are designed to only include the ptest in question so
> > > in theory are otherwise as minimal as the dependencies allow.
> > 
> > Alright, this I could try. But I fear there is a log more missing from my
> > plain poky and default machine target to get the selftests and tests running.
> 
> There is no secret magic config the autobuilder uses. You keep asking
> me for this and there isn't anything. It is actually starting to annoy
> me a bit as there isn't anything "hidden".

If I can't run the tests for poky the way yocto upstream does, then I'm afraid
I can't really help with the issues you've seeing with the tests.

> The configurations used are all from this file:
> 
> https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json

Ok so https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1948
for example uses yocto-autobuilder-helper at commit
af5d072a654a060c3ee61b5f394f52632e20200b. And the full repo
URL is visible not in the snippet there but in the full stdio download
of "Fetch yocto-autobuilder-helper" step. It was the web UI confusing me.

I'm sorry if these are all obvious to you and other developers but not
to me.

> Yes, there is a block of high level config around numbers of threads,
> disk space monitoring, pressure regulation values and so on but we
> purposefully keep the config to be as close to standard poky as we can.
>
> When we run selftest we do a couple of things. Firstly we split the
> machine and toolchain targets into separate areas.  We also split
> reproducibility to it's own target and test mirroring elsewhere too.
> This results in a slightly more complex selftest invocation:
> 
> OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15
> 
> The only test which I don't think we run anywhere any more is the
> test_checkpkg target.
> 
> You can see all this from the logs buildbot shows from it's UI on the
> autobuilder too.

Thanks, this explains alot!

I'm still wondering how the build hosts are setup with "modprobe vgem" and that
build user account is in "render" group (on Ubuntu at least). Should these
be added to some documentation too?

> > This magic is somewhere in the autobuilder related git repositories, but from plain
> > poky checkout with a specific commit from master branch I don't know which versions
> > and repos to use so that the tests would be passing.
> > 
> > With these modifications in local.conf:
> > 
> > IMAGE_CLASSES += "testimage"
> > TEST_RUNQEMUPARAMS += "slirp"
> 
> We do not use slirp on the autobuilder. We never have and we're
> unlikely ever to do so and it is not something we officially support
> for this. This is likely the biggest source of problems.

Sadly I can't figure out how to setup runqemu and testimage.bbclass and
oe-selftests networking without slirp and for my testing needs it has been good
enough, now also with oe-selftests.

> I appreciate that gives some networking challenges for people in
> constrained environments but we did that primarily to allow for
> simplifications in the rest of the setup.
> 
> > IMAGE_FEATURES += "ssh-server-dropbear"
> 
> I've already explained that this one does likely cause problems. We
> simply don't run many tests against minimal images. 
>
> > # update kernel to latest available in poky
> > PREFERRED_VERSION_linux-yocto = ""
> 
> Not sure why this is needed?

I'm trying to test the 6.4 kernel update and reproduce the issues you've seen
there too.

> > SANITY_TESTED_DISTROS = ""
> 
> This one we've discussed. It really should be fixed in a better way but
> isn't anywhere near the top of the priority list.

I'll try to get to this.

Cheers,

-Mikko

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Richard Purdie]

On Wed, 2023-08-23 at 14:47 +0300, Mikko Rapeli wrote:
> On Wed, Aug 23, 2023 at 11:49:39AM +0100, Richard Purdie wrote:
> > On Wed, 2023-08-23 at 12:47 +0300, Mikko Rapeli wrote:
> > > On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> > > > On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > > > > Hi,
> > > > > 
> > > > > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > > > > will this work when running multiple instances of qemu ?
> > > > > > e.g.  try bitbake core-image-ptest-all
> > > > > 
> > > > > I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> > > > > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> > > > > needed to test core-image-minimal:
> > > > > 
> > > > > Error: 
> > > > >  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
> > > > >   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> > > > >   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> > > > >   - conflicting requests
> > > > > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> > > > > 
> > > > > oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> > > > > just be skipped.
> > > > 
> > > > The autobuilder actually does that, the minimal image is just tested
> > > > with the small number of non-network tests. The main thing was to test
> > > > it does actually boot to a login prompt. We have other tests which test
> > > > the other areas with other images.
> > > 
> > > Yes, granted it's enough to test that boot to serial console login works.
> > > 
> > > > The reason for the above is that there will be ptest openssh images
> > > > which conflict with the dropbear ones. You can likely avoid that by
> > > > using:
> > > > 
> > > > IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"
> > > > 
> > > > The ptest images are designed to only include the ptest in question so
> > > > in theory are otherwise as minimal as the dependencies allow.
> > > 
> > > Alright, this I could try. But I fear there is a log more missing from my
> > > plain poky and default machine target to get the selftests and tests running.
> > 
> > There is no secret magic config the autobuilder uses. You keep asking
> > me for this and there isn't anything. It is actually starting to annoy
> > me a bit as there isn't anything "hidden".
> 
> If I can't run the tests for poky the way yocto upstream does, then I'm afraid
> I can't really help with the issues you've seeing with the tests.

If that isn't possible we need to fix the docs.

> > The configurations used are all from this file:
> > 
> > https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json
> 
> Ok so https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1948
> for example uses yocto-autobuilder-helper at commit
> af5d072a654a060c3ee61b5f394f52632e20200b. And the full repo
> URL is visible not in the snippet there but in the full stdio download
> of "Fetch yocto-autobuilder-helper" step. It was the web UI confusing me.
> 
> I'm sorry if these are all obvious to you and other developers but not
> to me.
> 
> > Yes, there is a block of high level config around numbers of threads,
> > disk space monitoring, pressure regulation values and so on but we
> > purposefully keep the config to be as close to standard poky as we can.
> > 
> > When we run selftest we do a couple of things. Firstly we split the
> > machine and toolchain targets into separate areas.  We also split
> > reproducibility to it's own target and test mirroring elsewhere too.
> > This results in a slightly more complex selftest invocation:
> > 
> > OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15
> > 
> > The only test which I don't think we run anywhere any more is the
> > test_checkpkg target.
> > 
> > You can see all this from the logs buildbot shows from it's UI on the
> > autobuilder too.
> 
> Thanks, this explains alot!
> 
> I'm still wondering how the build hosts are setup with "modprobe vgem" and that
> build user account is in "render" group (on Ubuntu at least). Should these
> be added to some documentation too?

Yes, they should.

> 
> > > This magic is somewhere in the autobuilder related git repositories, but from plain
> > > poky checkout with a specific commit from master branch I don't know which versions
> > > and repos to use so that the tests would be passing.
> > > 
> > > With these modifications in local.conf:
> > > 
> > > IMAGE_CLASSES += "testimage"
> > > TEST_RUNQEMUPARAMS += "slirp"
> > 
> > We do not use slirp on the autobuilder. We never have and we're
> > unlikely ever to do so and it is not something we officially support
> > for this. This is likely the biggest source of problems.
> 
> Sadly I can't figure out how to setup runqemu and testimage.bbclass and
> oe-selftests networking without slirp and for my testing needs it has been good
> enough, now also with oe-selftests.

As root you need to run:

runqemu-gen-tapdevs <uid> <gid> <number tap devs>

where uid is your user ID and gid is the group you want to be able to
access those devices. Once done once at boot, the builds will use those
devices. The autobuilder has these pre-setup.

You raise a good point that this should be documented, probably in the
testing manual. As we're all aware, documentation is hard so we should
either file a bug for that (and the vgem issue) or write a docs patch.

https://docs.yoctoproject.org/test-manual/index.html

> > I appreciate that gives some networking challenges for people in
> > constrained environments but we did that primarily to allow for
> > simplifications in the rest of the setup.
> > 
> > > IMAGE_FEATURES += "ssh-server-dropbear"
> > 
> > I've already explained that this one does likely cause problems. We
> > simply don't run many tests against minimal images. 
> > 
> > > # update kernel to latest available in poky
> > > PREFERRED_VERSION_linux-yocto = ""
> > 
> > Not sure why this is needed?
> 
> I'm trying to test the 6.4 kernel update and reproduce the issues you've seen
> there too.

Ok, I did change the default to 6.4 now.

> > > SANITY_TESTED_DISTROS = ""
> > 
> > This one we've discussed. It really should be fixed in a better way but
> > isn't anywhere near the top of the priority list.
> 
> I'll try to get to this.

Thanks,

Richard

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Ross Burton]

On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
> -    if d.getVar("QEMU_USE_SLIRP"):
> +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
>         slirp = True

This means we can remove the QEMU_USE_SLIRP variable too, right?

Ross

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Mikko Rapeli]

Hi,

On Tue, Sep 12, 2023 at 11:25:28AM +0000, Ross Burton wrote:
> On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
> > -    if d.getVar("QEMU_USE_SLIRP"):
> > +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
> >         slirp = True
> 
> This means we can remove the QEMU_USE_SLIRP variable too, right?

Yes. TEST_RUNQEMUPARAMS += "slirp" would be enough to run slirp with testimage.bbclass.

Cheers,

-Mikko

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Ross Burton]

On 12 Sep 2023, at 12:29, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> 
> Hi,
> 
> On Tue, Sep 12, 2023 at 11:25:28AM +0000, Ross Burton wrote:
>> On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
>>> -    if d.getVar("QEMU_USE_SLIRP"):
>>> +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
>>>        slirp = True
>> 
>> This means we can remove the QEMU_USE_SLIRP variable too, right?
> 
> Yes. TEST_RUNQEMUPARAMS += "slirp" would be enough to run slirp with testimage.bbclass.

Can you send a followup to do that shortly?

Ross

Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

[Mikko Rapeli]

On Tue, Sep 12, 2023 at 11:30:57AM +0000, Ross Burton wrote:
> On 12 Sep 2023, at 12:29, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > 
> > Hi,
> > 
> > On Tue, Sep 12, 2023 at 11:25:28AM +0000, Ross Burton wrote:
> >> On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
> >>> -    if d.getVar("QEMU_USE_SLIRP"):
> >>> +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
> >>>        slirp = True
> >> 
> >> This means we can remove the QEMU_USE_SLIRP variable too, right?
> > 
> > Yes. TEST_RUNQEMUPARAMS += "slirp" would be enough to run slirp with testimage.bbclass.
> 
> Can you send a followup to do that shortly?

Yes, I can do that.

In my developer builds I have following changes from this patch series:

oeqa selftest runtime_test.py: append to TEST_RUNQEMUPARAMS
oeqa dnf_runtime.py: fix HTTP server IP address and port
testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

Rest are already applied or rejected.

Cheers,

-Mikko

[PATCH v2 3/9] selftest runtime_test.py: add testimage.bbclass slirp test

[Mikko Rapeli]

Builds core-image-minimal with testimage.bbclass and an ssh server
and then runs tests with slirp networking. Default test suite must
pass.

With full caches on x86_64 with x86_64 target the test passes in 416 seconds:

2023-08-21 09:01:12,295 - oe-selftest - INFO - Ran 1 test in 464.955s
2023-08-21 09:01:12,295 - oe-selftest - INFO - OK
2023-08-21 09:01:19,648 - oe-selftest - INFO - RESULTS:
2023-08-21 09:01:19,648 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_slirp: PASSED (416.62s)
2023-08-21 09:01:19,711 - oe-selftest - INFO - SUMMARY:
2023-08-21 09:01:19,711 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 464.956s
2023-08-21 09:01:19,711 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/cases/runtime_test.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

v2: use :append instead of overwriting variables

v1: https://lists.openembedded.org/g/openembedded-core/message/186427

diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 463679dd32..19c8f0e9ec 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -129,6 +129,21 @@ class TestImage(OESelftestTestCase):
         bitbake('core-image-full-cmdline socat')
         bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_slirp(self):
+        """
+        Summary: Check basic testimage functionality with qemu and slirp networking.
+        """
+
+        features = '''
+IMAGE_CLASSES:append = " testimage"
+IMAGE_FEATURES:append = " ssh-server-dropbear"
+TEST_RUNQEMUPARAMS:append = " slirp"
+'''
+        self.write_config(features)
+
+        bitbake('core-image-minimal')
+        bitbake('-c testimage core-image-minimal')
+
     def test_testimage_dnf(self):
         """
         Summary: Check package feeds functionality for dnf
-- 
2.34.1

Re: [OE-core] [PATCH v2 3/9] selftest runtime_test.py: add testimage.bbclass slirp test

[Alexander Kanavin]

I guess the extra space stuff should be going here instead?

Alex

On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
>
> Builds core-image-minimal with testimage.bbclass and an ssh server
> and then runs tests with slirp networking. Default test suite must
> pass.
>
> With full caches on x86_64 with x86_64 target the test passes in 416 seconds:
>
> 2023-08-21 09:01:12,295 - oe-selftest - INFO - Ran 1 test in 464.955s
> 2023-08-21 09:01:12,295 - oe-selftest - INFO - OK
> 2023-08-21 09:01:19,648 - oe-selftest - INFO - RESULTS:
> 2023-08-21 09:01:19,648 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_slirp: PASSED (416.62s)
> 2023-08-21 09:01:19,711 - oe-selftest - INFO - SUMMARY:
> 2023-08-21 09:01:19,711 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 464.956s
> 2023-08-21 09:01:19,711 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)
>
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/lib/oeqa/selftest/cases/runtime_test.py | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
>
> v2: use :append instead of overwriting variables
>
> v1: https://lists.openembedded.org/g/openembedded-core/message/186427
>
> diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
> index 463679dd32..19c8f0e9ec 100644
> --- a/meta/lib/oeqa/selftest/cases/runtime_test.py
> +++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
> @@ -129,6 +129,21 @@ class TestImage(OESelftestTestCase):
>          bitbake('core-image-full-cmdline socat')
>          bitbake('-c testimage core-image-full-cmdline')
>
> +    def test_testimage_slirp(self):
> +        """
> +        Summary: Check basic testimage functionality with qemu and slirp networking.
> +        """
> +
> +        features = '''
> +IMAGE_CLASSES:append = " testimage"
> +IMAGE_FEATURES:append = " ssh-server-dropbear"
> +TEST_RUNQEMUPARAMS:append = " slirp"
> +'''
> +        self.write_config(features)
> +
> +        bitbake('core-image-minimal')
> +        bitbake('-c testimage core-image-minimal')
> +
>      def test_testimage_dnf(self):
>          """
>          Summary: Check package feeds functionality for dnf
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186550): https://lists.openembedded.org/g/openembedded-core/message/186550
> Mute This Topic: https://lists.openembedded.org/mt/100910039/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 3/9] selftest runtime_test.py: add testimage.bbclass slirp test

[Mikko Rapeli]

Hi,

On Wed, Aug 23, 2023 at 08:45:49PM +0200, Alexander Kanavin wrote:
> I guess the extra space stuff should be going here instead?

Or some other image which is supposed to be compatible with oeqa runtime tests
and testimage.bbclass. I think it's better to just ignore this patch.

Cheers,

-Mikko

Re: [OE-core] [PATCH v2 3/9] selftest runtime_test.py: add testimage.bbclass slirp test

[Alexander Kanavin]

On Thu, 24 Aug 2023 at 08:41, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> On Wed, Aug 23, 2023 at 08:45:49PM +0200, Alexander Kanavin wrote:
> > I guess the extra space stuff should be going here instead?
>
> Or some other image which is supposed to be compatible with oeqa runtime tests
> and testimage.bbclass. I think it's better to just ignore this patch.

It's a useful test, and core-image-minimal is entirely valid image for
testing (it's used throughout autobuilder and selftest). The point is
simply that if you need to customize that image or any other image for
the purposes of the test, all such customizations should be collected
in the test itself.

Alex

[PATCH v2 4/9] oeqa dnf_runtime.py: fix HTTP server IP address and port

[Mikko Rapeli]

Use correct HTTPService parameters like apt.py when setting up the repo
server. These work with qemu tun and slirp networking. Fixes test
failure with slirp networking when executing testimage.bbclass
selftests "oe-selftest -r runtime_test.TestImage".

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
index 64a3502370..d2f0f88f7d 100644
--- a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
+++ b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
@@ -10,7 +10,8 @@ class DnfSelftest(DnfTest):
         import tempfile
         cls.temp_dir = tempfile.TemporaryDirectory(prefix="oeqa-remotefeeds-")
         cls.repo_server = HTTPService(os.path.join(cls.tc.td['WORKDIR'], 'oe-rootfs-repo'),
-                                      cls.tc.target.server_ip)
+                                      '0.0.0.0', port=cls.tc.target.server_port,
+                                      logger=cls.tc.logger)
         cls.repo_server.start()
 
     @classmethod
-- 
2.34.1

[PATCH v2 5/9] oeqa selftest context.py: whitespace fix

[Mikko Rapeli]

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/context.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index f2a5ba792f..4b33245758 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -226,14 +226,14 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
         machines = []
 
         bbpath = self.tc_kwargs['init']['td']['BBPATH'].split(':')
-    
+
         for path in bbpath:
             found_machines = glob.glob(os.path.join(path, 'conf', 'machine', '*.conf'))
             if found_machines:
                 for i in found_machines:
                     # eg: '/home/<user>/poky/meta-intel/conf/machine/intel-core2-32.conf'
                     machines.append(os.path.splitext(os.path.basename(i))[0])
-    
+
         return machines
 
     def _get_cases_paths(self, bbpath):
-- 
2.34.1

[PATCH v2 6/9] oeqa selftest context.py: remove warning from missing meta-selftest

[Mikko Rapeli]

It's not a warning but a handled case and layer gets added
automatically. Very few build configs have this layer enabled
by default.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/context.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index 4b33245758..bd03e765b0 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -306,7 +306,7 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
                 os.chdir(builddir)
 
             if not "meta-selftest" in self.tc.td["BBLAYERS"]:
-                self.tc.logger.warning("meta-selftest layer not found in BBLAYERS, adding it")
+                self.tc.logger.info("meta-selftest layer not found in BBLAYERS, adding it")
                 meta_selftestdir = os.path.join(
                     self.tc.td["BBLAYERS_FETCH_DIR"], 'meta-selftest')
                 if os.path.isdir(meta_selftestdir):
-- 
2.34.1

[PATCH v2 7/9] oeqa selftest context.py: fix git commands and set branch name

[Mikko Rapeli]

Check return values of each git command and set initial branch name
to avoid this warning on every test run:

NOTE: Starting bitbake server...
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint:   git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint:   git branch -m <name>

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/context.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index bd03e765b0..bc3915a3aa 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -104,7 +104,7 @@ class OESelftestTestContext(OETestContext):
         oe.path.copytree(builddir + "/cache", newbuilddir + "/cache")
         oe.path.copytree(selftestdir, newselftestdir)
 
-        subprocess.check_output("git init; git add *; git commit -a -m 'initial'", cwd=newselftestdir, shell=True)
+        subprocess.check_output("git init -b main && git add * && git commit -a -m 'initial'", cwd=newselftestdir, shell=True)
 
         # Tried to used bitbake-layers add/remove but it requires recipe parsing and hence is too slow
         subprocess.check_output("sed %s/conf/bblayers.conf -i -e 's#%s#%s#g'" % (newbuilddir, selftestdir, newselftestdir), cwd=newbuilddir, shell=True)
-- 
2.34.1

Re: [OE-core] [PATCH v2 7/9] oeqa selftest context.py: fix git commands and set branch name

[Ross Burton]

On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
> -        subprocess.check_output("git init; git add *; git commit -a -m 'initial'", cwd=newselftestdir, shell=True)
> +        subprocess.check_output("git init -b main && git add * && git commit -a -m 'initial'", cwd=newselftestdir, shell=True)

I believe the last time I tried to fix this we had a problem where older distros had a git which didn’t handle init -b (which was added in 2.28).

I’m not sure whether this isn’t a problem anymore…

Ross

[PATCH v2 8/9] oeqa selftest runtime_test.py: append to TEST_RUNQEMUPARAMS

[Mikko Rapeli]

Don't overwrite TEST_RUNQEMUPARAMS since user may have added "slirp"
or similar things to it in site.conf, auto.conf or local.conf. Use
:append to add to it. Fixes failing tests when only "slirp" networking
works with qemu and tun based one is broken on the host machine.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/cases/runtime_test.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 19c8f0e9ec..5f90bc658f 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -251,11 +251,11 @@ TEST_RUNQEMUPARAMS:append = " slirp"
         features += 'TEST_SUITES = "ping ssh virgl"\n'
         features += 'IMAGE_FEATURES:append = " ssh-server-dropbear"\n'
         features += 'IMAGE_INSTALL:append = " kmscube"\n'
-        features_gtk = features + 'TEST_RUNQEMUPARAMS = "gtk gl"\n'
+        features_gtk = features + 'TEST_RUNQEMUPARAMS:append = " gtk gl"\n'
         self.write_config(features_gtk)
         bitbake('core-image-minimal')
         bitbake('-c testimage core-image-minimal')
-        features_sdl = features + 'TEST_RUNQEMUPARAMS = "sdl gl"\n'
+        features_sdl = features + 'TEST_RUNQEMUPARAMS:append = " sdl gl"\n'
         self.write_config(features_sdl)
         bitbake('core-image-minimal')
         bitbake('-c testimage core-image-minimal')
@@ -283,7 +283,7 @@ TEST_RUNQEMUPARAMS:append = " slirp"
         features += 'TEST_SUITES = "ping ssh virgl"\n'
         features += 'IMAGE_FEATURES:append = " ssh-server-dropbear"\n'
         features += 'IMAGE_INSTALL:append = " kmscube"\n'
-        features += 'TEST_RUNQEMUPARAMS = "egl-headless"\n'
+        features += 'TEST_RUNQEMUPARAMS:append = " egl-headless"\n'
         self.write_config(features)
         bitbake('core-image-minimal')
         bitbake('-c testimage core-image-minimal')
-- 
2.34.1

[PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Mikko Rapeli]

If access to /dev/dri/renderD128 fails, then qemu with 3d graphics will
fail to start and errors are rather cryptic like:

qemu-system-x86_64: egl: render node init failed

To fix this, users likely need to

 * modprobe vgem
 * add their user to "render" group to write to /dev/dri/renderD128

If access is not available due to missing HW, driver or failing access,
then skip the test:

2023-08-22 14:41:20,591 - oe-selftest - INFO - test_testimage_virgl_headless (runtime_test.TestImage)
2023-08-22 14:41:20,603 - oe-selftest - INFO -  ... skipped 'Can not open "/dev/dri/renderD128" device'
2023-08-22 14:41:20,603 - oe-selftest - INFO - Can not open "/dev/dri/renderD128" device

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/cases/runtime_test.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 5f90bc658f..e72504773a 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -264,7 +264,8 @@ TEST_RUNQEMUPARAMS:append = " slirp"
     def test_testimage_virgl_headless(self):
         """
         Summary: Check host-assisted accelerate OpenGL functionality in qemu with egl-headless frontend
-        Expected: 1. Check that virgl kernel driver is loaded and 3d acceleration is enabled
+        Expected: 1. Check that virgl kernel driver is loaded (modprobe vgem, user part of "render" group and
+                     can open /dev/dri/renderD* device) and 3d acceleration is enabled
                   2. Check that kmscube demo runs without crashing.
         Product: oe-core
         Author: Alexander Kanavin <alex.kanavin@gmail.com>
@@ -276,6 +277,14 @@ TEST_RUNQEMUPARAMS:append = " slirp"
             distro.startswith('almalinux') or distro.startswith('rocky')):
             self.skipTest('virgl headless cannot be tested with %s' %(distro))
 
+        # test requires vgem driver and possibly "render" group rights to access device file
+        render_dev = "/dev/dri/renderD128"
+        try:
+            with open(render_dev, "w") as f:
+                f.close()
+        except IOError:
+            self.skipTest('Can not open "%s" device' % (render_dev))
+
         qemu_distrofeatures = get_bb_var('DISTRO_FEATURES', 'qemu-system-native')
         features = 'IMAGE_CLASSES += "testimage"\n'
         if 'opengl' not in qemu_distrofeatures:
-- 
2.34.1

Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Alexander Kanavin]

On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
>
> If access to /dev/dri/renderD128 fails, then qemu with 3d graphics will
> fail to start and errors are rather cryptic like:
>
> qemu-system-x86_64: egl: render node init failed
>
> To fix this, users likely need to
>
>  * modprobe vgem
>  * add their user to "render" group to write to /dev/dri/renderD128
>
> If access is not available due to missing HW, driver or failing access,
> then skip the test:

This should be a failure rather than a skip. Otherwise, autobuilder
testing will silently regress and no one will notice. The error
message should include information from IOError as well so that it's
clear that the issue is device file permissions.

Alex

Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Alexander Kanavin]

Oh, and device should be /dev/dri/render*, and all of this should be in runqemu.

Alex

On Wed, 23 Aug 2023 at 20:37, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> >
> > If access to /dev/dri/renderD128 fails, then qemu with 3d graphics will
> > fail to start and errors are rather cryptic like:
> >
> > qemu-system-x86_64: egl: render node init failed
> >
> > To fix this, users likely need to
> >
> >  * modprobe vgem
> >  * add their user to "render" group to write to /dev/dri/renderD128
> >
> > If access is not available due to missing HW, driver or failing access,
> > then skip the test:
>
> This should be a failure rather than a skip. Otherwise, autobuilder
> testing will silently regress and no one will notice. The error
> message should include information from IOError as well so that it's
> clear that the issue is device file permissions.
>
> Alex

Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Alexander Kanavin]

Better yet, how about fixing this at the source, i.e. qemu itself?
Sorry for the rapid-fire :)

Alex

On Wed, 23 Aug 2023 at 20:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> Oh, and device should be /dev/dri/render*, and all of this should be in runqemu.
>
> Alex
>
> On Wed, 23 Aug 2023 at 20:37, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >
> > On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > >
> > > If access to /dev/dri/renderD128 fails, then qemu with 3d graphics will
> > > fail to start and errors are rather cryptic like:
> > >
> > > qemu-system-x86_64: egl: render node init failed
> > >
> > > To fix this, users likely need to
> > >
> > >  * modprobe vgem
> > >  * add their user to "render" group to write to /dev/dri/renderD128
> > >
> > > If access is not available due to missing HW, driver or failing access,
> > > then skip the test:
> >
> > This should be a failure rather than a skip. Otherwise, autobuilder
> > testing will silently regress and no one will notice. The error
> > message should include information from IOError as well so that it's
> > clear that the issue is device file permissions.
> >
> > Alex

Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Alexander Kanavin]

I mean runqemu does already have

def check_render_nodes(self):

where this should be fixed. Sorry can't help myself :)

Alex

On Wed, 23 Aug 2023 at 20:40, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
>
> Better yet, how about fixing this at the source, i.e. qemu itself?
> Sorry for the rapid-fire :)
>
> Alex
>
> On Wed, 23 Aug 2023 at 20:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >
> > Oh, and device should be /dev/dri/render*, and all of this should be in runqemu.
> >
> > Alex
> >
> > On Wed, 23 Aug 2023 at 20:37, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > >
> > > On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > > >
> > > > If access to /dev/dri/renderD128 fails, then qemu with 3d graphics will
> > > > fail to start and errors are rather cryptic like:
> > > >
> > > > qemu-system-x86_64: egl: render node init failed
> > > >
> > > > To fix this, users likely need to
> > > >
> > > >  * modprobe vgem
> > > >  * add their user to "render" group to write to /dev/dri/renderD128
> > > >
> > > > If access is not available due to missing HW, driver or failing access,
> > > > then skip the test:
> > >
> > > This should be a failure rather than a skip. Otherwise, autobuilder
> > > testing will silently regress and no one will notice. The error
> > > message should include information from IOError as well so that it's
> > > clear that the issue is device file permissions.
> > >
> > > Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186629): https://lists.openembedded.org/g/openembedded-core/message/186629
> Mute This Topic: https://lists.openembedded.org/mt/100910043/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Mikko Rapeli]

Hi,

On Wed, Aug 23, 2023 at 08:37:47PM +0200, Alexander Kanavin wrote:
> On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> >
> > If access to /dev/dri/renderD128 fails, then qemu with 3d graphics will
> > fail to start and errors are rather cryptic like:
> >
> > qemu-system-x86_64: egl: render node init failed
> >
> > To fix this, users likely need to
> >
> >  * modprobe vgem
> >  * add their user to "render" group to write to /dev/dri/renderD128
> >
> > If access is not available due to missing HW, driver or failing access,
> > then skip the test:
> 
> This should be a failure rather than a skip. Otherwise, autobuilder
> testing will silently regress and no one will notice. The error
> message should include information from IOError as well so that it's
> clear that the issue is device file permissions.

Then this change is not needed at all. Test already fails if render device
is not available or if access fails. In my experience very few environments have vgem
driver loaded and "render" group permissions for the user running the builds and tests.

Cheers,

-Mikko

Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Alexander Kanavin]

On Thu, 24 Aug 2023 at 08:45, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> Then this change is not needed at all. Test already fails if render device
> is not available or if access fails. In my experience very few environments have vgem
> driver loaded and "render" group permissions for the user running the builds and tests.

check_render_nodes() in runqemu could however be improved to check for
permissions as well (it currently checks only presence).

Alex

Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Alexander Kanavin]

This patch made it to master-next - please don't. The issue that needs
to be fixed is unhelpful error messages, which should be addressed in
runqemu (I'll make a patch for it).

But the test should continue to fail rather than be skipped (until
it's on a distro where it can't pass, which is only RHEL clones at the
moment). Otherwise skipping the test quietly amounts to disabling it
on newly added autobuilder workers or other instances where selftest
runs, and regressions will follow that.

Alex

On Thu, 24 Aug 2023 at 08:57, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
>
> On Thu, 24 Aug 2023 at 08:45, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > Then this change is not needed at all. Test already fails if render device
> > is not available or if access fails. In my experience very few environments have vgem
> > driver loaded and "render" group permissions for the user running the builds and tests.
>
> check_render_nodes() in runqemu could however be improved to check for
> permissions as well (it currently checks only presence).
>
> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186639): https://lists.openembedded.org/g/openembedded-core/message/186639
> Mute This Topic: https://lists.openembedded.org/mt/100910043/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails

[Richard Purdie]

On Thu, 2023-09-07 at 10:59 +0200, Alexander Kanavin wrote:
> This patch made it to master-next - please don't. The issue that needs
> to be fixed is unhelpful error messages, which should be addressed in
> runqemu (I'll make a patch for it).
> 
> But the test should continue to fail rather than be skipped (until
> it's on a distro where it can't pass, which is only RHEL clones at the
> moment). Otherwise skipping the test quietly amounts to disabling it
> on newly added autobuilder workers or other instances where selftest
> runs, and regressions will follow that.

Agreed. Ross asked these be reconsidered as there was a worry some were
falling through the cracks. I know some had issues and need to dive
deeper into remembering what happened. I've removed that one locally.

Cheers,

Richard

Re: [OE-core] [PATCH v2 1/9] core-image-minimal: increase extra space to pass df.py oeqa runtime test

[Alexander Kanavin]

Either you need to use a different image in testing, or add the needed
pieces to this image in the same commit, explaining why.

Adding extra space because you have a local, custom configuration
where it is required is not correct.

Alex

On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
>
> testimage.bbclass requires that ssh communication with target works
> and then it runs oeqa runtime tests with that. Tests like df.py
> check that there is more than 5Mb free space on the rootfs on target. Sadly
> latest core-image-minimal only has 1.5 Mb free space if
> ssh-server-dropbear is added to it. Thus by default, core-image-minimal
> is now failing oeqa rutime df.py test.
>
> Fix this by increasing core-image-minimal rootfs size by 5Mb if
> testimage.bbclass is used which implies adding either
> ssh-server-dropbear or ssh-server-openssh to the target rootfs.
>
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/recipes-core/images/core-image-minimal.bb | 3 +++
>  1 file changed, 3 insertions(+)
>
> v2: no changes
>
> v1: https://lists.openembedded.org/g/openembedded-core/message/186423
>
> diff --git a/meta/recipes-core/images/core-image-minimal.bb b/meta/recipes-core/images/core-image-minimal.bb
> index 84343adcd8..c04612bc15 100644
> --- a/meta/recipes-core/images/core-image-minimal.bb
> +++ b/meta/recipes-core/images/core-image-minimal.bb
> @@ -10,3 +10,6 @@ inherit core-image
>
>  IMAGE_ROOTFS_SIZE ?= "8192"
>  IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("DISTRO_FEATURES", "systemd", " + 4096", "", d)}"
> +
> +# a bit more space needed for sshd to actually run the tests
> +IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("IMAGE_CLASSES", "testimage", " + 5120", "", d)}"
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186549): https://lists.openembedded.org/g/openembedded-core/message/186549
> Mute This Topic: https://lists.openembedded.org/mt/100910038/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 1/9] core-image-minimal: increase extra space to pass df.py oeqa runtime test

[Mikko Rapeli]

Hi,

On Wed, Aug 23, 2023 at 08:44:33PM +0200, Alexander Kanavin wrote:
> Either you need to use a different image in testing, or add the needed
> pieces to this image in the same commit, explaining why.
> 
> Adding extra space because you have a local, custom configuration
> where it is required is not correct.

Sorry I thought upstream would be testing core-image-minimal as well.
If that's not the case, then just drop this. Some other images need
to be tested instead, but I don't know which.

Cheers,

-Mikko

> Alex
> 
> On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> >
> > testimage.bbclass requires that ssh communication with target works
> > and then it runs oeqa runtime tests with that. Tests like df.py
> > check that there is more than 5Mb free space on the rootfs on target. Sadly
> > latest core-image-minimal only has 1.5 Mb free space if
> > ssh-server-dropbear is added to it. Thus by default, core-image-minimal
> > is now failing oeqa rutime df.py test.
> >
> > Fix this by increasing core-image-minimal rootfs size by 5Mb if
> > testimage.bbclass is used which implies adding either
> > ssh-server-dropbear or ssh-server-openssh to the target rootfs.
> >
> > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > ---
> >  meta/recipes-core/images/core-image-minimal.bb | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > v2: no changes
> >
> > v1: https://lists.openembedded.org/g/openembedded-core/message/186423
> >
> > diff --git a/meta/recipes-core/images/core-image-minimal.bb b/meta/recipes-core/images/core-image-minimal.bb
> > index 84343adcd8..c04612bc15 100644
> > --- a/meta/recipes-core/images/core-image-minimal.bb
> > +++ b/meta/recipes-core/images/core-image-minimal.bb
> > @@ -10,3 +10,6 @@ inherit core-image
> >
> >  IMAGE_ROOTFS_SIZE ?= "8192"
> >  IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("DISTRO_FEATURES", "systemd", " + 4096", "", d)}"
> > +
> > +# a bit more space needed for sshd to actually run the tests
> > +IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("IMAGE_CLASSES", "testimage", " + 5120", "", d)}"
> > --
> > 2.34.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#186549): https://lists.openembedded.org/g/openembedded-core/message/186549
> > Mute This Topic: https://lists.openembedded.org/mt/100910038/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >