[PATCH] test-manual: update runtime-testing Exporting Tests section

[PATCH] test-manual: update runtime-testing Exporting Tests section

[Barne Carstensen]

Signed-off-by: Barne Carstensen <barne.carstensen@danfoss.com>
---
 documentation/test-manual/runtime-testing.rst | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/documentation/test-manual/runtime-testing.rst b/documentation/test-manual/runtime-testing.rst
index 557e0530b..909448ab2 100644
--- a/documentation/test-manual/runtime-testing.rst
+++ b/documentation/test-manual/runtime-testing.rst
@@ -403,7 +403,7 @@ defined in :term:`TEST_SUITES`.
 If your image is already built, make sure the following are set in your
 ``local.conf`` file::
 
-   INHERIT += "testexport"
+   IMAGE_CLASSES += "testexport"
    TEST_TARGET_IP = "IP-address-for-the-test-target"
    TEST_SERVER_IP = "IP-address-for-the-test-server"
 
@@ -413,18 +413,23 @@ following BitBake command form::
    $ bitbake image -c testexport
 
 Exporting the tests places them in the :term:`Build Directory` in
-``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
+``tmp/testimage/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
 variable.
 
 You can now run the tests outside of the build environment::
 
-   $ cd tmp/testexport/image
-   $ ./runexported.py testdata.json
+   $ cd tmp/testimage/image
+   $ ./oe-test runtime
+
+.. note::
+
+   You might need to run the image under QEMU or deploy it to your
+   hardware before you can run the tests.
 
 Here is a complete example that shows IP addresses and uses the
 ``core-image-sato`` image::
 
-   INHERIT += "testexport"
+   IMAGE_CLASSES += "testexport"
    TEST_TARGET_IP = "192.168.7.2"
    TEST_SERVER_IP = "192.168.7.1"
 
@@ -435,8 +440,8 @@ Use BitBake to export the tests::
 Run the tests outside of
 the build environment using the following::
 
-   $ cd tmp/testexport/core-image-sato
-   $ ./runexported.py testdata.json
+   $ cd tmp/testimage/core-image-sato
+   $ ./oe-test runtime
 
 Writing New Tests
 =================

Re: [docs] [PATCH] test-manual: update runtime-testing Exporting Tests section

[Antonin Godard]

Hi,

On Tue Sep 16, 2025 at 1:01 PM CEST, Barne Carstensen via lists.yoctoproject.org wrote:
> Signed-off-by: Barne Carstensen <barne.carstensen@danfoss.com>
> ---
>  documentation/test-manual/runtime-testing.rst | 19 ++++++++++++-------
>  1 file changed, 12 insertions(+), 7 deletions(-)
>
> diff --git a/documentation/test-manual/runtime-testing.rst b/documentation/test-manual/runtime-testing.rst
> index 557e0530b..909448ab2 100644
> --- a/documentation/test-manual/runtime-testing.rst
> +++ b/documentation/test-manual/runtime-testing.rst
> @@ -403,7 +403,7 @@ defined in :term:`TEST_SUITES`.
>  If your image is already built, make sure the following are set in your
>  ``local.conf`` file::
>  
> -   INHERIT += "testexport"
> +   IMAGE_CLASSES += "testexport"
>     TEST_TARGET_IP = "IP-address-for-the-test-target"
>     TEST_SERVER_IP = "IP-address-for-the-test-server"
>  
> @@ -413,18 +413,23 @@ following BitBake command form::
>     $ bitbake image -c testexport
>  
>  Exporting the tests places them in the :term:`Build Directory` in
> -``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
> +``tmp/testimage/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
>  variable.
>  
>  You can now run the tests outside of the build environment::
>  
> -   $ cd tmp/testexport/image
> -   $ ./runexported.py testdata.json
> +   $ cd tmp/testimage/image
> +   $ ./oe-test runtime
> +
> +.. note::
> +
> +   You might need to run the image under QEMU or deploy it to your
> +   hardware before you can run the tests.
>  
>  Here is a complete example that shows IP addresses and uses the
>  ``core-image-sato`` image::
>  
> -   INHERIT += "testexport"
> +   IMAGE_CLASSES += "testexport"
>     TEST_TARGET_IP = "192.168.7.2"
>     TEST_SERVER_IP = "192.168.7.1"
>  
> @@ -435,8 +440,8 @@ Use BitBake to export the tests::
>  Run the tests outside of
>  the build environment using the following::
>  
> -   $ cd tmp/testexport/core-image-sato
> -   $ ./runexported.py testdata.json
> +   $ cd tmp/testimage/core-image-sato
> +   $ ./oe-test runtime
>  
>  Writing New Tests
>  =================

I've tested these changes and it looks like this applies since ed4238487c81
("testexport: Fix to work as an image class"). So can be backported to walnascar
and scarthgap.

I will include this in the commit message, as I think the patch itself is
correct. Next time, can you please include a commit message body describing the
change and ideally referencing the commit that implied this change?

I've tested this on a core-image-minimal/qemuarm64 build and was able
to run some tests with your modifications. So:

Tested-by: Antonin Godard <antonin.godard@bootlin.com>
Reviewed-by: Antonin Godard <antonin.godard@bootlin.com>

I just had issues, I wonder if you also encountered them:

    File ".../tmp/testimage/core-image-minimal/meta/lib/oeqa/runtime/cases/storage.py", line 129, in setUpClass
      self.test_dir = os.path.join(self.mount_point, "oeqa")
                      ^^
  NameError: name 'os' is not defined

I have the feeling that this works when using `bitbake -c testimage <image>`
directly, because in bitbake the os module is available by default, and OE-Core
test were written with this in mind?

But in an exported environment, you obviously don't have this module imported by
default, like any other Python program (even after having sourced the
environment). In that case this would maybe mean that:

- we should import os in these files.
- or we consider that these tests were made for OECore testing with testimage
  only.

Thanks!
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

RE: [docs] [PATCH] test-manual: update runtime-testing Exporting Tests section

[Barne Carstensen]

Hello,

Thank you for reviewing and testing. Sure, I will add more information to the commit message next time. With the "commit that implied this change" you mean the commit in oe-core which switched from runexported.py to oe-test? I did not notice the error, btw.

Barne


Classified as Business
-----Original Message-----
From: Antonin Godard <antonin.godard@bootlin.com>
Sent: Mittwoch, 17. September 2025 16:04
To: Barne Carstensen <barne.carstensen@danfoss.com>; docs@lists.yoctoproject.org
Subject: Re: [docs] [PATCH] test-manual: update runtime-testing Exporting Tests section

[You don't often get email from antonin.godard@bootlin.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Hi,

On Tue Sep 16, 2025 at 1:01 PM CEST, Barne Carstensen via lists.yoctoproject.org wrote:
> Signed-off-by: Barne Carstensen <barne.carstensen@danfoss.com>
> ---
>  documentation/test-manual/runtime-testing.rst | 19
> ++++++++++++-------
>  1 file changed, 12 insertions(+), 7 deletions(-)
>
> diff --git a/documentation/test-manual/runtime-testing.rst
> b/documentation/test-manual/runtime-testing.rst
> index 557e0530b..909448ab2 100644
> --- a/documentation/test-manual/runtime-testing.rst
> +++ b/documentation/test-manual/runtime-testing.rst
> @@ -403,7 +403,7 @@ defined in :term:`TEST_SUITES`.
>  If your image is already built, make sure the following are set in
> your  ``local.conf`` file::
>
> -   INHERIT += "testexport"
> +   IMAGE_CLASSES += "testexport"
>     TEST_TARGET_IP = "IP-address-for-the-test-target"
>     TEST_SERVER_IP = "IP-address-for-the-test-server"
>
> @@ -413,18 +413,23 @@ following BitBake command form::
>     $ bitbake image -c testexport
>
>  Exporting the tests places them in the :term:`Build Directory` in
> -``tmp/testexport/``\ image, which is controlled by the
> :term:`TEST_EXPORT_DIR`
> +``tmp/testimage/``\ image, which is controlled by the
> +:term:`TEST_EXPORT_DIR`
>  variable.
>
>  You can now run the tests outside of the build environment::
>
> -   $ cd tmp/testexport/image
> -   $ ./runexported.py testdata.json
> +   $ cd tmp/testimage/image
> +   $ ./oe-test runtime
> +
> +.. note::
> +
> +   You might need to run the image under QEMU or deploy it to your
> +   hardware before you can run the tests.
>
>  Here is a complete example that shows IP addresses and uses the
> ``core-image-sato`` image::
>
> -   INHERIT += "testexport"
> +   IMAGE_CLASSES += "testexport"
>     TEST_TARGET_IP = "192.168.7.2"
>     TEST_SERVER_IP = "192.168.7.1"
>
> @@ -435,8 +440,8 @@ Use BitBake to export the tests::
>  Run the tests outside of
>  the build environment using the following::
>
> -   $ cd tmp/testexport/core-image-sato
> -   $ ./runexported.py testdata.json
> +   $ cd tmp/testimage/core-image-sato
> +   $ ./oe-test runtime
>
>  Writing New Tests
>  =================

I've tested these changes and it looks like this applies since ed4238487c81
("testexport: Fix to work as an image class"). So can be backported to walnascar and scarthgap.

I will include this in the commit message, as I think the patch itself is correct. Next time, can you please include a commit message body describing the change and ideally referencing the commit that implied this change?

I've tested this on a core-image-minimal/qemuarm64 build and was able to run some tests with your modifications. So:

Tested-by: Antonin Godard <antonin.godard@bootlin.com>
Reviewed-by: Antonin Godard <antonin.godard@bootlin.com>

I just had issues, I wonder if you also encountered them:

    File ".../tmp/testimage/core-image-minimal/meta/lib/oeqa/runtime/cases/storage.py", line 129, in setUpClass
      self.test_dir = os.path.join(self.mount_point, "oeqa")
                      ^^
  NameError: name 'os' is not defined

I have the feeling that this works when using `bitbake -c testimage <image>` directly, because in bitbake the os module is available by default, and OE-Core test were written with this in mind?

But in an exported environment, you obviously don't have this module imported by default, like any other Python program (even after having sourced the environment). In that case this would maybe mean that:

- we should import os in these files.
- or we consider that these tests were made for OECore testing with testimage
  only.

Thanks!
Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com/

Re: [PATCH] test-manual: update runtime-testing Exporting Tests section

[Antonin Godard]

On Tue, 16 Sep 2025 11:01:22 +0000, Barne Carstensen wrote:
> 


Applied, thanks!

[1/1] test-manual: update runtime-testing Exporting Tests section
      commit: 2caa8e581feaf3640bea68108f9a02583b17b21b

Best regards,
-- 
Antonin Godard <antonin.godard@bootlin.com>