[Buildroot] [PATCH 0/3] Fix 3 Python test cases

[Buildroot] [PATCH 0/3] Fix 3 Python test cases

[Thomas Petazzoni via buildroot]

Hello,

This small patch series fixes 3 Python test cases. They fail because
we use an initramfs in all Python tests (and in many other tests). The
issue of initramfs is that if the initramfs filesystem grows too
large, the kernel fails to decompress it, and simply continues booting
with a partially uncompressed initramfs: some files are simply
missing.

In this patch series, we fix just the 3 failing test cases, for the
sake of having a reasonably minimal solution to apply in master. We
switch those test cases to use an initramfs to using an ext2 image
that is exposed by Qemu as a regular hard-drive.

Long term, we probably want to switch all tests to using this
strategy, so that we don't face a weird limit caused by
initramfs. That being said, there is no silver bullet: even if we
switch to ext2, we will have to define, for each test, the size of the
ext2 filesystem image. So there will still be some sort of arbitrary
limit that will have to be tweaked from time to time as a given test
grows in size (more dependencies, dependencies growing, etc.).

Best regards,

Thomas Petazzoni

Thomas Petazzoni (3):
  support/testing/tests/package/test_python_boto3: use ext2 instead of
    cpio
  support/testing/tests/package/test_python_botocore: use ext2 instead
    of cpio
  support/testing/tests/package/test_python_s3transfer: use ext2 instead
    of cpio

 support/testing/tests/package/test_python_boto3.py   | 12 ++++++++++++
 .../testing/tests/package/test_python_botocore.py    | 12 ++++++++++++
 .../testing/tests/package/test_python_s3transfer.py  | 12 ++++++++++++
 3 files changed, 36 insertions(+)

-- 
2.37.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 1/3] support/testing/tests/package/test_python_boto3: use ext2 instead of cpio

[Thomas Petazzoni via buildroot]

The CPIO filesystem generated by the test_python_boto3 test is too
large, and doesn't fit as an initramfs in the 256MB of RAM available
in the versatilepb machine. This causes a "Initramfs unpacking failed:
write error" when booting, and many files being missing from the root
filesystem, ultimately causing the test to fail.

It would make sense to switch all test cases to use ext2 + a
hard-drive, but for now, let's fix the few test cases that are causing
problems.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/2884635041

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/tests/package/test_python_boto3.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/support/testing/tests/package/test_python_boto3.py b/support/testing/tests/package/test_python_boto3.py
index 756495fec9..7f24399304 100644
--- a/support/testing/tests/package/test_python_boto3.py
+++ b/support/testing/tests/package/test_python_boto3.py
@@ -1,4 +1,5 @@
 from tests.package.test_python import TestPythonPackageBase
+import os
 
 
 class TestPythonPy3Boto3(TestPythonPackageBase):
@@ -7,6 +8,17 @@ class TestPythonPy3Boto3(TestPythonPackageBase):
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_BOTO3=y
+        BR2_TARGET_ROOTFS_EXT2=y
+        BR2_TARGET_ROOTFS_EXT2_SIZE="120M"
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
     sample_scripts = ["tests/package/sample_python_boto3.py"]
     timeout = 10
+
+    def login(self):
+        ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file],
+                           kernel_cmdline=["rootwait", "root=/dev/sda"])
+        self.emulator.login()
-- 
2.37.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/3] support/testing/tests/package/test_python_boto3: use ext2 instead of cpio

[Yann E. MORIN]

Thomas, All,

On 2022-08-15 23:48 +0200, Thomas Petazzoni via buildroot spake thusly:
> The CPIO filesystem generated by the test_python_boto3 test is too
> large, and doesn't fit as an initramfs in the 256MB of RAM available
> in the versatilepb machine. This causes a "Initramfs unpacking failed:
> write error" when booting, and many files being missing from the root
> filesystem, ultimately causing the test to fail.
> 
> It would make sense to switch all test cases to use ext2 + a
> hard-drive, but for now, let's fix the few test cases that are causing
> problems.
> 
> Fixes:
> 
>   https://gitlab.com/buildroot.org/buildroot/-/jobs/2884635041
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/testing/tests/package/test_python_boto3.py | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/support/testing/tests/package/test_python_boto3.py b/support/testing/tests/package/test_python_boto3.py
> index 756495fec9..7f24399304 100644
> --- a/support/testing/tests/package/test_python_boto3.py
> +++ b/support/testing/tests/package/test_python_boto3.py
> @@ -1,4 +1,5 @@
>  from tests.package.test_python import TestPythonPackageBase
> +import os
>  
>  
>  class TestPythonPy3Boto3(TestPythonPackageBase):
> @@ -7,6 +8,17 @@ class TestPythonPy3Boto3(TestPythonPackageBase):
>          """
>          BR2_PACKAGE_PYTHON3=y
>          BR2_PACKAGE_PYTHON_BOTO3=y
> +        BR2_TARGET_ROOTFS_EXT2=y
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="120M"
> +        # BR2_TARGET_ROOTFS_TAR is not set

BR2_TARGET_ROOTFS_TAR is already unset in TestPythonPackageBase.config,
so disabling it here is superfluous.

But this will however still build the cpio archive, because it is
enabled in TestPythonPackageBase.config.

We can disable it with:

    class TestPythonPy3Boto3(TestPythonPackageBase):
        config = TestPythonPackageBase.config.replace('BR2_TARGET_ROOTFS_CPIO=y\n', '') + \
            """
            blablabla
            """

If you believe that it is acceptable, I can fix when applying. Or we can
continue to uselessly build the cpio archive, which does not take very
long to assemble...

Regards,
Yann E. MORIN.

>          """
>      sample_scripts = ["tests/package/sample_python_boto3.py"]
>      timeout = 10
> +
> +    def login(self):
> +        ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2")
> +        self.emulator.boot(arch="armv5",
> +                           kernel="builtin",
> +                           options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file],
> +                           kernel_cmdline=["rootwait", "root=/dev/sda"])
> +        self.emulator.login()
> -- 
> 2.37.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/3] support/testing/tests/package/test_python_boto3: use ext2 instead of cpio

[Thomas Petazzoni via buildroot]

Hello,

On Tue, 16 Aug 2022 18:26:26 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> >  class TestPythonPy3Boto3(TestPythonPackageBase):
> > @@ -7,6 +8,17 @@ class TestPythonPy3Boto3(TestPythonPackageBase):
> >          """
> >          BR2_PACKAGE_PYTHON3=y
> >          BR2_PACKAGE_PYTHON_BOTO3=y
> > +        BR2_TARGET_ROOTFS_EXT2=y
> > +        BR2_TARGET_ROOTFS_EXT2_SIZE="120M"
> > +        # BR2_TARGET_ROOTFS_TAR is not set  
> 
> BR2_TARGET_ROOTFS_TAR is already unset in TestPythonPackageBase.config,
> so disabling it here is superfluous.

Ah indeed.

> But this will however still build the cpio archive, because it is
> enabled in TestPythonPackageBase.config.
> 
> We can disable it with:
> 
>     class TestPythonPy3Boto3(TestPythonPackageBase):
>         config = TestPythonPackageBase.config.replace('BR2_TARGET_ROOTFS_CPIO=y\n', '') + \
>             """
>             blablabla
>             """
> 
> If you believe that it is acceptable, I can fix when applying. Or we can
> continue to uselessly build the cpio archive, which does not take very
> long to assemble...

I think we can continue to uselessly build the cpio archive. As I said,
my long term proposal would rather be to switch all tests to use ext2 +
a hard drive. We still need to define an image size, but at least we
can a hard and clear error if the filesystem no longer fits rather than
these weird errors caused by initramfs being partially uncompressed.

Thanks for the review!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/3] support/testing/tests/package/test_python_botocore: use ext2 instead of cpio

[Thomas Petazzoni via buildroot]

The CPIO filesystem generated by the test_python_botocore test is too
large, and doesn't fit as an initramfs in the 256MB of RAM available
in the versatilepb machine. This causes a "Initramfs unpacking failed:
write error" when booting, and many files being missing from the root
filesystem, ultimately causing the test to fail.

It would make sense to switch all test cases to use ext2 + a
hard-drive, but for now, let's fix the few test cases that are causing
problems.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/2884635042

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 .../testing/tests/package/test_python_botocore.py    | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/support/testing/tests/package/test_python_botocore.py b/support/testing/tests/package/test_python_botocore.py
index 6336c5658f..d198c3e682 100644
--- a/support/testing/tests/package/test_python_botocore.py
+++ b/support/testing/tests/package/test_python_botocore.py
@@ -1,4 +1,5 @@
 from tests.package.test_python import TestPythonPackageBase
+import os
 
 
 class TestPythonPy3Botocore(TestPythonPackageBase):
@@ -7,6 +8,17 @@ class TestPythonPy3Botocore(TestPythonPackageBase):
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_BOTOCORE=y
+        BR2_TARGET_ROOTFS_EXT2=y
+        BR2_TARGET_ROOTFS_EXT2_SIZE="120M"
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
     sample_scripts = ["tests/package/sample_python_botocore.py"]
     timeout = 10
+
+    def login(self):
+        ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file],
+                           kernel_cmdline=["rootwait", "root=/dev/sda"])
+        self.emulator.login()
-- 
2.37.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 3/3] support/testing/tests/package/test_python_s3transfer: use ext2 instead of cpio

[Thomas Petazzoni via buildroot]

The CPIO filesystem generated by the test_python_s3transfer test is
too large, and doesn't fit as an initramfs in the 256MB of RAM
available in the versatilepb machine. This causes a "Initramfs
unpacking failed: write error" when booting, and many files being
missing from the root filesystem, ultimately causing the test to fail.

It would make sense to switch all test cases to use ext2 + a
hard-drive, but for now, let's fix the few test cases that are causing
problems.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/2884635126

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 .../testing/tests/package/test_python_s3transfer.py  | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/support/testing/tests/package/test_python_s3transfer.py b/support/testing/tests/package/test_python_s3transfer.py
index dc669444ce..cc03ae513a 100644
--- a/support/testing/tests/package/test_python_s3transfer.py
+++ b/support/testing/tests/package/test_python_s3transfer.py
@@ -1,4 +1,5 @@
 from tests.package.test_python import TestPythonPackageBase
+import os
 
 
 class TestPythonPy3S3transfer(TestPythonPackageBase):
@@ -7,6 +8,17 @@ class TestPythonPy3S3transfer(TestPythonPackageBase):
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_S3TRANSFER=y
+        BR2_TARGET_ROOTFS_EXT2=y
+        BR2_TARGET_ROOTFS_EXT2_SIZE="120M"
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
     sample_scripts = ["tests/package/sample_python_s3transfer.py"]
     timeout = 10
+
+    def login(self):
+        ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file],
+                           kernel_cmdline=["rootwait", "root=/dev/sda"])
+        self.emulator.login()
-- 
2.37.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 0/3] Fix 3 Python test cases

[Yann E. MORIN]

Thomas, All,

On 2022-08-15 23:48 +0200, Thomas Petazzoni via buildroot spake thusly:
[--SNIP--]
> Thomas Petazzoni (3):
>   support/testing/tests/package/test_python_boto3: use ext2 instead of
>     cpio
>   support/testing/tests/package/test_python_botocore: use ext2 instead
>     of cpio
>   support/testing/tests/package/test_python_s3transfer: use ext2 instead
>     of cpio

All threeour applied to master, after removing the superfluous line
disabling rootfs-tar.

Regards,
Yann E. MORIN.

>  support/testing/tests/package/test_python_boto3.py   | 12 ++++++++++++
>  .../testing/tests/package/test_python_botocore.py    | 12 ++++++++++++
>  .../testing/tests/package/test_python_s3transfer.py  | 12 ++++++++++++
>  3 files changed, 36 insertions(+)
> 
> -- 
> 2.37.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot