* [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore @ 2022-06-15 17:12 Raphaël Mélotte 2022-06-15 17:12 ` [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 Raphaël Mélotte 2022-07-20 21:50 ` [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Thomas Petazzoni via buildroot 0 siblings, 2 replies; 5+ messages in thread From: Raphaël Mélotte @ 2022-06-15 17:12 UTC (permalink / raw) To: buildroot; +Cc: Raphaël Mélotte, Ricardo Martincoski With new python-botocore versions, the size of the package will increase and the 256MiB memory limit from the default versatilepb machine will no longer be sufficient to do runtime tests. Switch to using an aarch64 machine, and use 512MiB memory. The configuration and login method are taken from TestOpenJdk. While at it, add the python-botocore test files to the DEVELOPERS file. Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be> --- DEVELOPERS | 2 ++ .../tests/package/test_python_botocore.py | 24 ++++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/DEVELOPERS b/DEVELOPERS index 86755bfe18..c5ad5ce29b 100644 --- a/DEVELOPERS +++ b/DEVELOPERS @@ -2432,7 +2432,9 @@ F: package/python-jmespath/ F: package/python-pymupdf/ F: package/python-rsa/ F: package/python-s3transfer/ +F: support/testing/tests/package/sample_python_botocore.py F: support/testing/tests/package/sample_python_rsa.py +F: support/testing/tests/package/test_python_botocore.py F: support/testing/tests/package/test_python_rsa.py N: Refik Tuzakli <tuzakli.refik@gmail.com> diff --git a/support/testing/tests/package/test_python_botocore.py b/support/testing/tests/package/test_python_botocore.py index 6336c5658f..535ad5496d 100644 --- a/support/testing/tests/package/test_python_botocore.py +++ b/support/testing/tests/package/test_python_botocore.py @@ -1,12 +1,34 @@ +import os from tests.package.test_python import TestPythonPackageBase class TestPythonPy3Botocore(TestPythonPackageBase): __test__ = True - config = TestPythonPackageBase.config + \ + + config = \ """ + BR2_aarch64=y + BR2_TOOLCHAIN_EXTERNAL=y + BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0" + BR2_LINUX_KERNEL=y + BR2_LINUX_KERNEL_CUSTOM_VERSION=y + BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.10.34" + BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y + BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux.config" + BR2_TARGET_ROOTFS_CPIO=y + BR2_TARGET_ROOTFS_CPIO_GZIP=y BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_BOTOCORE=y """ + sample_scripts = ["tests/package/sample_python_botocore.py"] timeout = 10 + + def login(self): + img = os.path.join(self.builddir, "images", "rootfs.cpio.gz") + kern = os.path.join(self.builddir, "images", "Image") + self.emulator.boot(arch="aarch64", + kernel=kern, + kernel_cmdline=["console=ttyAMA0"], + options=["-M", "virt", "-cpu", "cortex-a57", "-m", "512M", "-initrd", img]) + self.emulator.login() -- 2.35.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 2022-06-15 17:12 [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Raphaël Mélotte @ 2022-06-15 17:12 ` Raphaël Mélotte 2022-07-23 16:56 ` Arnout Vandecappelle 2022-07-20 21:50 ` [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Thomas Petazzoni via buildroot 1 sibling, 1 reply; 5+ messages in thread From: Raphaël Mélotte @ 2022-06-15 17:12 UTC (permalink / raw) To: buildroot; +Cc: Raphaël Mélotte, Asaf Kahlon Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be> --- package/python-botocore/python-botocore.hash | 2 +- package/python-botocore/python-botocore.mk | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/package/python-botocore/python-botocore.hash b/package/python-botocore/python-botocore.hash index 9d5f706fef..bb806fe251 100644 --- a/package/python-botocore/python-botocore.hash +++ b/package/python-botocore/python-botocore.hash @@ -1,4 +1,4 @@ # sha256 from https://pypi.org/pypi/botocore/json -sha256 011360e79a4b843aa6591573cfa61e8eddc99b91adab1dfdb9a2b7f2c8511193 botocore-1.22.2.tar.gz +sha256 db6667b8dfd175d16187653942cd91dd1f0cf36adc0ea9d7a0805ba4d2a3321f botocore-1.27.8.tar.gz # Locally computed sha256 checksums sha256 0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594 LICENSE.txt diff --git a/package/python-botocore/python-botocore.mk b/package/python-botocore/python-botocore.mk index 1628e05562..459eefeab1 100644 --- a/package/python-botocore/python-botocore.mk +++ b/package/python-botocore/python-botocore.mk @@ -4,9 +4,9 @@ # ################################################################################ -PYTHON_BOTOCORE_VERSION = 1.22.2 +PYTHON_BOTOCORE_VERSION = 1.27.8 PYTHON_BOTOCORE_SOURCE = botocore-$(PYTHON_BOTOCORE_VERSION).tar.gz -PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/0d/95/f2e74e4a91994d9ebe45746f75a8c5cafd723a88270a9772d8d66e1a8ba1 +PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/d1/b0/ea92703bbbf82353d571636fe6697b162c3881576986d6474119d03fce53 PYTHON_BOTOCORE_SETUP_TYPE = setuptools PYTHON_BOTOCORE_LICENSE = Apache-2.0 PYTHON_BOTOCORE_LICENSE_FILES = LICENSE.txt -- 2.35.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 2022-06-15 17:12 ` [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 Raphaël Mélotte @ 2022-07-23 16:56 ` Arnout Vandecappelle 0 siblings, 0 replies; 5+ messages in thread From: Arnout Vandecappelle @ 2022-07-23 16:56 UTC (permalink / raw) To: Raphaël Mélotte, buildroot; +Cc: Asaf Kahlon On 15/06/2022 19:12, Raphaël Mélotte wrote: > Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be> Applied to master, thanks. Regards, Arnout > --- > package/python-botocore/python-botocore.hash | 2 +- > package/python-botocore/python-botocore.mk | 4 ++-- > 2 files changed, 3 insertions(+), 3 deletions(-) > > diff --git a/package/python-botocore/python-botocore.hash b/package/python-botocore/python-botocore.hash > index 9d5f706fef..bb806fe251 100644 > --- a/package/python-botocore/python-botocore.hash > +++ b/package/python-botocore/python-botocore.hash > @@ -1,4 +1,4 @@ > # sha256 from https://pypi.org/pypi/botocore/json > -sha256 011360e79a4b843aa6591573cfa61e8eddc99b91adab1dfdb9a2b7f2c8511193 botocore-1.22.2.tar.gz > +sha256 db6667b8dfd175d16187653942cd91dd1f0cf36adc0ea9d7a0805ba4d2a3321f botocore-1.27.8.tar.gz > # Locally computed sha256 checksums > sha256 0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594 LICENSE.txt > diff --git a/package/python-botocore/python-botocore.mk b/package/python-botocore/python-botocore.mk > index 1628e05562..459eefeab1 100644 > --- a/package/python-botocore/python-botocore.mk > +++ b/package/python-botocore/python-botocore.mk > @@ -4,9 +4,9 @@ > # > ################################################################################ > > -PYTHON_BOTOCORE_VERSION = 1.22.2 > +PYTHON_BOTOCORE_VERSION = 1.27.8 > PYTHON_BOTOCORE_SOURCE = botocore-$(PYTHON_BOTOCORE_VERSION).tar.gz > -PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/0d/95/f2e74e4a91994d9ebe45746f75a8c5cafd723a88270a9772d8d66e1a8ba1 > +PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/d1/b0/ea92703bbbf82353d571636fe6697b162c3881576986d6474119d03fce53 > PYTHON_BOTOCORE_SETUP_TYPE = setuptools > PYTHON_BOTOCORE_LICENSE = Apache-2.0 > PYTHON_BOTOCORE_LICENSE_FILES = LICENSE.txt _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore 2022-06-15 17:12 [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Raphaël Mélotte 2022-06-15 17:12 ` [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 Raphaël Mélotte @ 2022-07-20 21:50 ` Thomas Petazzoni via buildroot 2022-08-05 7:15 ` Raphaël Mélotte 1 sibling, 1 reply; 5+ messages in thread From: Thomas Petazzoni via buildroot @ 2022-07-20 21:50 UTC (permalink / raw) To: Raphaël Mélotte; +Cc: Ricardo Martincoski, buildroot Hello Raphaël, On Wed, 15 Jun 2022 19:12:15 +0200 Raphaël Mélotte <raphael.melotte@mind.be> wrote: > With new python-botocore versions, the size of the package will > increase and the 256MiB memory limit from the default versatilepb > machine will no longer be sufficient to do runtime tests. > > Switch to using an aarch64 machine, and use 512MiB memory. > > The configuration and login method are taken from TestOpenJdk. > > While at it, add the python-botocore test files to the DEVELOPERS > file. > > Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be> Thanks a lot for the proposal. I understand the need for more memory, but I'm wondering if building a full aarch64 kernel just for this is really the right solution. What about instead trying to pass a custom option to Qemu? support/testing/infra/emulator.py goes like this: def boot(self, arch, kernel=None, kernel_cmdline=None, options=None): if arch in ["armv7", "armv5"]: qemu_arch = "arm" else: qemu_arch = arch qemu_cmd = ["qemu-system-{}".format(qemu_arch), "-serial", "stdio", "-display", "none", "-m", "256"] if options: qemu_cmd += options So if we passe options=["-m", "512"], this should override the default -m 256. Then the question is how can the Python test infrastructure pass this down to self.emulator.boot(). Something like this perhaps: diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py index f29aff6624..bd0b21f193 100644 --- a/support/testing/tests/package/test_python.py +++ b/support/testing/tests/package/test_python.py @@ -11,11 +11,14 @@ class TestPythonBase(infra.basetest.BRTest): """ interpreter = "python" - def login(self): + def login(self, qemu_extra_opts=None): cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio") + qemu_opts = ["-initrd", cpio_file] + if qemu_extra_opts: + qemu_opts += qemu_extra_opts self.emulator.boot(arch="armv5", kernel="builtin", - options=["-initrd", cpio_file]) + options=qemu_opts) self.emulator.login() def version_test(self, version, timeout=-1): @@ -80,6 +83,7 @@ class TestPythonPackageBase(TestPythonBase): "{sample_scripts}") sample_scripts = None timeout = -1 + qemu_extra_opts = None def __init__(self, names): """Add the scripts to the target in build time.""" @@ -102,6 +106,6 @@ class TestPythonPackageBase(TestPythonBase): self.assertRunOk(cmd, timeout=self.timeout) def test_run(self): - self.login() + self.login(qemu_extra_opts=self.qemu_extra_opts) self.check_sample_scripts_exist() self.run_sample_scripts() diff --git a/support/testing/tests/package/test_python_botocore.py b/support/testing/tests/package/test_python_botocore.py index 6336c5658f..a7c35a6ed6 100644 --- a/support/testing/tests/package/test_python_botocore.py +++ b/support/testing/tests/package/test_python_botocore.py @@ -10,3 +10,4 @@ class TestPythonPy3Botocore(TestPythonPackageBase): """ sample_scripts = ["tests/package/sample_python_botocore.py"] timeout = 10 + qemu_extra_opts = ["-m", "512"] (Completely untested, and also I'm not sure if it's the most Pythonic way to do that, I'm not a Python developer). What do you think? Best regards, Thomas -- Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore 2022-07-20 21:50 ` [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Thomas Petazzoni via buildroot @ 2022-08-05 7:15 ` Raphaël Mélotte 0 siblings, 0 replies; 5+ messages in thread From: Raphaël Mélotte @ 2022-08-05 7:15 UTC (permalink / raw) To: Thomas Petazzoni; +Cc: Ricardo Martincoski, buildroot Hello Thomas, On 7/20/22 23:50, Thomas Petazzoni wrote: > > Thanks a lot for the proposal. Thanks for looking at it! > I understand the need for more memory, > but I'm wondering if building a full aarch64 kernel just for this is > really the right solution. > > What about instead trying to pass a custom option to Qemu? > > support/testing/infra/emulator.py goes like this: > > def boot(self, arch, kernel=None, kernel_cmdline=None, options=None): > if arch in ["armv7", "armv5"]: > qemu_arch = "arm" > else: > qemu_arch = arch > > qemu_cmd = ["qemu-system-{}".format(qemu_arch), > "-serial", "stdio", > "-display", "none", > "-m", "256"] > > if options: > qemu_cmd += options > > So if we passe options=["-m", "512"], this should override the default > -m 256. That's what I tried at first, but the problem is that qemu refuses to use more than 256MiB for the versatilepb machine (I should have made that clearer in the commit message, sorry). This is the place in qemu that checks that no more than 256MiB is used: === if (machine->ram_size > 0x10000000) { /* Device starting at address 0x10000000, * and memory cannot overlap with devices. * Refuse to run rather than behaving very confusingly. */ error_report("versatilepb: memory size must not exceed 256MB"); exit(1); } ==== So I had no another choice than to use a different one. I could use a different board than "aarch64-virt" if needed, I only chose it because it was already used elsewhere. What do you think? Kind regards, Raphaël _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-08-05 7:15 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-06-15 17:12 [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Raphaël Mélotte 2022-06-15 17:12 ` [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 Raphaël Mélotte 2022-07-23 16:56 ` Arnout Vandecappelle 2022-07-20 21:50 ` [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Thomas Petazzoni via buildroot 2022-08-05 7:15 ` Raphaël Mélotte
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox