[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Peter Korsgaard]

Build for x86-64 as public containers in general are only available for
x86-64.  Docker needs a number of kernel options enabled, so use a custom
kernel config based on the qemu one.

Docker needs entropy at startup, so enable the virtio-rng-pci device to
expose entropy to the guest.  The default RAM amount (128M) is not enough to
run docker / docker-compose, so bump to 512MB.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 .gitlab-ci.yml                                     |   1 +
 support/testing/conf/docker-compose-kernel.config  | 103 +++++++++++++++++++++
 support/testing/conf/docker-compose.yml            |   4 +
 .../testing/tests/package/test_docker_compose.py   |  70 ++++++++++++++
 4 files changed, 178 insertions(+)
 create mode 100644 support/testing/conf/docker-compose-kernel.config
 create mode 100644 support/testing/conf/docker-compose.yml
 create mode 100644 support/testing/tests/package/test_docker_compose.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8f1e4ae804..69d6291039 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -318,6 +318,7 @@ tests.init.test_systemd.TestInitSystemSystemdRwFull: *runtime_test
 tests.init.test_systemd.TestInitSystemSystemdRwIfupdown: *runtime_test
 tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test
 tests.package.test_atop.TestAtop: *runtime_test
+tests.package.test_docker_compose.TestDockerCompose: *runtime_test
 tests.package.test_dropbear.TestDropbear: *runtime_test
 tests.package.test_ipython.TestIPythonPy2: *runtime_test
 tests.package.test_ipython.TestIPythonPy3: *runtime_test
diff --git a/support/testing/conf/docker-compose-kernel.config b/support/testing/conf/docker-compose-kernel.config
new file mode 100644
index 0000000000..9468ef8db8
--- /dev/null
+++ b/support/testing/conf/docker-compose-kernel.config
@@ -0,0 +1,103 @@
+# CONFIG_SWAP is not set
+CONFIG_SYSVIPC=y
+CONFIG_POSIX_MQUEUE=y
+# CONFIG_CROSS_MEMORY_ATTACH is not set
+CONFIG_HIGH_RES_TIMERS=y
+CONFIG_CGROUPS=y
+CONFIG_MEMCG=y
+CONFIG_BLK_CGROUP=y
+CONFIG_CGROUP_SCHED=y
+CONFIG_CGROUP_PIDS=y
+CONFIG_CGROUP_FREEZER=y
+CONFIG_CGROUP_DEVICE=y
+CONFIG_CGROUP_CPUACCT=y
+# CONFIG_COMPAT_BRK is not set
+# CONFIG_X86_EXTENDED_PLATFORM is not set
+CONFIG_IOSF_MBI=y
+CONFIG_MCORE2=y
+# CONFIG_X86_MCE is not set
+# CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS is not set
+CONFIG_LEGACY_VSYSCALL_NONE=y
+# CONFIG_SUSPEND is not set
+# CONFIG_ACPI is not set
+CONFIG_CPU_IDLE=y
+CONFIG_PCI_MSI=y
+# CONFIG_VIRTUALIZATION is not set
+# CONFIG_IOSCHED_CFQ is not set
+# CONFIG_MQ_IOSCHED_DEADLINE is not set
+# CONFIG_MQ_IOSCHED_KYBER is not set
+# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
+CONFIG_TRANSPARENT_HUGEPAGE=y
+CONFIG_NET=y
+CONFIG_PACKET=y
+CONFIG_UNIX=y
+CONFIG_INET=y
+CONFIG_IP_MULTICAST=y
+CONFIG_IP_ADVANCED_ROUTER=y
+# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
+# CONFIG_INET_XFRM_MODE_TUNNEL is not set
+# CONFIG_INET_XFRM_MODE_BEET is not set
+# CONFIG_IPV6 is not set
+CONFIG_NETFILTER=y
+CONFIG_NF_CONNTRACK=y
+# CONFIG_NF_CONNTRACK_PROCFS is not set
+# CONFIG_NF_CT_PROTO_DCCP is not set
+# CONFIG_NF_CT_PROTO_SCTP is not set
+# CONFIG_NF_CT_PROTO_UDPLITE is not set
+CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
+CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
+CONFIG_NETFILTER_XT_MATCH_IPVS=y
+CONFIG_IP_VS=y
+CONFIG_IP_VS_RR=y
+CONFIG_IP_VS_NFCT=y
+CONFIG_IP_NF_IPTABLES=y
+CONFIG_IP_NF_FILTER=y
+CONFIG_IP_NF_NAT=y
+CONFIG_IP_NF_TARGET_MASQUERADE=y
+CONFIG_BRIDGE=y
+# CONFIG_BRIDGE_IGMP_SNOOPING is not set
+# CONFIG_WIRELESS is not set
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+CONFIG_VIRTIO_BLK=y
+CONFIG_NETDEVICES=y
+CONFIG_DUMMY=y
+CONFIG_MACVLAN=y
+CONFIG_VXLAN=y
+CONFIG_VETH=y
+CONFIG_VIRTIO_NET=y
+# CONFIG_ETHERNET is not set
+# CONFIG_WLAN is not set
+# CONFIG_INPUT_KEYBOARD is not set
+# CONFIG_INPUT_MOUSE is not set
+# CONFIG_SERIO is not set
+CONFIG_VT_HW_CONSOLE_BINDING=y
+# CONFIG_DEVMEM is not set
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+# CONFIG_SERIAL_8250_LPSS is not set
+# CONFIG_SERIAL_8250_MID is not set
+# CONFIG_HW_RANDOM_INTEL is not set
+# CONFIG_HW_RANDOM_AMD is not set
+# CONFIG_HW_RANDOM_VIA is not set
+CONFIG_HW_RANDOM_VIRTIO=y
+# CONFIG_DEVPORT is not set
+# CONFIG_HWMON is not set
+# CONFIG_USB_SUPPORT is not set
+CONFIG_RTC_CLASS=y
+CONFIG_VIRTIO_PCI=y
+# CONFIG_VIRTIO_PCI_LEGACY is not set
+# CONFIG_X86_PLATFORM_DEVICES is not set
+# CONFIG_IOMMU_SUPPORT is not set
+CONFIG_EXT4_FS=y
+CONFIG_OVERLAY_FS=y
+CONFIG_OVERLAY_FS_REDIRECT_DIR=y
+CONFIG_TMPFS=y
+# CONFIG_MISC_FILESYSTEMS is not set
+# CONFIG_NETWORK_FILESYSTEMS is not set
+CONFIG_KEYS=y
+CONFIG_SECURITYFS=y
+# CONFIG_CRYPTO_ECHAINIV is not set
+# CONFIG_CRYPTO_HW is not set
+# CONFIG_UNUSED_SYMBOLS is not set
+CONFIG_UNWINDER_FRAME_POINTER=y
diff --git a/support/testing/conf/docker-compose.yml b/support/testing/conf/docker-compose.yml
new file mode 100644
index 0000000000..49ff2677da
--- /dev/null
+++ b/support/testing/conf/docker-compose.yml
@@ -0,0 +1,4 @@
+version: '3'
+services:
+  busybox:
+    image: "busybox:latest"
diff --git a/support/testing/tests/package/test_docker_compose.py b/support/testing/tests/package/test_docker_compose.py
new file mode 100644
index 0000000000..8bf3ae00b5
--- /dev/null
+++ b/support/testing/tests/package/test_docker_compose.py
@@ -0,0 +1,70 @@
+import os
+
+import infra.basetest
+
+
+class TestDockerCompose(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        """
+        BR2_x86_64=y
+        BR2_x86_core2=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+        BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+        BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-x86-64-core2-full-2018.05.tar.bz2"
+        BR2_TOOLCHAIN_EXTERNAL_GCC_6=y
+        BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_16=y
+        BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
+        # BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
+        BR2_TOOLCHAIN_EXTERNAL_CXX=y
+        BR2_SYSTEM_DHCP="eth0"
+        BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
+        BR2_ROOTFS_POST_SCRIPT_ARGS="{}"
+        BR2_LINUX_KERNEL=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.19"
+        BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
+        BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="{}"
+        BR2_PACKAGE_CA_CERTIFICATES=y
+        BR2_PACKAGE_CGROUPFS_MOUNT=y
+        BR2_PACKAGE_DOCKER_CLI=y
+        BR2_PACKAGE_DOCKER_COMPOSE=y
+        BR2_PACKAGE_DOCKER_ENGINE=y
+        BR2_TARGET_ROOTFS_EXT2=y
+        BR2_TARGET_ROOTFS_EXT2_SIZE="512M"
+        # BR2_TARGET_ROOTFS_TAR is not set
+        """.format(
+            infra.filepath("tests/package/copy-sample-script-to-target.sh"),
+            infra.filepath("conf/docker-compose.yml"),
+            infra.filepath("conf/docker-compose-kernel.config"))
+
+    def wait_for_dockerd(self):
+        # dockerd takes a while to start up
+        _, _ = self.emulator.run('while [ ! -e /var/run/docker.sock ]; do sleep 1; done', 120)
+
+    def docker_test(self):
+        # will download container if not available, which may take some time
+        _, exit_code = self.emulator.run('docker run --rm busybox:latest /bin/true', 120)
+        self.assertEqual(exit_code, 0)
+
+    def docker_compose_test(self):
+        # will download container if not available, which may take some time
+        _, exit_code = self.emulator.run('docker-compose up', 120)
+        self.assertEqual(exit_code, 0)
+
+    def test_run(self):
+        kernel = os.path.join(self.builddir, "images", "bzImage")
+        rootfs = os.path.join(self.builddir, "images", "rootfs.ext2")
+        self.emulator.boot(arch="x86_64",
+                           kernel=kernel,
+                           kernel_cmdline=["root=/dev/vda", "console=ttyS0"],
+                           options=["-cpu", "core2duo",
+                                    "-m", "512M",
+                                    "-device", "virtio-rng-pci",
+                                    "-drive", "file={},format=raw,if=virtio".format(rootfs),
+                                    "-net", "nic,model=virtio",
+                                    "-net", "user"])
+        self.emulator.login()
+        self.wait_for_dockerd()
+        self.docker_test()
+        self.docker_compose_test()
-- 
2.11.0

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Matthew Weber]

Peter,

On Mon, Feb 4, 2019 at 10:56 AM Peter Korsgaard <peter@korsgaard.com> wrote:
>
> Build for x86-64 as public containers in general are only available for
> x86-64.  Docker needs a number of kernel options enabled, so use a custom
> kernel config based on the qemu one.
>
> Docker needs entropy at startup, so enable the virtio-rng-pci device to
> expose entropy to the guest.

Another option is enabling BR2_PACKAGE_HAVEGED to let the target compensate.

> The default RAM amount (128M) is not enough to
> run docker / docker-compose, so bump to 512MB.
>
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Very cool, I was just setting up a similar test case.  Few notes below.

Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>

> ---
>  .gitlab-ci.yml                                     |   1 +
>  support/testing/conf/docker-compose-kernel.config  | 103 +++++++++++++++++++++
>  support/testing/conf/docker-compose.yml            |   4 +
>  .../testing/tests/package/test_docker_compose.py   |  70 ++++++++++++++
>  4 files changed, 178 insertions(+)
>  create mode 100644 support/testing/conf/docker-compose-kernel.config
>  create mode 100644 support/testing/conf/docker-compose.yml
>  create mode 100644 support/testing/tests/package/test_docker_compose.py
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 8f1e4ae804..69d6291039 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -318,6 +318,7 @@ tests.init.test_systemd.TestInitSystemSystemdRwFull: *runtime_test
>  tests.init.test_systemd.TestInitSystemSystemdRwIfupdown: *runtime_test
>  tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test
>  tests.package.test_atop.TestAtop: *runtime_test
> +tests.package.test_docker_compose.TestDockerCompose: *runtime_test
>  tests.package.test_dropbear.TestDropbear: *runtime_test
>  tests.package.test_ipython.TestIPythonPy2: *runtime_test
>  tests.package.test_ipython.TestIPythonPy3: *runtime_test
> diff --git a/support/testing/conf/docker-compose-kernel.config b/support/testing/conf/docker-compose-kernel.config
> new file mode 100644
> index 0000000000..9468ef8db8
> --- /dev/null
> +++ b/support/testing/conf/docker-compose-kernel.config
> @@ -0,0 +1,103 @@
> +# CONFIG_SWAP is not set
> +CONFIG_SYSVIPC=y
> +CONFIG_POSIX_MQUEUE=y
> +# CONFIG_CROSS_MEMORY_ATTACH is not set
> +CONFIG_HIGH_RES_TIMERS=y
> +CONFIG_CGROUPS=y
> +CONFIG_MEMCG=y
> +CONFIG_BLK_CGROUP=y
> +CONFIG_CGROUP_SCHED=y
> +CONFIG_CGROUP_PIDS=y
> +CONFIG_CGROUP_FREEZER=y
> +CONFIG_CGROUP_DEVICE=y
> +CONFIG_CGROUP_CPUACCT=y
> +# CONFIG_COMPAT_BRK is not set
> +# CONFIG_X86_EXTENDED_PLATFORM is not set
> +CONFIG_IOSF_MBI=y
> +CONFIG_MCORE2=y
> +# CONFIG_X86_MCE is not set
> +# CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS is not set
> +CONFIG_LEGACY_VSYSCALL_NONE=y
> +# CONFIG_SUSPEND is not set
> +# CONFIG_ACPI is not set
> +CONFIG_CPU_IDLE=y
> +CONFIG_PCI_MSI=y
> +# CONFIG_VIRTUALIZATION is not set
> +# CONFIG_IOSCHED_CFQ is not set
> +# CONFIG_MQ_IOSCHED_DEADLINE is not set
> +# CONFIG_MQ_IOSCHED_KYBER is not set
> +# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
> +CONFIG_TRANSPARENT_HUGEPAGE=y
> +CONFIG_NET=y
> +CONFIG_PACKET=y
> +CONFIG_UNIX=y
> +CONFIG_INET=y
> +CONFIG_IP_MULTICAST=y
> +CONFIG_IP_ADVANCED_ROUTER=y
> +# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
> +# CONFIG_INET_XFRM_MODE_TUNNEL is not set
> +# CONFIG_INET_XFRM_MODE_BEET is not set
> +# CONFIG_IPV6 is not set
> +CONFIG_NETFILTER=y
> +CONFIG_NF_CONNTRACK=y
> +# CONFIG_NF_CONNTRACK_PROCFS is not set
> +# CONFIG_NF_CT_PROTO_DCCP is not set
> +# CONFIG_NF_CT_PROTO_SCTP is not set
> +# CONFIG_NF_CT_PROTO_UDPLITE is not set
> +CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
> +CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
> +CONFIG_NETFILTER_XT_MATCH_IPVS=y
> +CONFIG_IP_VS=y
> +CONFIG_IP_VS_RR=y
> +CONFIG_IP_VS_NFCT=y
> +CONFIG_IP_NF_IPTABLES=y
> +CONFIG_IP_NF_FILTER=y
> +CONFIG_IP_NF_NAT=y
> +CONFIG_IP_NF_TARGET_MASQUERADE=y
> +CONFIG_BRIDGE=y
> +# CONFIG_BRIDGE_IGMP_SNOOPING is not set
> +# CONFIG_WIRELESS is not set
> +CONFIG_DEVTMPFS=y
> +CONFIG_DEVTMPFS_MOUNT=y
> +CONFIG_VIRTIO_BLK=y
> +CONFIG_NETDEVICES=y
> +CONFIG_DUMMY=y
> +CONFIG_MACVLAN=y
> +CONFIG_VXLAN=y
> +CONFIG_VETH=y
> +CONFIG_VIRTIO_NET=y
> +# CONFIG_ETHERNET is not set
> +# CONFIG_WLAN is not set
> +# CONFIG_INPUT_KEYBOARD is not set
> +# CONFIG_INPUT_MOUSE is not set
> +# CONFIG_SERIO is not set
> +CONFIG_VT_HW_CONSOLE_BINDING=y
> +# CONFIG_DEVMEM is not set
> +CONFIG_SERIAL_8250=y
> +CONFIG_SERIAL_8250_CONSOLE=y
> +# CONFIG_SERIAL_8250_LPSS is not set
> +# CONFIG_SERIAL_8250_MID is not set
> +# CONFIG_HW_RANDOM_INTEL is not set
> +# CONFIG_HW_RANDOM_AMD is not set
> +# CONFIG_HW_RANDOM_VIA is not set
> +CONFIG_HW_RANDOM_VIRTIO=y
> +# CONFIG_DEVPORT is not set
> +# CONFIG_HWMON is not set
> +# CONFIG_USB_SUPPORT is not set
> +CONFIG_RTC_CLASS=y
> +CONFIG_VIRTIO_PCI=y
> +# CONFIG_VIRTIO_PCI_LEGACY is not set
> +# CONFIG_X86_PLATFORM_DEVICES is not set
> +# CONFIG_IOMMU_SUPPORT is not set
> +CONFIG_EXT4_FS=y
> +CONFIG_OVERLAY_FS=y
> +CONFIG_OVERLAY_FS_REDIRECT_DIR=y
> +CONFIG_TMPFS=y
> +# CONFIG_MISC_FILESYSTEMS is not set
> +# CONFIG_NETWORK_FILESYSTEMS is not set
> +CONFIG_KEYS=y
> +CONFIG_SECURITYFS=y
> +# CONFIG_CRYPTO_ECHAINIV is not set
> +# CONFIG_CRYPTO_HW is not set
> +# CONFIG_UNUSED_SYMBOLS is not set
> +CONFIG_UNWINDER_FRAME_POINTER=y

There is a script "./build/docker-engine*/contrib/check-config.sh"
which could be copied over to the target and used to do a docker
configuration check as part of the test case (would need config.gz
enabled).  I had started to add this as a br2-external post script, I
could send something after this merges.


> diff --git a/support/testing/conf/docker-compose.yml b/support/testing/conf/docker-compose.yml
> new file mode 100644
> index 0000000000..49ff2677da
> --- /dev/null
> +++ b/support/testing/conf/docker-compose.yml
> @@ -0,0 +1,4 @@
> +version: '3'
> +services:
> +  busybox:
> +    image: "busybox:latest"
> diff --git a/support/testing/tests/package/test_docker_compose.py b/support/testing/tests/package/test_docker_compose.py
> new file mode 100644
> index 0000000000..8bf3ae00b5
> --- /dev/null
> +++ b/support/testing/tests/package/test_docker_compose.py
> @@ -0,0 +1,70 @@
> +import os
> +
> +import infra.basetest
> +
> +
> +class TestDockerCompose(infra.basetest.BRTest):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        BR2_x86_64=y
> +        BR2_x86_core2=y
> +        BR2_TOOLCHAIN_EXTERNAL=y
> +        BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
> +        BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
> +        BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-x86-64-core2-full-2018.05.tar.bz2"
> +        BR2_TOOLCHAIN_EXTERNAL_GCC_6=y
> +        BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_16=y
> +        BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
> +        # BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
> +        BR2_TOOLCHAIN_EXTERNAL_CXX=y
> +        BR2_SYSTEM_DHCP="eth0"
> +        BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
> +        BR2_ROOTFS_POST_SCRIPT_ARGS="{}"
> +        BR2_LINUX_KERNEL=y
> +        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
> +        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.19"
> +        BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
> +        BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="{}"
> +        BR2_PACKAGE_CA_CERTIFICATES=y
> +        BR2_PACKAGE_CGROUPFS_MOUNT=y
> +        BR2_PACKAGE_DOCKER_CLI=y
> +        BR2_PACKAGE_DOCKER_COMPOSE=y
> +        BR2_PACKAGE_DOCKER_ENGINE=y
> +        BR2_TARGET_ROOTFS_EXT2=y
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="512M"
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        """.format(
> +            infra.filepath("tests/package/copy-sample-script-to-target.sh"),
> +            infra.filepath("conf/docker-compose.yml"),
> +            infra.filepath("conf/docker-compose-kernel.config"))
> +
> +    def wait_for_dockerd(self):
> +        # dockerd takes a while to start up
> +        _, _ = self.emulator.run('while [ ! -e /var/run/docker.sock ]; do sleep 1; done', 120)
> +
> +    def docker_test(self):
> +        # will download container if not available, which may take some time
> +        _, exit_code = self.emulator.run('docker run --rm busybox:latest /bin/true', 120)
> +        self.assertEqual(exit_code, 0)

Another way to test could be to directly grep /proc/self/cgroups for
the presence of docker.  Does the return code cover that the container
was all the way up?

Unrelated to above, I ran into errors like the following but I was
targeting my build for aarch64 in my runtime test.

Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, linux/amd64.
Building packages and commands for target, linux/arm64.
# cmd/trace
/accts/mlweber1/wip/TestDocker/build/host-go-1.11.5/pkg/tool/linux_amd64/link:
running /accts/mlweber1/wip/TestDocker/host/bin/aarch64-linux-gnu-gcc
failed: exit status 1
aarch64-linux-gnu-gcc: error: unrecognized command line option
'-Qunused-arguments'; did you mean '-Wunused-parameter'?
make[1]: *** [package/pkg-generic.mk:233:
/accts/mlweber1/wip/TestDocker/build/host-go-1.11.5/.stamp_built]
Error 2

Matt

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Thomas Petazzoni]

On Mon,  4 Feb 2019 17:55:57 +0100
Peter Korsgaard <peter@korsgaard.com> wrote:

> Build for x86-64 as public containers in general are only available for
> x86-64.  Docker needs a number of kernel options enabled, so use a custom
> kernel config based on the qemu one.

Do you need to duplicate the kernel config entirely, or can you use
board/qemu/x86_64/linux.config + a fragment ?

> Docker needs entropy at startup, so enable the virtio-rng-pci device to
> expose entropy to the guest.  The default RAM amount (128M) is not enough to
> run docker / docker-compose, so bump to 512MB.

Wow. Go is efficient :)

> diff --git a/support/testing/conf/docker-compose.yml b/support/testing/conf/docker-compose.yml
> new file mode 100644
> index 0000000000..49ff2677da
> --- /dev/null
> +++ b/support/testing/conf/docker-compose.yml
> @@ -0,0 +1,4 @@
> +version: '3'
> +services:
> +  busybox:
> +    image: "busybox:latest"

Is conf/ really the right place for this ? Don't we have some other
place to store the per-tests artifacts that go on the target ? I don't
think we have a good organization for this at the moment, so this will
do for now.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

Hi,

 > On Mon,  4 Feb 2019 17:55:57 +0100
 > Peter Korsgaard <peter@korsgaard.com> wrote:

 >> Build for x86-64 as public containers in general are only available for
 >> x86-64.  Docker needs a number of kernel options enabled, so use a custom
 >> kernel config based on the qemu one.

 > Do you need to duplicate the kernel config entirely, or can you use
 > board/qemu/x86_64/linux.config + a fragment ?

I didn't really look closely. I recently setup docker / docker-compose at
$WORK, so I based the linux config on that.

I can have a look at basing it on the qemu config, but that means that
we have to be a bit careful when bumping the qemu config, so I'm not
sure if it makes sense.


 >> Docker needs entropy at startup, so enable the virtio-rng-pci device to
 >> expose entropy to the guest.  The default RAM amount (128M) is not enough to
 >> run docker / docker-compose, so bump to 512MB.

 > Wow. Go is efficient :)

;)

 >> diff --git a/support/testing/conf/docker-compose.yml b/support/testing/conf/docker-compose.yml
 >> new file mode 100644
 >> index 0000000000..49ff2677da
 >> --- /dev/null
 >> +++ b/support/testing/conf/docker-compose.yml
 >> @@ -0,0 +1,4 @@
 >> +version: '3'
 >> +services:
 >> +  busybox:
 >> +    image: "busybox:latest"

 > Is conf/ really the right place for this ? Don't we have some other
 > place to store the per-tests artifacts that go on the target ? I don't
 > think we have a good organization for this at the moment, so this will
 > do for now.

Indeed. We have a number of more-or-less random files under conf/, and
then E.G. tests/package/test_python_*.py stores files for the target
also under tests/package/sample_*.py

I can move the files under tests/package if you prefer, but I'm not sure
if that is really cleaner.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[ricardo.martincoski at gmail.com]

Hello,

On Mon, Feb 04, 2019 at 03:11 PM, Matthew Weber wrote:

> On Mon, Feb 4, 2019 at 10:56 AM Peter Korsgaard wrote:
>>
>> Build for x86-64 as public containers in general are only available for
>> x86-64.  Docker needs a number of kernel options enabled, so use a custom
>> kernel config based on the qemu one.
>>
>> Docker needs entropy at startup, so enable the virtio-rng-pci device to
>> expose entropy to the guest.
> 
> Another option is enabling BR2_PACKAGE_HAVEGED to let the target compensate.

This is already used for 2 python test cases (pynacl and crossbar).
I don't have specific knowledge to say which is the best approach. For the
python test cases I just searched for *any* way to have enough entropy.


Regards,
Ricardo

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Ricardo Martincoski]

Hello,

+ Yann

On Mon, Feb 04, 2019 at 06:20 PM, Peter Korsgaard wrote:

>>>>>> "Thomas" == Thomas Petazzoni writes:
> 
>  > On Mon,  4 Feb 2019 17:55:57 +0100
>  > Peter Korsgaard wrote:
> 
>  >> Build for x86-64 as public containers in general are only available for
>  >> x86-64.  Docker needs a number of kernel options enabled, so use a custom
>  >> kernel config based on the qemu one.
> 
>  > Do you need to duplicate the kernel config entirely, or can you use
>  > board/qemu/x86_64/linux.config + a fragment ?
> 
> I didn't really look closely. I recently setup docker / docker-compose at
> $WORK, so I based the linux config on that.
> 
> I can have a look at basing it on the qemu config, but that means that
> we have to be a bit careful when bumping the qemu config, so I'm not
> sure if it makes sense.

The nice thing of using a fragment is that we would have an example of the
configs that one needs to enable in the kernel in order to use docker in the
target.
It should be *theoretically* straight-forward created from the docker
documentation. I don't know how well it is documented.
Any config that docker needs that is already in the base config (so showing in
both the base config and the fragment) should be correctly set by the merge of
configs, I guess.

But maybe we find some practical problem when trying to accomplish this
theoretical task, perhaps when changing the major version of the kernel.
Or even you think that maintaining such example config is too much out-of-scope
for the Buildroot source tree.

So... up to you guys. And it can be done in a followup patch.

[snip]
>  >> +++ b/support/testing/conf/docker-compose.yml
>  >> @@ -0,0 +1,4 @@
>  >> +version: '3'
>  >> +services:
>  >> +  busybox:
>  >> +    image: "busybox:latest"
> 
>  > Is conf/ really the right place for this ? Don't we have some other
>  > place to store the per-tests artifacts that go on the target ? I don't
>  > think we have a good organization for this at the moment, so this will
>  > do for now.
> 
> Indeed. We have a number of more-or-less random files under conf/, and
> then E.G. tests/package/test_python_*.py stores files for the target
> also under tests/package/sample_*.py

I obviously prefer the fixtures besides the test cases, since I always send test
cases in this way, AFAICR.
I just never bothered to submit a patch changing all test cases to do so.
We can choose one or another and standardize.

> 
> I can move the files under tests/package if you prefer, but I'm not sure
> if that is really cleaner.

If we decide to apply that series that Yann once submitted changing the
test cases to each package directory, I think it would make more sense to keep
the fixtures also in the same directory.

package/pkg1/Config.in
package/pkg1/pkg1.mk
package/pkg1/test_pkg1.py
package/pkg1/sample_pkg1.anything
package/pkg1/S50_init_script

Or something like this. I am not suggesting any naming convention.


Regards,
Ricardo

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Peter Korsgaard]

>>>>> "Matthew" == Matthew Weber <matthew.weber@rockwellcollins.com> writes:

 > Peter,
 > On Mon, Feb 4, 2019 at 10:56 AM Peter Korsgaard <peter@korsgaard.com> wrote:
 >> 
 >> Build for x86-64 as public containers in general are only available for
 >> x86-64.  Docker needs a number of kernel options enabled, so use a custom
 >> kernel config based on the qemu one.
 >> 
 >> Docker needs entropy at startup, so enable the virtio-rng-pci device to
 >> expose entropy to the guest.

 > Another option is enabling BR2_PACKAGE_HAVEGED to let the target compensate.

True, but given that most real systems have a hw rng, this is probably
closer to real life.

 >> The default RAM amount (128M) is not enough to
 >> run docker / docker-compose, so bump to 512MB.
 >> 
 >> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

 > Very cool, I was just setting up a similar test case.  Few notes below.

 > Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>

 > There is a script "./build/docker-engine*/contrib/check-config.sh"
 > which could be copied over to the target and used to do a docker
 > configuration check as part of the test case (would need config.gz
 > enabled).  I had started to add this as a br2-external post script, I
 > could send something after this merges.

Yes, I had originally used this script to figure out the kernel options
to use:

From the moby check-config script:
    (https://github.com/moby/moby/blob/e2de2123399f494cb41a4cb62392999c80c2e99c/contrib/check-config.sh)

    info: reading kernel config from /proc/config.gz ...

    Generally Necessary:
    - cgroup hierarchy: properly mounted [/sys/fs/cgroup]
    - CONFIG_NAMESPACES: enabled
    - CONFIG_NET_NS: enabled
    - CONFIG_PID_NS: enabled
    - CONFIG_IPC_NS: enabled
    - CONFIG_UTS_NS: enabled
    - CONFIG_CGROUPS: enabled
    - CONFIG_CGROUP_CPUACCT: enabled
    - CONFIG_CGROUP_DEVICE: enabled
    - CONFIG_CGROUP_FREEZER: enabled
    - CONFIG_CGROUP_SCHED: enabled
    - CONFIG_CPUSETS: enabled
    - CONFIG_MEMCG: enabled
    - CONFIG_KEYS: enabled
    - CONFIG_VETH: enabled
    - CONFIG_BRIDGE: enabled
    - CONFIG_BRIDGE_NETFILTER: enabled
    - CONFIG_NF_NAT_IPV4: enabled
    - CONFIG_IP_NF_FILTER: enabled
    - CONFIG_IP_NF_TARGET_MASQUERADE: enabled
    - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
    - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
    - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled
    - CONFIG_IP_NF_NAT: enabled
    - CONFIG_NF_NAT: enabled
    - CONFIG_NF_NAT_NEEDED: enabled
    - CONFIG_POSIX_MQUEUE: enabled

    Optional Features:
    - CONFIG_USER_NS: missing
    - CONFIG_SECCOMP: enabled
    - CONFIG_CGROUP_PIDS: enabled
    - CONFIG_MEMCG_SWAP: missing
    - CONFIG_MEMCG_SWAP_ENABLED: missing
    - CONFIG_LEGACY_VSYSCALL_NONE: enabled
        (containers using eglibc <= 2.13 will not work. Switch to
         "CONFIG_VSYSCALL_[NATIVE|EMULATE]" or use "vsyscall=[native|emulate]"
         on kernel command line. Note that this will disable ASLR for the,
         VDSO which may assist in exploiting security vulnerabilities.)
    - CONFIG_BLK_CGROUP: enabled
    - CONFIG_BLK_DEV_THROTTLING: missing
    - CONFIG_IOSCHED_CFQ: missing
    - CONFIG_CFQ_GROUP_IOSCHED: missing
    - CONFIG_CGROUP_PERF: missing
    - CONFIG_CGROUP_HUGETLB: missing
    - CONFIG_NET_CLS_CGROUP: missing
    - CONFIG_CGROUP_NET_PRIO: missing
    - CONFIG_CFS_BANDWIDTH: missing
    - CONFIG_FAIR_GROUP_SCHED: enabled
    - CONFIG_RT_GROUP_SCHED: missing
    - CONFIG_IP_VS: enabled
    - CONFIG_IP_VS_NFCT: enabled
    - CONFIG_IP_VS_RR: enabled
    - CONFIG_EXT4_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: missing
    - CONFIG_EXT4_FS_SECURITY: missing
        enable these ext4 configs if you are using ext3 or ext4 as backing filesystem
    - Network Drivers:
      - "overlay":
        - CONFIG_VXLAN: enabled
          Optional (for encrypted networks):
          - CONFIG_CRYPTO: enabled
          - CONFIG_CRYPTO_AEAD: missing
          - CONFIG_CRYPTO_GCM: missing
          - CONFIG_CRYPTO_SEQIV: missing
          - CONFIG_CRYPTO_GHASH: missing
          - CONFIG_XFRM: missing
          - CONFIG_XFRM_USER: missing
          - CONFIG_XFRM_ALGO: missing
          - CONFIG_INET_ESP: missing
          - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
      - "ipvlan":
        - CONFIG_IPVLAN: missing
      - "macvlan":
        - CONFIG_MACVLAN: enabled
        - CONFIG_DUMMY: enabled
      - "ftp,tftp client in container":
        - CONFIG_NF_NAT_FTP: missing
        - CONFIG_NF_CONNTRACK_FTP: missing
        - CONFIG_NF_NAT_TFTP: missing
        - CONFIG_NF_CONNTRACK_TFTP: missing
    - Storage Drivers:
      - "aufs":
        - CONFIG_AUFS_FS: missing
      - "btrfs":
        - CONFIG_BTRFS_FS: missing
        - CONFIG_BTRFS_FS_POSIX_ACL: missing
      - "devicemapper":
        - CONFIG_BLK_DEV_DM: missing
        - CONFIG_DM_THIN_PROVISIONING: missing
      - "overlay":
        - CONFIG_OVERLAY_FS: enabled
      - "zfs":
        - /dev/zfs: missing
        - zfs command: missing
        - zpool command: missing

    Limits:
    - /proc/sys/kernel/keys/root_maxkeys: 1000000

I did indeed afterwards clean up the kernel config a bit, and dropped
CONFIG_IKCONFIG.


 >> diff --git a/support/testing/conf/docker-compose.yml b/support/testing/conf/docker-compose.yml
 >> new file mode 100644
 >> index 0000000000..49ff2677da
 >> --- /dev/null
 >> +++ b/support/testing/conf/docker-compose.yml
 >> @@ -0,0 +1,4 @@
 >> +version: '3'
 >> +services:
 >> +  busybox:
 >> +    image: "busybox:latest"
 >> diff --git a/support/testing/tests/package/test_docker_compose.py
 >> b/support/testing/tests/package/test_docker_compose.py
 >> new file mode 100644
 >> index 0000000000..8bf3ae00b5
 >> --- /dev/null
 >> +++ b/support/testing/tests/package/test_docker_compose.py
 >> @@ -0,0 +1,70 @@
 >> +import os
 >> +
 >> +import infra.basetest
 >> +
 >> +
 >> +class TestDockerCompose(infra.basetest.BRTest):
 >> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
 >> +        """
 >> +        BR2_x86_64=y
 >> +        BR2_x86_core2=y
 >> +        BR2_TOOLCHAIN_EXTERNAL=y
 >> +        BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
 >> +        BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
 >> +
 >> BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-x86-64-core2-full-2018.05.tar.bz2"
 >> +        BR2_TOOLCHAIN_EXTERNAL_GCC_6=y
 >> +        BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_16=y
 >> +        BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
 >> +        # BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
 >> +        BR2_TOOLCHAIN_EXTERNAL_CXX=y
 >> +        BR2_SYSTEM_DHCP="eth0"
 >> +        BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
 >> +        BR2_ROOTFS_POST_SCRIPT_ARGS="{}"
 >> +        BR2_LINUX_KERNEL=y
 >> +        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
 >> +        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.19"
 >> +        BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
 >> +        BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="{}"
 >> +        BR2_PACKAGE_CA_CERTIFICATES=y
 >> +        BR2_PACKAGE_CGROUPFS_MOUNT=y
 >> +        BR2_PACKAGE_DOCKER_CLI=y
 >> +        BR2_PACKAGE_DOCKER_COMPOSE=y
 >> +        BR2_PACKAGE_DOCKER_ENGINE=y
 >> +        BR2_TARGET_ROOTFS_EXT2=y
 >> +        BR2_TARGET_ROOTFS_EXT2_SIZE="512M"
 >> +        # BR2_TARGET_ROOTFS_TAR is not set
 >> +        """.format(
 >> +            infra.filepath("tests/package/copy-sample-script-to-target.sh"),
 >> +            infra.filepath("conf/docker-compose.yml"),
 >> +            infra.filepath("conf/docker-compose-kernel.config"))
 >> +
 >> +    def wait_for_dockerd(self):
 >> +        # dockerd takes a while to start up
 >> +        _, _ = self.emulator.run('while [ ! -e /var/run/docker.sock ]; do sleep 1; done', 120)
 >> +
 >> +    def docker_test(self):
 >> +        # will download container if not available, which may take some time
 >> +        _, exit_code = self.emulator.run('docker run --rm busybox:latest /bin/true', 120)
 >> +        self.assertEqual(exit_code, 0)

 > Another way to test could be to directly grep /proc/self/cgroups for
 > the presence of docker.  Does the return code cover that the container
 > was all the way up?

The return code is the return code from executing /bin/true inside the
container, so it should only succeed if we were really able to
successfully execute it.


 > Unrelated to above, I ran into errors like the following but I was
 > targeting my build for aarch64 in my runtime test.

 > Building Go toolchain3 using go_bootstrap and Go toolchain2.
 > Building packages and commands for host, linux/amd64.
 > Building packages and commands for target, linux/arm64.
 > # cmd/trace
 > /accts/mlweber1/wip/TestDocker/build/host-go-1.11.5/pkg/tool/linux_amd64/link:
 > running /accts/mlweber1/wip/TestDocker/host/bin/aarch64-linux-gnu-gcc
 > failed: exit status 1
 > aarch64-linux-gnu-gcc: error: unrecognized command line option
 > '-Qunused-arguments'; did you mean '-Wunused-parameter'?
 > make[1]: *** [package/pkg-generic.mk:233:
 > /accts/mlweber1/wip/TestDocker/build/host-go-1.11.5/.stamp_built]
 > Error 2

Hmm, I haven't seen such error before. Googling around, it seems to be a
command line argument supported by clang:

https://github.com/Ericsson/codechecker/issues/985

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Peter Korsgaard]

>>>>> "ricardo" == ricardo martincoski <ricardo.martincoski@gmail.com> writes:

 > Hello,
 > On Mon, Feb 04, 2019 at 03:11 PM, Matthew Weber wrote:

 >> On Mon, Feb 4, 2019 at 10:56 AM Peter Korsgaard wrote:
 >>> 
 >>> Build for x86-64 as public containers in general are only available for
 >>> x86-64.  Docker needs a number of kernel options enabled, so use a custom
 >>> kernel config based on the qemu one.
 >>> 
 >>> Docker needs entropy at startup, so enable the virtio-rng-pci device to
 >>> expose entropy to the guest.
 >> 
 >> Another option is enabling BR2_PACKAGE_HAVEGED to let the target compensate.

 > This is already used for 2 python test cases (pynacl and crossbar).
 > I don't have specific knowledge to say which is the best approach. For the
 > python test cases I just searched for *any* way to have enough entropy.

I don't think it is really clear cut. Haveged is probably not something
you should be using on a real system, and it only starts up relatively
late so there could be issues with early user space (E.G. I have had
issues with running mkfs.ext4 very early), but the virtio-rng-pci device
is only available for a limited set of qemu targets (E.G. the ones that
emulate a PCI bus).

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Peter Korsgaard]

>>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:

 > Build for x86-64 as public containers in general are only available for
 > x86-64.  Docker needs a number of kernel options enabled, so use a custom
 > kernel config based on the qemu one.

 > Docker needs entropy at startup, so enable the virtio-rng-pci device to
 > expose entropy to the guest.  The default RAM amount (128M) is not enough to
 > run docker / docker-compose, so bump to 512MB.

 > Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] support/testing: add docker / docker-compose tests

[Matthew Weber]

Peter,

On Tue, Feb 5, 2019 at 4:05 PM Peter Korsgaard <peter@korsgaard.com> wrote:
>
> >>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:
>
>  > Build for x86-64 as public containers in general are only available for
>  > x86-64.  Docker needs a number of kernel options enabled, so use a custom
>  > kernel config based on the qemu one.
>
>  > Docker needs entropy at startup, so enable the virtio-rng-pci device to
>  > expose entropy to the guest.  The default RAM amount (128M) is not enough to
>  > run docker / docker-compose, so bump to 512MB.
>
>  > Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
>
> Committed, thanks.

It's great to have a reference test case for docker!