[PATCH 0/4] support OEQA inside container

[PATCH 0/4] support OEQA inside container

[André Draszik]

Hi,

With these patches it is possible to run bitbake inside a docker
container and still successfully execute the OEQA tests that require
the target to talk to a temporary http server spawned on the machine
running bitake, e.g. using crops.

This wasn't possible so far because:
* we need to map a port from outside the container into the container
  for the target to be able to reach the http server
* we need to pass the IP address of the docker host to the target,
  e.g. in the OPKG repository URL, but we can't bind the http server
  to that IP address, as the http server is running inside the container
  with a different IP address - the bind would fail

To make this work, TEST_SERVER_IP needs to be set to the IP address of
the docker host, and the port that is mapped into the container can now
also be specified in TEST_SERVER_IP, similar to TEST_TARGET_IP

Cheers,
Andre'

[PATCH 1/4] oeqa/utils/httpserver: allow to pass in listening port

[André Draszik]

Being able to specify the listening port is useful when
running OEQA from within a docker container, e.g.
crops or any other solution.
In that case, a port on the outside must be mapped to a
specific port inside the container. If no port is specified
for the http server module in this case, the http server
would choose a random port, which is unlikely to be mapped
and thus won't be reachable from the outside.

Signed-off-by: André Draszik <git@andred.net>
---
 meta/lib/oeqa/utils/httpserver.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/utils/httpserver.py b/meta/lib/oeqa/utils/httpserver.py
index aa435590f0..58d3c3b3f8 100644
--- a/meta/lib/oeqa/utils/httpserver.py
+++ b/meta/lib/oeqa/utils/httpserver.py
@@ -22,10 +22,10 @@ class HTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
 
 class HTTPService(object):
 
-    def __init__(self, root_dir, host='', logger=None):
+    def __init__(self, root_dir, host='', port=0, logger=None):
         self.root_dir = root_dir
         self.host = host
-        self.port = 0
+        self.port = port
         self.logger = logger
 
     def start(self):
-- 
2.24.0

[PATCH 2/4] oeqa/runtime/context.py: support listening port in TEST_SERVER_IP

[André Draszik]

Similar to the existing possibility to specify a port in
TEST_TARGET_IP, allow TEST_SERVER_IP to also contain a
port.

The intention is for this port to be passed into e.g.
the http server from the apt / dnf / opkg tests, or
any other (custom) tests that might need the target to
connect to a service spawned by bitbake / oeqa, where
bitbake is actually running inside a docker container.

Signed-off-by: André Draszik <git@andred.net>
---
 meta/lib/oeqa/runtime/context.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/lib/oeqa/runtime/context.py b/meta/lib/oeqa/runtime/context.py
index ef738a3359..2ecb1a8f01 100644
--- a/meta/lib/oeqa/runtime/context.py
+++ b/meta/lib/oeqa/runtime/context.py
@@ -98,6 +98,12 @@ class OERuntimeTestContextExecutor(OETestContextExecutor):
                 target_ip = target_ip_port[0]
                 kwargs['port'] = target_ip_port[1]
 
+        if server_ip:
+            server_ip_port = server_ip.split(':')
+            if len(server_ip_port) == 2:
+                server_ip = server_ip_port[0]
+                kwargs['server_port'] = int(server_ip_port[1])
+
         if target_type == 'simpleremote':
             target = OESSHTarget(logger, target_ip, server_ip, **kwargs)
         elif target_type == 'qemu':
-- 
2.24.0

[PATCH 3/4] oeqa/target/ssh oeqa/target/qemu: expose server listening port to tests

[André Draszik]

Allow tests to access the listening port as just introduced.
Note that when using qemu this infrastructure shouldn't be
needed, but we still need to set the port to 0 so that
a listening port is determined automatically (e.g. by the
python http server).

Signed-off-by: André Draszik <git@andred.net>
---
 meta/lib/oeqa/core/target/qemu.py | 1 +
 meta/lib/oeqa/core/target/ssh.py  | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/core/target/qemu.py b/meta/lib/oeqa/core/target/qemu.py
index 081c627b01..758703c0d1 100644
--- a/meta/lib/oeqa/core/target/qemu.py
+++ b/meta/lib/oeqa/core/target/qemu.py
@@ -24,6 +24,7 @@ class OEQemuTarget(OESSHTarget):
                 user, port)
 
         self.server_ip = server_ip
+        self.server_port = 0
         self.machine = machine
         self.rootfs = rootfs
         self.kernel = kernel
diff --git a/meta/lib/oeqa/core/target/ssh.py b/meta/lib/oeqa/core/target/ssh.py
index 51032ef1a9..63fc9468b3 100644
--- a/meta/lib/oeqa/core/target/ssh.py
+++ b/meta/lib/oeqa/core/target/ssh.py
@@ -15,7 +15,7 @@ from . import OETarget
 
 class OESSHTarget(OETarget):
     def __init__(self, logger, ip, server_ip, timeout=300, user='root',
-                 port=None, **kwargs):
+                 port=None, server_port=0, **kwargs):
         if not logger:
             logger = logging.getLogger('target')
             logger.setLevel(logging.INFO)
@@ -30,6 +30,7 @@ class OESSHTarget(OETarget):
         super(OESSHTarget, self).__init__(logger)
         self.ip = ip
         self.server_ip = server_ip
+        self.server_port = server_port
         self.timeout = timeout
         self.user = user
         ssh_options = [
-- 
2.24.0

[PATCH 4/4] oeqa/runtime/apt dnf opkg: support running from within docker container

[André Draszik]

If the user specified an http port to use for serving
files, use that instead of a random one. At the same
time, have the http server bind to all interfaces.

Binding to the server_ip might not always be possible,
e.g. in the case of running bitbake / oeqa from within
a docker container. In this case, the ip address is valid
outside the container, but not inside, and hence can't
be bound to. So switch to simply binding to all interfaces.

Signed-off-by: André Draszik <git@andred.net>
---
 meta/lib/oeqa/runtime/cases/apt.py  | 4 +++-
 meta/lib/oeqa/runtime/cases/dnf.py  | 3 ++-
 meta/lib/oeqa/runtime/cases/opkg.py | 4 +++-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/runtime/cases/apt.py b/meta/lib/oeqa/runtime/cases/apt.py
index 74a940d80f..c5378d90c3 100644
--- a/meta/lib/oeqa/runtime/cases/apt.py
+++ b/meta/lib/oeqa/runtime/cases/apt.py
@@ -22,7 +22,9 @@ class AptRepoTest(AptTest):
     @classmethod
     def setUpClass(cls):
         service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
-        cls.repo_server = HTTPService(service_repo, cls.tc.target.server_ip, logger=cls.tc.logger)
+        cls.repo_server = HTTPService(service_repo,
+                                      '0.0.0.0', port=cls.tc.target.server_port,
+                                      logger=cls.tc.logger)
         cls.repo_server.start()
 
     @classmethod
diff --git a/meta/lib/oeqa/runtime/cases/dnf.py b/meta/lib/oeqa/runtime/cases/dnf.py
index de3759995e..f40c63026e 100644
--- a/meta/lib/oeqa/runtime/cases/dnf.py
+++ b/meta/lib/oeqa/runtime/cases/dnf.py
@@ -53,7 +53,8 @@ class DnfRepoTest(DnfTest):
     @classmethod
     def setUpClass(cls):
         cls.repo_server = HTTPService(os.path.join(cls.tc.td['WORKDIR'], 'oe-testimage-repo'),
-                                      cls.tc.target.server_ip, logger=cls.tc.logger)
+                                      '0.0.0.0', port=cls.tc.target.server_port,
+                                      logger=cls.tc.logger)
         cls.repo_server.start()
 
     @classmethod
diff --git a/meta/lib/oeqa/runtime/cases/opkg.py b/meta/lib/oeqa/runtime/cases/opkg.py
index 750706161b..9cfee1cd88 100644
--- a/meta/lib/oeqa/runtime/cases/opkg.py
+++ b/meta/lib/oeqa/runtime/cases/opkg.py
@@ -25,7 +25,9 @@ class OpkgRepoTest(OpkgTest):
         if cls.tc.td["MULTILIB_VARIANTS"]:
             allarchfeed = cls.tc.td["TUNE_PKGARCH"]
         service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_IPK'], allarchfeed)
-        cls.repo_server = HTTPService(service_repo, cls.tc.target.server_ip, logger=cls.tc.logger)
+        cls.repo_server = HTTPService(service_repo,
+                                      '0.0.0.0', port=cls.tc.target.server_port,
+                                      logger=cls.tc.logger)
         cls.repo_server.start()
 
     @classmethod
-- 
2.24.0