qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images
@ 2016-06-28 15:42 Alex Bennée
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 1/5] tests/docker/docker.py: docker_dir outside build Alex Bennée
                   ` (5 more replies)
  0 siblings, 6 replies; 10+ messages in thread
From: Alex Bennée @ 2016-06-28 15:42 UTC (permalink / raw)
  To: qemu-devel; +Cc: famz, riku.voipio, Alex Bennée

This is the latest iteration of my qemu-user support inside Docker.
They apply directly on top of master. I've made the changes suggested
in the last review and split apart another patch. I've also added a
new update command so a tagged image can be updated with the latest
QEMU.

To use, first ensure you build the debian-bootstrap image:

    DEB_ARCH=armhf DEB_TYPE=testing \
      ./tests/docker/docker.py build qemu:debian-bootstrap \
      ./tests/docker/dockerfiles/debian-bootstrap.docker \
      --include-executable=./arm-linux-user/qemu-arm

And then run the test quick target:

    make docker-test-quick@debian-bootstrap J=9 V=1

To update the installed QEMU:

    ./tests/docker/docker.py update qemu:debian-bootstrap \
      ./arm-linux-user/qemu-arm

To run a throwaway container:

    docker run -t -i --rm qemu:debian-bootstrap

I'll leave it up to you how we cleanly integrate multi-arch builds
into the Make system ;-)

Alex Bennée (5):
  tests/docker/docker.py: docker_dir outside build
  tests/docker/docker.py: support --include-executable
  tests/docker/docker.py: check and run .pre script
  tests/docker/dockerfiles: new debian-bootstrap.docker
  tests/docker/docker.py: add update operation

 tests/docker/docker.py                           | 144 +++++++++++++++++++++--
 tests/docker/dockerfiles/debian-bootstrap.docker |  21 ++++
 tests/docker/dockerfiles/debian-bootstrap.pre    |   5 +
 3 files changed, 161 insertions(+), 9 deletions(-)
 create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
 create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre

-- 
2.7.4

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Qemu-devel] [PATCH v3 1/5] tests/docker/docker.py: docker_dir outside build
  2016-06-28 15:42 [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Alex Bennée
@ 2016-06-28 15:42 ` Alex Bennée
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 2/5] tests/docker/docker.py: support --include-executable Alex Bennée
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2016-06-28 15:42 UTC (permalink / raw)
  To: qemu-devel; +Cc: famz, riku.voipio, Alex Bennée

Instead of letting the build_image create the temporary working dir we
move the creation to the build command. This is preparation for the
later patches where additional files can be added to the build context
before the build step is run.

We also ensure we remove the build context after we are done (mkdtemp
doesn't do this automatically for you).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>

---
v2
  - new for v2
v3
  - add r-b tag
---
 tests/docker/docker.py | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 0151362..ae40bb3 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,7 +20,7 @@ import atexit
 import uuid
 import argparse
 import tempfile
-from shutil import copy
+from shutil import copy, rmtree
 
 def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
@@ -87,20 +87,20 @@ class Docker(object):
         labels = json.loads(resp)[0]["Config"].get("Labels", {})
         return labels.get("com.qemu.dockerfile-checksum", "")
 
-    def build_image(self, tag, dockerfile, df_path, quiet=True, argv=None):
+    def build_image(self, tag, docker_dir, dockerfile, quiet=True, argv=None):
         if argv == None:
             argv = []
-        tmp_dir = tempfile.mkdtemp(prefix="docker_build")
 
-        tmp_df = tempfile.NamedTemporaryFile(dir=tmp_dir, suffix=".docker")
+        tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker")
         tmp_df.write(dockerfile)
 
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
                      _text_checksum(dockerfile))
         tmp_df.flush()
+
         self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
-                 [tmp_dir],
+                 [docker_dir],
                  quiet=quiet)
 
     def image_matches_dockerfile(self, tag, dockerfile):
@@ -164,10 +164,15 @@ class BuildCommand(SubCommand):
         if dkr.image_matches_dockerfile(tag, dockerfile):
             if not args.quiet:
                 print "Image is up to date."
-            return 0
+        else:
+            # Create a docker context directory for the build
+            docker_dir = tempfile.mkdtemp(prefix="docker_build")
+
+            dkr.build_image(tag, docker_dir, dockerfile,
+                            quiet=args.quiet, argv=argv)
+
+            rmtree(docker_dir)
 
-        dkr.build_image(tag, dockerfile, args.dockerfile,
-                        quiet=args.quiet, argv=argv)
         return 0
 
 class CleanCommand(SubCommand):
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [Qemu-devel] [PATCH v3 2/5] tests/docker/docker.py: support --include-executable
  2016-06-28 15:42 [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Alex Bennée
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 1/5] tests/docker/docker.py: docker_dir outside build Alex Bennée
@ 2016-06-28 15:42 ` Alex Bennée
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 3/5] tests/docker/docker.py: check and run .pre script Alex Bennée
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2016-06-28 15:42 UTC (permalink / raw)
  To: qemu-devel; +Cc: famz, riku.voipio, Alex Bennée

When passed the path to a binary we copy it and any linked libraries (if
it is dynamically linked) into the docker build context. These can then
be included by a dockerfile with the line:

  # Copy all of context into container
  ADD . /

This is mainly intended for setting up foreign architecture docker
images which use qemu-$arch to do cross-architecture linux-user
execution. It also relies on the host and guest file-system following
reasonable multi-arch layouts so the copied libraries don't clash with
the guest ones.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - change name of option
  - require full path to executable
  - clean-up the copy code
v3
  - safely deal with static builds
  - quietly pass if makedirs not needed
  - split copy routine and so lib resolver
---
 tests/docker/docker.py | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index ae40bb3..96d906e 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,6 +20,7 @@ import atexit
 import uuid
 import argparse
 import tempfile
+import re
 from shutil import copy, rmtree
 
 def _text_checksum(text):
@@ -38,6 +39,54 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
+def _copy_with_mkdir(src, root_dir, sub_path):
+    """Copy src into root_dir, creating sub_path as needed."""
+    dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
+    try:
+        os.makedirs(dest_dir)
+    except OSError:
+        # we can safely ignore already created directories
+        pass
+
+    dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
+    copy(src, dest_file)
+
+
+def _get_so_libs(executable):
+    """Return a list of libraries associated with an executable.
+
+    The paths may be symbolic links which would need to be resolved to
+    ensure theright data is copied."""
+
+    libs = []
+    ldd_re = re.compile(r"(/.*/)(\S*)")
+    try:
+        ldd_output = subprocess.check_output(["ldd", executable])
+        for line in ldd_output.split("\n"):
+            search = ldd_re.search(line)
+            if search and len(search.groups()) == 2:
+                so_path = search.groups()[0]
+                so_lib = search.groups()[1]
+                libs.append("%s/%s" % (so_path, so_lib))
+    except subprocess.CalledProcessError:
+        print "%s had no associated libraries (static build?)" % (executable)
+
+    return libs
+
+def _copy_binary_with_libs(src, dest_dir):
+    """Copy a binary executable and all its dependant libraries.
+
+    This does rely on the host file-system being fairly multi-arch
+    aware so the file don't clash with the guests layout."""
+
+    _copy_with_mkdir(src, dest_dir, "/usr/bin")
+
+    libs = _get_so_libs(src)
+    if libs:
+        for l in libs:
+            so_path = os.path.dirname(l)
+            _copy_with_mkdir(l , dest_dir, so_path)
+
 class Docker(object):
     """ Running Docker commands """
     def __init__(self):
@@ -151,6 +200,10 @@ class BuildCommand(SubCommand):
     """ Build docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
     name = "build"
     def args(self, parser):
+        parser.add_argument("--include-executable", "-e",
+                            help="""Specify a binary that will be copied to the
+                            container together with all its dependent
+                            libraries""")
         parser.add_argument("tag",
                             help="Image Tag")
         parser.add_argument("dockerfile",
@@ -168,6 +221,11 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Do we include a extra binary?
+            if args.include_executable:
+                _copy_binary_with_libs(args.include_executable,
+                                       docker_dir)
+
             dkr.build_image(tag, docker_dir, dockerfile,
                             quiet=args.quiet, argv=argv)
 
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [Qemu-devel] [PATCH v3 3/5] tests/docker/docker.py: check and run .pre script
  2016-06-28 15:42 [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Alex Bennée
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 1/5] tests/docker/docker.py: docker_dir outside build Alex Bennée
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 2/5] tests/docker/docker.py: support --include-executable Alex Bennée
@ 2016-06-28 15:42 ` Alex Bennée
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 4/5] tests/docker/dockerfiles: new debian-bootstrap.docker Alex Bennée
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2016-06-28 15:42 UTC (permalink / raw)
  To: qemu-devel; +Cc: famz, riku.voipio, Alex Bennée

The docker script will now search for an associated $dockerfile.pre
script which gets run in the same build context as the dockerfile will
be. This is to support pre-seeding the build context before running the
docker build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v3
  - new for v3, split from previous patch
  - use check_call when running the pre-script
---
 tests/docker/docker.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 96d906e..244901d 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -221,6 +221,12 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Is there a .pre file to run in the build context?
+            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
+            if os.path.exists(docker_pre):
+                subprocess.check_call(os.path.realpath(docker_pre),
+                                      cwd=docker_dir)
+
             # Do we include a extra binary?
             if args.include_executable:
                 _copy_binary_with_libs(args.include_executable,
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [Qemu-devel] [PATCH v3 4/5] tests/docker/dockerfiles: new debian-bootstrap.docker
  2016-06-28 15:42 [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Alex Bennée
                   ` (2 preceding siblings ...)
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 3/5] tests/docker/docker.py: check and run .pre script Alex Bennée
@ 2016-06-28 15:42 ` Alex Bennée
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 5/5] tests/docker/docker.py: add update operation Alex Bennée
  2016-07-08  6:15 ` [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Fam Zheng
  5 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2016-06-28 15:42 UTC (permalink / raw)
  To: qemu-devel; +Cc: famz, riku.voipio, Alex Bennée

Together with the debian-bootstrap.pre script can now build an arbitrary
architecture of Debian using debootstrap. This allows debootstrap to set
up its first stage before the container is built.

To build a container you need a command line like:

  DEB_ARCH=armhf DEB_TYPE=testing \
    ./tests/docker/docker.py build \
    --include-executable=arm-linux-user/qemu-arm debian:armhf \
    ./tests/docker/dockerfiles/debian-bootstrap.docker

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - use .pre script instead of embedded HOST_CMD
  - make default image include all QEMU build-deps
v3
  - split docker.py from introduction of bootstrap
---
 tests/docker/dockerfiles/debian-bootstrap.docker | 21 +++++++++++++++++++++
 tests/docker/dockerfiles/debian-bootstrap.pre    |  5 +++++
 2 files changed, 26 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
 create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre

diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
new file mode 100644
index 0000000..3a9125e
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -0,0 +1,21 @@
+# Create Debian Bootstrap Image
+#
+# This is intended to be pre-poluated by:
+#  - a first stage debootstrap (see debian-bootstrap.pre)
+#  - a native qemu-$arch that binfmt_misc will run
+FROM scratch
+
+# Add everything from the context into the container
+ADD . /
+
+# Patch all mounts as docker already has stuff set up
+RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
+
+# Run stage 2
+RUN /debootstrap/debootstrap --second-stage
+
+# At this point we can install additional packages if we want
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
+RUN apt-get update
+RUN apt-get -y build-dep qemu
diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
new file mode 100755
index 0000000..6f42da6
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -0,0 +1,5 @@
+#!/bin/sh
+#
+# Simple wrapper for debootstrap, run in the docker build context
+#
+fakeroot debootstrap --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [Qemu-devel] [PATCH v3 5/5] tests/docker/docker.py: add update operation
  2016-06-28 15:42 [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Alex Bennée
                   ` (3 preceding siblings ...)
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 4/5] tests/docker/dockerfiles: new debian-bootstrap.docker Alex Bennée
@ 2016-06-28 15:42 ` Alex Bennée
  2016-07-08  6:15 ` [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Fam Zheng
  5 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2016-06-28 15:42 UTC (permalink / raw)
  To: qemu-devel; +Cc: famz, riku.voipio, Alex Bennée

This adds a new operation to the docker script to allow updating of
binaries in an existing container. This is because it would be
inefficient to re-build the whole container just for an update to the
QEMU binary.

To update the executable run:

    ./tests/docker/docker.py update \
        debian:armhf ./arm-linux-user/qemu-arm

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v3
  - new in v3
---
 tests/docker/docker.py | 59 +++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 244901d..a8753f8 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -21,6 +21,8 @@ import uuid
 import argparse
 import tempfile
 import re
+from tarfile import TarFile, TarInfo
+from StringIO import StringIO
 from shutil import copy, rmtree
 
 def _text_checksum(text):
@@ -94,9 +96,11 @@ class Docker(object):
         self._instances = []
         atexit.register(self._kill_instances)
 
-    def _do(self, cmd, quiet=True, **kwargs):
+    def _do(self, cmd, quiet=True, infile=None, **kwargs):
         if quiet:
             kwargs["stdout"] = subprocess.PIPE
+        if infile:
+            kwargs["stdin"] = infile
         return subprocess.call(self._command + cmd, **kwargs)
 
     def _do_kill_instances(self, only_known, only_active=True):
@@ -152,6 +156,11 @@ class Docker(object):
                  [docker_dir],
                  quiet=quiet)
 
+    def update_image(self, tag, tarball, quiet=True):
+        "Update a tagged image using "
+
+        self._do(["build", "-t", tag, "-"], quiet=quiet, infile=tarball)
+
     def image_matches_dockerfile(self, tag, dockerfile):
         try:
             checksum = self.get_image_dockerfile_checksum(tag)
@@ -239,6 +248,54 @@ class BuildCommand(SubCommand):
 
         return 0
 
+class UpdateCommand(SubCommand):
+    """ Update a docker image with new executables. Arguments: <tag> <executable>"""
+    name = "update"
+    def args(self, parser):
+        parser.add_argument("tag",
+                            help="Image Tag")
+        parser.add_argument("executable",
+                            help="Executable to copy")
+
+    def run(self, args, argv):
+        # Create a temporary tarball with our whole build context and
+        # dockerfile for the update
+        tmp = tempfile.NamedTemporaryFile(suffix="dckr.tar.gz")
+        tmp_tar = TarFile(fileobj=tmp, mode='w')
+
+        # Add the executable to the tarball
+        bn = os.path.basename(args.executable)
+        ff = "/usr/bin/%s" % bn
+        tmp_tar.add(args.executable, arcname=ff)
+
+        # Add any associated libraries
+        libs = _get_so_libs(args.executable)
+        if libs:
+            for l in libs:
+                tmp_tar.add(os.path.realpath(l), arcname=l)
+
+        # Create a Docker buildfile
+        df = StringIO()
+        df.write("FROM %s\n" % args.tag)
+        df.write("ADD . /\n")
+        df.seek(0)
+
+        df_tar = TarInfo(name="Dockerfile")
+        df_tar.size = len(df.buf)
+        tmp_tar.addfile(df_tar, fileobj=df)
+
+        tmp_tar.close()
+
+        # reset the file pointers
+        tmp.flush()
+        tmp.seek(0)
+
+        # Run the build with our tarball context
+        dkr = Docker()
+        dkr.update_image(args.tag, tmp, quiet=args.quiet)
+
+        return 0
+
 class CleanCommand(SubCommand):
     """Clean up docker instances"""
     name = "clean"
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images
  2016-06-28 15:42 [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Alex Bennée
                   ` (4 preceding siblings ...)
  2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 5/5] tests/docker/docker.py: add update operation Alex Bennée
@ 2016-07-08  6:15 ` Fam Zheng
  2016-07-08  7:53   ` Alex Bennée
  5 siblings, 1 reply; 10+ messages in thread
From: Fam Zheng @ 2016-07-08  6:15 UTC (permalink / raw)
  To: Alex Bennée; +Cc: qemu-devel, riku.voipio

On Tue, 06/28 16:42, Alex Bennée wrote:
> This is the latest iteration of my qemu-user support inside Docker.
> They apply directly on top of master. I've made the changes suggested
> in the last review and split apart another patch. I've also added a
> new update command so a tagged image can be updated with the latest
> QEMU.
> 
> To use, first ensure you build the debian-bootstrap image:
> 
>     DEB_ARCH=armhf DEB_TYPE=testing \
>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>       --include-executable=./arm-linux-user/qemu-arm

Alex,

Sorry for the late reply.

Is there a way to setup or detect the binfmt_misc configuration on the system?
I think it's better to error out (and hint how to fix that) if qemu-arm is not
registered. Now it only fails complaining about "wrong exec format" after the
long debootstrap process is done.

Fam

> 
> And then run the test quick target:
> 
>     make docker-test-quick@debian-bootstrap J=9 V=1
> 
> To update the installed QEMU:
> 
>     ./tests/docker/docker.py update qemu:debian-bootstrap \
>       ./arm-linux-user/qemu-arm
> 
> To run a throwaway container:
> 
>     docker run -t -i --rm qemu:debian-bootstrap
> 
> I'll leave it up to you how we cleanly integrate multi-arch builds
> into the Make system ;-)
> 
> Alex Bennée (5):
>   tests/docker/docker.py: docker_dir outside build
>   tests/docker/docker.py: support --include-executable
>   tests/docker/docker.py: check and run .pre script
>   tests/docker/dockerfiles: new debian-bootstrap.docker
>   tests/docker/docker.py: add update operation
> 
>  tests/docker/docker.py                           | 144 +++++++++++++++++++++--
>  tests/docker/dockerfiles/debian-bootstrap.docker |  21 ++++
>  tests/docker/dockerfiles/debian-bootstrap.pre    |   5 +
>  3 files changed, 161 insertions(+), 9 deletions(-)
>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
> 
> -- 
> 2.7.4
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images
  2016-07-08  6:15 ` [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Fam Zheng
@ 2016-07-08  7:53   ` Alex Bennée
  2016-07-08  9:14     ` Fam Zheng
  0 siblings, 1 reply; 10+ messages in thread
From: Alex Bennée @ 2016-07-08  7:53 UTC (permalink / raw)
  To: Fam Zheng; +Cc: qemu-devel, riku.voipio


Fam Zheng <famz@redhat.com> writes:

> On Tue, 06/28 16:42, Alex Bennée wrote:
>> This is the latest iteration of my qemu-user support inside Docker.
>> They apply directly on top of master. I've made the changes suggested
>> in the last review and split apart another patch. I've also added a
>> new update command so a tagged image can be updated with the latest
>> QEMU.
>>
>> To use, first ensure you build the debian-bootstrap image:
>>
>>     DEB_ARCH=armhf DEB_TYPE=testing \
>>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>>       --include-executable=./arm-linux-user/qemu-arm
>
> Alex,
>
> Sorry for the late reply.
>
> Is there a way to setup or detect the binfmt_misc configuration on the
> system?

The trick is doing this in a distribution agnostic way. Certainly
Debian-a-like systems have scripts that allow you to query the state of
binmfmt_misc.

> I think it's better to error out (and hint how to fix that) if qemu-arm is not
> registered. Now it only fails complaining about "wrong exec format" after the
> long debootstrap process is done.

Would it be enough to warn if basename of the included executable
doesn't appear anywhere in binfmt_misc?

>
> Fam
>
>>
>> And then run the test quick target:
>>
>>     make docker-test-quick@debian-bootstrap J=9 V=1
>>
>> To update the installed QEMU:
>>
>>     ./tests/docker/docker.py update qemu:debian-bootstrap \
>>       ./arm-linux-user/qemu-arm
>>
>> To run a throwaway container:
>>
>>     docker run -t -i --rm qemu:debian-bootstrap
>>
>> I'll leave it up to you how we cleanly integrate multi-arch builds
>> into the Make system ;-)
>>
>> Alex Bennée (5):
>>   tests/docker/docker.py: docker_dir outside build
>>   tests/docker/docker.py: support --include-executable
>>   tests/docker/docker.py: check and run .pre script
>>   tests/docker/dockerfiles: new debian-bootstrap.docker
>>   tests/docker/docker.py: add update operation
>>
>>  tests/docker/docker.py                           | 144 +++++++++++++++++++++--
>>  tests/docker/dockerfiles/debian-bootstrap.docker |  21 ++++
>>  tests/docker/dockerfiles/debian-bootstrap.pre    |   5 +
>>  3 files changed, 161 insertions(+), 9 deletions(-)
>>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
>>
>> --
>> 2.7.4
>>


--
Alex Bennée

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images
  2016-07-08  7:53   ` Alex Bennée
@ 2016-07-08  9:14     ` Fam Zheng
  2016-07-08 10:10       ` Alex Bennée
  0 siblings, 1 reply; 10+ messages in thread
From: Fam Zheng @ 2016-07-08  9:14 UTC (permalink / raw)
  To: Alex Bennée; +Cc: qemu-devel, riku.voipio

On Fri, 07/08 08:53, Alex Bennée wrote:
> 
> Fam Zheng <famz@redhat.com> writes:
> 
> > On Tue, 06/28 16:42, Alex Bennée wrote:
> >> This is the latest iteration of my qemu-user support inside Docker.
> >> They apply directly on top of master. I've made the changes suggested
> >> in the last review and split apart another patch. I've also added a
> >> new update command so a tagged image can be updated with the latest
> >> QEMU.
> >>
> >> To use, first ensure you build the debian-bootstrap image:
> >>
> >>     DEB_ARCH=armhf DEB_TYPE=testing \
> >>       ./tests/docker/docker.py build qemu:debian-bootstrap \
> >>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
> >>       --include-executable=./arm-linux-user/qemu-arm
> >
> > Alex,
> >
> > Sorry for the late reply.
> >
> > Is there a way to setup or detect the binfmt_misc configuration on the
> > system?
> 
> The trick is doing this in a distribution agnostic way. Certainly
> Debian-a-like systems have scripts that allow you to query the state of
> binmfmt_misc.

What about changing the docker file from "FROM scratch" to "FROM debian"? Then
I think you can query the state in the container. (What is the query command
called, BTW?)

In addition, what about the idea I proposed in v2: moving the debootstrap
command to a dockerfile "RUN" directive? Does it work?  I can experiment with
that in this weekend, but if you know that already, that's even better. I think
one hurdle in this series is the required build step.

Fam

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images
  2016-07-08  9:14     ` Fam Zheng
@ 2016-07-08 10:10       ` Alex Bennée
  0 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2016-07-08 10:10 UTC (permalink / raw)
  To: Fam Zheng; +Cc: qemu-devel, riku.voipio


Fam Zheng <famz@redhat.com> writes:

> On Fri, 07/08 08:53, Alex Bennée wrote:
>>
>> Fam Zheng <famz@redhat.com> writes:
>>
>> > On Tue, 06/28 16:42, Alex Bennée wrote:
>> >> This is the latest iteration of my qemu-user support inside Docker.
>> >> They apply directly on top of master. I've made the changes suggested
>> >> in the last review and split apart another patch. I've also added a
>> >> new update command so a tagged image can be updated with the latest
>> >> QEMU.
>> >>
>> >> To use, first ensure you build the debian-bootstrap image:
>> >>
>> >>     DEB_ARCH=armhf DEB_TYPE=testing \
>> >>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>> >>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>> >>       --include-executable=./arm-linux-user/qemu-arm
>> >
>> > Alex,
>> >
>> > Sorry for the late reply.
>> >
>> > Is there a way to setup or detect the binfmt_misc configuration on the
>> > system?
>>
>> The trick is doing this in a distribution agnostic way. Certainly
>> Debian-a-like systems have scripts that allow you to query the state of
>> binmfmt_misc.
>
> What about changing the docker file from "FROM scratch" to "FROM debian"? Then
> I think you can query the state in the container.

But that would pull in the host debian build so you'd get clashes.

> (What is the query command
> called, BTW?)

11:08 alex@zen/x86_64  [qemu.git/mttcg/base-patches-v4] >/usr/sbin/update-binfmts --display qemu-arm
qemu-arm (enabled):
     package = qemu-user-binfmt
        type = magic
      offset = 0
       magic = \x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00
        mask = \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
 interpreter = /usr/bin/qemu-arm
    detector =

> In addition, what about the idea I proposed in v2: moving the debootstrap
> command to a dockerfile "RUN" directive? Does it work?  I can experiment with
> that in this weekend, but if you know that already, that's even better. I think
> one hurdle in this series is the required build step.

The initial run needs a working shell so that's why we do the two stage
seed and then setup.

>
> Fam


--
Alex Bennée

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2016-07-08 10:10 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-06-28 15:42 [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Alex Bennée
2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 1/5] tests/docker/docker.py: docker_dir outside build Alex Bennée
2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 2/5] tests/docker/docker.py: support --include-executable Alex Bennée
2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 3/5] tests/docker/docker.py: check and run .pre script Alex Bennée
2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 4/5] tests/docker/dockerfiles: new debian-bootstrap.docker Alex Bennée
2016-06-28 15:42 ` [Qemu-devel] [PATCH v3 5/5] tests/docker/docker.py: add update operation Alex Bennée
2016-07-08  6:15 ` [Qemu-devel] [PATCH v3 0/5] Support building qemu-user powered docker test images Fam Zheng
2016-07-08  7:53   ` Alex Bennée
2016-07-08  9:14     ` Fam Zheng
2016-07-08 10:10       ` Alex Bennée

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).