[Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Alex Bennée]

Hi,

This is a re-spin of the previous series built on top of
fam/docker.next. I've made the changes suggested in the last review
and split the first patch apart to separate (and fix) the build
directory changes first.

Now it no longer messes with the docker file you can actually
cross-build tests. First ensure you build the debian-bootstrap image:

    DEB_ARCH=armhf DEB_TYPE=testing \
      ./tests/docker/docker.py build qemu:debian-bootstrap \
      ./tests/docker/dockerfiles/debian-bootstrap.docker \
      --include-executable=./arm-linux-user/qemu-arm

And then run the test quick target:

    make docker-test-quick@debian-bootstrap J=9 V=1

I'll leave it up to you how we cleanly integrate multi-arch builds
into the Make system ;-)

Alex Bennée (3):
  tests/docker/docker.py: docker_dir outside build
  tests/docker/docker.py: support --include-executable
  add debian-bootstrap.docker target (and pre script)

 tests/docker/docker.py                           | 68 +++++++++++++++++++++---
 tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++++
 tests/docker/dockerfiles/debian-bootstrap.pre    |  5 ++
 3 files changed, 86 insertions(+), 8 deletions(-)
 create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
 create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre

-- 
2.7.4

[Qemu-devel] [PATCH v2 1/3] tests/docker/docker.py: docker_dir outside build

[Alex Bennée]

Instead of letting the build_image create the temporary working dir we
move the creation to the build command. This is preparation for the
later patches where additional files can be added to the build context
before the build step is run.

We also ensure we remove the build context after we are done (mkdtemp
doesn't do this automatically for you).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - new for v2
---
 tests/docker/docker.py | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 0151362..ae40bb3 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,7 +20,7 @@ import atexit
 import uuid
 import argparse
 import tempfile
-from shutil import copy
+from shutil import copy, rmtree
 
 def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
@@ -87,20 +87,20 @@ class Docker(object):
         labels = json.loads(resp)[0]["Config"].get("Labels", {})
         return labels.get("com.qemu.dockerfile-checksum", "")
 
-    def build_image(self, tag, dockerfile, df_path, quiet=True, argv=None):
+    def build_image(self, tag, docker_dir, dockerfile, quiet=True, argv=None):
         if argv == None:
             argv = []
-        tmp_dir = tempfile.mkdtemp(prefix="docker_build")
 
-        tmp_df = tempfile.NamedTemporaryFile(dir=tmp_dir, suffix=".docker")
+        tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker")
         tmp_df.write(dockerfile)
 
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
                      _text_checksum(dockerfile))
         tmp_df.flush()
+
         self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
-                 [tmp_dir],
+                 [docker_dir],
                  quiet=quiet)
 
     def image_matches_dockerfile(self, tag, dockerfile):
@@ -164,10 +164,15 @@ class BuildCommand(SubCommand):
         if dkr.image_matches_dockerfile(tag, dockerfile):
             if not args.quiet:
                 print "Image is up to date."
-            return 0
+        else:
+            # Create a docker context directory for the build
+            docker_dir = tempfile.mkdtemp(prefix="docker_build")
+
+            dkr.build_image(tag, docker_dir, dockerfile,
+                            quiet=args.quiet, argv=argv)
+
+            rmtree(docker_dir)
 
-        dkr.build_image(tag, dockerfile, args.dockerfile,
-                        quiet=args.quiet, argv=argv)
         return 0
 
 class CleanCommand(SubCommand):
-- 
2.7.4

[Qemu-devel] [PATCH v2 2/3] tests/docker/docker.py: support --include-executable

[Alex Bennée]

When passed the path to a binary we copy it and any linked libraries
into the docker build context. These can then be included by a
dockerfile with the line:

  # Copy all of context into container
  ADD . /

This is mainly intended for setting up foreign architecture docker
images which use qemu-$arch to do cross-architecture linux-user
execution. It also relies on the host and guest file-system following
reasonable multi-arch layouts so the copied libraries don't clash with
the guest ones.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - change name of option
  - require full path to executable
  - clean-up the copy code
---
 tests/docker/docker.py | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index ae40bb3..ed6fa45 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,6 +20,7 @@ import atexit
 import uuid
 import argparse
 import tempfile
+import re
 from shutil import copy, rmtree
 
 def _text_checksum(text):
@@ -38,6 +39,38 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
+def _copy_with_mkdir(src, root_dir, sub_path):
+    """Copy src into root_dir, creating sub_path as needed."""
+    dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
+    try:
+        os.makedirs(dest_dir)
+    except OSError:
+        print "%s already created" % (dest_dir)
+
+    dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
+    copy(src, dest_file)
+
+
+def _copy_binary_with_libs(src, dest_dir):
+    """Copy a binary executable and all its dependant libraries.
+
+    This does rely on the host file-system being fairly multi-arch
+    aware so the file don't clash with the guests layout."""
+
+    _copy_with_mkdir(src, dest_dir, "/usr/bin")
+
+    # do ldd bit here
+    ldd_re = re.compile(r"(/.*/)(\S*)")
+    ldd_output = subprocess.check_output(["ldd", src])
+    for line in ldd_output.split("\n"):
+        search = ldd_re.search(line)
+        if search and len(search.groups()) == 2:
+            so_path = search.groups()[0]
+            so_lib = search.groups()[1]
+            _copy_with_mkdir("%s/%s" % (so_path, so_lib),
+                             dest_dir, so_path)
+
+
 class Docker(object):
     """ Running Docker commands """
     def __init__(self):
@@ -151,6 +184,10 @@ class BuildCommand(SubCommand):
     """ Build docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
     name = "build"
     def args(self, parser):
+        parser.add_argument("--include-executable", "-e",
+                            help="""Specify a binary that will be copied to the
+                            container together with all its dependent
+                            libraries""")
         parser.add_argument("tag",
                             help="Image Tag")
         parser.add_argument("dockerfile",
@@ -168,6 +205,11 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Do we include a extra binary?
+            if args.include_executable:
+                _copy_binary_with_libs(args.include_executable,
+                                       docker_dir)
+
             dkr.build_image(tag, docker_dir, dockerfile,
                             quiet=args.quiet, argv=argv)
 
-- 
2.7.4

[Qemu-devel] [PATCH v2 3/3] add debian-bootstrap.docker target (and pre script)

[Alex Bennée]

Together with the debian-bootstrap.pre script can now build an arbitrary
architecture of Debian using debootstrap. The docker script will now
search for an associated $dockerfile.pre script which gets run in the
same build context as the dockerfile will be. This allows debootstrap to
set up its first stage before the container is built.

To build a container you need a command line like:

  DEB_ARCH=armhf DEB_TYPE=testing \
    ./tests/docker/docker.py build \
    --include-executable=arm-linux-user/qemu-arm debian:armhf \
    ./tests/docker/dockerfiles/debian-bootstrap.docker

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - use .pre script instead of embedded HOST_CMD
  - make default image include all QEMU build-deps
---
 tests/docker/docker.py                           |  5 +++++
 tests/docker/dockerfiles/debian-bootstrap.docker | 21 +++++++++++++++++++++
 tests/docker/dockerfiles/debian-bootstrap.pre    |  5 +++++
 3 files changed, 31 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
 create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index ed6fa45..43041eb 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -205,6 +205,11 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Is there a .pre file to run in the build context?
+            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
+            if os.path.exists(docker_pre):
+                subprocess.call(os.path.realpath(docker_pre), cwd=docker_dir)
+
             # Do we include a extra binary?
             if args.include_executable:
                 _copy_binary_with_libs(args.include_executable,
diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
new file mode 100644
index 0000000..3a9125e
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -0,0 +1,21 @@
+# Create Debian Bootstrap Image
+#
+# This is intended to be pre-poluated by:
+#  - a first stage debootstrap (see debian-bootstrap.pre)
+#  - a native qemu-$arch that binfmt_misc will run
+FROM scratch
+
+# Add everything from the context into the container
+ADD . /
+
+# Patch all mounts as docker already has stuff set up
+RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
+
+# Run stage 2
+RUN /debootstrap/debootstrap --second-stage
+
+# At this point we can install additional packages if we want
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
+RUN apt-get update
+RUN apt-get -y build-dep qemu
diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
new file mode 100755
index 0000000..6f42da6
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -0,0 +1,5 @@
+#!/bin/sh
+#
+# Simple wrapper for debootstrap, run in the docker build context
+#
+fakeroot debootstrap --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian
-- 
2.7.4

Re: [Qemu-devel] [PATCH v2 1/3] tests/docker/docker.py: docker_dir outside build

[Fam Zheng]

On Wed, 06/08 17:35, Alex Bennée wrote:
> Instead of letting the build_image create the temporary working dir we
> move the creation to the build command. This is preparation for the
> later patches where additional files can be added to the build context
> before the build step is run.
> 
> We also ensure we remove the build context after we are done (mkdtemp
> doesn't do this automatically for you).
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v2
>   - new for v2
> ---
>  tests/docker/docker.py | 21 +++++++++++++--------
>  1 file changed, 13 insertions(+), 8 deletions(-)
> 
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 0151362..ae40bb3 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -20,7 +20,7 @@ import atexit
>  import uuid
>  import argparse
>  import tempfile
> -from shutil import copy
> +from shutil import copy, rmtree
>  
>  def _text_checksum(text):
>      """Calculate a digest string unique to the text content"""
> @@ -87,20 +87,20 @@ class Docker(object):
>          labels = json.loads(resp)[0]["Config"].get("Labels", {})
>          return labels.get("com.qemu.dockerfile-checksum", "")
>  
> -    def build_image(self, tag, dockerfile, df_path, quiet=True, argv=None):
> +    def build_image(self, tag, docker_dir, dockerfile, quiet=True, argv=None):
>          if argv == None:
>              argv = []
> -        tmp_dir = tempfile.mkdtemp(prefix="docker_build")
>  
> -        tmp_df = tempfile.NamedTemporaryFile(dir=tmp_dir, suffix=".docker")
> +        tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker")
>          tmp_df.write(dockerfile)
>  
>          tmp_df.write("\n")
>          tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
>                       _text_checksum(dockerfile))
>          tmp_df.flush()
> +
>          self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
> -                 [tmp_dir],
> +                 [docker_dir],
>                   quiet=quiet)
>  
>      def image_matches_dockerfile(self, tag, dockerfile):
> @@ -164,10 +164,15 @@ class BuildCommand(SubCommand):
>          if dkr.image_matches_dockerfile(tag, dockerfile):
>              if not args.quiet:
>                  print "Image is up to date."
> -            return 0
> +        else:
> +            # Create a docker context directory for the build
> +            docker_dir = tempfile.mkdtemp(prefix="docker_build")
> +
> +            dkr.build_image(tag, docker_dir, dockerfile,
> +                            quiet=args.quiet, argv=argv)
> +
> +            rmtree(docker_dir)

Maybe we could use a "finally" here to make sure docker_dir is always removed,
but that is not a duty of this patch.

Reviewed-by: Fam Zheng <famz@redhat.com>

>  
> -        dkr.build_image(tag, dockerfile, args.dockerfile,
> -                        quiet=args.quiet, argv=argv)
>          return 0
>  
>  class CleanCommand(SubCommand):
> -- 
> 2.7.4
> 

Re: [Qemu-devel] [PATCH v2 2/3] tests/docker/docker.py: support --include-executable

[Fam Zheng]

On Wed, 06/08 17:35, Alex Bennée wrote:
> When passed the path to a binary we copy it and any linked libraries
> into the docker build context. These can then be included by a
> dockerfile with the line:
> 
>   # Copy all of context into container
>   ADD . /
> 
> This is mainly intended for setting up foreign architecture docker
> images which use qemu-$arch to do cross-architecture linux-user
> execution. It also relies on the host and guest file-system following
> reasonable multi-arch layouts so the copied libraries don't clash with
> the guest ones.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v2
>   - change name of option
>   - require full path to executable
>   - clean-up the copy code
> ---
>  tests/docker/docker.py | 42 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
> 
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index ae40bb3..ed6fa45 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -20,6 +20,7 @@ import atexit
>  import uuid
>  import argparse
>  import tempfile
> +import re
>  from shutil import copy, rmtree
>  
>  def _text_checksum(text):
> @@ -38,6 +39,38 @@ def _guess_docker_command():
>      raise Exception("Cannot find working docker command. Tried:\n%s" % \
>                      commands_txt)
>  
> +def _copy_with_mkdir(src, root_dir, sub_path):
> +    """Copy src into root_dir, creating sub_path as needed."""
> +    dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
> +    try:
> +        os.makedirs(dest_dir)
> +    except OSError:
> +        print "%s already created" % (dest_dir)

Is this error necessary? If it doesn't hurt (i.e. multiple libraries are copied
into it), I think we can safely say "pass". Or, add "if not
os.path.isdir(dest_dir):" above os.makedirs.

> +
> +    dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
> +    copy(src, dest_file)
> +
> +
> +def _copy_binary_with_libs(src, dest_dir):
> +    """Copy a binary executable and all its dependant libraries.
> +
> +    This does rely on the host file-system being fairly multi-arch
> +    aware so the file don't clash with the guests layout."""
> +
> +    _copy_with_mkdir(src, dest_dir, "/usr/bin")
> +
> +    # do ldd bit here
> +    ldd_re = re.compile(r"(/.*/)(\S*)")
> +    ldd_output = subprocess.check_output(["ldd", src])
> +    for line in ldd_output.split("\n"):
> +        search = ldd_re.search(line)
> +        if search and len(search.groups()) == 2:
> +            so_path = search.groups()[0]
> +            so_lib = search.groups()[1]
> +            _copy_with_mkdir("%s/%s" % (so_path, so_lib),
> +                             dest_dir, so_path)
> +
> +
>  class Docker(object):
>      """ Running Docker commands """
>      def __init__(self):
> @@ -151,6 +184,10 @@ class BuildCommand(SubCommand):
>      """ Build docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
>      name = "build"
>      def args(self, parser):
> +        parser.add_argument("--include-executable", "-e",
> +                            help="""Specify a binary that will be copied to the
> +                            container together with all its dependent
> +                            libraries""")
>          parser.add_argument("tag",
>                              help="Image Tag")
>          parser.add_argument("dockerfile",
> @@ -168,6 +205,11 @@ class BuildCommand(SubCommand):
>              # Create a docker context directory for the build
>              docker_dir = tempfile.mkdtemp(prefix="docker_build")
>  
> +            # Do we include a extra binary?

s/a extra/an extra/

> +            if args.include_executable:
> +                _copy_binary_with_libs(args.include_executable,
> +                                       docker_dir)
> +
>              dkr.build_image(tag, docker_dir, dockerfile,
>                              quiet=args.quiet, argv=argv)
>  
> -- 
> 2.7.4
> 

Re: [Qemu-devel] [PATCH v2 3/3] add debian-bootstrap.docker target (and pre script)

[Fam Zheng]

On Wed, 06/08 17:35, Alex Bennée wrote:
> Together with the debian-bootstrap.pre script can now build an arbitrary
> architecture of Debian using debootstrap. The docker script will now
> search for an associated $dockerfile.pre script which gets run in the
> same build context as the dockerfile will be. This allows debootstrap to
> set up its first stage before the container is built.
> 
> To build a container you need a command line like:
> 
>   DEB_ARCH=armhf DEB_TYPE=testing \
>     ./tests/docker/docker.py build \
>     --include-executable=arm-linux-user/qemu-arm debian:armhf \
>     ./tests/docker/dockerfiles/debian-bootstrap.docker
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v2
>   - use .pre script instead of embedded HOST_CMD
>   - make default image include all QEMU build-deps
> ---
>  tests/docker/docker.py                           |  5 +++++

Could you split the docker.py into a separate patch?

>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 +++++++++++++++++++++
>  tests/docker/dockerfiles/debian-bootstrap.pre    |  5 +++++
>  3 files changed, 31 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
> 
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index ed6fa45..43041eb 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -205,6 +205,11 @@ class BuildCommand(SubCommand):
>              # Create a docker context directory for the build
>              docker_dir = tempfile.mkdtemp(prefix="docker_build")
>  
> +            # Is there a .pre file to run in the build context?
> +            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
> +            if os.path.exists(docker_pre):
> +                subprocess.call(os.path.realpath(docker_pre), cwd=docker_dir)

We should probably abort here if exit code is non-zero. Use
subprocess.check_call?

Fam

> +
>              # Do we include a extra binary?
>              if args.include_executable:
>                  _copy_binary_with_libs(args.include_executable,
> diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
> new file mode 100644
> index 0000000..3a9125e
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-bootstrap.docker
> @@ -0,0 +1,21 @@
> +# Create Debian Bootstrap Image
> +#
> +# This is intended to be pre-poluated by:
> +#  - a first stage debootstrap (see debian-bootstrap.pre)
> +#  - a native qemu-$arch that binfmt_misc will run
> +FROM scratch
> +
> +# Add everything from the context into the container
> +ADD . /
> +
> +# Patch all mounts as docker already has stuff set up
> +RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
> +
> +# Run stage 2
> +RUN /debootstrap/debootstrap --second-stage
> +
> +# At this point we can install additional packages if we want
> +# Duplicate deb line as deb-src
> +RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
> +RUN apt-get update
> +RUN apt-get -y build-dep qemu
> diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
> new file mode 100755
> index 0000000..6f42da6
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-bootstrap.pre
> @@ -0,0 +1,5 @@
> +#!/bin/sh
> +#
> +# Simple wrapper for debootstrap, run in the docker build context
> +#
> +fakeroot debootstrap --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian
> -- 
> 2.7.4
> 

Re: [Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Fam Zheng]

On Wed, 06/08 17:35, Alex Bennée wrote:
> Hi,
> 
> This is a re-spin of the previous series built on top of
> fam/docker.next. I've made the changes suggested in the last review
> and split the first patch apart to separate (and fix) the build
> directory changes first.
> 
> Now it no longer messes with the docker file you can actually
> cross-build tests. First ensure you build the debian-bootstrap image:
> 
>     DEB_ARCH=armhf DEB_TYPE=testing \
>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>       --include-executable=./arm-linux-user/qemu-arm

This can take long depending on the network - I had to explicitly use a close
mirror in the pre script to test this, not sure why the mirror redirector
doesn't work. Eventually I get this error once the pre script succeeds and
docker build starts:

Sending build context to Docker daemon 235.1 MB
Step 1 : FROM scratch
 ---> 
Step 2 : ADD . /
 ---> 807bfa810b0c
Removing intermediate container e57ded00b227
Step 3 : RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
 ---> Running in fd80232b38fc
rpc error: code = 2 desc = "oci runtime error: exec format error"

IIUC the sed is an armhf binary? Is something missing in the docker file? I
cannot find where the copied qemu-arm is used...

Fam

> 
> And then run the test quick target:
> 
>     make docker-test-quick@debian-bootstrap J=9 V=1
> 
> I'll leave it up to you how we cleanly integrate multi-arch builds
> into the Make system ;-)
> 
> Alex Bennée (3):
>   tests/docker/docker.py: docker_dir outside build
>   tests/docker/docker.py: support --include-executable
>   add debian-bootstrap.docker target (and pre script)
> 
>  tests/docker/docker.py                           | 68 +++++++++++++++++++++---
>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++++
>  tests/docker/dockerfiles/debian-bootstrap.pre    |  5 ++
>  3 files changed, 86 insertions(+), 8 deletions(-)
>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
> 
> -- 
> 2.7.4
> 

Re: [Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Wed, 06/08 17:35, Alex Bennée wrote:
>> Hi,
>>
>> This is a re-spin of the previous series built on top of
>> fam/docker.next. I've made the changes suggested in the last review
>> and split the first patch apart to separate (and fix) the build
>> directory changes first.
>>
>> Now it no longer messes with the docker file you can actually
>> cross-build tests. First ensure you build the debian-bootstrap image:
>>
>>     DEB_ARCH=armhf DEB_TYPE=testing \
>>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>>       --include-executable=./arm-linux-user/qemu-arm
>
> This can take long depending on the network - I had to explicitly use a close
> mirror in the pre script to test this, not sure why the mirror redirector
> doesn't work. Eventually I get this error once the pre script succeeds and
> docker build starts:
>
> Sending build context to Docker daemon 235.1 MB
> Step 1 : FROM scratch
>  --->
> Step 2 : ADD . /
>  ---> 807bfa810b0c
> Removing intermediate container e57ded00b227
> Step 3 : RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
>  ---> Running in fd80232b38fc
> rpc error: code = 2 desc = "oci runtime error: exec format error"
>
> IIUC the sed is an armhf binary? Is something missing in the docker file? I
> cannot find where the copied qemu-arm is used...

Ahh I realise I missed out the implicit binfmt_misc needs to be set up.
So on my Ubuntu system just having qemu-user installed means the host
system binfmt_misc is set up for armhf binaries to run
/usr/bin/qemu-arm. I haven't experimented with explicitly setting up
binfmt_misc in the container because it was already working :-/

>
> Fam
>
>>
>> And then run the test quick target:
>>
>>     make docker-test-quick@debian-bootstrap J=9 V=1
>>
>> I'll leave it up to you how we cleanly integrate multi-arch builds
>> into the Make system ;-)
>>
>> Alex Bennée (3):
>>   tests/docker/docker.py: docker_dir outside build
>>   tests/docker/docker.py: support --include-executable
>>   add debian-bootstrap.docker target (and pre script)
>>
>>  tests/docker/docker.py                           | 68 +++++++++++++++++++++---
>>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++++
>>  tests/docker/dockerfiles/debian-bootstrap.pre    |  5 ++
>>  3 files changed, 86 insertions(+), 8 deletions(-)
>>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
>>
>> --
>> 2.7.4
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 3/3] add debian-bootstrap.docker target (and pre script)

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Wed, 06/08 17:35, Alex Bennée wrote:
>> Together with the debian-bootstrap.pre script can now build an arbitrary
>> architecture of Debian using debootstrap. The docker script will now
>> search for an associated $dockerfile.pre script which gets run in the
>> same build context as the dockerfile will be. This allows debootstrap to
>> set up its first stage before the container is built.
>>
>> To build a container you need a command line like:
>>
>>   DEB_ARCH=armhf DEB_TYPE=testing \
>>     ./tests/docker/docker.py build \
>>     --include-executable=arm-linux-user/qemu-arm debian:armhf \
>>     ./tests/docker/dockerfiles/debian-bootstrap.docker
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>
>> ---
>> v2
>>   - use .pre script instead of embedded HOST_CMD
>>   - make default image include all QEMU build-deps
>> ---
>>  tests/docker/docker.py                           |  5 +++++
>
> Could you split the docker.py into a separate patch?

Sure.

>
>>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 +++++++++++++++++++++
>>  tests/docker/dockerfiles/debian-bootstrap.pre    |  5 +++++
>>  3 files changed, 31 insertions(+)
>>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
>>
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index ed6fa45..43041eb 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -205,6 +205,11 @@ class BuildCommand(SubCommand):
>>              # Create a docker context directory for the build
>>              docker_dir = tempfile.mkdtemp(prefix="docker_build")
>>
>> +            # Is there a .pre file to run in the build context?
>> +            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
>> +            if os.path.exists(docker_pre):
>> +                subprocess.call(os.path.realpath(docker_pre), cwd=docker_dir)
>
> We should probably abort here if exit code is non-zero. Use
> subprocess.check_call?

OK.

>
> Fam
>
>> +
>>              # Do we include a extra binary?
>>              if args.include_executable:
>>                  _copy_binary_with_libs(args.include_executable,
>> diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
>> new file mode 100644
>> index 0000000..3a9125e
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/debian-bootstrap.docker
>> @@ -0,0 +1,21 @@
>> +# Create Debian Bootstrap Image
>> +#
>> +# This is intended to be pre-poluated by:
>> +#  - a first stage debootstrap (see debian-bootstrap.pre)
>> +#  - a native qemu-$arch that binfmt_misc will run
>> +FROM scratch
>> +
>> +# Add everything from the context into the container
>> +ADD . /
>> +
>> +# Patch all mounts as docker already has stuff set up
>> +RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
>> +
>> +# Run stage 2
>> +RUN /debootstrap/debootstrap --second-stage
>> +
>> +# At this point we can install additional packages if we want
>> +# Duplicate deb line as deb-src
>> +RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>> +RUN apt-get update
>> +RUN apt-get -y build-dep qemu
>> diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
>> new file mode 100755
>> index 0000000..6f42da6
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/debian-bootstrap.pre
>> @@ -0,0 +1,5 @@
>> +#!/bin/sh
>> +#
>> +# Simple wrapper for debootstrap, run in the docker build context
>> +#
>> +fakeroot debootstrap --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian
>> --
>> 2.7.4
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 2/3] tests/docker/docker.py: support --include-executable

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Wed, 06/08 17:35, Alex Bennée wrote:
>> When passed the path to a binary we copy it and any linked libraries
>> into the docker build context. These can then be included by a
>> dockerfile with the line:
>>
>>   # Copy all of context into container
>>   ADD . /
>>
>> This is mainly intended for setting up foreign architecture docker
>> images which use qemu-$arch to do cross-architecture linux-user
>> execution. It also relies on the host and guest file-system following
>> reasonable multi-arch layouts so the copied libraries don't clash with
>> the guest ones.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>
>> ---
>> v2
>>   - change name of option
>>   - require full path to executable
>>   - clean-up the copy code
>> ---
>>  tests/docker/docker.py | 42 ++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 42 insertions(+)
>>
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index ae40bb3..ed6fa45 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -20,6 +20,7 @@ import atexit
>>  import uuid
>>  import argparse
>>  import tempfile
>> +import re
>>  from shutil import copy, rmtree
>>
>>  def _text_checksum(text):
>> @@ -38,6 +39,38 @@ def _guess_docker_command():
>>      raise Exception("Cannot find working docker command. Tried:\n%s" % \
>>                      commands_txt)
>>
>> +def _copy_with_mkdir(src, root_dir, sub_path):
>> +    """Copy src into root_dir, creating sub_path as needed."""
>> +    dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
>> +    try:
>> +        os.makedirs(dest_dir)
>> +    except OSError:
>> +        print "%s already created" % (dest_dir)
>
> Is this error necessary? If it doesn't hurt (i.e. multiple libraries are copied
> into it), I think we can safely say "pass". Or, add "if not
> os.path.isdir(dest_dir):" above os.makedirs.


OK.


>
>> +
>> +    dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
>> +    copy(src, dest_file)
>> +
>> +
>> +def _copy_binary_with_libs(src, dest_dir):
>> +    """Copy a binary executable and all its dependant libraries.
>> +
>> +    This does rely on the host file-system being fairly multi-arch
>> +    aware so the file don't clash with the guests layout."""
>> +
>> +    _copy_with_mkdir(src, dest_dir, "/usr/bin")
>> +
>> +    # do ldd bit here
>> +    ldd_re = re.compile(r"(/.*/)(\S*)")
>> +    ldd_output = subprocess.check_output(["ldd", src])
>> +    for line in ldd_output.split("\n"):
>> +        search = ldd_re.search(line)
>> +        if search and len(search.groups()) == 2:
>> +            so_path = search.groups()[0]
>> +            so_lib = search.groups()[1]
>> +            _copy_with_mkdir("%s/%s" % (so_path, so_lib),
>> +                             dest_dir, so_path)
>> +
>> +
>>  class Docker(object):
>>      """ Running Docker commands """
>>      def __init__(self):
>> @@ -151,6 +184,10 @@ class BuildCommand(SubCommand):
>>      """ Build docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
>>      name = "build"
>>      def args(self, parser):
>> +        parser.add_argument("--include-executable", "-e",
>> +                            help="""Specify a binary that will be copied to the
>> +                            container together with all its dependent
>> +                            libraries""")
>>          parser.add_argument("tag",
>>                              help="Image Tag")
>>          parser.add_argument("dockerfile",
>> @@ -168,6 +205,11 @@ class BuildCommand(SubCommand):
>>              # Create a docker context directory for the build
>>              docker_dir = tempfile.mkdtemp(prefix="docker_build")
>>
>> +            # Do we include a extra binary?
>
> s/a extra/an extra/

OK

>
>> +            if args.include_executable:
>> +                _copy_binary_with_libs(args.include_executable,
>> +                                       docker_dir)
>> +
>>              dkr.build_image(tag, docker_dir, dockerfile,
>>                              quiet=args.quiet, argv=argv)
>>
>> --
>> 2.7.4
>>

Thanks

--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Fam Zheng]

On Mon, 06/13 10:22, Alex Bennée wrote:
> 
> Fam Zheng <famz@redhat.com> writes:
> 
> > On Wed, 06/08 17:35, Alex Bennée wrote:
> >> Hi,
> >>
> >> This is a re-spin of the previous series built on top of
> >> fam/docker.next. I've made the changes suggested in the last review
> >> and split the first patch apart to separate (and fix) the build
> >> directory changes first.
> >>
> >> Now it no longer messes with the docker file you can actually
> >> cross-build tests. First ensure you build the debian-bootstrap image:
> >>
> >>     DEB_ARCH=armhf DEB_TYPE=testing \
> >>       ./tests/docker/docker.py build qemu:debian-bootstrap \
> >>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
> >>       --include-executable=./arm-linux-user/qemu-arm
> >
> > This can take long depending on the network - I had to explicitly use a close
> > mirror in the pre script to test this, not sure why the mirror redirector
> > doesn't work. Eventually I get this error once the pre script succeeds and
> > docker build starts:
> >
> > Sending build context to Docker daemon 235.1 MB
> > Step 1 : FROM scratch
> >  --->
> > Step 2 : ADD . /
> >  ---> 807bfa810b0c
> > Removing intermediate container e57ded00b227
> > Step 3 : RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
> >  ---> Running in fd80232b38fc
> > rpc error: code = 2 desc = "oci runtime error: exec format error"
> >
> > IIUC the sed is an armhf binary? Is something missing in the docker file? I
> > cannot find where the copied qemu-arm is used...
> 
> Ahh I realise I missed out the implicit binfmt_misc needs to be set up.
> So on my Ubuntu system just having qemu-user installed means the host
> system binfmt_misc is set up for armhf binaries to run
> /usr/bin/qemu-arm. I haven't experimented with explicitly setting up
> binfmt_misc in the container because it was already working :-/

A question related to this: I assume if we add a debian image it has everything
to initialize the debootstrap? Then is it possible to convert everything in the
pre script into Dockerfile RUN directives? That way it's more self-contained,
and can benefit from the docker checksum trick provided in docker.py.

Not that I don't like the pre script in any way (actually I am imagining using
a pre script to bootstrap an osx cross build image).

Fam

Re: [Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Riku Voipio]

On 13 June 2016 at 12:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Fam Zheng <famz@redhat.com> writes:
>
>> On Wed, 06/08 17:35, Alex Bennée wrote:
>>> Hi,
>>>
>>> This is a re-spin of the previous series built on top of
>>> fam/docker.next. I've made the changes suggested in the last review
>>> and split the first patch apart to separate (and fix) the build
>>> directory changes first.
>>>
>>> Now it no longer messes with the docker file you can actually
>>> cross-build tests. First ensure you build the debian-bootstrap image:
>>>
>>>     DEB_ARCH=armhf DEB_TYPE=testing \
>>>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>>>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>>>       --include-executable=./arm-linux-user/qemu-arm
>>
>> This can take long depending on the network - I had to explicitly use a close
>> mirror in the pre script to test this, not sure why the mirror redirector
>> doesn't work. Eventually I get this error once the pre script succeeds and
>> docker build starts:
>>
>> Sending build context to Docker daemon 235.1 MB
>> Step 1 : FROM scratch
>>  --->
>> Step 2 : ADD . /
>>  ---> 807bfa810b0c
>> Removing intermediate container e57ded00b227
>> Step 3 : RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
>>  ---> Running in fd80232b38fc
>> rpc error: code = 2 desc = "oci runtime error: exec format error"
>>
>> IIUC the sed is an armhf binary? Is something missing in the docker file? I
>> cannot find where the copied qemu-arm is used...
>
> Ahh I realise I missed out the implicit binfmt_misc needs to be set up.
> So on my Ubuntu system just having qemu-user installed means the host
> system binfmt_misc is set up for armhf binaries to run
> /usr/bin/qemu-arm. I haven't experimented with explicitly setting up
> binfmt_misc in the container because it was already working :-/

The containment leaks here. If your host has registered binfmt_misc,
identical settings will be visible in container. You can't change the
settings inside docker, unless you start with --privileged. And
changing binfmt_misc settings inside docker will also change the
settings in the host. So one needs to tiptoe around setting it up...

>>
>> Fam
>>
>>>
>>> And then run the test quick target:
>>>
>>>     make docker-test-quick@debian-bootstrap J=9 V=1
>>>
>>> I'll leave it up to you how we cleanly integrate multi-arch builds
>>> into the Make system ;-)
>>>
>>> Alex Bennée (3):
>>>   tests/docker/docker.py: docker_dir outside build
>>>   tests/docker/docker.py: support --include-executable
>>>   add debian-bootstrap.docker target (and pre script)
>>>
>>>  tests/docker/docker.py                           | 68 +++++++++++++++++++++---
>>>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++++
>>>  tests/docker/dockerfiles/debian-bootstrap.pre    |  5 ++
>>>  3 files changed, 86 insertions(+), 8 deletions(-)
>>>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>>>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
>>>
>>> --
>>> 2.7.4
>>>
>
>
> --
> Alex Bennée

Re: [Qemu-devel] [PATCH v2 2/3] tests/docker/docker.py: support --include-executable

[Riku Voipio]

On 8 June 2016 at 19:35, Alex Bennée <alex.bennee@linaro.org> wrote:
> When passed the path to a binary we copy it and any linked libraries
> into the docker build context. These can then be included by a
> dockerfile with the line:
>
>   # Copy all of context into container
>   ADD . /
>
> This is mainly intended for setting up foreign architecture docker
> images which use qemu-$arch to do cross-architecture linux-user
> execution. It also relies on the host and guest file-system following
> reasonable multi-arch layouts so the copied libraries don't clash with
> the guest ones.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> ---
> v2
>   - change name of option
>   - require full path to executable
>   - clean-up the copy code
> ---
>  tests/docker/docker.py | 42 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index ae40bb3..ed6fa45 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -20,6 +20,7 @@ import atexit
>  import uuid
>  import argparse
>  import tempfile
> +import re
>  from shutil import copy, rmtree
>
>  def _text_checksum(text):
> @@ -38,6 +39,38 @@ def _guess_docker_command():
>      raise Exception("Cannot find working docker command. Tried:\n%s" % \
>                      commands_txt)
>
> +def _copy_with_mkdir(src, root_dir, sub_path):
> +    """Copy src into root_dir, creating sub_path as needed."""
> +    dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
> +    try:
> +        os.makedirs(dest_dir)
> +    except OSError:
> +        print "%s already created" % (dest_dir)
> +
> +    dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
> +    copy(src, dest_file)
> +
> +
> +def _copy_binary_with_libs(src, dest_dir):
> +    """Copy a binary executable and all its dependant libraries.
> +
> +    This does rely on the host file-system being fairly multi-arch
> +    aware so the file don't clash with the guests layout."""
> +
> +    _copy_with_mkdir(src, dest_dir, "/usr/bin")
> +
> +    # do ldd bit here
> +    ldd_re = re.compile(r"(/.*/)(\S*)")
> +    ldd_output = subprocess.check_output(["ldd", src])

This fails if qemu-user has been built as --static. I think we can
just try, catch and continue.

> +    for line in ldd_output.split("\n"):
> +        search = ldd_re.search(line)
> +        if search and len(search.groups()) == 2:
> +            so_path = search.groups()[0]
> +            so_lib = search.groups()[1]
> +            _copy_with_mkdir("%s/%s" % (so_path, so_lib),
> +                             dest_dir, so_path)
> +
> +
>  class Docker(object):
>      """ Running Docker commands """
>      def __init__(self):
> @@ -151,6 +184,10 @@ class BuildCommand(SubCommand):
>      """ Build docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
>      name = "build"
>      def args(self, parser):
> +        parser.add_argument("--include-executable", "-e",
> +                            help="""Specify a binary that will be copied to the
> +                            container together with all its dependent
> +                            libraries""")
>          parser.add_argument("tag",
>                              help="Image Tag")
>          parser.add_argument("dockerfile",
> @@ -168,6 +205,11 @@ class BuildCommand(SubCommand):
>              # Create a docker context directory for the build
>              docker_dir = tempfile.mkdtemp(prefix="docker_build")
>
> +            # Do we include a extra binary?
> +            if args.include_executable:
> +                _copy_binary_with_libs(args.include_executable,
> +                                       docker_dir)
> +
>              dkr.build_image(tag, docker_dir, dockerfile,
>                              quiet=args.quiet, argv=argv)
>
> --
> 2.7.4
>

Re: [Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Alex Bennée]

Riku Voipio <riku.voipio@linaro.org> writes:

> On 13 June 2016 at 12:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Fam Zheng <famz@redhat.com> writes:
>>
>>> On Wed, 06/08 17:35, Alex Bennée wrote:
>>>> Hi,
>>>>
>>>> This is a re-spin of the previous series built on top of
>>>> fam/docker.next. I've made the changes suggested in the last review
>>>> and split the first patch apart to separate (and fix) the build
>>>> directory changes first.
>>>>
>>>> Now it no longer messes with the docker file you can actually
>>>> cross-build tests. First ensure you build the debian-bootstrap image:
>>>>
>>>>     DEB_ARCH=armhf DEB_TYPE=testing \
>>>>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>>>>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>>>>       --include-executable=./arm-linux-user/qemu-arm
>>>
>>> This can take long depending on the network - I had to explicitly use a close
>>> mirror in the pre script to test this, not sure why the mirror redirector
>>> doesn't work. Eventually I get this error once the pre script succeeds and
>>> docker build starts:
>>>
>>> Sending build context to Docker daemon 235.1 MB
>>> Step 1 : FROM scratch
>>>  --->
>>> Step 2 : ADD . /
>>>  ---> 807bfa810b0c
>>> Removing intermediate container e57ded00b227
>>> Step 3 : RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
>>>  ---> Running in fd80232b38fc
>>> rpc error: code = 2 desc = "oci runtime error: exec format error"
>>>
>>> IIUC the sed is an armhf binary? Is something missing in the docker file? I
>>> cannot find where the copied qemu-arm is used...
>>
>> Ahh I realise I missed out the implicit binfmt_misc needs to be set up.
>> So on my Ubuntu system just having qemu-user installed means the host
>> system binfmt_misc is set up for armhf binaries to run
>> /usr/bin/qemu-arm. I haven't experimented with explicitly setting up
>> binfmt_misc in the container because it was already working :-/
>
> The containment leaks here. If your host has registered binfmt_misc,
> identical settings will be visible in container. You can't change the
> settings inside docker, unless you start with --privileged. And
> changing binfmt_misc settings inside docker will also change the
> settings in the host. So one needs to tiptoe around setting it up...

Hmm well that makes it interesting. FWIW I'd stick to setting up binfmt
outside the container and defaulting to the "just works" setup depending
on how $HOST sets up binfmt_misc. The original patch actually queried
binfmt_misc to work out the exact path it needed to install qemu-arm in
the $GUEST container. For now we just default to /usr/bin which works on
Debian-types without problem.

>
>>>
>>> Fam
>>>
>>>>
>>>> And then run the test quick target:
>>>>
>>>>     make docker-test-quick@debian-bootstrap J=9 V=1
>>>>
>>>> I'll leave it up to you how we cleanly integrate multi-arch builds
>>>> into the Make system ;-)
>>>>
>>>> Alex Bennée (3):
>>>>   tests/docker/docker.py: docker_dir outside build
>>>>   tests/docker/docker.py: support --include-executable
>>>>   add debian-bootstrap.docker target (and pre script)
>>>>
>>>>  tests/docker/docker.py                           | 68 +++++++++++++++++++++---
>>>>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++++
>>>>  tests/docker/dockerfiles/debian-bootstrap.pre    |  5 ++
>>>>  3 files changed, 86 insertions(+), 8 deletions(-)
>>>>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>>>>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
>>>>
>>>> --
>>>> 2.7.4
>>>>
>>
>>
>> --
>> Alex Bennée


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Riku Voipio]

On 13 June 2016 at 15:30, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Riku Voipio <riku.voipio@linaro.org> writes:
>
>> On 13 June 2016 at 12:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>> Fam Zheng <famz@redhat.com> writes:
>>>
>>>> On Wed, 06/08 17:35, Alex Bennée wrote:
>>>>> Hi,
>>>>>
>>>>> This is a re-spin of the previous series built on top of
>>>>> fam/docker.next. I've made the changes suggested in the last review
>>>>> and split the first patch apart to separate (and fix) the build
>>>>> directory changes first.
>>>>>
>>>>> Now it no longer messes with the docker file you can actually
>>>>> cross-build tests. First ensure you build the debian-bootstrap image:
>>>>>
>>>>>     DEB_ARCH=armhf DEB_TYPE=testing \
>>>>>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>>>>>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>>>>>       --include-executable=./arm-linux-user/qemu-arm
>>>>
>>>> This can take long depending on the network - I had to explicitly use a close
>>>> mirror in the pre script to test this, not sure why the mirror redirector
>>>> doesn't work. Eventually I get this error once the pre script succeeds and
>>>> docker build starts:
>>>>
>>>> Sending build context to Docker daemon 235.1 MB
>>>> Step 1 : FROM scratch
>>>>  --->
>>>> Step 2 : ADD . /
>>>>  ---> 807bfa810b0c
>>>> Removing intermediate container e57ded00b227
>>>> Step 3 : RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
>>>>  ---> Running in fd80232b38fc
>>>> rpc error: code = 2 desc = "oci runtime error: exec format error"
>>>>
>>>> IIUC the sed is an armhf binary? Is something missing in the docker file? I
>>>> cannot find where the copied qemu-arm is used...
>>>
>>> Ahh I realise I missed out the implicit binfmt_misc needs to be set up.
>>> So on my Ubuntu system just having qemu-user installed means the host
>>> system binfmt_misc is set up for armhf binaries to run
>>> /usr/bin/qemu-arm. I haven't experimented with explicitly setting up
>>> binfmt_misc in the container because it was already working :-/
>>
>> The containment leaks here. If your host has registered binfmt_misc,
>> identical settings will be visible in container. You can't change the
>> settings inside docker, unless you start with --privileged. And
>> changing binfmt_misc settings inside docker will also change the
>> settings in the host. So one needs to tiptoe around setting it up...
>
> Hmm well that makes it interesting. FWIW I'd stick to setting up binfmt
> outside the container and defaulting to the "just works" setup depending
> on how $HOST sets up binfmt_misc. The original patch actually queried
> binfmt_misc to work out the exact path it needed to install qemu-arm in
> the $GUEST container. For now we just default to /usr/bin which works on
> Debian-types without problem.

For some value of without problem - you have qemu-user installed, so
you have /usr/bin/qemu-arm registered, while I have qemu-user-static
installed and thus /usr/bin/qemu-arm-static registered.

We have the ./scripts/qemu-binfmt-conf.sh we could run if no binfmt's
appear registered, or at least print an error message for the enduser
to run it.

>>
>>>>
>>>> Fam
>>>>
>>>>>
>>>>> And then run the test quick target:
>>>>>
>>>>>     make docker-test-quick@debian-bootstrap J=9 V=1
>>>>>
>>>>> I'll leave it up to you how we cleanly integrate multi-arch builds
>>>>> into the Make system ;-)
>>>>>
>>>>> Alex Bennée (3):
>>>>>   tests/docker/docker.py: docker_dir outside build
>>>>>   tests/docker/docker.py: support --include-executable
>>>>>   add debian-bootstrap.docker target (and pre script)
>>>>>
>>>>>  tests/docker/docker.py                           | 68 +++++++++++++++++++++---
>>>>>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++++
>>>>>  tests/docker/dockerfiles/debian-bootstrap.pre    |  5 ++
>>>>>  3 files changed, 86 insertions(+), 8 deletions(-)
>>>>>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>>>>>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
>>>>>
>>>>> --
>>>>> 2.7.4
>>>>>
>>>
>>>
>>> --
>>> Alex Bennée
>
>
> --
> Alex Bennée

Re: [Qemu-devel] [PATCH v2 0/3] Support building qemu-user powered docker test images

[Alex Bennée]

Riku Voipio <riku.voipio@linaro.org> writes:

> On 13 June 2016 at 15:30, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Riku Voipio <riku.voipio@linaro.org> writes:
>>
>>> On 13 June 2016 at 12:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>>
>>>> Fam Zheng <famz@redhat.com> writes:
>>>>
>>>>> On Wed, 06/08 17:35, Alex Bennée wrote:
>>>>>> Hi,
>>>>>>
>>>>>> This is a re-spin of the previous series built on top of
>>>>>> fam/docker.next. I've made the changes suggested in the last review
>>>>>> and split the first patch apart to separate (and fix) the build
>>>>>> directory changes first.
>>>>>>
>>>>>> Now it no longer messes with the docker file you can actually
>>>>>> cross-build tests. First ensure you build the debian-bootstrap image:
>>>>>>
>>>>>>     DEB_ARCH=armhf DEB_TYPE=testing \
>>>>>>       ./tests/docker/docker.py build qemu:debian-bootstrap \
>>>>>>       ./tests/docker/dockerfiles/debian-bootstrap.docker \
>>>>>>       --include-executable=./arm-linux-user/qemu-arm
>>>>>
>>>>> This can take long depending on the network - I had to explicitly use a close
>>>>> mirror in the pre script to test this, not sure why the mirror redirector
>>>>> doesn't work. Eventually I get this error once the pre script succeeds and
>>>>> docker build starts:
>>>>>
>>>>> Sending build context to Docker daemon 235.1 MB
>>>>> Step 1 : FROM scratch
>>>>>  --->
>>>>> Step 2 : ADD . /
>>>>>  ---> 807bfa810b0c
>>>>> Removing intermediate container e57ded00b227
>>>>> Step 3 : RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
>>>>>  ---> Running in fd80232b38fc
>>>>> rpc error: code = 2 desc = "oci runtime error: exec format error"
>>>>>
>>>>> IIUC the sed is an armhf binary? Is something missing in the docker file? I
>>>>> cannot find where the copied qemu-arm is used...
>>>>
>>>> Ahh I realise I missed out the implicit binfmt_misc needs to be set up.
>>>> So on my Ubuntu system just having qemu-user installed means the host
>>>> system binfmt_misc is set up for armhf binaries to run
>>>> /usr/bin/qemu-arm. I haven't experimented with explicitly setting up
>>>> binfmt_misc in the container because it was already working :-/
>>>
>>> The containment leaks here. If your host has registered binfmt_misc,
>>> identical settings will be visible in container. You can't change the
>>> settings inside docker, unless you start with --privileged. And
>>> changing binfmt_misc settings inside docker will also change the
>>> settings in the host. So one needs to tiptoe around setting it up...
>>
>> Hmm well that makes it interesting. FWIW I'd stick to setting up binfmt
>> outside the container and defaulting to the "just works" setup depending
>> on how $HOST sets up binfmt_misc. The original patch actually queried
>> binfmt_misc to work out the exact path it needed to install qemu-arm in
>> the $GUEST container. For now we just default to /usr/bin which works on
>> Debian-types without problem.
>
> For some value of without problem - you have qemu-user installed, so
> you have /usr/bin/qemu-arm registered, while I have qemu-user-static
> installed and thus /usr/bin/qemu-arm-static registered.
>
> We have the ./scripts/qemu-binfmt-conf.sh we could run if no binfmt's
> appear registered, or at least print an error message for the enduser
> to run it.

I noticed that has been updated recently. I shall have a look and see if
I can make the pre-script return an error if the binfmt conditions
aren't met. Hopefully it doesn't require root permissions to check all
this out ;-)

>
>>>
>>>>>
>>>>> Fam
>>>>>
>>>>>>
>>>>>> And then run the test quick target:
>>>>>>
>>>>>>     make docker-test-quick@debian-bootstrap J=9 V=1
>>>>>>
>>>>>> I'll leave it up to you how we cleanly integrate multi-arch builds
>>>>>> into the Make system ;-)
>>>>>>
>>>>>> Alex Bennée (3):
>>>>>>   tests/docker/docker.py: docker_dir outside build
>>>>>>   tests/docker/docker.py: support --include-executable
>>>>>>   add debian-bootstrap.docker target (and pre script)
>>>>>>
>>>>>>  tests/docker/docker.py                           | 68 +++++++++++++++++++++---
>>>>>>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++++
>>>>>>  tests/docker/dockerfiles/debian-bootstrap.pre    |  5 ++
>>>>>>  3 files changed, 86 insertions(+), 8 deletions(-)
>>>>>>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>>>>>>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
>>>>>>
>>>>>> --
>>>>>> 2.7.4
>>>>>>
>>>>
>>>>
>>>> --
>>>> Alex Bennée
>>
>>
>> --
>> Alex Bennée


--
Alex Bennée