[Qemu-devel] [PATCH 0/6] Expand the building of qemu-user docker images

[Qemu-devel] [PATCH 0/6] Expand the building of qemu-user docker images

[Alex Bennée]

Hi Fam,

These are a little bit of a work in progress so maybe not all ready to
be taken yet although review would be welcome.

This adds some tweaks to allow us to build the cross-arch debian
images with direct make invocations and then see which images we can
test against from the "make docker" help text. Now:

  make docker-image-debian-stable-arm64

Will do exactly as you expect. I can't get:

  make docker-test-quick@debian-stable-arm64

To work but I suspect this needs some fixing to the generated rules.

Comments?

Alex Bennée (6):
  tests/docker/Makefile.include: fix diff-index call
  tests/docker/test-user: a simple linux-user test
  tests/docker: add optional libs to travis.docker
  tests/docker/Makefile: Add a rule for Debian user images
  tests/docker/docker.py: expand images command
  tests/docker/Makefile.include: expand docker help text

 tests/docker/Makefile.include          | 29 ++++++++++++++++++++++++++---
 tests/docker/docker.py                 | 27 ++++++++++++++++++++++-----
 tests/docker/dockerfiles/travis.docker |  9 ++++++++-
 tests/docker/test-user                 | 18 ++++++++++++++++++
 4 files changed, 74 insertions(+), 9 deletions(-)
 create mode 100755 tests/docker/test-user

-- 
2.10.1

[Qemu-devel] [PATCH 1/6] tests/docker/Makefile.include: fix diff-index call

[Alex Bennée]

The whole thing is wrapped inside a call quiet-command as well as being
the actual call taking a --quiet argument so the redirect is
superfluous. For reasons I have yet to determine this also seems to be
causing the source preparation step to skip stashing work tree stuff.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
TODO:
  - properly understand the failure
---
 tests/docker/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 3f15d5a..d91e28b 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -20,7 +20,7 @@ IMAGES ?= %
 # Make archive from git repo $1 to tar.gz $2
 make-archive-maybe = $(if $(wildcard $1/*), \
 	$(call quiet-command, \
-		(cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
+		(cd $1; if git diff-index --quiet HEAD; then \
 			git archive -1 HEAD --format=tar.gz; \
 		else \
 			git archive -1 $$(git stash create) --format=tar.gz; \
-- 
2.10.1

Re: [Qemu-devel] [PATCH 1/6] tests/docker/Makefile.include: fix diff-index call

[Fam Zheng]

On Fri, 10/28 17:33, Alex Bennée wrote:
> The whole thing is wrapped inside a call quiet-command as well as being
> the actual call taking a --quiet argument so the redirect is
> superfluous. For reasons I have yet to determine this also seems to be
> causing the source preparation step to skip stashing work tree stuff.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> TODO:
>   - properly understand the failure

Yep, I don't see the bug on my machine (Fedora 24, git 2.7.4). What about the
removed "--"? Does that make a difference for you?

> ---
>  tests/docker/Makefile.include | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 3f15d5a..d91e28b 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -20,7 +20,7 @@ IMAGES ?= %
>  # Make archive from git repo $1 to tar.gz $2
>  make-archive-maybe = $(if $(wildcard $1/*), \
>  	$(call quiet-command, \
> -		(cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
> +		(cd $1; if git diff-index --quiet HEAD; then \
>  			git archive -1 HEAD --format=tar.gz; \
>  		else \
>  			git archive -1 $$(git stash create) --format=tar.gz; \
> -- 
> 2.10.1
> 
> 

Re: [Qemu-devel] [PATCH 1/6] tests/docker/Makefile.include: fix diff-index call

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Fri, 10/28 17:33, Alex Bennée wrote:
>> The whole thing is wrapped inside a call quiet-command as well as being
>> the actual call taking a --quiet argument so the redirect is
>> superfluous. For reasons I have yet to determine this also seems to be
>> causing the source preparation step to skip stashing work tree stuff.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>
>> ---
>> TODO:
>>   - properly understand the failure
>
> Yep, I don't see the bug on my machine (Fedora 24, git 2.7.4). What about the
> removed "--"? Does that make a difference for you?

The -- seems superfluous as we are not giving it a list of files to
check against. I was seeing the problem with known files with changes
which were not committed.

My setup is Ubuntu 16.04 with git 2.10.1 (I track upstream stable).

>
>> ---
>>  tests/docker/Makefile.include | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 3f15d5a..d91e28b 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -20,7 +20,7 @@ IMAGES ?= %
>>  # Make archive from git repo $1 to tar.gz $2
>>  make-archive-maybe = $(if $(wildcard $1/*), \
>>  	$(call quiet-command, \
>> -		(cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
>> +		(cd $1; if git diff-index --quiet HEAD; then \
>>  			git archive -1 HEAD --format=tar.gz; \
>>  		else \
>>  			git archive -1 $$(git stash create) --format=tar.gz; \
>> --
>> 2.10.1
>>
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH 1/6] tests/docker/Makefile.include: fix diff-index call

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Fri, 10/28 17:33, Alex Bennée wrote:
>> The whole thing is wrapped inside a call quiet-command as well as being
>> the actual call taking a --quiet argument so the redirect is
>> superfluous. For reasons I have yet to determine this also seems to be
>> causing the source preparation step to skip stashing work tree stuff.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>
>> ---
>> TODO:
>>   - properly understand the failure
>
> Yep, I don't see the bug on my machine (Fedora 24, git 2.7.4). What about the
> removed "--"? Does that make a difference for you?

Nope - it seems to be the &>/dev/null that triggers the problem - but
only in the Makefile. Running the commands from the command line works
as expected.

However why do we need the redirect here anyway considering the call
quiet-command?

>
>> ---
>>  tests/docker/Makefile.include | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 3f15d5a..d91e28b 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -20,7 +20,7 @@ IMAGES ?= %
>>  # Make archive from git repo $1 to tar.gz $2
>>  make-archive-maybe = $(if $(wildcard $1/*), \
>>  	$(call quiet-command, \
>> -		(cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
>> +		(cd $1; if git diff-index --quiet HEAD; then \
>>  			git archive -1 HEAD --format=tar.gz; \
>>  		else \
>>  			git archive -1 $$(git stash create) --format=tar.gz; \
>> --
>> 2.10.1
>>
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH 1/6] tests/docker/Makefile.include: fix diff-index call

[Fam Zheng]

On Tue, 11/01 10:02, Alex Bennée wrote:
> 
> Fam Zheng <famz@redhat.com> writes:
> 
> > On Fri, 10/28 17:33, Alex Bennée wrote:
> >> The whole thing is wrapped inside a call quiet-command as well as being
> >> the actual call taking a --quiet argument so the redirect is
> >> superfluous. For reasons I have yet to determine this also seems to be
> >> causing the source preparation step to skip stashing work tree stuff.
> >>
> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >>
> >> ---
> >> TODO:
> >>   - properly understand the failure
> >
> > Yep, I don't see the bug on my machine (Fedora 24, git 2.7.4). What about the
> > removed "--"? Does that make a difference for you?
> 
> Nope - it seems to be the &>/dev/null that triggers the problem - but
> only in the Makefile. Running the commands from the command line works
> as expected.
> 
> However why do we need the redirect here anyway considering the call
> quiet-command?

It's to force suppressing output, while quiet-command only suppresses the
echoing of the command itself. Since we have "--quiet" here, does "2>/dev/null"
work for you?

Fam

> 
> >
> >> ---
> >>  tests/docker/Makefile.include | 2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> >> index 3f15d5a..d91e28b 100644
> >> --- a/tests/docker/Makefile.include
> >> +++ b/tests/docker/Makefile.include
> >> @@ -20,7 +20,7 @@ IMAGES ?= %
> >>  # Make archive from git repo $1 to tar.gz $2
> >>  make-archive-maybe = $(if $(wildcard $1/*), \
> >>  	$(call quiet-command, \
> >> -		(cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
> >> +		(cd $1; if git diff-index --quiet HEAD; then \
> >>  			git archive -1 HEAD --format=tar.gz; \
> >>  		else \
> >>  			git archive -1 $$(git stash create) --format=tar.gz; \
> >> --
> >> 2.10.1
> >>
> >>
> 
> 
> --
> Alex Bennée

Re: [Qemu-devel] [PATCH 1/6] tests/docker/Makefile.include: fix diff-index call

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Tue, 11/01 10:02, Alex Bennée wrote:
>>
>> Fam Zheng <famz@redhat.com> writes:
>>
>> > On Fri, 10/28 17:33, Alex Bennée wrote:
>> >> The whole thing is wrapped inside a call quiet-command as well as being
>> >> the actual call taking a --quiet argument so the redirect is
>> >> superfluous. For reasons I have yet to determine this also seems to be
>> >> causing the source preparation step to skip stashing work tree stuff.
>> >>
>> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >>
>> >> ---
>> >> TODO:
>> >>   - properly understand the failure
>> >
>> > Yep, I don't see the bug on my machine (Fedora 24, git 2.7.4). What about the
>> > removed "--"? Does that make a difference for you?
>>
>> Nope - it seems to be the &>/dev/null that triggers the problem - but
>> only in the Makefile. Running the commands from the command line works
>> as expected.
>>
>> However why do we need the redirect here anyway considering the call
>> quiet-command?
>
> It's to force suppressing output, while quiet-command only suppresses the
> echoing of the command itself. Since we have "--quiet" here, does "2>/dev/null"
> work for you?

That works!

-		(cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
+		(cd $1; if git diff-index --quiet HEAD -- 2>/dev/null; then \


>
> Fam
>
>>
>> >
>> >> ---
>> >>  tests/docker/Makefile.include | 2 +-
>> >>  1 file changed, 1 insertion(+), 1 deletion(-)
>> >>
>> >> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> >> index 3f15d5a..d91e28b 100644
>> >> --- a/tests/docker/Makefile.include
>> >> +++ b/tests/docker/Makefile.include
>> >> @@ -20,7 +20,7 @@ IMAGES ?= %
>> >>  # Make archive from git repo $1 to tar.gz $2
>> >>  make-archive-maybe = $(if $(wildcard $1/*), \
>> >>  	$(call quiet-command, \
>> >> -		(cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
>> >> +		(cd $1; if git diff-index --quiet HEAD; then \
>> >>  			git archive -1 HEAD --format=tar.gz; \
>> >>  		else \
>> >>  			git archive -1 $$(git stash create) --format=tar.gz; \
>> >> --
>> >> 2.10.1
>> >>
>> >>
>>
>>
>> --
>> Alex Bennée


--
Alex Bennée

[Qemu-devel] [PATCH 2/6] tests/docker/test-user: a simple linux-user test

[Alex Bennée]

This is useful for checking an appropriately updated cross-arch docker
actually starts without having to do a full build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/test-user | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
 create mode 100755 tests/docker/test-user

diff --git a/tests/docker/test-user b/tests/docker/test-user
new file mode 100755
index 0000000..4871de7
--- /dev/null
+++ b/tests/docker/test-user
@@ -0,0 +1,18 @@
+#!/bin/bash -e
+#
+# A very simple test that checks qemu-linux-user binaries work
+# correctly in a cross-arch docker container.
+#
+# Copyright (c) 2016 Linaro.
+#
+# Authors:
+#  Alex Bennée <alex.bennee@linaro.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+uname -a
+exit 0
-- 
2.10.1

Re: [Qemu-devel] [PATCH 2/6] tests/docker/test-user: a simple linux-user test

[Fam Zheng]

On Fri, 10/28 17:33, Alex Bennée wrote:
> This is useful for checking an appropriately updated cross-arch docker
> actually starts without having to do a full build.

Can we call it "test-pass"?

Fam

> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/test-user | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
>  create mode 100755 tests/docker/test-user
> 
> diff --git a/tests/docker/test-user b/tests/docker/test-user
> new file mode 100755
> index 0000000..4871de7
> --- /dev/null
> +++ b/tests/docker/test-user
> @@ -0,0 +1,18 @@
> +#!/bin/bash -e
> +#
> +# A very simple test that checks qemu-linux-user binaries work
> +# correctly in a cross-arch docker container.
> +#
> +# Copyright (c) 2016 Linaro.
> +#
> +# Authors:
> +#  Alex Bennée <alex.bennee@linaro.org>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +. common.rc
> +
> +uname -a
> +exit 0
> -- 
> 2.10.1
> 
> 

[Qemu-devel] [PATCH 3/6] tests/docker: add optional libs to travis.docker

[Alex Bennée]

In our .travis.yml we install a bunch of extra libraries which are not
part of the packaged QEMU's builddeps. We include them here to make our
docker container more closely match the state of the system after docker
has installed the rest of the build dependencies.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/travis.docker | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
index e4983ae..6bef722 100644
--- a/tests/docker/dockerfiles/travis.docker
+++ b/tests/docker/dockerfiles/travis.docker
@@ -2,5 +2,12 @@ FROM quay.io/travisci/travis-ruby
 RUN apt-get update
 RUN apt-get -y build-dep qemu
 RUN apt-get -y build-dep device-tree-compiler
-RUN apt-get -y install python2.7 dh-autoreconf
+# Additional optional libs not in QEMU's builddep
+RUN apt-get -y install libbrlapi-dev liblttng-ust-dev libcap-ng-dev \
+                       libnfs-dev libnss3-dev libpixman-1-dev \
+                       libpng12-dev librados-dev libseccomp-dev \
+                       libspice-protocol-dev libspice-server-dev \
+                       libssh2-1-dev liburcu-dev libusb-1.0-0-dev \
+                       libvte-2.90-dev
+RUN apt-get -y install python2.7 dh-autoreconf sparse
 ENV FEATURES pyyaml
-- 
2.10.1

Re: [Qemu-devel] [PATCH 3/6] tests/docker: add optional libs to travis.docker

[Fam Zheng]

On Fri, 10/28 17:33, Alex Bennée wrote:
> In our .travis.yml we install a bunch of extra libraries which are not
> part of the packaged QEMU's builddeps. We include them here to make our
> docker container more closely match the state of the system after docker
> has installed the rest of the build dependencies.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/dockerfiles/travis.docker | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
> index e4983ae..6bef722 100644
> --- a/tests/docker/dockerfiles/travis.docker
> +++ b/tests/docker/dockerfiles/travis.docker
> @@ -2,5 +2,12 @@ FROM quay.io/travisci/travis-ruby
>  RUN apt-get update
>  RUN apt-get -y build-dep qemu
>  RUN apt-get -y build-dep device-tree-compiler
> -RUN apt-get -y install python2.7 dh-autoreconf
> +# Additional optional libs not in QEMU's builddep
> +RUN apt-get -y install libbrlapi-dev liblttng-ust-dev libcap-ng-dev \
> +                       libnfs-dev libnss3-dev libpixman-1-dev \
> +                       libpng12-dev librados-dev libseccomp-dev \
> +                       libspice-protocol-dev libspice-server-dev \
> +                       libssh2-1-dev liburcu-dev libusb-1.0-0-dev \
> +                       libvte-2.90-dev
> +RUN apt-get -y install python2.7 dh-autoreconf sparse

Any reason to use two apt-get lines?

Also, can you put these package names in $PACKAGES and dump 'dpkg -l $PACKAGES'
to /packages.txt like in ubuntu.docker? That way we can print it in reports by
adding SHOW_ENV=1 in make invocation.

Fam

>  ENV FEATURES pyyaml
> -- 
> 2.10.1
> 
> 

[Qemu-devel] [PATCH 4/6] tests/docker/Makefile: Add a rule for Debian user images

[Alex Bennée]

This allows you to create Debian images powered by a qemu linux-user
binary. Later patches will list these in the help file but for now:

  make docker-image-debian-stable-arm64

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index d91e28b..bb9e078 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -64,6 +64,22 @@ $(foreach i,$(DOCKER_IMAGES), \
 	) \
 )
 
+# Additional Debian qemu-user images
+#
+# These images are built using the generic debian-bootstrap image and a qemu-user binary
+#
+docker-image-debian-%: DEB_TYPE = $(shell echo '$@' | sed -e 's/docker-image-debian-\([^@]*\)-\(.*\)/\1/')
+docker-image-debian-%: DEB_ARCH = $(shell echo '$@' | sed -e 's/docker-image-debian-\([^@]*\)-\(.*\)/\2/')
+docker-image-debian-%: QEMU_ARCH = $(subst arm64,aarch64,$(subst armhf,arm,$(DEB_ARCH)))
+docker-image-debian-%: QEMU_EXEC=$(BUILD_DIR)/$(QEMU_ARCH)-linux-user/qemu-$(QEMU_ARCH)
+docker-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
+	$(call quiet-command,\
+		DEB_ARCH=$(DEB_ARCH) DEB_TYPE=$(DEB_TYPE) \
+		$(SRC_PATH)/tests/docker/docker.py build qemu:debian-$* $< \
+		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
+		--include-executable=$(QEMU_EXEC),\
+		"BUILD USER","$*")
+
 docker:
 	@echo 'Build QEMU and run tests inside Docker containers'
 	@echo
-- 
2.10.1

[Qemu-devel] [PATCH 5/6] tests/docker/docker.py: expand images command

[Alex Bennée]

Modern docker cli commands can filter by repository but as we want to
work on all instances do this by hand in python:

   /tests/docker/docker.py images --repo qemu

Will return a space delimited list for the benefit of our Makefile
recipes.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 37d8319..03c0238 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -105,12 +105,15 @@ class Docker(object):
         signal.signal(signal.SIGTERM, self._kill_instances)
         signal.signal(signal.SIGHUP, self._kill_instances)
 
-    def _do(self, cmd, quiet=True, infile=None, **kwargs):
+    def _do(self, cmd, quiet=True, infile=None, output=False, **kwargs):
         if quiet:
             kwargs["stdout"] = DEVNULL
         if infile:
             kwargs["stdin"] = infile
-        return subprocess.call(self._command + cmd, **kwargs)
+        if output:
+            return subprocess.check_output(self._command + cmd, **kwargs)
+        else:
+            return subprocess.call(self._command + cmd, **kwargs)
 
     def _do_kill_instances(self, only_known, only_active=True):
         cmd = ["ps", "-q"]
@@ -188,8 +191,8 @@ class Docker(object):
             self._instances.remove(label)
         return ret
 
-    def command(self, cmd, argv, quiet):
-        return self._do([cmd] + argv, quiet=quiet)
+    def command(self, cmd, argv, quiet, output=False):
+        return self._do([cmd] + argv, quiet=quiet, output=output)
 
 class SubCommand(object):
     """A SubCommand template base class"""
@@ -325,8 +328,22 @@ class CleanCommand(SubCommand):
 class ImagesCommand(SubCommand):
     """Run "docker images" command"""
     name = "images"
+    def args(self, parser):
+        parser.add_argument("--repo",
+                            help="List images in the given repository")
+
     def run(self, args, argv):
-        return Docker().command("images", argv, args.quiet)
+        if args.repo:
+            images=[]
+            image_re = re.compile(r"^"+re.escape(args.repo)+r"\s+(\S*)")
+            output = Docker().command("images", argv, args.quiet, output=True)
+            for line in output.split("\n"):
+                search = image_re.search(line)
+                if search:
+                    images.append(search.group(1))
+            print ' '.join(map(str,images))
+        else:
+            return Docker().command("images", argv, args.quiet)
 
 def main():
     parser = argparse.ArgumentParser(description="A Docker helper",
-- 
2.10.1

Re: [Qemu-devel] [PATCH 5/6] tests/docker/docker.py: expand images command

[Fam Zheng]

On Fri, 10/28 17:33, Alex Bennée wrote:
> Modern docker cli commands can filter by repository but as we want to
> work on all instances do this by hand in python:
> 
>    /tests/docker/docker.py images --repo qemu
> 
> Will return a space delimited list for the benefit of our Makefile
> recipes.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/docker.py | 27 ++++++++++++++++++++++-----
>  1 file changed, 22 insertions(+), 5 deletions(-)
> 
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 37d8319..03c0238 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -105,12 +105,15 @@ class Docker(object):
>          signal.signal(signal.SIGTERM, self._kill_instances)
>          signal.signal(signal.SIGHUP, self._kill_instances)
>  
> -    def _do(self, cmd, quiet=True, infile=None, **kwargs):
> +    def _do(self, cmd, quiet=True, infile=None, output=False, **kwargs):
>          if quiet:
>              kwargs["stdout"] = DEVNULL
>          if infile:
>              kwargs["stdin"] = infile
> -        return subprocess.call(self._command + cmd, **kwargs)
> +        if output:
> +            return subprocess.check_output(self._command + cmd, **kwargs)
> +        else:
> +            return subprocess.call(self._command + cmd, **kwargs)
>  
>      def _do_kill_instances(self, only_known, only_active=True):
>          cmd = ["ps", "-q"]
> @@ -188,8 +191,8 @@ class Docker(object):
>              self._instances.remove(label)
>          return ret
>  
> -    def command(self, cmd, argv, quiet):
> -        return self._do([cmd] + argv, quiet=quiet)
> +    def command(self, cmd, argv, quiet, output=False):
> +        return self._do([cmd] + argv, quiet=quiet, output=output)
>  
>  class SubCommand(object):
>      """A SubCommand template base class"""
> @@ -325,8 +328,22 @@ class CleanCommand(SubCommand):
>  class ImagesCommand(SubCommand):
>      """Run "docker images" command"""
>      name = "images"
> +    def args(self, parser):
> +        parser.add_argument("--repo",
> +                            help="List images in the given repository")
> +
>      def run(self, args, argv):
> -        return Docker().command("images", argv, args.quiet)
> +        if args.repo:
> +            images=[]
> +            image_re = re.compile(r"^"+re.escape(args.repo)+r"\s+(\S*)")
> +            output = Docker().command("images", argv, args.quiet, output=True)
> +            for line in output.split("\n"):
> +                search = image_re.search(line)
> +                if search:
> +                    images.append(search.group(1))
> +            print ' '.join(map(str,images))

s/,/, /

Otherwise looks good.

> +        else:
> +            return Docker().command("images", argv, args.quiet)
>  
>  def main():
>      parser = argparse.ArgumentParser(description="A Docker helper",
> -- 
> 2.10.1
> 

[Qemu-devel] [PATCH 6/6] tests/docker/Makefile.include: expand docker help text

[Alex Bennée]

No we can query what has and hasn't been built we can make this list
available in the help text. This is useful as some of the bootstrapped
builds can take some time to build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index bb9e078..a19991b 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -80,6 +80,10 @@ docker-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 		--include-executable=$(QEMU_EXEC),\
 		"BUILD USER","$*")
 
+DOCKER_USER_IMAGES := debian-stable-armhf debian-stable-arm64
+DOCKER_BUILT_IMAGES := $(shell $(SRC_PATH)/tests/docker/docker.py images --repo qemu)
+DOCKER_UNBUILT_IMAGES := $(filter-out debian-bootstrap, $(filter-out $(DOCKER_BUILT_IMAGES), $(DOCKER_USER_IMAGES) $(DOCKER_IMAGES)))
+
 docker:
 	@echo 'Build QEMU and run tests inside Docker containers'
 	@echo
@@ -96,8 +100,11 @@ docker:
 	@echo '    docker-image-IMAGE:  Build image "IMAGE".'
 	@echo '    docker-run:          For manually running a "TEST" with "IMAGE"'
 	@echo
-	@echo 'Available container images:'
-	@echo '    $(DOCKER_IMAGES)'
+	@echo 'Available built container images:'
+	@echo '    $(DOCKER_BUILT_IMAGES)'
+	@echo
+	@echo 'Available (but unbuilt) container images:'
+	@echo '    $(DOCKER_UNBUILT_IMAGES)'
 	@echo
 	@echo 'Available tests:'
 	@echo '    $(DOCKER_TESTS)'
-- 
2.10.1

Re: [Qemu-devel] [PATCH 6/6] tests/docker/Makefile.include: expand docker help text

[Fam Zheng]

On Fri, 10/28 17:33, Alex Bennée wrote:
> No we can query what has and hasn't been built we can make this list
> available in the help text. This is useful as some of the bootstrapped
> builds can take some time to build.

I'm not sure how useful it is to list built images without checking if it's in
sync with the dockerfile: we still rebuild outdated ones even they're built.

> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/Makefile.include | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index bb9e078..a19991b 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -80,6 +80,10 @@ docker-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
>  		--include-executable=$(QEMU_EXEC),\
>  		"BUILD USER","$*")
>  
> +DOCKER_USER_IMAGES := debian-stable-armhf debian-stable-arm64
> +DOCKER_BUILT_IMAGES := $(shell $(SRC_PATH)/tests/docker/docker.py images --repo qemu)
> +DOCKER_UNBUILT_IMAGES := $(filter-out debian-bootstrap, $(filter-out $(DOCKER_BUILT_IMAGES), $(DOCKER_USER_IMAGES) $(DOCKER_IMAGES)))
> +
>  docker:
>  	@echo 'Build QEMU and run tests inside Docker containers'
>  	@echo
> @@ -96,8 +100,11 @@ docker:
>  	@echo '    docker-image-IMAGE:  Build image "IMAGE".'
>  	@echo '    docker-run:          For manually running a "TEST" with "IMAGE"'
>  	@echo
> -	@echo 'Available container images:'
> -	@echo '    $(DOCKER_IMAGES)'
> +	@echo 'Available built container images:'
> +	@echo '    $(DOCKER_BUILT_IMAGES)'
> +	@echo
> +	@echo 'Available (but unbuilt) container images:'
> +	@echo '    $(DOCKER_UNBUILT_IMAGES)'
>  	@echo
>  	@echo 'Available tests:'
>  	@echo '    $(DOCKER_TESTS)'
> -- 
> 2.10.1
> 
> 

Re: [Qemu-devel] [PATCH 0/6] Expand the building of qemu-user docker images

[Fam Zheng]

On Fri, 10/28 17:33, Alex Bennée wrote:
> Hi Fam,
> 
> These are a little bit of a work in progress so maybe not all ready to
> be taken yet although review would be welcome.
> 
> This adds some tweaks to allow us to build the cross-arch debian
> images with direct make invocations and then see which images we can
> test against from the "make docker" help text. Now:
> 
>   make docker-image-debian-stable-arm64
> 
> Will do exactly as you expect. I can't get:
> 
>   make docker-test-quick@debian-stable-arm64
> 
> To work but I suspect this needs some fixing to the generated rules.
> 
> Comments?

Looks good, but what about binfmt_misc?

Fam

> 
> Alex Bennée (6):
>   tests/docker/Makefile.include: fix diff-index call
>   tests/docker/test-user: a simple linux-user test
>   tests/docker: add optional libs to travis.docker
>   tests/docker/Makefile: Add a rule for Debian user images
>   tests/docker/docker.py: expand images command
>   tests/docker/Makefile.include: expand docker help text
> 
>  tests/docker/Makefile.include          | 29 ++++++++++++++++++++++++++---
>  tests/docker/docker.py                 | 27 ++++++++++++++++++++++-----
>  tests/docker/dockerfiles/travis.docker |  9 ++++++++-
>  tests/docker/test-user                 | 18 ++++++++++++++++++
>  4 files changed, 74 insertions(+), 9 deletions(-)
>  create mode 100755 tests/docker/test-user
> 
> -- 
> 2.10.1
> 
> 

Re: [Qemu-devel] [PATCH 0/6] Expand the building of qemu-user docker images

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Fri, 10/28 17:33, Alex Bennée wrote:
>> Hi Fam,
>>
>> These are a little bit of a work in progress so maybe not all ready to
>> be taken yet although review would be welcome.
>>
>> This adds some tweaks to allow us to build the cross-arch debian
>> images with direct make invocations and then see which images we can
>> test against from the "make docker" help text. Now:
>>
>>   make docker-image-debian-stable-arm64
>>
>> Will do exactly as you expect. I can't get:
>>
>>   make docker-test-quick@debian-stable-arm64
>>
>> To work but I suspect this needs some fixing to the generated rules.
>>
>> Comments?
>
> Looks good, but what about binfmt_misc?

Currently the binfmt_misc check is done when the image is built (by the
pre script). We could expand docker.py to check and use that?

>
> Fam
>
>>
>> Alex Bennée (6):
>>   tests/docker/Makefile.include: fix diff-index call
>>   tests/docker/test-user: a simple linux-user test
>>   tests/docker: add optional libs to travis.docker
>>   tests/docker/Makefile: Add a rule for Debian user images
>>   tests/docker/docker.py: expand images command
>>   tests/docker/Makefile.include: expand docker help text
>>
>>  tests/docker/Makefile.include          | 29 ++++++++++++++++++++++++++---
>>  tests/docker/docker.py                 | 27 ++++++++++++++++++++++-----
>>  tests/docker/dockerfiles/travis.docker |  9 ++++++++-
>>  tests/docker/test-user                 | 18 ++++++++++++++++++
>>  4 files changed, 74 insertions(+), 9 deletions(-)
>>  create mode 100755 tests/docker/test-user
>>
>> --
>> 2.10.1
>>
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH 0/6] Expand the building of qemu-user docker images

[Fam Zheng]

On Mon, 10/31 11:40, Alex Bennée wrote:
> 
> Fam Zheng <famz@redhat.com> writes:
> 
> > On Fri, 10/28 17:33, Alex Bennée wrote:
> >> Hi Fam,
> >>
> >> These are a little bit of a work in progress so maybe not all ready to
> >> be taken yet although review would be welcome.
> >>
> >> This adds some tweaks to allow us to build the cross-arch debian
> >> images with direct make invocations and then see which images we can
> >> test against from the "make docker" help text. Now:
> >>
> >>   make docker-image-debian-stable-arm64
> >>
> >> Will do exactly as you expect. I can't get:
> >>
> >>   make docker-test-quick@debian-stable-arm64
> >>
> >> To work but I suspect this needs some fixing to the generated rules.
> >>
> >> Comments?
> >
> > Looks good, but what about binfmt_misc?
> 
> Currently the binfmt_misc check is done when the image is built (by the
> pre script). We could expand docker.py to check and use that?

I was just thinking that the real convenience only comes after the automated
setup of binfmt_misc (or a help text in "make docker" documentation).

What do you think?

Fam