[Qemu-devel] [PATCH for 2.10 v2 0/6] Various docker fixes

[Qemu-devel] [PATCH for 2.10 v2 0/6] Various docker fixes

[Alex Bennée]

Hi,

Phillipe pointed out a few anomalies with the travis.docker image
during review which has led to a couple more patches. As they are bug
fixes I didn't race to get a pull-req out today for hard-freeze. I
will roll a pull-req by the end of the week if there are no objections
as I'll be away for the rc1/2 cycles.

Alex Bennée (6):
  docker: ensure NOUSER for travis images
  docker: fix dirty/stash detection on some systems
  docker: ignore submodules when checking diff
  docker: docker.py make --no-cache skip checksum test
  docker: don't install device-tree-compiler build-deps in travis.docker
  docker: reduce noise when building travis.docker

 tests/docker/Makefile.include          | 3 ++-
 tests/docker/docker.py                 | 3 ++-
 tests/docker/dockerfiles/travis.docker | 6 ++++--
 3 files changed, 8 insertions(+), 4 deletions(-)

-- 
2.13.0

[Qemu-devel] [PATCH for 2.10 v2 1/6] docker: ensure NOUSER for travis images

[Alex Bennée]

While adding the current user is a useful default behaviour for
creating new images it is not appropriate for Travis which already has
a default user.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index aaab1a4208..d7dafdbd27 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -71,6 +71,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9
 docker-image-debian-s390x-cross: docker-image-debian9
 docker-image-debian-win32-cross: docker-image-debian8-mxe
 docker-image-debian-win64-cross: docker-image-debian8-mxe
+docker-image-travis: NOUSER=1
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
-- 
2.13.0

[Qemu-devel] [PATCH for 2.10 v2 2/6] docker: fix dirty/stash detection on some systems

[Alex Bennée]

I reported this a while ago but evidently forgot to push the patch
upstream. Without this I wasn't seeing the dirty tree state submitted
to the docker build which is very confusing.

Reference:
  Subject: [PATCH 1/6] tests/docker/Makefile.include: fix diff-index call
  Date: Fri, 28 Oct 2016 17:33:34 +0100
  Message-Id: <20161028163339.31096-2-alex.bennee@linaro.org>

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index d7dafdbd27..ff580edfad 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -20,7 +20,7 @@ IMAGES ?= %
 # Make archive from git repo $1 to tar.gz $2
 make-archive-maybe = $(if $(wildcard $1/*), \
 	$(call quiet-command, \
-		(cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
+		(cd $1; if git diff-index --quiet HEAD -- 2>/dev/null; then \
 			git archive -1 HEAD --format=tar.gz; \
 		else \
 			git archive -1 $$(git stash create) --format=tar.gz; \
-- 
2.13.0

[Qemu-devel] [PATCH for 2.10 v2 3/6] docker: ignore submodules when checking diff

[Alex Bennée]

The QEMU tree often has dirty submodules in it but this will confuse
things as git stash is not going to save their state directly. They
are likely just dirty from build files anyway. Ignore them in the
diff-index command.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 tests/docker/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index ff580edfad..2a560a5327 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -20,7 +20,7 @@ IMAGES ?= %
 # Make archive from git repo $1 to tar.gz $2
 make-archive-maybe = $(if $(wildcard $1/*), \
 	$(call quiet-command, \
-		(cd $1; if git diff-index --quiet HEAD -- 2>/dev/null; then \
+		(cd $1; if git diff-index --quiet --ignore-submodules HEAD -- 2>/dev/null; then \
 			git archive -1 HEAD --format=tar.gz; \
 		else \
 			git archive -1 $$(git stash create) --format=tar.gz; \
-- 
2.13.0

[Qemu-devel] [PATCH for 2.10 v2 4/6] docker: docker.py make --no-cache skip checksum test

[Alex Bennée]

If you invoke with NOCACHE=1 we pass --no-cache in the argv to
docker.py but may still not force a rebuild if the dockerfile checksum
hasn't changed. By testing for its presence we can force builds
without having to manually remove the docker image.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index ee40ca04d9..aab1648cc5 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -261,7 +261,8 @@ class BuildCommand(SubCommand):
         tag = args.tag
 
         dkr = Docker()
-        if dkr.image_matches_dockerfile(tag, dockerfile):
+        if "--no-cache" not in argv and \
+           dkr.image_matches_dockerfile(tag, dockerfile):
             if not args.quiet:
                 print "Image is up to date."
         else:
-- 
2.13.0

[Qemu-devel] [PATCH for 2.10 v2 5/6] docker: don't install device-tree-compiler build-deps in travis.docker

[Alex Bennée]

Installing the device-tree-compiler build-deps is a little extreme. We
only actually need the binary so include it with the other packages.

Suggested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/travis.docker | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
index 636fa590a5..57ac8e1419 100644
--- a/tests/docker/dockerfiles/travis.docker
+++ b/tests/docker/dockerfiles/travis.docker
@@ -1,6 +1,5 @@
 FROM quay.io/travisci/travis-ruby
 RUN apt-get update
 RUN apt-get -y build-dep qemu
-RUN apt-get -y build-dep device-tree-compiler
-RUN apt-get -y install python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
+RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
 ENV FEATURES pyyaml
-- 
2.13.0

Re: [Qemu-devel] [PATCH for 2.10 v2 5/6] docker: don't install device-tree-compiler build-deps in travis.docker

[Philippe Mathieu-Daudé]

On 07/25/2017 10:34 AM, Alex Bennée wrote:
> Installing the device-tree-compiler build-deps is a little extreme. We
> only actually need the binary so include it with the other packages.
> 
> Suggested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

before:

$ docker images qemu:travis --no-trunc 
--format="{{.Repository}}:{{.Tag}}\t{{.Size}}"

qemu:travis	6.16GB

after:

qemu:travis	5.92GB

not a big win :(

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

(not a "bugfix" btw)

> ---
>   tests/docker/dockerfiles/travis.docker | 3 +--
>   1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
> index 636fa590a5..57ac8e1419 100644
> --- a/tests/docker/dockerfiles/travis.docker
> +++ b/tests/docker/dockerfiles/travis.docker
> @@ -1,6 +1,5 @@
>   FROM quay.io/travisci/travis-ruby
>   RUN apt-get update
>   RUN apt-get -y build-dep qemu
> -RUN apt-get -y build-dep device-tree-compiler
> -RUN apt-get -y install python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
> +RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
>   ENV FEATURES pyyaml
> 

Re: [Qemu-devel] [PATCH for 2.10 v2 5/6] docker: don't install device-tree-compiler build-deps in travis.docker

[Philippe Mathieu-Daudé]

On 07/25/2017 11:17 AM, Philippe Mathieu-Daudé wrote:
> before:
> 
> $ docker images qemu:travis --no-trunc 
> --format="{{.Repository}}:{{.Tag}}\t{{.Size}}"
> 
> qemu:travis    6.16GB
> 
> after:
> 
> qemu:travis    5.92GB

I mis-selected, the correct size this patch applied:

qemu:travis	6.03GB

> 
> not a big win :(

Re: [Qemu-devel] [PATCH for 2.10 v2 5/6] docker: don't install device-tree-compiler build-deps in travis.docker

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 07/25/2017 10:34 AM, Alex Bennée wrote:
>> Installing the device-tree-compiler build-deps is a little extreme. We
>> only actually need the binary so include it with the other packages.
>>
>> Suggested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> before:
>
> $ docker images qemu:travis --no-trunc
> --format="{{.Repository}}:{{.Tag}}\t{{.Size}}"
>
> qemu:travis	6.16GB
>
> after:
>
> qemu:travis	5.92GB
>
> not a big win :(
>
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> (not a "bugfix" btw)

It's arguable - it was certainly wrong as we just need the compiler and
I'm not sure build-dep actually would have pulled it on (although the
qemu build-dep probably does).

>
>> ---
>>   tests/docker/dockerfiles/travis.docker | 3 +--
>>   1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
>> index 636fa590a5..57ac8e1419 100644
>> --- a/tests/docker/dockerfiles/travis.docker
>> +++ b/tests/docker/dockerfiles/travis.docker
>> @@ -1,6 +1,5 @@
>>   FROM quay.io/travisci/travis-ruby
>>   RUN apt-get update
>>   RUN apt-get -y build-dep qemu
>> -RUN apt-get -y build-dep device-tree-compiler
>> -RUN apt-get -y install python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
>> +RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
>>   ENV FEATURES pyyaml
>>


--
Alex Bennée

[Qemu-devel] [PATCH for 2.10 v2 6/6] docker: reduce noise when building travis.docker

[Alex Bennée]

Set the DEBIAN_FRONTEND and locale env vars to stop apt complaining so
much as we build the image.

Suggested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/travis.docker | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
index 57ac8e1419..605b6e429b 100644
--- a/tests/docker/dockerfiles/travis.docker
+++ b/tests/docker/dockerfiles/travis.docker
@@ -1,4 +1,7 @@
 FROM quay.io/travisci/travis-ruby
+ENV DEBIAN_FRONTEND noninteractive
+ENV LANG en_US.UTF-8
+ENV LC_ALL en_US.UTF-8
 RUN apt-get update
 RUN apt-get -y build-dep qemu
 RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
-- 
2.13.0

Re: [Qemu-devel] [PATCH for 2.10 v2 6/6] docker: reduce noise when building travis.docker

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

On 07/25/2017 08:34 AM, Alex Bennée wrote:
> Set the DEBIAN_FRONTEND and locale env vars to stop apt complaining so
> much as we build the image.
> 
> Suggested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/dockerfiles/travis.docker | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
> index 57ac8e1419..605b6e429b 100644
> --- a/tests/docker/dockerfiles/travis.docker
> +++ b/tests/docker/dockerfiles/travis.docker
> @@ -1,4 +1,7 @@
>  FROM quay.io/travisci/travis-ruby
> +ENV DEBIAN_FRONTEND noninteractive
> +ENV LANG en_US.UTF-8
> +ENV LC_ALL en_US.UTF-8

LANG is pointless if LC_ALL is set, as LC_ALL takes precedence.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [PATCH for 2.10 v2 6/6] docker: reduce noise when building travis.docker

[Philippe Mathieu-Daudé]

On 07/25/2017 10:34 AM, Alex Bennée wrote:
> Set the DEBIAN_FRONTEND and locale env vars to stop apt complaining so
> much as we build the image.
> 
> Suggested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   tests/docker/dockerfiles/travis.docker | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
> index 57ac8e1419..605b6e429b 100644
> --- a/tests/docker/dockerfiles/travis.docker
> +++ b/tests/docker/dockerfiles/travis.docker
> @@ -1,4 +1,7 @@
>   FROM quay.io/travisci/travis-ruby
> +ENV DEBIAN_FRONTEND noninteractive
> +ENV LANG en_US.UTF-8
> +ENV LC_ALL en_US.UTF-8

can be one-lined:
ENV DEBIAN_FRONTEND=noninteractive LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
maybe add "default env vars set using travis (see [link])" in message?

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

(not a bugfix)

>   RUN apt-get update
>   RUN apt-get -y build-dep qemu
>   RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
> 

Re: [Qemu-devel] [PATCH for 2.10 v2 0/6] Various docker fixes

[Fam Zheng]

On Tue, 07/25 14:34, Alex Bennée wrote:
> Hi,
> 
> Phillipe pointed out a few anomalies with the travis.docker image
> during review which has led to a couple more patches. As they are bug
> fixes I didn't race to get a pull-req out today for hard-freeze. I
> will roll a pull-req by the end of the week if there are no objections
> as I'll be away for the rc1/2 cycles.
> 
> Alex Bennée (6):
>   docker: ensure NOUSER for travis images
>   docker: fix dirty/stash detection on some systems
>   docker: ignore submodules when checking diff
>   docker: docker.py make --no-cache skip checksum test
>   docker: don't install device-tree-compiler build-deps in travis.docker
>   docker: reduce noise when building travis.docker

Thanks, I've queued patches 1, 4, 5, 6 for 2.11. I intend to drop the code
touched by 2 and 3 with a dedicated script:

https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg06349.html
https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg06358.html

(It would be great if you can give some test/review there :)

Fam

> 
>  tests/docker/Makefile.include          | 3 ++-
>  tests/docker/docker.py                 | 3 ++-
>  tests/docker/dockerfiles/travis.docker | 6 ++++--
>  3 files changed, 8 insertions(+), 4 deletions(-)
> 
> -- 
> 2.13.0
> 

Re: [Qemu-devel] [PATCH for 2.10 v2 0/6] Various docker fixes

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Tue, 07/25 14:34, Alex Bennée wrote:
>> Hi,
>>
>> Phillipe pointed out a few anomalies with the travis.docker image
>> during review which has led to a couple more patches. As they are bug
>> fixes I didn't race to get a pull-req out today for hard-freeze. I
>> will roll a pull-req by the end of the week if there are no objections
>> as I'll be away for the rc1/2 cycles.
>>
>> Alex Bennée (6):
>>   docker: ensure NOUSER for travis images
>>   docker: fix dirty/stash detection on some systems
>>   docker: ignore submodules when checking diff
>>   docker: docker.py make --no-cache skip checksum test
>>   docker: don't install device-tree-compiler build-deps in travis.docker
>>   docker: reduce noise when building travis.docker
>
> Thanks, I've queued patches 1, 4, 5, 6 for 2.11. I intend to drop the code
> touched by 2 and 3 with a dedicated script:
>
> https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg06349.html
> https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg06358.html
>
> (It would be great if you can give some test/review there :)

Cool - I shall have a look.

>
> Fam
>
>>
>>  tests/docker/Makefile.include          | 3 ++-
>>  tests/docker/docker.py                 | 3 ++-
>>  tests/docker/dockerfiles/travis.docker | 6 ++++--
>>  3 files changed, 8 insertions(+), 4 deletions(-)
>>
>> --
>> 2.13.0
>>


--
Alex Bennée