[Qemu-devel] [PULL 0/2] Docker patches

[Qemu-devel] [PULL 0/2] Docker patches

[Fam Zheng]

The following changes since commit b0fbe46ad82982b289a44ee2495b59b0bad8a842:

  Update version for v2.11.0-rc0 release (2017-11-07 16:05:28 +0000)

are available in the git repository at:

  git://github.com/famz/qemu.git tags/docker-pull-request

for you to fetch changes up to 6423795efc5b665c595d9a0bf93cfbbca00362e9:

  docker: correctly escape $BACKEND in the help output (2017-11-08 10:59:42 +0800)

----------------------------------------------------------------

Hi Peter,

Here are two fixes on docker testing (the one saying "improved" is to
make it actually "correct"). Thanks.

Fam

----------------------------------------------------------------

Fam Zheng (1):
  docker: Improved image checksum

Philippe Mathieu-Daudé (1):
  docker: correctly escape $BACKEND in the help output

 tests/docker/Makefile.include |  2 +-
 tests/docker/docker.py        | 24 +++++++++++++++++++++++-
 2 files changed, 24 insertions(+), 2 deletions(-)

-- 
2.13.6

[Qemu-devel] [PULL 1/2] docker: Improved image checksum

[Fam Zheng]

When a base image locally defined by QEMU, such as in the debian images,
is updated, the dockerfile checksum mechanism in docker.py still skips
updating the derived image, because it only looks at the literal content
of the dockerfile, without considering changes to the base image.

For example we have a recent fix e58c1f9b35e81 that fixed
debian-win64-cross by updating its base image, debian8-mxe, but due to
above "feature" of docker.py the image in question is automatically NOT
rebuilt unless you add NOCACHE=1. It is noticed on Shippable:

https://app.shippable.com/github/qemu/qemu/runs/541/2/console

because after the fix is merged, the error still occurs, and the log
shows the container image is, as explained above, not updated.

This is because at the time docker.py was written, there wasn't any
dependencies between QEMU's docker images.

Now improve this to preprocess any "FROM qemu:*" directives in the
dockerfiles while doing checksum, and inline the base image's dockerfile
content, recursively. This ensures any changes on the depended _QEMU_
images are taken into account.

This means for external images that we expect to retrieve from docker
registries, we still do it as before. It is not perfect, because
registry images can get updated too. Technically we could substitute the
image name with its hex ID as obtained with $(docker images $IMAGE
--format="{{.Id}}"), but --format is not supported by RHEL 7, so leave
it for now.

Reported-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20171103131229.4737-1-famz@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/docker.py | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 08122ca17d..1246ba9578 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -105,6 +105,28 @@ def _copy_binary_with_libs(src, dest_dir):
             so_path = os.path.dirname(l)
             _copy_with_mkdir(l , dest_dir, so_path)
 
+def _read_qemu_dockerfile(img_name):
+    df = os.path.join(os.path.dirname(__file__), "dockerfiles",
+                      img_name + ".docker")
+    return open(df, "r").read()
+
+def _dockerfile_preprocess(df):
+    out = ""
+    for l in df.splitlines():
+        if len(l.strip()) == 0 or l.startswith("#"):
+            continue
+        from_pref = "FROM qemu:"
+        if l.startswith(from_pref):
+            # TODO: Alternatively we could replace this line with "FROM $ID"
+            # where $ID is the image's hex id obtained with
+            #    $ docker images $IMAGE --format="{{.Id}}"
+            # but unfortunately that's not supported by RHEL 7.
+            inlining = _read_qemu_dockerfile(l[len(from_pref):])
+            out += _dockerfile_preprocess(inlining)
+            continue
+        out += l + "\n"
+    return out
+
 class Docker(object):
     """ Running Docker commands """
     def __init__(self):
@@ -196,7 +218,7 @@ class Docker(object):
             checksum = self.get_image_dockerfile_checksum(tag)
         except Exception:
             return False
-        return checksum == _text_checksum(dockerfile)
+        return checksum == _text_checksum(_dockerfile_preprocess(dockerfile))
 
     def run(self, cmd, keep, quiet):
         label = uuid.uuid1().hex
-- 
2.13.6

[Qemu-devel] [PULL 2/2] docker: correctly escape $BACKEND in the help output

[Fam Zheng]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

In Makefiles the $ must be escaped as $$ in shell uses.

Since 8a2390a4f47:

 $ make docker
     [...]
     NETWORK=1            Enable virtual network interface with default backend.
     NETWORK=ACKEND     Enable virtual network interface with ACKEND.

Once escaped:

 $ make docker
     [...]
     NETWORK=1            Enable virtual network interface with default backend.
     NETWORK=$BACKEND     Enable virtual network interface with $BACKEND.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Message-Id: <20171108024719.8389-1-f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index f1a398e9fa..de87341528 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -109,7 +109,7 @@ docker:
 	@echo '    DEBUG=1              Stop and drop to shell in the created container'
 	@echo '                         before running the command.'
 	@echo '    NETWORK=1            Enable virtual network interface with default backend.'
-	@echo '    NETWORK=$BACKEND     Enable virtual network interface with $BACKEND.'
+	@echo '    NETWORK=$$BACKEND     Enable virtual network interface with $$BACKEND.'
 	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
 	@echo '    NOCACHE=1            Ignore cache when build images.'
 	@echo '    EXECUTABLE=<path>    Include executable in image.'
-- 
2.13.6

Re: [Qemu-devel] [PULL 0/2] Docker patches

[Peter Maydell]

On 10 November 2017 at 13:42, Fam Zheng <famz@redhat.com> wrote:
> The following changes since commit b0fbe46ad82982b289a44ee2495b59b0bad8a842:
>
>   Update version for v2.11.0-rc0 release (2017-11-07 16:05:28 +0000)
>
> are available in the git repository at:
>
>   git://github.com/famz/qemu.git tags/docker-pull-request
>
> for you to fetch changes up to 6423795efc5b665c595d9a0bf93cfbbca00362e9:
>
>   docker: correctly escape $BACKEND in the help output (2017-11-08 10:59:42 +0800)
>
> ----------------------------------------------------------------
>
> Hi Peter,
>
> Here are two fixes on docker testing (the one saying "improved" is to
> make it actually "correct"). Thanks.
>
> Fam
>
> ----------------------------------------------------------------
>
> Fam Zheng (1):
>   docker: Improved image checksum
>
> Philippe Mathieu-Daudé (1):
>   docker: correctly escape $BACKEND in the help output
>
>  tests/docker/Makefile.include |  2 +-
>  tests/docker/docker.py        | 24 +++++++++++++++++++++++-
>  2 files changed, 24 insertions(+), 2 deletions(-)
>

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 0/2] Docker patches

[Fam Zheng]

The following changes since commit 392fba9f583223786f844dce9b2e7f9a0ce0147a:

  Merge remote-tracking branch 'remotes/stsquad/tags/pull-travis-updates-010618-1' into staging (2018-06-01 17:32:30 +0100)

are available in the Git repository at:

  git://github.com/famz/qemu.git tags/docker-pull-request

for you to fetch changes up to 63dba573fa65bf830ec3f78a8f3a6b0697ce1637:

  docker: Update fedora image to 28 (2018-06-04 14:39:18 +0800)

----------------------------------------------------------------

Hi Peter,

One improvement from Alex and one patch to track the Fedora latest
release.

----------------------------------------------------------------

Alex Bennée (1):
  tests/docker/Makefile.include: handle empty TARGET_LIST

Fam Zheng (1):
  docker: Update fedora image to 28

 rules.mak                              | 3 +++
 tests/docker/Makefile.include          | 2 +-
 tests/docker/dockerfiles/fedora.docker | 2 +-
 3 files changed, 5 insertions(+), 2 deletions(-)

-- 
2.17.0

Re: [Qemu-devel] [PULL 0/2] Docker patches

[Peter Maydell]

On 4 June 2018 at 08:10, Fam Zheng <famz@redhat.com> wrote:
> The following changes since commit 392fba9f583223786f844dce9b2e7f9a0ce0147a:
>
>   Merge remote-tracking branch 'remotes/stsquad/tags/pull-travis-updates-010618-1' into staging (2018-06-01 17:32:30 +0100)
>
> are available in the Git repository at:
>
>   git://github.com/famz/qemu.git tags/docker-pull-request
>
> for you to fetch changes up to 63dba573fa65bf830ec3f78a8f3a6b0697ce1637:
>
>   docker: Update fedora image to 28 (2018-06-04 14:39:18 +0800)
>
> ----------------------------------------------------------------
>
> Hi Peter,
>
> One improvement from Alex and one patch to track the Fedora latest
> release.
>
> ----------------------------------------------------------------
>
> Alex Bennée (1):
>   tests/docker/Makefile.include: handle empty TARGET_LIST
>
> Fam Zheng (1):
>   docker: Update fedora image to 28
>
>  rules.mak                              | 3 +++
>  tests/docker/Makefile.include          | 2 +-
>  tests/docker/dockerfiles/fedora.docker | 2 +-
>  3 files changed, 5 insertions(+), 2 deletions(-)
>

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 0/2] Docker patches

[Fam Zheng]

The following changes since commit 6181478f6395cdd9d6ffd99623d0c9f39ea53606:

  Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging (2017-02-28 08:46:03 +0000)

are available in the git repository at:

  git://github.com/famz/qemu.git tags/docker-pull-request

for you to fetch changes up to 1ed9251515382a1fc9c6187fee8d7d7bf141dc90:

  .shippable: add s390x-cross target (2017-02-28 20:31:01 +0800)

----------------------------------------------------------------

Two more test/build automation patches from Alex for the freeze. Theoretically
these are okay to merge even after freeze but on the other hand I see no reason
to delay them.

----------------------------------------------------------------

Alex Bennée (2):
  new: dockerfiles/debian-s390-cross
  .shippable: add s390x-cross target

 .shippable.yml                                     |  2 ++
 tests/docker/dockerfiles/debian-s390x-cross.docker | 22 ++++++++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-s390x-cross.docker

-- 
2.9.3

Re: [Qemu-devel] [PULL 0/2] Docker patches

[Peter Maydell]

On 28 February 2017 at 12:44, Fam Zheng <famz@redhat.com> wrote:
> The following changes since commit 6181478f6395cdd9d6ffd99623d0c9f39ea53606:
>
>   Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging (2017-02-28 08:46:03 +0000)
>
> are available in the git repository at:
>
>   git://github.com/famz/qemu.git tags/docker-pull-request
>
> for you to fetch changes up to 1ed9251515382a1fc9c6187fee8d7d7bf141dc90:
>
>   .shippable: add s390x-cross target (2017-02-28 20:31:01 +0800)
>
> ----------------------------------------------------------------
>
> Two more test/build automation patches from Alex for the freeze. Theoretically
> these are okay to merge even after freeze but on the other hand I see no reason
> to delay them.
>
> ----------------------------------------------------------------
>
> Alex Bennée (2):
>   new: dockerfiles/debian-s390-cross
>   .shippable: add s390x-cross target
>
>  .shippable.yml                                     |  2 ++
>  tests/docker/dockerfiles/debian-s390x-cross.docker | 22 ++++++++++++++++++++++
>  2 files changed, 24 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debian-s390x-cross.docker

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 0/2] Docker patches

[Fam Zheng]

The following changes since commit 42e0d60f1615ef63d16e41bb1668805560c37870:

  Merge remote-tracking branch 'remotes/riku/tags/pull-linux-user-20160804' into staging (2016-08-04 18:36:05 +0100)

are available in the git repository at:

  git@github.com:famz/qemu tags/docker-pull-request

for you to fetch changes up to 4a93f78ed086d6f0bceb3f0b281009e6026935b7:

  docker: Add "--enable-werror" to configure command line (2016-08-05 16:34:55 +0800)

----------------------------------------------------------------

Two bug fixes. One fixes breakage on RHEL 7, reported by Paolo, the other one
fixes the missing --enable-werror in docker build, which was intended since the
beginning, but was left behind during later iterations of the initial patch
series (lost .git in the docker src copy).

----------------------------------------------------------------

Fam Zheng (2):
  docker: Be compatible with older docker
  docker: Add "--enable-werror" to configure command line

 tests/docker/Makefile.include | 5 ++---
 tests/docker/common.rc        | 1 +
 2 files changed, 3 insertions(+), 3 deletions(-)

-- 
2.7.4

Re: [Qemu-devel] [PULL 0/2] Docker patches

[Peter Maydell]

On 5 August 2016 at 10:04, Fam Zheng <famz@redhat.com> wrote:
> The following changes since commit 42e0d60f1615ef63d16e41bb1668805560c37870:
>
>   Merge remote-tracking branch 'remotes/riku/tags/pull-linux-user-20160804' into staging (2016-08-04 18:36:05 +0100)
>
> are available in the git repository at:
>
>   git@github.com:famz/qemu tags/docker-pull-request
>
> for you to fetch changes up to 4a93f78ed086d6f0bceb3f0b281009e6026935b7:
>
>   docker: Add "--enable-werror" to configure command line (2016-08-05 16:34:55 +0800)
>
> ----------------------------------------------------------------
>
> Two bug fixes. One fixes breakage on RHEL 7, reported by Paolo, the other one
> fixes the missing --enable-werror in docker build, which was intended since the
> beginning, but was left behind during later iterations of the initial patch
> series (lost .git in the docker src copy).
>
> ----------------------------------------------------------------

Applied, thanks.

-- PMM