[Qemu-devel] [RFC PATCH 0/3] Better docker dependency checking

[Qemu-devel] [RFC PATCH 0/3] Better docker dependency checking

[Alex Bennée]

Hi,

These patches apply on top of my tcg/testing patch series and attempt
to make the dependancy checking a bit smarter within the confines of
what Make can achieve. The basic idea is although the debian-sid-image
will always rebuild by making it a conditional dependancy we can only
trigger this when we need to rebuild the target image. Together with
the top-level initial build this mean you can now run make check-tcg
with high -j's without issue.

The first patch in the series fixes the checksum code as it was broken
anyway. Currently we don't properly handle extra_files and executables
but this should be fairly easy to add. I've not done it at the moment
because we don't use these exotic builds in any of the default
targets.

Alex Bennée (3):
  docker: docker.py don't conflate checksums for extra_files
  dcoker: docker.py add check sub-command
  docker: remove SUBMAKE_NOCACHE_HACK for debian-sid based images

 tests/docker/Makefile.include | 25 ++++++++++++++-----------
 tests/docker/docker.py        | 30 +++++++++++++++++++++++++++---
 tests/tcg/Makefile.include    |  1 -
 3 files changed, 41 insertions(+), 15 deletions(-)

-- 
2.17.1

[Qemu-devel] [RFC PATCH 1/3] docker: docker.py don't conflate checksums for extra_files

[Alex Bennée]

This just gets confusing especially as the helper function doesn't
even take into account any extra files (or the executable). Currently
the actual check just ignores them and also passes the result through
_dockerfile_preprocess so we fix that too.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 37bfa98005..80efa641c5 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -200,8 +200,10 @@ class Docker(object):
 
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
-                     _text_checksum("\n".join([dockerfile] +
-                                    extra_files_cksum)))
+                     _text_checksum(_dockerfile_preprocess(dockerfile)))
+        for f, c in extra_files_cksum:
+            tmp_df.write("LABEL com.qemu.%s-checksum=%s" % (f, c))
+
         tmp_df.flush()
 
         self._do_check(["build", "-t", tag, "-f", tmp_df.name] + argv + \
@@ -316,7 +318,7 @@ class BuildCommand(SubCommand):
                 _copy_binary_with_libs(args.include_executable, docker_dir)
             for filename in args.extra_files or []:
                 _copy_with_mkdir(filename, docker_dir)
-                cksum += [_file_checksum(filename)]
+                cksum += [(filename, _file_checksum(filename))]
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
-- 
2.17.1

[Qemu-devel] [RFC PATCH 2/3] dcoker: docker.py add check sub-command

[Alex Bennée]

This command allows you to check if we need to re-build a docker
image. If the image isn't in the repository or the checksums don't
match then we return false and some text (for processing in
makefiles).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 80efa641c5..c2763957c9 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -441,6 +441,28 @@ class CcCommand(SubCommand):
         return Docker().command("run", cmd, args.quiet)
 
 
+class CheckCommand(SubCommand):
+    """Check if we need to re-build a docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
+    name = "check"
+    def args(self, parser):
+        parser.add_argument("tag",
+                            help="Image Tag")
+        parser.add_argument("dockerfile",
+                            help="Dockerfile name")
+
+    def run(self, args, argv):
+        dockerfile = open(args.dockerfile, "rb").read()
+        tag = args.tag
+
+        dkr = Docker()
+        if dkr.image_matches_dockerfile(tag, dockerfile):
+            if not args.quiet:
+                print "Image is up to date."
+            return 0
+        else:
+            print "Image needs updating"
+            return 1
+
 def main():
     parser = argparse.ArgumentParser(description="A Docker helper",
             usage="%s <subcommand> ..." % os.path.basename(sys.argv[0]))
-- 
2.17.1

[Qemu-devel] [RFC PATCH 3/3] docker: remove SUBMAKE_NOCACHE_HACK for debian-sid based images

[Alex Bennée]

Now we can conditionally make docker-image-debian-sid a dependency
depending on if we actually need to re-build our image we can remove
this hack and use that mechanism.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 25 ++++++++++++++-----------
 tests/tcg/Makefile.include    |  1 -
 2 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 3393e9f97d..2ce72de180 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -66,18 +66,21 @@ docker-image-debian-win64-cross: docker-image-debian8-mxe
 # its children. However when doing check-tcg we ensure everything is
 # built at top-level make so we can dispense with NOCACHE here to
 # avoid excessive re-builds.
-ifneq ($(SUBMAKE_NOCACHE_HACK),y)
 docker-image-debian-sid: NOCACHE=1
-endif
-
-docker-image-debian-alpha-cross: docker-image-debian-sid
-docker-image-debian-hppa-cross: docker-image-debian-sid
-docker-image-debian-m68k-cross: docker-image-debian-sid
-docker-image-debian-sh4-cross: docker-image-debian-sid
-docker-image-debian-sparc64-cross: docker-image-debian-sid
-docker-image-debian-mips64-cross: docker-image-debian-sid
-docker-image-debian-riscv64-cross: docker-image-debian-sid
-docker-image-debian-powerpc-cross: docker-image-debian-sid
+
+# $1 = dockerfile, $2 = dependancy
+cond-dock-dep = $(if \
+	$(call $(DOCKER_SCRIPT) check --quiet \
+		-t qemu:$1 $(DOCKER_FILES_DIR)/$1.docker),$2)
+
+docker-image-debian-alpha-cross: $(call cond-dock-dep,debian-alpha-cross,docker-image-debian-sid)
+docker-image-debian-hppa-cross: $(call cond-dock-dep,debian-hppa-cross,docker-image-debian-sid)
+docker-image-debian-m68k-cross: $(call cond-dock-dep,debian-m68k-cross,docker-image-debian-sid)
+docker-image-debian-sh4-cross: $(call cond-dock-dep,debian-sh4-cross,docker-image-debian-sid)
+docker-image-debian-sparc64-cross: $(call cond-dock-dep,debian-sparc64-cross,docker-image-debian-sid)
+docker-image-debian-mips64-cross: $(call cond-dock-dep,debian-mip64-cross,docker-image-debian-sid)
+docker-image-debian-riscv64-cross: $(call cond-dock-dep,debian-riscv64-cross,docker-image-debian-sid)
+docker-image-debian-powerpc-cross: $(call cond-dock-dep,debian-powerpc-cross,docker-image-debian-sid)
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/tcg/Makefile.include b/tests/tcg/Makefile.include
index 951f40c28d..4851df0951 100644
--- a/tests/tcg/Makefile.include
+++ b/tests/tcg/Makefile.include
@@ -39,7 +39,6 @@ ifeq ($(HAVE_USER_DOCKER)$(GUEST_BUILD),y)
 ifneq ($(DOCKER_IMAGE),)
 
 # We also need the Docker make rules to depend on
-SUBMAKE_NOCACHE_HACK=y
 include $(SRC_PATH)/tests/docker/Makefile.include
 
 DOCKER_COMPILE_CMD="$(DOCKER_SCRIPT) cc --user $(shell id -u) \
-- 
2.17.1