[PATCH] docker.py: fix fetching of FROM layers

[PATCH] docker.py: fix fetching of FROM layers

[Alex Bennée]

This worked on a system that was already bootstrapped because the
stage 2 images already existed even if they wouldn't be used. What we
should have pulled down was the FROM line containers first because
building on gitlab doesn't have the advantage of using our build
system to build the pre-requisite bits.

We still pull the image we want to build just in case we can use the
cached data.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 2d67bbd15a..c9f20d8d09 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -306,14 +306,18 @@ class Docker(object):
         checksum = _text_checksum(_dockerfile_preprocess(dockerfile))
 
         if registry is not None:
-            # see if we can fetch a cache copy, may fail...
-            pull_args = ["pull", "%s/%s" % (registry, tag)]
-            if self._do(pull_args, quiet=quiet) == 0:
+            sources = re.findall("FROM qemu\/(.*)", dockerfile)
+            # Fetch any cache layers we can, may fail
+            for s in sources:
+                pull_args = ["pull", "%s/qemu/%s" % (registry, s)]
+                if self._do(pull_args, quiet=quiet) != 0:
+                    registry = None
+                    break
+            # Make substitutions
+            if registry is not None:
                 dockerfile = dockerfile.replace("FROM qemu/",
                                                 "FROM %s/qemu/" %
                                                 (registry))
-            else:
-                registry = None
 
         tmp_df = tempfile.NamedTemporaryFile(mode="w+t",
                                              encoding='utf-8',
@@ -339,6 +343,8 @@ class Docker(object):
             build_args += ["--build-arg", "BUILDKIT_INLINE_CACHE=1"]
 
         if registry is not None:
+            pull_args = ["pull", "%s/%s" % (registry, tag)]
+            self._do(pull_args, quiet=quiet)
             cache = "%s/%s" % (registry, tag)
             build_args += ["--cache-from", cache]
         build_args += argv
-- 
2.20.1

Re: [PATCH] docker.py: fix fetching of FROM layers

[Daniel P. Berrangé]

On Mon, Jul 13, 2020 at 03:58:01PM +0100, Alex Bennée wrote:
> This worked on a system that was already bootstrapped because the
> stage 2 images already existed even if they wouldn't be used. What we
> should have pulled down was the FROM line containers first because
> building on gitlab doesn't have the advantage of using our build
> system to build the pre-requisite bits.
> 
> We still pull the image we want to build just in case we can use the
> cached data.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/docker.py | 16 +++++++++++-----
>  1 file changed, 11 insertions(+), 5 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] docker.py: fix fetching of FROM layers

[Philippe Mathieu-Daudé]

On 7/13/20 4:58 PM, Alex Bennée wrote:
> This worked on a system that was already bootstrapped because the
> stage 2 images already existed even if they wouldn't be used. What we
> should have pulled down was the FROM line containers first because
> building on gitlab doesn't have the advantage of using our build
> system to build the pre-requisite bits.
> 
> We still pull the image we want to build just in case we can use the
> cached data.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/docker.py | 16 +++++++++++-----
>  1 file changed, 11 insertions(+), 5 deletions(-)
> 

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>