[PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug

[PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug

[Stefan Hajnoczi]

QEMU's CI pipeline involves building container images that will be used
to run builds and tests. A recent Docker change triggered the following
error:

  $ docker push "$TAG"
  ...
  error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9

https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372

This happens because Docker now pushes out-of-order and the GitLab
Container Registry rejects due to an unknown reference:
https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5

It is unclear at this point whether GitLab will modify the behavior of
Container Registry or whether Docker will ship a fix.

The current workaround is to disable the provenance attestation that is
involved in this issue. QEMU's CI pipeline container images are used
internally for testing and are not widely distributed. Provenance
attestation can be disabled as there are no external consumers of these
images. Expect to revert this commit in the future when GitLab or Docker
have released their own fixes.

Cc: Alex Bennée <alex.bennee@linaro.org>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Thomas Huth <thuth@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
Note that this is an urgent patch so that the CI will pass again and I
can tag v11.1.0-rc3 later today. Currently it is failing due to this
issue!

 .gitlab-ci.d/container-template.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 8c7311cca57..88317d650e2 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -14,7 +14,7 @@
     - echo "TAG:$TAG"
     - echo "COMMON_TAG:$COMMON_TAG"
     - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
-      --build-arg BUILDKIT_INLINE_CACHE=1
+      --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false
       -f "tests/docker/dockerfiles/$NAME.docker" "."
     - docker push "$TAG"
   after_script:
-- 
2.55.0

Re: [PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug

[Daniel P. Berrangé]

On Tue, Aug 04, 2026 at 12:54:14PM -0400, Stefan Hajnoczi wrote:
> QEMU's CI pipeline involves building container images that will be used
> to run builds and tests. A recent Docker change triggered the following
> error:
> 
>   $ docker push "$TAG"
>   ...
>   error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372
> 
> This happens because Docker now pushes out-of-order and the GitLab
> Container Registry rejects due to an unknown reference:
> https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5
> 
> It is unclear at this point whether GitLab will modify the behavior of
> Container Registry or whether Docker will ship a fix.

The forum post links to an issue where this comment:

  https://github.com/moby/buildkit/issues/7007#issuecomment-5178990414

suggests we can just set

  BUILDX_NO_DEFAULT_OCI_ARTIFACT=true

...except I don't think they've updated the docker docker image
with this fix yet. Once they do, that will be better as we can
set it in the GitLab env such that it applies to all branches,
instead of havnig to cherry-pick 

> 
> The current workaround is to disable the provenance attestation that is
> involved in this issue. QEMU's CI pipeline container images are used
> internally for testing and are not widely distributed. Provenance
> attestation can be disabled as there are no external consumers of these
> images. Expect to revert this commit in the future when GitLab or Docker
> have released their own fixes.
> 
> Cc: Alex Bennée <alex.bennee@linaro.org>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> Note that this is an urgent patch so that the CI will pass again and I
> can tag v11.1.0-rc3 later today. Currently it is failing due to this
> issue!
> 
>  .gitlab-ci.d/container-template.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index 8c7311cca57..88317d650e2 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -14,7 +14,7 @@
>      - echo "TAG:$TAG"
>      - echo "COMMON_TAG:$COMMON_TAG"
>      - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
> -      --build-arg BUILDKIT_INLINE_CACHE=1
> +      --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false
>        -f "tests/docker/dockerfiles/$NAME.docker" "."
>      - docker push "$TAG"
>    after_script:

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1799 bytes --]

On Tue, Aug 04, 2026 at 12:54:14PM -0400, Stefan Hajnoczi wrote:
> QEMU's CI pipeline involves building container images that will be used
> to run builds and tests. A recent Docker change triggered the following
> error:
> 
>   $ docker push "$TAG"
>   ...
>   error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372
> 
> This happens because Docker now pushes out-of-order and the GitLab
> Container Registry rejects due to an unknown reference:
> https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5
> 
> It is unclear at this point whether GitLab will modify the behavior of
> Container Registry or whether Docker will ship a fix.
> 
> The current workaround is to disable the provenance attestation that is
> involved in this issue. QEMU's CI pipeline container images are used
> internally for testing and are not widely distributed. Provenance
> attestation can be disabled as there are no external consumers of these
> images. Expect to revert this commit in the future when GitLab or Docker
> have released their own fixes.
> 
> Cc: Alex Bennée <alex.bennee@linaro.org>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> Note that this is an urgent patch so that the CI will pass again and I
> can tag v11.1.0-rc3 later today. Currently it is failing due to this
> issue!
> 
>  .gitlab-ci.d/container-template.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Thanks, applied to my staging tree:
https://gitlab.com/stefanha/qemu/commits/staging

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]