docker-image-debian-all-test-cross errors

docker-image-debian-all-test-cross errors

[Richard Henderson]

Has anyone else been seeing

Error: Failed to fetch 
http://deb.debian.org/debian/pool/main/d/dpkg/libdpkg-perl_1.22.21_all.deb  404  Not Found 
[IP: 151.101.22.132 80]
Error: Failed to fetch 
http://deb.debian.org/debian/pool/main/d/dpkg/dpkg-dev_1.22.21_all.deb  404  Not Found 
[IP: 151.101.22.132 80]
Error: Failed to fetch 
http://deb.debian.org/debian/pool/main/p/python3.13/libpython3.13_3.13.5-2_amd64.deb  404 
Not Found [IP: 151.101.22.132 80]
Error: Failed to fetch 
http://deb.debian.org/debian/pool/main/g/glibc/libc6-i386_2.41-12_amd64.deb  404  Not 
Found [IP: 151.101.22.132 80]
Error: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Error: building at STEP "RUN DEBIAN_FRONTEND=noninteractive eatmydata   apt install -y 
--no-install-recommends   clang  dpkg-dev  gdb-multiarch  libclang-rt-dev": while running 
runtime: exit status 100
make: *** [/home/rth/qemu/src/tests/docker/Makefile.include:43: 
docker-image-debian-all-test-cross] Error 100

This has been consistently failing for me for a couple of weeks...


r~

Re: docker-image-debian-all-test-cross errors

[Mark Cave-Ayland]

On 04/07/2026 22:20, Richard Henderson wrote:

> Has anyone else been seeing
> 
> Error: Failed to fetch http://deb.debian.org/debian/pool/main/d/dpkg/libdpkg- 
> perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
> Error: Failed to fetch http://deb.debian.org/debian/pool/main/d/dpkg/dpkg- 
> dev_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
> Error: Failed to fetch http://deb.debian.org/debian/pool/main/p/python3.13/ 
> libpython3.13_3.13.5-2_amd64.deb  404 Not Found [IP: 151.101.22.132 80]
> Error: Failed to fetch http://deb.debian.org/debian/pool/main/g/glibc/libc6- 
> i386_2.41-12_amd64.deb  404  Not Found [IP: 151.101.22.132 80]
> Error: Unable to fetch some archives, maybe run apt-get update or try with --fix- 
> missing?
> Error: building at STEP "RUN DEBIAN_FRONTEND=noninteractive eatmydata   apt install - 
> y --no-install-recommends   clang  dpkg-dev  gdb-multiarch  libclang-rt-dev": while 
> running runtime: exit status 100
> make: *** [/home/rth/qemu/src/tests/docker/Makefile.include:43: docker-image-debian- 
> all-test-cross] Error 100
> 
> This has been consistently failing for me for a couple of weeks...
>  
> r~

I wonder if docker build has cached the existing layers, and since apt-get update 
only runs in the first part of the Dockerfile then the subsequent layers try to 
access an old base image?

Does running "docker builder prune" (possibly with --all) help at all? That should 
purge the cache and force docker to rebuild all the layers again.


ATB,

Mark.

Re: docker-image-debian-all-test-cross errors

[Richard Henderson]

On 7/4/26 14:41, Mark Cave-Ayland wrote:
> On 04/07/2026 22:20, Richard Henderson wrote:
> 
>> Has anyone else been seeing
>>
>> Error: Failed to fetch http://deb.debian.org/debian/pool/main/d/dpkg/libdpkg- 
>> perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
>> Error: Failed to fetch http://deb.debian.org/debian/pool/main/d/dpkg/dpkg- 
>> dev_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
>> Error: Failed to fetch http://deb.debian.org/debian/pool/main/p/python3.13/ 
>> libpython3.13_3.13.5-2_amd64.deb  404 Not Found [IP: 151.101.22.132 80]
>> Error: Failed to fetch http://deb.debian.org/debian/pool/main/g/glibc/libc6- 
>> i386_2.41-12_amd64.deb  404  Not Found [IP: 151.101.22.132 80]
>> Error: Unable to fetch some archives, maybe run apt-get update or try with --fix- missing?
>> Error: building at STEP "RUN DEBIAN_FRONTEND=noninteractive eatmydata   apt install - y 
>> --no-install-recommends   clang  dpkg-dev  gdb-multiarch  libclang-rt-dev": while 
>> running runtime: exit status 100
>> make: *** [/home/rth/qemu/src/tests/docker/Makefile.include:43: docker-image-debian- 
>> all-test-cross] Error 100
>>
>> This has been consistently failing for me for a couple of weeks...
>>
>> r~
> 
> I wonder if docker build has cached the existing layers, and since apt-get update only 
> runs in the first part of the Dockerfile then the subsequent layers try to access an old 
> base image?
> 
> Does running "docker builder prune" (possibly with --all) help at all? That should purge 
> the cache and force docker to rebuild all the layers again.

No, it doesn't.


r~

Re: docker-image-debian-all-test-cross errors

[Daniel P. Berrangé]

On Sat, Jul 04, 2026 at 09:41:34PM -0700, Richard Henderson wrote:
> On 7/4/26 14:41, Mark Cave-Ayland wrote:
> > On 04/07/2026 22:20, Richard Henderson wrote:
> > 
> > > Has anyone else been seeing
> > > 
> > > Error: Failed to fetch
> > > http://deb.debian.org/debian/pool/main/d/dpkg/libdpkg-
> > > perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
> > > Error: Failed to fetch
> > > http://deb.debian.org/debian/pool/main/d/dpkg/dpkg-
> > > dev_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
> > > Error: Failed to fetch
> > > http://deb.debian.org/debian/pool/main/p/python3.13/
> > > libpython3.13_3.13.5-2_amd64.deb  404 Not Found [IP: 151.101.22.132
> > > 80]
> > > Error: Failed to fetch
> > > http://deb.debian.org/debian/pool/main/g/glibc/libc6-
> > > i386_2.41-12_amd64.deb  404  Not Found [IP: 151.101.22.132 80]
> > > Error: Unable to fetch some archives, maybe run apt-get update or try with --fix- missing?
> > > Error: building at STEP "RUN DEBIAN_FRONTEND=noninteractive
> > > eatmydata   apt install - y --no-install-recommends   clang 
> > > dpkg-dev  gdb-multiarch  libclang-rt-dev": while running runtime:
> > > exit status 100
> > > make: *** [/home/rth/qemu/src/tests/docker/Makefile.include:43:
> > > docker-image-debian- all-test-cross] Error 100
> > > 
> > > This has been consistently failing for me for a couple of weeks...
> > > 
> > > r~
> > 
> > I wonder if docker build has cached the existing layers, and since
> > apt-get update only runs in the first part of the Dockerfile then the
> > subsequent layers try to access an old base image?
> > 
> > Does running "docker builder prune" (possibly with --all) help at all?
> > That should purge the cache and force docker to rebuild all the layers
> > again.
> 
> No, it doesn't.

Note QEMU will preferentially to use 'podman' if both docker and
podman are installed. So 'docker builder prune' might be purging
the wrong cache.  Try 'podman builder prune', or manually run
"podman pull docker.io/library/debian:13-slim" to force refresh
the base image.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: docker-image-debian-all-test-cross errors

[Richard Henderson]

On 7/6/26 01:34, Daniel P. Berrangé wrote:
> Note QEMU will preferentially to use 'podman' if both docker and
> podman are installed. So 'docker builder prune' might be purging
> the wrong cache.  Try 'podman builder prune', or manually run
> "podman pull docker.io/library/debian:13-slim" to force refresh
> the base image.

Thanks, that did it.

r~

Re: docker-image-debian-all-test-cross errors

[Michael Tokarev]

On 05.07.2026 00:20, Richard Henderson wrote:
> Has anyone else been seeing
> 
> Error: Failed to fetch http://deb.debian.org/debian/pool/main/d/dpkg/ 
> libdpkg-perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
...

dpkg (and all related packages) has been upgraded from 1.22.21 to
1.22.22 version quite a while (on Mar-2026).  Initially debian trixie
had dpkg 1.22.21.

I wonder where it is getting the old version reference from?
Should the previous layer be refreshed to point to the current trixie
point release?

/mjt

Re: docker-image-debian-all-test-cross errors

[Mark Cave-Ayland]

On 05/07/2026 10:07, Michael Tokarev wrote:

> On 05.07.2026 00:20, Richard Henderson wrote:
>> Has anyone else been seeing
>>
>> Error: Failed to fetch http://deb.debian.org/debian/pool/main/d/dpkg/ libdpkg- 
>> perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
> ...
> 
> dpkg (and all related packages) has been upgraded from 1.22.21 to
> 1.22.22 version quite a while (on Mar-2026).  Initially debian trixie
> had dpkg 1.22.21.
> 
> I wonder where it is getting the old version reference from?
> Should the previous layer be refreshed to point to the current trixie
> point release?
> 
> /mjt

If that is the case then I think you can fix it by doing a manual update of the base 
image to pull in the latest update locally i.e. "docker pull 
docker.io/library/debian:13-slim" and then rebuilding the docker images once again?


ATB,

Mark.

Re: docker-image-debian-all-test-cross errors

[Alex Bennée]

Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:

> On 05/07/2026 10:07, Michael Tokarev wrote:
>
>> On 05.07.2026 00:20, Richard Henderson wrote:
>>> Has anyone else been seeing
>>>
>>> Error: Failed to fetch
>>> http://deb.debian.org/debian/pool/main/d/dpkg/ libdpkg-
>>> perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
>> ...
>> dpkg (and all related packages) has been upgraded from 1.22.21 to
>> 1.22.22 version quite a while (on Mar-2026).  Initially debian trixie
>> had dpkg 1.22.21.
>> I wonder where it is getting the old version reference from?
>> Should the previous layer be refreshed to point to the current trixie
>> point release?
>> /mjt
>
> If that is the case then I think you can fix it by doing a manual
> update of the base image to pull in the latest update locally i.e.
> "docker pull docker.io/library/debian:13-slim" and then rebuilding the
> docker images once again?

You can always trigger a local build:

  make docker-image-all-test-cross NOCACHE=1

it looks like the source tree build of images has been broken again as
UNCHECKED_GOALS isn't being honoured for me.

>
>
> ATB,
>
> Mark.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: docker-image-debian-all-test-cross errors

[Pierrick Bouvier]

On 7/6/2026 1:35 AM, Alex Bennée wrote:
> Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:
> 
>> On 05/07/2026 10:07, Michael Tokarev wrote:
>>
>>> On 05.07.2026 00:20, Richard Henderson wrote:
>>>> Has anyone else been seeing
>>>>
>>>> Error: Failed to fetch
>>>> http://deb.debian.org/debian/pool/main/d/dpkg/ libdpkg-
>>>> perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
>>> ...
>>> dpkg (and all related packages) has been upgraded from 1.22.21 to
>>> 1.22.22 version quite a while (on Mar-2026).  Initially debian trixie
>>> had dpkg 1.22.21.
>>> I wonder where it is getting the old version reference from?
>>> Should the previous layer be refreshed to point to the current trixie
>>> point release?
>>> /mjt
>>
>> If that is the case then I think you can fix it by doing a manual
>> update of the base image to pull in the latest update locally i.e.
>> "docker pull docker.io/library/debian:13-slim" and then rebuilding the
>> docker images once again?
> 
> You can always trigger a local build:
> 
>   make docker-image-all-test-cross NOCACHE=1
> 
> it looks like the source tree build of images has been broken again as
> UNCHECKED_GOALS isn't being honoured for me.
> 
>>
>>
>> ATB,
>>
>> Mark.
> 

The right fix is to modify our dockerfiles to always combine apt update
and apt install, to avoid those kind of errors.

See this doc for more details:
https://docs.docker.com/build/building/best-practices/#apt-get

Regards,
Pierrick

Re: docker-image-debian-all-test-cross errors

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 7/6/2026 1:35 AM, Alex Bennée wrote:
>> Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:
>> 
>>> On 05/07/2026 10:07, Michael Tokarev wrote:
>>>
>>>> On 05.07.2026 00:20, Richard Henderson wrote:
>>>>> Has anyone else been seeing
>>>>>
>>>>> Error: Failed to fetch
>>>>> http://deb.debian.org/debian/pool/main/d/dpkg/ libdpkg-
>>>>> perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
>>>> ...
>>>> dpkg (and all related packages) has been upgraded from 1.22.21 to
>>>> 1.22.22 version quite a while (on Mar-2026).  Initially debian trixie
>>>> had dpkg 1.22.21.
>>>> I wonder where it is getting the old version reference from?
>>>> Should the previous layer be refreshed to point to the current trixie
>>>> point release?
>>>> /mjt
>>>
>>> If that is the case then I think you can fix it by doing a manual
>>> update of the base image to pull in the latest update locally i.e.
>>> "docker pull docker.io/library/debian:13-slim" and then rebuilding the
>>> docker images once again?
>> 
>> You can always trigger a local build:
>> 
>>   make docker-image-all-test-cross NOCACHE=1
>> 
>> it looks like the source tree build of images has been broken again as
>> UNCHECKED_GOALS isn't being honoured for me.
>> 
>>>
>>>
>>> ATB,
>>>
>>> Mark.
>> 
>
> The right fix is to modify our dockerfiles to always combine apt update
> and apt install, to avoid those kind of errors.

We do for the main part of the dependencies from lcitool. We then do a
second step for the cross compilers for which there are shenanigans to
deal with non-x86 arches before installing them.

But the original failure from perl should have been picked up by the
first bit.

>
> See this doc for more details:
> https://docs.docker.com/build/building/best-practices/#apt-get
>
> Regards,
> Pierrick

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: docker-image-debian-all-test-cross errors

[Pierrick Bouvier]

On 7/7/2026 2:29 AM, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 7/6/2026 1:35 AM, Alex Bennée wrote:
>>> Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:
>>>
>>>> On 05/07/2026 10:07, Michael Tokarev wrote:
>>>>
>>>>> On 05.07.2026 00:20, Richard Henderson wrote:
>>>>>> Has anyone else been seeing
>>>>>>
>>>>>> Error: Failed to fetch
>>>>>> http://deb.debian.org/debian/pool/main/d/dpkg/ libdpkg-
>>>>>> perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
>>>>> ...
>>>>> dpkg (and all related packages) has been upgraded from 1.22.21 to
>>>>> 1.22.22 version quite a while (on Mar-2026).  Initially debian trixie
>>>>> had dpkg 1.22.21.
>>>>> I wonder where it is getting the old version reference from?
>>>>> Should the previous layer be refreshed to point to the current trixie
>>>>> point release?
>>>>> /mjt
>>>>
>>>> If that is the case then I think you can fix it by doing a manual
>>>> update of the base image to pull in the latest update locally i.e.
>>>> "docker pull docker.io/library/debian:13-slim" and then rebuilding the
>>>> docker images once again?
>>>
>>> You can always trigger a local build:
>>>
>>>   make docker-image-all-test-cross NOCACHE=1
>>>
>>> it looks like the source tree build of images has been broken again as
>>> UNCHECKED_GOALS isn't being honoured for me.
>>>
>>>>
>>>>
>>>> ATB,
>>>>
>>>> Mark.
>>>
>>
>> The right fix is to modify our dockerfiles to always combine apt update
>> and apt install, to avoid those kind of errors.
> 
> We do for the main part of the dependencies from lcitool. We then do a
> second step for the cross compilers for which there are shenanigans to
> deal with non-x86 arches before installing them.
> 
> But the original failure from perl should have been picked up by the
> first bit.
>

If you look at original failure from Richard, it's this reason:

```
Error: Unable to fetch some archives, maybe run apt-get update or try
with --fix-missing?
Error: building at STEP "RUN DEBIAN_FRONTEND=noninteractive eatmydata
apt install -y --no-install-recommends   clang  dpkg-dev  gdb-multiarch
libclang-rt-dev"
```

apt is even polite enough to say we should do an apt update before :)

I'll send a series fixing this, I found 3 dockerfiles that have missing
apt update.

>>
>> See this doc for more details:
>> https://docs.docker.com/build/building/best-practices/#apt-get
>>
>> Regards,
>> Pierrick
> 

Regards,
Pierrick

Re: docker-image-debian-all-test-cross errors

[Pierrick Bouvier]

On 7/7/2026 9:34 AM, Pierrick Bouvier wrote:
> On 7/7/2026 2:29 AM, Alex Bennée wrote:
>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>
>>> On 7/6/2026 1:35 AM, Alex Bennée wrote:
>>>> Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:
>>>>
>>>>> On 05/07/2026 10:07, Michael Tokarev wrote:
>>>>>
>>>>>> On 05.07.2026 00:20, Richard Henderson wrote:
>>>>>>> Has anyone else been seeing
>>>>>>>
>>>>>>> Error: Failed to fetch
>>>>>>> http://deb.debian.org/debian/pool/main/d/dpkg/ libdpkg-
>>>>>>> perl_1.22.21_all.deb  404  Not Found [IP: 151.101.22.132 80]
>>>>>> ...
>>>>>> dpkg (and all related packages) has been upgraded from 1.22.21 to
>>>>>> 1.22.22 version quite a while (on Mar-2026).  Initially debian trixie
>>>>>> had dpkg 1.22.21.
>>>>>> I wonder where it is getting the old version reference from?
>>>>>> Should the previous layer be refreshed to point to the current trixie
>>>>>> point release?
>>>>>> /mjt
>>>>>
>>>>> If that is the case then I think you can fix it by doing a manual
>>>>> update of the base image to pull in the latest update locally i.e.
>>>>> "docker pull docker.io/library/debian:13-slim" and then rebuilding the
>>>>> docker images once again?
>>>>
>>>> You can always trigger a local build:
>>>>
>>>>   make docker-image-all-test-cross NOCACHE=1
>>>>
>>>> it looks like the source tree build of images has been broken again as
>>>> UNCHECKED_GOALS isn't being honoured for me.
>>>>
>>>>>
>>>>>
>>>>> ATB,
>>>>>
>>>>> Mark.
>>>>
>>>
>>> The right fix is to modify our dockerfiles to always combine apt update
>>> and apt install, to avoid those kind of errors.
>>
>> We do for the main part of the dependencies from lcitool. We then do a
>> second step for the cross compilers for which there are shenanigans to
>> deal with non-x86 arches before installing them.
>>
>> But the original failure from perl should have been picked up by the
>> first bit.
>>
> 
> If you look at original failure from Richard, it's this reason:
> 
> ```
> Error: Unable to fetch some archives, maybe run apt-get update or try
> with --fix-missing?
> Error: building at STEP "RUN DEBIAN_FRONTEND=noninteractive eatmydata
> apt install -y --no-install-recommends   clang  dpkg-dev  gdb-multiarch
> libclang-rt-dev"
> ```
> 
> apt is even polite enough to say we should do an apt update before :)
> 
> I'll send a series fixing this, I found 3 dockerfiles that have missing
> apt update.
>

Series sent:
https://lore.kernel.org/qemu-devel/20260707214655.1138626-1-pierrick.bouvier@oss.qualcomm.com/T/#

Regards,
Pierrick