Re: FreeBSD 14.1 aarch64 iso URL is down

Re: FreeBSD 14.1 aarch64 iso URL is down

[Stefan Hajnoczi]

On Sat, Jun 21, 2025 at 7:59 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:

(I forgot to CC qemu-devel)

>
> Hi,
> This might only be temporary, but the CI is getting HTTP 404 Not Found
> for the following URL:
> https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso
>
> https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848
>
> Stefan

Re: FreeBSD 14.1 aarch64 iso URL is down

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 585 bytes --]

On Sat, Jun 21, 2025, 6:01 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Sat, Jun 21, 2025 at 7:59 PM Stefan Hajnoczi <stefanha@gmail.com>
> wrote:
>
> (I forgot to CC qemu-devel)
>
> >
> > Hi,
> > This might only be temporary, but the CI is getting HTTP 404 Not Found
> > for the following URL:
> >
> https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso
> >
> > https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848
> >
> > Stefan
>

Time to bump the version to 14.3.

Warner

>

[-- Attachment #2: Type: text/html, Size: 1575 bytes --]

Re: FreeBSD 14.1 aarch64 iso URL is down

[Thomas Huth]

On 22/06/2025 03.46, Warner Losh wrote:
> 
> 
> On Sat, Jun 21, 2025, 6:01 PM Stefan Hajnoczi <stefanha@gmail.com 
> <mailto:stefanha@gmail.com>> wrote:
> 
>     On Sat, Jun 21, 2025 at 7:59 PM Stefan Hajnoczi <stefanha@gmail.com
>     <mailto:stefanha@gmail.com>> wrote:
> 
>     (I forgot to CC qemu-devel)
> 
>      >
>      > Hi,
>      > This might only be temporary, but the CI is getting HTTP 404 Not Found
>      > for the following URL:
>      > https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
>     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso <https://
>     download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
>     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso>
>      >
>      > https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848
>     <https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848>
>      >
>      > Stefan
> 
> 
> Time to bump the version to 14.3.

Hmm, while we're used to refresh the CI images for the *host* environments, 
it's rather ugly to see images for the *guests* of the functional tests 
disappear. Maybe we should rather remove that test if the URL is not stable?

  Thomas

Re: FreeBSD 14.1 aarch64 iso URL is down

[Warner Losh]

On Tue, Jun 24, 2025 at 10:02 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 22/06/2025 03.46, Warner Losh wrote:
> >
> >
> > On Sat, Jun 21, 2025, 6:01 PM Stefan Hajnoczi <stefanha@gmail.com
> > <mailto:stefanha@gmail.com>> wrote:
> >
> >     On Sat, Jun 21, 2025 at 7:59 PM Stefan Hajnoczi <stefanha@gmail.com
> >     <mailto:stefanha@gmail.com>> wrote:
> >
> >     (I forgot to CC qemu-devel)
> >
> >      >
> >      > Hi,
> >      > This might only be temporary, but the CI is getting HTTP 404 Not Found
> >      > for the following URL:
> >      > https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso <https://
> >     download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso>
> >      >
> >      > https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848
> >     <https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848>
> >      >
> >      > Stefan
> >
> >
> > Time to bump the version to 14.3.
>
> Hmm, while we're used to refresh the CI images for the *host* environments,
> it's rather ugly to see images for the *guests* of the functional tests
> disappear. Maybe we should rather remove that test if the URL is not stable?

Yes. Most of our images have a shelf life of about a year to 18 months. And QEMU
should be testing all the 'supported' FreeBSD images, just like for
other projects.
The question becomes how can we, the FreeBSD Project, remove the friction that's
here now because we timeout / move the older images as they pass out of support.

We've also just shifted to a more frequent release cadence, so the
releases have gone
from living for  18-24 months down to just 12. We just released
FreeBSD 14.3, and 14.1
is only a year old. So what's the best way of dealing with this? We
could have a 14-latest
but the md5 would change...

So I'm open to making a change, but I need help crafting something
that will work, since
I'm not clever enough to suggest something here.

Warner

Re: FreeBSD 14.1 aarch64 iso URL is down

[Stefan Hajnoczi]

On Tue, Jun 24, 2025 at 12:28 PM Warner Losh <imp@bsdimp.com> wrote:
>
> On Tue, Jun 24, 2025 at 10:02 AM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 22/06/2025 03.46, Warner Losh wrote:
> > >
> > >
> > > On Sat, Jun 21, 2025, 6:01 PM Stefan Hajnoczi <stefanha@gmail.com
> > > <mailto:stefanha@gmail.com>> wrote:
> > >
> > >     On Sat, Jun 21, 2025 at 7:59 PM Stefan Hajnoczi <stefanha@gmail.com
> > >     <mailto:stefanha@gmail.com>> wrote:
> > >
> > >     (I forgot to CC qemu-devel)
> > >
> > >      >
> > >      > Hi,
> > >      > This might only be temporary, but the CI is getting HTTP 404 Not Found
> > >      > for the following URL:
> > >      > https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> > >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso <https://
> > >     download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> > >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso>
> > >      >
> > >      > https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848
> > >     <https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848>
> > >      >
> > >      > Stefan
> > >
> > >
> > > Time to bump the version to 14.3.
> >
> > Hmm, while we're used to refresh the CI images for the *host* environments,
> > it's rather ugly to see images for the *guests* of the functional tests
> > disappear. Maybe we should rather remove that test if the URL is not stable?
>
> Yes. Most of our images have a shelf life of about a year to 18 months. And QEMU
> should be testing all the 'supported' FreeBSD images, just like for
> other projects.
> The question becomes how can we, the FreeBSD Project, remove the friction that's
> here now because we timeout / move the older images as they pass out of support.
>
> We've also just shifted to a more frequent release cadence, so the
> releases have gone
> from living for  18-24 months down to just 12. We just released
> FreeBSD 14.3, and 14.1
> is only a year old. So what's the best way of dealing with this? We
> could have a 14-latest
> but the md5 would change...
>
> So I'm open to making a change, but I need help crafting something
> that will work, since
> I'm not clever enough to suggest something here.

A test run should be repeatable. If a test passes on a given qemu.git
commit then it should continue to pass when run again. Using -latest
breaks this property, so let's avoid it.

Ideas:
1. FreeBSD provides convenient permanent URLs.
2. QEMU uses a permanent FreeBSD ISO mirror URL instead. Need to find
a mirror that is fast and reliable.
3. Someone agrees to regularly update the URL in QEMU's test suite so
that breakage isn't exposed. IMO the least desirable solution because
an old copy of the test will start failing after 12 months.

Stefan

Re: FreeBSD 14.1 aarch64 iso URL is down

[Warner Losh]

On Tue, Jun 24, 2025 at 11:16 AM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Tue, Jun 24, 2025 at 12:28 PM Warner Losh <imp@bsdimp.com> wrote:
> >
> > On Tue, Jun 24, 2025 at 10:02 AM Thomas Huth <thuth@redhat.com> wrote:
> > >
> > > On 22/06/2025 03.46, Warner Losh wrote:
> > > >
> > > >
> > > > On Sat, Jun 21, 2025, 6:01 PM Stefan Hajnoczi <stefanha@gmail.com
> > > > <mailto:stefanha@gmail.com>> wrote:
> > > >
> > > >     On Sat, Jun 21, 2025 at 7:59 PM Stefan Hajnoczi <stefanha@gmail.com
> > > >     <mailto:stefanha@gmail.com>> wrote:
> > > >
> > > >     (I forgot to CC qemu-devel)
> > > >
> > > >      >
> > > >      > Hi,
> > > >      > This might only be temporary, but the CI is getting HTTP 404 Not Found
> > > >      > for the following URL:
> > > >      > https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> > > >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso <https://
> > > >     download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> > > >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso>
> > > >      >
> > > >      > https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848
> > > >     <https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848>
> > > >      >
> > > >      > Stefan
> > > >
> > > >
> > > > Time to bump the version to 14.3.
> > >
> > > Hmm, while we're used to refresh the CI images for the *host* environments,
> > > it's rather ugly to see images for the *guests* of the functional tests
> > > disappear. Maybe we should rather remove that test if the URL is not stable?
> >
> > Yes. Most of our images have a shelf life of about a year to 18 months. And QEMU
> > should be testing all the 'supported' FreeBSD images, just like for
> > other projects.
> > The question becomes how can we, the FreeBSD Project, remove the friction that's
> > here now because we timeout / move the older images as they pass out of support.
> >
> > We've also just shifted to a more frequent release cadence, so the
> > releases have gone
> > from living for  18-24 months down to just 12. We just released
> > FreeBSD 14.3, and 14.1
> > is only a year old. So what's the best way of dealing with this? We
> > could have a 14-latest
> > but the md5 would change...
> >
> > So I'm open to making a change, but I need help crafting something
> > that will work, since
> > I'm not clever enough to suggest something here.
>
> A test run should be repeatable. If a test passes on a given qemu.git
> commit then it should continue to pass when run again. Using -latest
> breaks this property, so let's avoid it.
>
> Ideas:
> 1. FreeBSD provides convenient permanent URLs.
> 2. QEMU uses a permanent FreeBSD ISO mirror URL instead. Need to find
> a mirror that is fast and reliable.
> 3. Someone agrees to regularly update the URL in QEMU's test suite so
> that breakage isn't exposed. IMO the least desirable solution because
> an old copy of the test will start failing after 12 months.

So there's two issues at play.

FreeBSD does maintain all our archival releases forever. They never change.
But, we don't have permanent links to them today. We start with one URL and
then migrate to a second one when they transition from supported to unsupported.
We do this, in part, to make sure people upgrade. So in effect, this breakage
means that our notion is "working" in the sense that the FreeBSD project's goals
of making people "keep up to date."

This does, I realize, clash with the views that QEMU wants to have some stable
way to test images over time, even if the upstream's notion of supported or not
changes.

One easy idea might be to 'prestage' the 'legacy' releases when they
are supported
on the 'legacy' server so that tests can be written with the legacy
path so that these
tests always work, now and in the future.

So, this is terrible from a FreeBSD point of view. We'd like it if
qemu always tested
all of our releases, as well as snapshots of the tip of the spear.
There's got to be some
way to have some shared responsibility that we can automate. FreeBSD could test
the most recent release of qemu against a bunch of images in our CI
cluster. But we
don't actually have a CI cluster we could put that into (our focus is
just a little different)
today. Ideally, your (3) above would happen as we rotate in new
versions and out old
versions of FreeBSD. But honestly, I'm the person (or one of the
people) that should
be keeping his eye on the ball, but we see how well that has worked
out. So the question
becomes is this a management failure (eg, someone/something needs to prompt me
or others in the FreeBSD project to update it via reminders, release
checklists, etc)?
Or is it something that can fixed by automations somehow? I don't know...

Warner

Re: FreeBSD 14.1 aarch64 iso URL is down

[Stefan Hajnoczi]

On Tue, Jun 24, 2025 at 1:41 PM Warner Losh <imp@bsdimp.com> wrote:
>
> On Tue, Jun 24, 2025 at 11:16 AM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >
> > On Tue, Jun 24, 2025 at 12:28 PM Warner Losh <imp@bsdimp.com> wrote:
> > >
> > > On Tue, Jun 24, 2025 at 10:02 AM Thomas Huth <thuth@redhat.com> wrote:
> > > >
> > > > On 22/06/2025 03.46, Warner Losh wrote:
> > > > >
> > > > >
> > > > > On Sat, Jun 21, 2025, 6:01 PM Stefan Hajnoczi <stefanha@gmail.com
> > > > > <mailto:stefanha@gmail.com>> wrote:
> > > > >
> > > > >     On Sat, Jun 21, 2025 at 7:59 PM Stefan Hajnoczi <stefanha@gmail.com
> > > > >     <mailto:stefanha@gmail.com>> wrote:
> > > > >
> > > > >     (I forgot to CC qemu-devel)
> > > > >
> > > > >      >
> > > > >      > Hi,
> > > > >      > This might only be temporary, but the CI is getting HTTP 404 Not Found
> > > > >      > for the following URL:
> > > > >      > https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> > > > >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso <https://
> > > > >     download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> > > > >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso>
> > > > >      >
> > > > >      > https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848
> > > > >     <https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848>
> > > > >      >
> > > > >      > Stefan
> > > > >
> > > > >
> > > > > Time to bump the version to 14.3.
> > > >
> > > > Hmm, while we're used to refresh the CI images for the *host* environments,
> > > > it's rather ugly to see images for the *guests* of the functional tests
> > > > disappear. Maybe we should rather remove that test if the URL is not stable?
> > >
> > > Yes. Most of our images have a shelf life of about a year to 18 months. And QEMU
> > > should be testing all the 'supported' FreeBSD images, just like for
> > > other projects.
> > > The question becomes how can we, the FreeBSD Project, remove the friction that's
> > > here now because we timeout / move the older images as they pass out of support.
> > >
> > > We've also just shifted to a more frequent release cadence, so the
> > > releases have gone
> > > from living for  18-24 months down to just 12. We just released
> > > FreeBSD 14.3, and 14.1
> > > is only a year old. So what's the best way of dealing with this? We
> > > could have a 14-latest
> > > but the md5 would change...
> > >
> > > So I'm open to making a change, but I need help crafting something
> > > that will work, since
> > > I'm not clever enough to suggest something here.
> >
> > A test run should be repeatable. If a test passes on a given qemu.git
> > commit then it should continue to pass when run again. Using -latest
> > breaks this property, so let's avoid it.
> >
> > Ideas:
> > 1. FreeBSD provides convenient permanent URLs.
> > 2. QEMU uses a permanent FreeBSD ISO mirror URL instead. Need to find
> > a mirror that is fast and reliable.
> > 3. Someone agrees to regularly update the URL in QEMU's test suite so
> > that breakage isn't exposed. IMO the least desirable solution because
> > an old copy of the test will start failing after 12 months.
>
> So there's two issues at play.
>
> FreeBSD does maintain all our archival releases forever. They never change.
> But, we don't have permanent links to them today. We start with one URL and
> then migrate to a second one when they transition from supported to unsupported.
> We do this, in part, to make sure people upgrade. So in effect, this breakage
> means that our notion is "working" in the sense that the FreeBSD project's goals
> of making people "keep up to date."
>
> This does, I realize, clash with the views that QEMU wants to have some stable
> way to test images over time, even if the upstream's notion of supported or not
> changes.
>
> One easy idea might be to 'prestage' the 'legacy' releases when they
> are supported
> on the 'legacy' server so that tests can be written with the legacy
> path so that these
> tests always work, now and in the future.
>
> So, this is terrible from a FreeBSD point of view. We'd like it if
> qemu always tested
> all of our releases, as well as snapshots of the tip of the spear.
> There's got to be some
> way to have some shared responsibility that we can automate. FreeBSD could test
> the most recent release of qemu against a bunch of images in our CI
> cluster. But we
> don't actually have a CI cluster we could put that into (our focus is
> just a little different)
> today. Ideally, your (3) above would happen as we rotate in new
> versions and out old
> versions of FreeBSD. But honestly, I'm the person (or one of the
> people) that should
> be keeping his eye on the ball, but we see how well that has worked
> out. So the question
> becomes is this a management failure (eg, someone/something needs to prompt me
> or others in the FreeBSD project to update it via reminders, release
> checklists, etc)?
> Or is it something that can fixed by automations somehow? I don't know...

How about doing both:
1. Making the "legacy" URL available immediately so that anything that
needs a permalink can use it (but they will explicitly specify the
word "legacy" in the URL, which is a hint that it's not the latest and
greatest release).
2. You set up a calendar reminder to send a patch updating QEMU's test
suite to the latest FreeBSD release every 12 months. A shell script
could perform the steps of updating the URL, committing the change,
and sending a patch email.

That way FreeBSD's latest release will be tested in a timely manner
and a snapshot of the QEMU test suite will still pass after 12 months.

What do you think?

Stefan

Re: FreeBSD 14.1 aarch64 iso URL is down

[Warner Losh]

On Tue, Jun 24, 2025 at 3:25 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Tue, Jun 24, 2025 at 1:41 PM Warner Losh <imp@bsdimp.com> wrote:
> >
> > On Tue, Jun 24, 2025 at 11:16 AM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >
> > > On Tue, Jun 24, 2025 at 12:28 PM Warner Losh <imp@bsdimp.com> wrote:
> > > >
> > > > On Tue, Jun 24, 2025 at 10:02 AM Thomas Huth <thuth@redhat.com> wrote:
> > > > >
> > > > > On 22/06/2025 03.46, Warner Losh wrote:
> > > > > >
> > > > > >
> > > > > > On Sat, Jun 21, 2025, 6:01 PM Stefan Hajnoczi <stefanha@gmail.com
> > > > > > <mailto:stefanha@gmail.com>> wrote:
> > > > > >
> > > > > >     On Sat, Jun 21, 2025 at 7:59 PM Stefan Hajnoczi <stefanha@gmail.com
> > > > > >     <mailto:stefanha@gmail.com>> wrote:
> > > > > >
> > > > > >     (I forgot to CC qemu-devel)
> > > > > >
> > > > > >      >
> > > > > >      > Hi,
> > > > > >      > This might only be temporary, but the CI is getting HTTP 404 Not Found
> > > > > >      > for the following URL:
> > > > > >      > https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> > > > > >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso <https://
> > > > > >     download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/
> > > > > >     FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso>
> > > > > >      >
> > > > > >      > https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848
> > > > > >     <https://gitlab.com/qemu-project/qemu/-/jobs/10424901718#L848>
> > > > > >      >
> > > > > >      > Stefan
> > > > > >
> > > > > >
> > > > > > Time to bump the version to 14.3.
> > > > >
> > > > > Hmm, while we're used to refresh the CI images for the *host* environments,
> > > > > it's rather ugly to see images for the *guests* of the functional tests
> > > > > disappear. Maybe we should rather remove that test if the URL is not stable?
> > > >
> > > > Yes. Most of our images have a shelf life of about a year to 18 months. And QEMU
> > > > should be testing all the 'supported' FreeBSD images, just like for
> > > > other projects.
> > > > The question becomes how can we, the FreeBSD Project, remove the friction that's
> > > > here now because we timeout / move the older images as they pass out of support.
> > > >
> > > > We've also just shifted to a more frequent release cadence, so the
> > > > releases have gone
> > > > from living for  18-24 months down to just 12. We just released
> > > > FreeBSD 14.3, and 14.1
> > > > is only a year old. So what's the best way of dealing with this? We
> > > > could have a 14-latest
> > > > but the md5 would change...
> > > >
> > > > So I'm open to making a change, but I need help crafting something
> > > > that will work, since
> > > > I'm not clever enough to suggest something here.
> > >
> > > A test run should be repeatable. If a test passes on a given qemu.git
> > > commit then it should continue to pass when run again. Using -latest
> > > breaks this property, so let's avoid it.
> > >
> > > Ideas:
> > > 1. FreeBSD provides convenient permanent URLs.
> > > 2. QEMU uses a permanent FreeBSD ISO mirror URL instead. Need to find
> > > a mirror that is fast and reliable.
> > > 3. Someone agrees to regularly update the URL in QEMU's test suite so
> > > that breakage isn't exposed. IMO the least desirable solution because
> > > an old copy of the test will start failing after 12 months.
> >
> > So there's two issues at play.
> >
> > FreeBSD does maintain all our archival releases forever. They never change.
> > But, we don't have permanent links to them today. We start with one URL and
> > then migrate to a second one when they transition from supported to unsupported.
> > We do this, in part, to make sure people upgrade. So in effect, this breakage
> > means that our notion is "working" in the sense that the FreeBSD project's goals
> > of making people "keep up to date."
> >
> > This does, I realize, clash with the views that QEMU wants to have some stable
> > way to test images over time, even if the upstream's notion of supported or not
> > changes.
> >
> > One easy idea might be to 'prestage' the 'legacy' releases when they
> > are supported
> > on the 'legacy' server so that tests can be written with the legacy
> > path so that these
> > tests always work, now and in the future.
> >
> > So, this is terrible from a FreeBSD point of view. We'd like it if
> > qemu always tested
> > all of our releases, as well as snapshots of the tip of the spear.
> > There's got to be some
> > way to have some shared responsibility that we can automate. FreeBSD could test
> > the most recent release of qemu against a bunch of images in our CI
> > cluster. But we
> > don't actually have a CI cluster we could put that into (our focus is
> > just a little different)
> > today. Ideally, your (3) above would happen as we rotate in new
> > versions and out old
> > versions of FreeBSD. But honestly, I'm the person (or one of the
> > people) that should
> > be keeping his eye on the ball, but we see how well that has worked
> > out. So the question
> > becomes is this a management failure (eg, someone/something needs to prompt me
> > or others in the FreeBSD project to update it via reminders, release
> > checklists, etc)?
> > Or is it something that can fixed by automations somehow? I don't know...
>
> How about doing both:
> 1. Making the "legacy" URL available immediately so that anything that
> needs a permalink can use it (but they will explicitly specify the
> word "legacy" in the URL, which is a hint that it's not the latest and
> greatest release).
> 2. You set up a calendar reminder to send a patch updating QEMU's test
> suite to the latest FreeBSD release every 12 months. A shell script
> could perform the steps of updating the URL, committing the change,
> and sending a patch email.
>
> That way FreeBSD's latest release will be tested in a timely manner
> and a snapshot of the QEMU test suite will still pass after 12 months.
>
> What do you think?

So, ideally, we could try two URLs.  The "download" path has the full
CDN that we operate behind it. It's the least load on our
infrastructure. However, if that fails, the "archive" path will have
the same images. The "archive" path could be used all the time if the
two URL strategy can't be used. The downloads will be slower, but the
images are already transferred to the archive machine (not cdn) about
a week or so after the release. If the load is going to be super low,
this would be acceptable (and I suspect that it would be since we
cache these images). This would allow older releases to keep working
(once we transition to either the two or one url strategies), and all
the other nice features having the same tests today and 4 years from
now, should we ever need to test that, say for regressions. What's the
anticipated load, measured in downloads per day say, this testing
generates?

As for the second one, I've added reminders as well as a request to
our release team to ask me if I've done it a month after the release.
We'll see how well that works out since the update takes minutes.

Speaking of which, has someone done the update? I'm a bit behind on my
qemu stuff and haven't been paying attention.

Warner

Re: FreeBSD 14.1 aarch64 iso URL is down

[Thomas Huth]

On 26/06/2025 04.53, Warner Losh wrote:
> [...] What's the
> anticipated load, measured in downloads per day say, this testing
> generates?

Ideally, the functional tests download the assets once and then cache them. 
However, it's currently broken in the non-shared CI runners of the 
qemu-project (it's working in forked repos with the shared runners), so 
expect a fistful of downloads per day (I guess 5 downloads per day would be 
a reasonable number to expect).

> Speaking of which, has someone done the update? I'm a bit behind on my
> qemu stuff and haven't been paying attention.

AFAICT nobody has sent a patch yet, so it's currently still broken.

  Thomas

Re: FreeBSD 14.1 aarch64 iso URL is down

[Stefan Hajnoczi]

On Thu, Jun 26, 2025 at 2:02 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 26/06/2025 04.53, Warner Losh wrote:
> > [...] What's the
> > anticipated load, measured in downloads per day say, this testing
> > generates?
>
> Ideally, the functional tests download the assets once and then cache them.
> However, it's currently broken in the non-shared CI runners of the
> qemu-project (it's working in forked repos with the shared runners), so
> expect a fistful of downloads per day (I guess 5 downloads per day would be
> a reasonable number to expect).
>
> > Speaking of which, has someone done the update? I'm a bit behind on my
> > qemu stuff and haven't been paying attention.
>
> AFAICT nobody has sent a patch yet, so it's currently still broken.

Hi Thomas,
Thanks for sending a patch to use the archive server.

Warner: If you want to update the test environment to 14.3 or change
which URLs QEMU's test suite uses, please feel free to send additional
patches. For now Thomas' patch will let the test pass again.

Stefan

Re: FreeBSD 14.1 aarch64 iso URL is down

[Daniel P. Berrangé]

On Tue, Jun 24, 2025 at 11:41:28AM -0600, Warner Losh wrote:

> FreeBSD does maintain all our archival releases forever. They never change.
> But, we don't have permanent links to them today. We start with one URL and
> then migrate to a second one when they transition from supported to unsupported.
> We do this, in part, to make sure people upgrade. So in effect, this breakage
> means that our notion is "working" in the sense that the FreeBSD project's goals
> of making people "keep up to date."
> 
> This does, I realize, clash with the views that QEMU wants to have some stable
> way to test images over time, even if the upstream's notion of supported or not
> changes.
> 
> One easy idea might be to 'prestage' the 'legacy' releases when they
> are supported
> on the 'legacy' server so that tests can be written with the legacy
> path so that these
> tests always work, now and in the future.
> 
> So, this is terrible from a FreeBSD point of view. We'd like it if
> qemu always tested
> all of our releases, as well as snapshots of the tip of the spear.

FWIW, there are two distinct POV to testing which are clashing here.

What you're describing, IMHO, is a desire for QEMU to perform what
I would consider integration testing against all FreeBSD releases,
and forthcoming releases of FreeBSD.

What QEMU's functional test suite is aiming to do is provide
sufficient coverage of QEMU's functionality that we avoid
shipping regressions.

Where it gets fuzzy is that a functional test suite has overlap
with, and can be a decent proxy for, an integration test suite.

The key difference I see is around expectations for the results
of the test harness.

For QEMU's functional test suite, an overriding concern is that
a failure of the test suite *MUST* reflect a fault in QEMU.

We want to minimize (ideally eliminate) any failures caused by
factors outside QEMU. A failure should be something that can
be immediately referred back to the author of the PULL that
triggered it, without needing triage to determine if is it a
failure caused by something outside QEMU. A functional test
failure should generally gate the merging of a PULL request,
given that it should reflect a clear QEMU fault.

With this in mind, we don't ever want to be testing unreleased
snapshots, and even for released images, we always want to
fixate on a specific image hash. Similarly the execution  env
of the test suite is a docker container that has fairly well
constrained  software, though currently we do not fixate
our container images on particular package versions, which
has caused us painful spurious failures at times.


An integration test suite, by contrast, should be open to the
idea that failures can be cause by any moving part in the stack,
whether the host OS, QEMU, or the guest OS. Accepting that,
however, means taking on a significantly higher burden in the
triage of failures - that can easily become a full time job for
one or more people, so diagnose problems and then herd cats to
get it fixed in whichever piece was at fault.

This makes integration testing mostly unsuitable for use as a
gating test for merging PULL requests. It would run asynchronously
and problems could potentially take a long time to resolve, though
ideally by resolves by time of rrelease.

> There's got to be some
> way to have some shared responsibility that we can automate. FreeBSD could test
> the most recent release of qemu against a bunch of images in our CI
> cluster. But we
> don't actually have a CI cluster we could put that into (our focus is
> just a little different)
> today.

The issue of CI resources also impacts QEMU :-( We have to be wary that
our upstream testing is using our own limited CI resources, and any
contributors using GitLab CI also have limited quota.

The human constraint is probably the overriding concern I would
have from the QEMU side though. I'd love it if QEMU did full
integration testing across all guestOS we can get our hands on,
both current & forthcoming FreeBSD/Linux releases, and the
countless historical releases of many OS. Realistically we just
don't have the human resources to manage such a testing effort,
even if we found the hardware to support it.

Putting my Fedora hat on, we rebase QEMU in Fedora rawhide when
rc0 comes out, and Fedora's QA team rely on the rawhide QEMU in
doing release testing. While this isn't always timely enough to
prevent QEMU bugs getting into Fedora which then impact Fedora
releases, it is the best we can do given constraints of both
projects.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|