Re: Moving QEMU downloads to GitLab Releases?

[Warner Losh <imp@bsdimp.com>]

[-- Attachment #1: Type: text/plain, Size: 3739 bytes --]

On Mon, Oct 11, 2021 at 4:59 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Mon, Oct 11, 2021 at 09:21:30AM +0200, Gerd Hoffmann wrote:
> >   Hi,
> >
> > > > I guess the main question is who is using the ROM/BIOS sources in the
> > > > tarballs, and would this 2-step process work for those users? If
> there
> > > > are distros relying on them then maybe there are some more logistics
> to
> > > > consider since the make-release downloads are both time-consuming and
> > > > prone to network errors/stalls since it relies on the availability
> of a
> > > > good number of different hosts.
> > >
> > > Great, maybe Gerd can comment on splitting out firmware.
> >
> > I think the binaries are mostly a convenience feature for developers.
> > Distros typically build from source anyway, and typically they build
> > from upstream source instead of qemu submodule.
> >
> > The idea to split them out to a separate repo is exists for quite a
> > while.  I have an old qemu-firmware repo laying around on my disk, which
> > basically moves roms/ + submodules and the binaries built from it over.
> >
> > I think with the switch to gitlab it doesn't make sense any more to
> > commit pre-built firmware binaries to a git repo.  Instead we should
> > build the firmware in gitlab ci, i.e. effectively move the build rules
> > we have right now in roms/Makefile to .gitlab-ci.d/, then publish the
> > firmware binaries as build artifacts or gitlab pages.
> >
> > When done just remove the pre-build binaries from git and add a helper
> > script to fetch firmware binaries from gitlab instead.
> >
> > > QEMU mirrors firmware sources on GitLab too. We're able to provide the
> > > same level of download availability on our mirror firmware repos as for
> > > the main qemu.git repo.
> >
> > I think enabling CI for the firmware mirrors should work given that it
> > is possible to specify a custom CI/CD configuration file, i.e. use
> > something like
> >
> >     /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml
> >
> > or
> >
> >     /qemu-project/qemu-firmware/seabios.yml
> >
> > as config file for the
> >
> >     /qemu-project/seabios/
> >
> > mirror.  Then we can publish binaries using gitlab pages at
> >
> >     https://qemu-project.gitlab.io/seabios/
> >
> > That way we also don't need the roms/ submodules any more, i.e. we
> > can remove both sources and binaries for the firmware from the
> > release tarballs.
>
> Thanks!
>
> For developer convenience it would be nice to avoid manual steps after
> git clone qemu.git. Maybe ./configure should check for firmware blobs
> and automatically download them. There could be a ./configure
> --disable-firmware-download option that distros can use to skip the
> download when building everything from source.
>

One thing to keep in mind about the downstream consumers: Many
of them have two phases to their build process that run asynchronously
to each other. There is a fetch phase that grabs everything, and a build
phase that builds the binaries. The second phase may not have access
to the internet for a variety of reasons (these days being a security
measure, for good or ill). Please make sure that any such plans
allow for this model.

Today, that's all dealt with by grabbing tarballs from github which
is how the submodules are dealt with. So long as the images
had well known names, and could be fetched with the normal
wget/cgit/fetch programs, that would suffice. Requiring use of
some weird bespoke script would cause friction for the downstreams
using qemu.

So while I'm all for making things a little more independent,
let's not do it in a way that makes life difficult for downstreams.
There are more there than just a couple of big distro builders.

Warner

[-- Attachment #2: Type: text/html, Size: 4819 bytes --]