Re: Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 4167 bytes --]

On Mon, Oct 11, 2021 at 08:28:34AM -0600, Warner Losh wrote:
> On Mon, Oct 11, 2021 at 4:59 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> > On Mon, Oct 11, 2021 at 09:21:30AM +0200, Gerd Hoffmann wrote:
> > >   Hi,
> > >
> > > > > I guess the main question is who is using the ROM/BIOS sources in the
> > > > > tarballs, and would this 2-step process work for those users? If
> > there
> > > > > are distros relying on them then maybe there are some more logistics
> > to
> > > > > consider since the make-release downloads are both time-consuming and
> > > > > prone to network errors/stalls since it relies on the availability
> > of a
> > > > > good number of different hosts.
> > > >
> > > > Great, maybe Gerd can comment on splitting out firmware.
> > >
> > > I think the binaries are mostly a convenience feature for developers.
> > > Distros typically build from source anyway, and typically they build
> > > from upstream source instead of qemu submodule.
> > >
> > > The idea to split them out to a separate repo is exists for quite a
> > > while.  I have an old qemu-firmware repo laying around on my disk, which
> > > basically moves roms/ + submodules and the binaries built from it over.
> > >
> > > I think with the switch to gitlab it doesn't make sense any more to
> > > commit pre-built firmware binaries to a git repo.  Instead we should
> > > build the firmware in gitlab ci, i.e. effectively move the build rules
> > > we have right now in roms/Makefile to .gitlab-ci.d/, then publish the
> > > firmware binaries as build artifacts or gitlab pages.
> > >
> > > When done just remove the pre-build binaries from git and add a helper
> > > script to fetch firmware binaries from gitlab instead.
> > >
> > > > QEMU mirrors firmware sources on GitLab too. We're able to provide the
> > > > same level of download availability on our mirror firmware repos as for
> > > > the main qemu.git repo.
> > >
> > > I think enabling CI for the firmware mirrors should work given that it
> > > is possible to specify a custom CI/CD configuration file, i.e. use
> > > something like
> > >
> > >     /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml
> > >
> > > or
> > >
> > >     /qemu-project/qemu-firmware/seabios.yml
> > >
> > > as config file for the
> > >
> > >     /qemu-project/seabios/
> > >
> > > mirror.  Then we can publish binaries using gitlab pages at
> > >
> > >     https://qemu-project.gitlab.io/seabios/
> > >
> > > That way we also don't need the roms/ submodules any more, i.e. we
> > > can remove both sources and binaries for the firmware from the
> > > release tarballs.
> >
> > Thanks!
> >
> > For developer convenience it would be nice to avoid manual steps after
> > git clone qemu.git. Maybe ./configure should check for firmware blobs
> > and automatically download them. There could be a ./configure
> > --disable-firmware-download option that distros can use to skip the
> > download when building everything from source.
> >
> 
> One thing to keep in mind about the downstream consumers: Many
> of them have two phases to their build process that run asynchronously
> to each other. There is a fetch phase that grabs everything, and a build
> phase that builds the binaries. The second phase may not have access
> to the internet for a variety of reasons (these days being a security
> measure, for good or ill). Please make sure that any such plans
> allow for this model.
> 
> Today, that's all dealt with by grabbing tarballs from github which
> is how the submodules are dealt with. So long as the images
> had well known names, and could be fetched with the normal
> wget/cgit/fetch programs, that would suffice. Requiring use of
> some weird bespoke script would cause friction for the downstreams
> using qemu.
> 
> So while I'm all for making things a little more independent,
> let's not do it in a way that makes life difficult for downstreams.
> There are more there than just a couple of big distro builders.

I think this is fine. Firmware blobs aren't needed to build QEMU, only
to run the system emulator.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]