On distro packaging of stub domains (Re: Notes from Xen BoF at Debconf15)

[Ian Campbell <ian.campbell@citrix.com>]

On Tue, 2015-09-08 at 10:24 +0100, Ian Campbell wrote:
> Stubdomains

When I passed these notes around internally for sanity checking we ended up
discussing this issue, since we decided it would be better to move this to
xen-devel I'm quoting the thread below with permission.

It was a bit tricky to flatten the thread into a single mail, but it wasn't
too "branchy" so I think this will be ok. Hopefully I've not misrepresented
what anyone said by how I've arranged it here...

> ===========
> 
> Hard to do in a packaging environment (is really its own partial
> architecture). Rump kernels are no different in this regard.
>
> No clever ideas were put forward.

-----------------------------------------------------------------------

George Dunplap:

> Meaning, hard to reuse binaries from existing qemu package rather than
> using the Xen build system to download and build bespoke qemu images.
> 
> Are the grub packagers doing anything with grub-xen?
> 
> If people want to use the existing qemu package, it does seem like
> making a qemu-xen-stubdom package would be the most sensible thing to do.

-----------------------------------------------------------------------

Me in reply to George:

> From the PoV of a distro a qemu-xen-stubdom package is duplicating a load
> of source code (e.g. qemu and all the libraries which it uses, as well as
> the stubdom libc and kernel etc)), which is disliked by distros, who wish
> to use their existing packages for things.
> 
> But you can't just use the existing source packages for all those libraries
> rebuilt for "arch=stubdom" since "stubdom" is not an architecture which the
> distro understands. (An arch to a distro is a processor architecture + libc
> + calling convention ABI). arch=stubdom could never be a full arch to the
> distro (hence "partial architecture").
> 
> Security teams also don't like things which contain duplicates of source
> code.
> 
> grub-xen is completely self contained, it doesn't rely on anything outside
> of grub.git(or .tar). That's not because they've imported a load of 3rd
> party libraries, it really is just that grub does PV by itself in a
> completely self contained way, but grub is simple enough to get away with
> this.

-----------------------------------------------------------------------

In a short diversion on the final ("grub-xen is completely...") para George
said:

> Do you mean that it's easier to just toss grub-xen into the grub
> package, because it's not very big and doesn't require the maintainer to
> know anything at all about Xen (hence the point about importing 3rd
> party libraries)?

To which I replied:

> Size is not the issue. The fact that grub-xen (or indeed grub-*) has no
> build dependencies at all is what matters.

-----------------------------------------------------------------------

The main thread of the conversation (actually the same mails as above) was in reply to the "From the PoV of a distro" where George said:

> Sorry, this comment came from an RPM perspective, where a single .spec
> file will build a suite of separate rpms (with the same prefix).  The
> CentOS xen.spec will build xen-[version], xen-hypervisor-[version],
> xen-runtime-[version], xen-libs-[version], xen-ocaml-[version], &c.
> 
> CentOS is a bit of a special case, but I presume that in Fedora, if they
> wanted to, they could build a qemu-xenpv package as part of the normal
> qemu RPM build process that would generate a runnable PV image as part
> of the normal rpmbuild of the qemu.spec.
> 
> Is this not possible with debian packages?  Aren't there at least
> separate ${lib} and ${lib}-devel packages?

-----------------------------------------------------------------------

Then I:

> I think we are talking at cross purposes. In Debian a single source package
> does indeed build $lib and $lib-dev etc.
> 
> But, those packages are for arch={i386,amd64,armhf}. They are not for
> arch={stubdom-i386,stubdom-amd64} and there is no such arch in Debian (nor
> CentOS nor Fedora I expect).
> 
> For the qemu source package to be able build a runnable qemu-xenpv binary
> it would need to get all the libFOO and libBAR which qemu needs from
> somewhere, and they would need to be built for the stubdom-{i386,amd64}
> arches, not for i386 or amd64. All the builddeps for qemu-pv need to come
> from somewhere and the existing i386 or amd64 libraries do not satisfy the
> build dep for a pv stubdom.
> 
> (To be clear, I'm talking about mini-os/rump kernel stub doms here, not
> Linux ones of any sort)

-----------------------------------------------------------------------

To which George said:

> I think I see what you're saying -- if building a qemu stubdom *only*
> involved the qemu codebase itself, then it would be really easy for the
> qemu package maintainers to add qemu-xenpv (or something) which would
> just build qemu to run in PV mode.  But in fact, building qemu stubdoms
> involves building all of the other libraries upon which qemu depends for
> stubdoms as well; which again, we don't want duplicated inside the qumu
> package, and so would mean re-building all those other packages for
> stubdoms as well.
> 
> Interestingly, if we cast a vision for a world where unikernels are the
> norm, then it might actually make sense for a distro like Debian to go
> through and re-build core libraries against rump kernels, so that then
> applications like qemu (or apache, or whatever) could link against them
> and build a bootable image, perhaps capable of running either on Xen or
> KVM.
> 
> That's obviously not going to happen overnight -- but it would be
> interesting for Someone to give a try, just to see how difficult it is.
>  If it's not that difficult to build libraries against rumpkernels and
> package them up so they can be linked to from applications, then the
> road to Debian / Fedora-built unikernel applications (including qemu)
> may not actually be that long.

-----------------------------------------------------------------------

Wei and I both replied. Wei said:

> The vision is that rump kernel toolchains will be packaged to distros,
> so that people can build rump kernel with all necessary source tarballs.
> That is not exactly the same as what you just said. What you said is
> probably next step down the road.
> 
> > That's obviously not going to happen overnight -- but it would be
> > interesting for Someone to give a try, just to see how difficult it is.
> 
> Most faffs will be fighting with build systems of individual packages.
> Also some software has less tested configuration that doesn't actually
> work on source code level. For example, I'm very well aware some QEMU
> bugs that prevents it from building with rump kernel.
> 
> All those fixes to build systems and source code can only happen as
> developers realise rump kernel is a thing that they care about. It's a
> bit unrealistic to expect a single team to fix every single piece of
> software.

and I said:

> The first hurdle would be how much of the distro is cross-compilable, since
> rump kernels necessarily imply that.
> 
> That in turn requires the rumpkernel toolchain to look like any other cross
> toolchain i.e. you call -gcc, otherwise every package would need
> tweaking to use the rumpkernel build system.
> 
> Probably that then implies a load of config.{sub,guess} updates for every
> package etc but those are normal for crossing any new architecture and I
> think are reasonably well understood these days.
-----------------------------------------------------------------------

At this point we decided this should go on list, and here we are...