Re: xen.git build system (Re: [HACKATHON] Toolstack session)

[Wei Liu <wei.liu2@citrix.com>]

On Wed, Apr 27, 2016 at 06:03:54PM +0100, George Dunlap wrote:
> On 26/04/16 14:44, Wei Liu wrote:
> > Hi all
> > 
> > I spent some time this morning to work out the details of xen.git build
> > system.
> > 
> > * How build system works at the moment?
> >   1. Stubdom.mk.in and Tools.mk.in define FETCHER variable.
> >   2. m4/fetcher.m4 checks for wget or ftp, which becomes FETCHER.
> >   3. StdGNU.mk defines GIT. It can be overwritten by setting envar
> >      when building.
> >   4. scripts/git-checkout.sh is used to checkout git tree.
> >   5. Invocation of git-checkout.sh in Makefile, tools/Makefile and
> >      tools/firmware/Makefile.
> >   6. Direct invocation of GIT in Makefile, tools/Makefile,
> >      tools/firmware/Makefile in the subtree force update targets.
> >   7. stubdom/Makefile and tools/firmware/etherboot/Makefile invoke FETCHER.
> > 
> > * What will be cloned?
> >   1. mini-os
> >   2. qemu-trad
> >   3. qemu-xen -- can be skipped with --with-system-qemu
> >   4. seabios -- can be skipped with --with-system-seabios
> >   5. ovmf -- can be skipped with --with-system-ovmf
> > 
> > * What needs to be fetched?
> >   1. Stubdom needs:
> >      - newlib
> >      - zlib
> >      - libpci
> >      - lwip
> >      - gmp
> >      - polarssl
> >      - tpmemu
> >      - ocaml
> >      - grub
> >   2. tools/firmware/etherboot needs:
> >      - ipxe
> >   A dumb way of dealing with these tarballs might be just to commit them
> >   in tree. That way we can just eliminate fetcher all together.
> 
> Commit the tarballs in-tree?
> 

The extracted source code, actually.

> I don't think we want to do that; but we certainly could consider
> including them in our release tarball.
> 

This would do, too.

> >> Wei: what downstream consumers expect from the build system. Xen has a top 
> >> level makefile that builds everything, by pulling other projects source 
> >> code. Trying to make it cleaner.
> >>
> >> George: someone recommended to pull grub2 into the Xen build system, but it 
> >> was seen as too much. Try to remove other pieces that Xen build system pull 
> >> in in order to perform a build. Package Raisin in a way that includes all 
> >> the needed dependencies (source).
> >>
> >> Doug: Gentoo/Yocto build system based on the one from Debian. It's not good 
> >> to pull things from the internet when performing a build. Yocto build system 
> >> disables network access when performing a build, custom patches are needed 
> >> in order to fix that. XenServer has to do the same.
> >>
> > 
> > I can provide patches to add a --disable-download configure option. That
> > would basically stub out GIT and FETCHER to be /bin/false (Linux) or
> > /usr/bin/false (*BSD). The default would still be --enable-download.
> > 
> > Is such big switch good enough as first step?
> 
> I'm not sure that we necessarily need a "--disable-download" option.
> The thing that has to be patched out is that configure will fail if it
> can't find wget.
> 

It will be a side effect of --disable-download because FETCHER will be
set to false.  I like to make things explicit in this case.

Can you elaborate on how every thing fit together in your vision? There
must be something else that you have not expressed explicitly to make
sure the whole build environment (xen build system + the build host)
won't fetch external stuff.

I'm open to ideas on how things -- xen build system and build host --
fit together and make adequate adjustment to the xen build system. But
unfortunately I also envisage different people have different views on
how the world functions.

> >> How to fix:
> >>  - Everything controlled by the Xen build system, make it clear what will be 
> >> downloaded, have a target to download the required sources.
> > 
> > What I don't really get is whether this implies a dedicated target to
> > download components (and recursively search for dependencies) *and*
> > place them in the right place in xen.git build system. This option
> > doesn't seem maintainable to me. The anticipation is that  we might add
> > more stuff in xen.git. We can't track what every package needs and
> > provide some options to work out where to put those dependencies.
> > 
> > I think having xen.git build system only track top-level components
> > (such as seabios, ovmf etc) is doable.
> 
> I don't think anyone was suggesting that we download all the
> sub-dependencies of everything recursively from scratch.  The
> expectation is that Xen will build in a "typical" distro environment,
> and that we'll download only things that we develop (xen, minios,
> qemu-trad, qemu-upstream, minios), things we expect a typical distro not
> to have (seabios, ovmf, ipxe), or things that need custom patches /
> recompilation (newlib and all the components recompiled against newlib
> for stubdom).
> 
> It might be worth going back to revisit some of these decisions --
> seabios is available in many distros now, and I don't think we've had
> any hard requirements missing from major seabios releases for a while
> now; it might be time to start thinking about taking seabios out of the
> Xen build, for instance.
> 

Seabios, ovmf are already handled. IPXE, no, because we need patching
it. It would be trivial to provide --with-system-ipxe though.

I propose downstream consumers to go over the list of stuff I listed
above and explicitly express their thoughts on individual components, so
that we can have concrete actions on each items.

> I suppose mini-os, like rumpkernels, is a bit of a special case because
> everything really does require recompilation, almost like a distro.
> 

Yes, they will need some special treatment in order for downstream to
consume them.

> But to be honest, I was a bit confused about how discussed "make
> download" target was supposed to work in practice.  The SOP for

I don't have good idea either. We need to collect more data points on
this.

Wei.

> rpm-based project seems to try as much as possible to only include
> upstream tarballs with patches.  When rpmbuild time comes, *all* the
> building happens without access to the internet; all tarballs must
> already be present in the SOURCES directory.  So having a separate "make
> download" wouldn't really help CentOS at least -- unless it gave you a
> list of files to copy into your SOURCES directory, and copy out again in
> your spec file.
> 
>  -George
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel