Re: xen.git build system (Re: [HACKATHON] Toolstack session)

[George Dunlap <george.dunlap@citrix.com>]

On 26/04/16 14:44, Wei Liu wrote:
> Hi all
> 
> I spent some time this morning to work out the details of xen.git build
> system.
> 
> * How build system works at the moment?
>   1. Stubdom.mk.in and Tools.mk.in define FETCHER variable.
>   2. m4/fetcher.m4 checks for wget or ftp, which becomes FETCHER.
>   3. StdGNU.mk defines GIT. It can be overwritten by setting envar
>      when building.
>   4. scripts/git-checkout.sh is used to checkout git tree.
>   5. Invocation of git-checkout.sh in Makefile, tools/Makefile and
>      tools/firmware/Makefile.
>   6. Direct invocation of GIT in Makefile, tools/Makefile,
>      tools/firmware/Makefile in the subtree force update targets.
>   7. stubdom/Makefile and tools/firmware/etherboot/Makefile invoke FETCHER.
> 
> * What will be cloned?
>   1. mini-os
>   2. qemu-trad
>   3. qemu-xen -- can be skipped with --with-system-qemu
>   4. seabios -- can be skipped with --with-system-seabios
>   5. ovmf -- can be skipped with --with-system-ovmf
> 
> * What needs to be fetched?
>   1. Stubdom needs:
>      - newlib
>      - zlib
>      - libpci
>      - lwip
>      - gmp
>      - polarssl
>      - tpmemu
>      - ocaml
>      - grub
>   2. tools/firmware/etherboot needs:
>      - ipxe
>   A dumb way of dealing with these tarballs might be just to commit them
>   in tree. That way we can just eliminate fetcher all together.

Commit the tarballs in-tree?

I don't think we want to do that; but we certainly could consider
including them in our release tarball.

>> Wei: what downstream consumers expect from the build system. Xen has a top 
>> level makefile that builds everything, by pulling other projects source 
>> code. Trying to make it cleaner.
>>
>> George: someone recommended to pull grub2 into the Xen build system, but it 
>> was seen as too much. Try to remove other pieces that Xen build system pull 
>> in in order to perform a build. Package Raisin in a way that includes all 
>> the needed dependencies (source).
>>
>> Doug: Gentoo/Yocto build system based on the one from Debian. It's not good 
>> to pull things from the internet when performing a build. Yocto build system 
>> disables network access when performing a build, custom patches are needed 
>> in order to fix that. XenServer has to do the same.
>>
> 
> I can provide patches to add a --disable-download configure option. That
> would basically stub out GIT and FETCHER to be /bin/false (Linux) or
> /usr/bin/false (*BSD). The default would still be --enable-download.
> 
> Is such big switch good enough as first step?

I'm not sure that we necessarily need a "--disable-download" option.
The thing that has to be patched out is that configure will fail if it
can't find wget.

>> How to fix:
>>  - Everything controlled by the Xen build system, make it clear what will be 
>> downloaded, have a target to download the required sources.
> 
> What I don't really get is whether this implies a dedicated target to
> download components (and recursively search for dependencies) *and*
> place them in the right place in xen.git build system. This option
> doesn't seem maintainable to me. The anticipation is that  we might add
> more stuff in xen.git. We can't track what every package needs and
> provide some options to work out where to put those dependencies.
> 
> I think having xen.git build system only track top-level components
> (such as seabios, ovmf etc) is doable.

I don't think anyone was suggesting that we download all the
sub-dependencies of everything recursively from scratch.  The
expectation is that Xen will build in a "typical" distro environment,
and that we'll download only things that we develop (xen, minios,
qemu-trad, qemu-upstream, minios), things we expect a typical distro not
to have (seabios, ovmf, ipxe), or things that need custom patches /
recompilation (newlib and all the components recompiled against newlib
for stubdom).

It might be worth going back to revisit some of these decisions --
seabios is available in many distros now, and I don't think we've had
any hard requirements missing from major seabios releases for a while
now; it might be time to start thinking about taking seabios out of the
Xen build, for instance.

I suppose mini-os, like rumpkernels, is a bit of a special case because
everything really does require recompilation, almost like a distro.

But to be honest, I was a bit confused about how discussed "make
download" target was supposed to work in practice.  The SOP for
rpm-based project seems to try as much as possible to only include
upstream tarballs with patches.  When rpmbuild time comes, *all* the
building happens without access to the internet; all tarballs must
already be present in the SOURCES directory.  So having a separate "make
download" wouldn't really help CentOS at least -- unless it gave you a
list of files to copy into your SOURCES directory, and copy out again in
your spec file.

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel