3.4.0 wants to download Linux despite its presence

3.4.0 wants to download Linux despite its presence

[Michael David Crawford]

I tried at first to build the integrated sources, but couldn't get it to 
work.  I saw a list message that said it was broken, so I tried building 
the separate xen 3.4.0 and Linux tarballs.

However, no matter what I try, the build hits the Mercurial server to 
download the Linux kernel sources.  I have tried a number of things, but 
I am completely unable to prevent the download.

May I respectfully suggest that it's A REALLY BAD IDEA to automatically 
download dependencies of you find them missing?

Much better would be to just list what the dependencies are in the 
README, as well as where to get them.  If the build finds them missing, 
then it should just say so and stop.

I've been trying to get a working build of Xen for WEEKS now, with very 
little success.  I am able to boot Xen and a pv_ops Dom0, but that's as 
far as I get.

Please help me.

I really, really want to be able to help out by contributing useful code 
to the project, but the troubles I keep having with the build have had 
me stymied.

Mike
-- 
Michael David Crawford
mdc@prgmr.com

    prgmr.com - We Don't Assume You Are Stupid.

       Xen-Powered Virtual Private Servers: http://prgmr.com/xen

Re: 3.4.0 wants to download Linux despite its presence

[Ian Jackson]

Michael David Crawford writes ("[Xen-devel] 3.4.0 wants to download Linux despite its presence"):
> However, no matter what I try, the build hits the Mercurial server to 
> download the Linux kernel sources.  I have tried a number of things, but 
> I am completely unable to prevent the download.

The build system is rather unfortunate.  You can completely prevent
the Xen build from building a kernel by setting XKERNELS=''
You'll have to build the kernel yourself then of course.

> May I respectfully suggest that it's A REALLY BAD IDEA to automatically 
> download dependencies of you find them missing?

Sadly it's not just the kernel which is done this way.  There are a
handful of tarballs of various kinds too.  If you want to deal with
those offline too you'll find that you can set XEN_EXTFILES_URL.

> Much better would be to just list what the dependencies are in the 
> README, as well as where to get them.  If the build finds them missing, 
> then it should just say so and stop.

I would actually agree.  Having a part of the build system which
brings all the pieces together automatically for you would be good,
but it would be sensible for xen-unstable.hg to be less enthusiastic
about downloading stuff.

I'm not sure the community really agrees with me though.

Ian.

Re: 3.4.0 wants to download Linux despite its presence

[Keir Fraser]

On 20/05/2009 07:57, "Ian Jackson" <Ian.Jackson@eu.citrix.com> wrote:

>> Much better would be to just list what the dependencies are in the
>> README, as well as where to get them.  If the build finds them missing,
>> then it should just say so and stop.
> 
> I would actually agree.  Having a part of the build system which
> brings all the pieces together automatically for you would be good,
> but it would be sensible for xen-unstable.hg to be less enthusiastic
> about downloading stuff.

Perhaps we should more prominently advertise 'make install-xen
install-tools'. This should generally result in no downloads apart from
qemu, and that's a special-case dependency since it ties quite closely to
xen-unstable -- we can't just use any qemu tree. And of course you can
define XEN_EXTFILES_URL so as to make any download attempt fail, and then
you can go grab the packages yourself as you see fit and assemble your
requirements list. Imagine the noise on the mailing list if xen-unstable
didn't try to assemble itself automatically out of the box.

 -- Keir

Re: 3.4.0 wants to download Linux despite its presence

[Jeremy Fitzhardinge]

Keir Fraser wrote:
> On 20/05/2009 07:57, "Ian Jackson" <Ian.Jackson@eu.citrix.com> wrote:
>
>   
>>> Much better would be to just list what the dependencies are in the
>>> README, as well as where to get them.  If the build finds them missing,
>>> then it should just say so and stop.
>>>       
>> I would actually agree.  Having a part of the build system which
>> brings all the pieces together automatically for you would be good,
>> but it would be sensible for xen-unstable.hg to be less enthusiastic
>> about downloading stuff.
>>     
>
> Perhaps we should more prominently advertise 'make install-xen
> install-tools'. This should generally result in no downloads apart from
> qemu, and that's a special-case dependency since it ties quite closely to
> xen-unstable -- we can't just use any qemu tree. And of course you can
> define XEN_EXTFILES_URL so as to make any download attempt fail, and then
> you can go grab the packages yourself as you see fit and assemble your
> requirements list. Imagine the noise on the mailing list if xen-unstable
> didn't try to assemble itself automatically out of the box.
>   

What about adding "make fetch-dependencies", and have the default build 
say something like "X is missing.  Type "make fetch-dependencies"."?

    J

Re: 3.4.0 wants to download Linux despite its presence

[Keir Fraser]

On 20/05/2009 14:12, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:

>> Perhaps we should more prominently advertise 'make install-xen
>> install-tools'. This should generally result in no downloads apart from
>> qemu, and that's a special-case dependency since it ties quite closely to
>> xen-unstable -- we can't just use any qemu tree. And of course you can
>> define XEN_EXTFILES_URL so as to make any download attempt fail, and then
>> you can go grab the packages yourself as you see fit and assemble your
>> requirements list. Imagine the noise on the mailing list if xen-unstable
>> didn't try to assemble itself automatically out of the box.
>>   
> 
> What about adding "make fetch-dependencies", and have the default build
> say something like "X is missing.  Type "make fetch-dependencies"."?

I could live with that, yes.

 -- Keir

Re: 3.4.0 wants to download Linux despite its presence

[Michael David Crawford]

Ian Jackson wrote:
> Michael David Crawford writes ("[Xen-devel] 3.4.0 wants to download Linux despite its presence"):
>> However, no matter what I try, the build hits the Mercurial server to 
>> download the Linux kernel sources.  I have tried a number of things, but 
>> I am completely unable to prevent the download.
> 
> The build system is rather unfortunate.  You can completely prevent
> the Xen build from building a kernel by setting XKERNELS=''
> You'll have to build the kernel yourself then of course.

I'm afraid that didn't work - it still downloaded from Mercurial, and is 
building it now.

I'm just going to let it build, but use a kernel that I previously built.

Jeremy's suggestion to "make fetch-dependencies" is I think a good one.

I've tried "make xen" and "make tools", rather than "make world", but 
the impression I have is that they leave the build incomplete somehow. 
I was never able to get a DomU to load and would get all sorts of arcane 
messages.

I promise to keep trying though - I really *want* Xen to work.

I don't think it's so bad to automatically fetch all the dependencies 
other that Linux.

One reason I wanted to have my own Linux kernel was so I could use a 
RAID driver that I backported to 2.6.18 from the current release.  I 
would imagine many users would have specific requirements for their 
Linux kernel that the default Xen Linux cannot satisfy.

Mike
-- 
Michael David Crawford
mdc@prgmr.com

    prgmr.com - We Don't Assume You Are Stupid.

       Xen-Powered Virtual Private Servers: http://prgmr.com/xen

Re: 3.4.0 wants to download Linux despite its presence

[Patrick Colp]

Doing make xen and make tools should be just fine (I do this regularly, as 
I don't want or need Linux to be constantly rebuilt and reinstalled when 
I'm only changing things to Xen and/or tools).

Did you install them after building (make install-xen and make 
install-tools)? I think you need to be sudo for it to install properly.


Patrick


Michael David Crawford wrote:
> Ian Jackson wrote:
>> Michael David Crawford writes ("[Xen-devel] 3.4.0 wants to download 
>> Linux despite its presence"):
>>> However, no matter what I try, the build hits the Mercurial server to 
>>> download the Linux kernel sources.  I have tried a number of things, 
>>> but I am completely unable to prevent the download.
>>
>> The build system is rather unfortunate.  You can completely prevent
>> the Xen build from building a kernel by setting XKERNELS=''
>> You'll have to build the kernel yourself then of course.
> 
> I'm afraid that didn't work - it still downloaded from Mercurial, and is 
> building it now.
> 
> I'm just going to let it build, but use a kernel that I previously built.
> 
> Jeremy's suggestion to "make fetch-dependencies" is I think a good one.
> 
> I've tried "make xen" and "make tools", rather than "make world", but 
> the impression I have is that they leave the build incomplete somehow. I 
> was never able to get a DomU to load and would get all sorts of arcane 
> messages.
> 
> I promise to keep trying though - I really *want* Xen to work.
> 
> I don't think it's so bad to automatically fetch all the dependencies 
> other that Linux.
> 
> One reason I wanted to have my own Linux kernel was so I could use a 
> RAID driver that I backported to 2.6.18 from the current release.  I 
> would imagine many users would have specific requirements for their 
> Linux kernel that the default Xen Linux cannot satisfy.
> 
> Mike

Re: 3.4.0 wants to download Linux despite its presence

[Michael David Crawford]

Patrick Colp wrote:
> Doing make xen and make tools should be just fine (I do this regularly, 
> as I don't want or need Linux to be constantly rebuilt and reinstalled 
> when I'm only changing things to Xen and/or tools).

I don't know for sure, but I have the theory that "make world" will 
build things that "make xen" and "make tools" don't, which will then 
persist in being installed when later tools builds miss some things.

> Did you install them after building (make install-xen and make 
> install-tools)? I think you need to be sudo for it to install properly.

I think I tried those just a couple times, but killed them when they 
started to build the Linux kernel again.

I've been using install.sh, but that turns out not to check that all the 
required files actually exist - it just installs whatever is in 
dist/install, even if something is missing.

I have a "make world" going right now, and will do a "make install" when 
that's done.  I decided to just let it build Linux, just so I can be 
certain everything else builds.

Mike
-- 
Michael David Crawford
mdc@prgmr.com

    prgmr.com - We Don't Assume You Are Stupid.

       Xen-Powered Virtual Private Servers: http://prgmr.com/xen

Re: 3.4.0 wants to download Linux despite its presence

[Patrick Colp]

Michael David Crawford wrote:
> Patrick Colp wrote:
>> Doing make xen and make tools should be just fine (I do this regularly, 
>> as I don't want or need Linux to be constantly rebuilt and reinstalled 
>> when I'm only changing things to Xen and/or tools).
> 
> I don't know for sure, but I have the theory that "make world" will 
> build things that "make xen" and "make tools" don't, which will then 
> persist in being installed when later tools builds miss some things.

Hmm... I don't know if I've ever used "make world", actually. If I want to 
build everything, I just do "make" (it seems the difference is that world 
will do a clean and kdelete first).

The only difference I can see between using world vs xen and tools is that 
world also builds the stubdom (which can be done separately with "make 
stubdom").

So perhaps the thing to do if you want stubdom support too is to do:

sudo make install-xen
sudo make install-tools
sudo make install-stubdom


>> Did you install them after building (make install-xen and make 
>> install-tools)? I think you need to be sudo for it to install properly.
> 
> I think I tried those just a couple times, but killed them when they 
> started to build the Linux kernel again.

I don't think they should do that... but maybe they need to build the linux 
kernel once. I'm not sure, since I did build and install the linux kernel 
(but only once, ever after I just need to do xen and tools).


> I have a "make world" going right now, and will do a "make install" when 
> that's done.  I decided to just let it build Linux, just so I can be 
> certain everything else builds.

Yes, this certainly seems like the safe thing to do.


Patrick

Re: 3.4.0 wants to download Linux despite its presence

[Michael David Crawford]

>> I don't know for sure, but I have the theory that "make world" will 
>> build things that "make xen" and "make tools" don't, which will then 
>> persist in being installed when later tools builds miss some things.

This seems to actually be the case: when I just did "make xen" and "make 
world", the only thing in the dist directory was the the install directory.

Just now I had a "make world" complete successfully, and now the dist 
directory *also* contains check, COPYING, install.sh and README.

Mike
-- 
Michael David Crawford
mdc@prgmr.com

    prgmr.com - We Don't Assume You Are Stupid.

       Xen-Powered Virtual Private Servers: http://prgmr.com/xen

Re: 3.4.0 wants to download Linux despite its presence

[Marc - A. Dahlhaus [ Administration | Westermann GmbH ]]

Am Freitag, den 22.05.2009, 02:11 -0700 schrieb Michael David Crawford:
> Patrick Colp wrote:
> > Doing make xen and make tools should be just fine (I do this regularly, 
> > as I don't want or need Linux to be constantly rebuilt and reinstalled 
> > when I'm only changing things to Xen and/or tools).
> 
> I don't know for sure, but I have the theory that "make world" will 
> build things that "make xen" and "make tools" don't, which will then 
> persist in being installed when later tools builds miss some things.
> 
> > Did you install them after building (make install-xen and make 
> > install-tools)? I think you need to be sudo for it to install properly.
> 
> I think I tried those just a couple times, but killed them when they 
> started to build the Linux kernel again.
> 
> I've been using install.sh, but that turns out not to check that all the 
> required files actually exist - it just installs whatever is in 
> dist/install, even if something is missing.
> 
> I have a "make world" going right now, and will do a "make install" when 
> that's done.  I decided to just let it build Linux, just so I can be 
> certain everything else builds.
> 
> Mike

Hello Mike,

just don't use "make world" as it includes a call to "make clean" which
will delete the included linux tree before the actual build and it will
dowload the linux tree again during the build phase. Use "make dist"
instead.

Marc