building xen without mercury repository access

building xen without mercury repository access

[Guillaume Rousse]

Hello, I'm mandriva package maintainer.

I just tried to update mandriva xen package. I noticed the build
procedure changed since 3.1.0: instead of downloading pristine kernel
sources, and patching them locally, the makefile now directly fetches
modified kernel from xen mercury repository. And even if a working
directory exists, it apparently still attempts to sync it before
building... Which is quite curious for a tagged release (but I may have
misunderstood the process).

As our packages are supposed to be self-contained, I need to download
once for all the xenified kernel sources. What command should I run ?
Running "sh buildconfigs/select-repository linux-2.6.18-xen.hg .:.." in
the top-level directory, as suggested by makefile, fails miserabily:
select-repository: Searching `.:..' for linux-2.6.18-xen.hg
select-repository: Ignoring `.'
not found!
select-repository: Unable to determine Xen repository parent.
-- 
Guillaume Rousse
Moyens Informatiques - INRIA Futurs
Tel: 01 69 35 69 62

Re: building xen without mercury repository access

[Keir Fraser]

On 5/2/08 17:54, "Guillaume Rousse" <Guillaume.Rousse@inria.fr> wrote:

> I just tried to update mandriva xen package. I noticed the build
> procedure changed since 3.1.0: instead of downloading pristine kernel
> sources, and patching them locally, the makefile now directly fetches
> modified kernel from xen mercury repository.

Yes, that's right. Preferably the repository linux-2.6.18-xen.hg is placed
adjacent (at same directory level) to your xen repository.

> And even if a working
> directory exists, it apparently still attempts to sync it before
> building... Which is quite curious for a tagged release (but I may have
> misunderstood the process).

Not sure what you mean. We don't automatically 'hg pull' in the Linux
directory to pull upstream patches down, or anything like that.

There is no tag in the linux-2.6.18-xen.hg because guest development is now
divorced from the core Xen release process. Xen releases comprise hypervisor
and tools only. The point being that the big vendors are now maintaining
their own Xen patchsets against whatever Linux version they like. If you do
not want to do that then actually linux-2.6.18-xen.hg is pretty stable
anyway.

> As our packages are supposed to be self-contained, I need to download
> once for all the xenified kernel sources. What command should I run ?
> Running "sh buildconfigs/select-repository linux-2.6.18-xen.hg .:.." in
> the top-level directory, as suggested by makefile, fails miserabily:
> select-repository: Searching `.:..' for linux-2.6.18-xen.hg
> select-repository: Ignoring `.'
> not found!
> select-repository: Unable to determine Xen repository parent.

 # hg clone -r RELEASE-3.2.0 http://xenbits.xensource.com/xen-3.2-testing.hg
 # hg clone -r 406 http://xenbits.xensource.com/linux-2.6.18-xen.hg
 # cd xen-3.2-testing.hg && make -j8 world

 -- Keir

Re: building xen without mercury repository access

[Guillaume Rousse]

Keir Fraser a écrit :
> On 5/2/08 17:54, "Guillaume Rousse" <Guillaume.Rousse@inria.fr> wrote:
> 
>> I just tried to update mandriva xen package. I noticed the build
>> procedure changed since 3.1.0: instead of downloading pristine kernel
>> sources, and patching them locally, the makefile now directly fetches
>> modified kernel from xen mercury repository.
> 
> Yes, that's right. Preferably the repository linux-2.6.18-xen.hg is placed
> adjacent (at same directory level) to your xen repository.
> 
>> And even if a working
>> directory exists, it apparently still attempts to sync it before
>> building... Which is quite curious for a tagged release (but I may have
>> misunderstood the process).
> 
> Not sure what you mean. We don't automatically 'hg pull' in the Linux
> directory to pull upstream patches down, or anything like that.
That:
make[2]: entrant dans le répertoire «
/home/guillomovitch/cooker/xen/BUILD/xen-3.2.0 »
[..]
Pulling changes from http://xenbits.xensource.com/linux-2.6.18-xen.hg
into linux-2.6.18-xen.hg.
pulling from http://xenbits.xensource.com/linux-2.6.18-xen.hg
searching for changes
no changes found

However, I just found out than using the repository was just the default
behaviour. The following env variables allow to use a local pristine
kernel tarball:
export XEN_LINUX_SOURCE=tarball
export KETCHUP=/bin/true

It's just a bit difficult to grab in the makefile to find it :/
-- 
Guillaume Rousse
Moyens Informatiques - INRIA Futurs
Tel: 01 69 35 69 62

Re: building xen without mercury repository access

[Keir Fraser]

On 6/2/08 22:10, "Guillaume Rousse" <Guillaume.Rousse@inria.fr> wrote:

> That:
> make[2]: entrant dans le répertoire «
> /home/guillomovitch/cooker/xen/BUILD/xen-3.2.0 »
> [..]
> Pulling changes from http://xenbits.xensource.com/linux-2.6.18-xen.hg
> into linux-2.6.18-xen.hg.
> pulling from http://xenbits.xensource.com/linux-2.6.18-xen.hg
> searching for changes
> no changes found

Yes, I was wrong. :-)

> However, I just found out than using the repository was just the default
> behaviour. The following env variables allow to use a local pristine
> kernel tarball:
> export XEN_LINUX_SOURCE=tarball
> export KETCHUP=/bin/true
> 
> It's just a bit difficult to grab in the makefile to find it :/

You can override them on the make command line.

 -- Keir

Re: building xen without mercury repository access

[Guillaume Rousse]

Keir Fraser a écrit :
>> However, I just found out than using the repository was just the default
>> behaviour. The following env variables allow to use a local pristine
>> kernel tarball:
>> export XEN_LINUX_SOURCE=tarball
>> export KETCHUP=/bin/true
>>
>> It's just a bit difficult to grab in the makefile to find it :/
> 
> You can override them on the make command line.
The problem is not to set them, but to find their existence and their
effect, as they are documented nowhere. Having all of them centralized
and documented in a single configuration file, which would be included
by all makefile, would be simpler.

And technically, you can't override all of those variables on command
line, given the high number of recursive make invocation losing some of
them (pae switch, for instance). It's safer to rely on environment
there. Which is troublesome when your build process involves two
distinct steps (building and installing), each starting with a fresh
environment. I've been hit many time by issues such as the following
sequence resulting in a non-pae kernel being installed...
pae=y make kernels
make linux-2.6-xen-install

Here again, a configuration file would make easier to ensure consistency.
-- 
Guillaume Rousse
Moyens Informatiques - INRIA Futurs
Tel: 01 69 35 69 62

Re: building xen without mercury repository access

[Yves-Gaël Chény]

Guillaume Rousse a écrit :
> Keir Fraser a écrit :
>   
>>> However, I just found out than using the repository was just the default
>>> behaviour. The following env variables allow to use a local pristine
>>> kernel tarball:
>>> export XEN_LINUX_SOURCE=tarball
>>> export KETCHUP=/bin/true
>>>
>>> It's just a bit difficult to grab in the makefile to find it :/
>>>       
>> You can override them on the make command line.
>>     
> The problem is not to set them, but to find their existence and their
> effect, as they are documented nowhere. Having all of them centralized
> and documented in a single configuration file, which would be included
> by all makefile, would be simpler.
>
> And technically, you can't override all of those variables on command
> line, given the high number of recursive make invocation losing some of
> them (pae switch, for instance). It's safer to rely on environment
> there. Which is troublesome when your build process involves two
> distinct steps (building and installing), each starting with a fresh
> environment. I've been hit many time by issues such as the following
> sequence resulting in a non-pae kernel being installed...
> pae=y make kernels
> make linux-2.6-xen-install
>
> Here again, a configuration file would make easier to ensure consistency.
>   
Agree with you Guillaume.

-- 
Yves-Gaël Chény
40 rue du Perray
44 000 Nantes
Tel : 06 73 93 06 11
      09 51 15 57 81
yves@antredugeek.fr
www.antredugeek.fr

       (o_
(o_    //\
(/)_   V_/_

Re: building xen without mercury repository access

[Guillaume Rousse]

Keir Fraser a écrit :
>> However, I just found out than using the repository was just the default
>> behaviour. The following env variables allow to use a local pristine
>> kernel tarball:
>> export XEN_LINUX_SOURCE=tarball
>> export KETCHUP=/bin/true
>>
>> It's just a bit difficult to grab in the makefile to find it :/
I also found I had to explicitely use IMAGE_TARGET=bzImage, because
default value IMAGE_TARGET=vmlinuz was wrong.

Anyway, using pristine kernel source tarball is not enough. How can I
fetch just the xen dom0 patchset against it ? cloning the whole
linux-2.6.18-xen.hg repository is a bit overkill for me (beside the
additional constraints of preventing network access during build).
-- 
Guillaume Rousse
Moyens Informatiques - INRIA Futurs
Tel: 01 69 35 69 62

Re: building xen without mercury repository access

[Valter Douglas Lisbôa Jr.]

On Monday 11 February 2008 20:32:27 Guillaume Rousse wrote:
>
> Anyway, using pristine kernel source tarball is not enough. How can I
> fetch just the xen dom0 patchset against it ? cloning the whole
> linux-2.6.18-xen.hg repository is a bit overkill for me (beside the
> additional constraints of preventing network access during build).
Oh! I lost the start of this thread, but I interested in this question too. I 
only could make the patch set in dom0 with a entire retrival of the mercury 
repository and diff it with a vanilla source. I agree this is a unnecessary 
step, and I wanna find a better solution.

Re: building xen without mercury repository access

[Keir Fraser]

On 11/2/08 22:32, "Guillaume Rousse" <Guillaume.Rousse@inria.fr> wrote:

> Anyway, using pristine kernel source tarball is not enough. How can I
> fetch just the xen dom0 patchset against it ? cloning the whole
> linux-2.6.18-xen.hg repository is a bit overkill for me (beside the
> additional constraints of preventing network access during build).

We should probably stick a tarball of our Linux tree on www.xen.org.

 -- Keir

Re: building xen without mercury repository access

[Ian Jackson]

Keir Fraser writes ("Re: [Xen-devel] building xen without mercury repository access"):
> On 11/2/08 22:32, "Guillaume Rousse" <Guillaume.Rousse@inria.fr> wrote:
> > Anyway, using pristine kernel source tarball is not enough. How can I
> > fetch just the xen dom0 patchset against it ? cloning the whole
> > linux-2.6.18-xen.hg repository is a bit overkill for me (beside the
> > additional constraints of preventing network access during build).
> 
> We should probably stick a tarball of our Linux tree on www.xen.org.

Yes.  But which version of our hg tree ?  There aren't any release
tags and a nightly tarball from tip would look scary even if it's
actually quite stable.

Ian.

Re: building xen without mercury repository access

[Guillaume Rousse]

Ian Jackson a écrit :
> Keir Fraser writes ("Re: [Xen-devel] building xen without mercury repository access"):
>> On 11/2/08 22:32, "Guillaume Rousse" <Guillaume.Rousse@inria.fr> wrote:
>>> Anyway, using pristine kernel source tarball is not enough. How can I
>>> fetch just the xen dom0 patchset against it ? cloning the whole
>>> linux-2.6.18-xen.hg repository is a bit overkill for me (beside the
>>> additional constraints of preventing network access during build).
>> We should probably stick a tarball of our Linux tree on www.xen.org.
> 
> Yes.  But which version of our hg tree ?  There aren't any release
> tags and a nightly tarball from tip would look scary even if it's
> actually quite stable.
I don't really care, as long as it is properly identified. Keir
suggested revision 406, it's OK for me (I don't track internal changes,
I'm not a kernel hacker).
Of course, if you could have releases, it would be better :)
-- 
Guillaume Rousse
Moyens Informatiques - INRIA Futurs
Tel: 01 69 35 69 62

Re: building xen without mercury repository access

[Keir Fraser]

On 12/2/08 10:43, "Guillaume Rousse" <Guillaume.Rousse@inria.fr> wrote:

>> Yes.  But which version of our hg tree ?  There aren't any release
>> tags and a nightly tarball from tip would look scary even if it's
>> actually quite stable.
> I don't really care, as long as it is properly identified. Keir
> suggested revision 406, it's OK for me (I don't track internal changes,
> I'm not a kernel hacker).
> Of course, if you could have releases, it would be better :)

I think in future we will hold off big changes to the Linux repo coming up
to a release, and we will snap the Linux repo for a tarball when we do the
Xen release. So at least something reasonably stable is available for people
to build with new Xen, at little extra effort for us.

In this case I have tagged the current Linux repo tip as 'xen-3.2.0' and we
will a roll a tarball of that.

 -- Keir

Re: building xen without mercury repository access

[Ian Jackson]

Keir Fraser writes ("Re: [Xen-devel] building xen without mercury repository access"):
> In this case I have tagged the current Linux repo tip as 'xen-3.2.0' and we
> will a roll a tarball of that.

This is now available here:
 http://bits.xensource.com/oss-xen/release/3.2.0/linux-2.6.18-xen-3.2.0.tar.gz
and we should have a link to it from xen.org soon.

Ian.

Re: building xen without mercury repository access

[Maximilian Wilhelm]

Am Tuesday, den 12 February hub Ian Jackson folgendes in die Tasten:

> Keir Fraser writes ("Re: [Xen-devel] building xen without mercury repository access"):
> > In this case I have tagged the current Linux repo tip as 'xen-3.2.0' and we
> > will a roll a tarball of that.

> This is now available here:
>  http://bits.xensource.com/oss-xen/release/3.2.0/linux-2.6.18-xen-3.2.0.tar.gz
> and we should have a link to it from xen.org soon.

So what would I now have to do if I want to build Xen 3.2 with e.g.
Kernel 2.6.24?

Do I really have to generate a patchset with diffing the
xen-kernel-tree with vanilla 2.6.18?

Ciao
Max
-- 
	Follow the white penguin.

Re: building xen without mercury repository access

[Keir Fraser]

On 12/2/08 11:25, "Maximilian Wilhelm" <max@rfc2324.org> wrote:

>> This is now available here:
>>  
>> http://bits.xensource.com/oss-xen/release/3.2.0/linux-2.6.18-xen-3.2.0.tar.gz
>> and we should have a link to it from xen.org soon.
> 
> So what would I now have to do if I want to build Xen 3.2 with e.g.
> Kernel 2.6.24?
> 
> Do I really have to generate a patchset with diffing the
> xen-kernel-tree with vanilla 2.6.18?

2.6.24 has support for running on Xen as a domU. So you do not need to apply
any patchset on the vanilla kernel. The work to let upstream kernel run as
dom0 is still in progress. Until then you have to use our 2.6.18 or,
preferably, a kernel provided by your distro. The kernel does not have to
specifically a '3.2 kernel' -- any reasonably modern Xen-patched kernel
should do.

 -- Keir

Re: building xen without mercury repository access

[Guillaume Rousse]

Ian Jackson a écrit :
> Keir Fraser writes ("Re: [Xen-devel] building xen without mercury repository access"):
>> In this case I have tagged the current Linux repo tip as 'xen-3.2.0' and we
>> will a roll a tarball of that.
> 
> This is now available here:
>  http://bits.xensource.com/oss-xen/release/3.2.0/linux-2.6.18-xen-3.2.0.tar.gz
> and we should have a link to it from xen.org soon.
Cool. However, as it is actually a 2.6.18.8 kernel, could you please
change archive name and top-level directory name to match its content ?
It would make managing its content a bit easier, thanks.

-- 
Guillaume Rousse
Moyens Informatiques - INRIA Futurs
Tel: 01 69 35 69 62