Deficiencies in guest bootloader design

Deficiencies in guest bootloader design

[Michael Paesold]

I have tried to use a dynamically activated block device with pygrub 
(actually drbd, but could be enbd, nbd, or simply any vbd other than phy: or 
file:). That does not work.

The problem is that the bootloader is not called from Xend, after virtual 
block devices are setup, but instead it is called directly from xm, i.e. in 
xm/create.py.

If I have looked correctly, (1) a bootloader cannot be used for devices such 
as nbd: or enbd: devices, and (2) a bootloader cannot be used when accessing 
Xend via XML-RPC (or libvirt or similar), because the whole notion of a 
"bootloader" at domain creation time is only available in "xm", but not in 
Xend.

It should be fixable, since at least on guest *restart*, 
xend/XendDomainInfo.py already uses the bootloader to re-extract the kernel 
image from the vbd.

Is there anyone working on bootloader improvements? If not, is a change to 
move the bootloader stuff from xm to Xend acceptable? In that case I could 
try to come up with a patch.

There are some issues though. In XendDomainInfo.initDomain, image.create is 
done before self.createDevices. Can this be changed easily?

Thanks for any help or guidance.

Best Regards,
Michael Paesold 

Re: Deficiencies in guest bootloader design

[Matt Ayres]

Michael Paesold wrote:
> I have tried to use a dynamically activated block device with pygrub 
> (actually drbd, but could be enbd, nbd, or simply any vbd other than 
> phy: or file:). That does not work.
> 

Have you looked at domUloader?  I inquired about a recent version and 
was told there was no project page, yet.  The patches I have will not 
patch against 3.0-testing, way too many rejects.

Thank you,
Matt Ayres

Re: Deficiencies in guest bootloader design

[Ewan Mellor]

On Thu, Apr 13, 2006 at 04:29:23PM +0200, Michael Paesold wrote:

> I have tried to use a dynamically activated block device with pygrub 
> (actually drbd, but could be enbd, nbd, or simply any vbd other than phy: 
> or file:). That does not work.
> 
> The problem is that the bootloader is not called from Xend, after virtual 
> block devices are setup, but instead it is called directly from xm, i.e. in 
> xm/create.py.
> 
> If I have looked correctly, (1) a bootloader cannot be used for devices 
> such as nbd: or enbd: devices, and (2) a bootloader cannot be used when 
> accessing Xend via XML-RPC (or libvirt or similar), because the whole 
> notion of a "bootloader" at domain creation time is only available in "xm", 
> but not in Xend.
> 
> It should be fixable, since at least on guest *restart*, 
> xend/XendDomainInfo.py already uses the bootloader to re-extract the kernel 
> image from the vbd.
> 
> Is there anyone working on bootloader improvements? If not, is a change to 
> move the bootloader stuff from xm to Xend acceptable? In that case I could 
> try to come up with a patch.

Certainly such a patch would be accepted.  There was a long thread discussing
bootloaders recently -- you should dig it out of the archive and see what the
conclusions were -- there seem to be a few competing ideas for bootloader
support in Xen (pygrub and domUloader being the two prominent ones).  If you
were going to put some effort into this, it would be appreciated by a number
of the Xen users, certainly.

> There are some issues though. In XendDomainInfo.initDomain, image.create is 
> done before self.createDevices. Can this be changed easily?

Off the top of my head, I can't think why this would be too much of a problem.
Why do you ask though?  If the bootloader is running inside the guest, then
that is happening after both of those things, because at the point at which we
are doing initDomain, the guest hasn't even started running yet.  Why does
ordering here concern you?

Don't forget that VMX domains handle their devices differently to para-virt
domains, and the VMX folks won't appreciate it if you break their code ;-)

Cheers,

Ewan.

Re: Deficiencies in guest bootloader design

[Anthony Liguori]

Ewan Mellor wrote:
> On Thu, Apr 13, 2006 at 04:29:23PM +0200, Michael Paesold wrote:
>
>   
>> I have tried to use a dynamically activated block device with pygrub 
>> (actually drbd, but could be enbd, nbd, or simply any vbd other than phy: 
>> or file:). That does not work.
>>
>> The problem is that the bootloader is not called from Xend, after virtual 
>> block devices are setup, but instead it is called directly from xm, i.e. in 
>> xm/create.py.
>>
>> If I have looked correctly, (1) a bootloader cannot be used for devices 
>> such as nbd: or enbd: devices, and (2) a bootloader cannot be used when 
>> accessing Xend via XML-RPC (or libvirt or similar), because the whole 
>> notion of a "bootloader" at domain creation time is only available in "xm", 
>> but not in Xend.
>>
>> It should be fixable, since at least on guest *restart*, 
>> xend/XendDomainInfo.py already uses the bootloader to re-extract the kernel 
>> image from the vbd.
>>
>> Is there anyone working on bootloader improvements? If not, is a change to 
>> move the bootloader stuff from xm to Xend acceptable? In that case I could 
>> try to come up with a patch.
>>     
>
> Certainly such a patch would be accepted.
>   

Just as a warning, it's not going to be simple to run something like 
pygrub from Xend.  The reason it runs from xm now is that it needs to 
interact with the user.  If it were launched from Xend, it need to find 
a way to attach to a PTY that xenconsoled would use so that it can 
display the text over the normal console before actually launching the 
domain.

It's certainly not impossible but it touches quite a few things.

Regards,

Anthony Liguori

>> There are some issues though. In XendDomainInfo.initDomain, image.create is 
>> done before self.createDevices. Can this be changed easily?
>>     
>
> Off the top of my head, I can't think why this would be too much of a problem.
> Why do you ask though?  If the bootloader is running inside the guest, then
> that is happening after both of those things, because at the point at which we
> are doing initDomain, the guest hasn't even started running yet.  Why does
> ordering here concern you?
>
> Don't forget that VMX domains handle their devices differently to para-virt
> domains, and the VMX folks won't appreciate it if you break their code ;-)
>
> Cheers,
>
> Ewan.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

Re: Deficiencies in guest bootloader design

[Ewan Mellor]

On Thu, Apr 13, 2006 at 10:24:44AM -0500, Anthony Liguori wrote:

> Ewan Mellor wrote:
> >On Thu, Apr 13, 2006 at 04:29:23PM +0200, Michael Paesold wrote:
> >
> >  
> >>I have tried to use a dynamically activated block device with pygrub 
> >>(actually drbd, but could be enbd, nbd, or simply any vbd other than phy: 
> >>or file:). That does not work.
> >>
> >>The problem is that the bootloader is not called from Xend, after virtual 
> >>block devices are setup, but instead it is called directly from xm, i.e. 
> >>in xm/create.py.
> >>
> >>If I have looked correctly, (1) a bootloader cannot be used for devices 
> >>such as nbd: or enbd: devices, and (2) a bootloader cannot be used when 
> >>accessing Xend via XML-RPC (or libvirt or similar), because the whole 
> >>notion of a "bootloader" at domain creation time is only available in 
> >>"xm", but not in Xend.
> >>
> >>It should be fixable, since at least on guest *restart*, 
> >>xend/XendDomainInfo.py already uses the bootloader to re-extract the 
> >>kernel image from the vbd.
> >>
> >>Is there anyone working on bootloader improvements? If not, is a change 
> >>to move the bootloader stuff from xm to Xend acceptable? In that case I 
> >>could try to come up with a patch.
> >>    
> >
> >Certainly such a patch would be accepted.
> >  
> 
> Just as a warning, it's not going to be simple to run something like 
> pygrub from Xend.  The reason it runs from xm now is that it needs to 
> interact with the user.  If it were launched from Xend, it need to find 
> a way to attach to a PTY that xenconsoled would use so that it can 
> display the text over the normal console before actually launching the 
> domain.
> 
> It's certainly not impossible but it touches quite a few things.

So how _is_ the virtual frame buffer coming along?  ;-)

Ewan.

Re: Deficiencies in guest bootloader design

[Anthony Liguori]

Ewan Mellor wrote:
> On Thu, Apr 13, 2006 at 10:24:44AM -0500, Anthony Liguori wrote: 
>   
>> Just as a warning, it's not going to be simple to run something like 
>> pygrub from Xend.  The reason it runs from xm now is that it needs to 
>> interact with the user.  If it were launched from Xend, it need to find 
>> a way to attach to a PTY that xenconsoled would use so that it can 
>> display the text over the normal console before actually launching the 
>> domain.
>>
>> It's certainly not impossible but it touches quite a few things.
>>     
>
> So how _is_ the virtual frame buffer coming along?  ;-)
>   

Yeah, I've talked to Jeremy about the fact that the virtual frame buffer 
just makes the situation 100 times worse :-)

Regards,

Anthony Liguori

> Ewan.
>   

Re: Deficiencies in guest bootloader design

[Michael Paesold]

Ewan Mellor wrote:

>> Is there anyone working on bootloader improvements? If not, is a change 
>> to
>> move the bootloader stuff from xm to Xend acceptable? In that case I 
>> could
>> try to come up with a patch.
>
> Certainly such a patch would be accepted.

I already read Anthony's reply -- so I see some obstacles now :-)

> There was a long thread discussing
> bootloaders recently -- you should dig it out of the archive and see what 
> the
> conclusions were -- there seem to be a few competing ideas for bootloader
> support in Xen (pygrub and domUloader being the two prominent ones).  If 
> you
> were going to put some effort into this, it would be appreciated by a 
> number
> of the Xen users, certainly.

I am going to do that now.

>> There are some issues though. In XendDomainInfo.initDomain, image.create 
>> is
>> done before self.createDevices. Can this be changed easily?
>
> Off the top of my head, I can't think why this would be too much of a 
> problem.
> Why do you ask though?  If the bootloader is running inside the guest, 
> then
> that is happening after both of those things, because at the point at 
> which we
> are doing initDomain, the guest hasn't even started running yet.  Why does
> ordering here concern you?

Well, with pygrub at least, the bootloader is not really executed in the 
guest, but in dom0. It extracts kernel and initrd into temporary files which 
are than booted as if they were directly specified in the guest config. So 
to extract the kernel, we would have to first setup the vbds.

I am going to check the archives now.

Best Regards,
Michael Paesold 

Re: Deficiencies in guest bootloader design

[Michael Paesold]

Matt Ayres wrote:
> Michael Paesold wrote:
>> I have tried to use a dynamically activated block device with pygrub 
>> (actually drbd, but could be enbd, nbd, or simply any vbd other than 
>> phy: or file:). That does not work.
>>
> 
> Have you looked at domUloader?  I inquired about a recent version and 
> was told there was no project page, yet.  The patches I have will not 
> patch against 3.0-testing, way too many rejects.

No, not yet. Where did you get the patches from, this mailing list? -- in 
that case I should find them in the archives.

I am going to take a look. Thanks.

Best Regards,
Michael Paesold

Re: Deficiencies in guest bootloader design

[Michael Paesold]

Anthony Liguori wrote:
> Ewan Mellor wrote:
>> On Thu, Apr 13, 2006 at 04:29:23PM +0200, Michael Paesold wrote:
 >>
>>> I have tried to use a dynamically activated block device with pygrub 
>>> (actually drbd, but could be enbd, nbd, or simply any vbd other than 
>>> phy: or file:). That does not work.
...
>>> Is there anyone working on bootloader improvements? If not, is a 
>>> change to move the bootloader stuff from xm to Xend acceptable? In 
>>> that case I could try to come up with a patch.
>>>     
>>
>> Certainly such a patch would be accepted.
> 
> Just as a warning, it's not going to be simple to run something like 
> pygrub from Xend.  The reason it runs from xm now is that it needs to 
> interact with the user.  If it were launched from Xend, it need to find 
> a way to attach to a PTY that xenconsoled would use so that it can 
> display the text over the normal console before actually launching the 
> domain.
> 
> It's certainly not impossible but it touches quite a few things.

Thanks for that important hint. Hmm, output of pygrub is only really 
important in the case when xm create is called with -c. In that case the 
console must be setup anyway, but I guess that this only happens after at 
least part of the guest initialization.

<digression>
After reading through past discussion, I have a feeling that porting real 
boot loaders to xen (e.g. grub) would be a better long term solution.

Just for booting linux, pygrub is really ok, and I prefer it over 
domUloader because it relies on existing infrastructure in domU (menu.lst) 
to get the correct kernel to boot ("yum update" inside domU will do the 
right thing once grub configuration is setup correctly).

On the other hand pygrub only handles ext2/3 and reiserfs, so booting a bsd 
guest won't work easily.
</digression>

Best Regards,
Michael Paesold

Re: Deficiencies in guest bootloader design

[Michael Paesold]

Ewan Mellor wrote:
>> Is there anyone working on bootloader improvements? If not, is a change to 
>> move the bootloader stuff from xm to Xend acceptable? In that case I could 
>> try to come up with a patch.
> 
> Certainly such a patch would be accepted.  There was a long thread discussing
> bootloaders recently -- you should dig it out of the archive and see what the
> conclusions were -- there seem to be a few competing ideas for bootloader
> support in Xen (pygrub and domUloader being the two prominent ones).  If you
> were going to put some effort into this, it would be appreciated by a number
> of the Xen users, certainly.

I have read through the threads about pygrub/domUloader in the archives, 
although I have not found discussion about the initial inclusion of pygrub. 
I mostly see discussions about the actual features of the boot loaders 
themselves, not really about integration relevant to my question.

I see the most important question is that of interaction with the user, 
since that seems to be the main reason for bootloader being invoked from xm 
instead of Xend. I will study the code to see what can be done.

I appreciate any input or help!

Best Regards,
Michael