[RFC] hvm-stub for ia64

[RFC] hvm-stub for ia64

[tgingold]

Hi,

this is my initial work on hvm-stub domains.
HVM-stub domains are a mix of PV and HVM: it requires VT extension but
look externally as a PV domain.

All the io emulation is done within the domain by a stub firmware.  This
layer translates raw IO accesses to PV commands, ie it can run unmodified
linux without running qemu-dm.

Currently it works only for ia64 and can boot linux (serial console).
I cross-post to xen-devel because I can be of a general interest.

I'd plan to submit patches soon but here is a preview.

Part 1: tools/ - modification to start an hvmstub domain (small)
Part 2: xen/ - IO reflection (small)
Part 3: extra/stubfw - stub firmware (large)

Change to common files are rather small and this is a good news.
The stub firmware is more a proof of concept and need serious enhancement.
It is just minimal to be able to boot linux (ie serial console and one ide
disk).

I am not sure where should go stubfw, so if anyone has suggestions...

Comments are welcome,
Tristan.

Re: [RFC] hvm-stub for ia64

[Keir Fraser]

On 22/11/07 03:59, "tgingold@free.fr" <tgingold@free.fr> wrote:

> Part 1: tools/ - modification to start an hvmstub domain (small)
> Part 2: xen/ - IO reflection (small)
> Part 3: extra/stubfw - stub firmware (large)
> 
> Change to common files are rather small and this is a good news.
> The stub firmware is more a proof of concept and need serious enhancement.
> It is just minimal to be able to boot linux (ie serial console and one ide
> disk).
> 
> I am not sure where should go stubfw, so if anyone has suggestions...

Does the hvmstub have to look different to an hvm guest from the p.o.v. of
the tools? It's rather a shame to introduce a third kind of domain at that
level.

How do you think this appraoch compares with a separate stub domain per HVM
guest? That's the approach being developed on the x86 side, but it would be
nice to not increase divergence between x86 and ia64.

 -- Keir

Re: [Xen-devel] [RFC] hvm-stub for ia64

[tgingold]

Quoting Keir Fraser <Keir.Fraser@cl.cam.ac.uk>:

> Does the hvmstub have to look different to an hvm guest from the p.o.v. of
> the tools? It's rather a shame to introduce a third kind of domain at that
> level.
There are of course differences: qemu shouldn't be started.  And the builder
C function is different.  I can try to merge with the hvm builder function
but I am not sure it is worth.  The amount of new code is still rather small
for tools.

> How do you think this appraoch compares with a separate stub domain per HVM
> guest? That's the approach being developed on the x86 side, but it would be
> nice to not increase divergence between x86 and ia64.
I think my approach is much more simpler.  Just look at the amount of new code
for hypervisor and tools: very small.

There is nothing ia64 specific in hvmstub.  For performance, it requires
cheap virtual to physical (ie switching paging on and off) transitions in the
guest, but I think this is already true for x86.

The real work is in the stub firmware which could be partially shared between
x86 and ia64.  However I'd also like to implement more realistic machine
emulation for ia64 in order to be able to run more OSes.

Tristan.

Re: [RFC] hvm-stub for ia64

[Keir Fraser]

On 22/11/07 09:31, "tgingold@free.fr" <tgingold@free.fr> wrote:

>> Does the hvmstub have to look different to an hvm guest from the p.o.v. of
>> the tools? It's rather a shame to introduce a third kind of domain at that
>> level.
> There are of course differences: qemu shouldn't be started.  And the builder
> C function is different.  I can try to merge with the hvm builder function
> but I am not sure it is worth.  The amount of new code is still rather small
> for tools.

Presumably the old HVM builder for ia64 becomes dead code though? So really
this is the 'new' hvm building path. Do you need to support the old way (on
ia64) if this is checked in?

>> How do you think this appraoch compares with a separate stub domain per HVM
>> guest? That's the approach being developed on the x86 side, but it would be
>> nice to not increase divergence between x86 and ia64.
> I think my approach is much more simpler.  Just look at the amount of new code
> for hypervisor and tools: very small.

I doubt the magnitude of changes in tools/hypervisor is different between
the two approaches. It might even be smaller in the separate domain
approach.

 -- Keir

Re: [RFC] hvm-stub for ia64

[tgingold]

Quoting Keir Fraser <Keir.Fraser@cl.cam.ac.uk>:

> On 22/11/07 09:31, "tgingold@free.fr" <tgingold@free.fr> wrote:
>
> Presumably the old HVM builder for ia64 becomes dead code though? So really
> this is the 'new' hvm building path. Do you need to support the old way (on
> ia64) if this is checked in?
Yes, I don't plan to remove the old way on ia64.  I have not implemented
a VGA in the stub firmware and I don't plan to do it soon.  Only Windows
requires a VGA and I don't have Windows for ia64.

> I doubt the magnitude of changes in tools/hypervisor is different between
> the two approaches. It might even be smaller in the separate domain
> approach.
We are speaking about around 200 lines of code in the hypervisor and slightly
more for tools (300 lines mostly ia64 specific), right ?

Is the stub domain code already available ?  I'd like to have a look.

Tristan.

Re: [RFC] hvm-stub for ia64

[Keir Fraser]

On 22/11/07 10:00, "tgingold@free.fr" <tgingold@free.fr> wrote:

>> I doubt the magnitude of changes in tools/hypervisor is different between
>> the two approaches. It might even be smaller in the separate domain
>> approach.
> We are speaking about around 200 lines of code in the hypervisor and slightly
> more for tools (300 lines mostly ia64 specific), right ?
> 
> Is the stub domain code already available ?  I'd like to have a look.

No, it's not released yet. But same as your approach, all the complexity is
in the stub.

 -- Keir

Re: [RFC] hvm-stub for ia64

[Samuel Thibault]

tgingold@free.fr, le Thu 22 Nov 2007 11:00:24 +0100, a écrit :
> Quoting Keir Fraser <Keir.Fraser@cl.cam.ac.uk>:
> 
> > On 22/11/07 09:31, "tgingold@free.fr" <tgingold@free.fr> wrote:
> >
> > Presumably the old HVM builder for ia64 becomes dead code though? So really
> > this is the 'new' hvm building path. Do you need to support the old way (on
> > ia64) if this is checked in?
> Yes, I don't plan to remove the old way on ia64.  I have not implemented
> a VGA in the stub firmware and I don't plan to do it soon.

That's the main difference between your approach (reimplement a
firmware) and the approach being developped here at XenSource (embed
qemu in a tiny stubdomain).  Your VMX work in Xen is most probably
something useful, but reimplementing the hardware emulation support of
qemu looks to me like duplicate work.  I haven't ported my stubdomain to
ia64 yet, but that shouldn't be very hard now that it works for x86.

Samuel

Re: [RFC] hvm-stub for ia64

[tgingold]

Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:

> That's the main difference between your approach (reimplement a
> firmware) and the approach being developped here at XenSource (embed
> qemu in a tiny stubdomain).  Your VMX work in Xen is most probably
> something useful, but reimplementing the hardware emulation support of
> qemu looks to me like duplicate work.

Of course it doesn't reimplement the emulation.  The current biggest emulation
is the IDE and I have copied ide.c from qemu and made almost no modification.

The advantage is my approach is no need for scheduling between two domains
for hardware emulation.  With the current hvm model, I reach the EFI prompt
in about 10sec.  With my hvmstub it took less than 2 seconds. Very pleasant
to use!

Tristan.

Re: [RFC] hvm-stub for ia64

[Samuel Thibault]

tgingold@free.fr, le Thu 22 Nov 2007 12:12:31 +0100, a écrit :
> Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> 
> > That's the main difference between your approach (reimplement a
> > firmware) and the approach being developped here at XenSource (embed
> > qemu in a tiny stubdomain).  Your VMX work in Xen is most probably
> > something useful, but reimplementing the hardware emulation support of
> > qemu looks to me like duplicate work.
> 
> Of course it doesn't reimplement the emulation.  The current biggest emulation
> is the IDE and I have copied ide.c from qemu and made almost no modification.

That's still forking.  And you'll also need to do it for vga,
network, ...

> The advantage is my approach is no need for scheduling between two domains
> for hardware emulation.

Ah, yes, on ia64 you have room to load your firmware in the domain
itself, which makes an extra benefit (with the stubdomain we avoid the
need for linux scheduling, but not the need for xen scheduling).

> With the current hvm model, I reach the EFI prompt
> in about 10sec.  With my hvmstub it took less than 2 seconds.

And I wonder how much it would take with a stub domain, probably
somewhere in between.

Samuel

Re: [Xen-devel] [RFC] hvm-stub for ia64

[tgingold]

Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:

> tgingold@free.fr, le Thu 22 Nov 2007 12:12:31 +0100, a écrit :
> > Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> > Of course it doesn't reimplement the emulation.  The current biggest
> emulation
> > is the IDE and I have copied ide.c from qemu and made almost no
> modification.
>
> That's still forking.  And you'll also need to do it for vga,
> network, ...

To be fair, I'd need to know better how stubdomain is implemented.


> > The advantage is my approach is no need for scheduling between two domains
> > for hardware emulation.
>
> Ah, yes, on ia64 you have room to load your firmware in the domain
> itself, which makes an extra benefit (with the stubdomain we avoid the
> need for linux scheduling, but not the need for xen scheduling).
Should be the same on x86.  Firmware is loaded between 4GB-16MB and 4GB.
Enough room for stub firmware and EFI.

Tristan.

Re: [RFC] hvm-stub for ia64

[Samuel Thibault]

Hi,

tgingold@free.fr, le Thu 22 Nov 2007 14:04:47 +0100, a écrit :
> Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> > tgingold@free.fr, le Thu 22 Nov 2007 12:12:31 +0100, a écrit :
> > > Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> > > Of course it doesn't reimplement the emulation.  The current biggest
> > emulation
> > > is the IDE and I have copied ide.c from qemu and made almost no
> > modification.
> >
> > That's still forking.  And you'll also need to do it for vga,
> > network, ...
> 
> To be fair, I'd need to know better how stubdomain is implemented.

It's actually "just" a matter of compiling qemu against newlib, mini-os
and lwIP and make that run as a kernel in its own domain. That way it's
just the same code as the usual dm approach and mini-os, with some
#ifdefs.

Samuel

Re: [Xen-devel] [RFC] hvm-stub for ia64

[Tristan Gingold]

On Thu, Nov 22, 2007 at 10:20:44PM +0000, Samuel Thibault wrote:
> Hi,
> 
> It's actually "just" a matter of compiling qemu against newlib, mini-os
> and lwIP and make that run as a kernel in its own domain. That way it's
> just the same code as the usual dm approach and mini-os, with some
> #ifdefs.
I suppose you had to add vbd and full vif to mini-os ?
VGA will only be available with vnc ?

I suppose you also have to do a double save-and-restore ?

Tristan.

Re: [RFC] hvm-stub for ia64

[Samuel Thibault]

Hello,

Tristan Gingold, le Fri 23 Nov 2007 04:30:53 +0100, a écrit :
> On Thu, Nov 22, 2007 at 10:20:44PM +0000, Samuel Thibault wrote:
> > It's actually "just" a matter of compiling qemu against newlib, mini-os
> > and lwIP and make that run as a kernel in its own domain. That way it's
> > just the same code as the usual dm approach and mini-os, with some
> > #ifdefs.
> I suppose you had to add vbd and full vif to mini-os ?

Yes, and they will profit whichever project using mini-os.

> VGA will only be available with vnc ?

I only tested vnc for now, but I can't see a reason why vfb shouldn't
work.

> I suppose you also have to do a double save-and-restore ?

Of the domain states when saving/restoring them you mean?

Samuel

Re: [RFC] hvm-stub for ia64

[Samuel Thibault]

Hi,

If you want to give it a try, a repository is available on 
http://xenbits.xensource.com/ext/xen-minios-stubdom.hg

Samuel