RE: Re: [Xen-changelog] Indirect hypercalls through ahypercall transfer page.

RE: Re: [Xen-changelog] Indirect hypercalls through ahypercall transfer page.

[Ian Pratt]

 > On Wed, 2006-01-25 at 13:58 +0000, Xen patchbot -unstable wrote:
> > # HG changeset patch
> > # User kaf24@firebug.cl.cam.ac.uk
> > # Node ID 71914b64b5770285cf095c1906e346b814f4878b
> > # Parent  246b6444c6432193ea98c9f01de58e6740bde09c
> > Indirect hypercalls through a hypercall transfer page.
> 
> Sorry, I may have missed some conversation about this...
> 
> What is this patch about? I mean I can see how it works, but 
> what is the motivation for it?

The primary reason for this patch is that it will enable us to run a
xenified kernel on bare metal: a 'micro xen' would be loaded by the boot
loader alongside a xenified kernel image, enabling the kernel to
effectively run on the baremetal at full native performance (though
obviously you could only run one kernel on a machine at a time).

The nice thing about this approach is that a vendor could certify their
app running on an OS over Xen, and have good confidence that it would
then run stably on the OS running over 'microxen', as microxen just
doesn't implement the type checking enforcement and multiplexing that
real xen does. Doing things this way around is certinly the best way of
getting assurance without having to certify apps twice. It requires a
bit of a leap-of-faith that virtualization is going to be so common
place on servers that the paravirtualized kernel should be the one used
for certification, but it's a sound technical argument. Any scheme that
doesn't certify with the kernel actually running over the hypervisor
(whatever the hypervisor) is fairly delusional. 

Implementing microxen is pretty straight forward. One could implement a
prototype just by deleteing huge swathes of code from xen, but it
probably makes sense to go for something really minimalist. The OS
kernel would run in ring 0 (alongside microxen) and would use the xen
transfer page hypercall mechanism to simply jump into microxen to
implement the xen api using the normal privileged instructions (mov cr3
etc). Xen guests only use a handful of hypercalls, and most of the code
for high frequency ones can just be in-lined in the transfer page.

Another nice feature of this approach is that if the CPU vendors add
certain new features [e.g. a tagged TLB (Address Space Numbers),
multicast IPIs etc] the same xenified guest kernel would be able to
imediately take advantage of them. Further, as full virtualization
hardware assist (VT/Pacifica) improves in performance over the next few
years, it may make sense to start running xen-ified kernels in hvm
partitions using the microxen shim. We'd be able to tune what gets done
in hardware vs software by adjusting the shim, which is dropped in by
the domain builder.


Ian

Re: Re: [Xen-changelog] Indirect hypercalls through ahypercall transfer page.

[Mike D. Day]

Ian Pratt wrote:
>> What is this patch about? I mean I can see how it works, but 
>> what is the motivation for it?
> 
> The primary reason for this patch is that it will enable us to run a
> xenified kernel on bare metal: a 'micro xen' would be loaded by the boot
> loader alongside a xenified kernel image, enabling the kernel to
> effectively run on the baremetal at full native performance (though
> obviously you could only run one kernel on a machine at a time).

This is an elegant feature that would have been better introduced with 
the above explanation rather than a patchbot message.

If I missed that discussion, apologies.

Mike Day

RE: Re: [Xen-changelog] Indirect hypercalls through ahypercall transfer page.

[Ian Pratt]

> > The primary reason for this patch is that it will enable us 
> to run a 
> > xenified kernel on bare metal: a 'micro xen' would be loaded by the 
> > boot loader alongside a xenified kernel image, enabling the 
> kernel to 
> > effectively run on the baremetal at full native performance (though 
> > obviously you could only run one kernel on a machine at a time).
> 
> This is an elegant feature that would have been better 
> introduced with the above explanation rather than a patchbot message.
> 
> If I missed that discussion, apologies.

Mike, there was some discussion about this and the other new 'feature
flags' at the OS Ports session at the summit (and in particular in the
'linux merge' breakout group).

The hypercall transfer page is just one of several new feature flags
that should enable us to have a single xen-ified kernel that is capable
of running on bare metal, on pae and non-pae hypervisors, 64b
hypervisor, under HVM etc. Obviously many of these combinations will
require shadow pagetables to be enabled (and hence some loss of
performance), but it will be good to be able to support them.

The main feature flag that is currently missing is one to enable the
kernel's hypervisor vm hole size to be setable at guest boot time
(necessary for running non-pae kernels on a pae hypervisor). Hopefully
Novell are working on this... :)

Ian
 

Re: Re: [Xen-changelog] Indirect hypercalls through ahypercall transfer page.

[Mike D. Day]

Ian Pratt wrote:

> 
> Mike, there was some discussion about this and the other new 'feature
> flags' at the OS Ports session at the summit (and in particular in the
> 'linux merge' breakout group).

Thanks, I guess I missed that session.

> The hypercall transfer page is just one of several new feature flags
> that should enable us to have a single xen-ified kernel that is capable
> of running on bare metal, on pae and non-pae hypervisors, 64b

A 32-bit kernel running on a 64-bit xen, for example? So in this case 
xen fills the page with 32-bit hcalls?

Mike

RE: Re: [Xen-changelog] Indirect hypercalls through ahypercall transfer page.

[Ian Pratt]

 
> > The hypercall transfer page is just one of several new 
> feature flags 
> > that should enable us to have a single xen-ified kernel that is 
> > capable of running on bare metal, on pae and non-pae 
> hypervisors, 64b
> 
> A 32-bit kernel running on a 64-bit xen, for example? So in 
> this case xen fills the page with 32-bit hcalls?

Our current priority developing xen feature flags has been to enable
xenified kernels to run on bare metal and to allow non-PAE kernels on a
PAE hypervisor, but 32 on 64 is certainly something it would be good to
support too.

The key feature flag we're still missing is the ability for the guest
kernel to query the size of the 'hypervisor hole' at boot time, and then
adjust its VM layout appropriately. Conceptually, this is quite simple
for Linux -- just turn HYPERVISOR_VIRTSTART into a variable.
Unfortunately, things are a little more complicated: right at the top of
the kernel's virtual address space is the fixaddr region. The vsyscall
page lives within the fixaddr region, and its virtual address needs to
be known at *compile* time as it gets built into various elf objects.

A number of options are aviable to fix this 1) provide either kernel or
domain builder support for relocating the syscall page (sounds hard); 2)
move the vsyscall page out of fixmap and put it somewhere toward the
bottom of the virtual address space; 3) move the vsyscall page out of
fixmap and put it just below the kernel address space (i.e. top of user
space). The latter is the currently favoured option, as recommended by
Andi (it's also used by the x86_64 ia32 compat layer). I'm hoping we can
get this in before the SLES10 xen kernel gets finalized.


Ian

Re: Re: [Xen-changelog] Indirect hypercalls through ahypercall transfer page.

[Ulrich Drepper]

[-- Attachment #1.1: Type: text/plain, Size: 531 bytes --]

Ian Pratt wrote:

> The vsyscall
> page lives within the fixaddr region, and its virtual address needs to
> be known at *compile* time as it gets built into various elf objects.

That's not necessarily the case.

The FC x86 kernel has random vdso address support.  I don't know why
this isn't also enabled for x86-64 but there is no real reason for that.
   These are, after all, DSOs which need little fixup for a new location.

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-changelog] Indirect hypercalls through ahypercall transfer page.

[Ulrich Drepper]

[-- Attachment #1.1: Type: text/plain, Size: 756 bytes --]

Ulrich Drepper wrote:
> I don't know why
> this isn't also enabled for x86-64 but there is no real reason for that.

Actually, I remember.  It's Andi insistance on this stupid vsyscall
interface with fixed addresses.  This is absolutely unnecessary.  vdsos
are so much nicer to handle, need no special code.  If somebody would
finally be able to replace the vsyscall stuff x86-64 currently uses with
a vdso implementation (passing up the address in the auxiliary vector)
all would be nice and easy.  x86 and ppc already use the vdso, it's not
an unproven concept.  ppc even uses it in a way which would be needed to
implement the fast gettimeofday etc.

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel