Vanilla Linux 64-bit paravirt guest support

Vanilla Linux 64-bit paravirt guest support

[Michael Abd-El-Malek]

Hello,

Can someone tell me what the status of 64-bit paravirtualization support is in 
the vanilla Linux tree?  It seems that 2.6.24 only has 32-bit PV support.  And 
it seems that 2.6.25 will be getting 64-bit PV support.  Will it be complete? 
Some lkml posts seem to suggest that not all hypercalls are supported for 64-bit?

On a related note: is it possible to compile the PV network/block drivers for a 
HVM Linux guest with the newest kernels?  Compilation failed and a previous 
message on the list says that you can't do that.  The reason I'd want to do this 
is to let me run a 64-bit recent Linux kernel with PV drivers.

Thanks,
Mike

Re: Vanilla Linux 64-bit paravirt guest support

[Jeremy Fitzhardinge]

Michael Abd-El-Malek wrote:
> Hello,
>
> Can someone tell me what the status of 64-bit paravirtualization 
> support is in the vanilla Linux tree?  It seems that 2.6.24 only has 
> 32-bit PV support.  And it seems that 2.6.25 will be getting 64-bit PV 
> support.  Will it be complete?

No, 2.6.25 will not have 64-bit Xen support.  I'm hoping it will be 
ready for .26.

> On a related note: is it possible to compile the PV network/block 
> drivers for a HVM Linux guest with the newest kernels?  Compilation 
> failed and a previous message on the list says that you can't do 
> that.  The reason I'd want to do this is to let me run a 64-bit recent 
> Linux kernel with PV drivers. 

No, we don't support that for the moment, though it would be a useful 
thing to have.

    J

Re: Vanilla Linux 64-bit paravirt guest support

[Michael Abd-El-Malek]

Jeremy Fitzhardinge wrote:
> Michael Abd-El-Malek wrote:
>> Can someone tell me what the status of 64-bit paravirtualization 
>> support is in the vanilla Linux tree?  It seems that 2.6.24 only has 
>> 32-bit PV support.  And it seems that 2.6.25 will be getting 64-bit PV 
>> support.  Will it be complete?
> 
> No, 2.6.25 will not have 64-bit Xen support.  I'm hoping it will be 
> ready for .26.

Is 64-bit domU support available anywhere at the moment?  For example, what is 
the status of the git://git.et.redhat.com/xen-pvops-64.git tree?  I pulled that 
tree and tried building 64-bit Xen domU support (since this tree allows you to 
configure the kernel with that capability, unlike the vanilla Linux tree).  But 
compilation failed in enlighten.c because xen_smp_ops isn't defined in x86_64.

>> On a related note: is it possible to compile the PV network/block 
>> drivers for a HVM Linux guest with the newest kernels?  Compilation 
>> failed and a previous message on the list says that you can't do 
>> that.  The reason I'd want to do this is to let me run a 64-bit recent 
>> Linux kernel with PV drivers. 
> 
> No, we don't support that for the moment, though it would be a useful 
> thing to have.

Is there a reason this wouldn't work, or is it just "scripting changes" that 
need to be done?

Thanks,
Mike

Re: Vanilla Linux 64-bit paravirt guest support

[Jeremy Fitzhardinge]

Michael Abd-El-Malek wrote:
> Is 64-bit domU support available anywhere at the moment?  For example, 
> what is the status of the git://git.et.redhat.com/xen-pvops-64.git 
> tree?  I pulled that tree and tried building 64-bit Xen domU support 
> (since this tree allows you to configure the kernel with that 
> capability, unlike the vanilla Linux tree).  But compilation failed in 
> enlighten.c because xen_smp_ops isn't defined in x86_64.

Redhat have some patches which they're shipping in Fedora 9.  Once F9 is 
out the door, I'm hoping they'll polish them into an upstreamable form.  
I don't know whether that git tree represents what's in F9, or if that's 
somewhere else; at the very least I'd expect you'd be able to pull the 
patches out of the srpm.

>>> On a related note: is it possible to compile the PV network/block 
>>> drivers for a HVM Linux guest with the newest kernels?  Compilation 
>>> failed and a previous message on the list says that you can't do 
>>> that.  The reason I'd want to do this is to let me run a 64-bit 
>>> recent Linux kernel with PV drivers. 
>>
>> No, we don't support that for the moment, though it would be a useful 
>> thing to have.
>
> Is there a reason this wouldn't work, or is it just "scripting 
> changes" that need to be done? 

In order to simplify the initial upstreaming work, I removed the support 
for pv-in-hvm, since it adds a moderate amount of extra complexity.  
We'd need to put that back in - it shouldn't be too hard, but it hasn't 
been a priority so far.

    J

Re: Vanilla Linux 64-bit paravirt guest support

[Mark McLoughlin]

On Thu, 2008-04-10 at 09:01 -0500, Jeremy Fitzhardinge wrote:
> Michael Abd-El-Malek wrote:
> > Is 64-bit domU support available anywhere at the moment?  For example, 
> > what is the status of the git://git.et.redhat.com/xen-pvops-64.git 
> > tree?  I pulled that tree and tried building 64-bit Xen domU support 
> > (since this tree allows you to configure the kernel with that 
> > capability, unlike the vanilla Linux tree).  But compilation failed in 
> > enlighten.c because xen_smp_ops isn't defined in x86_64.

Try building without CONFIG_SMP, it doesn't support that yet.

> Redhat have some patches which they're shipping in Fedora 9.  Once F9 is 
> out the door, I'm hoping they'll polish them into an upstreamable form.  
> I don't know whether that git tree represents what's in F9, or if that's 
> somewhere else; at the very least I'd expect you'd be able to pull the 
> patches out of the srpm.

Yep, this tree:

  http://git.et.redhat.com/?p=xen-pvops-64.git

is the work-in-progress x86_64 tree.

This tree:

  http://git.et.redhat.com/?p=linux-2.6-fedora-pvops.git

is what we're actually shipping for F-9. It includes the x86_64 work,
but some other paravirt_ops patches too, most of which are queued up
upstream.

Cheers,
Mark.

Re: Vanilla Linux 64-bit paravirt guest support

[Jeremy Fitzhardinge]

Mark McLoughlin wrote:
> s what we're actually shipping for F-9. It includes the x86_64 work,
> but some other paravirt_ops patches too, most of which are queued up
> upstream.

What else do you have?  Hm.  I was really hoping to drop /proc/xen/...

    J

Re: Vanilla Linux 64-bit paravirt guest support

[Michael Abd-El-Malek]

On Apr 10, 2008, at 10:11 AM, Mark McLoughlin wrote:
> On Thu, 2008-04-10 at 09:01 -0500, Jeremy Fitzhardinge wrote:
>> Michael Abd-El-Malek wrote:
>>> Is 64-bit domU support available anywhere at the moment?  For  
>>> example,
>>> what is the status of the git://git.et.redhat.com/xen-pvops-64.git
>>> tree?  I pulled that tree and tried building 64-bit Xen domU support
>>> (since this tree allows you to configure the kernel with that
>>> capability, unlike the vanilla Linux tree).  But compilation  
>>> failed in
>>> enlighten.c because xen_smp_ops isn't defined in x86_64.
>
> Try building without CONFIG_SMP, it doesn't support that yet.

Other than SMP support, does the tree represent a fully functional 64- 
bit PV domU support? Does it also allow all hypercalls?  Put another  
way: is a 64-bit PV domU from that tree less capable than a 64-bit PV  
domU from Xen's linux-2.6.18.8 tree?

>> Redhat have some patches which they're shipping in Fedora 9.  Once  
>> F9 is
>> out the door, I'm hoping they'll polish them into an upstreamable  
>> form.
>> I don't know whether that git tree represents what's in F9, or if  
>> that's
>> somewhere else; at the very least I'd expect you'd be able to pull  
>> the
>> patches out of the srpm.
>
> Yep, this tree:
>
>  http://git.et.redhat.com/?p=xen-pvops-64.git
>
> is the work-in-progress x86_64 tree.
>
> This tree:
>
>  http://git.et.redhat.com/?p=linux-2.6-fedora-pvops.git
>
> is what we're actually shipping for F-9. It includes the x86_64 work,
> but some other paravirt_ops patches too, most of which are queued up
> upstream.

Which tree do you recommend I use?

Thanks,
Mike

Re: Vanilla Linux 64-bit paravirt guest support

[Mark McLoughlin]

On Thu, 2008-04-10 at 09:25 -0500, Jeremy Fitzhardinge wrote:
> Mark McLoughlin wrote:
> > s what we're actually shipping for F-9. It includes the x86_64 work,
> > but some other paravirt_ops patches too, most of which are queued up
> > upstream.
> 
> What else do you have?  Hm.  I was really hoping to drop /proc/xen/...

Okay, here's the list:

    xen x86_64: Initial x86_64 support for Xen paravirt_ops

Eduardo's x86_64 stuff

    xen x86_64: Add 64 bit version of privcmd_hypercall()
    xen: Add Xen's /sys/hypervisor interface
    xen: Add /proc/xen/xenbus
    xen: Add /proc/xen/privcmd
    xen: Add /proc/xen/capabilities
    xen: Add empty xenctrl module

This is /proc/xen and /sys/hypervisor stuff pulled from the dom0 tree.
In that tree we've got mostly full compatibility with the
linux-2.6.18-xen userspace interfaces AFAIR.

I probably don't need all of these for DomU, but I went for the safe bet
of including anything that userspace might be relying on.

Happy for us to drop /proc/xen in the future - this is just a "get
things going" thing.

    xen debug: Add xprintk to log directly via hypercall

Just for debugging

    xen: Add a vmlinuz target

bzImage is the way forward, but for now it was easier to just do stick
with vmlinuz - e.g. until there's support for bzImage Dom0 and we get
newer Xen HV and userspace in Fedora 

    xen blkfront: Delay wait for block devices until after the disk is added.
    xen: Add compatibility aliases for frontend drivers
    xen: Module autoprobing support for frontend drivers
    xen: Enable Xen console by default in domU
    xen pvfb: Para-virtual framebuffer, keyboard and pointer driver
    xen: Make xen-blkfront write its protocol ABI to xenstore
    xen: Do not pin/unpin PMD pages

All queued upstream

    xen x86_64: Only define load_user_cs_desc() on 32 bit
    xen execshield: fix endless GPF fault loop
    xen execshield: Add xen-specific load_user_cs_desc()
    squashfs: Fix build without CONFIG_SMP

Fedora specific (e.g. execshield)

Cheers,
Mark.

Re: Vanilla Linux 64-bit paravirt guest support

[Mark McLoughlin]

On Thu, 2008-04-10 at 10:32 -0400, Michael Abd-El-Malek wrote:
> On Apr 10, 2008, at 10:11 AM, Mark McLoughlin wrote:
> > On Thu, 2008-04-10 at 09:01 -0500, Jeremy Fitzhardinge wrote:
> >> Michael Abd-El-Malek wrote:
> >>> Is 64-bit domU support available anywhere at the moment?  For  
> >>> example,
> >>> what is the status of the git://git.et.redhat.com/xen-pvops-64.git
> >>> tree?  I pulled that tree and tried building 64-bit Xen domU support
> >>> (since this tree allows you to configure the kernel with that
> >>> capability, unlike the vanilla Linux tree).  But compilation  
> >>> failed in
> >>> enlighten.c because xen_smp_ops isn't defined in x86_64.
> >
> > Try building without CONFIG_SMP, it doesn't support that yet.
> 
> Other than SMP support, does the tree represent a fully functional 64- 
> bit PV domU support? 

No, it's a work-in-progress - ia32 emulation is also missing and we're
tracking down a nasty pagetable pinning bug atm.

Even then it would only be on par with 32-bit pv_ops DomU, which itself
doesn't yet have all the Xen features of 2.6.18 tree.

> Does it also allow all hypercalls?  Put another  
> way: is a 64-bit PV domU from that tree less capable than a 64-bit PV  
> domU from Xen's linux-2.6.18.8 tree?

It depends on how you define "less capable" - e.g. some might think a
tree based on a 1.5 year old kernel  is less capable even if it does
have more xen features ... :-)

> >> Redhat have some patches which they're shipping in Fedora 9.  Once  
> >> F9 is
> >> out the door, I'm hoping they'll polish them into an upstreamable  
> >> form.
> >> I don't know whether that git tree represents what's in F9, or if  
> >> that's
> >> somewhere else; at the very least I'd expect you'd be able to pull  
> >> the
> >> patches out of the srpm.
> >
> > Yep, this tree:
> >
> >  http://git.et.redhat.com/?p=xen-pvops-64.git
> >
> > is the work-in-progress x86_64 tree.
> >
> > This tree:
> >
> >  http://git.et.redhat.com/?p=linux-2.6-fedora-pvops.git
> >
> > is what we're actually shipping for F-9. It includes the x86_64 work,
> > but some other paravirt_ops patches too, most of which are queued up
> > upstream.
> 
> Which tree do you recommend I use?

I'd use the xen-pvops-64 tree unless you are specifically wanting to
help with Fedora's kernel-xen packages.

Cheers,
Mark.

Re: Vanilla Linux 64-bit paravirt guest support

[Jeremy Fitzhardinge]

Michael Abd-El-Malek wrote:
> On Apr 10, 2008, at 10:11 AM, Mark McLoughlin wrote:
>> On Thu, 2008-04-10 at 09:01 -0500, Jeremy Fitzhardinge wrote:
>>> Michael Abd-El-Malek wrote:
>>>> Is 64-bit domU support available anywhere at the moment?  For example,
>>>> what is the status of the git://git.et.redhat.com/xen-pvops-64.git
>>>> tree?  I pulled that tree and tried building 64-bit Xen domU support
>>>> (since this tree allows you to configure the kernel with that
>>>> capability, unlike the vanilla Linux tree).  But compilation failed in
>>>> enlighten.c because xen_smp_ops isn't defined in x86_64.
>>
>> Try building without CONFIG_SMP, it doesn't support that yet.
>
> Other than SMP support, does the tree represent a fully functional 
> 64-bit PV domU support? Does it also allow all hypercalls?  Put 
> another way: is a 64-bit PV domU from that tree less capable than a 
> 64-bit PV domU from Xen's linux-2.6.18.8 tree?

The 32-bit domU implementation is less capable than 2.6.18-xen.  For 
example, it does not yet support suspend/resume/migrate or ballooning, 
and a number of interesting features are not yet in a released kernel 
(pvfb).  Further out I hope we'll get things like dom0 support in as well.

(I'm not sure what you mean by "allow all hypercalls"; do you mean "all 
features"?)

    J

Re: Vanilla Linux 64-bit paravirt guest support

[Jeremy Fitzhardinge]

Mark McLoughlin wrote:
> On Thu, 2008-04-10 at 09:25 -0500, Jeremy Fitzhardinge wrote:
>   
>> Mark McLoughlin wrote:
>>     
>>> s what we're actually shipping for F-9. It includes the x86_64 work,
>>> but some other paravirt_ops patches too, most of which are queued up
>>> upstream.
>>>       
>> What else do you have?  Hm.  I was really hoping to drop /proc/xen/...
>>     
>
> Okay, here's the list:
>
>     xen x86_64: Initial x86_64 support for Xen paravirt_ops
>
> Eduardo's x86_64 stuff
>
>     xen x86_64: Add 64 bit version of privcmd_hypercall()
>     xen: Add Xen's /sys/hypervisor interface
>     xen: Add /proc/xen/xenbus
>     xen: Add /proc/xen/privcmd
>     xen: Add /proc/xen/capabilities
>     xen: Add empty xenctrl module
>
> This is /proc/xen and /sys/hypervisor stuff pulled from the dom0 tree.
> In that tree we've got mostly full compatibility with the
> linux-2.6.18-xen userspace interfaces AFAIR.
>
> I probably don't need all of these for DomU, but I went for the safe bet
> of including anything that userspace might be relying on.
>
> Happy for us to drop /proc/xen in the future - this is just a "get
> things going" thing.
>   

I'd rather fix userspace now, rather than let anyone get the impression 
that /proc/xen/ is going to be a supported interface.  We'll probably 
have to end up making /proc/xen/ exist for backwards compat, but we 
should make it clear that it isn't the canonical path.

Not that I've thought about where it should be.  /sys/hypervisor/xen?  
xendom0fs?

    J

Fix a typo in p2m.c

[Huang2, Wei]

This is a fix to typo in p2m.c. Courtesy to Xiao Wang
(sirouni@yahoo.com.cn) for pointing out this bug.


===================
diff -r 8d750b7acfa3 xen/arch/x86/mm/p2m.c
--- a/xen/arch/x86/mm/p2m.c     Thu Apr 10 11:11:25 2008 +0100
+++ b/xen/arch/x86/mm/p2m.c     Thu Apr 10 04:13:59 2008 -0500
@@ -941,7 +941,7 @@ void p2m_change_type_global(struct domai
                     mfn = l1e_get_pfn(l1e[i1]);
                     gfn = get_gpfn_from_mfn(mfn);
                     /* create a new 1le entry with the new type */
-                    flags = p2m_flags_to_type(nt);
+                    flags = p2m_type_to_flags(nt);
                     l1e_content = l1e_from_pfn(mfn, flags);
                     paging_write_p2m_entry(d, gfn, &l1e[i1],
                                            l1mfn, l1e_content, 1);

Re: Vanilla Linux 64-bit paravirt guest support

[Michael Abd-El-Malek]

On Apr 10, 2008, at 10:57 AM, Jeremy Fitzhardinge wrote:
> Michael Abd-El-Malek wrote:
>> On Apr 10, 2008, at 10:11 AM, Mark McLoughlin wrote:
>>> On Thu, 2008-04-10 at 09:01 -0500, Jeremy Fitzhardinge wrote:
>>>> Michael Abd-El-Malek wrote:
>>>>> Is 64-bit domU support available anywhere at the moment?  For  
>>>>> example,
>>>>> what is the status of the git://git.et.redhat.com/xen-pvops-64.git
>>>>> tree?  I pulled that tree and tried building 64-bit Xen domU  
>>>>> support
>>>>> (since this tree allows you to configure the kernel with that
>>>>> capability, unlike the vanilla Linux tree).  But compilation  
>>>>> failed in
>>>>> enlighten.c because xen_smp_ops isn't defined in x86_64.
>>>
>>> Try building without CONFIG_SMP, it doesn't support that yet.
>>
>> Other than SMP support, does the tree represent a fully functional  
>> 64-bit PV domU support? Does it also allow all hypercalls?  Put  
>> another way: is a 64-bit PV domU from that tree less capable than a  
>> 64-bit PV domU from Xen's linux-2.6.18.8 tree?
>
> The 32-bit domU implementation is less capable than 2.6.18-xen.  For  
> example, it does not yet support suspend/resume/migrate or  
> ballooning, and a number of interesting features are not yet in a  
> released kernel (pvfb).  Further out I hope we'll get things like  
> dom0 support in as well.

For my purposes, suspend/resume/migrate isn't required.  But I might  
need ballooning.  Would I be able to take the balloon driver from  
2.6.18.8 and use it in those new trees?  The new tree seems to have  
all the hypercalls and defines the increase_reservation/ 
decrease_reservation op codes.  I can add the stub code for these calls.

> (I'm not sure what you mean by "allow all hypercalls"; do you mean  
> "all features"?)

I was thinking along the lines of grant hypercalls for mapping another  
domU's grants.  HVM can't let me do that.  But I need to be able to do  
this for my work.

Thanks,
Mike

Re: Vanilla Linux 64-bit paravirt guest support

[Jeremy Fitzhardinge]

Michael Abd-El-Malek wrote:
> For my purposes, suspend/resume/migrate isn't required.  But I might 
> need ballooning.  Would I be able to take the balloon driver from 
> 2.6.18.8 and use it in those new trees?  The new tree seems to have 
> all the hypercalls and defines the 
> increase_reservation/decrease_reservation op codes.  I can add the 
> stub code for these calls.

I have pending patches for balloon support in x86.git 
(http://people.redhat.com/mingo/x86.git/README)

>> (I'm not sure what you mean by "allow all hypercalls"; do you mean 
>> "all features"?)
>
> I was thinking along the lines of grant hypercalls for mapping another 
> domU's grants.  HVM can't let me do that.  But I need to be able to do 
> this for my work. 

If you have a kernel driver, then you can use those hypercalls.

    J

Re: Vanilla Linux 64-bit paravirt guest support

[Bastian Blank]

On Thu, Apr 10, 2008 at 10:04:31AM -0500, Jeremy Fitzhardinge wrote:
> Not that I've thought about where it should be.  /sys/hypervisor/xen?  

Yes.

> xendom0fs?

xen_hypfs to follow the namingscheme used for the other implementation:

| # grep s390 /proc/filesystems
| nodev   s390_hypfs

It is not restricted to dom0.

Bastian

-- 
Conquest is easy. Control is not.
		-- Kirk, "Mirror, Mirror", stardate unknown

Re: Vanilla Linux 64-bit paravirt guest support

[Mark McLoughlin]

On Thu, 2008-04-10 at 15:11 +0100, Mark McLoughlin wrote:
> On Thu, 2008-04-10 at 09:01 -0500, Jeremy Fitzhardinge wrote:
> > Michael Abd-El-Malek wrote:
> > > Is 64-bit domU support available anywhere at the moment?  For example, 
> > > what is the status of the git://git.et.redhat.com/xen-pvops-64.git 
> > > tree?  I pulled that tree and tried building 64-bit Xen domU support 
> > > (since this tree allows you to configure the kernel with that 
> > > capability, unlike the vanilla Linux tree).  But compilation failed in 
> > > enlighten.c because xen_smp_ops isn't defined in x86_64.
> 
> Try building without CONFIG_SMP, it doesn't support that yet.
> 
> > Redhat have some patches which they're shipping in Fedora 9.  Once F9 is 
> > out the door, I'm hoping they'll polish them into an upstreamable form.  
> > I don't know whether that git tree represents what's in F9, or if that's 
> > somewhere else; at the very least I'd expect you'd be able to pull the 
> > patches out of the srpm.
> 
> Yep, this tree:
> 
>   http://git.et.redhat.com/?p=xen-pvops-64.git
> 
> is the work-in-progress x86_64 tree.

FYI, here's the config options that's currently needed to get this tree
working:

# CONFIG_SMP is not set
# CONFIG_NUMA is not set
# CONFIG_NEED_MULTIPLE_NODES is not set

# x86_64 breaks with a different CONFIG_PHYSICAL_START, currently
CONFIG_PHYSICAL_START=0x200000

# x86_64 breaks with CONFIG_SPARSEMEM_VMEMMAP, currently
# CONFIG_SPARSEMEM_VMEMMAP is not set

# 32-bit emulation isn't ready yet
# CONFIG_IA32_EMULATION is not set

Cheers,
Mark.