xen/xen vs xen/kvm nesting with pv drivers

xen/xen vs xen/kvm nesting with pv drivers

[Anthony Wright]

I tried to install Xen (4.7.0 with linux 4.7.2 Dom0) on an AWS virtual machine and it failed because while AWS uses Xen it requires that you use the PVHVM network driver. I then tried to install Xen on a Google Cloud virtual machine and despite also requiring you to use PV drivers, that succeeded because Google Cloud uses KVM.

I think this means that if you nest Xen in KVM you can use high performance drivers, but if you nest Xen in Xen you have to use slower drivers, which seems to be the wrong way around!

I'd like to be able to install Xen on an AWS virtual machine, and wondered what are the challenges to getting the pv drivers working in a nested environment. Is this a problem with the Dom0 kernel only expecting there to be a single XenStore, or is there also a problem in Xen?

Thanks,

Anthony Wright

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: xen/xen vs xen/kvm nesting with pv drivers

[Andrew Cooper]

On 06/09/16 13:47, Anthony Wright wrote:
> I tried to install Xen (4.7.0 with linux 4.7.2 Dom0) on an AWS virtual machine and it failed because while AWS uses Xen it requires that you use the PVHVM network driver. I then tried to install Xen on a Google Cloud virtual machine and despite also requiring you to use PV drivers, that succeeded because Google Cloud uses KVM.
>
> I think this means that if you nest Xen in KVM you can use high performance drivers, but if you nest Xen in Xen you have to use slower drivers, which seems to be the wrong way around!
>
> I'd like to be able to install Xen on an AWS virtual machine, and wondered what are the challenges to getting the pv drivers working in a nested environment. Is this a problem with the Dom0 kernel only expecting there to be a single XenStore, or is there also a problem in Xen?

Nesting Xen inside Xen and getting high-speed drivers at L1 is a hard
problem, which is why noone has tackled it yet.

The problems all revolve around L1's dom0.  It can't issue hypercalls to
L0, meaning that it cant find or connect the xenstore ring.  Even if it
could, there is the problem of multiple xenstores, which doesn't fit in
the current architecture.

It would be lovely if someone would work on this, but it is a very large
swamp.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: xen/xen vs xen/kvm nesting with pv drivers

[Anthony Wright]

On 06/09/2016 14:05, Andrew Cooper wrote:
> On 06/09/16 13:47, Anthony Wright wrote:
>> I tried to install Xen (4.7.0 with linux 4.7.2 Dom0) on an AWS virtual machine and it failed because while AWS uses Xen it requires that you use the PVHVM network driver. I then tried to install Xen on a Google Cloud virtual machine and despite also requiring you to use PV drivers, that succeeded because Google Cloud uses KVM.
>>
>> I think this means that if you nest Xen in KVM you can use high performance drivers, but if you nest Xen in Xen you have to use slower drivers, which seems to be the wrong way around!
>>
>> I'd like to be able to install Xen on an AWS virtual machine, and wondered what are the challenges to getting the pv drivers working in a nested environment. Is this a problem with the Dom0 kernel only expecting there to be a single XenStore, or is there also a problem in Xen?
> Nesting Xen inside Xen and getting high-speed drivers at L1 is a hard
> problem, which is why noone has tackled it yet.
>
> The problems all revolve around L1's dom0.  It can't issue hypercalls to
> L0, meaning that it cant find or connect the xenstore ring.  Even if it
> could, there is the problem of multiple xenstores, which doesn't fit in
> the current architecture.
>
> It would be lovely if someone would work on this, but it is a very large
> swamp.
>
> ~Andrew
Does the L1's Dom0 have to issue the hypercalls directly? Would it be
possible to get the L1's Dom0 to issue the request to the L1 hypervisor
and that to call the L0 hypervisor? This would seem to fit the current
architecture fairly closely. (Sorry if I've got the terminology wrong)

Regarding multiple XenStores, I appreciate there would be significant
problems, but you'd only have a maximum of two XenStores, one for the
xenback drivers (the current XenStore) and one for the xenfront drivers
(that talks to the parent hypervisor).

Anthony


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: xen/xen vs xen/kvm nesting with pv drivers

[Andrew Cooper]

On 06/09/16 14:32, Anthony Wright wrote:
> On 06/09/2016 14:05, Andrew Cooper wrote:
>> On 06/09/16 13:47, Anthony Wright wrote:
>>> I tried to install Xen (4.7.0 with linux 4.7.2 Dom0) on an AWS virtual machine and it failed because while AWS uses Xen it requires that you use the PVHVM network driver. I then tried to install Xen on a Google Cloud virtual machine and despite also requiring you to use PV drivers, that succeeded because Google Cloud uses KVM.
>>>
>>> I think this means that if you nest Xen in KVM you can use high performance drivers, but if you nest Xen in Xen you have to use slower drivers, which seems to be the wrong way around!
>>>
>>> I'd like to be able to install Xen on an AWS virtual machine, and wondered what are the challenges to getting the pv drivers working in a nested environment. Is this a problem with the Dom0 kernel only expecting there to be a single XenStore, or is there also a problem in Xen?
>> Nesting Xen inside Xen and getting high-speed drivers at L1 is a hard
>> problem, which is why noone has tackled it yet.
>>
>> The problems all revolve around L1's dom0.  It can't issue hypercalls to
>> L0, meaning that it cant find or connect the xenstore ring.  Even if it
>> could, there is the problem of multiple xenstores, which doesn't fit in
>> the current architecture.
>>
>> It would be lovely if someone would work on this, but it is a very large
>> swamp.
>>
>> ~Andrew
> Does the L1's Dom0 have to issue the hypercalls directly? Would it be
> possible to get the L1's Dom0 to issue the request to the L1 hypervisor
> and that to call the L0 hypervisor? This would seem to fit the current
> architecture fairly closely. (Sorry if I've got the terminology wrong)

In principle, L1 Xen could proxy hypercalls from L1 dom0 to L0 Xen.

However, event channels and grant maps affect the full L1 guest physical
space, and need to be managed by the L1 Xen, not the L1 dom0.  At that
point, you are talking about proxying the event/grant interface, but as
Xen tries to specifically stay out of the way of disk/network in dom0,
it isn't obvious where the split lives.

>
> Regarding multiple XenStores, I appreciate there would be significant
> problems, but you'd only have a maximum of two XenStores, one for the
> xenback drivers (the current XenStore) and one for the xenfront drivers
> (that talks to the parent hypervisor).

Until this point, event channels, grant maps and xenstore have been
global per-domain with no concept of separate namespaces.  As a result,
changing the existing code to work in a nested way will be very invasive.


Fundamentally, the problem is that Xen's virtual architecture does not
nest cleanly.  This is easy to identify in hindsight, but about 15 years
too late to act upon.  I don't have any good suggestions, short of
something radical like using virtio.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: xen/xen vs xen/kvm nesting with pv drivers

[Konrad Rzeszutek Wilk]

On Tue, Sep 06, 2016 at 03:36:28PM +0100, Andrew Cooper wrote:
> On 06/09/16 14:32, Anthony Wright wrote:
> > On 06/09/2016 14:05, Andrew Cooper wrote:
> >> On 06/09/16 13:47, Anthony Wright wrote:
> >>> I tried to install Xen (4.7.0 with linux 4.7.2 Dom0) on an AWS virtual machine and it failed because while AWS uses Xen it requires that you use the PVHVM network driver. I then tried to install Xen on a Google Cloud virtual machine and despite also requiring you to use PV drivers, that succeeded because Google Cloud uses KVM.
> >>>
> >>> I think this means that if you nest Xen in KVM you can use high performance drivers, but if you nest Xen in Xen you have to use slower drivers, which seems to be the wrong way around!
> >>>
> >>> I'd like to be able to install Xen on an AWS virtual machine, and wondered what are the challenges to getting the pv drivers working in a nested environment. Is this a problem with the Dom0 kernel only expecting there to be a single XenStore, or is there also a problem in Xen?
> >> Nesting Xen inside Xen and getting high-speed drivers at L1 is a hard
> >> problem, which is why noone has tackled it yet.
> >>
> >> The problems all revolve around L1's dom0.  It can't issue hypercalls to
> >> L0, meaning that it cant find or connect the xenstore ring.  Even if it
> >> could, there is the problem of multiple xenstores, which doesn't fit in
> >> the current architecture.
> >>
> >> It would be lovely if someone would work on this, but it is a very large
> >> swamp.
> >>
> >> ~Andrew
> > Does the L1's Dom0 have to issue the hypercalls directly? Would it be
> > possible to get the L1's Dom0 to issue the request to the L1 hypervisor
> > and that to call the L0 hypervisor? This would seem to fit the current
> > architecture fairly closely. (Sorry if I've got the terminology wrong)
> 
> In principle, L1 Xen could proxy hypercalls from L1 dom0 to L0 Xen.
> 
> However, event channels and grant maps affect the full L1 guest physical
> space, and need to be managed by the L1 Xen, not the L1 dom0.  At that
> point, you are talking about proxying the event/grant interface, but as
> Xen tries to specifically stay out of the way of disk/network in dom0,
> it isn't obvious where the split lives.
> 
> >
> > Regarding multiple XenStores, I appreciate there would be significant
> > problems, but you'd only have a maximum of two XenStores, one for the
> > xenback drivers (the current XenStore) and one for the xenfront drivers
> > (that talks to the parent hypervisor).
> 
> Until this point, event channels, grant maps and xenstore have been
> global per-domain with no concept of separate namespaces.  As a result,
> changing the existing code to work in a nested way will be very invasive.
> 
> 
> Fundamentally, the problem is that Xen's virtual architecture does not
> nest cleanly.  This is easy to identify in hindsight, but about 15 years
> too late to act upon.  I don't have any good suggestions, short of
> something radical like using virtio.

There was an paper along with an RFC implemention of PV drivers in
nested dom0. CCing a co-worker who may remember it better than me.

> 
> ~Andrew
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: xen/xen vs xen/kvm nesting with pv drivers

[Konrad Rzeszutek Wilk]

. snip..
> > >> It would be lovely if someone would work on this, but it is a very large
> > >> swamp.
> > >>
> > >> ~Andrew
> > > Does the L1's Dom0 have to issue the hypercalls directly? Would it be
> > > possible to get the L1's Dom0 to issue the request to the L1 hypervisor
> > > and that to call the L0 hypervisor? This would seem to fit the current
> > > architecture fairly closely. (Sorry if I've got the terminology wrong)
> > 
> > In principle, L1 Xen could proxy hypercalls from L1 dom0 to L0 Xen.
> > 
> > However, event channels and grant maps affect the full L1 guest physical
> > space, and need to be managed by the L1 Xen, not the L1 dom0.  At that
> > point, you are talking about proxying the event/grant interface, but as
> > Xen tries to specifically stay out of the way of disk/network in dom0,
> > it isn't obvious where the split lives.
> > 
> > >
> > > Regarding multiple XenStores, I appreciate there would be significant
> > > problems, but you'd only have a maximum of two XenStores, one for the
> > > xenback drivers (the current XenStore) and one for the xenfront drivers
> > > (that talks to the parent hypervisor).
> > 
> > Until this point, event channels, grant maps and xenstore have been
> > global per-domain with no concept of separate namespaces.  As a result,
> > changing the existing code to work in a nested way will be very invasive.
> > 
> > 
> > Fundamentally, the problem is that Xen's virtual architecture does not
> > nest cleanly.  This is easy to identify in hindsight, but about 15 years
> > too late to act upon.  I don't have any good suggestions, short of
> > something radical like using virtio.
> 
> There was an paper along with an RFC implemention of PV drivers in
> nested dom0. CCing a co-worker who may remember it better than me.
> 

http://code.google.com/p/xen-blanket/
http://xcloud.cs.cornell.edu/pubs/xen_blanket.pdf

Are the links.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel