PVHVM drivers in upstream linux kernel

PVHVM drivers in upstream linux kernel

[Juergen Gross]

Hi,

looking into the upstream linux sources I realized that the PVHVM
drivers of XEN are only available with the pvops kernel. Is this on
purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
configurable without having to enable CONFIG_PARAVIRT?

Juergen

Re: PVHVM drivers in upstream linux kernel

[David Vrabel]

On 02/12/14 09:39, Juergen Gross wrote:
> Hi,
> 
> looking into the upstream linux sources I realized that the PVHVM
> drivers of XEN are only available with the pvops kernel. Is this on
> purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
> configurable without having to enable CONFIG_PARAVIRT?

I suppose that would be possible but I don't think it's a useful
configuration because you would lose PV spinlocks for example.

David

Re: PVHVM drivers in upstream linux kernel

[Juergen Gross]

On 12/02/2014 11:54 AM, David Vrabel wrote:
> On 02/12/14 09:39, Juergen Gross wrote:
>> Hi,
>>
>> looking into the upstream linux sources I realized that the PVHVM
>> drivers of XEN are only available with the pvops kernel. Is this on
>> purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
>> configurable without having to enable CONFIG_PARAVIRT?
>
> I suppose that would be possible but I don't think it's a useful
> configuration because you would lose PV spinlocks for example.

Having no pv spinlocks on a single vcpu HVM guest isn't so bad. Having
no pv-drivers for disks and networking is worse.


Juergen

Re: PVHVM drivers in upstream linux kernel

[Ian Campbell]

On Tue, 2014-12-02 at 10:54 +0000, David Vrabel wrote:
> On 02/12/14 09:39, Juergen Gross wrote:
> > Hi,
> > 
> > looking into the upstream linux sources I realized that the PVHVM
> > drivers of XEN are only available with the pvops kernel. Is this on
> > purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
> > configurable without having to enable CONFIG_PARAVIRT?
> 
> I suppose that would be possible but I don't think it's a useful
> configuration because you would lose PV spinlocks for example.

IIRC the reason this hasn't been implemented until now is that
refactoring would be required to the various bits of driver code which
assumes PAE + PARAVIRT when they aren't strictly needed, e.g. grant
table code. Whether its worth the churn at this stage is debatable, but
I think the (in)ability to use PV spinlocks is a red-herring.

Adding PV drivers to an HVM guest is a useful thing to do, even without
PV spinlocks. PV IO gets you far more incremental benefit than the locks
do, adding PV IO paths is the number 1 thing which should be done to any
guest.

One actual usecase is installing from a distro installer which isn't
PAE, let alone PARAVIRT enabled[0], to get far enough that you can
install a more capable PVHVM kernel with more bells and whistles.

If there were distros around who refused wholesale to enable PARAVIRT
even in a non-default kernel then it would be more likely that they
could be convinced to enable a set of PV IO drivers, since they have 0
impact on a non-PARAVIRT system, and still give significant benefit to
Xen users. I don't know of any of the major distros are refusing
PARAVIRT in this way though.

Ian.

[0] The default i386 Debian installer falls into this camp, but you can
use the special PV Xen variant to install as PVHVM too so it's not so
critical.

Re: PVHVM drivers in upstream linux kernel

[Juergen Gross]

On 12/02/2014 12:05 PM, Ian Campbell wrote:
> On Tue, 2014-12-02 at 10:54 +0000, David Vrabel wrote:
>> On 02/12/14 09:39, Juergen Gross wrote:
>>> Hi,
>>>
>>> looking into the upstream linux sources I realized that the PVHVM
>>> drivers of XEN are only available with the pvops kernel. Is this on
>>> purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
>>> configurable without having to enable CONFIG_PARAVIRT?
>>
>> I suppose that would be possible but I don't think it's a useful
>> configuration because you would lose PV spinlocks for example.
>
> IIRC the reason this hasn't been implemented until now is that
> refactoring would be required to the various bits of driver code which
> assumes PAE + PARAVIRT when they aren't strictly needed, e.g. grant
> table code. Whether its worth the churn at this stage is debatable, but
> I think the (in)ability to use PV spinlocks is a red-herring.
>
> Adding PV drivers to an HVM guest is a useful thing to do, even without
> PV spinlocks. PV IO gets you far more incremental benefit than the locks
> do, adding PV IO paths is the number 1 thing which should be done to any
> guest.

I take this as an "ack" to change this. :-)

> One actual usecase is installing from a distro installer which isn't
> PAE, let alone PARAVIRT enabled[0], to get far enough that you can
> install a more capable PVHVM kernel with more bells and whistles.
>
> If there were distros around who refused wholesale to enable PARAVIRT
> even in a non-default kernel then it would be more likely that they
> could be convinced to enable a set of PV IO drivers, since they have 0
> impact on a non-PARAVIRT system, and still give significant benefit to
> Xen users. I don't know of any of the major distros are refusing
> PARAVIRT in this way though.

I think we have customers wanting to run a default kernel as domU. So it
isn't always the distro refusing paravirt, it might be the user...

Okay, how do the current config settings regarding Xen look like?

We have:
- XEN depending on PARAVIRT
- XEN_DOM0 depending on XEN and others
- XEN_BACKEND depending on XEN_DOM0
- various backend drivers depending on XEN_BACKEND
- XEN_PVHVM depending on XEN
- various frontend drivers depending on XEN (even if some are not
   depending on XEN according to Kconfig, they do as the complete
   drivers/xen directory is made only if CONFIG_XEN is defined)

To sort things out I'd suggest to:
- make XEN independent from PARAVIRT
- let XEN_DOM0 select XEN_BACKEND, PARAVIRT, XEN
- let XEN_BACKEND select PARAVIRT, XEN (I'd like to be able to build
   a driver domain without XEN_DOM0)
- introduce XEN_FRONTEND, let it select XEN
- let frontend drivers and drivers needed by those depend on
   XEN_FRONTEND
- let XEN_PVHVM select XEN_FRONTEND
- don't skip drivers/xen on make, as XEN might be selected via a
   config item in that directory


Juergen

Re: PVHVM drivers in upstream linux kernel

[Ian Campbell]

On Tue, 2014-12-02 at 12:33 +0100, Juergen Gross wrote:
> On 12/02/2014 12:05 PM, Ian Campbell wrote:
> > On Tue, 2014-12-02 at 10:54 +0000, David Vrabel wrote:
> >> On 02/12/14 09:39, Juergen Gross wrote:
> >>> Hi,
> >>>
> >>> looking into the upstream linux sources I realized that the PVHVM
> >>> drivers of XEN are only available with the pvops kernel. Is this on
> >>> purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
> >>> configurable without having to enable CONFIG_PARAVIRT?
> >>
> >> I suppose that would be possible but I don't think it's a useful
> >> configuration because you would lose PV spinlocks for example.
> >
> > IIRC the reason this hasn't been implemented until now is that
> > refactoring would be required to the various bits of driver code which
> > assumes PAE + PARAVIRT when they aren't strictly needed, e.g. grant
> > table code. Whether its worth the churn at this stage is debatable, but
> > I think the (in)ability to use PV spinlocks is a red-herring.
> >
> > Adding PV drivers to an HVM guest is a useful thing to do, even without
> > PV spinlocks. PV IO gets you far more incremental benefit than the locks
> > do, adding PV IO paths is the number 1 thing which should be done to any
> > guest.
> 
> I take this as an "ack" to change this. :-)

It's a best "IMHO being able to do this is a useful thing". I can't ack
the actual final patch, a) I'm not a relevant maintainer and b) I've not
seen the patch.

> Okay, how do the current config settings regarding Xen look like?

...I'll leave the mechanics down to you and the maintainers to thrash
out.

Ian.

Re: PVHVM drivers in upstream linux kernel

[Juergen Gross]

On 12/02/2014 12:36 PM, Ian Campbell wrote:
> On Tue, 2014-12-02 at 12:33 +0100, Juergen Gross wrote:
>> On 12/02/2014 12:05 PM, Ian Campbell wrote:
>>> On Tue, 2014-12-02 at 10:54 +0000, David Vrabel wrote:
>>>> On 02/12/14 09:39, Juergen Gross wrote:
>>>>> Hi,
>>>>>
>>>>> looking into the upstream linux sources I realized that the PVHVM
>>>>> drivers of XEN are only available with the pvops kernel. Is this on
>>>>> purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
>>>>> configurable without having to enable CONFIG_PARAVIRT?
>>>>
>>>> I suppose that would be possible but I don't think it's a useful
>>>> configuration because you would lose PV spinlocks for example.
>>>
>>> IIRC the reason this hasn't been implemented until now is that
>>> refactoring would be required to the various bits of driver code which
>>> assumes PAE + PARAVIRT when they aren't strictly needed, e.g. grant
>>> table code. Whether its worth the churn at this stage is debatable, but
>>> I think the (in)ability to use PV spinlocks is a red-herring.
>>>
>>> Adding PV drivers to an HVM guest is a useful thing to do, even without
>>> PV spinlocks. PV IO gets you far more incremental benefit than the locks
>>> do, adding PV IO paths is the number 1 thing which should be done to any
>>> guest.
>>
>> I take this as an "ack" to change this. :-)
>
> It's a best "IMHO being able to do this is a useful thing". I can't ack
> the actual final patch, a) I'm not a relevant maintainer and b) I've not
> seen the patch.

It was not meant to be taken as an ack for a not yet written patch. :-)

>
>> Okay, how do the current config settings regarding Xen look like?
>
> ...I'll leave the mechanics down to you and the maintainers to thrash
> out.

Yeah, stay tuned. :-)


Juergen

Re: PVHVM drivers in upstream linux kernel

[David Vrabel]

On 02/12/14 11:33, Juergen Gross wrote:
> 
> I think we have customers wanting to run a default kernel as domU. So it
> isn't always the distro refusing paravirt, it might be the user...

I don't think this is a sensible use case but I'm not adverse to
improving the set of Xen config options.

> To sort things out I'd suggest to:
> - make XEN independent from PARAVIRT
> - let XEN_DOM0 select XEN_BACKEND, PARAVIRT, XEN
> - let XEN_BACKEND select PARAVIRT, XEN (I'd like to be able to build
>   a driver domain without XEN_DOM0)
> - introduce XEN_FRONTEND, let it select XEN
> - let frontend drivers and drivers needed by those depend on
>   XEN_FRONTEND
> - let XEN_PVHVM select XEN_FRONTEND

Rather than looking at the current set of configuration options, can you
look at what user-visible functionality or use cases need to be covered
by a (potentially new) set of configuration options?

> - don't skip drivers/xen on make, as XEN might be selected via a
>   config item in that directory

I don't think this matters.

David

Re: PVHVM drivers in upstream linux kernel

[Juergen Gross]

On 12/02/2014 12:59 PM, David Vrabel wrote:
> On 02/12/14 11:33, Juergen Gross wrote:
>>
>> I think we have customers wanting to run a default kernel as domU. So it
>> isn't always the distro refusing paravirt, it might be the user...
>
> I don't think this is a sensible use case but I'm not adverse to
> improving the set of Xen config options.
>
>> To sort things out I'd suggest to:
>> - make XEN independent from PARAVIRT
>> - let XEN_DOM0 select XEN_BACKEND, PARAVIRT, XEN
>> - let XEN_BACKEND select PARAVIRT, XEN (I'd like to be able to build
>>    a driver domain without XEN_DOM0)
>> - introduce XEN_FRONTEND, let it select XEN
>> - let frontend drivers and drivers needed by those depend on
>>    XEN_FRONTEND
>> - let XEN_PVHVM select XEN_FRONTEND
>
> Rather than looking at the current set of configuration options, can you
> look at what user-visible functionality or use cases need to be covered
> by a (potentially new) set of configuration options?

I'd see:
- XEN_PV (selects PARAVIRT, XEN_FRONTEND): be able to run as pv-domain
   (x86 only)
- XEN_PVHVM (selects XEN_FRONTEND): be able to run as hvm-domain with
   pv-drivers
- XEN_BACKEND (selects PARAVIRT if x86): be able to run as driver domain
   (dom0 or other)
- XEN_DOM0 (selects PARAVIRT if x86, XEN_BACKEND): be able to run as
   dom0
- XEN_FRONTEND: be able to run as domU with pv-drivers


Juergen

Re: PVHVM drivers in upstream linux kernel

[David Vrabel]

On 02/12/14 13:00, Juergen Gross wrote:
> 
> I'd see:
> - XEN_PV (selects PARAVIRT, XEN_FRONTEND): be able to run as pv-domain
>   (x86 only)

Depends on PARAVIRT perhaps?

> - XEN_PVHVM (selects XEN_FRONTEND): be able to run as hvm-domain with
>   pv-drivers
> - XEN_BACKEND (selects PARAVIRT if x86): be able to run as driver domain
>   (dom0 or other)

Does not need to select PARAVIRT -- HVM domains can run backends.

> - XEN_DOM0 (selects PARAVIRT if x86, XEN_BACKEND): be able to run as
>   dom0

XEN_DOM0 depends on XEN_PV or XEN_PVH (if x86) and whatever ARM needs.

> - XEN_FRONTEND: be able to run as domU with pv-drivers

It may also be interesting to consider splitting the PV MMU stuff under
a PARAVIRT_MMU option.  This might address a reason why people want to
disable PARAVIRT completely.

David

Re: PVHVM drivers in upstream linux kernel

[Juergen Gross]

On 12/02/2014 03:09 PM, David Vrabel wrote:
> On 02/12/14 13:00, Juergen Gross wrote:
>>
>> I'd see:
>> - XEN_PV (selects PARAVIRT, XEN_FRONTEND): be able to run as pv-domain
>>    (x86 only)
>
> Depends on PARAVIRT perhaps?

Chicken and egg problem? :-)

I'd say select, as PARAVIRT isn't a primary function the user wants to
enable, it is a prerequisite for e.g. XEN_PV.

>
>> - XEN_PVHVM (selects XEN_FRONTEND): be able to run as hvm-domain with
>>    pv-drivers
>> - XEN_BACKEND (selects PARAVIRT if x86): be able to run as driver domain
>>    (dom0 or other)
>
> Does not need to select PARAVIRT -- HVM domains can run backends.

Okay.

>
>> - XEN_DOM0 (selects PARAVIRT if x86, XEN_BACKEND): be able to run as
>>    dom0
>
> XEN_DOM0 depends on XEN_PV or XEN_PVH (if x86) and whatever ARM needs.

I've removed XEN_PV as XEN_DOM0 shouldn't require XEN_FRONTEND.
We can add XEN_PARAVIRT instead which will be needed by XEN_DOM0 and
XEN_PV and will select PARAVIRT.

>
>> - XEN_FRONTEND: be able to run as domU with pv-drivers
>
> It may also be interesting to consider splitting the PV MMU stuff under
> a PARAVIRT_MMU option.  This might address a reason why people want to
> disable PARAVIRT completely.

Okay, seems sensible. Especially regarding XEN_PVH which I've omitted
here.

I'll try to assemble a complete config tree for review before starting
with patches. :-)


Juergen

Re: PVHVM drivers in upstream linux kernel

[Konrad Rzeszutek Wilk]

On Tue, Dec 02, 2014 at 10:39:59AM +0100, Juergen Gross wrote:
> Hi,
> 
> looking into the upstream linux sources I realized that the PVHVM
> drivers of XEN are only available with the pvops kernel. Is this on
> purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
> configurable without having to enable CONFIG_PARAVIRT?

Yes. For example on non-PAE kernels
> 
> Juergen
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: PVHVM drivers in upstream linux kernel

[Konrad Rzeszutek Wilk]

On Tue, Dec 02, 2014 at 11:05:14AM +0000, Ian Campbell wrote:
> On Tue, 2014-12-02 at 10:54 +0000, David Vrabel wrote:
> > On 02/12/14 09:39, Juergen Gross wrote:
> > > Hi,
> > > 
> > > looking into the upstream linux sources I realized that the PVHVM
> > > drivers of XEN are only available with the pvops kernel. Is this on
> > > purpose? Shouldn't the frontend drivers, xen/platform-pci.c etc. be
> > > configurable without having to enable CONFIG_PARAVIRT?
> > 
> > I suppose that would be possible but I don't think it's a useful
> > configuration because you would lose PV spinlocks for example.
> 
> IIRC the reason this hasn't been implemented until now is that
> refactoring would be required to the various bits of driver code which
> assumes PAE + PARAVIRT when they aren't strictly needed, e.g. grant
> table code. Whether its worth the churn at this stage is debatable, but
> I think the (in)ability to use PV spinlocks is a red-herring.
> 
> Adding PV drivers to an HVM guest is a useful thing to do, even without
> PV spinlocks. PV IO gets you far more incremental benefit than the locks
> do, adding PV IO paths is the number 1 thing which should be done to any
> guest.
> 
> One actual usecase is installing from a distro installer which isn't
> PAE, let alone PARAVIRT enabled[0], to get far enough that you can
> install a more capable PVHVM kernel with more bells and whistles.
> 
> If there were distros around who refused wholesale to enable PARAVIRT
> even in a non-default kernel then it would be more likely that they
> could be convinced to enable a set of PV IO drivers, since they have 0
> impact on a non-PARAVIRT system, and still give significant benefit to
> Xen users. I don't know of any of the major distros are refusing
> PARAVIRT in this way though.
> 
> Ian.
> 
> [0] The default i386 Debian installer falls into this camp, but you can
> use the special PV Xen variant to install as PVHVM too so it's not so
> critical.

And the Fedora 21 LiveISO (32-bit) does too.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: PVHVM drivers in upstream linux kernel

[Ian Campbell]

On Tue, 2014-12-02 at 10:11 -0500, Konrad Rzeszutek Wilk wrote:
> > [0] The default i386 Debian installer falls into this camp, but you can
> > use the special PV Xen variant to install as PVHVM too so it's not so
> > critical.
> 
> And the Fedora 21 LiveISO (32-bit) does too.

Interesting, I thought Fedora had switched to requiring PAE as a minimum
baseline everywhere 

Ian.

Re: PVHVM drivers in upstream linux kernel

[Konrad Rzeszutek Wilk]

On Tue, Dec 02, 2014 at 03:13:28PM +0000, Ian Campbell wrote:
> On Tue, 2014-12-02 at 10:11 -0500, Konrad Rzeszutek Wilk wrote:
> > > [0] The default i386 Debian installer falls into this camp, but you can
> > > use the special PV Xen variant to install as PVHVM too so it's not so
> > > critical.
> > 
> > And the Fedora 21 LiveISO (32-bit) does too.
> 
> Interesting, I thought Fedora had switched to requiring PAE as a minimum
> baseline everywhere 

It did, except for the GNOME based LiveCD. And oddly enough it will install
an PAE kernel (since the CPU is capable of it) . But of course it won't
bundle the Xen drivers in the initramfs, so when the new OS boots it
keels over as it unplugs the emulated ones.
> 
> Ian.
>