Xen's Linux kernel config options

Xen's Linux kernel config options

[Juergen Gross]

Hi,

This is a design proposal for a rework of the config options on the
Linux kernel which are related to Xen.

The need to do so arose from the fact that it is currently not
possible to build the Xen frontend drivers for a non-pvops kernel,
e.g. to run them in a HVM-domain. There are more drawbacks in the
current config options to which I'll come later.

Current Xen related config options are as follows:

Option                          Selects                 Depends
---------------------------------------------------------------------
XEN                             PARAVIRT_CLOCK(x86)     PARAVIRT(x86)
                                 XEN_HAVE_PVMMU(x86)
                                 SWIOTLB_XEN(arm,arm64)
   PCI_XEN(x86)                  SWIOTLB_XEN
   XEN_DOM0                                              PCI_XEN(x86)
     XEN_BACKEND
       XEN_BLKDEV_BACKEND
       XEN_PCIDEV_BACKEND(x86)
       XEN_SCSI_BACKEND
       XEN_NETDEV_BACKEND
     XEN_ACPI_HOTPLUG_MEMORY                             XEN_STUB
     XEN_ACPI_HOTPLUG_CPU                                XEN_STUB
     XEN_MCE_LOG(x86)
   XEN_PVHVM(x86)
     XEN_PVH(x86)
   XEN_MAX_DOMAIN_MEMORY(x86)
   XEN_SAVE_RESTORE(x86)
   XEN_DEBUG_FS(x86)
   XEN_FBDEV_FRONTEND            XEN_XENBUS_FRONTEND
                                 INPUT_XEN_KBDDEV_FRONTEND
   XEN_BLKDEV_FRONTEND           XEN_XENBUS_FRONTEND
   HVC_XEN
     HVC_XEN_FRONTEND            XEN_XENBUS_FRONTEND
   TCG_XEN                       XEN_XENBUS_FRONTEND
   XEN_PCIDEV_FRONTEND           PCI_XEN
                                 XEN_XENBUS_FRONTEND
   XEN_SCSI_FRONTEND             XEN_XENBUS_FRONTEND
   INPUT_XEN_KBDDEV_FRONTEND     XEN_XENBUS_FRONTEND
   XEN_WDT
   XEN_BALLOON
     XEN_SELFBALLOONING                                  XEN_TMEM
     XEN_BALLOON_MEMORY_HOTPLUG
     XEN_SCRUB_PAGES
   XEN_DEV_EVTCHN
   XENFS                         XEN_PRIVCMD
     XEN_COMPAT_XENFS
   XEN_SYS_HYPERVISOR
   XEN_XENBUS_FRONTEND
   XEN_GNTDEV
   XEN_GRANT_DEV_ALLOC
   SWIOTLB_XEN
   XEN_TMEM(x86)
   XEN_PRIVCMD
   XEN_STUB(x86_64)                                      BROKEN
   XEN_ACPI_PROCESSOR(x86)
   XEN_HAVE_PVMMU
   XEN_EFI(x64)
   XEN_NETDEV_FRONTEND           XEN_XENBUS_FRONTEND

Legend:
- The first column are the Xen config options. Indentation in this
   column reflects the dependency between those options (e.g.
   XEN_SCSI_BACKEND depends on XEN_BACKEND, which in turn depends on
   XEN_DOM0, which depends on XEN).
- The second column shows the options which are selected automatically
   if the corresponding option in the first column is configured.
- The last column shows additional dependencies which can't be shown via
   the indentation level of the first column.
- Some options or dependencies are architecture specific, they are
   listed in brackets (e.g. XEN_TMEM which is available for x86 only).
- Non-Xen options are mentioned only if they seem to be really relevant,
   like e.g. PARAVIRT or BROKEN.

There are several reasons to change the config options:
- Be able to build Xen frontend drivers for HVM domains without the need
   for choosing a pvops kernel: currently the frontend drivers need XEN
   configured which depends on PARAVIRT.
- Be able to build a Dom0 using XEN_PVH without having to configure
   XEN_HAVE_PVMMU.
- Be able to build HVM driver domains.
- Some features are available for x86 only, in spite of being not
   architecture specific, e.g. XEN_DEBUG_FS or XEN_TMEM.

As a first draft I'd suggest the following:

Option                          Selects                 Depends
----------------------------------------------------------------------
XEN                             SWIOTLB_XEN(arm,arm64)
   XEN_PV(x86)                   XEN_HAVE_PVMMU
                                 PARAVIRT
                                 PARAVIRT_CLOCK
   XEN_PVH(x86)                  XEN_PVHVM
                                 PARAVIRT
                                 PARAVIRT_CLOCK
   XEN_BACKEND                                           XEN_PV(x86) ||
                                                         XEN_PVH(x86) ||
                                                         XEN_PVHVM
     XEN_BLKDEV_BACKEND
     XEN_PCIDEV_BACKEND(x86)
     XEN_SCSI_BACKEND
     XEN_NETDEV_BACKEND
   PCI_XEN(x86)                  SWIOTLB_XEN
   XEN_DOM0                      XEN_BACKEND             XEN_PV(x86) ||
                                 PCI_XEN(x86)            XEN_PVH(x86)
     XEN_ACPI_HOTPLUG_MEMORY                             XEN_STUB
     XEN_ACPI_HOTPLUG_CPU                                XEN_STUB
     XEN_MCE_LOG(x86)
   XEN_MAX_DOMAIN_MEMORY(x86)
   XEN_SAVE_RESTORE(x86)
   XEN_DEBUG_FS
   XEN_WDT
   XEN_BALLOON
     XEN_SELFBALLOONING                                  XEN_TMEM
     XEN_BALLOON_MEMORY_HOTPLUG
     XEN_SCRUB_PAGES
   XENFS                         XEN_PRIVCMD
     XEN_COMPAT_XENFS
   XEN_SYS_HYPERVISOR
   XEN_DEV_EVTCHN
   XEN_GNTDEV
   XEN_GRANT_DEV_ALLOC
   SWIOTLB_XEN
   XEN_TMEM
   XEN_PRIVCMD
   XEN_STUB(x86_64)                                      BROKEN
   XEN_ACPI_PROCESSOR(x86)
   XEN_HAVE_PVMMU
   XEN_EFI(x64)
XEN_PVHVM
XEN_FRONTEND                                            XEN_PV ||
                                                         XEN_PVH ||
                                                         XEN_PVHVM
   XEN_XENBUS_FRONTEND
   XEN_FBDEV_FRONTEND            XEN_XENBUS_FRONTEND
                                 INPUT_XEN_KBDDEV_FRONTEND
   XEN_BLKDEV_FRONTEND           XEN_XENBUS_FRONTEND
   HVC_XEN_FRONTEND              XEN_XENBUS_FRONTEND
                                 HVC_XEN
   TCG_XEN                       XEN_XENBUS_FRONTEND
   XEN_PCIDEV_FRONTEND           PCI_XEN
                                 XEN_XENBUS_FRONTEND
   XEN_SCSI_FRONTEND             XEN_XENBUS_FRONTEND
   INPUT_XEN_KBDDEV_FRONTEND     XEN_XENBUS_FRONTEND
   XEN_NETDEV_FRONTEND           XEN_XENBUS_FRONTEND

There might be some further adjustments needed (should XEN_DEV_EVTCHN
and XEN_GNTDEV belong to XEN_BACKEND?), but these are minor nits. The
main difference to the current status is the XEN setting no longer
controlling all other options.

Changing the config options in this way surely will need some
adjustments in the drivers, but this can be discussed when we agree
on the future config dependencies.


Juergen

Re: Xen's Linux kernel config options

[Konrad Rzeszutek Wilk]

On Fri, Dec 12, 2014 at 02:17:27PM +0100, Juergen Gross wrote:
> Hi,
> 
> This is a design proposal for a rework of the config options on the
> Linux kernel which are related to Xen.
> 
> The need to do so arose from the fact that it is currently not
> possible to build the Xen frontend drivers for a non-pvops kernel,
> e.g. to run them in a HVM-domain. There are more drawbacks in the
> current config options to which I'll come later.
> 
> Current Xen related config options are as follows:
> 
> Option                          Selects                 Depends
> ---------------------------------------------------------------------
> XEN                             PARAVIRT_CLOCK(x86)     PARAVIRT(x86)
>                                 XEN_HAVE_PVMMU(x86)
>                                 SWIOTLB_XEN(arm,arm64)
>   PCI_XEN(x86)                  SWIOTLB_XEN
>   XEN_DOM0                                              PCI_XEN(x86)
>     XEN_BACKEND
>       XEN_BLKDEV_BACKEND
>       XEN_PCIDEV_BACKEND(x86)
>       XEN_SCSI_BACKEND
>       XEN_NETDEV_BACKEND
>     XEN_ACPI_HOTPLUG_MEMORY                             XEN_STUB
>     XEN_ACPI_HOTPLUG_CPU                                XEN_STUB
>     XEN_MCE_LOG(x86)
>   XEN_PVHVM(x86)
>     XEN_PVH(x86)
>   XEN_MAX_DOMAIN_MEMORY(x86)
>   XEN_SAVE_RESTORE(x86)
>   XEN_DEBUG_FS(x86)
>   XEN_FBDEV_FRONTEND            XEN_XENBUS_FRONTEND
>                                 INPUT_XEN_KBDDEV_FRONTEND
>   XEN_BLKDEV_FRONTEND           XEN_XENBUS_FRONTEND
>   HVC_XEN
>     HVC_XEN_FRONTEND            XEN_XENBUS_FRONTEND
>   TCG_XEN                       XEN_XENBUS_FRONTEND
>   XEN_PCIDEV_FRONTEND           PCI_XEN
>                                 XEN_XENBUS_FRONTEND
>   XEN_SCSI_FRONTEND             XEN_XENBUS_FRONTEND
>   INPUT_XEN_KBDDEV_FRONTEND     XEN_XENBUS_FRONTEND
>   XEN_WDT
>   XEN_BALLOON
>     XEN_SELFBALLOONING                                  XEN_TMEM
>     XEN_BALLOON_MEMORY_HOTPLUG
>     XEN_SCRUB_PAGES
>   XEN_DEV_EVTCHN
>   XENFS                         XEN_PRIVCMD
>     XEN_COMPAT_XENFS
>   XEN_SYS_HYPERVISOR
>   XEN_XENBUS_FRONTEND
>   XEN_GNTDEV
>   XEN_GRANT_DEV_ALLOC
>   SWIOTLB_XEN
>   XEN_TMEM(x86)
>   XEN_PRIVCMD
>   XEN_STUB(x86_64)                                      BROKEN
>   XEN_ACPI_PROCESSOR(x86)
>   XEN_HAVE_PVMMU
>   XEN_EFI(x64)
>   XEN_NETDEV_FRONTEND           XEN_XENBUS_FRONTEND
> 
> Legend:
> - The first column are the Xen config options. Indentation in this
>   column reflects the dependency between those options (e.g.
>   XEN_SCSI_BACKEND depends on XEN_BACKEND, which in turn depends on
>   XEN_DOM0, which depends on XEN).
> - The second column shows the options which are selected automatically
>   if the corresponding option in the first column is configured.
> - The last column shows additional dependencies which can't be shown via
>   the indentation level of the first column.
> - Some options or dependencies are architecture specific, they are
>   listed in brackets (e.g. XEN_TMEM which is available for x86 only).
> - Non-Xen options are mentioned only if they seem to be really relevant,
>   like e.g. PARAVIRT or BROKEN.
> 
> There are several reasons to change the config options:
> - Be able to build Xen frontend drivers for HVM domains without the need
>   for choosing a pvops kernel: currently the frontend drivers need XEN
>   configured which depends on PARAVIRT.
> - Be able to build a Dom0 using XEN_PVH without having to configure
>   XEN_HAVE_PVMMU.
> - Be able to build HVM driver domains.
> - Some features are available for x86 only, in spite of being not
>   architecture specific, e.g. XEN_DEBUG_FS or XEN_TMEM.
> 
> As a first draft I'd suggest the following:
> 
> Option                          Selects                 Depends
> ----------------------------------------------------------------------
> XEN                             SWIOTLB_XEN(arm,arm64)
>   XEN_PV(x86)                   XEN_HAVE_PVMMU
>                                 PARAVIRT
>                                 PARAVIRT_CLOCK
>   XEN_PVH(x86)                  XEN_PVHVM
>                                 PARAVIRT
>                                 PARAVIRT_CLOCK
>   XEN_BACKEND                                           XEN_PV(x86) ||
>                                                         XEN_PVH(x86) ||
>                                                         XEN_PVHVM
>     XEN_BLKDEV_BACKEND
>     XEN_PCIDEV_BACKEND(x86)
>     XEN_SCSI_BACKEND
>     XEN_NETDEV_BACKEND
>   PCI_XEN(x86)                  SWIOTLB_XEN
>   XEN_DOM0                      XEN_BACKEND             XEN_PV(x86) ||
>                                 PCI_XEN(x86)            XEN_PVH(x86)

Could XEN_DOM0 be removed completly? And instead we just have
an 'backend' and 'frontend' type options?


>     XEN_ACPI_HOTPLUG_MEMORY                             XEN_STUB
>     XEN_ACPI_HOTPLUG_CPU                                XEN_STUB
>     XEN_MCE_LOG(x86)
>   XEN_MAX_DOMAIN_MEMORY(x86)
>   XEN_SAVE_RESTORE(x86)
>   XEN_DEBUG_FS
>   XEN_WDT
>   XEN_BALLOON
>     XEN_SELFBALLOONING                                  XEN_TMEM
>     XEN_BALLOON_MEMORY_HOTPLUG
>     XEN_SCRUB_PAGES
>   XENFS                         XEN_PRIVCMD
>     XEN_COMPAT_XENFS
>   XEN_SYS_HYPERVISOR
>   XEN_DEV_EVTCHN
>   XEN_GNTDEV
>   XEN_GRANT_DEV_ALLOC
>   SWIOTLB_XEN
>   XEN_TMEM
>   XEN_PRIVCMD
>   XEN_STUB(x86_64)                                      BROKEN
>   XEN_ACPI_PROCESSOR(x86)
>   XEN_HAVE_PVMMU
>   XEN_EFI(x64)
> XEN_PVHVM
> XEN_FRONTEND                                            XEN_PV ||
>                                                         XEN_PVH ||
>                                                         XEN_PVHVM
>   XEN_XENBUS_FRONTEND
>   XEN_FBDEV_FRONTEND            XEN_XENBUS_FRONTEND
>                                 INPUT_XEN_KBDDEV_FRONTEND
>   XEN_BLKDEV_FRONTEND           XEN_XENBUS_FRONTEND
>   HVC_XEN_FRONTEND              XEN_XENBUS_FRONTEND
>                                 HVC_XEN
>   TCG_XEN                       XEN_XENBUS_FRONTEND
>   XEN_PCIDEV_FRONTEND           PCI_XEN
>                                 XEN_XENBUS_FRONTEND
>   XEN_SCSI_FRONTEND             XEN_XENBUS_FRONTEND
>   INPUT_XEN_KBDDEV_FRONTEND     XEN_XENBUS_FRONTEND
>   XEN_NETDEV_FRONTEND           XEN_XENBUS_FRONTEND
> 
> There might be some further adjustments needed (should XEN_DEV_EVTCHN
> and XEN_GNTDEV belong to XEN_BACKEND?), but these are minor nits. The
> main difference to the current status is the XEN setting no longer
> controlling all other options.
> 
> Changing the config options in this way surely will need some
> adjustments in the drivers, but this can be discussed when we agree
> on the future config dependencies.
> 
> 
> Juergen

Re: Xen's Linux kernel config options

[David Vrabel]

On 12/12/14 13:17, Juergen Gross wrote:
> 
> 
> As a first draft I'd suggest the following:

Some rough thoughts below.

> Option                          Selects                 Depends
> ----------------------------------------------------------------------
> XEN                             SWIOTLB_XEN(arm,arm64)

XEN should get you basic support for making hypercalls and allowing
guest to identify themselves as running on Xen.  I don't think it should
select SWIOTLB_XEN even on ARM as it is only needed for guests with
access to real hardware.

>   XEN_PV(x86)                   XEN_HAVE_PVMMU
>                                 PARAVIRT
>                                 PARAVIRT_CLOCK
>   XEN_PVH(x86)                  XEN_PVHVM
>                                 PARAVIRT
>                                 PARAVIRT_CLOCK
>   XEN_BACKEND                                           XEN_PV(x86) ||
>                                                         XEN_PVH(x86) ||
>                                                         XEN_PVHVM
>     XEN_BLKDEV_BACKEND
>     XEN_PCIDEV_BACKEND(x86)
>     XEN_SCSI_BACKEND
>     XEN_NETDEV_BACKEND
[...]
> XEN_PVHVM

Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.

> XEN_FRONTEND                                            XEN_PV ||
>                                                         XEN_PVH ||
>                                                         XEN_PVHVM

This enables all the basic infrastructure for frontends: event channels,
grant tables and Xenbus.

Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
possible to have frontend drivers without support for any of the
PV/PVHVM/PVH guest types.  Frontends only need event channels, grant
table and xenbus.

Perhaps have XEN_FRONTEND select XEN instead?

You need an additional option to enable the Xen platform PCI device.
This should depend on XEN_FRONTEND.

>   XEN_XENBUS_FRONTEND

Fold this into XEN_FRONTEND?

>   XEN_FBDEV_FRONTEND            XEN_XENBUS_FRONTEND
>                                 INPUT_XEN_KBDDEV_FRONTEND
>   XEN_BLKDEV_FRONTEND           XEN_XENBUS_FRONTEND
>   HVC_XEN_FRONTEND              XEN_XENBUS_FRONTEND
>                                 HVC_XEN
>   TCG_XEN                       XEN_XENBUS_FRONTEND
>   XEN_PCIDEV_FRONTEND           PCI_XEN
>                                 XEN_XENBUS_FRONTEND
>   XEN_SCSI_FRONTEND             XEN_XENBUS_FRONTEND
>   INPUT_XEN_KBDDEV_FRONTEND     XEN_XENBUS_FRONTEND
>   XEN_NETDEV_FRONTEND           XEN_XENBUS_FRONTEND
> 
> There might be some further adjustments needed (should XEN_DEV_EVTCHN
> and XEN_GNTDEV belong to XEN_BACKEND?), but these are minor nits. The
> main difference to the current status is the XEN setting no longer
> controlling all other options.

XEN_DEV_EVTCH and XEN_GNTDEV are useful to non-backends (e.g., for
interdomain comms).

David

Re: Xen's Linux kernel config options

[Juergen Gross]

On 12/12/2014 05:44 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Dec 12, 2014 at 02:17:27PM +0100, Juergen Gross wrote:
>> Hi,
>>
>> This is a design proposal for a rework of the config options on the
>> Linux kernel which are related to Xen.
>>
>> The need to do so arose from the fact that it is currently not
>> possible to build the Xen frontend drivers for a non-pvops kernel,
>> e.g. to run them in a HVM-domain. There are more drawbacks in the
>> current config options to which I'll come later.
>>
>> Current Xen related config options are as follows:
>>
>> Option                          Selects                 Depends
>> ---------------------------------------------------------------------
>> XEN                             PARAVIRT_CLOCK(x86)     PARAVIRT(x86)
>>                                  XEN_HAVE_PVMMU(x86)
>>                                  SWIOTLB_XEN(arm,arm64)
>>    PCI_XEN(x86)                  SWIOTLB_XEN
>>    XEN_DOM0                                              PCI_XEN(x86)
>>      XEN_BACKEND
>>        XEN_BLKDEV_BACKEND
>>        XEN_PCIDEV_BACKEND(x86)
>>        XEN_SCSI_BACKEND
>>        XEN_NETDEV_BACKEND
>>      XEN_ACPI_HOTPLUG_MEMORY                             XEN_STUB
>>      XEN_ACPI_HOTPLUG_CPU                                XEN_STUB
>>      XEN_MCE_LOG(x86)
>>    XEN_PVHVM(x86)
>>      XEN_PVH(x86)
>>    XEN_MAX_DOMAIN_MEMORY(x86)
>>    XEN_SAVE_RESTORE(x86)
>>    XEN_DEBUG_FS(x86)
>>    XEN_FBDEV_FRONTEND            XEN_XENBUS_FRONTEND
>>                                  INPUT_XEN_KBDDEV_FRONTEND
>>    XEN_BLKDEV_FRONTEND           XEN_XENBUS_FRONTEND
>>    HVC_XEN
>>      HVC_XEN_FRONTEND            XEN_XENBUS_FRONTEND
>>    TCG_XEN                       XEN_XENBUS_FRONTEND
>>    XEN_PCIDEV_FRONTEND           PCI_XEN
>>                                  XEN_XENBUS_FRONTEND
>>    XEN_SCSI_FRONTEND             XEN_XENBUS_FRONTEND
>>    INPUT_XEN_KBDDEV_FRONTEND     XEN_XENBUS_FRONTEND
>>    XEN_WDT
>>    XEN_BALLOON
>>      XEN_SELFBALLOONING                                  XEN_TMEM
>>      XEN_BALLOON_MEMORY_HOTPLUG
>>      XEN_SCRUB_PAGES
>>    XEN_DEV_EVTCHN
>>    XENFS                         XEN_PRIVCMD
>>      XEN_COMPAT_XENFS
>>    XEN_SYS_HYPERVISOR
>>    XEN_XENBUS_FRONTEND
>>    XEN_GNTDEV
>>    XEN_GRANT_DEV_ALLOC
>>    SWIOTLB_XEN
>>    XEN_TMEM(x86)
>>    XEN_PRIVCMD
>>    XEN_STUB(x86_64)                                      BROKEN
>>    XEN_ACPI_PROCESSOR(x86)
>>    XEN_HAVE_PVMMU
>>    XEN_EFI(x64)
>>    XEN_NETDEV_FRONTEND           XEN_XENBUS_FRONTEND
>>
>> Legend:
>> - The first column are the Xen config options. Indentation in this
>>    column reflects the dependency between those options (e.g.
>>    XEN_SCSI_BACKEND depends on XEN_BACKEND, which in turn depends on
>>    XEN_DOM0, which depends on XEN).
>> - The second column shows the options which are selected automatically
>>    if the corresponding option in the first column is configured.
>> - The last column shows additional dependencies which can't be shown via
>>    the indentation level of the first column.
>> - Some options or dependencies are architecture specific, they are
>>    listed in brackets (e.g. XEN_TMEM which is available for x86 only).
>> - Non-Xen options are mentioned only if they seem to be really relevant,
>>    like e.g. PARAVIRT or BROKEN.
>>
>> There are several reasons to change the config options:
>> - Be able to build Xen frontend drivers for HVM domains without the need
>>    for choosing a pvops kernel: currently the frontend drivers need XEN
>>    configured which depends on PARAVIRT.
>> - Be able to build a Dom0 using XEN_PVH without having to configure
>>    XEN_HAVE_PVMMU.
>> - Be able to build HVM driver domains.
>> - Some features are available for x86 only, in spite of being not
>>    architecture specific, e.g. XEN_DEBUG_FS or XEN_TMEM.
>>
>> As a first draft I'd suggest the following:
>>
>> Option                          Selects                 Depends
>> ----------------------------------------------------------------------
>> XEN                             SWIOTLB_XEN(arm,arm64)
>>    XEN_PV(x86)                   XEN_HAVE_PVMMU
>>                                  PARAVIRT
>>                                  PARAVIRT_CLOCK
>>    XEN_PVH(x86)                  XEN_PVHVM
>>                                  PARAVIRT
>>                                  PARAVIRT_CLOCK
>>    XEN_BACKEND                                           XEN_PV(x86) ||
>>                                                          XEN_PVH(x86) ||
>>                                                          XEN_PVHVM
>>      XEN_BLKDEV_BACKEND
>>      XEN_PCIDEV_BACKEND(x86)
>>      XEN_SCSI_BACKEND
>>      XEN_NETDEV_BACKEND
>>    PCI_XEN(x86)                  SWIOTLB_XEN
>>    XEN_DOM0                      XEN_BACKEND             XEN_PV(x86) ||
>>                                  PCI_XEN(x86)            XEN_PVH(x86)
>
> Could XEN_DOM0 be removed completly? And instead we just have
> an 'backend' and 'frontend' type options?

What about (physical) memory/cpu hotplug and mce?

We could rename XEN_DOM0 to XEN_HWDOM, however.

>
>
>>      XEN_ACPI_HOTPLUG_MEMORY                             XEN_STUB
>>      XEN_ACPI_HOTPLUG_CPU                                XEN_STUB
>>      XEN_MCE_LOG(x86)
>>    XEN_MAX_DOMAIN_MEMORY(x86)
>>    XEN_SAVE_RESTORE(x86)
>>    XEN_DEBUG_FS
>>    XEN_WDT
>>    XEN_BALLOON
>>      XEN_SELFBALLOONING                                  XEN_TMEM
>>      XEN_BALLOON_MEMORY_HOTPLUG
>>      XEN_SCRUB_PAGES
>>    XENFS                         XEN_PRIVCMD
>>      XEN_COMPAT_XENFS
>>    XEN_SYS_HYPERVISOR
>>    XEN_DEV_EVTCHN
>>    XEN_GNTDEV
>>    XEN_GRANT_DEV_ALLOC
>>    SWIOTLB_XEN
>>    XEN_TMEM
>>    XEN_PRIVCMD
>>    XEN_STUB(x86_64)                                      BROKEN
>>    XEN_ACPI_PROCESSOR(x86)
>>    XEN_HAVE_PVMMU
>>    XEN_EFI(x64)
>> XEN_PVHVM
>> XEN_FRONTEND                                            XEN_PV ||
>>                                                          XEN_PVH ||
>>                                                          XEN_PVHVM
>>    XEN_XENBUS_FRONTEND
>>    XEN_FBDEV_FRONTEND            XEN_XENBUS_FRONTEND
>>                                  INPUT_XEN_KBDDEV_FRONTEND
>>    XEN_BLKDEV_FRONTEND           XEN_XENBUS_FRONTEND
>>    HVC_XEN_FRONTEND              XEN_XENBUS_FRONTEND
>>                                  HVC_XEN
>>    TCG_XEN                       XEN_XENBUS_FRONTEND
>>    XEN_PCIDEV_FRONTEND           PCI_XEN
>>                                  XEN_XENBUS_FRONTEND
>>    XEN_SCSI_FRONTEND             XEN_XENBUS_FRONTEND
>>    INPUT_XEN_KBDDEV_FRONTEND     XEN_XENBUS_FRONTEND
>>    XEN_NETDEV_FRONTEND           XEN_XENBUS_FRONTEND
>>
>> There might be some further adjustments needed (should XEN_DEV_EVTCHN
>> and XEN_GNTDEV belong to XEN_BACKEND?), but these are minor nits. The
>> main difference to the current status is the XEN setting no longer
>> controlling all other options.
>>
>> Changing the config options in this way surely will need some
>> adjustments in the drivers, but this can be discussed when we agree
>> on the future config dependencies.
>>
>>
>> Juergen
>

Re: Xen's Linux kernel config options

[Juergen Gross]

On 12/12/2014 06:29 PM, David Vrabel wrote:
> On 12/12/14 13:17, Juergen Gross wrote:
>>
>>
>> As a first draft I'd suggest the following:
>
> Some rough thoughts below.
>
>> Option                          Selects                 Depends
>> ----------------------------------------------------------------------
>> XEN                             SWIOTLB_XEN(arm,arm64)
>
> XEN should get you basic support for making hypercalls and allowing
> guest to identify themselves as running on Xen.  I don't think it should
> select SWIOTLB_XEN even on ARM as it is only needed for guests with
> access to real hardware.

Okay.

>
>>    XEN_PV(x86)                   XEN_HAVE_PVMMU
>>                                  PARAVIRT
>>                                  PARAVIRT_CLOCK
>>    XEN_PVH(x86)                  XEN_PVHVM
>>                                  PARAVIRT
>>                                  PARAVIRT_CLOCK
>>    XEN_BACKEND                                           XEN_PV(x86) ||
>>                                                          XEN_PVH(x86) ||
>>                                                          XEN_PVHVM
>>      XEN_BLKDEV_BACKEND
>>      XEN_PCIDEV_BACKEND(x86)
>>      XEN_SCSI_BACKEND
>>      XEN_NETDEV_BACKEND
> [...]
>> XEN_PVHVM
>
> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.

Okay.

>
>> XEN_FRONTEND                                            XEN_PV ||
>>                                                          XEN_PVH ||
>>                                                          XEN_PVHVM
>
> This enables all the basic infrastructure for frontends: event channels,
> grant tables and Xenbus.
>
> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> possible to have frontend drivers without support for any of the
> PV/PVHVM/PVH guest types.  Frontends only need event channels, grant
> table and xenbus.

Okay.

> Perhaps have XEN_FRONTEND select XEN instead?

Hmm, with XEN not selecting anything else automatically (apart from
stuff needed by frontends AND backends) this would be okay, I think.

> You need an additional option to enable the Xen platform PCI device.
> This should depend on XEN_FRONTEND.

Yep.

>
>>    XEN_XENBUS_FRONTEND
>
> Fold this into XEN_FRONTEND?

Hmm, what about Xenstore in an own domain? I'd say XEN_FRONTEND should
select XEN_XENBUS_FRONTEND, but XEN_XENBUS_FRONTEND should be available
to non-frontend domains as well.

>
>>    XEN_FBDEV_FRONTEND            XEN_XENBUS_FRONTEND
>>                                  INPUT_XEN_KBDDEV_FRONTEND
>>    XEN_BLKDEV_FRONTEND           XEN_XENBUS_FRONTEND
>>    HVC_XEN_FRONTEND              XEN_XENBUS_FRONTEND
>>                                  HVC_XEN
>>    TCG_XEN                       XEN_XENBUS_FRONTEND
>>    XEN_PCIDEV_FRONTEND           PCI_XEN
>>                                  XEN_XENBUS_FRONTEND
>>    XEN_SCSI_FRONTEND             XEN_XENBUS_FRONTEND
>>    INPUT_XEN_KBDDEV_FRONTEND     XEN_XENBUS_FRONTEND
>>    XEN_NETDEV_FRONTEND           XEN_XENBUS_FRONTEND
>>
>> There might be some further adjustments needed (should XEN_DEV_EVTCHN
>> and XEN_GNTDEV belong to XEN_BACKEND?), but these are minor nits. The
>> main difference to the current status is the XEN setting no longer
>> controlling all other options.
>
> XEN_DEV_EVTCH and XEN_GNTDEV are useful to non-backends (e.g., for
> interdomain comms).

Okay.


Juergen

Re: Xen's Linux kernel config options

[Stefano Stabellini]

On Fri, 12 Dec 2014, David Vrabel wrote:
> On 12/12/14 13:17, Juergen Gross wrote:
> > 
> > 
> > As a first draft I'd suggest the following:
> 
> Some rough thoughts below.
> 
> > Option                          Selects                 Depends
> > ----------------------------------------------------------------------
> > XEN                             SWIOTLB_XEN(arm,arm64)
> 
> XEN should get you basic support for making hypercalls and allowing
> guest to identify themselves as running on Xen.  I don't think it should
> select SWIOTLB_XEN even on ARM as it is only needed for guests with
> access to real hardware.

In that case XEN_BACKEND should select SWIOTLB_XEN.

Re: Xen's Linux kernel config options

[Luis R. Rodriguez]

On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> On 12/12/14 13:17, Juergen Gross wrote:
>> XEN_PVHVM
>
> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.

FWIW, although it seems we do not want to let users just build
XEN_PVHVM hypervisors I have the changes required now to at least get
this to build so I do know what it takes.

>> XEN_FRONTEND                                            XEN_PV ||
>>                                                         XEN_PVH ||
>>                                                         XEN_PVHVM
>
> This enables all the basic infrastructure for frontends: event channels,
> grant tables and Xenbus.
>
> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> possible to have frontend drivers without support for any of the
> PV/PVHVM/PVH guest types.

David, can you elaborate on the type of Xen guest it would be on x86
its not PV, PVHVM, or PVH? I'm particularly curious about the
xen_domain_type and how it would end up to selected. As it is we tie
in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
completely removed from XEN_PVH we need quite a bit of code changes
which at least as code exercise I have completed already. If we want
at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
XEN_PVH are not available we need a bit more work.

>  Frontends only need event channels, grant
> table and xenbus.

Well xenbus_probe_initcall() will check for xen_domain() and that
won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
XEN_PVH set -- to start off with. Then
drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
bit of places as well, that won't be set unless xen_setup_features()
is called which right now is only done on x86 arch/x86/xen/enlighten.c
which as Juergen pointed out, is not needed if you don't have XEN_PV
or XEN_PVH. As it turns out this is incorrect though, its needed for
XEN_PVHVM as well and my split exercise in code addresses this. Now,
at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
don't call xen_setup_features() and its unclear to me where or how
that should happen in other cases.

> Perhaps have XEN_FRONTEND select XEN instead?

Right now if you enable CONFIG_XEN xen-head.S brings in and assumes a
big tamale of guest support on x86, there are quite a bit of other
code that also relies on CONFIG_XEN for similar purposes, and trying
to remove out XEN_FRONTEND dependency from XEN_PV, XEN_PVHVM, XEN_PVH
requires quite a bit work, most of which I think I've done, the only
puzzle remaining to me at least is what we want to do for the setup
for non XEN_PV, XEN_PVHVM, XEN_PVH Linux systems.

  Luis

Re: Xen's Linux kernel config options

[Luis R. Rodriguez]

On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
>> On 12/12/14 13:17, Juergen Gross wrote:
>>> XEN_PVHVM
>>
>> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
>
> FWIW, although it seems we do not want to let users just build
> XEN_PVHVM hypervisors I have the changes required now to at least get
> this to build so I do know what it takes.
>
>>> XEN_FRONTEND                                            XEN_PV ||
>>>                                                         XEN_PVH ||
>>>                                                         XEN_PVHVM
>>
>> This enables all the basic infrastructure for frontends: event channels,
>> grant tables and Xenbus.
>>
>> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
>> possible to have frontend drivers without support for any of the
>> PV/PVHVM/PVH guest types.
>
> David, can you elaborate on the type of Xen guest it would be on x86
> its not PV, PVHVM, or PVH? I'm particularly curious about the
> xen_domain_type and how it would end up to selected. As it is we tie
> in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> completely removed from XEN_PVH we need quite a bit of code changes
> which at least as code exercise I have completed already. If we want
> at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> XEN_PVH are not available we need a bit more work.

OK I think I see the issue. We have nothing quite like
xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
think I can that close the gap I'm observing.

>>  Frontends only need event channels, grant
>> table and xenbus.
>
> Well xenbus_probe_initcall() will check for xen_domain() and that
> won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> XEN_PVH set -- to start off with. Then
> drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> bit of places as well, that won't be set unless xen_setup_features()
> is called which right now is only done on x86 arch/x86/xen/enlighten.c
> which as Juergen pointed out, is not needed if you don't have XEN_PV
> or XEN_PVH. As it turns out this is incorrect though, its needed for
> XEN_PVHVM as well and my split exercise in code addresses this. Now,
> at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> don't call xen_setup_features() and its unclear to me where or how
> that should happen in other cases.

Yeah I think having an x86 equivalent of xen_guest_init() would solve
this, Stefano, thoughts?

>> Perhaps have XEN_FRONTEND select XEN instead?
>
> Right now if you enable CONFIG_XEN xen-head.S brings in and assumes a
> big tamale of guest support on x86, there are quite a bit of other
> code that also relies on CONFIG_XEN for similar purposes, and trying
> to remove out XEN_FRONTEND dependency from XEN_PV, XEN_PVHVM, XEN_PVH
> requires quite a bit work, most of which I think I've done, the only
> puzzle remaining to me at least is what we want to do for the setup
> for non XEN_PV, XEN_PVHVM, XEN_PVH Linux systems.

And with a xen_guest_init() this should still need to be resolved, and
I think I have this mostly addressed already in my dev branch, just
have to clean it up.

 Luis

Re: Xen's Linux kernel config options

[Stefano Stabellini]

On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> >> On 12/12/14 13:17, Juergen Gross wrote:
> >>> XEN_PVHVM
> >>
> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
> >
> > FWIW, although it seems we do not want to let users just build
> > XEN_PVHVM hypervisors I have the changes required now to at least get
> > this to build so I do know what it takes.
> >
> >>> XEN_FRONTEND                                            XEN_PV ||
> >>>                                                         XEN_PVH ||
> >>>                                                         XEN_PVHVM
> >>
> >> This enables all the basic infrastructure for frontends: event channels,
> >> grant tables and Xenbus.
> >>
> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> >> possible to have frontend drivers without support for any of the
> >> PV/PVHVM/PVH guest types.
> >
> > David, can you elaborate on the type of Xen guest it would be on x86
> > its not PV, PVHVM, or PVH? I'm particularly curious about the
> > xen_domain_type and how it would end up to selected. As it is we tie
> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> > completely removed from XEN_PVH we need quite a bit of code changes
> > which at least as code exercise I have completed already. If we want
> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> > XEN_PVH are not available we need a bit more work.
> 
> OK I think I see the issue. We have nothing quite like
> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
> think I can that close the gap I'm observing.
> 
> >>  Frontends only need event channels, grant
> >> table and xenbus.
> >
> > Well xenbus_probe_initcall() will check for xen_domain() and that
> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> > XEN_PVH set -- to start off with. Then
> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> > bit of places as well, that won't be set unless xen_setup_features()
> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
> > which as Juergen pointed out, is not needed if you don't have XEN_PV
> > or XEN_PVH. As it turns out this is incorrect though, its needed for
> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> > don't call xen_setup_features() and its unclear to me where or how
> > that should happen in other cases.
> 
> Yeah I think having an x86 equivalent of xen_guest_init() would solve
> this, Stefano, thoughts?

Having xen_guest_init() on x86 would be nice.  Being able to set
xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
of XEN_PV/PVH/PVHVM also makes sense from Linux POV.

That said, I don't see much value in removing XEN_PVHVM: why are we even
doing this? What is the improvement we are seeking?


> >> Perhaps have XEN_FRONTEND select XEN instead?
> >
> > Right now if you enable CONFIG_XEN xen-head.S brings in and assumes a
> > big tamale of guest support on x86, there are quite a bit of other
> > code that also relies on CONFIG_XEN for similar purposes, and trying
> > to remove out XEN_FRONTEND dependency from XEN_PV, XEN_PVHVM, XEN_PVH
> > requires quite a bit work, most of which I think I've done, the only
> > puzzle remaining to me at least is what we want to do for the setup
> > for non XEN_PV, XEN_PVHVM, XEN_PVH Linux systems.
> 
> And with a xen_guest_init() this should still need to be resolved, and
> I think I have this mostly addressed already in my dev branch, just
> have to clean it up.
> 
>  Luis
> 

Re: Xen's Linux kernel config options

[Luis R. Rodriguez]

On Tue, Feb 24, 2015 at 7:21 AM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
>> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
>> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
>> >> On 12/12/14 13:17, Juergen Gross wrote:
>> >>> XEN_PVHVM
>> >>
>> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
>> >
>> > FWIW, although it seems we do not want to let users just build
>> > XEN_PVHVM hypervisors I have the changes required now to at least get
>> > this to build so I do know what it takes.
>> >
>> >>> XEN_FRONTEND                                            XEN_PV ||
>> >>>                                                         XEN_PVH ||
>> >>>                                                         XEN_PVHVM
>> >>
>> >> This enables all the basic infrastructure for frontends: event channels,
>> >> grant tables and Xenbus.
>> >>
>> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
>> >> possible to have frontend drivers without support for any of the
>> >> PV/PVHVM/PVH guest types.
>> >
>> > David, can you elaborate on the type of Xen guest it would be on x86
>> > its not PV, PVHVM, or PVH? I'm particularly curious about the
>> > xen_domain_type and how it would end up to selected. As it is we tie
>> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
>> > completely removed from XEN_PVH we need quite a bit of code changes
>> > which at least as code exercise I have completed already. If we want
>> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
>> > XEN_PVH are not available we need a bit more work.
>>
>> OK I think I see the issue. We have nothing quite like
>> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
>> think I can that close the gap I'm observing.
>>
>> >>  Frontends only need event channels, grant
>> >> table and xenbus.
>> >
>> > Well xenbus_probe_initcall() will check for xen_domain() and that
>> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
>> > XEN_PVH set -- to start off with. Then
>> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
>> > bit of places as well, that won't be set unless xen_setup_features()
>> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
>> > which as Juergen pointed out, is not needed if you don't have XEN_PV
>> > or XEN_PVH. As it turns out this is incorrect though, its needed for
>> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
>> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
>> > don't call xen_setup_features() and its unclear to me where or how
>> > that should happen in other cases.
>>
>> Yeah I think having an x86 equivalent of xen_guest_init() would solve
>> this, Stefano, thoughts?
>
> Having xen_guest_init() on x86 would be nice.  Being able to set
> xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
> of XEN_PV/PVH/PVHVM also makes sense from Linux POV.

OK great, thanks for the feedback.

> That said, I don't see much value in removing XEN_PVHVM: why are we even
> doing this? What is the improvement we are seeking?

We would not, the above discussed about the possibility of letting
users enable XEN_PVHVM without XEN_PVH, that's all. As is the only
thing that can enable XEN_PVHVM is if you enable XEN_PVH. If we want
xen_guest_init() alone though we might need the decoupling though at
least at build time so that if XEN_PV or XEN_PVH is not selected we'd
at least have XEN_PVHVM. Thoughts?

 Luis

Re: Xen's Linux kernel config options

[Stefano Stabellini]

On Tue, 24 Feb 2015, Luis R. Rodriguez wrote:
> On Tue, Feb 24, 2015 at 7:21 AM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
> >> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> >> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> >> >> On 12/12/14 13:17, Juergen Gross wrote:
> >> >>> XEN_PVHVM
> >> >>
> >> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
> >> >
> >> > FWIW, although it seems we do not want to let users just build
> >> > XEN_PVHVM hypervisors I have the changes required now to at least get
> >> > this to build so I do know what it takes.
> >> >
> >> >>> XEN_FRONTEND                                            XEN_PV ||
> >> >>>                                                         XEN_PVH ||
> >> >>>                                                         XEN_PVHVM
> >> >>
> >> >> This enables all the basic infrastructure for frontends: event channels,
> >> >> grant tables and Xenbus.
> >> >>
> >> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> >> >> possible to have frontend drivers without support for any of the
> >> >> PV/PVHVM/PVH guest types.
> >> >
> >> > David, can you elaborate on the type of Xen guest it would be on x86
> >> > its not PV, PVHVM, or PVH? I'm particularly curious about the
> >> > xen_domain_type and how it would end up to selected. As it is we tie
> >> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> >> > completely removed from XEN_PVH we need quite a bit of code changes
> >> > which at least as code exercise I have completed already. If we want
> >> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> >> > XEN_PVH are not available we need a bit more work.
> >>
> >> OK I think I see the issue. We have nothing quite like
> >> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
> >> think I can that close the gap I'm observing.
> >>
> >> >>  Frontends only need event channels, grant
> >> >> table and xenbus.
> >> >
> >> > Well xenbus_probe_initcall() will check for xen_domain() and that
> >> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> >> > XEN_PVH set -- to start off with. Then
> >> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> >> > bit of places as well, that won't be set unless xen_setup_features()
> >> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
> >> > which as Juergen pointed out, is not needed if you don't have XEN_PV
> >> > or XEN_PVH. As it turns out this is incorrect though, its needed for
> >> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
> >> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> >> > don't call xen_setup_features() and its unclear to me where or how
> >> > that should happen in other cases.
> >>
> >> Yeah I think having an x86 equivalent of xen_guest_init() would solve
> >> this, Stefano, thoughts?
> >
> > Having xen_guest_init() on x86 would be nice.  Being able to set
> > xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
> > of XEN_PV/PVH/PVHVM also makes sense from Linux POV.
> 
> OK great, thanks for the feedback.
> 
> > That said, I don't see much value in removing XEN_PVHVM: why are we even
> > doing this? What is the improvement we are seeking?
> 
> We would not, the above discussed about the possibility of letting
> users enable XEN_PVHVM without XEN_PVH, that's all.

OK, that makes sense.


> As is the only thing that can enable XEN_PVHVM is if you enable
> XEN_PVH.

This is the bit that we need to change but it shouldn't be difficult.


> If we want
> xen_guest_init() alone though we might need the decoupling though at
> least at build time so that if XEN_PV or XEN_PVH is not selected we'd
> at least have XEN_PVHVM. Thoughts?

Today pv(h) and pvhvm have very different boot paths already: pv and pvh
initialize via xen_start_kernel while pvhvm via xen_hvm_guest_init.
They are already very decoupled. You just need to make sure that
init_hvm_pv_info() is called regardless on any XEN_PV/PVH/PVHVM configs.

Re: Xen's Linux kernel config options

[Stefano Stabellini]

On Wed, 25 Feb 2015, Stefano Stabellini wrote:
> On Tue, 24 Feb 2015, Luis R. Rodriguez wrote:
> > On Tue, Feb 24, 2015 at 7:21 AM, Stefano Stabellini
> > <stefano.stabellini@eu.citrix.com> wrote:
> > > On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
> > >> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > >> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> > >> >> On 12/12/14 13:17, Juergen Gross wrote:
> > >> >>> XEN_PVHVM
> > >> >>
> > >> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
> > >> >
> > >> > FWIW, although it seems we do not want to let users just build
> > >> > XEN_PVHVM hypervisors I have the changes required now to at least get
> > >> > this to build so I do know what it takes.
> > >> >
> > >> >>> XEN_FRONTEND                                            XEN_PV ||
> > >> >>>                                                         XEN_PVH ||
> > >> >>>                                                         XEN_PVHVM
> > >> >>
> > >> >> This enables all the basic infrastructure for frontends: event channels,
> > >> >> grant tables and Xenbus.
> > >> >>
> > >> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> > >> >> possible to have frontend drivers without support for any of the
> > >> >> PV/PVHVM/PVH guest types.
> > >> >
> > >> > David, can you elaborate on the type of Xen guest it would be on x86
> > >> > its not PV, PVHVM, or PVH? I'm particularly curious about the
> > >> > xen_domain_type and how it would end up to selected. As it is we tie
> > >> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> > >> > completely removed from XEN_PVH we need quite a bit of code changes
> > >> > which at least as code exercise I have completed already. If we want
> > >> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> > >> > XEN_PVH are not available we need a bit more work.
> > >>
> > >> OK I think I see the issue. We have nothing quite like
> > >> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
> > >> think I can that close the gap I'm observing.
> > >>
> > >> >>  Frontends only need event channels, grant
> > >> >> table and xenbus.
> > >> >
> > >> > Well xenbus_probe_initcall() will check for xen_domain() and that
> > >> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> > >> > XEN_PVH set -- to start off with. Then
> > >> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> > >> > bit of places as well, that won't be set unless xen_setup_features()
> > >> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
> > >> > which as Juergen pointed out, is not needed if you don't have XEN_PV
> > >> > or XEN_PVH. As it turns out this is incorrect though, its needed for
> > >> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
> > >> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> > >> > don't call xen_setup_features() and its unclear to me where or how
> > >> > that should happen in other cases.
> > >>
> > >> Yeah I think having an x86 equivalent of xen_guest_init() would solve
> > >> this, Stefano, thoughts?
> > >
> > > Having xen_guest_init() on x86 would be nice.  Being able to set
> > > xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
> > > of XEN_PV/PVH/PVHVM also makes sense from Linux POV.
> > 
> > OK great, thanks for the feedback.
> > 
> > > That said, I don't see much value in removing XEN_PVHVM: why are we even
> > > doing this? What is the improvement we are seeking?
> > 
> > We would not, the above discussed about the possibility of letting
> > users enable XEN_PVHVM without XEN_PVH, that's all.
> 
> OK, that makes sense.
> 
> 
> > As is the only thing that can enable XEN_PVHVM is if you enable
> > XEN_PVH.
> 
> This is the bit that we need to change but it shouldn't be difficult.
> 
> 
> > If we want
> > xen_guest_init() alone though we might need the decoupling though at
> > least at build time so that if XEN_PV or XEN_PVH is not selected we'd
> > at least have XEN_PVHVM. Thoughts?
> 
> Today pv(h) and pvhvm have very different boot paths already: pv and pvh
> initialize via xen_start_kernel while pvhvm via xen_hvm_guest_init.
> They are already very decoupled. You just need to make sure that
> init_hvm_pv_info() is called regardless on any XEN_PV/PVH/PVHVM configs.

Sorry for not replying to the point earlier -- I missed your v1
patch series submission.

After reading your series, I think we should get rid of XEN_PVHVM on x86
entirely: if XEN and XEN_FRONTEND are chosen by the user, what is the
point of one more separate option for XEN_PVHVM?

XEN_PVHVM only gives you vector callback, pv timers,
HVMOP_pagetable_dying and irq remapping support. I don't think it is
worth introducing one more option for those. They should be always built
and available if XEN_FRONTEND.

I suggest we remove XEN_PVHVM and we fold the few little remain features
offered by XEN_PVHVM in XEN_FRONTEND directly.

In terms of xen_domain types, it is still good idea to set xen_domain
to the right value no matter what. I think we should call
init_hvm_pv_info() even if XEN_FRONTEND/XEN_PVHVM is not selected.

Re: Xen's Linux kernel config options

[Luis R. Rodriguez]

On Wed, Feb 25, 2015 at 12:01:31PM +0000, Stefano Stabellini wrote:
> On Tue, 24 Feb 2015, Luis R. Rodriguez wrote:
> > On Tue, Feb 24, 2015 at 7:21 AM, Stefano Stabellini
> > <stefano.stabellini@eu.citrix.com> wrote:
> > > On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
> > >> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > >> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> > >> >> On 12/12/14 13:17, Juergen Gross wrote:
> > >> >>> XEN_PVHVM
> > >> >>
> > >> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
> > >> >
> > >> > FWIW, although it seems we do not want to let users just build
> > >> > XEN_PVHVM hypervisors I have the changes required now to at least get
> > >> > this to build so I do know what it takes.
> > >> >
> > >> >>> XEN_FRONTEND                                            XEN_PV ||
> > >> >>>                                                         XEN_PVH ||
> > >> >>>                                                         XEN_PVHVM
> > >> >>
> > >> >> This enables all the basic infrastructure for frontends: event channels,
> > >> >> grant tables and Xenbus.
> > >> >>
> > >> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> > >> >> possible to have frontend drivers without support for any of the
> > >> >> PV/PVHVM/PVH guest types.
> > >> >
> > >> > David, can you elaborate on the type of Xen guest it would be on x86
> > >> > its not PV, PVHVM, or PVH? I'm particularly curious about the
> > >> > xen_domain_type and how it would end up to selected. As it is we tie
> > >> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> > >> > completely removed from XEN_PVH we need quite a bit of code changes
> > >> > which at least as code exercise I have completed already. If we want
> > >> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> > >> > XEN_PVH are not available we need a bit more work.
> > >>
> > >> OK I think I see the issue. We have nothing quite like
> > >> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
> > >> think I can that close the gap I'm observing.
> > >>
> > >> >>  Frontends only need event channels, grant
> > >> >> table and xenbus.
> > >> >
> > >> > Well xenbus_probe_initcall() will check for xen_domain() and that
> > >> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> > >> > XEN_PVH set -- to start off with. Then
> > >> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> > >> > bit of places as well, that won't be set unless xen_setup_features()
> > >> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
> > >> > which as Juergen pointed out, is not needed if you don't have XEN_PV
> > >> > or XEN_PVH. As it turns out this is incorrect though, its needed for
> > >> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
> > >> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> > >> > don't call xen_setup_features() and its unclear to me where or how
> > >> > that should happen in other cases.
> > >>
> > >> Yeah I think having an x86 equivalent of xen_guest_init() would solve
> > >> this, Stefano, thoughts?
> > >
> > > Having xen_guest_init() on x86 would be nice.  Being able to set
> > > xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
> > > of XEN_PV/PVH/PVHVM also makes sense from Linux POV.
> > 
> > OK great, thanks for the feedback.
> > 
> > > That said, I don't see much value in removing XEN_PVHVM: why are we even
> > > doing this? What is the improvement we are seeking?
> > 
> > We would not, the above discussed about the possibility of letting
> > users enable XEN_PVHVM without XEN_PVH, that's all.
> 
> OK, that makes sense.
> 
> > As is the only thing that can enable XEN_PVHVM is if you enable
> > XEN_PVH.
> 
> This is the bit that we need to change but it shouldn't be difficult.
> 
> > If we want
> > xen_guest_init() alone though we might need the decoupling though at
> > least at build time so that if XEN_PV or XEN_PVH is not selected we'd
> > at least have XEN_PVHVM. Thoughts?
> 
> Today pv(h) and pvhvm have very different boot paths already: pv and pvh
> initialize via xen_start_kernel while pvhvm via xen_hvm_guest_init.

Ah I see, this helps a lot thanks!

> They are already very decoupled. You just need to make sure that
> init_hvm_pv_info() is called regardless on any XEN_PV/PVH/PVHVM configs.

I'm a bit confused about this given that as I see it right now
init_hvm_pv_info() is the Xen x86_hyper->init_platform() callback and that is
only called on init_hypervisor_platform() *iff* x86_hyper->detect() passed
(returned non-zero), but xen detect() returns 0 in the PV case:

static uint32_t __init xen_hvm_platform(void)                                   
{                                                                               
        if (xen_nopv)                                                           
                return 0;                                                       
                                                                                
**      if (xen_pv_domain())                                                    
**              return 0;                                                       
                                                                                
        return xen_cpuid_base();                                                
}

To be clear that is when xen_domain_type == XEN_PV_DOMAIN, and that should only
ever happen when xen_start_kernel() gets called prior to xen_hvm_platform().

Since as you note xen_start_kernel() should only be run for XEN_PV and
XEN_PVH, that is in essence the "PV"iness of the guest -- the Xen
hypervisor will look for this via the ELF entry note and kick it off
upon guest start via the hypervisor launch_vm(), otherwise we'd boot the kernel
through the usual means and init/main.c --> start_kernel() will evetually call
setup_arch() and in the end init_hypervisor_platform().

I see no way then in which init_hvm_pv_info() would be called for XEN_PV
and XEN_PVH. In fact should we not be able to just BUG_ON(xen_pv_domain()) above?

 Luis

Re: Xen's Linux kernel config options

[Konrad Rzeszutek Wilk]

> 
> I see no way then in which init_hvm_pv_info() would be called for XEN_PV
> and XEN_PVH. In fact should we not be able to just BUG_ON(xen_pv_domain()) above?

/me nods.

Re: Xen's Linux kernel config options

[Luis R. Rodriguez]

On Wed, Feb 25, 2015 at 02:27:08PM +0000, Stefano Stabellini wrote:
> On Wed, 25 Feb 2015, Stefano Stabellini wrote:
> > On Tue, 24 Feb 2015, Luis R. Rodriguez wrote:
> > > On Tue, Feb 24, 2015 at 7:21 AM, Stefano Stabellini
> > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
> > > >> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > > >> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> > > >> >> On 12/12/14 13:17, Juergen Gross wrote:
> > > >> >>> XEN_PVHVM
> > > >> >>
> > > >> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
> > > >> >
> > > >> > FWIW, although it seems we do not want to let users just build
> > > >> > XEN_PVHVM hypervisors I have the changes required now to at least get
> > > >> > this to build so I do know what it takes.
> > > >> >
> > > >> >>> XEN_FRONTEND                                            XEN_PV ||
> > > >> >>>                                                         XEN_PVH ||
> > > >> >>>                                                         XEN_PVHVM
> > > >> >>
> > > >> >> This enables all the basic infrastructure for frontends: event channels,
> > > >> >> grant tables and Xenbus.
> > > >> >>
> > > >> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> > > >> >> possible to have frontend drivers without support for any of the
> > > >> >> PV/PVHVM/PVH guest types.
> > > >> >
> > > >> > David, can you elaborate on the type of Xen guest it would be on x86
> > > >> > its not PV, PVHVM, or PVH? I'm particularly curious about the
> > > >> > xen_domain_type and how it would end up to selected. As it is we tie
> > > >> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> > > >> > completely removed from XEN_PVH we need quite a bit of code changes
> > > >> > which at least as code exercise I have completed already. If we want
> > > >> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> > > >> > XEN_PVH are not available we need a bit more work.
> > > >>
> > > >> OK I think I see the issue. We have nothing quite like
> > > >> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
> > > >> think I can that close the gap I'm observing.
> > > >>
> > > >> >>  Frontends only need event channels, grant
> > > >> >> table and xenbus.
> > > >> >
> > > >> > Well xenbus_probe_initcall() will check for xen_domain() and that
> > > >> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> > > >> > XEN_PVH set -- to start off with. Then
> > > >> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> > > >> > bit of places as well, that won't be set unless xen_setup_features()
> > > >> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
> > > >> > which as Juergen pointed out, is not needed if you don't have XEN_PV
> > > >> > or XEN_PVH. As it turns out this is incorrect though, its needed for
> > > >> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
> > > >> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> > > >> > don't call xen_setup_features() and its unclear to me where or how
> > > >> > that should happen in other cases.
> > > >>
> > > >> Yeah I think having an x86 equivalent of xen_guest_init() would solve
> > > >> this, Stefano, thoughts?
> > > >
> > > > Having xen_guest_init() on x86 would be nice.  Being able to set
> > > > xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
> > > > of XEN_PV/PVH/PVHVM also makes sense from Linux POV.
> > > 
> > > OK great, thanks for the feedback.
> > > 
> > > > That said, I don't see much value in removing XEN_PVHVM: why are we even
> > > > doing this? What is the improvement we are seeking?
> > > 
> > > We would not, the above discussed about the possibility of letting
> > > users enable XEN_PVHVM without XEN_PVH, that's all.
> > 
> > OK, that makes sense.
> > 
> > 
> > > As is the only thing that can enable XEN_PVHVM is if you enable
> > > XEN_PVH.
> > 
> > This is the bit that we need to change but it shouldn't be difficult.
> > 
> > 
> > > If we want
> > > xen_guest_init() alone though we might need the decoupling though at
> > > least at build time so that if XEN_PV or XEN_PVH is not selected we'd
> > > at least have XEN_PVHVM. Thoughts?
> > 
> > Today pv(h) and pvhvm have very different boot paths already: pv and pvh
> > initialize via xen_start_kernel while pvhvm via xen_hvm_guest_init.
> > They are already very decoupled. You just need to make sure that
> > init_hvm_pv_info() is called regardless on any XEN_PV/PVH/PVHVM configs.
> 
> Sorry for not replying to the point earlier -- I missed your v1
> patch series submission.
> 
> After reading your series, I think we should get rid of XEN_PVHVM on x86
> entirely: if XEN and XEN_FRONTEND are chosen by the user, what is the
> point of one more separate option for XEN_PVHVM?

Sure, agreed.

> XEN_PVHVM only gives you vector callback, pv timers,
> HVMOP_pagetable_dying and irq remapping support. I don't think it is
> worth introducing one more option for those. They should be always built
> and available if XEN_FRONTEND.
> 
> I suggest we remove XEN_PVHVM and we fold the few little remain features
> offered by XEN_PVHVM in XEN_FRONTEND directly.

OK since currently XEN_PVH selects XEN_PVHVM that would mean
XEN_PVH will now select XEN_FRONTEND. We also decided that
XEN_FRONTEND would select:

  XEN
  XEN_XENBUS_FRONTEND
  INPUT_XEN_KBDDEV_FRONTEND

So that would go in as collateral by having XEN_PVH select XEN_FRONTEND.
The only new one implicated here would be INPUT_XEN_KBDDEV_FRONTEND.
Also as per our agreed upon changes XEN_PVHVM should select:

  PARAVIRT
  PARAVIRT_CLOCK

So XEN_FRONTEND would select these as well then, the only new one being
PARAVIRT_CLOCK. XEN_PVHVM would depend on PCI && X86_LOCAL_APIC, so that
goes as a dependency on XEN_FRONTEND as well.

If we are OK with all this I will proceed with this.

> In terms of xen_domain types, it is still good idea to set xen_domain
> to the right value no matter what. I think we should call
> init_hvm_pv_info() even if XEN_FRONTEND/XEN_PVHVM is not selected.

Well so we currently do not call init_hvm_pv_info() without XEN_PVHVM,
the XEN_PVHVM -> XEN_FRONTEND change would implicate:

diff --git a/arch/x86/kernel/cpu/hypervisor.c b/arch/x86/kernel/cpu/hypervisor.c
index 36ce402..83a53c0 100644
--- a/arch/x86/kernel/cpu/hypervisor.c
+++ b/arch/x86/kernel/cpu/hypervisor.c
@@ -27,7 +27,7 @@
 
 static const __initconst struct hypervisor_x86 * const hypervisors[] =
 {
-#ifdef CONFIG_XEN_PVHVM
+#ifdef CONFIG_XEN_FRONTEND
 	&x86_hyper_xen_hvm,
 #endif
 	&x86_hyper_vmware,

Striving towards this change alone should get us what we need I think
for enabling us to build front end drivers. In fact doing a wide change
on the delta I had prepared so far I get front end drivers now compiling
happily with this change. I am not sure if we need then init_hvm_pv_info()
called elsewhere if this change is made as the only possibility would
be is if the guest is a XEN_PV or XEN_PVH. Let me know.

  Luis

Re: Xen's Linux kernel config options

[Stefano Stabellini]

On Wed, 25 Feb 2015, Luis R. Rodriguez wrote:
> On Wed, Feb 25, 2015 at 12:01:31PM +0000, Stefano Stabellini wrote:
> > On Tue, 24 Feb 2015, Luis R. Rodriguez wrote:
> > > On Tue, Feb 24, 2015 at 7:21 AM, Stefano Stabellini
> > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
> > > >> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > > >> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> > > >> >> On 12/12/14 13:17, Juergen Gross wrote:
> > > >> >>> XEN_PVHVM
> > > >> >>
> > > >> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
> > > >> >
> > > >> > FWIW, although it seems we do not want to let users just build
> > > >> > XEN_PVHVM hypervisors I have the changes required now to at least get
> > > >> > this to build so I do know what it takes.
> > > >> >
> > > >> >>> XEN_FRONTEND                                            XEN_PV ||
> > > >> >>>                                                         XEN_PVH ||
> > > >> >>>                                                         XEN_PVHVM
> > > >> >>
> > > >> >> This enables all the basic infrastructure for frontends: event channels,
> > > >> >> grant tables and Xenbus.
> > > >> >>
> > > >> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> > > >> >> possible to have frontend drivers without support for any of the
> > > >> >> PV/PVHVM/PVH guest types.
> > > >> >
> > > >> > David, can you elaborate on the type of Xen guest it would be on x86
> > > >> > its not PV, PVHVM, or PVH? I'm particularly curious about the
> > > >> > xen_domain_type and how it would end up to selected. As it is we tie
> > > >> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> > > >> > completely removed from XEN_PVH we need quite a bit of code changes
> > > >> > which at least as code exercise I have completed already. If we want
> > > >> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> > > >> > XEN_PVH are not available we need a bit more work.
> > > >>
> > > >> OK I think I see the issue. We have nothing quite like
> > > >> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
> > > >> think I can that close the gap I'm observing.
> > > >>
> > > >> >>  Frontends only need event channels, grant
> > > >> >> table and xenbus.
> > > >> >
> > > >> > Well xenbus_probe_initcall() will check for xen_domain() and that
> > > >> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> > > >> > XEN_PVH set -- to start off with. Then
> > > >> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> > > >> > bit of places as well, that won't be set unless xen_setup_features()
> > > >> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
> > > >> > which as Juergen pointed out, is not needed if you don't have XEN_PV
> > > >> > or XEN_PVH. As it turns out this is incorrect though, its needed for
> > > >> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
> > > >> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> > > >> > don't call xen_setup_features() and its unclear to me where or how
> > > >> > that should happen in other cases.
> > > >>
> > > >> Yeah I think having an x86 equivalent of xen_guest_init() would solve
> > > >> this, Stefano, thoughts?
> > > >
> > > > Having xen_guest_init() on x86 would be nice.  Being able to set
> > > > xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
> > > > of XEN_PV/PVH/PVHVM also makes sense from Linux POV.
> > > 
> > > OK great, thanks for the feedback.
> > > 
> > > > That said, I don't see much value in removing XEN_PVHVM: why are we even
> > > > doing this? What is the improvement we are seeking?
> > > 
> > > We would not, the above discussed about the possibility of letting
> > > users enable XEN_PVHVM without XEN_PVH, that's all.
> > 
> > OK, that makes sense.
> > 
> > > As is the only thing that can enable XEN_PVHVM is if you enable
> > > XEN_PVH.
> > 
> > This is the bit that we need to change but it shouldn't be difficult.
> > 
> > > If we want
> > > xen_guest_init() alone though we might need the decoupling though at
> > > least at build time so that if XEN_PV or XEN_PVH is not selected we'd
> > > at least have XEN_PVHVM. Thoughts?
> > 
> > Today pv(h) and pvhvm have very different boot paths already: pv and pvh
> > initialize via xen_start_kernel while pvhvm via xen_hvm_guest_init.
> 
> Ah I see, this helps a lot thanks!
> 
> > They are already very decoupled. You just need to make sure that
> > init_hvm_pv_info() is called regardless on any XEN_PV/PVH/PVHVM configs.
> 
> I'm a bit confused about this given that as I see it right now
> init_hvm_pv_info() is the Xen x86_hyper->init_platform() callback and that is
> only called on init_hypervisor_platform() *iff* x86_hyper->detect() passed
> (returned non-zero), but xen detect() returns 0 in the PV case:
> 
> static uint32_t __init xen_hvm_platform(void)                                   
> {                                                                               
>         if (xen_nopv)                                                           
>                 return 0;                                                       
>                                                                                 
> **      if (xen_pv_domain())                                                    
> **              return 0;                                                       
>                                                                                 
>         return xen_cpuid_base();                                                
> }
> 
> To be clear that is when xen_domain_type == XEN_PV_DOMAIN, and that should only
> ever happen when xen_start_kernel() gets called prior to xen_hvm_platform().
> 
> Since as you note xen_start_kernel() should only be run for XEN_PV and
> XEN_PVH, that is in essence the "PV"iness of the guest -- the Xen
> hypervisor will look for this via the ELF entry note and kick it off
> upon guest start via the hypervisor launch_vm(), otherwise we'd boot the kernel
> through the usual means and init/main.c --> start_kernel() will evetually call
> setup_arch() and in the end init_hypervisor_platform().
> 
> I see no way then in which init_hvm_pv_info() would be called for XEN_PV
> and XEN_PVH. In fact should we not be able to just BUG_ON(xen_pv_domain()) above?

Yes, that is true.

What I meant is that for HVM guests you could set xen_domain_type from
something like xen_hvm_platform (x86_hyper->detect()) indepedently from
PVH/PV/PVHVM config options. xen_domain_type = XEN_HVM_DOMAIN doesn't
necessarely mean that we are going to use PV on HVM interfaces, it only
means that we are running inside Xen HVM virtuam machine.

Re: Xen's Linux kernel config options

[Luis R. Rodriguez]

On Thu, Feb 26, 2015 at 11:19:17AM +0000, Stefano Stabellini wrote:
> On Wed, 25 Feb 2015, Luis R. Rodriguez wrote:
> > On Wed, Feb 25, 2015 at 12:01:31PM +0000, Stefano Stabellini wrote:
> > > On Tue, 24 Feb 2015, Luis R. Rodriguez wrote:
> > > > On Tue, Feb 24, 2015 at 7:21 AM, Stefano Stabellini
> > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
> > > > >> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > > > >> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> > > > >> >> On 12/12/14 13:17, Juergen Gross wrote:
> > > > >> >>> XEN_PVHVM
> > > > >> >>
> > > > >> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
> > > > >> >
> > > > >> > FWIW, although it seems we do not want to let users just build
> > > > >> > XEN_PVHVM hypervisors I have the changes required now to at least get
> > > > >> > this to build so I do know what it takes.
> > > > >> >
> > > > >> >>> XEN_FRONTEND                                            XEN_PV ||
> > > > >> >>>                                                         XEN_PVH ||
> > > > >> >>>                                                         XEN_PVHVM
> > > > >> >>
> > > > >> >> This enables all the basic infrastructure for frontends: event channels,
> > > > >> >> grant tables and Xenbus.
> > > > >> >>
> > > > >> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> > > > >> >> possible to have frontend drivers without support for any of the
> > > > >> >> PV/PVHVM/PVH guest types.
> > > > >> >
> > > > >> > David, can you elaborate on the type of Xen guest it would be on x86
> > > > >> > its not PV, PVHVM, or PVH? I'm particularly curious about the
> > > > >> > xen_domain_type and how it would end up to selected. As it is we tie
> > > > >> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> > > > >> > completely removed from XEN_PVH we need quite a bit of code changes
> > > > >> > which at least as code exercise I have completed already. If we want
> > > > >> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> > > > >> > XEN_PVH are not available we need a bit more work.
> > > > >>
> > > > >> OK I think I see the issue. We have nothing quite like
> > > > >> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
> > > > >> think I can that close the gap I'm observing.
> > > > >>
> > > > >> >>  Frontends only need event channels, grant
> > > > >> >> table and xenbus.
> > > > >> >
> > > > >> > Well xenbus_probe_initcall() will check for xen_domain() and that
> > > > >> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> > > > >> > XEN_PVH set -- to start off with. Then
> > > > >> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> > > > >> > bit of places as well, that won't be set unless xen_setup_features()
> > > > >> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
> > > > >> > which as Juergen pointed out, is not needed if you don't have XEN_PV
> > > > >> > or XEN_PVH. As it turns out this is incorrect though, its needed for
> > > > >> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
> > > > >> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> > > > >> > don't call xen_setup_features() and its unclear to me where or how
> > > > >> > that should happen in other cases.
> > > > >>
> > > > >> Yeah I think having an x86 equivalent of xen_guest_init() would solve
> > > > >> this, Stefano, thoughts?
> > > > >
> > > > > Having xen_guest_init() on x86 would be nice.  Being able to set
> > > > > xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
> > > > > of XEN_PV/PVH/PVHVM also makes sense from Linux POV.
> > > > 
> > > > OK great, thanks for the feedback.
> > > > 
> > > > > That said, I don't see much value in removing XEN_PVHVM: why are we even
> > > > > doing this? What is the improvement we are seeking?
> > > > 
> > > > We would not, the above discussed about the possibility of letting
> > > > users enable XEN_PVHVM without XEN_PVH, that's all.
> > > 
> > > OK, that makes sense.
> > > 
> > > > As is the only thing that can enable XEN_PVHVM is if you enable
> > > > XEN_PVH.
> > > 
> > > This is the bit that we need to change but it shouldn't be difficult.
> > > 
> > > > If we want
> > > > xen_guest_init() alone though we might need the decoupling though at
> > > > least at build time so that if XEN_PV or XEN_PVH is not selected we'd
> > > > at least have XEN_PVHVM. Thoughts?
> > > 
> > > Today pv(h) and pvhvm have very different boot paths already: pv and pvh
> > > initialize via xen_start_kernel while pvhvm via xen_hvm_guest_init.
> > 
> > Ah I see, this helps a lot thanks!
> > 
> > > They are already very decoupled. You just need to make sure that
> > > init_hvm_pv_info() is called regardless on any XEN_PV/PVH/PVHVM configs.
> > 
> > I'm a bit confused about this given that as I see it right now
> > init_hvm_pv_info() is the Xen x86_hyper->init_platform() callback and that is
> > only called on init_hypervisor_platform() *iff* x86_hyper->detect() passed
> > (returned non-zero), but xen detect() returns 0 in the PV case:
> > 
> > static uint32_t __init xen_hvm_platform(void)                                   
> > {                                                                               
> >         if (xen_nopv)                                                           
> >                 return 0;                                                       
> >                                                                                 
> > **      if (xen_pv_domain())                                                    
> > **              return 0;                                                       
> >                                                                                 
> >         return xen_cpuid_base();                                                
> > }
> > 
> > To be clear that is when xen_domain_type == XEN_PV_DOMAIN, and that should only
> > ever happen when xen_start_kernel() gets called prior to xen_hvm_platform().
> > 
> > Since as you note xen_start_kernel() should only be run for XEN_PV and
> > XEN_PVH, that is in essence the "PV"iness of the guest -- the Xen
> > hypervisor will look for this via the ELF entry note and kick it off
> > upon guest start via the hypervisor launch_vm(), otherwise we'd boot the kernel
> > through the usual means and init/main.c --> start_kernel() will evetually call
> > setup_arch() and in the end init_hypervisor_platform().
> > 
> > I see no way then in which init_hvm_pv_info() would be called for XEN_PV
> > and XEN_PVH. In fact should we not be able to just BUG_ON(xen_pv_domain()) above?
> 
> Yes, that is true.
> 
> What I meant is that for HVM guests you could set xen_domain_type from
> something like xen_hvm_platform (x86_hyper->detect()) indepedently from
> PVH/PV/PVHVM config options. xen_domain_type = XEN_HVM_DOMAIN doesn't
> necessarely mean that we are going to use PV on HVM interfaces, it only
> means that we are running inside Xen HVM virtuam machine.

Sure, that is clear to me, so in other words right now we have no path that
only pegs a simple Xen HVM virtual machine upon setup on init, correct?

  Luis

Re: Xen's Linux kernel config options

[Stefano Stabellini]

On Thu, 26 Feb 2015, Luis R. Rodriguez wrote:
> On Thu, Feb 26, 2015 at 11:19:17AM +0000, Stefano Stabellini wrote:
> > On Wed, 25 Feb 2015, Luis R. Rodriguez wrote:
> > > On Wed, Feb 25, 2015 at 12:01:31PM +0000, Stefano Stabellini wrote:
> > > > On Tue, 24 Feb 2015, Luis R. Rodriguez wrote:
> > > > > On Tue, Feb 24, 2015 at 7:21 AM, Stefano Stabellini
> > > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > > On Mon, 23 Feb 2015, Luis R. Rodriguez wrote:
> > > > > >> On Thu, Feb 19, 2015 at 3:43 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > > > > >> > On Fri, Dec 12, 2014 at 9:29 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> > > > > >> >> On 12/12/14 13:17, Juergen Gross wrote:
> > > > > >> >>> XEN_PVHVM
> > > > > >> >>
> > > > > >> >> Move XEN_PVHVM under XEN and have it select PARAVIRT and PARAVIRT_CLOCK.
> > > > > >> >
> > > > > >> > FWIW, although it seems we do not want to let users just build
> > > > > >> > XEN_PVHVM hypervisors I have the changes required now to at least get
> > > > > >> > this to build so I do know what it takes.
> > > > > >> >
> > > > > >> >>> XEN_FRONTEND                                            XEN_PV ||
> > > > > >> >>>                                                         XEN_PVH ||
> > > > > >> >>>                                                         XEN_PVHVM
> > > > > >> >>
> > > > > >> >> This enables all the basic infrastructure for frontends: event channels,
> > > > > >> >> grant tables and Xenbus.
> > > > > >> >>
> > > > > >> >> Don't make XEN_FRONTEND depend on any XEN_* variant.  It should be
> > > > > >> >> possible to have frontend drivers without support for any of the
> > > > > >> >> PV/PVHVM/PVH guest types.
> > > > > >> >
> > > > > >> > David, can you elaborate on the type of Xen guest it would be on x86
> > > > > >> > its not PV, PVHVM, or PVH? I'm particularly curious about the
> > > > > >> > xen_domain_type and how it would end up to selected. As it is we tie
> > > > > >> > in XEN_PVHVM at build time with XEN_PVH, in order to have XEN_PVHVM
> > > > > >> > completely removed from XEN_PVH we need quite a bit of code changes
> > > > > >> > which at least as code exercise I have completed already. If we want
> > > > > >> > at the very least xen_domain_type set when XEN_PV, XEN_PVHVM, and
> > > > > >> > XEN_PVH are not available we need a bit more work.
> > > > > >>
> > > > > >> OK I think I see the issue. We have nothing quite like
> > > > > >> xen_guest_init() on x86 enlighten.c, we do have this for ARM and I
> > > > > >> think I can that close the gap I'm observing.
> > > > > >>
> > > > > >> >>  Frontends only need event channels, grant
> > > > > >> >> table and xenbus.
> > > > > >> >
> > > > > >> > Well xenbus_probe_initcall() will check for xen_domain() and that
> > > > > >> > won't be set on x86 right now unless we have XEN_PV, XEN_PVHVM or
> > > > > >> > XEN_PVH set -- to start off with. Then
> > > > > >> > drivers/xen/xenbus/xenbus_client.c will check xen_feature in quite a
> > > > > >> > bit of places as well, that won't be set unless xen_setup_features()
> > > > > >> > is called which right now is only done on x86 arch/x86/xen/enlighten.c
> > > > > >> > which as Juergen pointed out, is not needed if you don't have XEN_PV
> > > > > >> > or XEN_PVH. As it turns out this is incorrect though, its needed for
> > > > > >> > XEN_PVHVM as well and my split exercise in code addresses this. Now,
> > > > > >> > at least in my code if you don't have XEN_PV, XEN_PVHVM, or XEN_PVH we
> > > > > >> > don't call xen_setup_features() and its unclear to me where or how
> > > > > >> > that should happen in other cases.
> > > > > >>
> > > > > >> Yeah I think having an x86 equivalent of xen_guest_init() would solve
> > > > > >> this, Stefano, thoughts?
> > > > > >
> > > > > > Having xen_guest_init() on x86 would be nice.  Being able to set
> > > > > > xen_domain_type to XEN_HVM_DOMAIN if we are running on Xen, regardless
> > > > > > of XEN_PV/PVH/PVHVM also makes sense from Linux POV.
> > > > > 
> > > > > OK great, thanks for the feedback.
> > > > > 
> > > > > > That said, I don't see much value in removing XEN_PVHVM: why are we even
> > > > > > doing this? What is the improvement we are seeking?
> > > > > 
> > > > > We would not, the above discussed about the possibility of letting
> > > > > users enable XEN_PVHVM without XEN_PVH, that's all.
> > > > 
> > > > OK, that makes sense.
> > > > 
> > > > > As is the only thing that can enable XEN_PVHVM is if you enable
> > > > > XEN_PVH.
> > > > 
> > > > This is the bit that we need to change but it shouldn't be difficult.
> > > > 
> > > > > If we want
> > > > > xen_guest_init() alone though we might need the decoupling though at
> > > > > least at build time so that if XEN_PV or XEN_PVH is not selected we'd
> > > > > at least have XEN_PVHVM. Thoughts?
> > > > 
> > > > Today pv(h) and pvhvm have very different boot paths already: pv and pvh
> > > > initialize via xen_start_kernel while pvhvm via xen_hvm_guest_init.
> > > 
> > > Ah I see, this helps a lot thanks!
> > > 
> > > > They are already very decoupled. You just need to make sure that
> > > > init_hvm_pv_info() is called regardless on any XEN_PV/PVH/PVHVM configs.
> > > 
> > > I'm a bit confused about this given that as I see it right now
> > > init_hvm_pv_info() is the Xen x86_hyper->init_platform() callback and that is
> > > only called on init_hypervisor_platform() *iff* x86_hyper->detect() passed
> > > (returned non-zero), but xen detect() returns 0 in the PV case:
> > > 
> > > static uint32_t __init xen_hvm_platform(void)                                   
> > > {                                                                               
> > >         if (xen_nopv)                                                           
> > >                 return 0;                                                       
> > >                                                                                 
> > > **      if (xen_pv_domain())                                                    
> > > **              return 0;                                                       
> > >                                                                                 
> > >         return xen_cpuid_base();                                                
> > > }
> > > 
> > > To be clear that is when xen_domain_type == XEN_PV_DOMAIN, and that should only
> > > ever happen when xen_start_kernel() gets called prior to xen_hvm_platform().
> > > 
> > > Since as you note xen_start_kernel() should only be run for XEN_PV and
> > > XEN_PVH, that is in essence the "PV"iness of the guest -- the Xen
> > > hypervisor will look for this via the ELF entry note and kick it off
> > > upon guest start via the hypervisor launch_vm(), otherwise we'd boot the kernel
> > > through the usual means and init/main.c --> start_kernel() will evetually call
> > > setup_arch() and in the end init_hypervisor_platform().
> > > 
> > > I see no way then in which init_hvm_pv_info() would be called for XEN_PV
> > > and XEN_PVH. In fact should we not be able to just BUG_ON(xen_pv_domain()) above?
> > 
> > Yes, that is true.
> > 
> > What I meant is that for HVM guests you could set xen_domain_type from
> > something like xen_hvm_platform (x86_hyper->detect()) indepedently from
> > PVH/PV/PVHVM config options. xen_domain_type = XEN_HVM_DOMAIN doesn't
> > necessarely mean that we are going to use PV on HVM interfaces, it only
> > means that we are running inside Xen HVM virtuam machine.
> 
> Sure, that is clear to me, so in other words right now we have no path that
> only pegs a simple Xen HVM virtual machine upon setup on init, correct?

That's right, only if PVHVM we do that via xen_hvm_platform
(x86_hyper->detect()).