A couple of HVMlite loose ends

A couple of HVMlite loose ends

[Roger Pau Monné]

Hello,

While working on a HVMlite Dom0 implementation I've found a couple of
loose ends with the design that I would like to comment because it's not
clear to me what's the best direction to take.

1. HVM CPUID and Dom0.

Sadly the way CPUID is handled inside of Xen varies between PV and HVM.
On PV guests AFAICT we mostly do black-listing (I think this is the
right term), which means we pick the native CPUID result and then
perform a series of filter operations in order to remove features which
should not be exposed to a PV guest. On the other hand, for HVM guests
we pre-populate an array (d->arch.cpuids) during domain build time, and
the contents of that array is what is returned to the guest when a CPUID
instruction is executed.

This is a problem for a HVMlite Dom0, since the code that populates this
array resides in libxc, and when creating a HVMlite Dom0 from Xen itself
(using a properly arranged Dom0 builder) this array doesn't get
populated at all, leading to wrong CPUID information being returned to
the guest. I can see two solutions to this problem:

 a) Switch CPUID handling for HVMlite Dom0 to the PV one, like it's done
for PVH Dom0.

 b) Duplicate the code in libxc into the Xen HVMlite Dom0 builder and
populate d->arch.cpuids.

I'm leaning towards option "b)", because I would like HVMlite to behave
as a HVM guest as much as possible, but I would like to hear opinions
from others before taking either route.

2. HVM MTRR and Dom0.

MTRR ranges are initialised from hvmloader, which means that although we
expose the MTRR functionality to HVMlite guests (and AFAICT the
functionality is fully complete/usable), the initial state in which a
guest finds the MTRR ranges is not expected, leading to errors. Again, I
see three ways to solve this:

 a) Mask the MTRR functionality from CPUID for HVMlite guests. This
requires adding a XEN_X86_EMU_MTRR flag to the bitmap introduced in arch
domain.

 b) Setting up the initial MTRR state from libxl/libxc for HVMlite DomU
and from the Xen domain builder for HVMlite Dom0. This again implies
some functional duplication of MTRR related code, since now we would
have 3 different places where MTRR is setup. One inside hvmloader for
classic HVM guests, another one inside of libxl/libxc for HVMlite DomU
and yet another one in the Dom0 building for HVMlite Dom0.

 c) Be aware of this fact and change the OS code so that it will setup
the initial MTRR ranges correctly.

Again, I'm leaning towards "b)", because that's the one that's closest
to native and what we do for classic HVM guests, but would like to hear
opinions.

Thanks, Roger.

Re: A couple of HVMlite loose ends

[David Vrabel]

On 13/01/16 15:49, Roger Pau Monné wrote:
> Hello,
> 
> While working on a HVMlite Dom0 implementation I've found a couple of
> loose ends with the design that I would like to comment because it's not
> clear to me what's the best direction to take.
> 
> 1. HVM CPUID and Dom0.

I think Andy's pending cpu feature series will address this.

> 2. HVM MTRR and Dom0.
> 
> MTRR ranges are initialised from hvmloader, which means that although we
> expose the MTRR functionality to HVMlite guests (and AFAICT the
> functionality is fully complete/usable), the initial state in which a
> guest finds the MTRR ranges is not expected, leading to errors. Again, I
> see three ways to solve this:
> 
>  a) Mask the MTRR functionality from CPUID for HVMlite guests. This
> requires adding a XEN_X86_EMU_MTRR flag to the bitmap introduced in arch
> domain.

I'd favour this approach I think, depending on if any changes were
needed in guests to support this (I assume not?).

Guests should use PAT instead.

David

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: A couple of HVMlite loose ends

[Juergen Gross]

On 13/01/16 16:49, Roger Pau Monné wrote:
> Hello,
> 
> While working on a HVMlite Dom0 implementation I've found a couple of
> loose ends with the design that I would like to comment because it's not
> clear to me what's the best direction to take.
> 
> 1. HVM CPUID and Dom0.
> 
> Sadly the way CPUID is handled inside of Xen varies between PV and HVM.
> On PV guests AFAICT we mostly do black-listing (I think this is the
> right term), which means we pick the native CPUID result and then
> perform a series of filter operations in order to remove features which
> should not be exposed to a PV guest. On the other hand, for HVM guests
> we pre-populate an array (d->arch.cpuids) during domain build time, and
> the contents of that array is what is returned to the guest when a CPUID
> instruction is executed.
> 
> This is a problem for a HVMlite Dom0, since the code that populates this
> array resides in libxc, and when creating a HVMlite Dom0 from Xen itself
> (using a properly arranged Dom0 builder) this array doesn't get
> populated at all, leading to wrong CPUID information being returned to
> the guest. I can see two solutions to this problem:
> 
>  a) Switch CPUID handling for HVMlite Dom0 to the PV one, like it's done
> for PVH Dom0.
> 
>  b) Duplicate the code in libxc into the Xen HVMlite Dom0 builder and
> populate d->arch.cpuids.

Make this "move the code from libxc into xen/common/lib<blah> and use
the code from libxc and the hypervisor".

> I'm leaning towards option "b)", because I would like HVMlite to behave
> as a HVM guest as much as possible, but I would like to hear opinions
> from others before taking either route.

I'd prefer the modified b) version, too.

> 2. HVM MTRR and Dom0.
> 
> MTRR ranges are initialised from hvmloader, which means that although we
> expose the MTRR functionality to HVMlite guests (and AFAICT the
> functionality is fully complete/usable), the initial state in which a
> guest finds the MTRR ranges is not expected, leading to errors. Again, I
> see three ways to solve this:
> 
>  a) Mask the MTRR functionality from CPUID for HVMlite guests. This
> requires adding a XEN_X86_EMU_MTRR flag to the bitmap introduced in arch
> domain.
> 
>  b) Setting up the initial MTRR state from libxl/libxc for HVMlite DomU
> and from the Xen domain builder for HVMlite Dom0. This again implies
> some functional duplication of MTRR related code, since now we would
> have 3 different places where MTRR is setup. One inside hvmloader for
> classic HVM guests, another one inside of libxl/libxc for HVMlite DomU
> and yet another one in the Dom0 building for HVMlite Dom0.

Same as above.

> 
>  c) Be aware of this fact and change the OS code so that it will setup
> the initial MTRR ranges correctly.
> 
> Again, I'm leaning towards "b)", because that's the one that's closest
> to native and what we do for classic HVM guests, but would like to hear
> opinions.

And again as above. :-)


Juergen

Re: A couple of HVMlite loose ends

[Roger Pau Monné]

El 13/01/16 a les 16.56, Juergen Gross ha escrit:
> On 13/01/16 16:49, Roger Pau Monné wrote:
>>  b) Setting up the initial MTRR state from libxl/libxc for HVMlite DomU
>> and from the Xen domain builder for HVMlite Dom0. This again implies
>> some functional duplication of MTRR related code, since now we would
>> have 3 different places where MTRR is setup. One inside hvmloader for
>> classic HVM guests, another one inside of libxl/libxc for HVMlite DomU
>> and yet another one in the Dom0 building for HVMlite Dom0.

Moving the code to xen/common/lib<blah> is not suitable in this case (or
so I think), because hvmloader runs inside of the guest (just like a
regular BIOS firmware), while we would need to do this from the outside
for HVMlite guests (using mtrr_def_type_msr_set, mtrr_fix_range_msr_set
and mtrr_var_range_msr_set).

Roger.

Re: A couple of HVMlite loose ends

[Juergen Gross]

On 13/01/16 17:01, Roger Pau Monné wrote:
> El 13/01/16 a les 16.56, Juergen Gross ha escrit:
>> On 13/01/16 16:49, Roger Pau Monné wrote:
>>>  b) Setting up the initial MTRR state from libxl/libxc for HVMlite DomU
>>> and from the Xen domain builder for HVMlite Dom0. This again implies
>>> some functional duplication of MTRR related code, since now we would
>>> have 3 different places where MTRR is setup. One inside hvmloader for
>>> classic HVM guests, another one inside of libxl/libxc for HVMlite DomU
>>> and yet another one in the Dom0 building for HVMlite Dom0.
> 
> Moving the code to xen/common/lib<blah> is not suitable in this case (or
> so I think), because hvmloader runs inside of the guest (just like a
> regular BIOS firmware), while we would need to do this from the outside
> for HVMlite guests (using mtrr_def_type_msr_set, mtrr_fix_range_msr_set
> and mtrr_var_range_msr_set).

Have a look how xen/common/libelf is being used from libxc and the
hypervisor. It is just code included via make instead of duplicating it
on source level.


Juergen

Re: A couple of HVMlite loose ends

[Roger Pau Monné]

El 13/01/16 a les 16.54, David Vrabel ha escrit:
> On 13/01/16 15:49, Roger Pau Monné wrote:
>> 2. HVM MTRR and Dom0.
>>
>> MTRR ranges are initialised from hvmloader, which means that although we
>> expose the MTRR functionality to HVMlite guests (and AFAICT the
>> functionality is fully complete/usable), the initial state in which a
>> guest finds the MTRR ranges is not expected, leading to errors. Again, I
>> see three ways to solve this:
>>
>>  a) Mask the MTRR functionality from CPUID for HVMlite guests. This
>> requires adding a XEN_X86_EMU_MTRR flag to the bitmap introduced in arch
>> domain.
> 
> I'd favour this approach I think, depending on if any changes were
> needed in guests to support this (I assume not?).

No, AFAIK PAT works perfectly fine on HVMlite, but IIRC MTRR is
mandatory on amd64 CPUs (although almost nobody uses it).

> Guests should use PAT instead.

This is a far shoot, but HVMlite guests are not required to run with
paging enabled, and as so we should provide ways to set memory
attributes taking that into account IMHO.

Roger.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: A couple of HVMlite loose ends

[Jan Beulich]

>>> On 13.01.16 at 16:54, <david.vrabel@citrix.com> wrote:
> On 13/01/16 15:49, Roger Pau Monné wrote:
>> While working on a HVMlite Dom0 implementation I've found a couple of
>> loose ends with the design that I would like to comment because it's not
>> clear to me what's the best direction to take.
>> 
>> 1. HVM CPUID and Dom0.
> 
> I think Andy's pending cpu feature series will address this.

The blacklisting vs whitelisting part yes, but the pre-population of
d->arch.cpuids[] is independent afaict.

>> 2. HVM MTRR and Dom0.
>> 
>> MTRR ranges are initialised from hvmloader, which means that although we
>> expose the MTRR functionality to HVMlite guests (and AFAICT the
>> functionality is fully complete/usable), the initial state in which a
>> guest finds the MTRR ranges is not expected, leading to errors. Again, I
>> see three ways to solve this:
>> 
>>  a) Mask the MTRR functionality from CPUID for HVMlite guests. This
>> requires adding a XEN_X86_EMU_MTRR flag to the bitmap introduced in arch
>> domain.
> 
> I'd favour this approach I think, depending on if any changes were
> needed in guests to support this (I assume not?).
> 
> Guests should use PAT instead.

+1

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: A couple of HVMlite loose ends

[Jan Beulich]

>>> On 13.01.16 at 16:49, <roger.pau@citrix.com> wrote:
> Hello,
> 
> While working on a HVMlite Dom0 implementation I've found a couple of
> loose ends with the design that I would like to comment because it's not
> clear to me what's the best direction to take.
> 
> 1. HVM CPUID and Dom0.
> 
> Sadly the way CPUID is handled inside of Xen varies between PV and HVM.
> On PV guests AFAICT we mostly do black-listing (I think this is the
> right term), which means we pick the native CPUID result and then
> perform a series of filter operations in order to remove features which
> should not be exposed to a PV guest. On the other hand, for HVM guests
> we pre-populate an array (d->arch.cpuids) during domain build time, and
> the contents of that array is what is returned to the guest when a CPUID
> instruction is executed.

This d->arch.cpuids[] mechanism is common to HVM and PV; the
exception really is Dom0.

> This is a problem for a HVMlite Dom0, since the code that populates this
> array resides in libxc, and when creating a HVMlite Dom0 from Xen itself
> (using a properly arranged Dom0 builder) this array doesn't get
> populated at all, leading to wrong CPUID information being returned to
> the guest. I can see two solutions to this problem:
> 
>  a) Switch CPUID handling for HVMlite Dom0 to the PV one, like it's done
> for PVH Dom0.

I think this is what needs to be done for the time being, unless ...

>  b) Duplicate the code in libxc into the Xen HVMlite Dom0 builder and
> populate d->arch.cpuids.

... this would be done by source code sharing.

Jan

Re: A couple of HVMlite loose ends

[Andrew Cooper]

On 13/01/16 15:49, Roger Pau Monné wrote:
> Hello,
>
> While working on a HVMlite Dom0 implementation I've found a couple of
> loose ends with the design that I would like to comment because it's not
> clear to me what's the best direction to take.
>
> 1. HVM CPUID and Dom0.
>
> Sadly the way CPUID is handled inside of Xen varies between PV and HVM.
> On PV guests AFAICT we mostly do black-listing (I think this is the
> right term), which means we pick the native CPUID result and then
> perform a series of filter operations in order to remove features which
> should not be exposed to a PV guest. On the other hand, for HVM guests
> we pre-populate an array (d->arch.cpuids) during domain build time, and
> the contents of that array is what is returned to the guest when a CPUID
> instruction is executed.
>
> This is a problem for a HVMlite Dom0, since the code that populates this
> array resides in libxc, and when creating a HVMlite Dom0 from Xen itself
> (using a properly arranged Dom0 builder) this array doesn't get
> populated at all, leading to wrong CPUID information being returned to
> the guest. I can see two solutions to this problem:
>
>  a) Switch CPUID handling for HVMlite Dom0 to the PV one, like it's done
> for PVH Dom0.
>
>  b) Duplicate the code in libxc into the Xen HVMlite Dom0 builder and
> populate d->arch.cpuids.
>
> I'm leaning towards option "b)", because I would like HVMlite to behave
> as a HVM guest as much as possible, but I would like to hear opinions
> from others before taking either route.

My phase 2 plans for cpuid involve having Xen generate a complete
maximum cpuid policy for each type of guest, having libxc copy this and
trim this down to suit the domain, then provide it back to Xen as the
end policy for the domain.

The current situation of having libxc do a guestimation based on what
dom0 can see is wrong, and I will be removing it in the longterm.  I
will also be removing any distinction between the hardware domain and
other domains, when it comes to cpuid handling.

In the short term, this isn't very helpful.  I would suggest an
intermediate "b)", which will end up coming back out when CPUID phase 2
is complete.

>
> 2. HVM MTRR and Dom0.
>
> MTRR ranges are initialised from hvmloader, which means that although we
> expose the MTRR functionality to HVMlite guests (and AFAICT the
> functionality is fully complete/usable), the initial state in which a
> guest finds the MTRR ranges is not expected, leading to errors. Again, I
> see three ways to solve this:

What errors?  OSes already need to deal with any quantity of crazy setup
from firmware.

>
>  a) Mask the MTRR functionality from CPUID for HVMlite guests. This
> requires adding a XEN_X86_EMU_MTRR flag to the bitmap introduced in arch
> domain.

MTRRs are x86 architectural features.

>
>  b) Setting up the initial MTRR state from libxl/libxc for HVMlite DomU
> and from the Xen domain builder for HVMlite Dom0. This again implies
> some functional duplication of MTRR related code, since now we would
> have 3 different places where MTRR is setup. One inside hvmloader for
> classic HVM guests, another one inside of libxl/libxc for HVMlite DomU
> and yet another one in the Dom0 building for HVMlite Dom0.
>
>  c) Be aware of this fact and change the OS code so that it will setup
> the initial MTRR ranges correctly.
>
> Again, I'm leaning towards "b)", because that's the one that's closest
> to native and what we do for classic HVM guests, but would like to hear
> opinions.

What is wrong with having them disabled by default?  They are not
generally useful to guests, given PAT and no SMM/firmware to get in the
way.  If a guest really wants to use them, it can always turn them on
and reconfigure them.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: A couple of HVMlite loose ends

[Andrew Cooper]

On 13/01/16 16:13, Jan Beulich wrote:
>>>> On 13.01.16 at 16:49, <roger.pau@citrix.com> wrote:
>> Hello,
>>
>> While working on a HVMlite Dom0 implementation I've found a couple of
>> loose ends with the design that I would like to comment because it's not
>> clear to me what's the best direction to take.
>>
>> 1. HVM CPUID and Dom0.
>>
>> Sadly the way CPUID is handled inside of Xen varies between PV and HVM.
>> On PV guests AFAICT we mostly do black-listing (I think this is the
>> right term), which means we pick the native CPUID result and then
>> perform a series of filter operations in order to remove features which
>> should not be exposed to a PV guest. On the other hand, for HVM guests
>> we pre-populate an array (d->arch.cpuids) during domain build time, and
>> the contents of that array is what is returned to the guest when a CPUID
>> instruction is executed.
> This d->arch.cpuids[] mechanism is common to HVM and PV; the
> exception really is Dom0.

Dom0 is not special when it comes to cpuid, and shouldn't be treated as
such.  My longter term CPUID plans will be fixing this.

~Andrew

Re: A couple of HVMlite loose ends

[Jan Beulich]

>>> On 13.01.16 at 17:17, <andrew.cooper3@citrix.com> wrote:
> On 13/01/16 16:13, Jan Beulich wrote:
>>>>> On 13.01.16 at 16:49, <roger.pau@citrix.com> wrote:
>>> Hello,
>>>
>>> While working on a HVMlite Dom0 implementation I've found a couple of
>>> loose ends with the design that I would like to comment because it's not
>>> clear to me what's the best direction to take.
>>>
>>> 1. HVM CPUID and Dom0.
>>>
>>> Sadly the way CPUID is handled inside of Xen varies between PV and HVM.
>>> On PV guests AFAICT we mostly do black-listing (I think this is the
>>> right term), which means we pick the native CPUID result and then
>>> perform a series of filter operations in order to remove features which
>>> should not be exposed to a PV guest. On the other hand, for HVM guests
>>> we pre-populate an array (d->arch.cpuids) during domain build time, and
>>> the contents of that array is what is returned to the guest when a CPUID
>>> instruction is executed.
>> This d->arch.cpuids[] mechanism is common to HVM and PV; the
>> exception really is Dom0.
> 
> Dom0 is not special when it comes to cpuid, and shouldn't be treated as
> such.  My longter term CPUID plans will be fixing this.

In some way it is - there's no need for hiding features from it, since
it can't be migrated.

Jan

Re: A couple of HVMlite loose ends

[Andrew Cooper]

On 13/01/16 16:26, Jan Beulich wrote:
>>>> On 13.01.16 at 17:17, <andrew.cooper3@citrix.com> wrote:
>> On 13/01/16 16:13, Jan Beulich wrote:
>>>>>> On 13.01.16 at 16:49, <roger.pau@citrix.com> wrote:
>>>> Hello,
>>>>
>>>> While working on a HVMlite Dom0 implementation I've found a couple of
>>>> loose ends with the design that I would like to comment because it's not
>>>> clear to me what's the best direction to take.
>>>>
>>>> 1. HVM CPUID and Dom0.
>>>>
>>>> Sadly the way CPUID is handled inside of Xen varies between PV and HVM.
>>>> On PV guests AFAICT we mostly do black-listing (I think this is the
>>>> right term), which means we pick the native CPUID result and then
>>>> perform a series of filter operations in order to remove features which
>>>> should not be exposed to a PV guest. On the other hand, for HVM guests
>>>> we pre-populate an array (d->arch.cpuids) during domain build time, and
>>>> the contents of that array is what is returned to the guest when a CPUID
>>>> instruction is executed.
>>> This d->arch.cpuids[] mechanism is common to HVM and PV; the
>>> exception really is Dom0.
>> Dom0 is not special when it comes to cpuid, and shouldn't be treated as
>> such.  My longter term CPUID plans will be fixing this.
> In some way it is - there's no need for hiding features from it, since
> it can't be migrated.

Thats perfectly fine and normal.  The same applies to all other domains
which wont migrate, or will only migrate to identical hardware.

~Andrew

Re: A couple of HVMlite loose ends

[Boris Ostrovsky]

On 01/13/2016 11:30 AM, Andrew Cooper wrote:
> On 13/01/16 16:26, Jan Beulich wrote:
>>>>> On 13.01.16 at 17:17, <andrew.cooper3@citrix.com> wrote:
>>> On 13/01/16 16:13, Jan Beulich wrote:
>>>>>>> On 13.01.16 at 16:49, <roger.pau@citrix.com> wrote:
>>>>> Hello,
>>>>>
>>>>> While working on a HVMlite Dom0 implementation I've found a couple of
>>>>> loose ends with the design that I would like to comment because it's not
>>>>> clear to me what's the best direction to take.
>>>>>
>>>>> 1. HVM CPUID and Dom0.
>>>>>
>>>>> Sadly the way CPUID is handled inside of Xen varies between PV and HVM.
>>>>> On PV guests AFAICT we mostly do black-listing (I think this is the
>>>>> right term), which means we pick the native CPUID result and then
>>>>> perform a series of filter operations in order to remove features which
>>>>> should not be exposed to a PV guest. On the other hand, for HVM guests
>>>>> we pre-populate an array (d->arch.cpuids) during domain build time, and
>>>>> the contents of that array is what is returned to the guest when a CPUID
>>>>> instruction is executed.
>>>> This d->arch.cpuids[] mechanism is common to HVM and PV; the
>>>> exception really is Dom0.
>>> Dom0 is not special when it comes to cpuid, and shouldn't be treated as
>>> such.  My longter term CPUID plans will be fixing this.
>> In some way it is - there's no need for hiding features from it, since
>> it can't be migrated.
> Thats perfectly fine and normal.  The same applies to all other domains
> which wont migrate, or will only migrate to identical hardware.

We don't use CPUID to only turn on and off features. It also, for 
example, describes topology -- something that we may use one day 
(possibly for dom0 too).


-boris

Re: A couple of HVMlite loose ends

[Andrew Cooper]

On 13/01/16 17:03, Boris Ostrovsky wrote:
> On 01/13/2016 11:30 AM, Andrew Cooper wrote:
>> On 13/01/16 16:26, Jan Beulich wrote:
>>>>>> On 13.01.16 at 17:17, <andrew.cooper3@citrix.com> wrote:
>>>> On 13/01/16 16:13, Jan Beulich wrote:
>>>>>>>> On 13.01.16 at 16:49, <roger.pau@citrix.com> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> While working on a HVMlite Dom0 implementation I've found a
>>>>>> couple of
>>>>>> loose ends with the design that I would like to comment because
>>>>>> it's not
>>>>>> clear to me what's the best direction to take.
>>>>>>
>>>>>> 1. HVM CPUID and Dom0.
>>>>>>
>>>>>> Sadly the way CPUID is handled inside of Xen varies between PV
>>>>>> and HVM.
>>>>>> On PV guests AFAICT we mostly do black-listing (I think this is the
>>>>>> right term), which means we pick the native CPUID result and then
>>>>>> perform a series of filter operations in order to remove features
>>>>>> which
>>>>>> should not be exposed to a PV guest. On the other hand, for HVM
>>>>>> guests
>>>>>> we pre-populate an array (d->arch.cpuids) during domain build
>>>>>> time, and
>>>>>> the contents of that array is what is returned to the guest when
>>>>>> a CPUID
>>>>>> instruction is executed.
>>>>> This d->arch.cpuids[] mechanism is common to HVM and PV; the
>>>>> exception really is Dom0.
>>>> Dom0 is not special when it comes to cpuid, and shouldn't be
>>>> treated as
>>>> such.  My longter term CPUID plans will be fixing this.
>>> In some way it is - there's no need for hiding features from it, since
>>> it can't be migrated.
>> Thats perfectly fine and normal.  The same applies to all other domains
>> which wont migrate, or will only migrate to identical hardware.
>
> We don't use CPUID to only turn on and off features. It also, for
> example, describes topology -- something that we may use one day
> (possibly for dom0 too).

It still doesn't make dom0 special compared to other guests.  All guests
(should) have accurate topology information represented in cpuid (I will
also be fixing this).

~Andrew