[RFC] When to use "domain creation flag" or "HVM param"?

[RFC] When to use "domain creation flag" or "HVM param"?

[Don Slutz]

Currently Jan Beulich is not happy with the addition of a new domain
creation flag.  Andrew Cooper is not happy with a HVM param.  I am stuck
in the middle.

This is from:

On 02/23/15 11:28, Jan Beulich wrote:
>> I was not sure on the way to go based on the emails.
>
> I think I said so before - we should come to some conclusion among
> maintainers here first. I'm afraid besides Andrew and me no-one
> really voiced any opinion; maybe worth for you to send out a
> separate request-for-comments on this specific aspect...



Below is some of the subsets of the email threads on this:



Message-ID: <54DA5C69.1060409@terremark.com>
References: <1412285417-19180-1-git-send-email-dslutz@verizon.com>
<1412285417-19180-5-git-send-email-dslutz@verizon.com>
<542DCA92.1030701@terremark.com> <542DD44F.6030101@terremark.com>
<54B8F1740200007800055B42@mail.emea.novell.com>
<54BFE768.3090309@terremark.com>
<54C0C39F0200007800057F73@mail.emea.novell.com> <54C6643B.1@terremark.com>
In-Reply-To: <54C6643B.1@terremark.com>
Subject: Re: [PATCH v8 4/7] xen: Add vmware_port support
Date: Tue, 10 Feb 2015 14:30:49 -0500
From: Don Slutz <dslutz@verizon.com>
To: Don Slutz <dslutz@verizon.com>, Jan Beulich <JBeulich@suse.com>
CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>, Andrew Cooper
<andrew.cooper3@citrix.com>, Ian Campbell <ian.campbell@citrix.com>,
George Dunlap <George.Dunlap@eu.citrix.com>, Ian Jackson
<ian.jackson@eu.citrix.com>, Stefano Stabellini
<stefano.stabellini@eu.citrix.com>, Eddie Dong <eddie.dong@intel.com>,
Jun Nakajima <jun.nakajima@intel.com>, Kevin Tian
<kevin.tian@intel.com>, xen-devel@lists.xen.org, Boris Ostrovsky
<boris.ostrovsky@oracle.com>, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com>, Keir Fraser <keir@xen.org>, Tim Deegan
<tim@xen.org>

On 01/26/15 10:58, Don Slutz wrote:
> On 01/22/15 03:32, Jan Beulich wrote:
>>>>> On 21.01.15 at 18:52, <dslutz@verizon.com> wrote:
>>> On 01/16/15 05:09, Jan Beulich wrote:
>>>>>>> On 03.10.14 at 00:40, <dslutz@verizon.com> wrote:
>>>>> This is a new domain_create() flag, DOMCRF_vmware_port.  It is
>>>>> passed to domctl as XEN_DOMCTL_CDF_vmware_port.
>>>> Can you explain why a HVM param isn't suitable here?
>>>>
>>> The issue is that you need this flag during construct_vmcb() and
>>> construct_vmcs().  While Intel has vmx_update_exception_bitmap()
>>> AMD does not.  So when HVM param's are setup and/or changed there
>>> currently is no way to adjust AMD's exception bitmap.
>>>
>>> So this is the simpler way.
>> But the less desirable one from a design/consistency perspective.
>> Unless other maintainers disagree, I'd like to see this changed.
>
> Ok, but will wait some time to see if "Unless other maintainers disagree"
>

While coding this is up I have hit issues that I need input on:

As a HVM_PARAM_ item, I would assume I should be following
what HVM_PARAM_VIRIDIAN does.  It has this comment:

            case HVM_PARAM_VIRIDIAN:
                /* This should only ever be set once by the tools and
read by the guest. */

Which is almost true.  However the code allows you to change from 0 to
non-zero any time in the life of the DomU.  I am assuming that this is
why xc_domain_save() and xc_domain_restore() save and restore this
HVM_PARAM_ item.

With the enable of vmware_port the same way, I feel it would be a bug
to allow the enable after "create" to not also adjust QEMU.  Currently
there is no way for the hypervisor to tell QEMU to enable vmware_port
handling.  So to avoid adding this code to xen and QEMU, it looks to
me that adding code to make this a true write only 1 time would be
needed so that you cannot use the hyper call to change later.

So, should I extend this change to cover other HVM_PARAM_?

Is all this additional code (xc_domain_save(), xc_domain_restore(),
write only 1 time) still better then a domain_create() flag?

   -Don Slutz





Message-ID: <54E2FF9202000078000607A7@mail.emea.novell.com>
References: <1412285417-19180-1-git-send-email-dslutz@verizon.com>
 <1412285417-19180-5-git-send-email-dslutz@verizon.com>
 <542DCA92.1030701@terremark.com> <542DD44F.6030101@terremark.com>
 <54B8F1740200007800055B42@mail.emea.novell.com>
 <54BFE768.3090309@terremark.com>
 <54C0C39F0200007800057F73@mail.emea.novell.com> <54C6643B.1@terremark.com>
 <54DA5C69.1060409@terremark.com>
 <54DB193F020000780005ED8F@mail.emea.novell.com>
 <54DB8B83.9000606@citrix.com>
In-Reply-To: <54DB8B83.9000606@citrix.com>Subject: Re: [PATCH v8 4/7]
xen: Add vmware_port support
Date: Tue, 17 Feb 2015 07:45:06 +0000
From: Jan Beulich <JBeulich@suse.com>
To: Andrew Cooper <andrew.cooper3@citrix.com>, Don Slutz
<dslutz@verizon.com>
CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>, Ian Campbell
<ian.campbell@citrix.com>, George Dunlap <George.Dunlap@eu.citrix.com>,
IanJackson <ian.jackson@eu.citrix.com>, Stefano Stabellini
<stefano.stabellini@eu.citrix.com>, Eddie Dong <eddie.dong@intel.com>,
JunNakajima <jun.nakajima@intel.com>, Kevin Tian <kevin.tian@intel.com>,
xen-devel@lists.xen.org, Boris Ostrovsky <boris.ostrovsky@oracle.com>,
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, Keir Fraser
<keir@xen.org>, Tim Deegan <tim@xen.org>

>>> On 11.02.15 at 18:04, <andrew.cooper3@citrix.com> wrote:
> On 11/02/15 07:56, Jan Beulich wrote:
>>>>> On 10.02.15 at 20:30, <dslutz@verizon.com> wrote:
>>> While coding this is up I have hit issues that I need input on:
>>>
>>> As a HVM_PARAM_ item, I would assume I should be following
>>> what HVM_PARAM_VIRIDIAN does.  It has this comment:
>>>
>>>             case HVM_PARAM_VIRIDIAN:
>>>                 /* This should only ever be set once by the tools and
>>> read by the guest. */
>>>
>>> Which is almost true.  However the code allows you to change from 0 to
>>> non-zero any time in the life of the DomU.  I am assuming that this is
>>> why xc_domain_save() and xc_domain_restore() save and restore this
>>> HVM_PARAM_ item.>> I was not sure on the way to go based on the emails.
>
> I think I said so before - we should come to some conclusion among
> maintainers here first. I'm afraid besides Andrew and me no-one
> really voiced any opinion; maybe worth for you to send out a
> separate request-for-comments on this specific aspect...
>>>
>>> With the enable of vmware_port the same way, I feel it would be a bug
>>> to allow the enable after "create" to not also adjust QEMU.  Currently
>>> there is no way for the hypervisor to tell QEMU to enable vmware_port
>>> handling.  So to avoid adding this code to xen and QEMU, it looks to
>>> me that adding code to make this a true write only 1 time would be
>>> needed so that you cannot use the hyper call to change later.
>>>
>>> So, should I extend this change to cover other HVM_PARAM_?
>>>
>>> Is all this additional code (xc_domain_save(), xc_domain_restore(),
>>> write only 1 time) still better then a domain_create() flag?
>> I suppose for your case it's indeed the right approach. Which other
>> params this may be true for as well I can't immediately say, but I'd
>> certainly like to ask for adjustments to others to be in separate
>> patches (and perhaps even a separate series), with proper
>> rationale for each of them. I guess Andrew will have further input
>> for you on this matter...
>
> My recommendation is still to use a creation flag.  The described
> problem is exactly the reason why I dislike the use of hvmparams for
> booleans like this which really do need to be consistent for the
> lifetime of the guest.

While I can see your point, HVM params are much better scalable
(and more obviously architecture specific) than creation flags...

Jan




Message-ID: <54EB4FE002000078000628F7@mail.emea.novell.com>
References: <1424127915-27004-1-git-send-email-dslutz@verizon.com>
 <1424127915-27004-6-git-send-email-dslutz@verizon.com>
In-Reply-To: <1424127915-27004-6-git-send-email-dslutz@verizon.com>
Subject: Re: [PATCH v9 05/13] xen: Add vmware_port support
Date: Mon, 23 Feb 2015 15:05:52 +0000
From: Jan Beulich <JBeulich@suse.com>
To: Don Slutz <dslutz@verizon.com>
CC: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>, Suravee
Suthikulpanit <suravee.suthikulpanit@amd.com>, Andrew Cooper
<andrew.cooper3@citrix.com>, Ian Campbell <ian.campbell@citrix.com>,
George Dunlap <George.Dunlap@eu.citrix.com>, Ian Jackson
<ian.jackson@eu.citrix.com>, Stefano Stabellini
<stefano.stabellini@eu.citrix.com>, Eddie Dong <eddie.dong@intel.com>,
Jun Nakajima <jun.nakajima@intel.com>, Kevin Tian
<kevin.tian@intel.com>, xen-devel@lists.xen.org, Boris Ostrovsky
<boris.ostrovsky@oracle.com>, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com>, Keir Fraser <keir@xen.org>, Tim Deegan
<tim@xen.org>

>>> On 17.02.15 at 00:05, <dslutz@verizon.com> wrote:
> This includes adding is_vmware_port_enabled
>
> This is a new domain_create() flag, DOMCRF_vmware_port.  It is
> passed to domctl as XEN_DOMCTL_CDF_vmware_port.

As indicated before, I don't think this is a good use case for a
domain creation flag. Some of the others we have may not be either,
but as I have said quite often recently - let's not make the situation
worse by following bad examples.



   -Don Slutz

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Tim Deegan]

At 15:08 -0500 on 23 Feb (1424700515), Don Slutz wrote:
> Currently Jan Beulich is not happy with the addition of a new domain
> creation flag.  Andrew Cooper is not happy with a HVM param.  I am stuck
> in the middle.

I prefer a new flag, for anything that's fixed for the life of the
domain.  We've already had too many bugs where HVM params changed
when people thought they wouldn't.

Jan, is your objection that we'll run out of XEN_DOMCTL_CDF_* bits?  I
think we can add more flag fields to DOMCTL_createdomain (or a v2) if
that becomes a problem.

Cheers,

Tim.


> On 02/23/15 11:28, Jan Beulich wrote:
> >> I was not sure on the way to go based on the emails.
> >
> > I think I said so before - we should come to some conclusion among
> > maintainers here first. I'm afraid besides Andrew and me no-one
> > really voiced any opinion; maybe worth for you to send out a
> > separate request-for-comments on this specific aspect...
> 
> 
> 
> Below is some of the subsets of the email threads on this:
> 
> 
> 
> Message-ID: <54DA5C69.1060409@terremark.com>
> References: <1412285417-19180-1-git-send-email-dslutz@verizon.com>
> <1412285417-19180-5-git-send-email-dslutz@verizon.com>
> <542DCA92.1030701@terremark.com> <542DD44F.6030101@terremark.com>
> <54B8F1740200007800055B42@mail.emea.novell.com>
> <54BFE768.3090309@terremark.com>
> <54C0C39F0200007800057F73@mail.emea.novell.com> <54C6643B.1@terremark.com>
> In-Reply-To: <54C6643B.1@terremark.com>
> Subject: Re: [PATCH v8 4/7] xen: Add vmware_port support
> Date: Tue, 10 Feb 2015 14:30:49 -0500
> From: Don Slutz <dslutz@verizon.com>
> To: Don Slutz <dslutz@verizon.com>, Jan Beulich <JBeulich@suse.com>
> CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>, Andrew Cooper
> <andrew.cooper3@citrix.com>, Ian Campbell <ian.campbell@citrix.com>,
> George Dunlap <George.Dunlap@eu.citrix.com>, Ian Jackson
> <ian.jackson@eu.citrix.com>, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com>, Eddie Dong <eddie.dong@intel.com>,
> Jun Nakajima <jun.nakajima@intel.com>, Kevin Tian
> <kevin.tian@intel.com>, xen-devel@lists.xen.org, Boris Ostrovsky
> <boris.ostrovsky@oracle.com>, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com>, Keir Fraser <keir@xen.org>, Tim Deegan
> <tim@xen.org>
> 
> On 01/26/15 10:58, Don Slutz wrote:
> > On 01/22/15 03:32, Jan Beulich wrote:
> >>>>> On 21.01.15 at 18:52, <dslutz@verizon.com> wrote:
> >>> On 01/16/15 05:09, Jan Beulich wrote:
> >>>>>>> On 03.10.14 at 00:40, <dslutz@verizon.com> wrote:
> >>>>> This is a new domain_create() flag, DOMCRF_vmware_port.  It is
> >>>>> passed to domctl as XEN_DOMCTL_CDF_vmware_port.
> >>>> Can you explain why a HVM param isn't suitable here?
> >>>>
> >>> The issue is that you need this flag during construct_vmcb() and
> >>> construct_vmcs().  While Intel has vmx_update_exception_bitmap()
> >>> AMD does not.  So when HVM param's are setup and/or changed there
> >>> currently is no way to adjust AMD's exception bitmap.
> >>>
> >>> So this is the simpler way.
> >> But the less desirable one from a design/consistency perspective.
> >> Unless other maintainers disagree, I'd like to see this changed.
> >
> > Ok, but will wait some time to see if "Unless other maintainers disagree"
> >
> 
> While coding this is up I have hit issues that I need input on:
> 
> As a HVM_PARAM_ item, I would assume I should be following
> what HVM_PARAM_VIRIDIAN does.  It has this comment:
> 
>             case HVM_PARAM_VIRIDIAN:
>                 /* This should only ever be set once by the tools and
> read by the guest. */
> 
> Which is almost true.  However the code allows you to change from 0 to
> non-zero any time in the life of the DomU.  I am assuming that this is
> why xc_domain_save() and xc_domain_restore() save and restore this
> HVM_PARAM_ item.
> 
> With the enable of vmware_port the same way, I feel it would be a bug
> to allow the enable after "create" to not also adjust QEMU.  Currently
> there is no way for the hypervisor to tell QEMU to enable vmware_port
> handling.  So to avoid adding this code to xen and QEMU, it looks to
> me that adding code to make this a true write only 1 time would be
> needed so that you cannot use the hyper call to change later.
> 
> So, should I extend this change to cover other HVM_PARAM_?
> 
> Is all this additional code (xc_domain_save(), xc_domain_restore(),
> write only 1 time) still better then a domain_create() flag?
> 
>    -Don Slutz
> 
> 
> 
> 
> 
> Message-ID: <54E2FF9202000078000607A7@mail.emea.novell.com>
> References: <1412285417-19180-1-git-send-email-dslutz@verizon.com>
>  <1412285417-19180-5-git-send-email-dslutz@verizon.com>
>  <542DCA92.1030701@terremark.com> <542DD44F.6030101@terremark.com>
>  <54B8F1740200007800055B42@mail.emea.novell.com>
>  <54BFE768.3090309@terremark.com>
>  <54C0C39F0200007800057F73@mail.emea.novell.com> <54C6643B.1@terremark.com>
>  <54DA5C69.1060409@terremark.com>
>  <54DB193F020000780005ED8F@mail.emea.novell.com>
>  <54DB8B83.9000606@citrix.com>
> In-Reply-To: <54DB8B83.9000606@citrix.com>Subject: Re: [PATCH v8 4/7]
> xen: Add vmware_port support
> Date: Tue, 17 Feb 2015 07:45:06 +0000
> From: Jan Beulich <JBeulich@suse.com>
> To: Andrew Cooper <andrew.cooper3@citrix.com>, Don Slutz
> <dslutz@verizon.com>
> CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>, Ian Campbell
> <ian.campbell@citrix.com>, George Dunlap <George.Dunlap@eu.citrix.com>,
> IanJackson <ian.jackson@eu.citrix.com>, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com>, Eddie Dong <eddie.dong@intel.com>,
> JunNakajima <jun.nakajima@intel.com>, Kevin Tian <kevin.tian@intel.com>,
> xen-devel@lists.xen.org, Boris Ostrovsky <boris.ostrovsky@oracle.com>,
> Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, Keir Fraser
> <keir@xen.org>, Tim Deegan <tim@xen.org>
> 
> >>> On 11.02.15 at 18:04, <andrew.cooper3@citrix.com> wrote:
> > On 11/02/15 07:56, Jan Beulich wrote:
> >>>>> On 10.02.15 at 20:30, <dslutz@verizon.com> wrote:
> >>> While coding this is up I have hit issues that I need input on:
> >>>
> >>> As a HVM_PARAM_ item, I would assume I should be following
> >>> what HVM_PARAM_VIRIDIAN does.  It has this comment:
> >>>
> >>>             case HVM_PARAM_VIRIDIAN:
> >>>                 /* This should only ever be set once by the tools and
> >>> read by the guest. */
> >>>
> >>> Which is almost true.  However the code allows you to change from 0 to
> >>> non-zero any time in the life of the DomU.  I am assuming that this is
> >>> why xc_domain_save() and xc_domain_restore() save and restore this
> >>> HVM_PARAM_ item.>> I was not sure on the way to go based on the emails.
> >
> > I think I said so before - we should come to some conclusion among
> > maintainers here first. I'm afraid besides Andrew and me no-one
> > really voiced any opinion; maybe worth for you to send out a
> > separate request-for-comments on this specific aspect...
> >>>
> >>> With the enable of vmware_port the same way, I feel it would be a bug
> >>> to allow the enable after "create" to not also adjust QEMU.  Currently
> >>> there is no way for the hypervisor to tell QEMU to enable vmware_port
> >>> handling.  So to avoid adding this code to xen and QEMU, it looks to
> >>> me that adding code to make this a true write only 1 time would be
> >>> needed so that you cannot use the hyper call to change later.
> >>>
> >>> So, should I extend this change to cover other HVM_PARAM_?
> >>>
> >>> Is all this additional code (xc_domain_save(), xc_domain_restore(),
> >>> write only 1 time) still better then a domain_create() flag?
> >> I suppose for your case it's indeed the right approach. Which other
> >> params this may be true for as well I can't immediately say, but I'd
> >> certainly like to ask for adjustments to others to be in separate
> >> patches (and perhaps even a separate series), with proper
> >> rationale for each of them. I guess Andrew will have further input
> >> for you on this matter...
> >
> > My recommendation is still to use a creation flag.  The described
> > problem is exactly the reason why I dislike the use of hvmparams for
> > booleans like this which really do need to be consistent for the
> > lifetime of the guest.
> 
> While I can see your point, HVM params are much better scalable
> (and more obviously architecture specific) than creation flags...
> 
> Jan
> 
> 
> 
> 
> Message-ID: <54EB4FE002000078000628F7@mail.emea.novell.com>
> References: <1424127915-27004-1-git-send-email-dslutz@verizon.com>
>  <1424127915-27004-6-git-send-email-dslutz@verizon.com>
> In-Reply-To: <1424127915-27004-6-git-send-email-dslutz@verizon.com>
> Subject: Re: [PATCH v9 05/13] xen: Add vmware_port support
> Date: Mon, 23 Feb 2015 15:05:52 +0000
> From: Jan Beulich <JBeulich@suse.com>
> To: Don Slutz <dslutz@verizon.com>
> CC: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>, Suravee
> Suthikulpanit <suravee.suthikulpanit@amd.com>, Andrew Cooper
> <andrew.cooper3@citrix.com>, Ian Campbell <ian.campbell@citrix.com>,
> George Dunlap <George.Dunlap@eu.citrix.com>, Ian Jackson
> <ian.jackson@eu.citrix.com>, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com>, Eddie Dong <eddie.dong@intel.com>,
> Jun Nakajima <jun.nakajima@intel.com>, Kevin Tian
> <kevin.tian@intel.com>, xen-devel@lists.xen.org, Boris Ostrovsky
> <boris.ostrovsky@oracle.com>, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com>, Keir Fraser <keir@xen.org>, Tim Deegan
> <tim@xen.org>
> 
> >>> On 17.02.15 at 00:05, <dslutz@verizon.com> wrote:
> > This includes adding is_vmware_port_enabled
> >
> > This is a new domain_create() flag, DOMCRF_vmware_port.  It is
> > passed to domctl as XEN_DOMCTL_CDF_vmware_port.
> 
> As indicated before, I don't think this is a good use case for a
> domain creation flag. Some of the others we have may not be either,
> but as I have said quite often recently - let's not make the situation
> worse by following bad examples.
> 
> 
> 
>    -Don Slutz

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Jan Beulich]

>>> On 24.02.15 at 11:24, <tim@xen.org> wrote:
> At 15:08 -0500 on 23 Feb (1424700515), Don Slutz wrote:
>> Currently Jan Beulich is not happy with the addition of a new domain
>> creation flag.  Andrew Cooper is not happy with a HVM param.  I am stuck
>> in the middle.
> 
> I prefer a new flag, for anything that's fixed for the life of the
> domain.  We've already had too many bugs where HVM params changed
> when people thought they wouldn't.
> 
> Jan, is your objection that we'll run out of XEN_DOMCTL_CDF_* bits?  I
> think we can add more flag fields to DOMCTL_createdomain (or a v2) if
> that becomes a problem.

In a couple of years we may end up with an x86-CPUID-like mess
of hundreds of flags. And apart from that scalability issue I also
dislike the gross mixing of arch specific and generic flags here.
Perhaps the already arch-specific XEN_DOMCTL_configure_domain
would be the better route then if HVM params are being disliked?

Jan

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Tim Deegan]

At 10:31 +0000 on 24 Feb (1424770274), Jan Beulich wrote:
> >>> On 24.02.15 at 11:24, <tim@xen.org> wrote:
> > At 15:08 -0500 on 23 Feb (1424700515), Don Slutz wrote:
> >> Currently Jan Beulich is not happy with the addition of a new domain
> >> creation flag.  Andrew Cooper is not happy with a HVM param.  I am stuck
> >> in the middle.
> > 
> > I prefer a new flag, for anything that's fixed for the life of the
> > domain.  We've already had too many bugs where HVM params changed
> > when people thought they wouldn't.
> > 
> > Jan, is your objection that we'll run out of XEN_DOMCTL_CDF_* bits?  I
> > think we can add more flag fields to DOMCTL_createdomain (or a v2) if
> > that becomes a problem.
> 
> In a couple of years we may end up with an x86-CPUID-like mess
> of hundreds of flags.

I'm not very worried about that.  Looking at the similar hvm params
now (booleans that shouldn't change at runtime), I only count 5 new flags.

> And apart from that scalability issue I also
> dislike the gross mixing of arch specific and generic flags here.

We could namespace that easily enough - start with 16 generic, 16
arch-specific, or just add an arch_flags field right now.

> Perhaps the already arch-specific XEN_DOMCTL_configure_domain
> would be the better route then if HVM params are being disliked?

That appears to return config info from Xen rather than setting it.
But I think I would prefer a create op that took arch-specific flags
to a second op that effectively has to be called exactly at create
time (with the risk that we'll mess up enfoorcing that and have asome
race between flag-setting and e.g. memory allocation).

Tim.

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Andrew Cooper]

On 24/02/15 10:31, Jan Beulich wrote:
>>>> On 24.02.15 at 11:24, <tim@xen.org> wrote:
>> At 15:08 -0500 on 23 Feb (1424700515), Don Slutz wrote:
>>> Currently Jan Beulich is not happy with the addition of a new domain
>>> creation flag.  Andrew Cooper is not happy with a HVM param.  I am stuck
>>> in the middle.
>> I prefer a new flag, for anything that's fixed for the life of the
>> domain.  We've already had too many bugs where HVM params changed
>> when people thought they wouldn't.
>>
>> Jan, is your objection that we'll run out of XEN_DOMCTL_CDF_* bits?  I
>> think we can add more flag fields to DOMCTL_createdomain (or a v2) if
>> that becomes a problem.
> In a couple of years we may end up with an x86-CPUID-like mess
> of hundreds of flags. And apart from that scalability issue I also
> dislike the gross mixing of arch specific and generic flags here.
> Perhaps the already arch-specific XEN_DOMCTL_configure_domain
> would be the better route then if HVM params are being disliked?

Given some recent consideration to the problem of domain architectural
state (x86 cpuid policy, arm gic/spi), a (set of?) configuration
hypercalls valid only during domain construction would perhaps be the
best way to proceed.

Extending createdomain itself is incompatible with XSM disaggregation
and having the architectural state in the migration stream.


The vmware backdoor is however slightly more complicated, in that it
also involves qemu.  What would be the effects be for a domain where Xen
believes the backdoor is active, but qemu is not running appropriately?

~Andrew

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Don Slutz]

On 02/24/15 05:50, Andrew Cooper wrote:
> On 24/02/15 10:31, Jan Beulich wrote:
>>>>> On 24.02.15 at 11:24, <tim@xen.org> wrote:
>>> At 15:08 -0500 on 23 Feb (1424700515), Don Slutz wrote:
>>>> Currently Jan Beulich is not happy with the addition of a new domain
>>>> creation flag.  Andrew Cooper is not happy with a HVM param.  I am stuck
>>>> in the middle.

> The vmware backdoor is however slightly more complicated, in that it
> also involves qemu.  What would be the effects be for a domain where Xen
> believes the backdoor is active, but qemu is not running appropriately?
> 

Right now for a QEMU that does not support vmport=on as a machine
property, QEMU will terminate and the domain will not be created.

If QEMU is started without this and Xen then activates the backdoor,
QEMU will act like any other unsupported ioport access.  This does mean
that you will lose the use of a mouse under current Xorg GUI.

I have not tried to hot-plug the vmport and vmmouse devices in QEMU.
That may take some changes to work.

    -Don Slutz

> ~Andrew
> 

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Tim Deegan]

At 10:50 +0000 on 24 Feb (1424771408), Andrew Cooper wrote:
> On 24/02/15 10:31, Jan Beulich wrote:
> >>>> On 24.02.15 at 11:24, <tim@xen.org> wrote:
> >> At 15:08 -0500 on 23 Feb (1424700515), Don Slutz wrote:
> >>> Currently Jan Beulich is not happy with the addition of a new domain
> >>> creation flag.  Andrew Cooper is not happy with a HVM param.  I am stuck
> >>> in the middle.
> >> I prefer a new flag, for anything that's fixed for the life of the
> >> domain.  We've already had too many bugs where HVM params changed
> >> when people thought they wouldn't.
> >>
> >> Jan, is your objection that we'll run out of XEN_DOMCTL_CDF_* bits?  I
> >> think we can add more flag fields to DOMCTL_createdomain (or a v2) if
> >> that becomes a problem.
> > In a couple of years we may end up with an x86-CPUID-like mess
> > of hundreds of flags. And apart from that scalability issue I also
> > dislike the gross mixing of arch specific and generic flags here.
> > Perhaps the already arch-specific XEN_DOMCTL_configure_domain
> > would be the better route then if HVM params are being disliked?
> 
> Given some recent consideration to the problem of domain architectural
> state (x86 cpuid policy, arm gic/spi), a (set of?) configuration
> hypercalls valid only during domain construction would perhaps be the
> best way to proceed.

That sounds like you're also arguing for using HVM params then.  :)

The nice thing about having a single set of flags at creation time is
that it avoids any worrying about what order the flag-setting and the
init of VM state happens in (e.g. turning on a feature after vcpus are
already assigned means extra code to update them; having the features
be settable in any order means handling all O(N^2) interactions).

> Extending createdomain itself is incompatible with XSM disaggregation

Hrm.  Possibly, but I think that might be splitting hairs.
A privileged VM creation component will already have to check its
arguments (e.g. memory size, #vcpus, boot image) against some policy.

> and having the architectural state in the migration stream.

Not at all -- since these flags are immutable, you can trivially send
them before any other state.  If a toolstack can't remember what it
did, we could add a what-were-the-creation-flags domctl.

Tim.

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Lars Kurth]

Tim, Andrew, Jan,
it seems as if we are slowly coming to some conclusion on this thread. If
I am mistaken, I am wondering whether it would make sense to have an IRC
meeting with all the involved stake-holders and report back to the list.
Regards
Lars

On 26/02/2015 10:52, "Tim Deegan" <tim@xen.org> wrote:

>At 10:50 +0000 on 24 Feb (1424771408), Andrew Cooper wrote:
>> On 24/02/15 10:31, Jan Beulich wrote:
>> >>>> On 24.02.15 at 11:24, <tim@xen.org> wrote:
>> >> At 15:08 -0500 on 23 Feb (1424700515), Don Slutz wrote:
>> >>> Currently Jan Beulich is not happy with the addition of a new domain
>> >>> creation flag.  Andrew Cooper is not happy with a HVM param.  I am
>>stuck
>> >>> in the middle.
>> >> I prefer a new flag, for anything that's fixed for the life of the
>> >> domain.  We've already had too many bugs where HVM params changed
>> >> when people thought they wouldn't.
>> >>
>> >> Jan, is your objection that we'll run out of XEN_DOMCTL_CDF_* bits?
>>I
>> >> think we can add more flag fields to DOMCTL_createdomain (or a v2) if
>> >> that becomes a problem.
>> > In a couple of years we may end up with an x86-CPUID-like mess
>> > of hundreds of flags. And apart from that scalability issue I also
>> > dislike the gross mixing of arch specific and generic flags here.
>> > Perhaps the already arch-specific XEN_DOMCTL_configure_domain
>> > would be the better route then if HVM params are being disliked?
>> 
>> Given some recent consideration to the problem of domain architectural
>> state (x86 cpuid policy, arm gic/spi), a (set of?) configuration
>> hypercalls valid only during domain construction would perhaps be the
>> best way to proceed.
>
>That sounds like you're also arguing for using HVM params then.  :)
>
>The nice thing about having a single set of flags at creation time is
>that it avoids any worrying about what order the flag-setting and the
>init of VM state happens in (e.g. turning on a feature after vcpus are
>already assigned means extra code to update them; having the features
>be settable in any order means handling all O(N^2) interactions).
>
>> Extending createdomain itself is incompatible with XSM disaggregation
>
>Hrm.  Possibly, but I think that might be splitting hairs.
>A privileged VM creation component will already have to check its
>arguments (e.g. memory size, #vcpus, boot image) against some policy.
>
>> and having the architectural state in the migration stream.
>
>Not at all -- since these flags are immutable, you can trivially send
>them before any other state.  If a toolstack can't remember what it
>did, we could add a what-were-the-creation-flags domctl.
>
>Tim.

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Julien Grall]

Hi,

On 26/02/15 11:09, Lars Kurth wrote:
> Tim, Andrew, Jan,
> it seems as if we are slowly coming to some conclusion on this thread. If
> I am mistaken, I am wondering whether it would make sense to have an IRC
> meeting with all the involved stake-holders and report back to the list.

I'm not sure where I should answer...

We have a similar problem on ARM where we have arch-specific information
(GIC version, number of interrupts) which changes between each domain.

On Xen 4.5, we took the approach to create a separate DOMCTL for passing
information. It has to be called before any VCPUs is created
(DOMCTL_set_max_vcpus) and make the code more complicate to handle
because we have to defer some domain initialization.

I took another approach for Xen 4.6 based on Jan suggestion [1]. A v3 as
been send recently [2] and we had some discussion about what is the best
approach.

I hope this will help to sort out a good approach for both ARM and x86.

Regards,

[1] http://lists.xen.org/archives/html/xen-devel/2014-11/msg00522.html
[2] http://lists.xen.org/archives/html/xen-devel/2015-01/msg01184.html

-- 
Julien Grall

Re: [RFC] When to use "domain creation flag" or "HVM param"?

[Tim Deegan]

At 15:33 +0000 on 26 Feb (1424961188), Julien Grall wrote:
> Hi,
> 
> On 26/02/15 11:09, Lars Kurth wrote:
> > Tim, Andrew, Jan,
> > it seems as if we are slowly coming to some conclusion on this thread. If
> > I am mistaken, I am wondering whether it would make sense to have an IRC
> > meeting with all the involved stake-holders and report back to the list.
> 
> I'm not sure where I should answer...
> 
> We have a similar problem on ARM where we have arch-specific information
> (GIC version, number of interrupts) which changes between each domain.
> 
> On Xen 4.5, we took the approach to create a separate DOMCTL for passing
> information. It has to be called before any VCPUs is created
> (DOMCTL_set_max_vcpus) and make the code more complicate to handle
> because we have to defer some domain initialization.
> 
> I took another approach for Xen 4.6 based on Jan suggestion [1]. A v3 as
> been send recently [2] and we had some discussion about what is the best
> approach.

This line (adding these immutable config options at create time) seems
like a good one to me.

For migration, we'd need a hypercall that lets the Xen tools extract
the correct values to pass to the receiving Xen.  Xen would fill in
the actual values used for anything (like this GIC option) that
was set to 'default' or 'don't care' on the initial create op.

Andrew Cooper had some reasons why we might want to split this into a
bare create op (which might do no more than allocate a domid) and a
set-config op that would take these and all other immutable flags.
I'm not wild for that but could be convinced either way -- I'll let
him fill in the details.

Cheers,

Tim.