Xen Project policy on feature flags

Xen Project policy on feature flags

[Stefano Stabellini]

Hello all,
I am writing to request a clarification on Xen feature flags
(XENFEAT_*) and backward compatibility:
    
is the hypervisor allowed to remove any feature flags in a future
release, even though doing so might break some existing guests?

For example one could write a PV on HVM guest that requires
XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
release remove that feature? Or is it now part of our ABI, therefore
maintained for backward compatibility, following the rule that we don't
break existing guests?


I always thought that any XENFEAT feature flags could be removed going
forward, if the hypervisor maintainers decide to do so. However Ian
Campbell is of the opposite opinion, so I think we should have a clear
policy regarding them.

In any case I think that it is generally useful to have optional flags
that advertise the presence of a feature but can also be removed going
forward. If XENFEAT feature flags are not them, then we might still want
to introduce them as a separate concept.

Cheers,

Stefano



P.S.
FYI the question has been raised in regards to
XENFEAT_grant_map_identity: we know now that it is not a complete
solution to the problem it was introduced to solve and we'll probably
want to remove it if/when we'll find a better alternative.
However if we cannot remove the feature in the future we would need to
get rid of it now, before is casted in stone in our ABI.

Re: Xen Project policy on feature flags

[Jan Beulich]

>>> On 26.09.14 at 15:24, <stefano.stabellini@eu.citrix.com> wrote:
> I am writing to request a clarification on Xen feature flags
> (XENFEAT_*) and backward compatibility:
>     
> is the hypervisor allowed to remove any feature flags in a future
> release, even though doing so might break some existing guests?
> 
> For example one could write a PV on HVM guest that requires
> XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
> release remove that feature? Or is it now part of our ABI, therefore
> maintained for backward compatibility, following the rule that we don't
> break existing guests?
> 
> 
> I always thought that any XENFEAT feature flags could be removed going
> forward, if the hypervisor maintainers decide to do so. However Ian
> Campbell is of the opposite opinion, so I think we should have a clear
> policy regarding them.
> 
> In any case I think that it is generally useful to have optional flags
> that advertise the presence of a feature but can also be removed going
> forward. If XENFEAT feature flags are not them, then we might still want
> to introduce them as a separate concept.

My view is that these are precisely there to indicate what the
hypervisor supports. I.e. while we can't remove the definition
from the public header, the hypervisor could stop advertising that
it's capable of a certain feature at any time. Consumers are
expected to check for the feature flag and deal with it being off.

Jan

Re: Xen Project policy on feature flags

[Andrew Cooper]

On 26/09/14 14:39, Jan Beulich wrote:
>>>> On 26.09.14 at 15:24, <stefano.stabellini@eu.citrix.com> wrote:
>> I am writing to request a clarification on Xen feature flags
>> (XENFEAT_*) and backward compatibility:
>>     
>> is the hypervisor allowed to remove any feature flags in a future
>> release, even though doing so might break some existing guests?
>>
>> For example one could write a PV on HVM guest that requires
>> XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
>> release remove that feature? Or is it now part of our ABI, therefore
>> maintained for backward compatibility, following the rule that we don't
>> break existing guests?
>>
>>
>> I always thought that any XENFEAT feature flags could be removed going
>> forward, if the hypervisor maintainers decide to do so. However Ian
>> Campbell is of the opposite opinion, so I think we should have a clear
>> policy regarding them.
>>
>> In any case I think that it is generally useful to have optional flags
>> that advertise the presence of a feature but can also be removed going
>> forward. If XENFEAT feature flags are not them, then we might still want
>> to introduce them as a separate concept.
> My view is that these are precisely there to indicate what the
> hypervisor supports. I.e. while we can't remove the definition
> from the public header, the hypervisor could stop advertising that
> it's capable of a certain feature at any time. Consumers are
> expected to check for the feature flag and deal with it being off.

Agreed.  It is an administrator policy whether their deployed Xen
supports all the features, or a subset.  (e.g. disabling features for
embedded or security-surface reasons)

In this case it is fine for a guest not to function if its minimum
required featureset not completely supported by Xen, but it should fail
with "Xen doesn't support required feature $FOO".

~Andrew

Re: Xen Project policy on feature flags

[Ian Campbell]

On Fri, 2014-09-26 at 14:39 +0100, Jan Beulich wrote:
> >>> On 26.09.14 at 15:24, <stefano.stabellini@eu.citrix.com> wrote:
> > I am writing to request a clarification on Xen feature flags
> > (XENFEAT_*) and backward compatibility:
> >     
> > is the hypervisor allowed to remove any feature flags in a future
> > release, even though doing so might break some existing guests?
> > 
> > For example one could write a PV on HVM guest that requires
> > XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
> > release remove that feature? Or is it now part of our ABI, therefore
> > maintained for backward compatibility, following the rule that we don't
> > break existing guests?
> > 
> > 
> > I always thought that any XENFEAT feature flags could be removed going
> > forward, if the hypervisor maintainers decide to do so. However Ian
> > Campbell is of the opposite opinion, so I think we should have a clear
> > policy regarding them.
> > 
> > In any case I think that it is generally useful to have optional flags
> > that advertise the presence of a feature but can also be removed going
> > forward. If XENFEAT feature flags are not them, then we might still want
> > to introduce them as a separate concept.
> 
> My view is that these are precisely there to indicate what the
> hypervisor supports. I.e. while we can't remove the definition
> from the public header, the hypervisor could stop advertising that
> it's capable of a certain feature at any time. Consumers are
> expected to check for the feature flag and deal with it being off.

What if there is no way to deal with it being off other than crashing or
refusing to work etc?

The context here is XENFEAT_grant_map_identity which turns out not to
work in practice (it doesn't actually allow non-LPAE arm32 dom0 to work,
which it was supposed to). We've not released with it yet (as it stands
it'll be included in 4.5).

We do have some other ideas for an alternative fix to the underlying
issue. But we've not actually tried them yet.

IMHO We need to decide whether to nuke that flag now or in the future
(i.e. post 4.5). Given that it doesn't actually work and that we've not
released with it I'm most in favour of not releasing with it, and given
that I'm in favour of removing it now and not waiting until the last
minute.

If we release with it then we will regress at least Linux 3.16 and 3.17
when we remove this flag.

I don't think we have many other flags which are "work at all" flags
rather than "enhance" flags.

Ian.

Re: Xen Project policy on feature flags

[Konrad Rzeszutek Wilk]

On Fri, Sep 26, 2014 at 03:19:32PM +0100, Ian Campbell wrote:
> On Fri, 2014-09-26 at 14:39 +0100, Jan Beulich wrote:
> > >>> On 26.09.14 at 15:24, <stefano.stabellini@eu.citrix.com> wrote:
> > > I am writing to request a clarification on Xen feature flags
> > > (XENFEAT_*) and backward compatibility:
> > >     
> > > is the hypervisor allowed to remove any feature flags in a future
> > > release, even though doing so might break some existing guests?
> > > 
> > > For example one could write a PV on HVM guest that requires
> > > XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
> > > release remove that feature? Or is it now part of our ABI, therefore
> > > maintained for backward compatibility, following the rule that we don't
> > > break existing guests?
> > > 
> > > 
> > > I always thought that any XENFEAT feature flags could be removed going
> > > forward, if the hypervisor maintainers decide to do so. However Ian
> > > Campbell is of the opposite opinion, so I think we should have a clear
> > > policy regarding them.
> > > 
> > > In any case I think that it is generally useful to have optional flags
> > > that advertise the presence of a feature but can also be removed going
> > > forward. If XENFEAT feature flags are not them, then we might still want
> > > to introduce them as a separate concept.
> > 
> > My view is that these are precisely there to indicate what the
> > hypervisor supports. I.e. while we can't remove the definition
> > from the public header, the hypervisor could stop advertising that
> > it's capable of a certain feature at any time. Consumers are
> > expected to check for the feature flag and deal with it being off.
> 
> What if there is no way to deal with it being off other than crashing or
> refusing to work etc?
> 
> The context here is XENFEAT_grant_map_identity which turns out not to
> work in practice (it doesn't actually allow non-LPAE arm32 dom0 to work,
> which it was supposed to). We've not released with it yet (as it stands
> it'll be included in 4.5).
> 
> We do have some other ideas for an alternative fix to the underlying
> issue. But we've not actually tried them yet.
> 
> IMHO We need to decide whether to nuke that flag now or in the future
> (i.e. post 4.5). Given that it doesn't actually work and that we've not
> released with it I'm most in favour of not releasing with it, and given
> that I'm in favour of removing it now and not waiting until the last
> minute.

I am in favour of reverting things. That is a simple way of fixing
bugs :-)

> 
> If we release with it then we will regress at least Linux 3.16 and 3.17
> when we remove this flag.

Wasn't this feature considered 'experimental' in Linux code?

Is the Linux code able to boot without this flag? Let me rephrase - will
it boot in the same fashion (And with the same bugs) as it did prior
to this functionality being introduced?

> 
> I don't think we have many other flags which are "work at all" flags
> rather than "enhance" flags.
> 
> Ian.
> 

Re: Xen Project policy on feature flags

[Stefano Stabellini]

On Fri, 26 Sep 2014, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 26, 2014 at 03:19:32PM +0100, Ian Campbell wrote:
> > On Fri, 2014-09-26 at 14:39 +0100, Jan Beulich wrote:
> > > >>> On 26.09.14 at 15:24, <stefano.stabellini@eu.citrix.com> wrote:
> > > > I am writing to request a clarification on Xen feature flags
> > > > (XENFEAT_*) and backward compatibility:
> > > >     
> > > > is the hypervisor allowed to remove any feature flags in a future
> > > > release, even though doing so might break some existing guests?
> > > > 
> > > > For example one could write a PV on HVM guest that requires
> > > > XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
> > > > release remove that feature? Or is it now part of our ABI, therefore
> > > > maintained for backward compatibility, following the rule that we don't
> > > > break existing guests?
> > > > 
> > > > 
> > > > I always thought that any XENFEAT feature flags could be removed going
> > > > forward, if the hypervisor maintainers decide to do so. However Ian
> > > > Campbell is of the opposite opinion, so I think we should have a clear
> > > > policy regarding them.
> > > > 
> > > > In any case I think that it is generally useful to have optional flags
> > > > that advertise the presence of a feature but can also be removed going
> > > > forward. If XENFEAT feature flags are not them, then we might still want
> > > > to introduce them as a separate concept.
> > > 
> > > My view is that these are precisely there to indicate what the
> > > hypervisor supports. I.e. while we can't remove the definition
> > > from the public header, the hypervisor could stop advertising that
> > > it's capable of a certain feature at any time. Consumers are
> > > expected to check for the feature flag and deal with it being off.
> > 
> > What if there is no way to deal with it being off other than crashing or
> > refusing to work etc?
> > 
> > The context here is XENFEAT_grant_map_identity which turns out not to
> > work in practice (it doesn't actually allow non-LPAE arm32 dom0 to work,
> > which it was supposed to). We've not released with it yet (as it stands
> > it'll be included in 4.5).
> > 
> > We do have some other ideas for an alternative fix to the underlying
> > issue. But we've not actually tried them yet.
> > 
> > IMHO We need to decide whether to nuke that flag now or in the future
> > (i.e. post 4.5). Given that it doesn't actually work and that we've not
> > released with it I'm most in favour of not releasing with it, and given
> > that I'm in favour of removing it now and not waiting until the last
> > minute.
> 
> I am in favour of reverting things. That is a simple way of fixing
> bugs :-)

Unfortunately in this case it is complicated :-)

The real cause of the problem are Zoltan's changes to netback to switch
to grant mapping operations on ARM. Unfortunately when we realized it,
Linux 3.15 had already been released so we tried to fix the issue
instead. It turns out that the fix doesn't work on non-LPAE dom0.
Reverting the LPAE-only fix still gives you a broken setup.


> > If we release with it then we will regress at least Linux 3.16 and 3.17
> > when we remove this flag.
> 
> Wasn't this feature considered 'experimental' in Linux code?

No


> Is the Linux code able to boot without this flag?

It can boot but later on it would encour into serious issues if the
kernel in question is used as dom0 kernel and you have dma-capable
network devices on your platform.


> Let me rephrase - will it boot in the same fashion (And with the same
> bugs) as it did prior to this functionality being introduced?

3.15 -> dom0 on ARM broken (if netback is used)
3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with CONFIG_ARM_LPAE

Reverting the XENFEAT_grant_map_identity related changes would give you
a system broken even with CONFIG_ARM_LPAE.
Reverting Zoltan's changes to netback would give you a working system.



> > I don't think we have many other flags which are "work at all" flags
> > rather than "enhance" flags.
> > 
> > Ian.
> > 
> 

Re: Xen Project policy on feature flags

[George Dunlap]

On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
>> Let me rephrase - will it boot in the same fashion (And with the same
>> bugs) as it did prior to this functionality being introduced?
> 3.15 -> dom0 on ARM broken (if netback is used)
> 3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with CONFIG_ARM_LPAE
>
> Reverting the XENFEAT_grant_map_identity related changes would give you
> a system broken even with CONFIG_ARM_LPAE.
> Reverting Zoltan's changes to netback would give you a working system.

So the *only* reason to include this flag at this point is to allow 
people to run an unmodified 3.17 kernel with netback, is that right?  It 
seems like a really jerk move to encourage people to put 3.17 on their 
systems, and then deliberately break it on a subsequent release.

Particularly with ARM, where people are used to doing platform-specific 
things, it seems like encouraging people to either use pre-3.15 or to 
revert Zoltan's changes would be a better option.

  -George

Re: Xen Project policy on feature flags

[Wei Liu]

On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
> On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
> >>Let me rephrase - will it boot in the same fashion (And with the same
> >>bugs) as it did prior to this functionality being introduced?
> >3.15 -> dom0 on ARM broken (if netback is used)
> >3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with CONFIG_ARM_LPAE
> >
> >Reverting the XENFEAT_grant_map_identity related changes would give you
> >a system broken even with CONFIG_ARM_LPAE.
> >Reverting Zoltan's changes to netback would give you a working system.
> 

FWIW reverting isn't practical as many more fixes have gone in.

I think a possible workaround is to copy directly xen-netback directory
from 3.14 and build it against new kernel. Netback itself is quite
self-contained.

> So the *only* reason to include this flag at this point is to allow people
> to run an unmodified 3.17 kernel with netback, is that right?  It seems like
> a really jerk move to encourage people to put 3.17 on their systems, and
> then deliberately break it on a subsequent release.
> 

I think this applies to all netback > 3.15. That means all unmodified
kernels released during our next release cycle.

(Not suggesting we should have this flag or not, just providing some
information to clarify current situation)

Wei.

> Particularly with ARM, where people are used to doing platform-specific
> things, it seems like encouraging people to either use pre-3.15 or to revert
> Zoltan's changes would be a better option.
> 
>  -George
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Xen Project policy on feature flags

[George Dunlap]

On 09/29/2014 10:31 AM, Wei Liu wrote:
> On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
>> On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
>>>> Let me rephrase - will it boot in the same fashion (And with the same
>>>> bugs) as it did prior to this functionality being introduced?
>>> 3.15 -> dom0 on ARM broken (if netback is used)
>>> 3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with CONFIG_ARM_LPAE
>>>
>>> Reverting the XENFEAT_grant_map_identity related changes would give you
>>> a system broken even with CONFIG_ARM_LPAE.
>>> Reverting Zoltan's changes to netback would give you a working system.
> FWIW reverting isn't practical as many more fixes have gone in.
>
> I think a possible workaround is to copy directly xen-netback directory
> from 3.14 and build it against new kernel. Netback itself is quite
> self-contained.

Could we provide a patch which would just disable the problematic behavior?

>
>> So the *only* reason to include this flag at this point is to allow people
>> to run an unmodified 3.17 kernel with netback, is that right?  It seems like
>> a really jerk move to encourage people to put 3.17 on their systems, and
>> then deliberately break it on a subsequent release.
>>
> I think this applies to all netback > 3.15. That means all unmodified
> kernels released during our next release cycle.

Well adding this flag won't make 3.15 and 3.16 work, will it?  Or do you 
expect the code to use this flag to trickle back to those kernels as well?

In any case, what that means all those kernels would be broken should we 
ever decide to remove the flag.  I think it's much better to ask someone 
to use an older kernel or apply a patch.

  -George

Re: Xen Project policy on feature flags

[Wei Liu]

On Mon, Sep 29, 2014 at 10:36:35AM +0100, George Dunlap wrote:
> On 09/29/2014 10:31 AM, Wei Liu wrote:
> >On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
> >>On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
> >>>>Let me rephrase - will it boot in the same fashion (And with the same
> >>>>bugs) as it did prior to this functionality being introduced?
> >>>3.15 -> dom0 on ARM broken (if netback is used)
> >>>3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with CONFIG_ARM_LPAE
> >>>
> >>>Reverting the XENFEAT_grant_map_identity related changes would give you
> >>>a system broken even with CONFIG_ARM_LPAE.
> >>>Reverting Zoltan's changes to netback would give you a working system.
> >FWIW reverting isn't practical as many more fixes have gone in.
> >
> >I think a possible workaround is to copy directly xen-netback directory
> >from 3.14 and build it against new kernel. Netback itself is quite
> >self-contained.
> 
> Could we provide a patch which would just disable the problematic behavior?
> 

Unfortunately no. Zoltan removed the other data path (copying from guest
to Dom0, which works for ARM) completely. It's not something as simple
as flipping a switch in netback to bring it back.

> >
> >>So the *only* reason to include this flag at this point is to allow people
> >>to run an unmodified 3.17 kernel with netback, is that right?  It seems like
> >>a really jerk move to encourage people to put 3.17 on their systems, and
> >>then deliberately break it on a subsequent release.
> >>
> >I think this applies to all netback > 3.15. That means all unmodified
> >kernels released during our next release cycle.
> 
> Well adding this flag won't make 3.15 and 3.16 work, will it?  Or do you

As I understand it, no, it won't make 3.15 and 3.16 work.

> expect the code to use this flag to trickle back to those kernels as well?
> 

This one is for Stefano.  I can't say for sure but that looks unlikely
to me.

Wei.

> In any case, what that means all those kernels would be broken should we
> ever decide to remove the flag.  I think it's much better to ask someone to
> use an older kernel or apply a patch.
> 
>  -George

Re: Xen Project policy on feature flags

[Stefano Stabellini]

On Mon, 29 Sep 2014, Wei Liu wrote:
> On Mon, Sep 29, 2014 at 10:36:35AM +0100, George Dunlap wrote:
> > On 09/29/2014 10:31 AM, Wei Liu wrote:
> > >On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
> > >>On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
> > >>>>Let me rephrase - will it boot in the same fashion (And with the same
> > >>>>bugs) as it did prior to this functionality being introduced?
> > >>>3.15 -> dom0 on ARM broken (if netback is used)
> > >>>3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with CONFIG_ARM_LPAE
> > >>>
> > >>>Reverting the XENFEAT_grant_map_identity related changes would give you
> > >>>a system broken even with CONFIG_ARM_LPAE.
> > >>>Reverting Zoltan's changes to netback would give you a working system.
> > >FWIW reverting isn't practical as many more fixes have gone in.
> > >
> > >I think a possible workaround is to copy directly xen-netback directory
> > >from 3.14 and build it against new kernel. Netback itself is quite
> > >self-contained.
> > 
> > Could we provide a patch which would just disable the problematic behavior?
> > 
> 
> Unfortunately no. Zoltan removed the other data path (copying from guest
> to Dom0, which works for ARM) completely. It's not something as simple
> as flipping a switch in netback to bring it back.
> 
> > >
> > >>So the *only* reason to include this flag at this point is to allow people
> > >>to run an unmodified 3.17 kernel with netback, is that right?  It seems like
> > >>a really jerk move to encourage people to put 3.17 on their systems, and
> > >>then deliberately break it on a subsequent release.
> > >>
> > >I think this applies to all netback > 3.15. That means all unmodified
> > >kernels released during our next release cycle.
> > 
> > Well adding this flag won't make 3.15 and 3.16 work, will it?  Or do you
> 
> As I understand it, no, it won't make 3.15 and 3.16 work.

No, the flag would only make kernels >= 3.17 work if they are built with
CONFIG_ARM_LPAE.



> > expect the code to use this flag to trickle back to those kernels as well?
> > 
> 
> This one is for Stefano.  I can't say for sure but that looks unlikely
> to me.

We are trying to find alternative solutions to the problem. I can't say
it is easy but I am working on an alternative approach.


> Wei.
> 
> > In any case, what that means all those kernels would be broken should we
> > ever decide to remove the flag.  I think it's much better to ask someone to
> > use an older kernel or apply a patch.
> > 
> >  -George
> 

Re: Xen Project policy on feature flags

[David Vrabel]

On 29/09/14 10:36, George Dunlap wrote:
> On 09/29/2014 10:31 AM, Wei Liu wrote:
>> On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
>>> On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
>>>>> Let me rephrase - will it boot in the same fashion (And with the same
>>>>> bugs) as it did prior to this functionality being introduced?
>>>> 3.15 -> dom0 on ARM broken (if netback is used)
>>>> 3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with
>>>> CONFIG_ARM_LPAE
>>>>
>>>> Reverting the XENFEAT_grant_map_identity related changes would give you
>>>> a system broken even with CONFIG_ARM_LPAE.
>>>> Reverting Zoltan's changes to netback would give you a working system.
>> FWIW reverting isn't practical as many more fixes have gone in.
>>
>> I think a possible workaround is to copy directly xen-netback directory
>> from 3.14 and build it against new kernel. Netback itself is quite
>> self-contained.
> 
> Could we provide a patch which would just disable the problematic behavior?

No.  This would require re-introducing the grant copy from-guest path to
netback.  This would be expensive since netback has seen significant
changes since (multi-queue support in particular).

It would also not fix the underlying ARM-specific bug and other users of
grant mapping would be similarly broken.

I think we should:

1. Revert XENFEAT_grant_map_identity.
2. Add the flush-cache-by-bus-address hypercall.
3. Add the Linux support this this cache operation and tag this for stable.
4. Backport the hypercall to 4.4.x.

I think this is critical to fix in 4.5 and should have a freeze
exception.  I would even consider slipping the 4.5 release to get this
fixed.

David

Re: Xen Project policy on feature flags

[Ian Campbell]

On Mon, 2014-09-29 at 11:05 +0100, David Vrabel wrote:
> On 29/09/14 10:36, George Dunlap wrote:
> > On 09/29/2014 10:31 AM, Wei Liu wrote:
> >> On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
> >>> On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
> >>>>> Let me rephrase - will it boot in the same fashion (And with the same
> >>>>> bugs) as it did prior to this functionality being introduced?
> >>>> 3.15 -> dom0 on ARM broken (if netback is used)
> >>>> 3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with
> >>>> CONFIG_ARM_LPAE
> >>>>
> >>>> Reverting the XENFEAT_grant_map_identity related changes would give you
> >>>> a system broken even with CONFIG_ARM_LPAE.
> >>>> Reverting Zoltan's changes to netback would give you a working system.
> >> FWIW reverting isn't practical as many more fixes have gone in.
> >>
> >> I think a possible workaround is to copy directly xen-netback directory
> >> from 3.14 and build it against new kernel. Netback itself is quite
> >> self-contained.
> > 
> > Could we provide a patch which would just disable the problematic behavior?
> 
> No.  This would require re-introducing the grant copy from-guest path to
> netback.  This would be expensive since netback has seen significant
> changes since (multi-queue support in particular).
> 
> It would also not fix the underlying ARM-specific bug and other users of
> grant mapping would be similarly broken.
> 
> I think we should:
> 
> 1. Revert XENFEAT_grant_map_identity.
> 2. Add the flush-cache-by-bus-address hypercall.
> 3. Add the Linux support this this cache operation and tag this for stable.
> 4. Backport the hypercall to 4.4.x.
> 
> I think this is critical to fix in 4.5 and should have a freeze
> exception.  I would even consider slipping the 4.5 release to get this
> fixed.

I agree with this plan of action.

If the new h/call doesn't make 4.5.0 for some reason then it absolutely
must make it for 4.5.1 (and I have no doubt that it would).

Ian.

Re: Xen Project policy on feature flags

[Konrad Rzeszutek Wilk]

On Mon, Sep 29, 2014 at 12:32:59PM +0100, Ian Campbell wrote:
> On Mon, 2014-09-29 at 11:05 +0100, David Vrabel wrote:
> > On 29/09/14 10:36, George Dunlap wrote:
> > > On 09/29/2014 10:31 AM, Wei Liu wrote:
> > >> On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
> > >>> On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
> > >>>>> Let me rephrase - will it boot in the same fashion (And with the same
> > >>>>> bugs) as it did prior to this functionality being introduced?
> > >>>> 3.15 -> dom0 on ARM broken (if netback is used)
> > >>>> 3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with
> > >>>> CONFIG_ARM_LPAE
> > >>>>
> > >>>> Reverting the XENFEAT_grant_map_identity related changes would give you
> > >>>> a system broken even with CONFIG_ARM_LPAE.
> > >>>> Reverting Zoltan's changes to netback would give you a working system.
> > >> FWIW reverting isn't practical as many more fixes have gone in.
> > >>
> > >> I think a possible workaround is to copy directly xen-netback directory
> > >> from 3.14 and build it against new kernel. Netback itself is quite
> > >> self-contained.
> > > 
> > > Could we provide a patch which would just disable the problematic behavior?
> > 
> > No.  This would require re-introducing the grant copy from-guest path to
> > netback.  This would be expensive since netback has seen significant
> > changes since (multi-queue support in particular).
> > 
> > It would also not fix the underlying ARM-specific bug and other users of
> > grant mapping would be similarly broken.
> > 
> > I think we should:
> > 
> > 1. Revert XENFEAT_grant_map_identity.
> > 2. Add the flush-cache-by-bus-address hypercall.
> > 3. Add the Linux support this this cache operation and tag this for stable.
> > 4. Backport the hypercall to 4.4.x.
> > 
> > I think this is critical to fix in 4.5 and should have a freeze
> > exception.  I would even consider slipping the 4.5 release to get this
> > fixed.
> 
> I agree with this plan of action.
> 
> If the new h/call doesn't make 4.5.0 for some reason then it absolutely
> must make it for 4.5.1 (and I have no doubt that it would).

And there goes my plan for an relaxed-release.

I don't recall seeing my answer about distangling CONFIG_ARM_LPAE and
the DMA_ADDR_64_BIT (or whatever it is called) config option. Which
was meant to allow an 32-bit OS to deal with 64-bit PCI devices - which
would have allowed us to still to program 64-bit PCI devices without
the page table support for it. Is that an option?

> 
> Ian.
> 

Re: Xen Project policy on feature flags

[Ian Campbell]

On Mon, 2014-09-29 at 10:55 -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, Sep 29, 2014 at 12:32:59PM +0100, Ian Campbell wrote:
> > On Mon, 2014-09-29 at 11:05 +0100, David Vrabel wrote:
> > > On 29/09/14 10:36, George Dunlap wrote:
> > > > On 09/29/2014 10:31 AM, Wei Liu wrote:
> > > >> On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
> > > >>> On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
> > > >>>>> Let me rephrase - will it boot in the same fashion (And with the same
> > > >>>>> bugs) as it did prior to this functionality being introduced?
> > > >>>> 3.15 -> dom0 on ARM broken (if netback is used)
> > > >>>> 3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with
> > > >>>> CONFIG_ARM_LPAE
> > > >>>>
> > > >>>> Reverting the XENFEAT_grant_map_identity related changes would give you
> > > >>>> a system broken even with CONFIG_ARM_LPAE.
> > > >>>> Reverting Zoltan's changes to netback would give you a working system.
> > > >> FWIW reverting isn't practical as many more fixes have gone in.
> > > >>
> > > >> I think a possible workaround is to copy directly xen-netback directory
> > > >> from 3.14 and build it against new kernel. Netback itself is quite
> > > >> self-contained.
> > > > 
> > > > Could we provide a patch which would just disable the problematic behavior?
> > > 
> > > No.  This would require re-introducing the grant copy from-guest path to
> > > netback.  This would be expensive since netback has seen significant
> > > changes since (multi-queue support in particular).
> > > 
> > > It would also not fix the underlying ARM-specific bug and other users of
> > > grant mapping would be similarly broken.
> > > 
> > > I think we should:
> > > 
> > > 1. Revert XENFEAT_grant_map_identity.
> > > 2. Add the flush-cache-by-bus-address hypercall.
> > > 3. Add the Linux support this this cache operation and tag this for stable.
> > > 4. Backport the hypercall to 4.4.x.
> > > 
> > > I think this is critical to fix in 4.5 and should have a freeze
> > > exception.  I would even consider slipping the 4.5 release to get this
> > > fixed.
> > 
> > I agree with this plan of action.
> > 
> > If the new h/call doesn't make 4.5.0 for some reason then it absolutely
> > must make it for 4.5.1 (and I have no doubt that it would).
> 
> And there goes my plan for an relaxed-release.
> 
> I don't recall seeing my answer about distangling CONFIG_ARM_LPAE and
> the DMA_ADDR_64_BIT (or whatever it is called) config option. Which
> was meant to allow an 32-bit OS to deal with 64-bit PCI devices - which
> would have allowed us to still to program 64-bit PCI devices without
> the page table support for it. Is that an option?

The root issue here is that the guest needs a virtual address mapping of
a 64-bit DMA address, which simply isn't possible with a non-LPAE guest
kernel.

Ian.

Re: Xen Project policy on feature flags

[Stefano Stabellini]

On Mon, 29 Sep 2014, Ian Campbell wrote:
> On Mon, 2014-09-29 at 10:55 -0400, Konrad Rzeszutek Wilk wrote:
> > On Mon, Sep 29, 2014 at 12:32:59PM +0100, Ian Campbell wrote:
> > > On Mon, 2014-09-29 at 11:05 +0100, David Vrabel wrote:
> > > > On 29/09/14 10:36, George Dunlap wrote:
> > > > > On 09/29/2014 10:31 AM, Wei Liu wrote:
> > > > >> On Mon, Sep 29, 2014 at 10:00:13AM +0100, George Dunlap wrote:
> > > > >>> On 09/26/2014 03:49 PM, Stefano Stabellini wrote:
> > > > >>>>> Let me rephrase - will it boot in the same fashion (And with the same
> > > > >>>>> bugs) as it did prior to this functionality being introduced?
> > > > >>>> 3.15 -> dom0 on ARM broken (if netback is used)
> > > > >>>> 3.17 -> dom0 on ARM is fixed, only if the kernel is compiled with
> > > > >>>> CONFIG_ARM_LPAE
> > > > >>>>
> > > > >>>> Reverting the XENFEAT_grant_map_identity related changes would give you
> > > > >>>> a system broken even with CONFIG_ARM_LPAE.
> > > > >>>> Reverting Zoltan's changes to netback would give you a working system.
> > > > >> FWIW reverting isn't practical as many more fixes have gone in.
> > > > >>
> > > > >> I think a possible workaround is to copy directly xen-netback directory
> > > > >> from 3.14 and build it against new kernel. Netback itself is quite
> > > > >> self-contained.
> > > > > 
> > > > > Could we provide a patch which would just disable the problematic behavior?
> > > > 
> > > > No.  This would require re-introducing the grant copy from-guest path to
> > > > netback.  This would be expensive since netback has seen significant
> > > > changes since (multi-queue support in particular).
> > > > 
> > > > It would also not fix the underlying ARM-specific bug and other users of
> > > > grant mapping would be similarly broken.
> > > > 
> > > > I think we should:
> > > > 
> > > > 1. Revert XENFEAT_grant_map_identity.
> > > > 2. Add the flush-cache-by-bus-address hypercall.
> > > > 3. Add the Linux support this this cache operation and tag this for stable.
> > > > 4. Backport the hypercall to 4.4.x.
> > > > 
> > > > I think this is critical to fix in 4.5 and should have a freeze
> > > > exception.  I would even consider slipping the 4.5 release to get this
> > > > fixed.
> > > 
> > > I agree with this plan of action.
> > > 
> > > If the new h/call doesn't make 4.5.0 for some reason then it absolutely
> > > must make it for 4.5.1 (and I have no doubt that it would).
> > 
> > And there goes my plan for an relaxed-release.
> > 
> > I don't recall seeing my answer about distangling CONFIG_ARM_LPAE and
> > the DMA_ADDR_64_BIT (or whatever it is called) config option. Which
> > was meant to allow an 32-bit OS to deal with 64-bit PCI devices - which
> > would have allowed us to still to program 64-bit PCI devices without
> > the page table support for it. Is that an option?
> 
> The root issue here is that the guest needs a virtual address mapping of
> a 64-bit DMA address, which simply isn't possible with a non-LPAE guest
> kernel.

FYI the virtual mapping is needed because required cache flushing ops
on ARM take a virtual address as argument.

Re: Xen Project policy on feature flags

[Jan Beulich]

>>> On 26.09.14 at 16:19, <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-09-26 at 14:39 +0100, Jan Beulich wrote:
>> >>> On 26.09.14 at 15:24, <stefano.stabellini@eu.citrix.com> wrote:
>> > I am writing to request a clarification on Xen feature flags
>> > (XENFEAT_*) and backward compatibility:
>> >     
>> > is the hypervisor allowed to remove any feature flags in a future
>> > release, even though doing so might break some existing guests?
>> > 
>> > For example one could write a PV on HVM guest that requires
>> > XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
>> > release remove that feature? Or is it now part of our ABI, therefore
>> > maintained for backward compatibility, following the rule that we don't
>> > break existing guests?
>> > 
>> > 
>> > I always thought that any XENFEAT feature flags could be removed going
>> > forward, if the hypervisor maintainers decide to do so. However Ian
>> > Campbell is of the opposite opinion, so I think we should have a clear
>> > policy regarding them.
>> > 
>> > In any case I think that it is generally useful to have optional flags
>> > that advertise the presence of a feature but can also be removed going
>> > forward. If XENFEAT feature flags are not them, then we might still want
>> > to introduce them as a separate concept.
>> 
>> My view is that these are precisely there to indicate what the
>> hypervisor supports. I.e. while we can't remove the definition
>> from the public header, the hypervisor could stop advertising that
>> it's capable of a certain feature at any time. Consumers are
>> expected to check for the feature flag and deal with it being off.
> 
> What if there is no way to deal with it being off other than crashing or
> refusing to work etc?

Generally that would mean bad coding on the kernel side. But
in this specific case...

> The context here is XENFEAT_grant_map_identity which turns out not to
> work in practice (it doesn't actually allow non-LPAE arm32 dom0 to work,
> which it was supposed to). We've not released with it yet (as it stands
> it'll be included in 4.5).
> 
> We do have some other ideas for an alternative fix to the underlying
> issue. But we've not actually tried them yet.
> 
> IMHO We need to decide whether to nuke that flag now or in the future
> (i.e. post 4.5). Given that it doesn't actually work and that we've not
> released with it I'm most in favour of not releasing with it, and given
> that I'm in favour of removing it now and not waiting until the last
> minute.
> 
> If we release with it then we will regress at least Linux 3.16 and 3.17
> when we remove this flag.

... I agree removing it now (by reverting respective commit(s)) would
be better.

> I don't think we have many other flags which are "work at all" flags
> rather than "enhance" flags.

Indeed, and in such a case it may be a bad choice to consider this a
feature flag in the first place.

Jan

Re: Xen Project policy on feature flags

[David Vrabel]

On 26/09/14 14:24, Stefano Stabellini wrote:
> Hello all,
> I am writing to request a clarification on Xen feature flags
> (XENFEAT_*) and backward compatibility:
>     
> is the hypervisor allowed to remove any feature flags in a future
> release, even though doing so might break some existing guests?
> 
> For example one could write a PV on HVM guest that requires
> XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
> release remove that feature? Or is it now part of our ABI, therefore
> maintained for backward compatibility, following the rule that we don't
> break existing guests?
> 
> 
> I always thought that any XENFEAT feature flags could be removed going
> forward, if the hypervisor maintainers decide to do so. However Ian
> Campbell is of the opposite opinion, so I think we should have a clear
> policy regarding them.

A guest that runs on Xen version V /must/ continue to run on V+1.

The is similar to the policy the Linux kernel has for the user space ABI.

This does permit support for features to be removed but only if no guest
would be broken by its removal.  But since it it is not possible to know
what guests people have running and what features they require, I can't
see how any feature could be safely removed.

> In any case I think that it is generally useful to have optional flags
> that advertise the presence of a feature but can also be removed going
> forward. If XENFEAT feature flags are not them, then we might still want
> to introduce them as a separate concept.

I don't think "optional" feature flags are any different.  You can
specify that guests must handle the feature being missing but that's no
guarantee that guest will actually implement the fallback.

David

Re: Xen Project policy on feature flags

[Konrad Rzeszutek Wilk]

On Fri, Sep 26, 2014 at 03:26:33PM +0100, David Vrabel wrote:
> On 26/09/14 14:24, Stefano Stabellini wrote:
> > Hello all,
> > I am writing to request a clarification on Xen feature flags
> > (XENFEAT_*) and backward compatibility:
> >     
> > is the hypervisor allowed to remove any feature flags in a future
> > release, even though doing so might break some existing guests?
> > 
> > For example one could write a PV on HVM guest that requires
> > XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
> > release remove that feature? Or is it now part of our ABI, therefore
> > maintained for backward compatibility, following the rule that we don't
> > break existing guests?
> > 
> > 
> > I always thought that any XENFEAT feature flags could be removed going
> > forward, if the hypervisor maintainers decide to do so. However Ian
> > Campbell is of the opposite opinion, so I think we should have a clear
> > policy regarding them.
> 
> A guest that runs on Xen version V /must/ continue to run on V+1.

Not if the feature is 'experimental' (I don't know offhand if this
is experimental or not).

> 
> The is similar to the policy the Linux kernel has for the user space ABI.

Sure, but there are also ioctls and such which can change between releases
such that application for V+1 will _not_ work with V. Look at 'perf'
as an example.

> 
> This does permit support for features to be removed but only if no guest
> would be broken by its removal.  But since it it is not possible to know
> what guests people have running and what features they require, I can't
> see how any feature could be safely removed.

The guest is already broken even by taking advantage of this feature.


> 
> > In any case I think that it is generally useful to have optional flags
> > that advertise the presence of a feature but can also be removed going
> > forward. If XENFEAT feature flags are not them, then we might still want
> > to introduce them as a separate concept.
> 
> I don't think "optional" feature flags are any different.  You can
> specify that guests must handle the feature being missing but that's no
> guarantee that guest will actually implement the fallback.

That would be a odd - as it would mean guests were written
to work with V+1 but not with V-1.

Granted that is the case for booting Linux as dom0. It can only work
with Xen 4.1 and later. But you could argue that the fallback for booting
with V-1 (Xen 4.0) is to not boot at all as it did not have the
minimum required features.

So was this feature an required feature or an enhancement?

> 
> David
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Xen Project policy on feature flags

[Stefano Stabellini]

On Fri, 26 Sep 2014, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 26, 2014 at 03:26:33PM +0100, David Vrabel wrote:
> > On 26/09/14 14:24, Stefano Stabellini wrote:
> > > Hello all,
> > > I am writing to request a clarification on Xen feature flags
> > > (XENFEAT_*) and backward compatibility:
> > >     
> > > is the hypervisor allowed to remove any feature flags in a future
> > > release, even though doing so might break some existing guests?
> > > 
> > > For example one could write a PV on HVM guest that requires
> > > XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
> > > release remove that feature? Or is it now part of our ABI, therefore
> > > maintained for backward compatibility, following the rule that we don't
> > > break existing guests?
> > > 
> > > 
> > > I always thought that any XENFEAT feature flags could be removed going
> > > forward, if the hypervisor maintainers decide to do so. However Ian
> > > Campbell is of the opposite opinion, so I think we should have a clear
> > > policy regarding them.
> > 
> > A guest that runs on Xen version V /must/ continue to run on V+1.
> 
> Not if the feature is 'experimental' (I don't know offhand if this
> is experimental or not).
> 
> > 
> > The is similar to the policy the Linux kernel has for the user space ABI.
> 
> Sure, but there are also ioctls and such which can change between releases
> such that application for V+1 will _not_ work with V. Look at 'perf'
> as an example.
> 
> > 
> > This does permit support for features to be removed but only if no guest
> > would be broken by its removal.  But since it it is not possible to know
> > what guests people have running and what features they require, I can't
> > see how any feature could be safely removed.
> 
> The guest is already broken even by taking advantage of this feature.

The guest is capable of recognizing whether the feature is available or
not and doesn't try to use the feature if it is not advertized by the
XENFEAT flag.

Unfortunately the fall back at the moment is breaking up farther along
the way.

 
> > > In any case I think that it is generally useful to have optional flags
> > > that advertise the presence of a feature but can also be removed going
> > > forward. If XENFEAT feature flags are not them, then we might still want
> > > to introduce them as a separate concept.
> > 
> > I don't think "optional" feature flags are any different.  You can
> > specify that guests must handle the feature being missing but that's no
> > guarantee that guest will actually implement the fallback.
> 
> That would be a odd - as it would mean guests were written
> to work with V+1 but not with V-1.
> 
> Granted that is the case for booting Linux as dom0. It can only work
> with Xen 4.1 and later. But you could argue that the fallback for booting
> with V-1 (Xen 4.0) is to not boot at all as it did not have the
> minimum required features.
> 
> So was this feature an required feature or an enhancement?

>From a practical perspective it is a required feature for Linux.

>From a theoretical and ABI standpoint, one could write a Dom0 kernel
that doesn't need this feature to work properly (and in fact you can
still have it by reverting Zoltan's changes or going back to 3.15).

Re: Xen Project policy on feature flags

[Konrad Rzeszutek Wilk]

On Fri, Sep 26, 2014 at 03:54:40PM +0100, Stefano Stabellini wrote:
> On Fri, 26 Sep 2014, Konrad Rzeszutek Wilk wrote:
> > On Fri, Sep 26, 2014 at 03:26:33PM +0100, David Vrabel wrote:
> > > On 26/09/14 14:24, Stefano Stabellini wrote:
> > > > Hello all,
> > > > I am writing to request a clarification on Xen feature flags
> > > > (XENFEAT_*) and backward compatibility:
> > > >     
> > > > is the hypervisor allowed to remove any feature flags in a future
> > > > release, even though doing so might break some existing guests?
> > > > 
> > > > For example one could write a PV on HVM guest that requires
> > > > XENFEAT_hvm_callback_vector (regardless of PVH), could a future Xen
> > > > release remove that feature? Or is it now part of our ABI, therefore
> > > > maintained for backward compatibility, following the rule that we don't
> > > > break existing guests?
> > > > 
> > > > 
> > > > I always thought that any XENFEAT feature flags could be removed going
> > > > forward, if the hypervisor maintainers decide to do so. However Ian
> > > > Campbell is of the opposite opinion, so I think we should have a clear
> > > > policy regarding them.
> > > 
> > > A guest that runs on Xen version V /must/ continue to run on V+1.
> > 
> > Not if the feature is 'experimental' (I don't know offhand if this
> > is experimental or not).
> > 
> > > 
> > > The is similar to the policy the Linux kernel has for the user space ABI.
> > 
> > Sure, but there are also ioctls and such which can change between releases
> > such that application for V+1 will _not_ work with V. Look at 'perf'
> > as an example.
> > 
> > > 
> > > This does permit support for features to be removed but only if no guest
> > > would be broken by its removal.  But since it it is not possible to know
> > > what guests people have running and what features they require, I can't
> > > see how any feature could be safely removed.
> > 
> > The guest is already broken even by taking advantage of this feature.
> 
> The guest is capable of recognizing whether the feature is available or
> not and doesn't try to use the feature if it is not advertized by the
> XENFEAT flag.
> 
> Unfortunately the fall back at the moment is breaking up farther along
> the way.

HA! To summarize:

 - With this feature available we break.
 - Without this feature available we break.

In either way removing this feature will mean we are still broken - it is
just a different shade of it.

> 
>  
> > > > In any case I think that it is generally useful to have optional flags
> > > > that advertise the presence of a feature but can also be removed going
> > > > forward. If XENFEAT feature flags are not them, then we might still want
> > > > to introduce them as a separate concept.
> > > 
> > > I don't think "optional" feature flags are any different.  You can
> > > specify that guests must handle the feature being missing but that's no
> > > guarantee that guest will actually implement the fallback.
> > 
> > That would be a odd - as it would mean guests were written
> > to work with V+1 but not with V-1.
> > 
> > Granted that is the case for booting Linux as dom0. It can only work
> > with Xen 4.1 and later. But you could argue that the fallback for booting
> > with V-1 (Xen 4.0) is to not boot at all as it did not have the
> > minimum required features.
> > 
> > So was this feature an required feature or an enhancement?
> 
> >From a practical perspective it is a required feature for Linux.

Hehe. We have a required feature that we want to be broken, eh :-)

Can the CONFIG_PCI_DMA_ARE_64_BIT (or whatever it is) be part of an
non-LPAE build? That would have fixed the issue too right?

> 
> >From a theoretical and ABI standpoint, one could write a Dom0 kernel
> that doesn't need this feature to work properly (and in fact you can
> still have it by reverting Zoltan's changes or going back to 3.15).

Oh boy. Have we dug ourselves in a hole.

Re: Xen Project policy on feature flags

[Jan Beulich]

>>> On 26.09.14 at 16:26, <david.vrabel@citrix.com> wrote:
> On 26/09/14 14:24, Stefano Stabellini wrote:
>> I always thought that any XENFEAT feature flags could be removed going
>> forward, if the hypervisor maintainers decide to do so. However Ian
>> Campbell is of the opposite opinion, so I think we should have a clear
>> policy regarding them.
> 
> A guest that runs on Xen version V /must/ continue to run on V+1.
> 
> The is similar to the policy the Linux kernel has for the user space ABI.

Linux has proven a number of times that it doesn't always care about
such compatibility. The most prominent example being a major sysfs
layout rework in 2.6.37 which broke many older systems of mine.

I view feature flags much more like CPUs announcing certain
capabilities via CPUID - this is no guarantee that all that functionality
will also be present in future CPUs.

Jan