Support situation for nestedhvm

Support situation for nestedhvm

[Elliott Mitchell]

I ran into the nestedhvm via the following path.  I was considering the
feasibility of shedding tasks from a desktop onto a server running Xen.
I was looking at `man xl.cfg` and noticed "nestedhvm".

Since one of the tasks the computer handled was running other OSes in
fully simulated environments, this seemed to be something I was looking
for.  No where did I ever see anything hinting "This configuration option
is completely unsupported and risky to use".


For an option like this, additional steps should have been needed to
enable it.

First, on Xen's command-line there needs to be something along the
lines of "enable_unsupported=nestedhvm,others".

Second, in xl.cfg perhaps there should be an `enable_unsupported` option
which is a list of such options.

Third, perhaps a build-time configuration option too?


The issue is the above.  At no point did I realize I had crossed the
support boundary.  Things simply started exploding without any warnings.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Support situation for nestedhvm

[Andrew Cooper]

On 07/11/2023 7:53 pm, Elliott Mitchell wrote:
> I ran into the nestedhvm via the following path.  I was considering the
> feasibility of shedding tasks from a desktop onto a server running Xen.
> I was looking at `man xl.cfg` and noticed "nestedhvm".
>
> Since one of the tasks the computer handled was running other OSes in
> fully simulated environments, this seemed to be something I was looking
> for.  No where did I ever see anything hinting "This configuration option
> is completely unsupported and risky to use".

This one is explicitly covered in SUPPORT.md, and has had XSAs out
against it in the past for being unexpectedly active when it oughtn't to
have been.

> Things simply started exploding without any warnings.

Things also explode if you try to create a VM with 10x more RAM than you
have, or if you try `./xenwatchdogd --help`, or `xl debug-keys c`, or
many other things. 

The xl manpage probably ought to state explicitly that the option is
experimental, but that the extent of what I'd consider reasonable here.

You can't solve educational matters with technical measures.

~Andrew

Re: Support situation for nestedhvm

[Alejandro Vallejo]

Hi,

On Tue, Nov 07, 2023 at 08:15:32PM +0000, Andrew Cooper wrote:
> On 07/11/2023 7:53 pm, Elliott Mitchell wrote:
> > I ran into the nestedhvm via the following path.  I was considering the
> > feasibility of shedding tasks from a desktop onto a server running Xen.
> > I was looking at `man xl.cfg` and noticed "nestedhvm".
> >
> > Since one of the tasks the computer handled was running other OSes in
> > fully simulated environments, this seemed to be something I was looking
> > for.  No where did I ever see anything hinting "This configuration option
> > is completely unsupported and risky to use".
> 
> This one is explicitly covered in SUPPORT.md, and has had XSAs out
> against it in the past for being unexpectedly active when it oughtn't to
> have been.
> 
> > Things simply started exploding without any warnings.
> 
> Things also explode if you try to create a VM with 10x more RAM than you
> have, or if you try `./xenwatchdogd --help`, or `xl debug-keys c`, or
> many other things. 
> 
> The xl manpage probably ought to state explicitly that the option is
> experimental, but that the extent of what I'd consider reasonable here.
> 
> You can't solve educational matters with technical measures.
> 
> ~Andrew
> 
No, but we can prevent users unexpectedly shooting themselves in the foot.

Elliott's point (as I understood it) was that we could have an
"experimental" switch, that would warn and error out when experimental
features are used without it. This is just cfg sugar coating for xl, and
would improve UX. Cargo uses the same sort of idea in the Rust ecosystem to
make a clear distinction between unstable features that may change and
stable ones that are meant to stay and just work.

Having "experimental=nestedhvm,foo" is one option, having "experimental=1"
be a required flag to enable experimental features is another. Heck, even
renaming "nestedhvm" to "experimental-nestedhvm" would be an improvement.

```
   Error: nestedhvm=1 is an unstable experimental feature not encouraged
          for production purposes. Enable it with experimental-nestedhvm=1
          instead.
```

We can't fix it overnight, but we can't make it _very_ clear it's unstable
to anyone currently using the feature.

Cheers,
Alejandro

Re: Support situation for nestedhvm

[Andrew Cooper]

On 09/11/2023 9:50 am, Alejandro Vallejo wrote:
> Hi,
>
> On Tue, Nov 07, 2023 at 08:15:32PM +0000, Andrew Cooper wrote:
>> On 07/11/2023 7:53 pm, Elliott Mitchell wrote:
>>> I ran into the nestedhvm via the following path.  I was considering the
>>> feasibility of shedding tasks from a desktop onto a server running Xen.
>>> I was looking at `man xl.cfg` and noticed "nestedhvm".
>>>
>>> Since one of the tasks the computer handled was running other OSes in
>>> fully simulated environments, this seemed to be something I was looking
>>> for.  No where did I ever see anything hinting "This configuration option
>>> is completely unsupported and risky to use".
>> This one is explicitly covered in SUPPORT.md, and has had XSAs out
>> against it in the past for being unexpectedly active when it oughtn't to
>> have been.
>>
>>> Things simply started exploding without any warnings.
>> Things also explode if you try to create a VM with 10x more RAM than you
>> have, or if you try `./xenwatchdogd --help`, or `xl debug-keys c`, or
>> many other things. 
>>
>> The xl manpage probably ought to state explicitly that the option is
>> experimental, but that the extent of what I'd consider reasonable here.
>>
>> You can't solve educational matters with technical measures.
>>
>> ~Andrew
>>
> No, but we can prevent users unexpectedly shooting themselves in the foot.

... and break OSSTest and XenRT while you're at it.

Like it or not, this knob is behaved in this way for 15 years.  You will
be doing harm for no benefit by trying to change it.

And if you need a cautionary tail on why this is a bad idea generally,
as well as a background on why I will firmly object to technical
countermeasures like this, read up on Xen's allow_unsafe command line
parameter.

~Andrew

Re: Support situation for nestedhvm

[Alejandro Vallejo]

On Thu, Nov 09, 2023 at 10:36:21AM +0000, Andrew Cooper wrote:
> On 09/11/2023 9:50 am, Alejandro Vallejo wrote:
> > Hi,
> >
> > On Tue, Nov 07, 2023 at 08:15:32PM +0000, Andrew Cooper wrote:
> >> On 07/11/2023 7:53 pm, Elliott Mitchell wrote:
> >>> I ran into the nestedhvm via the following path.  I was considering the
> >>> feasibility of shedding tasks from a desktop onto a server running Xen.
> >>> I was looking at `man xl.cfg` and noticed "nestedhvm".
> >>>
> >>> Since one of the tasks the computer handled was running other OSes in
> >>> fully simulated environments, this seemed to be something I was looking
> >>> for.  No where did I ever see anything hinting "This configuration option
> >>> is completely unsupported and risky to use".
> >> This one is explicitly covered in SUPPORT.md, and has had XSAs out
> >> against it in the past for being unexpectedly active when it oughtn't to
> >> have been.
> >>
> >>> Things simply started exploding without any warnings.
> >> Things also explode if you try to create a VM with 10x more RAM than you
> >> have, or if you try `./xenwatchdogd --help`, or `xl debug-keys c`, or
> >> many other things. 
> >>
> >> The xl manpage probably ought to state explicitly that the option is
> >> experimental, but that the extent of what I'd consider reasonable here.
> >>
> >> You can't solve educational matters with technical measures.
> >>
> >> ~Andrew
> >>
> > No, but we can prevent users unexpectedly shooting themselves in the foot.
> 
> ... and break OSSTest and XenRT while you're at it.
> 
> Like it or not, this knob is behaved in this way for 15 years.  You will
> be doing harm for no benefit by trying to change it.
Improving UX is a distinctively good benefit. A lot of people on this
mailing list may be aware of its quirks, but a user shouldn't need to be
that aware in order to set up a stable system.

> 
> And if you need a cautionary tail on why this is a bad idea generally,
> as well as a background on why I will firmly object to technical
> countermeasures like this, read up on Xen's allow_unsafe command line
> parameter.
> 
> ~Andrew
This?
  https://bugzilla.redhat.com/show_bug.cgi?id=858724

If so, that's very different. allow_unsafe caused previously accesible
remote hosts to become unbootable after an update, leaving anyone with a
remote host without IPMI interface dead in the water. It's nothing like
preventing spinning up a VM with a set of features that with high
likelihood a user doesn't want.

Both OSSTest and XenRT can simply adjust their nestedhvm knobs based on a
simple probing script.

Cheers,
Alejandro

Re: Support situation for nestedhvm

[Elliott Mitchell]

On Tue, Nov 07, 2023 at 08:15:32PM +0000, Andrew Cooper wrote:
> On 07/11/2023 7:53 pm, Elliott Mitchell wrote:
> > I ran into the nestedhvm via the following path.  I was considering the
> > feasibility of shedding tasks from a desktop onto a server running Xen.
> > I was looking at `man xl.cfg` and noticed "nestedhvm".
> >
> > Since one of the tasks the computer handled was running other OSes in
> > fully simulated environments, this seemed to be something I was looking
> > for.  No where did I ever see anything hinting "This configuration option
> > is completely unsupported and risky to use".
> 
> This one is explicitly covered in SUPPORT.md, and has had XSAs out
> against it in the past for being unexpectedly active when it oughtn't to
> have been.

$ wc -l SUPPORT.md 
1166 SUPPORT.md
$ grep -e^###\  SUPPORT.md | wc -l
129
$

Someone who was exclusively handling Xen system administration all the
time might be expected to memorize that.  Everyone else may recall some
of that, but will forget large portions rapidly.

> > Things simply started exploding without any warnings.
> 
> Things also explode if you try to create a VM with 10x more RAM than you
> have, or if you try `./xenwatchdogd --help`, or `xl debug-keys c`, or
> many other things. 

These of course /should/ be fixed.

> The xl manpage probably ought to state explicitly that the option is
> experimental, but that the extent of what I'd consider reasonable here.

This should have been done years ago.


On Thu, Nov 09, 2023 at 10:36:21AM +0000, Andrew Cooper wrote:
> On 09/11/2023 9:50 am, Alejandro Vallejo wrote:
> >
> > No, but we can prevent users unexpectedly shooting themselves in the foot.

> Like it or not, this knob is behaved in this way for 15 years.  You will
> be doing harm for no benefit by trying to change it.

Does your car feature spurs and ropes in order to be compatible with
horses?  Things do change over time.

> And if you need a cautionary tail on why this is a bad idea generally,
> as well as a background on why I will firmly object to technical
> countermeasures like this, read up on Xen's allow_unsafe command line
> parameter.

I can see why that experience would leave a mark.  Perhaps one of these
should create the domain with nestedhvm disabled, and overtly state it
needs to be enabled in two places.

Yet I reject your metaphor.  allow_unsafe was added as part of an urgent
security update.  This behavior would be released as part of a major
version upgrade.  Major upgrades need to be done with care since they're
the time when things break (hopefully this wouldn't break anything, but I
agree such could happen).


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445