Re: [PATCH 0/4] Allow schedulers to be selectable through Kconfig

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 18/12/15 17:56, Doug Goldstein wrote:
> On 12/18/15 4:45 AM, Ian Campbell wrote:
>> On Thu, 2015-12-17 at 14:59 -0600, Jonathan Creekmore wrote:
>>> Add machinery to allow the schedulers to be individually selectable
>>> through the Kconfig interface.
>> So I don't want to pick on this series or schedulers specifically here, but
>> instead discuss the general premise of configurability of hypervisor
>> binaries, and this happens to be the first. I'm CCing Doug and "THE REST"
>> from MAINTAINERS
>>
>> Currently (even with the current switch to Kconfig thus far) we have a
>> fairly small and manageable set of configurations which any given Xen
>> binary can be in and in terms of what users are actually running an even
>> smaller set I believe, most users fiddle with zero options and a small
>> number with one or two. I'd hazard a guess that the vast majority of Xen
>> binaries are using a single config today and that the second place is a
>> fairly distant second.
>>
>> This means we have avoided the combinatorial explosion of configuration
>> options which Linux suffers from (which result in things like randconfig
>> build robots because no one can keep track of it all).
>>
>> Just to be clear: I'm not at all opposed to more configurability for expert
>> users who have specific usecases, know what they are doing and are willing
>> to take responsibility for developments deviating form the norm.
>>
>> However I am very wary of putting shiny looking nobs in front of the
>> average user, since they will find them and they will inevitably play with
>> them and we will end up in the situation where every bug report involves an
>> additional RTT while we ask for their .config (ok, in reality we'd often
>> ask at the same time we inevitably have to ask for logs and other key info,
>> so I guess I'm exaggerating, but still its a worry I think).
>>
>> As well as support there is obviously a testing matrix impact.
>>
>> How would people feel about a CONFIG_STANDARD_FEATURESET[0] with the
>> majority of tweakables depending on !STANDARD_FEATURESET? It would default
>> Y with a help text which dissuades normal users from touching it ("Say Y,
>> unless you are willing to pick up the pieces yourself. We do not routinely
>> test or validate configurations without this option set. We expect you to
>> offer to fix issues which you find. Beware of the leopard.").[1]
>>
>> I might even go so far as to suggest that !STANDARD_FEATURESET
>> configurations would not be subject to security support (i.e. issues which
>> can _only_ arise with that option disabled are't covered).
>>
>> The bar for adding a new option which does not depend on
>> !STANDARD_FEATURESET would be _high_.
>>
>> Ian.
>>
>> [0] I really mean CONFIG_STANDARD_CONFIG, but that sounds dumb. I don't
>> really intend for it to only control "features". CONFIG_STANDARD might be
>> an alternative.
>> [1] Hrm, CONFIG_LEOPARD?
>>
>>         http://www.goodreads.com/quotes/40705-but-the-plans-were-on-display-on-display-i-eventually
>>
>>     ;-)
>>
> So I know this is going to be a thread hijack but I think its important
> for me to explain where Jonathan Creekmore and I are coming from with
> our recently submitted patches.
>
> We recently took over a project based on an old version of Xen that has
> a lot of customization. Part of our goal was to understand those
> modifications and re-implement them in a fashion that would be
> acceptable to upstream. So for example in the code base we worked with
> all other schedulers were entirely ripped out of the code base. Jonathan
> has reworked this to make it possible to disable each scheduler
> independently. Additionally Jonathan improved the way that extra
> schedulers can be added in the future.
>
> Effectively we want to engage the Xen community as much as possible and
> come up with solutions that are workable for the community as well as
> our needs. Now here's why I believe in this approach. I've looked at
> some of the other XenProject derived projects (e.g. XenServer, Qubes,
> OpenXT) and Linux distros shipping Xen (Ubuntu, Debian, Gentoo, Yocto,
> etc) and I saw that they all contain a number of patches against
> XenProject. To me this weakens the overall project by everyone
> maintaining their own fork. Everyone's goal should be to work on
> upstreaming changes in a fashion that is acceptable to the community in
> an agnostic fashion. Recently I took Joanna Rutkowska to task for not
> contributing work upstream and maintaining downstream changes and
> expecting Xen to confirm to her project's needs [1]. But on the flip
> side of the coin Ian Campbell has been helping Debian to upstream
> changes they need [2].
>
> In the end I really see the primary people that build Xen on their own
> as project maintainers (XenServer, Qubes, OpenXT) or distro maintainers
> (Ubuntu, Debian, Gentoo, Yocto) or "expert" users. Most people will use
> Xen as it comes packaged for them already and the Kconfig changes give
> the project/distro/experts the flexibility they need to build up Xen
> without maintaining downstream patches. So these won't really be shiny
> knobs for users to twiddle.
>
> What provides the defaults that users should use are the defconfig
> files. The reason this doesn't work out so well for Linux is that there
> are really not defaults that work for everyone. I believe for a long
> time the x86_64 defconfig was a setup for Linus' development laptop and
> most of the other arches defconfigs were setup for their respective
> maintainers as well. But in the case of Xen the defconfig will be the
> defaults and we don't even require the user to run `make *config` so it
> really is more of an "expert" option already vs Linux where you must
> first run `make *config` before building your kernel.
>
> [1]
> http://lists.xenproject.org/archives/html/xen-devel/2015-11/msg00813.html
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=805508

The distro packages and maintainers are likely to be the vast majority
of users of the config system, and they by definition know what they are
doing.

Next we get onto the few people casually using Xen.  As the defaults run
automatically, they need not even know that kconfig exists under the
hood, unless they go looking for it.

Anyone who runs `make menuconfig` thinks they know what they are doing,
and will ignore a wall of text telling them that things may be
dangerous.  They implicitly acknowledge responsibility for shooting
themselves in the foot.  At the end of the day, technical measures do
not help in defending against stupidity.

The fact that the defconfig gets applied automatically mitigates against
people playing with options simply because the need to to get something
out in the end.  This is a strict improvement over the current state of
play for Linux.

There will be options which should live behind a CONFIG_EXPERT (or
equivalent) (e.g. lock profiling?) but these should be in the minority. 
Another advantage we have is that, given Xen's size, it is never going
to have the quantity of options currently available for Linux.

As long as the defconfig files are kept up to date as new features are
added, everything will be fine.

~Andrew

P.S. One very good side effect of the Linux options with randconfig is
that it helps force clean APIs on the code.  The code in Xen has grown
organically without these stricter boundaries, and I will be looking to
clean bits of this up going forwards.