[PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Dario Faggioli]

Hey,

"Just" as per the subject, I wrote feature documents for (almost) all our
schedulers. No big deal, I'd say, apart from the fact that I'm declaring
Credit2 **Supperted**, instead of experimental.

In fact, it's being tested by OSSTest for ages, and it's undergone a huge
amount of development, testing and benchmarking lately, and its current status
is finally good enough, IMO.

As a consequence of that, in patch 2, I'm also touching the code, but that's
just for removing a printk, so it should not be too big of a deal I hope.

This wiki page is listed among the references in Credit2's feature document:
https://wiki.xenproject.org/wiki/Credit2_Scheduler_Development.
If you go there checking it out, bear in mind that I'm updating and
re-organazing it by quite a bit, like right now.

Sorry if the Cc-list is a bit large, but that's the best I and
get_maintainers.pl could come up with. :-P

Thanks and Regards,
Dario
---
Dario Faggioli (3):
      docs: Credit1 feature document.
      docs: Credit2 feature document.
      docs: RTDS feature document.

 docs/features/credit.pandoc  |   99 +++++++++++++++++++++++++++++++++
 docs/features/credit2.pandoc |  123 +++++++++++++++++++++++++++++++++++++++++
 docs/features/rtds.pandoc    |  125 ++++++++++++++++++++++++++++++++++++++++++
 xen/common/sched_credit2.c   |    2 -
 4 files changed, 347 insertions(+), 2 deletions(-)
 create mode 100644 docs/features/credit.pandoc
 create mode 100644 docs/features/credit2.pandoc
 create mode 100644 docs/features/rtds.pandoc
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 1/3] docs: Credit1 feature document.

[Dario Faggioli]

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Lars Kurth <lars.kurth@citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
---
 docs/features/credit.pandoc |   99 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100644 docs/features/credit.pandoc

diff --git a/docs/features/credit.pandoc b/docs/features/credit.pandoc
new file mode 100644
index 0000000..fed0da2
--- /dev/null
+++ b/docs/features/credit.pandoc
@@ -0,0 +1,99 @@
+% Credit
+% Revision 1
+
+\clearpage
+
+# Basics
+---------------- ----------------------------------------------------
+         Status: e.g. **Supported**
+
+Architecture(s): e.g. x86, arm
+
+      Component: e.g. Hypervisor
+---------------- ----------------------------------------------------
+
+# Overview
+
+Credit (also known as Credit1) is the default virtual CPU (vCPU) scheduler
+of the Xen hypervisor. The job of an hypervisor's scheduler is to decide,
+among all the various vCPUs of the various virtual machines, which ones
+should execute on the host's physical CPUs (pCPUs), at any given point in
+time.
+
+# User details
+
+Xen supports multiple schedulers. As said, Credit is the default, so it
+is used automatically, unless the `sched=$SCHED` (with `$SCHED` different
+than `credit`) parameter is passed to Xen via the bootloader.
+
+Once the system is live, for creating a cpupool with Credit as its
+scheduler, either compile a cpupool configuration file, as described
+in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified in
+`tools/examples/cpupool`), or use just `xl` directly:
+
+    xl cpupool-create name=\"pool1\" sched=\"credit\" cpus=[4,8]
+
+Two kind of interactions with the scheduler are possible:
+
+* checking or changing the global parameters, via, e.g.:
+    * `xl sched-credit -s`
+    * `xl sched-credit -s -p pool1`
+    * `xl sched-credit -s -t 20`
+* checking or changing a VM's scheduling parameters, via, e.g.:
+    * `xl sched-credit -d vm1`
+    * `xl sched-credit -d vm1 -w 512`
+
+# Technical details
+
+Implementation entirely lives in the hypervisor. Xen has a pluggable, hook
+based, architecture for schedulers. Actual Credit code is all inside one
+file: `xen/common/sched_credit.c`.
+
+# Limitations
+
+In Credit, a vCPU has a priority, a status (i.e., active or inactive),
+a weight and some credits... and all these things interact in a rather
+involved way. Also, with years of use, things have gotten even more
+complex (due to, e.g., the introduction of boosting, caps and vCPU
+soft-affinity).
+
+Dealing with such complexity is starting to be an issue. Odd behavior
+or subtle scheduling anomalies, that is not always possible to act upon,
+have been identified already. [1][2][3]
+
+A certain lack of scalability and difficulties and weakness in dealing
+with mixed workloads and VMs with low latency requirements are other
+known problems. [4] For all these reasons, effort is ongoing to have
+Credit2 become the new default scheduler.
+
+# Testing
+
+Any change to Credit code must to be tested by doing at least the following:
+
+* create a few virtual machine and verify that they boot and can
+  run some basic workload (e.g., login into them and run simple commands),
+* shutdown/reboot the virtual machines,
+* shutdown the system.
+
+Ideally, all the above steps should **also** be performed in a configuration
+that includes cpupools, better if with pools using different schedulers, and
+by also doing the following:
+
+* move the virtual machines between cpupools.
+
+# References
+
+* [potential non-ideal behavior on hyperthreaded systems](https://lists.xenproject.org/archives/html/xen-devel/2014-07/msg01848.html) [1]
+* [long standing BOOST vs. migration bug](https://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html) [2]
+* [priority handling issues](https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg01362.html) [3]
+* "Scheduler development update", XenSummit Asia 2009 [whitepaper](http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf) [4]
+* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf)
+* [Xen Project Scheduler](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+2016-10-10 1        Xen 4.8  Document written
+---------- -------- -------- -------------------------------------------


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 1/3] docs: Credit1 feature document.

[Wei Liu]

On Thu, Oct 13, 2016 at 12:02:28PM +0100, Dario Faggioli wrote:
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
> ---
> Cc: George Dunlap <George.Dunlap@eu.citrix.com>
> Cc: Wei Liu <wei.liu2@citrix.com>
> Cc: Lars Kurth <lars.kurth@citrix.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Cc: Stefano Stabellini <sstabellini@kernel.org>
> ---
>  docs/features/credit.pandoc |   99 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 99 insertions(+)
>  create mode 100644 docs/features/credit.pandoc
> 
> diff --git a/docs/features/credit.pandoc b/docs/features/credit.pandoc
> new file mode 100644
> index 0000000..fed0da2
> --- /dev/null
> +++ b/docs/features/credit.pandoc
> @@ -0,0 +1,99 @@
> +% Credit
> +% Revision 1
> +
> +\clearpage
> +
> +# Basics
> +---------------- ----------------------------------------------------
> +         Status: e.g. **Supported**
> +

Delete the "e.g." in all three documents, please.

> +Architecture(s): e.g. x86, arm
> +

And this should probably be "any".

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 1/3] docs: Credit1 feature document.

[Andrew Cooper]

On 13/10/16 12:02, Dario Faggioli wrote:
> diff --git a/docs/features/credit.pandoc b/docs/features/credit.pandoc
> new file mode 100644
> index 0000000..fed0da2
> --- /dev/null
> +++ b/docs/features/credit.pandoc

Simply "Credit" as a top level feature isn't very descriptive.  Can you
see about working scheduler somewhere into the name?

> @@ -0,0 +1,99 @@
> +% Credit
> +% Revision 1
> +
> +\clearpage
> +
> +# Basics
> +---------------- ----------------------------------------------------
> +         Status: e.g. **Supported**
> +
> +Architecture(s): e.g. x86, arm
> +
> +      Component: e.g. Hypervisor
> +---------------- ----------------------------------------------------

You should drop the e.g.'s.  In cases like this where it really is just
a software algorithm, I would suggest setting the architecture to all,
or omitting the line entirely.  I would expect that any new architecture
is going gain all the schedulers without modification?

> +
> +# Overview
> +
> +Credit (also known as Credit1) is the default virtual CPU (vCPU) scheduler
> +of the Xen hypervisor. The job of an hypervisor's scheduler is to decide,
> +among all the various vCPUs of the various virtual machines, which ones
> +should execute on the host's physical CPUs (pCPUs), at any given point in
> +time.

A lot of this is generic to all schedulers.

I wonder whether it might be better to have a schedulers meta-feature
doc which deals with the common scheduler parts, including interactions
on the Xen command line, xl, etc.

> +
> +# User details
> +
> +Xen supports multiple schedulers. As said, Credit is the default, so it
> +is used automatically, unless the `sched=$SCHED` (with `$SCHED` different
> +than `credit`) parameter is passed to Xen via the bootloader.
> +
> +Once the system is live, for creating a cpupool with Credit as its
> +scheduler, either compile a cpupool configuration file, as described
> +in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified in
> +`tools/examples/cpupool`), or use just `xl` directly:

I should see about ensuring that cross-references work with the
HTML-generated versions of these docs.  You might be able to get away
with just putting in a plain hyperlink here.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 1/3] docs: Credit1 feature document.

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4137 bytes --]

On Thu, 2016-10-13 at 12:47 +0100, Andrew Cooper wrote:
> On 13/10/16 12:02, Dario Faggioli wrote:
> > 
> > diff --git a/docs/features/credit.pandoc
> > b/docs/features/credit.pandoc
> > new file mode 100644
> > index 0000000..fed0da2
> > --- /dev/null
> > +++ b/docs/features/credit.pandoc
> 
> Simply "Credit" as a top level feature isn't very descriptive.  Can
> you
> see about working scheduler somewhere into the name?
> 
Yep, I wasn't sure whether or not to do that. Re-thinking things, I
agree that'd be better. I'll do.

> > @@ -0,0 +1,99 @@
> > +% Credit
> > +% Revision 1
> > +
> > +\clearpage
> > +
> > +# Basics
> > +---------------- -----------------------------------------------
> > -----
> > +         Status: e.g. **Supported**
> > +
> > +Architecture(s): e.g. x86, arm
> > +
> > +      Component: e.g. Hypervisor
> > +---------------- -----------------------------------------------
> > -----
> 
> You should drop the e.g.'s.  
>
Which I was sure I'd have done... sorry.

> In cases like this where it really is just
> a software algorithm, I would suggest setting the architecture to
> all,
> or omitting the line entirely.  
>
Omitting the line is what I also was considering myself. Again, will
do.

> > +# Overview
> > +
> > +Credit (also known as Credit1) is the default virtual CPU (vCPU)
> > scheduler
> > +of the Xen hypervisor. The job of an hypervisor's scheduler is to
> > decide,
> > +among all the various vCPUs of the various virtual machines, which
> > ones
> > +should execute on the host's physical CPUs (pCPUs), at any given
> > point in
> > +time.
> 
> A lot of this is generic to all schedulers.
> 
Not really. Well, sure some is, but, at the end, this period is pretty
much the only one that is present, identical to itself, in all the
three documents (and I certainly can see about shortening or removing
it, if we don't want that).


And in fact, the rest...

> I wonder whether it might be better to have a schedulers meta-feature
> doc which deals with the common scheduler parts, including
> interactions
> on the Xen command line, xl, etc.
> 
...may look similar, but they're subtle differences spread around. And
the more subtle those differences, the higher the amount of cross-
referencing between different documents would be, making it more
difficult to read and understand what the situation is for one specific
scheduler.

xl interface is a good example: sub-commands are very similar, but then
the scheduling parameters are different for each scheduler.

The way in which you create a cpupool is the same (modulo the name=""),
but doesn't necessarily have to be, e.g., if we start allowing
specifing some of the global parameters of the scheduler on the command
line (e.g., "create a Credit cpupool, but with timeslice=10"). Not
possible right now, but doable, and even convenient (I've already have
plans for that :-P).

So, FWIW, I would stick with different documents.

> > +Once the system is live, for creating a cpupool with Credit as its
> > +scheduler, either compile a cpupool configuration file, as
> > described
> > +in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified in
> > +`tools/examples/cpupool`), or use just `xl` directly:
> 
> I should see about ensuring that cross-references work with the
> HTML-generated versions of these docs.  You might be able to get away
> with just putting in a plain hyperlink here.
> 
I thought about that, but then ended up following suit from your
docs/feature/migration.pandoc.

I'll turn this in links if that's what you think is best. Personally, I
's say it makes the _text_ document a bit less readable, but I guess
the version we care about is the _HTML_ one?

Anyway, I'm basically ok with anything. :-)

Thanks and Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 2/3] docs: Credit2 feature document.

[Dario Faggioli]

Since we are marking the feature as 'Supported', remove the
"this is experimental software" warning in the code at once.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Anshul Makkar <anshul.makkar@citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Lars Kurth <lars.kurth@citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
---
 docs/features/credit2.pandoc |  123 ++++++++++++++++++++++++++++++++++++++++++
 xen/common/sched_credit2.c   |    2 -
 2 files changed, 123 insertions(+), 2 deletions(-)
 create mode 100644 docs/features/credit2.pandoc

diff --git a/docs/features/credit2.pandoc b/docs/features/credit2.pandoc
new file mode 100644
index 0000000..6203cec
--- /dev/null
+++ b/docs/features/credit2.pandoc
@@ -0,0 +1,123 @@
+% Credit2
+% Revision 1
+
+\clearpage
+
+# Basics
+---------------- ----------------------------------------------------
+         Status: e.g. **Supported**
+
+Architecture(s): e.g. x86, arm
+
+      Component: e.g. Hypervisor
+---------------- ----------------------------------------------------
+
+# Overview
+
+Credit2 is one of the virtual CPU (vCPU) scheduler available in the
+Xen hypervisor. The job of an hypervisor's virtual CPU scheduler is
+to decide, among all the various vCPUs of the various virtual machines,
+which ones should execute on the host's physical CPUs (pCPUs), at any
+given point in time.
+
+Credit2 was designed as a general purpose scheduler, with particular
+focus on improving handling of mixed workloads, scalability and
+support for low latency applications inside VMs, with respect to
+Credit1.
+
+# User details
+
+Credit2 is not in use by default. In order to use it as the Xen
+scheduler the following parameter should be passed to the hypervisor
+at boot:
+
+    `sched=credit2`
+
+Other parameters are available for tuning the behavior of Credit2
+(see `docs/misc/xen-command-line.markdown` for a complete list and
+for their meaning).
+
+Once the system is live, for creating a cpupool with Credit2 as
+its scheduler, either compile a cpupool configuration file, as
+described in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified
+in `tools/examples/cpupool`), or use just `xl` directly:
+
+    xl cpupool-create name=\"pool1\" sched=\"credit2\" cpus=[1,2]
+
+Two kind of interactions with the scheduler are possible:
+
+* checking or changing the global parameters, via, e.g.:
+    * `xl sched-credit2 -s`
+    * `xl sched-credit2 -s -p pool1`
+    * `xl sched-credit2 -s -r 100`
+* checking or changing a VM scheduling parameters, via, e.g.:
+    * `xl sched-credit2 -d vm1`
+    * `xl sched-credit2 -d vm1 -w 1024`
+
+# Technical details
+
+The implementation entirely lives in the hypervisor. Xen has a
+pluggable, hook based, architecture for schedulers. Credit2 code
+is all inside one file: `xen/common/sched_credit2.c`
+
+Global scheduling parameters, such as context switching rate
+limiting, is only available from Xen 4.8 onward. In libxl, the
+`LIBXL\_HAVE\_SCHED\_CREDIT2\_PARAMS` symbol is introduced to
+indicate their availability.
+
+# Limitations
+
+The Credit1 scheduler supports vCPU hard-affinity, vCPU soft-affinity
+and caps (see `docs/man/xl.pod.1.in` for more details). In Credit2,
+vCPU hard affinity is supported starting from Xen 4.8, while soft-affinity
+and caps are being implemented, but not supported yet in any released
+hypervisor.
+
+# Testing
+
+Any change done in Credit2 wants to be tested by doing at least the
+following:
+
+* boot the system with `sched=credit2`,
+* create a few virtual machine and verify that they boot and can
+  run some basic workload (e.g., login into them and run simple commands),
+* shutdown/reboot the virtual machines,
+* shutdown/reboot the system.
+
+Ideally, all the above steps should **also** be performed in a configuration
+where Credit2 is used as the scheduler of a cpupool, and by also doing the
+following:
+
+* move a virtual machine inside and outside a Credit2 cpupool.
+
+# Areas for improvement
+
+* Close the feature gap with Credit1 (i.e., finishing implementing vCPU
+  soft-affinity and caps);
+* vCPUs' reservations (similar to caps, but providing a vCPU with guarantees
+  about some pCPU time it will always be able to execute for);
+* benchmarking for assessing the best combination of values for the various
+  parameters (`sched\_credit2\_migrate\_resist`, `credit2\_balance\_over`,
+  `credit2\_balance\_under`)
+
+# Known issues
+
+* I/O oriented benchmarks (like network and disk throughput) have given
+  contradictory and non-conclusive results so far. Need to run more of
+  those.
+
+# References
+
+* "Scheduler development update", XenSummit Asia 2009 [whitepaper](http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf)
+* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf)
+* "Scope and Performance of Credit-2 Scheduler" [XPDS16 Presentation](http://www.slideshare.net/xen_com_mgr/xpds16-scope-and-performance-of-credit2-scheduler-anshul-makkar-ctirix-systems-uk-ltd)
+* [Credit2 Scheduler Development](https://wiki.xenproject.org/wiki/Credit2_Scheduler_Development)
+* [Xen Project Scheduler](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+2016-10-10 1        Xen 4.8  Document written
+---------- -------- -------- -------------------------------------------
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index fe46e80..1f26553 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -2954,8 +2954,6 @@ csched2_init(struct scheduler *ops)
     struct csched2_private *prv;
 
     printk("Initializing Credit2 scheduler\n");
-    printk(" WARNING: This is experimental software in development.\n" \
-           " Use at your own risk.\n");
 
     printk(XENLOG_INFO " load_precision_shift: %d\n"
            XENLOG_INFO " load_window_shift: %d\n"


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 3/3] docs: RTDS feature document.

[Dario Faggioli]

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: Meng Xu <mengxu@cis.upenn.edu>
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Lars Kurth <lars.kurth@citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
---
 docs/features/rtds.pandoc |  125 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 125 insertions(+)
 create mode 100644 docs/features/rtds.pandoc

diff --git a/docs/features/rtds.pandoc b/docs/features/rtds.pandoc
new file mode 100644
index 0000000..76aa165
--- /dev/null
+++ b/docs/features/rtds.pandoc
@@ -0,0 +1,125 @@
+% RTDS
+% Revision 1
+
+\clearpage
+
+# Basics
+---------------- ----------------------------------------------------
+         Status: e.g. **Experimental**
+
+Architecture(s): e.g. x86, arm
+
+      Component: e.g. Hypervisor
+---------------- ----------------------------------------------------
+
+# Overview
+
+RTDS is one of the virtual CPU (vCPU) scheduler available in the Xen
+hypervisor. The job of an hypervisor's virtual CPU scheduler is to
+decide, among all the various vCPUs of the various virtual machines,
+which ones should execute on the host's physical CPUs (pCPUs), at any
+given point in time.
+
+RTDS is a real--time scheduler, so its purpose is enabling
+**deterministic** scheduling of the virtual machine's vCPUs. It has
+been originally developed in the context of the RT-Xen project.
+
+# User details
+
+RTDS is not in use by default. In order to use it as the Xen scheduler
+the following parameter should be passed to the hypervisor at boot:
+
+    `sched=rtds`
+
+Once the system is live, for creating a cpupool with RTDS as its
+scheduler, either compile a cpupool configuration file, as described
+in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified in
+`tools/examples/cpupool`), or use just `xl` directly:
+
+    xl cpupool-create name=\"pool-rt\" sched=\"rtds\" cpus=[4,5,6,8]
+
+For checking or changing a VM's scheduling parameters from xl, do
+as follows:
+    * `xl sched-rtds -d vm-rt`
+    * `xl sched-rtds -d vm-rt -t 10000 -b 25000`
+
+It is possible, for a multiple vCPUs VM, to change the parameters of
+each vCPU individually:
+    * `xl sched-rtds -d vm-rt -v 0 -p 20000 -b 10000 -v 1 -p 45000 -b 12000`
+
+# Technical details
+
+The implementation entirely lives in the hypervisor. Xen has a
+pluggable, hook based, architecture for schedulers. RTDS code
+is all inside one file: `xen/common/sched_rt.c`
+
+In libxl, the availability of the RTDS scheduler is advertised by
+the presence of the LIBXL\_HAVE\_SCHED\_RTDS symbol. The ability of
+specifying different scheduling parameters for each vcpu has been
+introduced later, and is available if the following symbols are defined:
+    * `LIBXL\_HAVE\_VCPU\_SCHED\_PARAMS`,
+    * `LIBXL\_HAVE\_SCHED\_RTDS\_VCPU\_PARAMS`.
+
+# Limitations
+
+RTDS is a special purpose scheduling. This is by design, and not at
+all a limitation, but it is certainly something to keep in mind when
+thinking about using it. The purpose of the scheduler is enabling
+deterministic and statically analyzable behavior (as per the
+real-time academic literature), according to the scheduling parameters
+assigned to each vCPU.
+
+Using RTDS a the Xen scheduler, and/or for general purpose workloads
+is definitely possible, but the vCPU scheduling parameters (of both
+Domain0 and of the various VMs) would probably require tweaking, with
+respect to their default values.
+
+# Testing
+
+Any change done in RTDS must be tested by doing the following:
+
+* create a cpupool with RTDS as its scheduler,
+* create a few virtual machines a move them in and out of the pool,
+* create a few virtual machines, directly inside the pool, and verify
+  that they boot and can run some basic workload (e.g., login into them
+  and run simple commands),
+* shutdown/reboot the virtual machines,
+
+The fact that the system boots fine when passing `sched=rtds` to Xen
+should also be verified.
+
+Finally, to check that the scheduler is working properly (although only
+at a macroscopic level), the following should be done:
+
+* create a VM with 1 vCPU and put it in the RTDS cpupool,
+* set the scheduling parameters such as it has a 50% reservation, with
+  `xl sched-rtds -d vm -t 100000 -b 50000`,
+* run a CPU-burning process inside the VM (e.g., `yes`),
+* check with `xentop` (in Domain0) that the VM is getting no more than
+  50% pCPU time.
+
+# Areas for improvement
+
+* Work-conserving mode to be added;
+* performance assessment, especially focusing on what level of real-time
+  behavior the scheduler enables.
+
+# Known issues
+
+* OSSTest reports occasional failures on ARM.
+
+# References
+
+* "RT-Xen: Real-Time Virtualization" [XPDS14 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/2014_Xen_Developer_Summit_0.pdf)
+* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf)
+* [RT-Xen Project](https://sites.google.com/site/realtimexen/)
+* [RTDS-Based-Scheduler](https://wiki.xenproject.org/wiki/RTDS-Based-Scheduler)
+* [Xen Project Scheduler](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+2016-10-13 1        Xen 4.8  Document written
+---------- -------- -------- -------------------------------------------


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Andrew Cooper]

On 13/10/16 12:01, Dario Faggioli wrote:
> Hey,
>
> "Just" as per the subject, I wrote feature documents for (almost) all our
> schedulers. No big deal, I'd say, apart from the fact that I'm declaring
> Credit2 **Supperted**, instead of experimental.

Supperted?  That's like supported right? ;p


It is fine for you to propose that a feature should be upgraded to
supported, and this is probably the best way to formally do so.

However, final agreement of a feature becoming supported should include
input from the security team. (At the end of the day, it is us with
extra work if the feature isn't up to scratch.)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[George Dunlap]

On 13/10/16 12:28, Andrew Cooper wrote:
> On 13/10/16 12:01, Dario Faggioli wrote:
>> Hey,
>>
>> "Just" as per the subject, I wrote feature documents for (almost) all our
>> schedulers. No big deal, I'd say, apart from the fact that I'm declaring
>> Credit2 **Supperted**, instead of experimental.
> 
> Supperted?  That's like supported right? ;p
> 
> 
> It is fine for you to propose that a feature should be upgraded to
> supported, and this is probably the best way to formally do so.
> 
> However, final agreement of a feature becoming supported should include
> input from the security team. (At the end of the day, it is us with
> extra work if the feature isn't up to scratch.)

Yes, interesting idea.  I don't think in our discussions of feature
lifecycle maturity we'd talked about that yet.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1214 bytes --]

On Thu, 2016-10-13 at 12:28 +0100, Andrew Cooper wrote:
> On 13/10/16 12:01, Dario Faggioli wrote:
> > "Just" as per the subject, I wrote feature documents for (almost)
> > all our
> > schedulers. No big deal, I'd say, apart from the fact that I'm
> > declaring
> > Credit2 **Supperted**, instead of experimental.
> 
> Supperted?  That's like supported right? ;p
> 
Nah, 'supperted' is the new 'supported', didn't you know? :-P

> It is fine for you to propose that a feature should be upgraded to
> supported, and this is probably the best way to formally do so.
> 
> However, final agreement of a feature becoming supported should
> include
> input from the security team. (At the end of the day, it is us with
> extra work if the feature isn't up to scratch.)
> 
Ok, so, if that's the case, what's the process: resend (this patch) --
or some other kind of formal request-- with security@xenproject.org
Cc-ed?

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Wei Liu]

On Thu, Oct 13, 2016 at 01:44:28PM +0100, Dario Faggioli wrote:
> On Thu, 2016-10-13 at 12:28 +0100, Andrew Cooper wrote:
> > On 13/10/16 12:01, Dario Faggioli wrote:
> > > "Just" as per the subject, I wrote feature documents for (almost)
> > > all our
> > > schedulers. No big deal, I'd say, apart from the fact that I'm
> > > declaring
> > > Credit2 **Supperted**, instead of experimental.
> > 
> > Supperted?  That's like supported right? ;p
> > 
> Nah, 'supperted' is the new 'supported', didn't you know? :-P
> 
> > It is fine for you to propose that a feature should be upgraded to
> > supported, and this is probably the best way to formally do so.
> > 
> > However, final agreement of a feature becoming supported should
> > include
> > input from the security team. (At the end of the day, it is us with
> > extra work if the feature isn't up to scratch.)
> > 
> Ok, so, if that's the case, what's the process: resend (this patch) --
> or some other kind of formal request-- with security@xenproject.org
> Cc-ed?
> 

If you want these to be applied more quickly the best thing to do is to
leave the status experimental and later send another patch with
security@ CC'ed to change the status.

Wei.

> Regards,
> Dario
> -- 
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 907 bytes --]

On Thu, 2016-10-13 at 13:46 +0100, Wei Liu wrote:
> On Thu, Oct 13, 2016 at 01:44:28PM +0100, Dario Faggioli wrote:
> > 
> > Ok, so, if that's the case, what's the process: resend (this patch)
> > --
> > or some other kind of formal request-- with security@xenproject.org
> > Cc-ed?
> > 
> 
> If you want these to be applied more quickly the best thing to do is
> to
> leave the status experimental and later send another patch with
> security@ CC'ed to change the status.
> 
I like this as a way forward... lemme see if I can do that before going
to catch a train that will make me catch a plane! :-D

Thanks and Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Stefano Stabellini]

On Thu, 13 Oct 2016, Andrew Cooper wrote:
> On 13/10/16 12:01, Dario Faggioli wrote:
> > Hey,
> >
> > "Just" as per the subject, I wrote feature documents for (almost) all our
> > schedulers. No big deal, I'd say, apart from the fact that I'm declaring
> > Credit2 **Supperted**, instead of experimental.
> 
> Supperted?  That's like supported right? ;p
> 
> 
> It is fine for you to propose that a feature should be upgraded to
> supported, and this is probably the best way to formally do so.
> 
> However, final agreement of a feature becoming supported should include
> input from the security team. (At the end of the day, it is us with
> extra work if the feature isn't up to scratch.)

Is this new? If so, should we formalize the change in process somewhere
(patch to governance, etc.)?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Konrad Rzeszutek Wilk]

On October 13, 2016 2:13:19 PM EDT, Stefano Stabellini <sstabellini@kernel.org> wrote:
>On Thu, 13 Oct 2016, Andrew Cooper wrote:
>> On 13/10/16 12:01, Dario Faggioli wrote:
>> > Hey,
>> >
>> > "Just" as per the subject, I wrote feature documents for (almost)
>all our
>> > schedulers. No big deal, I'd say, apart from the fact that I'm
>declaring
>> > Credit2 **Supperted**, instead of experimental.
>> 
>> Supperted?  That's like supported right? ;p
>> 
>> 
>> It is fine for you to propose that a feature should be upgraded to
>> supported, and this is probably the best way to formally do so.
>> 
>> However, final agreement of a feature becoming supported should
>include
>> input from the security team. (At the end of the day, it is us with
>> extra work if the feature isn't up to scratch.)
>
>Is this new? If so, should we formalize the change in process somewhere
>(patch to governance, etc.)?

This came about when we had .. XSA7? Which was the tmem one and came with the idea that anything that moves to Supported has to pass the security audit pass.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Stefano Stabellini]

On Thu, 13 Oct 2016, Konrad Rzeszutek Wilk wrote:
> On October 13, 2016 2:13:19 PM EDT, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >On Thu, 13 Oct 2016, Andrew Cooper wrote:
> >> On 13/10/16 12:01, Dario Faggioli wrote:
> >> > Hey,
> >> >
> >> > "Just" as per the subject, I wrote feature documents for (almost)
> >all our
> >> > schedulers. No big deal, I'd say, apart from the fact that I'm
> >declaring
> >> > Credit2 **Supperted**, instead of experimental.
> >> 
> >> Supperted?  That's like supported right? ;p
> >> 
> >> 
> >> It is fine for you to propose that a feature should be upgraded to
> >> supported, and this is probably the best way to formally do so.
> >> 
> >> However, final agreement of a feature becoming supported should
> >include
> >> input from the security team. (At the end of the day, it is us with
> >> extra work if the feature isn't up to scratch.)
> >
> >Is this new? If so, should we formalize the change in process somewhere
> >(patch to governance, etc.)?
> 
> This came about when we had .. XSA7? Which was the tmem one and came with the idea that anything that moves to Supported has to pass the security audit pass.

Make sense. In that case we should definitely write it down somewhere. I
like the idea of keeping these info on pandoc on a git repo, like Lars
did with the governance.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Andrew Cooper]

On 13/10/2016 22:06, Stefano Stabellini wrote:
>
>>>>> Credit2 **Supperted**, instead of experimental.
>>>> Supperted?  That's like supported right? ;p
>>>>
>>>>
>>>> It is fine for you to propose that a feature should be upgraded to
>>>> supported, and this is probably the best way to formally do so.
>>>>
>>>> However, final agreement of a feature becoming supported should
>>> include
>>>> input from the security team. (At the end of the day, it is us with
>>>> extra work if the feature isn't up to scratch.)
>>> Is this new? If so, should we formalize the change in process somewhere
>>> (patch to governance, etc.)?
>> This came about when we had .. XSA7? Which was the tmem one and came with the idea that anything that moves to Supported has to pass the security audit pass.

Off the top of my head, XSA-7 is the Intel silicon sysret bug.  ITYM
XSA-15, but your point still stands.

> Make sense. In that case we should definitely write it down somewhere.

My entire purpose with introducing feature docs to start with was to try
and do for feature what we now do very well for
docs/misc/xen-command-line.markdown  i.e. to keep the documentation
easy, and up to date.

This does involve the committers, maintainers and reviewers being
mindful to prompt for an update when necessary.  However, it is very
evident from the replies on-list these days that it has become second
nature to a lot of people to either patch the document in the first
place, or prompt others to do the same in submitted patches.

Overall, this is a very good position to be in, for the community as a
whole.  I am certainly hoping that in another couple of years, we as a
community have exactly the same "second nature" mindset when it comes to
keeping the feature docs up to date.

> I like the idea of keeping these info on pandoc on a git repo, like Lars
> did with the governance.

I should hasten to add that perhaps picking on the security team in
isolation was a poor move on my part, for which I apologise. There are
multiple involved parties with different interests, all of which need
attending to.

The feature submitter (should) have a vested interest in getting code
submitted (even as experimental/tech preview to start with), so that
others can trial/debug/extend the feature.  Even having code upstream
but disabled-by-default in Kconfig is a far better position for others
to start from, than to be provided with instructions saying "there was a
patch series several months ago on a mailing list".

The reviewers/maintainers (should) have a vested interest in keeping the
overall quality up.  Due to their appointment within the community, they
have demonstrated knowledge and expertise in their respective areas, and
the overall project relies on them to ask questions such as "How does $X
work with $Y?" or "Have you considered $Z as an alternative?", and to
come to a fair and unbiased opinion to the quality and worthiness of a
feature.

The release manager has conflicted vested interests.  On the one hand,
getting more features in a release is better on paper and in the press,
but can certainly backfire when features aren't up to scratch. 
Ultimately, if features in a release are substandard, the downstreams
and end users will be the ones to bear the pain.  It is in the release
managers best interest to ensure that the features which are finally
accepted are actually ready.

Nothing is ever perfect, but a whole lot of stuff is perfectly good in
common case, and useful for people in general.  This is why the feature
doc template specifically has sections for know issues to be fixed, and
areas for future improvement.  Any documentation (so long as it isn't
factually incorrect) is better than nothing.

There seems to be general consensus that a status of "Supported" means
"with security support", and I agree with this assessment.  Ultimately,
the security team is accountable to whatever the project as a whole
declares "Supported", but as the security team are all commiters and
maintainer there is a large overlap of responsibility.  OTOH, the
security team are also responsible for managing the fallout when things
go wrong.

XSAs are bad publicity, and are a problem for downstreams (remember that
we have plenty of downstreams who measure quantity of servers in units
of a datacenter).  Another problem we as a community have is that no
support status is written down; there is a 13ish? year backlog in this
regard.  Writing details down in feature docs is intended to get
everyone on the same, un-ambiguous, page.

We also have an unfortunate habit of new features appearing, being
accepted, and (intentionally or unintentionally) available to guests. 
Fastforward a bit, and it is discovered that said feature resembles a
sieve in terms of security holes, and the common recourse is to publish
an XSA stating "turn it off and don't use it with untrusted guests", or
"here is a patch to actually turn it off, and still don't use it with
untrusted guests", because we really can't re-engineer a feature in a
security patch.  (Sorry to pick on TMEM here, but it is now 4 years of
development later and it still has a lot of development left to go. 
Reworking features in the light of a security hole can be very
complicated, and lots of work.)

There should be a high barrier to "Supported" status, because the cost
of getting it wrong is equally high.  However, there are perfectly
legitimate intermediate stages such as "Supported in these limited set
of circumstances".  A number of features are not plausibly for use in
production environments, but otherwise function fine for
development/investigatory purposes.  In these cases, something like "no
security support, but believed to be working fine" might be appropriate.

Having said all of this, I don't wish to deter feature submitters from
striving for "Supported".  Attaining "Supported" should be a accolade
that a feature is good from a technical point of view, with no important
missing parts.  I will certainly continue to strive for my features to
fit into this category, and I hope that the rest of the community will
hold my code to the same high standard that I hold others to.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Stefano Stabellini]

On Fri, 14 Oct 2016, Andrew Cooper wrote:
> > I like the idea of keeping these info on pandoc on a git repo, like Lars
> > did with the governance.
> 
> I should hasten to add that perhaps picking on the security team in
> isolation was a poor move on my part, for which I apologise. There are
> multiple involved parties with different interests, all of which need
> attending to.

No worries, on my part I just think that we should write down these
things to remember, set expectations and to avoid surprises.


> The feature submitter (should) have a vested interest in getting code
> submitted (even as experimental/tech preview to start with), so that
> others can trial/debug/extend the feature.  Even having code upstream
> but disabled-by-default in Kconfig is a far better position for others
> to start from, than to be provided with instructions saying "there was a
> patch series several months ago on a mailing list".

Undeniably true.

 
> The reviewers/maintainers (should) have a vested interest in keeping the
> overall quality up.  Due to their appointment within the community, they
> have demonstrated knowledge and expertise in their respective areas, and
> the overall project relies on them to ask questions such as "How does $X
> work with $Y?" or "Have you considered $Z as an alternative?", and to
> come to a fair and unbiased opinion to the quality and worthiness of a
> feature.
> 
> The release manager has conflicted vested interests.  On the one hand,
> getting more features in a release is better on paper and in the press,
> but can certainly backfire when features aren't up to scratch. 
> Ultimately, if features in a release are substandard, the downstreams
> and end users will be the ones to bear the pain.  It is in the release
> managers best interest to ensure that the features which are finally
> accepted are actually ready.
> 
> Nothing is ever perfect, but a whole lot of stuff is perfectly good in
> common case, and useful for people in general.  This is why the feature
> doc template specifically has sections for know issues to be fixed, and
> areas for future improvement.  Any documentation (so long as it isn't
> factually incorrect) is better than nothing.
> 
> There seems to be general consensus that a status of "Supported" means
> "with security support", and I agree with this assessment.  Ultimately,
> the security team is accountable to whatever the project as a whole
> declares "Supported", but as the security team are all commiters and
> maintainer there is a large overlap of responsibility.  OTOH, the
> security team are also responsible for managing the fallout when things
> go wrong.
> 
> XSAs are bad publicity, and are a problem for downstreams (remember that
> we have plenty of downstreams who measure quantity of servers in units
> of a datacenter).  Another problem we as a community have is that no
> support status is written down; there is a 13ish? year backlog in this
> regard.  Writing details down in feature docs is intended to get
> everyone on the same, un-ambiguous, page.
> 
> We also have an unfortunate habit of new features appearing, being
> accepted, and (intentionally or unintentionally) available to guests. 
> Fastforward a bit, and it is discovered that said feature resembles a
> sieve in terms of security holes, and the common recourse is to publish
> an XSA stating "turn it off and don't use it with untrusted guests", or
> "here is a patch to actually turn it off, and still don't use it with
> untrusted guests", because we really can't re-engineer a feature in a
> security patch.  (Sorry to pick on TMEM here, but it is now 4 years of
> development later and it still has a lot of development left to go. 
> Reworking features in the light of a security hole can be very
> complicated, and lots of work.)

Yes, that's definitely a position we should try not to be in.


> There should be a high barrier to "Supported" status, because the cost
> of getting it wrong is equally high.  However, there are perfectly
> legitimate intermediate stages such as "Supported in these limited set
> of circumstances".  A number of features are not plausibly for use in
> production environments, but otherwise function fine for
> development/investigatory purposes.  In these cases, something like "no
> security support, but believed to be working fine" might be appropriate.

I agree on this. I think we need an intermediate step: "working but not
supported for security" is completely reasonable. When we say that it is
"working", it should be because we have automated testing for it (I
don't know if I would go as far as requiring it to be in OSSTest, any
automated testing, even third party, would do). If it is not
automatically tested, then it is just "best effort".

Finally some features involve some sort of ABI exposed to the guest. At
some point when the feature is "Supported", the ABI will be most
certainly maintained for backward compatibility. But the feature could
be fully working, automatically tested, but still the ABI could be not
stable. Think of the early days of libxl and the initial ARM support in
Linux. Both working, tested, but the ABI wasn't stable. Often asking
contributors to make the ABI stable to begin with could be setting the
bar too high for an initial contribution.

Something like:

"best effort" < "working (unstable ABI)" < "working (stable ABI)" < "supported"


> Having said all of this, I don't wish to deter feature submitters from
> striving for "Supported".  Attaining "Supported" should be a accolade
> that a feature is good from a technical point of view, with no important
> missing parts.  I will certainly continue to strive for my features to
> fit into this category, and I hope that the rest of the community will
> hold my code to the same high standard that I hold others to.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Jan Beulich]

>>> On 14.10.16 at 02:58, <sstabellini@kernel.org> wrote:
> On Fri, 14 Oct 2016, Andrew Cooper wrote:
>> There should be a high barrier to "Supported" status, because the cost
>> of getting it wrong is equally high.  However, there are perfectly
>> legitimate intermediate stages such as "Supported in these limited set
>> of circumstances".  A number of features are not plausibly for use in
>> production environments, but otherwise function fine for
>> development/investigatory purposes.  In these cases, something like "no
>> security support, but believed to be working fine" might be appropriate.
> 
> I agree on this. I think we need an intermediate step: "working but not
> supported for security" is completely reasonable. When we say that it is
> "working", it should be because we have automated testing for it (I
> don't know if I would go as far as requiring it to be in OSSTest, any
> automated testing, even third party, would do). If it is not
> automatically tested, then it is just "best effort".

I don't think this is a reasonable expectation - how would you envision
testing the dozens of command line options alone, not to speak of
things depending on hardware characteristics?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[George Dunlap]

On 14/10/16 07:36, Jan Beulich wrote:
>>>> On 14.10.16 at 02:58, <sstabellini@kernel.org> wrote:
>> On Fri, 14 Oct 2016, Andrew Cooper wrote:
>>> There should be a high barrier to "Supported" status, because the cost
>>> of getting it wrong is equally high.  However, there are perfectly
>>> legitimate intermediate stages such as "Supported in these limited set
>>> of circumstances".  A number of features are not plausibly for use in
>>> production environments, but otherwise function fine for
>>> development/investigatory purposes.  In these cases, something like "no
>>> security support, but believed to be working fine" might be appropriate.
>>
>> I agree on this. I think we need an intermediate step: "working but not
>> supported for security" is completely reasonable. When we say that it is
>> "working", it should be because we have automated testing for it (I
>> don't know if I would go as far as requiring it to be in OSSTest, any
>> automated testing, even third party, would do). If it is not
>> automatically tested, then it is just "best effort".
> 
> I don't think this is a reasonable expectation - how would you envision
> testing the dozens of command line options alone, not to speak of
> things depending on hardware characteristics?

Well there may be situations where we can make reasonable exceptions.
But it would certainly be a lot better if a feature wasn't considered
"done" until there was something in place to make sure that it worked
and continued to work, other than "we hope people use it and report any
bugs they find".

The more interesting aspect of Stefano's suggestion here is whether
there should be two levels of "supported" -- "supported" as in, "this
works but it's not in our security boundary", and "supported" as in,
"this works and it *is* in our security boundary".

But as we don't *yet* have such a decision-making process in place, I
think we need to approach each change in an ad-hoc manner.  Having a
discussion about *credit2* which includes the security aspects makes
sense, and I don't think we need to wait until we've got a generalized
framework to make a reasonable decision about that.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Jan Beulich]

>>> On 14.10.16 at 11:59, <george.dunlap@citrix.com> wrote:
> On 14/10/16 07:36, Jan Beulich wrote:
>>>>> On 14.10.16 at 02:58, <sstabellini@kernel.org> wrote:
>>> On Fri, 14 Oct 2016, Andrew Cooper wrote:
>>>> There should be a high barrier to "Supported" status, because the cost
>>>> of getting it wrong is equally high.  However, there are perfectly
>>>> legitimate intermediate stages such as "Supported in these limited set
>>>> of circumstances".  A number of features are not plausibly for use in
>>>> production environments, but otherwise function fine for
>>>> development/investigatory purposes.  In these cases, something like "no
>>>> security support, but believed to be working fine" might be appropriate.
>>>
>>> I agree on this. I think we need an intermediate step: "working but not
>>> supported for security" is completely reasonable. When we say that it is
>>> "working", it should be because we have automated testing for it (I
>>> don't know if I would go as far as requiring it to be in OSSTest, any
>>> automated testing, even third party, would do). If it is not
>>> automatically tested, then it is just "best effort".
>> 
>> I don't think this is a reasonable expectation - how would you envision
>> testing the dozens of command line options alone, not to speak of
>> things depending on hardware characteristics?
> 
> Well there may be situations where we can make reasonable exceptions.
> But it would certainly be a lot better if a feature wasn't considered
> "done" until there was something in place to make sure that it worked
> and continued to work, other than "we hope people use it and report any
> bugs they find".

Perhaps an earlier question here is what a "feature" is. My command
line option example was specifically chosen to make it possibly very
small scope, yet indicate an area where we would likely say "using
this and that option is not supported" (depending on the instance
with either of the two meanings you name below).

> The more interesting aspect of Stefano's suggestion here is whether
> there should be two levels of "supported" -- "supported" as in, "this
> works but it's not in our security boundary", and "supported" as in,
> "this works and it *is* in our security boundary".

To me the question is how far that would matter for people
wanting to use Xen in production: I for one wouldn't feel well
using features which aren't security supported.

> But as we don't *yet* have such a decision-making process in place, I
> think we need to approach each change in an ad-hoc manner.  Having a
> discussion about *credit2* which includes the security aspects makes
> sense, and I don't think we need to wait until we've got a generalized
> framework to make a reasonable decision about that.

Sure.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Stefano Stabellini]

On Fri, 14 Oct 2016, Jan Beulich wrote:
> >>> On 14.10.16 at 11:59, <george.dunlap@citrix.com> wrote:
> > On 14/10/16 07:36, Jan Beulich wrote:
> >>>>> On 14.10.16 at 02:58, <sstabellini@kernel.org> wrote:
> >>> On Fri, 14 Oct 2016, Andrew Cooper wrote:
> >>>> There should be a high barrier to "Supported" status, because the cost
> >>>> of getting it wrong is equally high.  However, there are perfectly
> >>>> legitimate intermediate stages such as "Supported in these limited set
> >>>> of circumstances".  A number of features are not plausibly for use in
> >>>> production environments, but otherwise function fine for
> >>>> development/investigatory purposes.  In these cases, something like "no
> >>>> security support, but believed to be working fine" might be appropriate.
> >>>
> >>> I agree on this. I think we need an intermediate step: "working but not
> >>> supported for security" is completely reasonable. When we say that it is
> >>> "working", it should be because we have automated testing for it (I
> >>> don't know if I would go as far as requiring it to be in OSSTest, any
> >>> automated testing, even third party, would do). If it is not
> >>> automatically tested, then it is just "best effort".
> >> 
> >> I don't think this is a reasonable expectation - how would you envision
> >> testing the dozens of command line options alone, not to speak of
> >> things depending on hardware characteristics?
> > 
> > Well there may be situations where we can make reasonable exceptions.
> > But it would certainly be a lot better if a feature wasn't considered
> > "done" until there was something in place to make sure that it worked
> > and continued to work, other than "we hope people use it and report any
> > bugs they find".
> 
> Perhaps an earlier question here is what a "feature" is. My command
> line option example was specifically chosen to make it possibly very
> small scope, yet indicate an area where we would likely say "using
> this and that option is not supported" (depending on the instance
> with either of the two meanings you name below).

This is another one of those cases where we need to apply our judgement
on a case by case basis.


> > The more interesting aspect of Stefano's suggestion here is whether
> > there should be two levels of "supported" -- "supported" as in, "this
> > works but it's not in our security boundary", and "supported" as in,
> > "this works and it *is* in our security boundary".
> 
> To me the question is how far that would matter for people
> wanting to use Xen in production: I for one wouldn't feel well
> using features which aren't security supported.

Sure, but Xen isn't only used in high availability production systems.
For example one use case for Xen on ARM is IoT, a space where most
devices don't even have a way to be securely updated yet. (I hope that
people using Xen on ARM in those scenarios do have a way to patch theirs
devices.)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/3 for 4.8] docs: feature documents for the schedulers

[Stefano Stabellini]

On Fri, 14 Oct 2016, George Dunlap wrote:
> On 14/10/16 07:36, Jan Beulich wrote:
> >>>> On 14.10.16 at 02:58, <sstabellini@kernel.org> wrote:
> >> On Fri, 14 Oct 2016, Andrew Cooper wrote:
> >>> There should be a high barrier to "Supported" status, because the cost
> >>> of getting it wrong is equally high.  However, there are perfectly
> >>> legitimate intermediate stages such as "Supported in these limited set
> >>> of circumstances".  A number of features are not plausibly for use in
> >>> production environments, but otherwise function fine for
> >>> development/investigatory purposes.  In these cases, something like "no
> >>> security support, but believed to be working fine" might be appropriate.
> >>
> >> I agree on this. I think we need an intermediate step: "working but not
> >> supported for security" is completely reasonable. When we say that it is
> >> "working", it should be because we have automated testing for it (I
> >> don't know if I would go as far as requiring it to be in OSSTest, any
> >> automated testing, even third party, would do). If it is not
> >> automatically tested, then it is just "best effort".
> > 
> > I don't think this is a reasonable expectation - how would you envision
> > testing the dozens of command line options alone, not to speak of
> > things depending on hardware characteristics?
>
> Well there may be situations where we can make reasonable exceptions.
> But it would certainly be a lot better if a feature wasn't considered
> "done" until there was something in place to make sure that it worked
> and continued to work, other than "we hope people use it and report any
> bugs they find".

It is difficult to generalize, because, as Jan wrote, some things come
with dozen of command line options, some other things come with none.
This is were we'll have to apply our judgment on a case by case basis.
But indeed the basic idea is that it is "done" when there is some
testing for it, where "some" is case specific. Community testing is
great, but automated testing is more reliable and predictable.

At the same time this is an Open Source community and we might get
contributions from people that aren't paid to work on Xen. We cannot
ask all contributors to write automated testing scripts, and maybe offer
security support. A contributor might write good code for a new feature
but refuse to write the testing for it. I think that's OK, we can take
that code, but then we need to clarify the state of that feature with
the community.

On one hand we don't want users to think a feature is fully stable and
supported when actually it is not. It negatively impacts Xen brand and
reputation. On the other hand we don't want to discourage contributions
by asking to commit to security support, automated testing, ABI
stability etc. from the start. Rome wasn't built in a day :-)
Can you imagine if we accepted ARM patches in Xen only after Xen on ARM
was ready for automated testing and security support?

The best way to achieve both goals at the same time is to clarify the
right level of quality/stability/support for each feature.


> The more interesting aspect of Stefano's suggestion here is whether
> there should be two levels of "supported" -- "supported" as in, "this
> works but it's not in our security boundary", and "supported" as in,
> "this works and it *is* in our security boundary".

I think the security team should be free to refuse to offer support for
something, which otherwise might be considered working and stable. It
might not even be a vote of distrust: after all the security team has
only a finite amount of resources, maybe not enough to cover all areas
of the project.


> But as we don't *yet* have such a decision-making process in place, I
> think we need to approach each change in an ad-hoc manner.  Having a
> discussion about *credit2* which includes the security aspects makes
> sense, and I don't think we need to wait until we've got a generalized
> framework to make a reasonable decision about that.

By no means I meant to delay Dario's work.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel