[RFC v2 for-4.6 0/2] In-tree feature documentation

[RFC v2 for-4.6 0/2] In-tree feature documentation

[Andrew Cooper]

An issue which Xen has is an uncertain support statement for features.
Given the success seen with docs/misc/xen-command-line.markdown, and in
particular keeping it up to date, introduce a similar system for
features.

Patch 1 introduces a proposed template (and a makefile tweak to include
the new docs/features subdirectory), while patch 2 is a feature document
covering the topic of migration.

v2 Adds %Revision and #History table, following feedback from v1.

This is tagged RFC as I expect people to have different views as to what
is useful to include.  I would particilarly appreciate feedback on the
template before it starts getting used widely.

Lars: Does this look like a reasonable counterpart to your formal
support statement document?

Jim: Per your request at the summit for new information, is patch 2
suitable?

Andrew Cooper (2):
  docs: Template for feature documents
  docs: Migration feature document

 docs/Makefile                  |    2 +-
 docs/features/migration.pandoc |  100 ++++++++++++++++++++++++++++++++++++++++
 docs/features/template.pandoc  |   67 +++++++++++++++++++++++++++
 3 files changed, 168 insertions(+), 1 deletion(-)
 create mode 100644 docs/features/migration.pandoc
 create mode 100644 docs/features/template.pandoc

-- 
1.7.10.4

[PATCH v2 for-4.6 1/2] docs: Template for feature documents

[Andrew Cooper]

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
v2: Added %Revision and #History table
---
 docs/Makefile                 |    2 +-
 docs/features/template.pandoc |   67 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 docs/features/template.pandoc

diff --git a/docs/Makefile b/docs/Makefile
index 272292c..5d620e5 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -16,7 +16,7 @@ MARKDOWNSRC-y := $(sort $(shell find misc -name '*.markdown' -print))
 
 TXTSRC-y := $(sort $(shell find misc -name '*.txt' -print))
 
-PANDOCSRC-y := $(sort $(shell find specs -name '*.pandoc' -print))
+PANDOCSRC-y := $(sort $(shell find features/ misc/ specs/ -name '*.pandoc' -print))
 
 # Documentation targets
 DOC_MAN1 := $(patsubst man/%.pod.1,man1/%.1,$(MAN1SRC-y))
diff --git a/docs/features/template.pandoc b/docs/features/template.pandoc
new file mode 100644
index 0000000..7479036
--- /dev/null
+++ b/docs/features/template.pandoc
@@ -0,0 +1,67 @@
+% Template for feature documents
+% Revision $N
+
+\clearpage
+
+This is a suggested template for formatting of a Xen feature document in
+tree.
+
+The purpose of this document is to provide a concrete support statement
+for the feature (indicating its security status), as well as brief user
+and technical documentation.
+
+# Basics
+
+A table with an overview of the support status and applicability.
+
+---------------- ----------------------------------------------------
+         Status: e.g. **Supported**/**Tech Preview**/**Experimental**
+
+Architecture(s): e.g. x86, arm
+
+   Component(s): e.g. Hypervisor, toolstack, guest
+
+       Hardware: _where applicable_
+---------------- ----------------------------------------------------
+
+# Overview
+
+A short description the feature, similar to an abstract for a
+paper/presentation.
+
+# User information
+
+Information for a user attempting to use the feature.  Should include
+how to enable the feature (is it enabled by default? If not, how to turn
+it on?), and how to interact with the feature (typically via `xl`).
+
+# Limitations
+
+Information concerning incompatibilities with other features or hardware
+combinations.
+
+# Technical information
+
+Information for a developer or power user.  Should include where to look
+in-tree for detailed documents and code.
+
+# Areas for improvement
+
+List of enhancements which could be undertaken, e.g. to improve the
+feature itself, or improve interaction with other features.
+
+# Known issues
+
+List of known issues or bugs.  For tech preview or experimental
+features, this section must contain the list of items needing fixing for
+its status to be upgraded.
+
+# History
+
+A table of changes to the document, in chronological order.
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+YYYY-MM-DD N        Xen X.Y  ...
+---------- -------- -------- -------------------------------------------
-- 
1.7.10.4

Re: [PATCH v2 for-4.6 1/2] docs: Template for feature documents

[Ian Campbell]

On Tue, 2015-08-25 at 11:40 +0100, Andrew Cooper wrote:
> 
> +A table with an overview of the support status and applicability.
> +
> +---------------- ----------------------------------------------------
> +         Status: e.g. **Supported**/**Tech Preview**/**Experimental**

These should be well defined somewhere and that definition linked to in the
template. Perhaps this is the "formal support statement document" mentioned
in the cover mail but which I haven't stumbled over in my (overflowing)
post vacation INBOX 

Re: [PATCH v2 for-4.6 1/2] docs: Template for feature documents

[Andrew Cooper]

On 01/09/15 14:41, Ian Campbell wrote:
> On Tue, 2015-08-25 at 11:40 +0100, Andrew Cooper wrote:
>> +A table with an overview of the support status and applicability.
>> +
>> +---------------- ----------------------------------------------------
>> +         Status: e.g. **Supported**/**Tech Preview**/**Experimental**
> These should be well defined somewhere and that definition linked to in the
> template. Perhaps this is the "formal support statement document" mentioned
> in the cover mail but which I haven't stumbled over in my (overflowing)
> post vacation INBOX 

It is referring to a thread of Lars', which is attempting to nail down
these definitions.

~Andrew

[PATCH v2 for-4.6 2/2] docs: Migration feature document

[Andrew Cooper]

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
v2:
 * %Revision and #History, following template review
 * Grammar fixes
---
 docs/features/migration.pandoc |  100 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 100 insertions(+)
 create mode 100644 docs/features/migration.pandoc

diff --git a/docs/features/migration.pandoc b/docs/features/migration.pandoc
new file mode 100644
index 0000000..0fd227f
--- /dev/null
+++ b/docs/features/migration.pandoc
@@ -0,0 +1,100 @@
+% Migration
+% Revision 1
+
+\clearpage
+
+# Basics
+--------------- -------------
+        Status: **Supported**
+
+  Architecture: x86
+
+     Component: Toolstack
+--------------- -------------
+
+# Overview
+
+Migration is a mechanism to move a virtual machine while the VM is
+running.  Live migration moves a running virtual machine between two
+physical servers, but the same mechanism can be used for non-live
+migrate (pause and copy) and suspend/resume from disk.
+
+# User details
+
+No hardware requirements, although hypervisor logdirty support is
+required for live migration.
+
+From the command line, `xl migrate/save/restore` are the top level
+interactions.  e.g.
+
+    xl create my-vm.cfg
+    xl migrate my-vm localhost
+
+or
+
+    xl create my-vm.cfg
+    xl save my-vm /path/to/save/file
+    xl restore /path/to/save/file
+
+Xen 4.6 sees the instruction of Migration v2.  There is no change for
+people using `xl`, although the `libxl` API has had an extension.
+
+# Technical details
+
+Migration is formed of several layers.  `libxc` is responsible for the
+contents of the VM (ram, vcpus, etc) and the live migration loop, while
+`libxl` is responsible for items such as emulator state.
+
+The format of the migration v2 stream is specified in two documents, and
+is architecture neutral.  Compatibility with legacy streams is
+maintained via the `convert-legacy-stream` script which transforms a
+legacy stream into a migration v2 stream.
+
+* Documents
+    * `docs/specs/libxc-migration-stream.pandoc`
+    * `docs/specs/libxl-migration-stream.pandoc`
+* `libxc`
+    * `tools/libxc/xc_sr_*.[hc]`
+* `libxl`
+    * `tools/libxl/libxl_stream_{read,write}.c`
+* Scripts
+    * `tools/python/xen/migration/*.py`
+    * `tools/python/scripts/convert-legacy-stream`
+    * `tools/python/scripts/verify-stream-v2`
+
+Users of the `libxl` API have a new parameter `stream_version` in
+`domain_restore_params` which is used to distinguish between legacy and
+v2 migration streams, and hence whether legacy conversion is required.
+
+# Limitations
+
+Hypervisor logdirty support is incompatible with hardware passthrough,
+as IOMMU faults cannot be used to track writes.
+
+While not a bug in migration specifically, VMs are very sensitive to
+changes in cpuid information, and cpuid levelling support currently has
+its issues.  Extreme care should be taken when migrating VMs between
+non-identical CPUs until the cpuid levelling improvements are complete.
+
+# Areas for improvement
+
+* Arm support
+* Linear P2M support for x86 PV
+* Live looping parameters
+
+# Known issues
+
+* x86 HVM guest physmap operations (not reflected in logdirty bitmap)
+* x86 HVM with PoD pages (attempts to map cause PoD allocations)
+* x86 HVM with nested-virt (no relevant information included in the
+  stream)
+* x86 PV ballooning (P2M marked dirty, target frame not marked)
+* x86 PV P2M structure changes (not noticed, stale mappings used)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+2015-10-24 1        Xen 4.6  Document written
+---------- -------- -------- -------------------------------------------
-- 
1.7.10.4

Re: [PATCH v2 for-4.6 2/2] docs: Migration feature document

[Jim Fehlig]

On 08/25/2015 04:40 AM, Andrew Cooper wrote:
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> v2:
>   * %Revision and #History, following template review
>   * Grammar fixes
> ---
>   docs/features/migration.pandoc |  100 ++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 100 insertions(+)
>   create mode 100644 docs/features/migration.pandoc
>
> diff --git a/docs/features/migration.pandoc b/docs/features/migration.pandoc
> new file mode 100644
> index 0000000..0fd227f
> --- /dev/null
> +++ b/docs/features/migration.pandoc
> @@ -0,0 +1,100 @@
> +% Migration
> +% Revision 1
> +
> +\clearpage
> +
> +# Basics
> +--------------- -------------
> +        Status: **Supported**
> +
> +  Architecture: x86
> +
> +     Component: Toolstack
> +--------------- -------------
> +
> +# Overview
> +
> +Migration is a mechanism to move a virtual machine while the VM is
> +running.  Live migration moves a running virtual machine between two
> +physical servers, but the same mechanism can be used for non-live
> +migrate (pause and copy) and suspend/resume from disk.

s/migrate/migration/ ?

> +
> +# User details
> +
> +No hardware requirements, although hypervisor logdirty support is
> +required for live migration.
> +
> +From the command line, `xl migrate/save/restore` are the top level
> +interactions.  e.g.
> +
> +    xl create my-vm.cfg
> +    xl migrate my-vm localhost
> +
> +or
> +
> +    xl create my-vm.cfg
> +    xl save my-vm /path/to/save/file
> +    xl restore /path/to/save/file
> +
> +Xen 4.6 sees the instruction of Migration v2.  There is no change for

Xen 4.6 sees the introduction of Migration v2.
Or, Xen 4.6 introduces Migration v2.

> +people using `xl`, although the `libxl` API has had an extension.
> +
> +# Technical details
> +
> +Migration is formed of several layers.  `libxc` is responsible for the
> +contents of the VM (ram, vcpus, etc) and the live migration loop, while
> +`libxl` is responsible for items such as emulator state.
> +
> +The format of the migration v2 stream is specified in two documents, and
> +is architecture neutral.  Compatibility with legacy streams is
> +maintained via the `convert-legacy-stream` script which transforms a
> +legacy stream into a migration v2 stream.
> +
> +* Documents
> +    * `docs/specs/libxc-migration-stream.pandoc`
> +    * `docs/specs/libxl-migration-stream.pandoc`
> +* `libxc`
> +    * `tools/libxc/xc_sr_*.[hc]`
> +* `libxl`
> +    * `tools/libxl/libxl_stream_{read,write}.c`
> +* Scripts
> +    * `tools/python/xen/migration/*.py`
> +    * `tools/python/scripts/convert-legacy-stream`
> +    * `tools/python/scripts/verify-stream-v2`
> +
> +Users of the `libxl` API have a new parameter `stream_version` in
> +`domain_restore_params` which is used to distinguish between legacy and
> +v2 migration streams, and hence whether legacy conversion is required.

A question better asked when you posted the patches, but is there a way to 
detect if the receiver is V2 capable? I suspect sending a V2 stream to a 
receiver capable only of V1 is not advised :-).

Also, commit id introducing the change would be helpful info. Looks like 
3a9ace01 in this case.

> +
> +# Limitations
> +
> +Hypervisor logdirty support is incompatible with hardware passthrough,
> +as IOMMU faults cannot be used to track writes.
> +
> +While not a bug in migration specifically, VMs are very sensitive to
> +changes in cpuid information, and cpuid levelling support currently has
> +its issues.  Extreme care should be taken when migrating VMs between
> +non-identical CPUs until the cpuid levelling improvements are complete.
> +
> +# Areas for improvement
> +
> +* Arm support
> +* Linear P2M support for x86 PV
> +* Live looping parameters

* cpuid levelling

Regards,
Jim

> +
> +# Known issues
> +
> +* x86 HVM guest physmap operations (not reflected in logdirty bitmap)
> +* x86 HVM with PoD pages (attempts to map cause PoD allocations)
> +* x86 HVM with nested-virt (no relevant information included in the
> +  stream)
> +* x86 PV ballooning (P2M marked dirty, target frame not marked)
> +* x86 PV P2M structure changes (not noticed, stale mappings used)
> +
> +# History
> +
> +------------------------------------------------------------------------
> +Date       Revision Version  Notes
> +---------- -------- -------- -------------------------------------------
> +2015-10-24 1        Xen 4.6  Document written
> +---------- -------- -------- -------------------------------------------

Re: [PATCH v2 for-4.6 2/2] docs: Migration feature document

[Andrew Cooper]

On 27/08/15 03:15, Jim Fehlig wrote:
> On 08/25/2015 04:40 AM, Andrew Cooper wrote:
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> ---
>> v2:
>>   * %Revision and #History, following template review
>>   * Grammar fixes
>> ---
>>   docs/features/migration.pandoc |  100
>> ++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 100 insertions(+)
>>   create mode 100644 docs/features/migration.pandoc
>>
>> diff --git a/docs/features/migration.pandoc
>> b/docs/features/migration.pandoc
>> new file mode 100644
>> index 0000000..0fd227f
>> --- /dev/null
>> +++ b/docs/features/migration.pandoc
>> @@ -0,0 +1,100 @@
>> +% Migration
>> +% Revision 1
>> +
>> +\clearpage
>> +
>> +# Basics
>> +--------------- -------------
>> +        Status: **Supported**
>> +
>> +  Architecture: x86
>> +
>> +     Component: Toolstack
>> +--------------- -------------
>> +
>> +# Overview
>> +
>> +Migration is a mechanism to move a virtual machine while the VM is
>> +running.  Live migration moves a running virtual machine between two
>> +physical servers, but the same mechanism can be used for non-live
>> +migrate (pause and copy) and suspend/resume from disk.
>
> s/migrate/migration/ ?

Either is valid here, although it would be better to leave the entire
sentence in the same tense.

>
>> +
>> +# User details
>> +
>> +No hardware requirements, although hypervisor logdirty support is
>> +required for live migration.
>> +
>> +From the command line, `xl migrate/save/restore` are the top level
>> +interactions.  e.g.
>> +
>> +    xl create my-vm.cfg
>> +    xl migrate my-vm localhost
>> +
>> +or
>> +
>> +    xl create my-vm.cfg
>> +    xl save my-vm /path/to/save/file
>> +    xl restore /path/to/save/file
>> +
>> +Xen 4.6 sees the instruction of Migration v2.  There is no change for
>
> Xen 4.6 sees the introduction of Migration v2.
> Or, Xen 4.6 introduces Migration v2.

Oops.  I did mean s/instruction/introduction/.

>
>> +people using `xl`, although the `libxl` API has had an extension.
>> +
>> +# Technical details
>> +
>> +Migration is formed of several layers.  `libxc` is responsible for the
>> +contents of the VM (ram, vcpus, etc) and the live migration loop, while
>> +`libxl` is responsible for items such as emulator state.
>> +
>> +The format of the migration v2 stream is specified in two documents,
>> and
>> +is architecture neutral.  Compatibility with legacy streams is
>> +maintained via the `convert-legacy-stream` script which transforms a
>> +legacy stream into a migration v2 stream.
>> +
>> +* Documents
>> +    * `docs/specs/libxc-migration-stream.pandoc`
>> +    * `docs/specs/libxl-migration-stream.pandoc`
>> +* `libxc`
>> +    * `tools/libxc/xc_sr_*.[hc]`
>> +* `libxl`
>> +    * `tools/libxl/libxl_stream_{read,write}.c`
>> +* Scripts
>> +    * `tools/python/xen/migration/*.py`
>> +    * `tools/python/scripts/convert-legacy-stream`
>> +    * `tools/python/scripts/verify-stream-v2`
>> +
>> +Users of the `libxl` API have a new parameter `stream_version` in
>> +`domain_restore_params` which is used to distinguish between legacy and
>> +v2 migration streams, and hence whether legacy conversion is required.
>
> A question better asked when you posted the patches, but is there a
> way to detect if the receiver is V2 capable? I suspect sending a V2
> stream to a receiver capable only of V1 is not advised :-).

libxl has no concept of asking the far side for information, and
therefore no concept of advertising its features.

For querying the local libxl at build time, LIBXL_HAVE_SRM_V2 and
LIBXL_HAVE_SRM_V1 from 3a9ace01

>
> Also, commit id introducing the change would be helpful info. Looks
> like 3a9ace01 in this case.

See the reply (I am about to write) on the 0/0 thread, but I hope that
putting git sha's in this document is going to be impossible.

>
>> +
>> +# Limitations
>> +
>> +Hypervisor logdirty support is incompatible with hardware passthrough,
>> +as IOMMU faults cannot be used to track writes.
>> +
>> +While not a bug in migration specifically, VMs are very sensitive to
>> +changes in cpuid information, and cpuid levelling support currently has
>> +its issues.  Extreme care should be taken when migrating VMs between
>> +non-identical CPUs until the cpuid levelling improvements are complete.
>> +
>> +# Areas for improvement
>> +
>> +* Arm support
>> +* Linear P2M support for x86 PV
>> +* Live looping parameters
>
> * cpuid levelling

The topic of cpuid levelling is covered in the paragraph above, and is
logically separate from migration.  XenServer for examples uses it work
around various guest kernel bugs.

It is just as broken in the plain boot case as the migrate case, and I
don't wish to confuse the issues.

~Andrew

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Jim Fehlig]

On 08/25/2015 04:40 AM, Andrew Cooper wrote:
> An issue which Xen has is an uncertain support statement for features.
> Given the success seen with docs/misc/xen-command-line.markdown, and in
> particular keeping it up to date, introduce a similar system for
> features.
>
> Patch 1 introduces a proposed template (and a makefile tweak to include
> the new docs/features subdirectory), while patch 2 is a feature document
> covering the topic of migration.
>
> v2 Adds %Revision and #History table, following feedback from v1.
>
> This is tagged RFC as I expect people to have different views as to what
> is useful to include.  I would particilarly appreciate feedback on the
> template before it starts getting used widely.
>
> Lars: Does this look like a reasonable counterpart to your formal
> support statement document?
>
> Jim: Per your request at the summit for new information, is patch 2
> suitable?

Yes. It provides excellent info, with pointers to dig deeper for those interested.

Your proposal does raise the bar for feature contribution, but I'm not active 
enough in the Xen community to know if folks are supportive of the additional 
overhead. Would new features require a feature doc before committing?

My comment at the summit was more along the lines of a summary of libxl changes 
for external consumers. A libxl interface changelog of sorts, for consumers 
unable to keep up with xen-devel. But I also noted this may be a personal 
problem. You've already mentioned 'git log' :-).

Regards,
Jim

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Andrew Cooper]

On 27/08/15 03:44, Jim Fehlig wrote:
> On 08/25/2015 04:40 AM, Andrew Cooper wrote:
>> An issue which Xen has is an uncertain support statement for features.
>> Given the success seen with docs/misc/xen-command-line.markdown, and in
>> particular keeping it up to date, introduce a similar system for
>> features.
>>
>> Patch 1 introduces a proposed template (and a makefile tweak to include
>> the new docs/features subdirectory), while patch 2 is a feature document
>> covering the topic of migration.
>>
>> v2 Adds %Revision and #History table, following feedback from v1.
>>
>> This is tagged RFC as I expect people to have different views as to what
>> is useful to include.  I would particilarly appreciate feedback on the
>> template before it starts getting used widely.
>>
>> Lars: Does this look like a reasonable counterpart to your formal
>> support statement document?
>>
>> Jim: Per your request at the summit for new information, is patch 2
>> suitable?
>
> Yes. It provides excellent info, with pointers to dig deeper for those
> interested.
>
> Your proposal does raise the bar for feature contribution, but I'm not
> active enough in the Xen community to know if folks are supportive of
> the additional overhead. Would new features require a feature doc
> before committing?

Unless there are some screaming objections, I am hoping yes, and that
the document becomes the authoritative support statement for the
feature.  (This is currently a real problem working out the security
status of bugs in features lacking a concrete statement of support.)

Writing one of these documents is not hard (I hope for there to soon be
many examples to refer to), and I would like to see it become
common-practice to have as $N/$N on a series.

~Andrew

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Ian Jackson]

Andrew Cooper writes ("[RFC v2 for-4.6 0/2] In-tree feature documentation"):
> An issue which Xen has is an uncertain support statement for features.
> Given the success seen with docs/misc/xen-command-line.markdown, and in
> particular keeping it up to date, introduce a similar system for
> features.
> 
> Patch 1 introduces a proposed template (and a makefile tweak to include
> the new docs/features subdirectory), while patch 2 is a feature document
> covering the topic of migration.
> 
> v2 Adds %Revision and #History table, following feedback from v1.
> 
> This is tagged RFC as I expect people to have different views as to what
> is useful to include.  I would particilarly appreciate feedback on the
> template before it starts getting used widely.
> 
> Lars: Does this look like a reasonable counterpart to your formal
> support statement document?
> 
> Jim: Per your request at the summit for new information, is patch 2
> suitable?

I have read both patches.

I do wonder whether cross-referencing all the "issues" is a good idea.
It seems like it might be a lot of work to keep them in step.

I don't have anything else to add to the comments that others have
made.

Overall I think this is a good template.  The extra overhead may even
be negative.  The work of writing up a feature in the style of this
document may obviate the need to put much of the same information in a
0/N or a design document, and the existing template is quite
lightweight.

Ian.

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Andrew Cooper]

On 27/08/15 15:52, Ian Jackson wrote:
> Andrew Cooper writes ("[RFC v2 for-4.6 0/2] In-tree feature documentation"):
>> An issue which Xen has is an uncertain support statement for features.
>> Given the success seen with docs/misc/xen-command-line.markdown, and in
>> particular keeping it up to date, introduce a similar system for
>> features.
>>
>> Patch 1 introduces a proposed template (and a makefile tweak to include
>> the new docs/features subdirectory), while patch 2 is a feature document
>> covering the topic of migration.
>>
>> v2 Adds %Revision and #History table, following feedback from v1.
>>
>> This is tagged RFC as I expect people to have different views as to what
>> is useful to include.  I would particilarly appreciate feedback on the
>> template before it starts getting used widely.
>>
>> Lars: Does this look like a reasonable counterpart to your formal
>> support statement document?
>>
>> Jim: Per your request at the summit for new information, is patch 2
>> suitable?
> I have read both patches.
>
> I do wonder whether cross-referencing all the "issues" is a good idea.
> It seems like it might be a lot of work to keep them in step.

I don't expect all the issues to be enumerated (generally, they would be
found the first time someone falls over the issue), but where known
interaction issues exist, we need to have some place to leave them.

There are plenty of examples where known issues are documented somewhere
in the xen-devel archives, or in an individuals head, and neither of
these are useful places for the information to exist.

I would hope that over time the number of issues decreases to 0 (that
would be the day!), but a good second is at least to have a formal
acknowledge that there are issues, which can be located by the next
unfortunate sole to hit the issue.

>
> I don't have anything else to add to the comments that others have
> made.
>
> Overall I think this is a good template.  The extra overhead may even
> be negative.  The work of writing up a feature in the style of this
> document may obviate the need to put much of the same information in a
> 0/N or a design document, and the existing template is quite
> lightweight.

I hadn't considered this side benefit when designing the template.

~Andrew

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Ian Jackson]

Andrew Cooper writes ("Re: [RFC v2 for-4.6 0/2] In-tree feature documentation"):
> On 27/08/15 15:52, Ian Jackson wrote:
> > I do wonder whether cross-referencing all the "issues" is a good idea.
> > It seems like it might be a lot of work to keep them in step.
> 
> I don't expect all the issues to be enumerated (generally, they would be
> found the first time someone falls over the issue), but where known
> interaction issues exist, we need to have some place to leave them.

I was prompted to ask this because it seemed to me that some of the
issues were discussed in other parts of the text or in other patches
as well as in `issues'.

> There are plenty of examples where known issues are documented somewhere
> in the xen-devel archives, or in an individuals head, and neither of
> these are useful places for the information to exist.

I agree with this.  I think things should be in the tree, but once.

Ian.

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Andrew Cooper]

On 27/08/15 18:58, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [RFC v2 for-4.6 0/2] In-tree feature documentation"):
>> On 27/08/15 15:52, Ian Jackson wrote:
>>> I do wonder whether cross-referencing all the "issues" is a good idea.
>>> It seems like it might be a lot of work to keep them in step.
>> I don't expect all the issues to be enumerated (generally, they would be
>> found the first time someone falls over the issue), but where known
>> interaction issues exist, we need to have some place to leave them.
> I was prompted to ask this because it seemed to me that some of the
> issues were discussed in other parts of the text or in other patches
> as well as in `issues'.

Which issues are you concerned about?

>
>> There are plenty of examples where known issues are documented somewhere
>> in the xen-devel archives, or in an individuals head, and neither of
>> these are useful places for the information to exist.
> I agree with this.  I think things should be in the tree, but once.

I am certainly not advocating needless repetition, but I don't see
anything in the submitted text which would quality.  I will happily
correct the text if I am mistaken.

~Andrew

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Lars Kurth]

> On 27 Aug 2015, at 15:52, Ian Jackson <ian.jackson@eu.citrix.com> wrote:
> 
> Andrew Cooper writes ("[RFC v2 for-4.6 0/2] In-tree feature documentation"):
>> An issue which Xen has is an uncertain support statement for features.
>> Given the success seen with docs/misc/xen-command-line.markdown, and in
>> particular keeping it up to date, introduce a similar system for
>> features.
>> 
>> Patch 1 introduces a proposed template (and a makefile tweak to include
>> the new docs/features subdirectory), while patch 2 is a feature document
>> covering the topic of migration.
>> 
>> v2 Adds %Revision and #History table, following feedback from v1.
>> 
>> This is tagged RFC as I expect people to have different views as to what
>> is useful to include.  I would particilarly appreciate feedback on the
>> template before it starts getting used widely.
>> 
>> Lars: Does this look like a reasonable counterpart to your formal
>> support statement document?
>> 
>> Jim: Per your request at the summit for new information, is patch 2
>> suitable?
> 
> I have read both patches.

Me too

> I do wonder whether cross-referencing all the "issues" is a good idea.
> It seems like it might be a lot of work to keep them in step.

There is a risk that these may go stale. I am wondering, whether if we do have features, we can come up with some conventions that allow us to grep for the issues on the list. Just an idea.

We could have a unique feature ID in the #basics section. Migration (as in the first line of migration.pandoc) is probably too generic in this example (too many false negatives). But if there was a unique enough feature identifier that can be grep'ed in commit logs, on xen-devel@, ... that may help.

> Overall I think this is a good template.  The extra overhead may even
> be negative.  The work of writing up a feature in the style of this
> document may obviate the need to put much of the same information in a
> 0/N or a design document, and the existing template is quite
> lightweight.

I agree. I don't really have any additional comments Andrew. So feel free to 

We may need some extra tags/headings, if we were to include things such as 

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Andrew Cooper]

On 28/08/15 18:16, Lars Kurth wrote:
>> On 27 Aug 2015, at 15:52, Ian Jackson <ian.jackson@eu.citrix.com> wrote:
>>
>> Andrew Cooper writes ("[RFC v2 for-4.6 0/2] In-tree feature documentation"):
>>> An issue which Xen has is an uncertain support statement for features.
>>> Given the success seen with docs/misc/xen-command-line.markdown, and in
>>> particular keeping it up to date, introduce a similar system for
>>> features.
>>>
>>> Patch 1 introduces a proposed template (and a makefile tweak to include
>>> the new docs/features subdirectory), while patch 2 is a feature document
>>> covering the topic of migration.
>>>
>>> v2 Adds %Revision and #History table, following feedback from v1.
>>>
>>> This is tagged RFC as I expect people to have different views as to what
>>> is useful to include.  I would particilarly appreciate feedback on the
>>> template before it starts getting used widely.
>>>
>>> Lars: Does this look like a reasonable counterpart to your formal
>>> support statement document?
>>>
>>> Jim: Per your request at the summit for new information, is patch 2
>>> suitable?
>> I have read both patches.
> Me too
>
>> I do wonder whether cross-referencing all the "issues" is a good idea.
>> It seems like it might be a lot of work to keep them in step.
> There is a risk that these may go stale. I am wondering, whether if we do have features, we can come up with some conventions that allow us to grep for the issues on the list. Just an idea.
>
> We could have a unique feature ID in the #basics section. Migration (as in the first line of migration.pandoc) is probably too generic in this example (too many false negatives). But if there was a unique enough feature identifier that can be grep'ed in commit logs, on xen-devel@, ... that may help.

This feels like over-engineering a solution.  Maintaining a set of
unique features will be extra burden on the core maintainers, as well as
a extra burden on submitters to know how to work this brand new system.

I would hope that few supported features have "issues" as identified in
the migration document.

I expect this section to be far more useful for experimental and tech
preview features.  In such cases issues are perfectly fine (It is far
better to have some code people can play with, with a set of known
restrictions, than to have no code at all), and can serve as a todo list
before its status can be elevated to supported.

>
>> Overall I think this is a good template.  The extra overhead may even
>> be negative.  The work of writing up a feature in the style of this
>> document may obviate the need to put much of the same information in a
>> 0/N or a design document, and the existing template is quite
>> lightweight.
> I agree. I don't really have any additional comments Andrew. So feel free to 
>
> We may need some extra tags/headings, if we were to include things such as 

I would like to avoid making the system overly prescriptive (as this
makes it harder for contributers), but I would be perfectly happy
reviewing a new series where patch 0/$N simply says "refer to patch $Y
for the feature document".

~Andrew

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Andrew Cooper]

On 28/08/15 18:40, Andrew Cooper wrote:
> On 28/08/15 18:16, Lars Kurth wrote:
>>> On 27 Aug 2015, at 15:52, Ian Jackson <ian.jackson@eu.citrix.com> wrote:
>>>
>>> Andrew Cooper writes ("[RFC v2 for-4.6 0/2] In-tree feature documentation"):
>>>> An issue which Xen has is an uncertain support statement for features.
>>>> Given the success seen with docs/misc/xen-command-line.markdown, and in
>>>> particular keeping it up to date, introduce a similar system for
>>>> features.
>>>>
>>>> Patch 1 introduces a proposed template (and a makefile tweak to include
>>>> the new docs/features subdirectory), while patch 2 is a feature document
>>>> covering the topic of migration.
>>>>
>>>> v2 Adds %Revision and #History table, following feedback from v1.
>>>>
>>>> This is tagged RFC as I expect people to have different views as to what
>>>> is useful to include.  I would particilarly appreciate feedback on the
>>>> template before it starts getting used widely.
>>>>
>>>> Lars: Does this look like a reasonable counterpart to your formal
>>>> support statement document?
>>>>
>>>> Jim: Per your request at the summit for new information, is patch 2
>>>> suitable?
>>> I have read both patches.
>> Me too
>>
>>> I do wonder whether cross-referencing all the "issues" is a good idea.
>>> It seems like it might be a lot of work to keep them in step.
>> There is a risk that these may go stale. I am wondering, whether if we do have features, we can come up with some conventions that allow us to grep for the issues on the list. Just an idea.
>>
>> We could have a unique feature ID in the #basics section. Migration (as in the first line of migration.pandoc) is probably too generic in this example (too many false negatives). But if there was a unique enough feature identifier that can be grep'ed in commit logs, on xen-devel@, ... that may help.
> This feels like over-engineering a solution.  Maintaining a set of
> unique features will be extra burden on the core maintainers, as well as
> a extra burden on submitters to know how to work this brand new system.
>
> I would hope that few supported features have "issues" as identified in
> the migration document.
>
> I expect this section to be far more useful for experimental and tech
> preview features.  In such cases issues are perfectly fine (It is far
> better to have some code people can play with, with a set of known
> restrictions, than to have no code at all), and can serve as a todo list
> before its status can be elevated to supported.

In addition.

I expect that maintainers would have an idea of the documentation which
exists in tree, and refer to it when reviewing new code in the area.  I
certainly will be.

Nowadays, it feels that enough people reviewing new submissions have an
instinct of "changes to command line parameters should patch
docs/misc/xen-command-line.markdown".  This is a very good position as
far as keeping the command line documentation up to date goes, and was
the entire principle behind introducing that document in the first place.

I would hope that after a while, similar instincts would develop for
major modification or addition of new features.

~Andrew

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Lars Kurth]

> On 28 Aug 2015, at 18:40, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> 
> On 28/08/15 18:16, Lars Kurth wrote:
>>> On 27 Aug 2015, at 15:52, Ian Jackson <ian.jackson@eu.citrix.com> wrote:
>>> 
>>> Andrew Cooper writes ("[RFC v2 for-4.6 0/2] In-tree feature documentation"):
>>>> An issue which Xen has is an uncertain support statement for features.
>>>> Given the success seen with docs/misc/xen-command-line.markdown, and in
>>>> particular keeping it up to date, introduce a similar system for
>>>> features.
>>>> 
>>>> Patch 1 introduces a proposed template (and a makefile tweak to include
>>>> the new docs/features subdirectory), while patch 2 is a feature document
>>>> covering the topic of migration.
>>>> 
>>>> v2 Adds %Revision and #History table, following feedback from v1.
>>>> 
>>>> This is tagged RFC as I expect people to have different views as to what
>>>> is useful to include.  I would particilarly appreciate feedback on the
>>>> template before it starts getting used widely.
>>>> 
>>>> Lars: Does this look like a reasonable counterpart to your formal
>>>> support statement document?
>>>> 
>>>> Jim: Per your request at the summit for new information, is patch 2
>>>> suitable?
>>> I have read both patches.
>> Me too
>> 
>>> I do wonder whether cross-referencing all the "issues" is a good idea.
>>> It seems like it might be a lot of work to keep them in step.
>> There is a risk that these may go stale. I am wondering, whether if we do have features, we can come up with some conventions that allow us to grep for the issues on the list. Just an idea.
>> 
>> We could have a unique feature ID in the #basics section. Migration (as in the first line of migration.pandoc) is probably too generic in this example (too many false negatives). But if there was a unique enough feature identifier that can be grep'ed in commit logs, on xen-devel@, ... that may help.
> 
> This feels like over-engineering a solution.  Maintaining a set of
> unique features will be extra burden on the core maintainers, as well as
> a extra burden on submitters to know how to work this brand new system.

As I said, it was just an idea to discuss.
 
> I would hope that few supported features have "issues" as identified in
> the migration document.
> 
> I expect this section to be far more useful for experimental and tech
> preview features.  In such cases issues are perfectly fine (It is far
> better to have some code people can play with, with a set of known
> restrictions, than to have no code at all), and can serve as a todo list
> before its status can be elevated to supported.
> 
>> 
>>> Overall I think this is a good template.  The extra overhead may even
>>> be negative.  The work of writing up a feature in the style of this
>>> document may obviate the need to put much of the same information in a
>>> 0/N or a design document, and the existing template is quite
>>> lightweight.
>> I agree. I don't really have any additional comments Andrew. So feel free to 
>> 
>> We may need some extra tags/headings, if we were to include things such as 

This end of the mail was deleted by mistake. I meant to say was ...

We may need some extra tags/headings, if we were to include things such as supported limits for memory, vCPUs, ... I remember, you raised the point that some of the theoretical limits are not always tested. 

* Maybe a Testing section, which outlines what is tested automatically and what should be manually tested when updated. It could be optional.
* Maybe another optional references section: that would allow us to link to 3rd party specs, presentations, etc.

Lars

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Andrew Cooper]

On 28/08/15 18:51, Lars Kurth wrote:
>> On 28 Aug 2015, at 18:40, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>
>> On 28/08/15 18:16, Lars Kurth wrote:
>>>> On 27 Aug 2015, at 15:52, Ian Jackson <ian.jackson@eu.citrix.com> wrote:
>>>>
>>>> Andrew Cooper writes ("[RFC v2 for-4.6 0/2] In-tree feature documentation"):
>>>>> An issue which Xen has is an uncertain support statement for features.
>>>>> Given the success seen with docs/misc/xen-command-line.markdown, and in
>>>>> particular keeping it up to date, introduce a similar system for
>>>>> features.
>>>>>
>>>>> Patch 1 introduces a proposed template (and a makefile tweak to include
>>>>> the new docs/features subdirectory), while patch 2 is a feature document
>>>>> covering the topic of migration.
>>>>>
>>>>> v2 Adds %Revision and #History table, following feedback from v1.
>>>>>
>>>>> This is tagged RFC as I expect people to have different views as to what
>>>>> is useful to include.  I would particilarly appreciate feedback on the
>>>>> template before it starts getting used widely.
>>>>>
>>>>> Lars: Does this look like a reasonable counterpart to your formal
>>>>> support statement document?
>>>>>
>>>>> Jim: Per your request at the summit for new information, is patch 2
>>>>> suitable?
>>>> I have read both patches.
>>> Me too
>>>
>>>> I do wonder whether cross-referencing all the "issues" is a good idea.
>>>> It seems like it might be a lot of work to keep them in step.
>>> There is a risk that these may go stale. I am wondering, whether if we do have features, we can come up with some conventions that allow us to grep for the issues on the list. Just an idea.
>>>
>>> We could have a unique feature ID in the #basics section. Migration (as in the first line of migration.pandoc) is probably too generic in this example (too many false negatives). But if there was a unique enough feature identifier that can be grep'ed in commit logs, on xen-devel@, ... that may help.
>> This feels like over-engineering a solution.  Maintaining a set of
>> unique features will be extra burden on the core maintainers, as well as
>> a extra burden on submitters to know how to work this brand new system.
> As I said, it was just an idea to discuss.
>  
>> I would hope that few supported features have "issues" as identified in
>> the migration document.
>>
>> I expect this section to be far more useful for experimental and tech
>> preview features.  In such cases issues are perfectly fine (It is far
>> better to have some code people can play with, with a set of known
>> restrictions, than to have no code at all), and can serve as a todo list
>> before its status can be elevated to supported.
>>
>>>> Overall I think this is a good template.  The extra overhead may even
>>>> be negative.  The work of writing up a feature in the style of this
>>>> document may obviate the need to put much of the same information in a
>>>> 0/N or a design document, and the existing template is quite
>>>> lightweight.
>>> I agree. I don't really have any additional comments Andrew. So feel free to 
>>>
>>> We may need some extra tags/headings, if we were to include things such as 
> This end of the mail was deleted by mistake. I meant to say was ...
>
> We may need some extra tags/headings, if we were to include things such as supported limits for memory, vCPUs, ... I remember, you raised the point that some of the theoretical limits are not always tested.

Absolutely.  Not everyone has a server with 123TB of RAM to hand, or
even 16TB which is default current limit.  (For this issue, testing from
both Citrix and Oracle indicates a bug when more than 5TB of RAM is used.)

Therefore, a distinction between the theoretical limit and
currently-tested limit is very useful.  I expect the the commercial
stakeholders will be in a position to routinely test at far above the
limit available to direct consumers of the Xen project.


For the in-tree statement of limits, I have not put much though to how
to represent them yet, but I am not sure that the feature template
proposed in #1 will be a great fit.  I suspect we will want something a
little different.

>
> * Maybe a Testing section, which outlines what is tested automatically and what should be manually tested when updated. It could be optional.

Ooh - that is a good idea.

Even if it is just a paragraph about suggested things to test when
modifying, it will be useful to contributors.

> * Maybe another optional references section: that would allow us to link to 3rd party specs, presentations, etc.

I should really put some references in the migration document, shouldn't I.

~Andrew

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Lars Kurth]

On 28/08/2015 19:18, "Andrew Cooper" <andrew.cooper3@citrix.com> wrote:

>On 28/08/15 18:51, Lars Kurth wrote:
>>>
>>
>> We may need some extra tags/headings, if we were to include things such
>>as supported limits for memory, vCPUs, ... I remember, you raised the
>>point that some of the theoretical limits are not always tested.
>
>Absolutely.  Not everyone has a server with 123TB of RAM to hand, or
>even 16TB which is default current limit.  (For this issue, testing from
>both Citrix and Oracle indicates a bug when more than 5TB of RAM is used.)
>
>Therefore, a distinction between the theoretical limit and
>currently-tested limit is very useful.  I expect the the commercial
>stakeholders will be in a position to routinely test at far above the
>limit available to direct consumers of the Xen project.
>
>For the in-tree statement of limits, I have not put much though to how
>to represent them yet, but I am not sure that the feature template
>proposed in #1 will be a great fit.  I suspect we will want something a
>little different.

Maybe a master document for system stuff, with a slightly different format.

I may have missed this: what kind of mark-up is being used?
http://pandoc.org/ seems to support a few: may need to add a README with a
couple of pointers.

And I am assuming some sort of index is produced when these docs are
built: correct?

Lars

Re: [RFC v2 for-4.6 0/2] In-tree feature documentation

[Andrew Cooper]

On 28/08/15 19:52, Lars Kurth wrote:
>
> On 28/08/2015 19:18, "Andrew Cooper" <andrew.cooper3@citrix.com> wrote:
>
>> On 28/08/15 18:51, Lars Kurth wrote:
>>> We may need some extra tags/headings, if we were to include things such
>>> as supported limits for memory, vCPUs, ... I remember, you raised the
>>> point that some of the theoretical limits are not always tested.
>> Absolutely.  Not everyone has a server with 123TB of RAM to hand, or
>> even 16TB which is default current limit.  (For this issue, testing from
>> both Citrix and Oracle indicates a bug when more than 5TB of RAM is used.)
>>
>> Therefore, a distinction between the theoretical limit and
>> currently-tested limit is very useful.  I expect the the commercial
>> stakeholders will be in a position to routinely test at far above the
>> limit available to direct consumers of the Xen project.
>>
>> For the in-tree statement of limits, I have not put much though to how
>> to represent them yet, but I am not sure that the feature template
>> proposed in #1 will be a great fit.  I suspect we will want something a
>> little different.
> Maybe a master document for system stuff, with a slightly different format.

Possibly.  Nothing prevents us from having different types of
documentation with different expected layouts.

We can see what feels best when we get to it.

>
> I may have missed this: what kind of mark-up is being used?
> http://pandoc.org/ seems to support a few: may need to add a README with a
> couple of pointers.

http://pandoc.org/demo/example9/pandocs-markdown.html

Pandoc extends plain markdown in a number of ways commonly found
elsewhere, including the ability to insert raw LaTeX if needs be.

The eagle-eyed might have spotted a cunningly positioned \clearpage
which aids the clarity of the generated pdf file.

>
> And I am assuming some sort of index is produced when these docs are
> built: correct?

For the HTML ones, yes.  One example is:

http://xenbits.xen.org/docs/unstable/specs/libxc-migration-stream.html

Which is currently generated from a pandoc document in tree.


By default in-tree, all pandoc stuff will be rendered to pdf and html. 
I am not aware whether the pdf versions are served from xenbits, but the
html ones certainly are.

~Andrew