[PATCH 0 of 6 DOCDAY] docs updates

[PATCH 0 of 6 DOCDAY] docs updates

[Ian Campbell]

Here is some documentation I wrote as part of the doc day yesterday
(and this morning if I'm honest).

I posted a bunch of it yesterday but this incorporates some additional
stuff.

* Import PVHVM unplug protocol specification from
  qemu-xen-unstable.git.
* Install markdown documentation as both HTML and TXT.
* Updated unmodified devices README based on information on wiki.
* Document the xl vif configuration syntax in a manner similar to the
  existing disk configuration.
* Add some basic xl configuration file examples

[PATCH 1 of 6 DOCDAY] docs: import HVM emulated device unplug protocol spec

[Ian Campbell]

# HG changeset patch
# User Ian Campbell <ian.campbell@citrix.com>
# Date 1319646021 -3600
# Node ID 03b570c3792bb6abce46de5c9ab560ac169117c3
# Parent  c681dd5aecf3da3c6fd0e4d8a760a9cd18617033
docs: import HVM emulated device unplug protocol spec

Convert to markdown as I go.

Currently this lives in qemu-xen.git i386-dm/README.hvm-pv-magic-ioport-disable
and I can never find it when I want it. As we transition to upstream qemu this
location becomes less useful.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

diff -r c681dd5aecf3 -r 03b570c3792b docs/misc/hvm-emulated-unplug.markdown
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/docs/misc/hvm-emulated-unplug.markdown	Wed Oct 26 17:20:21 2011 +0100
@@ -0,0 +1,68 @@
+#Xen HVM emulated device unplug protocol
+
+The protocol covers three basic things:
+
+ * Disconnecting emulated devices.
+ * Getting log messages out of the drivers and into dom0.
+ * Allowing dom0 to block the loading of specific drivers.  This is
+   intended as a backwards-compatibility thing: if we discover a bug
+   in some old version of the drivers, then rather than working around
+   it in Xen, we have the option of just making those drivers fall
+   back to emulated mode.
+
+The current protocol works like this (from the point of view of
+drivers):
+
+1. When the drivers first come up, they check whether the unplug logic
+   is available by reading a two-byte magic number from IO port `0x10`.
+   These should be `0x49d2`.  If the magic number doesn't match, the
+   drivers don't do anything.
+
+2. The drivers read a one-byte protocol version from IO port `0x12`.  If
+   this is 0, skip to 6.
+
+3. The drivers write a two-byte product number to IO port `0x12`.  At
+   the moment, the only drivers using this protocol are our
+   closed-source ones, which use product number 1.
+
+4. The drivers write a four-byte build number to IO port `0x10`.
+
+5. The drivers check the magic number by reading two bytes from `0x10`
+   again.  If it's changed from `0x49d2` to `0xd249`, the drivers are
+   blacklisted and should not load.
+
+6. The drivers write a two-byte bitmask of devices to unplug to IO
+   port `0x10`.  The defined fields are:
+
+  * `1` -- All IDE disks (not including CD drives)
+  * `2` -- All emulated NICs
+  * `4` -- All IDE disks except for the primary master (not including CD
+	   drives)
+
+   The relevant emulated devices then disappear from the relevant
+   buses.  For most guest operating systems, you want to do this
+   before device enumeration happens.
+
+Once the drivers have checked the magic number, they can send log
+messages to qemu which will be logged to wherever qemu's logs go
+(`/var/log/xen/qemu-dm.log` on normal Xen, dom0 syslog on XenServer).
+These messages are written to IO port `0x12` a byte at a time, and are
+terminated by newlines.  There's a fairly aggressive rate limiter on
+these messages, so they shouldn't be used for anything even vaguely
+high-volume, but they're rather useful for debugging and support.
+
+It is still permitted for a driver to use this logging feature if it
+is blacklisted, but *ONLY* if it has checked the magic number and found
+it to be `0x49d2` or `0xd249`.
+
+This isn't exactly a pretty protocol, but it does solve the problem.
+
+The blacklist is, from qemu's point of view, handled mostly through
+xenstore.  A driver version is considered to be blacklisted if
+`/mh/driver-blacklist/{product_name}/{build_number}` exists and is
+readable, where `{build_number}` is the build number from step 4 as a
+decimal number.  `{product_name}` is a string corresponding to the
+product number in step 3.
+
+The master registry of product names and numbers is in
+qemu-xen-unstable's xenstore.c.

[PATCH 2 of 6 DOCDAY] docs: add some scaffolding for building HTML docs from markdown

[Ian Campbell]

# HG changeset patch
# User Ian Campbell <ian.campbell@citrix.com>
# Date 1319646217 -3600
# Node ID 52058c9448935818e2a810a438244e06f40b4e0e
# Parent  03b570c3792bb6abce46de5c9ab560ac169117c3
docs: add some scaffolding for building HTML docs from markdown

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

diff -r 03b570c3792b -r 52058c944893 docs/Docs.mk
--- a/docs/Docs.mk	Wed Oct 26 17:20:21 2011 +0100
+++ b/docs/Docs.mk	Wed Oct 26 17:23:37 2011 +0100
@@ -7,3 +7,4 @@ DOXYGEN		:= doxygen
 POD2MAN		:= pod2man
 DOT		:= dot
 NEATO		:= neato
+MARKDOWN	:= markdown
diff -r 03b570c3792b -r 52058c944893 docs/Makefile
--- a/docs/Makefile	Wed Oct 26 17:20:21 2011 +0100
+++ b/docs/Makefile	Wed Oct 26 17:23:37 2011 +0100
@@ -11,10 +11,11 @@ DOC_MAN1SRC	:= $(wildcard man/*.pod.1)
 DOC_MAN1	:= $(patsubst man/%.pod.1,man1/%.1,$(DOC_MAN1SRC))
 DOC_MAN5	:= $(patsubst man/%.pod.5,man5/%.5,$(DOC_MAN5SRC))
 DOC_TEX		:= src/user.tex src/interface.tex
+DOC_MARKDOWN	:= $(wildcard misc/*.markdown)
 DOC_PS		:= $(patsubst src/%.tex,ps/%.ps,$(DOC_TEX))
 DOC_PDF		:= $(patsubst src/%.tex,pdf/%.pdf,$(DOC_TEX))
-DOC_HTML	:= $(patsubst src/%.tex,html/%/index.html,$(DOC_TEX))
-
+DOC_HTML	:= $(patsubst src/%.tex,html/%/index.html,$(DOC_TEX)) \
+		   $(patsubst %.markdown,html/%.html,$(DOC_MARKDOWN))
 GFX = $(patsubst %.fig, %.eps, $(wildcard figs/*.fig))
 
 .PHONY: all
@@ -37,9 +38,7 @@ ps: $(DOC_PS)
 pdf: $(DOC_PDF)
 
 .PHONY: html
-html:
-	@if which $(LATEX2HTML) 1>/dev/null 2>/dev/null; then \
-	$(MAKE) $(DOC_HTML); fi
+html: $(DOC_HTML)
 
 .PHONY: python-dev-docs
 python-dev-docs:
@@ -110,7 +109,16 @@ ps/%.ps: %.dvi
 	$(FIG2DEV) -L eps $< $@
 
 html/%/index.html: src/%.tex
-	$(INSTALL_DIR) $(@D)
+	@$(INSTALL_DIR) $(@D)
+	@set -e ; if which $(LATEX2HTML) 1>/dev/null 2>/dev/null; then \
+        echo "Running latex2html to generate $*/index.html ... "; \
 	$(LATEX2HTML) -split 0 -show_section_numbers -toc_depth 3 -nonavigation \
 	-numbered_footnotes -local_icons -noinfo -math -dir $(@D) \
-	$< 1>/dev/null 2>/dev/null
+	$< 1>/dev/null 2>/dev/null ;fi
+
+html/%.html: %.markdown
+	@$(INSTALL_DIR) $(@D)
+	@set -e ; if which $(MARKDOWN) 1>/dev/null 2>/dev/null; then \
+	echo "Running markdown to generate $*.html ... "; \
+	$(MARKDOWN) $< > $@.tmp ; \
+	$(call move-if-changed,$@.tmp,$@) ; fi

[PATCH 3 of 6 DOCDAY] unmodified_drivers: update README from http://wiki.xen.org/xenwiki/UnmodifiedDrivers

[Ian Campbell]

# HG changeset patch
# User Ian Campbell <ian.campbell@citrix.com>
# Date 1319647443 -3600
# Node ID b49eb99dc11aff809566f4b5bd53053aa5daef7d
# Parent  52058c9448935818e2a810a438244e06f40b4e0e
unmodified_drivers: update README from http://wiki.xen.org/xenwiki/UnmodifiedDrivers

Add reference to the fact that these drivers are for "classic-Xen" kernels only
and do not work with PVops but point towards the PVHVM functionality in
mainstream.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

diff -r 52058c944893 -r b49eb99dc11a unmodified_drivers/linux-2.6/README
--- a/unmodified_drivers/linux-2.6/README	Wed Oct 26 17:23:37 2011 +0100
+++ b/unmodified_drivers/linux-2.6/README	Wed Oct 26 17:44:03 2011 +0100
@@ -1,10 +1,19 @@
-To build:
+These drivers provide paravirtualised drivers for pre-PVops Linux
+kernels which have the classic-Xen patches applied. They do not work
+with kernels which are not patched with the classic-Xen patches. In
+particular this includes all PVops kernels.
 
-1. ./mkbuildtree [<arch>]
+For 2.6.36 and newer you should use the "PVHVM" functionality which is
+available in the mainline kernel.
+
+To build do one of:
+
+   ./mkbuildtree [<arch>]
    NB. You can override paths to Xen sources and a (stub) XenLinux
        build tree via the XEN and XL environment variable.
 
-2. make -C /path/to/kernel/build M=$PWD modules
+- or -
+   make -C /path/to/kernel/build M=$PWD modules
    NB. This is your native kernel build tree (or a distro provided
        stub), not the XenLinux sources referred to in step 1.
    NB. If you are cross compiling, you need to set ARCH and

[PATCH 4 of 6 DOCDAY] docs: add a document describing the xl vif syntax

[Ian Campbell]

# HG changeset patch
# User Ian Campbell <ian.campbell@citrix.com>
# Date 1319705668 -3600
# Node ID 996143b104f0c027ad45b9e04bdccd342e7a1b94
# Parent  b49eb99dc11aff809566f4b5bd53053aa5daef7d
docs: add a document describing the xl vif syntax

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

diff -r b49eb99dc11a -r 996143b104f0 docs/misc/xl-network-configuration.markdown
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/docs/misc/xl-network-configuration.markdown	Thu Oct 27 09:54:28 2011 +0100
@@ -0,0 +1,126 @@
+# XL Network Configuration
+
+## Syntax Overview
+
+This document specifies the xl config file format vif configuration
+option.  It has the following form:
+
+        vif = [ '<vifspec>', '<vifspec>', ... ]
+
+where each vifspec is in this form:
+    
+        [<key>=<value>|<flag>,]
+
+For example:
+
+        'mac=00:16:3E:74:3d:76,model=rtl8139,bridge=xenbr0'
+        'mac=00:16:3E:74:34:32'
+        '' # The empty string
+
+These might be specified in the domain config file like this:
+
+        vif = [ 'mac=00:16:3E:74:34:32', 'mac=00:16:3e:5f:48:e4,bridge=xenbr1' ]
+
+More formally, the string is a series of comma-separated keyword/value
+pairs. All keywords are optional.
+
+Each device has a `DEVID` which is its index within the vif list, starting from 0.
+
+## Keywords
+
+### mac
+
+If specified then this option specifies the MAC address inside the
+guest of this VIF device. The value is a 48-bit number represented as
+six groups of two hexadecimal digits, separated by colons (:).
+
+The default if this keyword is not specified is to be automatically
+generate a MAC address inside the space assigned to Xen's
+[Organizationally Unique Identifier][oui] (00:16:3e).
+
+If you are choosing a MAC address then it is strongly recommend to
+follow one of the following strategies:
+
+  * Generate a random sequence of 6 byte, set the locally administered
+    bit (bit 2 of the first byte) and clear the multicast bit (bit 1
+    of the first byte). In other words the first byte should have the
+    bit pattern xxxxxx10 (where x is a randomly generated bit) and the
+    remaining 5 bytes are randomly generated See
+    [http://en.wikipedia.org/wiki/MAC_address] for more details the
+    structure of a MAC address.
+  * Allocate an address from within the space defined by your
+    organization's OUI (if you have one) following your organization's
+    procedures for doing so.
+  * Allocate an address from within the space defined by Xen's OUI
+    (00:16:3e). Taking care not to clash with other users of the
+    physical network segment where this VIF will reside.
+
+If you have an OUI for your own use then that is the preferred
+strategy. Otherwise in general you should prefer to generate a random
+MAC and set the locally administered bit since this allows for more
+bits of randomness than using the Xen OUI.
+
+### bridge
+
+Specifies the name of the network bridge which this VIF should be
+added to. The default is `xenbr0`. The bridge must be configured using
+your distribution's network configuration tools. See the [wiki][net]
+for guidance and examples.
+
+### type
+
+This keyword is valid for HVM guests only.
+
+Specifies the type of device to valid values are:
+
+  * `ioemu` (default) -- this device will be provided as an emulate
+    device to the guest and also as a paravirtualised device which the
+    guest may choose to use instead if it has suitable drivers
+    available.
+  * `vif` -- this device will be provided as a paravirtualised device
+    only.
+
+### model
+
+This keyword is valid for HVM guest devices with `type=ioemu` only.
+
+Specifies the type device to emulated for this guest. Valid values
+are:
+
+  * `rtl8139` (default) -- Realtek RTL8139
+  * `e1000` -- Intel E1000 
+  * in principal any device supported by your device model
+
+### vifname
+
+This keyword is valid for HVM guest devices with `type=ioemu` only.
+
+Specifies the backend device name for an emulated device. The default
+is `tapDOMID.DEVID` where `DOMID` is the guest domain ID and `DEVID`
+is the device number.
+
+### script
+
+Specifies the hotplug script to run to configure this device (e.g. to
+add it to the relevant bridge). Defaults to
+`XEN_SCRIPT_DIR/vif-bridge` but can be set to any script. Some example
+scripts are installed in `XEN_SCRIPT_DIR`.
+
+### ip
+
+Specifies the IP address for the device, the default is not to
+specify an IP address.
+
+What, if any, effect this has depends on the hotplug script which is
+configured. A typically behaviour (exhibited by the example hotplug
+scripts) if set might be to configure firewall rules to allow only the
+specified IP address to be used by the guest (blocking all others).
+
+### backend
+
+Specifies the backend domain which this device should attach to. This
+defaults to domain 0. Specifying another domain requires setting up a
+driver domain which is outside the scope of this document.
+
+[oui]: http://en.wikipedia.org/wiki/Organizationally_Unique_Identifier
+[net]: http://wiki.xen.org/xenwiki/HostConfiguration/Networking

[PATCH 5 of 6 DOCDAY] xl: add some example configuration files

[Ian Campbell]

# HG changeset patch
# User Ian Campbell <ian.campbell@citrix.com>
# Date 1319705714 -3600
# Node ID 74e5af602abf03022e8e118bb58876e13812612f
# Parent  996143b104f0c027ad45b9e04bdccd342e7a1b94
xl: add some example configuration files

These are much more minimal than the equivalent xmexamples but should
eventually reference a more complete document describing the available options.

These are based on the guest configuration files that I actually use day-to-day.

There is an unresolved reference to a document describing the complete syntax.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

diff -r 996143b104f0 -r 74e5af602abf tools/examples/xlexample.hvm
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tools/examples/xlexample.hvm	Thu Oct 27 09:55:14 2011 +0100
@@ -0,0 +1,39 @@
+# =====================================================================
+# Example HVM guest configuration
+# =====================================================================
+#
+# This is a fairly minimal example of what is required for an
+# HVM guest. For a more complete guide see <XXX Document TBD>
+
+# Guest name
+name = "example.hvm"
+
+# 128-bit UUID for the domain as a hexadecimal number.
+# Use "uuidgen" to generate one if required.
+# The default behavior is to generate a new UUID each time the guest is started.
+#uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
+
+# Initial memory allocation (MB)
+memory = 128
+
+# Maximum memory (MB)
+# If this is greater than `memory' then the slack will start ballooned
+# (this assumes guest kernel support for ballooning)
+#maxmem = 512
+
+# Number of VCPUS
+vcpus = 2
+
+# Network devices
+# A list of 'vifspec' entries as described in
+# docs/misc/xl-network-configuration.markdown
+vif = [ '' ]
+
+# Disk Devices
+# A list of `diskspec' entries as described in
+# docs/misc/xl-disk-configuration.txt
+disk = [ '/dev/vg/guest-volume,raw,xvda,rw' ]
+
+# Guest VGA console configuration, either SDL or VNC
+sdl = 1
+#vnc = 1
diff -r 996143b104f0 -r 74e5af602abf tools/examples/xlexample.pvlinux
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tools/examples/xlexample.pvlinux	Thu Oct 27 09:55:14 2011 +0100
@@ -0,0 +1,44 @@
+# =====================================================================
+# Example PV Linux guest configuration
+# =====================================================================
+#
+# This is a fairly minimal example of what is required for a
+# Paravirtualised Linux guest. For a more complete guide see <XXX Document TBD>
+
+# Guest name
+name = "example.pvlinux"
+
+# 128-bit UUID for the domain as a hexadecimal number.
+# Use "uuidgen" to generate one if required.
+# The default behavior is to generate a new UUID each time the guest is started.
+#uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
+
+# Kernel image to boot
+kernel = "/boot/vmlinuz"
+
+# Ramdisk (optional)
+#ramdisk = "/boot/initrd.gz"
+
+# Kernel command line options
+extra = "root=/dev/xvda1"
+
+# Initial memory allocation (MB)
+memory = 128
+
+# Maximum memory (MB)
+# If this is greater than `memory' then the slack will start ballooned
+# (this assumes guest kernel support for ballooning)
+#maxmem = 512
+
+# Number of VCPUS
+vcpus = 2
+
+# Network devices
+# A list of 'vifspec' entries as described in
+# docs/misc/xl-network-configuration.markdown
+vif = [ '' ]
+
+# Disk Devices
+# A list of `diskspec' entries as described in
+# docs/misc/xl-disk-configuration.txt
+disk = [ '/dev/vg/guest-volume,raw,xvda,rw' ]

[PATCH 6 of 6 DOCDAY] docs: install text documentation

[Ian Campbell]

# HG changeset patch
# User Ian Campbell <ian.campbell@citrix.com>
# Date 1319705722 -3600
# Node ID 8ddcfa3917a695cd6d533376e11e0e6e4da8d673
# Parent  74e5af602abf03022e8e118bb58876e13812612f
docs: install text documentation

Including markdown docs.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

diff -r 74e5af602abf -r 8ddcfa3917a6 .hgignore
--- a/.hgignore	Thu Oct 27 09:55:14 2011 +0100
+++ b/.hgignore	Thu Oct 27 09:55:22 2011 +0100
@@ -57,6 +57,7 @@
 ^docs/user/labels\.pl$
 ^docs/user/user\.css$
 ^docs/user/user\.html$
+^docs/txt/.*$
 ^docs/xen-api/vm_lifecycle.eps$
 ^docs/xen-api/xenapi-datamodel-graph.eps$
 ^docs/xen-api/xenapi.out$
diff -r 74e5af602abf -r 8ddcfa3917a6 docs/Makefile
--- a/docs/Makefile	Thu Oct 27 09:55:14 2011 +0100
+++ b/docs/Makefile	Thu Oct 27 09:55:22 2011 +0100
@@ -16,13 +16,16 @@ DOC_PS		:= $(patsubst src/%.tex,ps/%.ps,
 DOC_PDF		:= $(patsubst src/%.tex,pdf/%.pdf,$(DOC_TEX))
 DOC_HTML	:= $(patsubst src/%.tex,html/%/index.html,$(DOC_TEX)) \
 		   $(patsubst %.markdown,html/%.html,$(DOC_MARKDOWN))
+DOC_TXT         := $(patsubst %.txt,txt/%.txt,$(wildcard misc/*.txt)) \
+		   $(patsubst %.markdown,txt/%.txt,$(DOC_MARKDOWN))
+
 GFX = $(patsubst %.fig, %.eps, $(wildcard figs/*.fig))
 
 .PHONY: all
 all: build
 
 .PHONY: build
-build: ps pdf html man-pages
+build: ps pdf html txt man-pages
 	@if which $(DOT) 1>/dev/null 2>/dev/null ; then              \
 	$(MAKE) -C xen-api build ; else                              \
         echo "Graphviz (dot) not installed; skipping xen-api." ; fi
@@ -40,6 +43,9 @@ pdf: $(DOC_PDF)
 .PHONY: html
 html: $(DOC_HTML)
 
+.PHONY: txt
+txt: $(DOC_TXT)
+
 .PHONY: python-dev-docs
 python-dev-docs:
 	@mkdir -v -p api/tools/python
@@ -122,3 +128,13 @@ html/%.html: %.markdown
 	echo "Running markdown to generate $*.html ... "; \
 	$(MARKDOWN) $< > $@.tmp ; \
 	$(call move-if-changed,$@.tmp,$@) ; fi
+
+txt/%.txt: %.txt
+	$(INSTALL_DIR) $(@D)
+	cp $< $@.tmp
+	$(call move-if-changed,$@.tmp,$@)
+
+txt/%.txt: %.markdown
+	$(INSTALL_DIR) $(@D)
+	cp $< $@.tmp
+	$(call move-if-changed,$@.tmp,$@)

Re: [PATCH 1 of 6 DOCDAY] docs: import HVM emulated device unplug protocol spec

[Ian Jackson]

Ian Campbell writes ("[Xen-devel] [PATCH 1 of 6 DOCDAY] docs: import HVM emulated device unplug protocol spec"):
> docs: import HVM emulated device unplug protocol spec

Applied all six, thanks.

Ian.