[OSSTEST PATCH 1/4] examine: Do not try to find old version

[OSSTEST PATCH 1/4] examine: Do not try to find old version

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-daily-branch | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cr-daily-branch b/cr-daily-branch
index 39483cd..cf00db8 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -88,7 +88,7 @@ check_tested () {
 
 if [ "x$OLD_REVISION" = x ]; then
     case $branch in
-	distros-*)
+	distros-*|examine)
 	    OSSTEST_NO_BASELINE=y
 	    OLD_REVISION=flight-`check_tested`
 	    ;;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 2/4] README: Better documentation of recipes, db, etc.

[Ian Jackson]

CC: George Dunlap <george.dunlap@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 60 insertions(+), 15 deletions(-)

diff --git a/README b/README
index ffe0018..93129e3 100644
--- a/README
+++ b/README
@@ -8,32 +8,55 @@ Terminology
 
 "flight":
 
-    Each run of osstest is referred to as a "flight". Each flight is
-    given a unique ID (a number or name).
+    Each run of osstest is referred to as a "flight".  Each flight is
+    given a unique ID.
+
+    Standalone mode generally uses a flight named "standalone", which
+    is frequently erased and reused.  In "Executive" mode (used for
+    production osstest instances) flights are numbered and the
+    metadata is permanently recorded.
 
 "job":
 
-    Each flight consists of one or more "jobs". These are a sequence
+    Each flight consists of one or more "jobs".  These are a sequence
     of test steps run in order and correspond to a column in the test
-    report grid. They have names like "build-amd64" or
-    "test-amd64-amd64-pv". A job can depend on the output of another
+    report grid.  They have names like "build-amd64" or
+    "test-amd64-amd64-pv".  A job can depend on the output of another
     job in the flight -- e.g. most test-* jobs depend on one or more
     build-* jobs.
 
+    A job has a named "recipe", which indicates what things need to be
+    done in what order: generally, a sequence of ts-* scripts.  The
+    recipes are defined by their implementations in sg-run-job.
+
+    The set of jobs for any particular kind of flight is defined in
+    make-flight, make-*-flight, and mfi-*.
+
 "step":
 
-    Each job consists of multiple "steps" which is an individual test
-    operation, such as "build the hypervisor", "install a guest",
-    "start a guest", "migrate a guest", etc. A step corresponds to a
-    cell in the results grid. A given step can be reused in multiple
-    different jobs, e.g. the "xen build" step is used in several
-    different build-* jobs. This reuse can be seen in the rows of the
-    results grid.
+    Running a job consists of running its individual "steps".  Each
+    step is an individual test operation, such as "build the
+    hypervisor", "install a guest", "start a guest", "migrate a
+    guest", etc.
+
+    Generally a step corresponds to one execution of some ts-* script.
+
+    steps have a "testid" which is used to uniquely identify the same
+    operation in different test runs.  This is used for identifying
+    regressions, bisection, and so on.
+
+    The steps for a job, including the testids, are defined by the
+    recipe in sg-run-job (see "job", above).
+
+    Each step run in a job becomes a cell in the HTML results grid.  A
+    given step might appear in multiple different jobs, e.g. the "xen
+    build" step is used in several different build-* jobs.  This can
+    be seen in the rows of the results grid.
 
 "runvar":
 
-    A runvar is a named textual variable associated with each job in a
-    given flight. They serve as both the inputs and outputs to the
+    A runvar is a named textual variable associated with a given job
+    and flight. They serve as both the inputs and outputs to the
     job.
 
     For example a Xen build job may have input runvars "tree_xen" (the
@@ -55,6 +78,15 @@ Terminology
         * the parameters of the guest to test (e.g. distro, PV vs HVM
           etc).
 
+    Runvar names often have structure, being assembled out of various
+    pieces; and the names are sometimes parsed, too.
+
+flights, jobs and runvars are kept in the database (in standalone
+mode, "standalone.db").  As for steps, only steps which have been
+started are recorded in the db; steps which have yet to be attempted
+are not in the database.  Also, if a ts-* script is run by hand, this
+is not recorded as a step.
+
 Operation
 =========
 
@@ -73,12 +105,25 @@ referenced by each job's configuration. It then runs each of these in
 turn, taking into account the prerequisites etc, by calling the
 relevant "ts-*" scripts.
 
+Each ts-* is a convenient unit of test execution and reporting.  Most
+ts-* scripts are written in perl, although simple shell wrappers are
+sometimes used.  ts-* scripts expect OSSTEST_FLIGHT and OSSTEST_JOB to
+be set in the environment.  Most of them also expect to be told extra
+information on the command line - notably, which host(s) to operate
+on.
+
+In automatic operation, the command line arguments for each test
+step's ts-* script invocation are fixed by the recipe defined in
+sg-run-job.
+
 When running in standalone mode it is possible to run any of these
 steps by hand, ("mg-execute-flight", "sg-run-job", "ts-*") although
 you will need to find the correct inputs (some of which are documented
 below) and perhaps take care of prerequisites yourself (e.g. running
 "./sg-run-job test-armhf-armhf-xl" means you must have done
-"./sg-runjob build-armhf" and "build-armhf-pvops" first.
+"./sg-runjob build-armhf" and "build-armhf-pvops" first.  When running
+a ts-* script manually one often wants to specify "host=<hostname>".
+(See the head comment for "selecthost" in Osstest/TestSupport.pm.)
 
 Results
 =======
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 3/4] README.dev: Miscellaneous minor improvements

[Ian Jackson]

Clarify documentation on mg-blockage, and also suggest using
allocation instead.

No longer advise removing crontab as a way to shut osstest down.  The
stop file approach works fine and is less disruptive.

Minor formatting changes.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.dev | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/README.dev b/README.dev
index 09b9531..6d37ada 100644
--- a/README.dev
+++ b/README.dev
@@ -22,11 +22,14 @@ Removing machines for servicing/outage
 ======================================
 
 mg-blockage <date-from> <date-to> <hostflag> [<extra-xinfo>]
-OSSTEST_CONFIG=production-config ./mg-blockage '2014-05-21 0955' '2014-05-21 1400' equiv-marilith
 
+Eg
+ OSSTEST_CONFIG=production-config ./mg-blockage '2014-05-21 0955' '2014-05-21 1400' equiv-marilith
 
 Keeps running for the duration, so run it in a screen on the osstest VM.
 
+Or you can use mg-allocate.
+
 Commisioning a new machine
 ==========================
 
@@ -70,17 +73,16 @@ blessing and add the production ones, e.g.
 
 $ mg-hosts setflags HOSTA HOSTB -- \!blessed-commission blessed-{real,play,adhoc}
 
-Shutting down
-=============
 
-crontab -r (remove)
-crontab < crontab (restore afterwards)
+Shutting down the whole system
+==============================
 
-you can create a file "stop" in ~osstest/testing.git or
+You can create a file "stop" in ~osstest/testing.git or
 ~osstest/for-blah/something.git to cause that cron job to not start
 any new things. Put a note in the stop file to say why...  (in the
 former case, disables everything; in the latter, just that "branch")
 
+
 Creating a new branch
 =====================
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 4/4] README.dev: Improve instructions for new machine commissioning

[Ian Jackson]

Discuss preliminary setup (hardware and BIOS).

Invent "mudcake0" and "mudcake1" as machines being added and use them
consistently in the example runes.

Recommend to run a host examination flight first.  (NB I am still
testing that this actually works as expected.  Host examination is a
bit new.)

Consistently suggest use of a $basis, and use of $flight variable.

Do not prepend runes with OSSTEST_CONFIG setting.  Instead, we expect
the user to have it set already (and mention this).

Use "cs-adjust-flight new:" rather than the two-step process.

Formatting improvements etc.

CC: Xudong Hao <xudong.hao@intel.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.dev | 113 ++++++++++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 90 insertions(+), 23 deletions(-)

diff --git a/README.dev b/README.dev
index 6d37ada..dc0dafd 100644
--- a/README.dev
+++ b/README.dev
@@ -33,45 +33,112 @@ Or you can use mg-allocate.
 Commisioning a new machine
 ==========================
 
+Firstly, arrange that it is hooked up to network, serial, and pdu.
+
+(In the Xen Project Massachusetts test lab, make sure that ansible
+playbook hosts/ has the wiring information; running `make install'
+will then arrange for /root/osstest-hosts-config to contain the Ether,
+Serial and PowerMethod configuration, and also configure the DNS and
+the DHCP server.)
+
+Go through the BIOS set up, using the serial console.  Check that:
+ - After AC power loss, machine is set to turn on
+   (NOT "last state" which is usually the default)   
+ - Serial console is enabled (usually with "redirection after boot")
+ - Boot order is 1. primary network 2. primary disk
+ - Virtualisation extensions (and VT-d if applicable) are enabled
+ - ILOM is disabled (or, us using a not-connected network port)
+
+In the Massachusetts instance you access the machines' serial consoles
+with something like
+   ssh -vt root@serial0.test-lab sympathy -r mudcake0
+
+Most BIOSes can be entered by hammering on, alternately, ESC 2
+or ESC 0 or DEL or something.
+
+The program "xduplic-copier" can be useful when used with xterm to
+type into two serial consoles at once - that lets you make sure the
+two BIOSes are set up completely identically.
+
+If the machine's ethernet address is not already in the relevant
+database, it can be collected via the BIOS and should be recorded.
+(In Massachusetts, in ansible playbook/hosts/testnet.info.)
+
+Secondly, set it up in the osstest database
+-------------------------------------------
+
+Most of these tools can operate on multiple hosts, and we like to have
+hosts in at least pairs.  Let us imagine we are adding two hosts
+mudcake0 and mudcake1.
+
 Base it on an existing machine:
 
-$ mg-hosts create-like marilith-n5 marilith-n4
+ $ ./mg-hosts create-like merlot0 mudcake0,mudcake1
+
+Allocate it for the duration of setup:
+
+ $ ./mg-allocate mudcake{0,1}		# this shell syntax provides
+ 					# two arguments, mudcake0 mudcake1
+
+Set up the host flags:
+
+ $ ./mg-hosts setflags mudcake{0,1} -- blessed-commission-mudcake equiv-mudcake arch-{amd64,i386} arch-xen-{amd64,i386} hvm hvm-intel purpose-{build,test} suite-{jessie,stretch}
+
+(Consult mg-hosts showflags to see which flags are appropriate.)
+
+Set up any necessary host properties:
+
+ $ ./mg-hosts setprops mudcake{0,1} -- ....
+
+For example, one might need something like this:
+
+ $ ./mg-hosts setprops marilith-n4 -- dhcp-watch-method 'leases dhcp3 dns-cam1.uk.xensource.com:5556'
+
+(Many of these things are not needed in Massachusetts as the ansible
+playbook will provide it via an autogenerated config file - see above.)
+
+Create the tftp directory:
 
-Allocate it for the duration of setup
-$ mg-allocate HOST
+ $ ./mg-hosts mknetbootdir mudcake{0,1}
 
-Set up the props and flags
-$ mg-hosts setprops ...
-$ mg-hosts setflags ...
+Run commissioning test flights
+------------------------------
 
-$ mg-hosts setprops marilith-n4 -- dhcp-watch-method 'leases dhcp3 dns-cam1.uk.xensource.com:5556'
-Handles multiple hosts:
-$ mg-hosts setprops HOSTA HOSTB -- "PROP" "VALUE"
+Firstly, a basic "host examination" that checks that we can install
+and boot Xen:
 
-$ mg-hosts mknetbootdir HOSTA HOSTB
+ $ basis=113124   # pick last good xen-unstable or osstest flight
+ $ flight=`./make-hosts-flight play xen-unstable blessed-commission-mudcake commission-mudcake $basis`; echo $flight
+ 113155
+ $ ./mg-execute-flight -Bcommission-mudcake -Eian.jackson@citrix.com $flight
 
-Create a flight:
-$ FLIGHT=`OSSTEST_CONFIG=production-config ./cs-flight-create commission xen-unstable`
-$ echo $FLIGHT
-20974
+This will email the specified address.  The examination should pass,
+completely.  If it does not then you may need to change the BIOS
+serial arrangements, or do other debugging.
 
-Clone an existing flight, e.g. 20855 was previous successful flight
-$ OSSTEST_CONFIG=production-config ./cs-adjust-flight $FLIGHT copy 20855 
+It is best to run mg-execute-flight in screen, so that it will survive
+ssh session disconnection.  You should be running out of an osstest
+tree (testing.git clone) with good recent code, with an appropriate
+OSSTEST_CONFIG setting (eg OSSTEST_CONFIG=production-config, in
+Massachusetts).  You can run it as yourself, though, if you have the
+right permissions set up.
 
-"commission" is a blessing, hosts should be blessed with it, e.g.
+If that works, a more thorough test:
 
-$ mg-hosts setflags HOSTA HOSTB -- blessed-commission
+ $ basis=113124   # pick last good xen-unstable or osstest flight
+ $ flight=`./cs-adjust-flight new:commission-mudcake copy $basis`; echo $flight
+ $ ./mg-execute-flight -Bcommission-mudcake -Eian.jackson@citrix.com -f$basis $flight
 
-Can use anything e.g. commission-$classofmachine if doing multiple
-commissions in parallel.
+This should show no regressions.  (Or, at least, none that are a cause
+for concern.)
 
-Run it, e.g. in a screen session:
-OSSTEST_CONFIG=production-config ./mg-execute-flight  -Bcommission -Eian.campbell@citrix.com -f20855 $FLIGHT
+Bless
+-----
 
 Once machines are ready for production use remove the commission
 blessing and add the production ones, e.g.
 
-$ mg-hosts setflags HOSTA HOSTB -- \!blessed-commission blessed-{real,play,adhoc}
+ $ ./mg-hosts setflags mudcake{0,1} -- \!blessed-commission blessed-{real,play,adhoc}
 
 
 Shutting down the whole system
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel