[PATCH RFC 0/9] Running benchmarks via OSSTest

[Dario Faggioli <dario.faggioli@citrix.com>]

Hello Everyone,

This is something I've been working in the latest few days, and it's about
making it possible to run benchmarks via OSSTest. I've (of course) tested it in
standalone mode, but nothing forbids to integrate it in our push gate, as we
were saying at the hackathon. Of course, there are a lot of things to be
decided. Where to run benchmarks, which benchmarks, how frequently, what to do
with the results, etc, but it's a start!  :-P

So, in standalone mode, after applying the patches, you can do the following,
and you'll have a PV guest running unixbench for a while, and the results
fetched for you on the OSSTest standalone controller stash area:

# Retrieve the benchmark archive
./mg-unixbench-download
# Reset, create the benchmarking flight, install the host and build Xen and Linux
./standalone-reset -t bench
./standalone run-job -R -h ghoul3 build-amd64   # consider --lvextendmax=XXGB, if necessary
./standalone run-job -h ghoul3 build-amd64-pvops
# Or set-paths --if build happened already-- like this:
#./standalone set-paths build-amd64
#./standalone set-paths build-amd64-pvops
# Install Xen on the host
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-xen-install
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-host-reboot
# Install (and start) the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-debian-install
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-debian-fixup debian
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-guest-start debian
# Prepare the benchmarking environment in the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-prep debian
# Run the benchmark in the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-run debian
# Retrieve the benchmark results:
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-reslts debian

A few jobs are being created. In case you want to use one, instead of invoking
all the ts-s, run the following:

# Retrieve the benchmark archive
./mg-unixbench-download    
# Reset, create the benchmarking flight, install the host and build Xen and Linux
./standalone-reset -t bench
./standalone run-job -R -h ghoul3 build-amd64   # consider --lvextendmax=XXGB, if necessary
./standalone run-job -h ghoul3 build-amd64-pvops
# Run the benchmarking job:
./standalone run-job -h ghoul3 bench-unixbench-amd64-amd64-xl-credit2

Almost all the loginc to use an HVM guest are there, a just need a few fixups
in ts-debian-hvm-install that I'll submit as a separate patch.

I tested both the ways in standalone mode, and it seems to be working here.

And that's it... So, what do you think about it? IanC? IanJ?

I now it's still a sort of a prototype, but I'd love to hear your thoughts
before proceeding any further.

One thing about the last two patches. They are there to show how the code will
look like if we take a slightly different path than what's outlined in the
first part of the series. The idea is, in fact, to have the mg-xxx-download and
the ts-xxx-{prep,run,reslts} scripts to be more generic, accept one (or more)
benchmark name(s) as parameter and prepare the target (host or guest) for them.
That would make the single scripts slightly more complex, but will avoid having
to have 4 scripts (downloading, preparing, running and fetching results) for
each new benchmark we want to introduce support for.

Personally, I think I prefer them to be generic, i.e., something similar to
what they look like after all the series is applied, as compared to how they
look only until patch 7. This is exacly an example of something I'd like to
hear your thoughts about before going forward. :-)

The series is also available here:

 git://xenbits.xen.org/people/dariof/osstest.git benchmarking-with-osstest-RFC

Regards,
Dario
---
Dario Faggioli (9):
      mg-unixbench-download: new script for downloading the unixbench archive
      ts-unixbench-prep: prep the environment for running unixbench
      ts-unixbench-run: kick off the benchmark on the target
      ts-unixbench-reslts: for retrieving the results
      sg-run-job: new recipe for a unixbench job
      make-bench-flight: to create a benchmarking flight
      make-flight: introduce a new -t option
      mg-unixbench-download: make it generic
      ts-unixbench-prep: make it generic


 ap-common             |    8 ++++
 make-bench-flight     |   92 ++++++++++++++++++++++++++++++++++++++++++++++
 mg-unixbench-download |   49 ++++++++++++++++++++++++
 sg-run-job            |   28 ++++++++++++++
 standalone-reset      |    9 +++-
 ts-unixbench-prep     |   99 +++++++++++++++++++++++++++++++++++++++++++++++++
 ts-unixbench-reslts   |   56 ++++++++++++++++++++++++++++
 ts-unixbench-run      |   45 ++++++++++++++++++++++
 8 files changed, 383 insertions(+), 3 deletions(-)
 create mode 100755 make-bench-flight
 create mode 100755 mg-unixbench-download
 create mode 100755 ts-unixbench-prep
 create mode 100755 ts-unixbench-reslts
 create mode 100755 ts-unixbench-run

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-------------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)