[PATCH v2 0/4] Smoke testing for Git

[PATCH v2 0/4] Smoke testing for Git

[Ævar Arnfjörð Bjarmason]

v2 of the non-RFC version of the smoke testing for Git series. It goes
on top of Git (due to pu changes in t/README).

Changes since v1:

  * We can't include GIT-BUILD-OPTIONS. Change by Thomas Rast

  * Add SMOKE_{COMMENT,TAGS}= to submit a comment or list of tags with
    the smoke report. There are examples of this at
    http://smoke.git.nix.is/app/projects/smoke_reports/1

  * Do --foo="$(BAR)", not --foo=$(BAR) to avoid quoting issues

  * Minor changes not worth noting, like rewording commit messages a
    bit.

Ævar Arnfjörð Bjarmason (4):
  tests: Infrastructure for Git smoke testing
  t/README: Document the Smoke testing
  t/Makefile: Can't include GIT-BUILD-OPTIONS, it's a .sh
  t/README: Add SMOKE_{COMMENT,TAGS}= to smoke_report target

 t/Makefile |   41 ++++++++++++++++++++++++++++++++-
 t/README   |   73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 t/harness  |   21 +++++++++++++++++
 3 files changed, 134 insertions(+), 1 deletions(-)
 create mode 100755 t/harness

[PATCH v2 1/4] tests: Infrastructure for Git smoke testing

[Ævar Arnfjörð Bjarmason]

Add the capability to send smoke reports from the Git test suite.

Currently we only notice bugs in the test suite when it's run
manually. Bugs in Git that only occur on obscure platforms or setups
that the core developers aren't using can thus go unnoticed.

This series aims to change that. With it, anyone that's interested in
avoiding bitrot in Git can volunteer to run a smoke tester. A smoke
tester periodically compiles the latest version of Git, runs the test
suite, and submits a report to a central server indicating how the
test run went.

A smoke tester might run something like this in cron:

    #!/bin/sh
    cd ~/g/git
    git fetch
    for branch in maint master next pu; do
        git checkout origin/$i &&
        make clean all &&
        cd t &&
        make smoke_report
    done

The smoker might want to compile git with non-default flags, include
bisecting functionality or run the tests under valgrind. Doing that is
outside the scope of this patch, this just adds a report submission
mechanism. But including a canonical smoke runner is something we'll
want to include eventually.

What this does now is add smoke and smoke_report targets to t/Makefile
(this example only uses a few tests for demonstration):

    $ make clean smoke
    rm -f -r 'trash directory'.* test-results
    rm -f t????/cvsroot/CVSROOT/?*
    rm -f -r valgrind/bin
    rm -f .prove
    perl ./harness --git-version="1.7.2.1.173.gc9b40" \
                    --no-verbose \
                    --archive="test-results/git-smoke.tar.gz" \
                    t0000-basic.sh t0001-init.sh t0002-gitfile.sh t0003-attributes.sh t0004-unwritable.sh t0005-signals.sh t0006-date.sh
    t0000-basic.sh ....... ok
    t0001-init.sh ........ ok
    t0002-gitfile.sh ..... ok
    t0003-attributes.sh .. ok
    t0004-unwritable.sh .. ok
    t0005-signals.sh ..... ok
    t0006-date.sh ........ ok
    All tests successful.

    Test Summary Report
    -------------------
    t0000-basic.sh     (Wstat: 0 Tests: 46 Failed: 0)
      TODO passed:   5
    Files=7, Tests=134,  3 wallclock secs ( 0.06 usr  0.05 sys +  0.23 cusr  1.33 csys =  1.67 CPU)
    Result: PASS

    TAP Archive created at /home/avar/g/git/t/test-results/git-smoke.tar.gz

The smoke target uses TAP::Harness::Archive to aggregate the test
results into a tarball. The tarball contains two things, the output of
every test file that was run, and a metadata file:

Tarball contents:

    $ tar xzvf git-smoke.tar.gz
    t0004-unwritable.sh
    t0001-init.sh
    t0002-gitfile.sh
    t0005-signals.sh
    t0000-basic.sh
    t0003-attributes.sh
    t0006-date.sh
    meta.yml

A test report:

    $ cat t0005-signals.sh
    ok 1 - sigchain works
    # passed all 1 test(s)
    1..1

A metadata file:

    ---
    extra_properties:
    file_attributes:
      -
        description: t0000-basic.sh
        end_time: 1280437324.61398
        start_time: 1280437324.22186
      -
        description: t0001-init.sh
        end_time: 1280437325.12346
        start_time: 1280437324.62393
      -
        description: t0002-gitfile.sh
        end_time: 1280437325.29428
        start_time: 1280437325.13646
      -
        description: t0003-attributes.sh
        end_time: 1280437325.59678
        start_time: 1280437325.30565
      -
        description: t0004-unwritable.sh
        end_time: 1280437325.77376
        start_time: 1280437325.61003
      -
        description: t0005-signals.sh
        end_time: 1280437325.85426
        start_time: 1280437325.78727
      -
        description: t0006-date.sh
        end_time: 1280437326.2362
        start_time: 1280437325.86768
    file_order:
      - t0000-basic.sh
      - t0001-init.sh
      - t0002-gitfile.sh
      - t0003-attributes.sh
      - t0004-unwritable.sh
      - t0005-signals.sh
      - t0006-date.sh
    start_time: 1280437324
    stop_time: 1280437326

The "extra_properties" hash is where we'll stick Git-specific info,
like whether Git was compiled with gettext or the fallback regex
engine, and what branch we're compiling. Currently no metadata like
this is included.

The entire tarball is then submitted to a central smokebox at
smoke.git.nix.is. This is done with curl(1) via the "smoke_report"
target:

    $ make smoke_report
    curl \
                    -H "Expect: " \
                    -F project=Git \
                    -F architecture=x86_64 \
                    -F platform=Linux \
                    -F revision="1.7.2.1.173.gc9b40" \
                    -F report_file=@test-results/git-smoke.tar.gz \
                    http://smoke.git.nix.is/app/projects/process_add_report/1 \
            | grep -v ^Redirecting
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  117k  100    63  100  117k      3   6430  0:00:21  0:00:18  0:00:03     0
    Reported #8 added.

Reports are then made available on the smokebox via a web interface:

    http://smoke.git.nix.is/app/projects/smoke_reports/1

The smoke reports are also mirrored to a Git repository hosted on
GitHub:

    http://github.com/gitsmoke/smoke-reports

The Smolder SQLite database that contains metadata about the reports
is also made available:

    http://github.com/gitsmoke/smoke-database

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/Makefile |   35 ++++++++++++++++++++++++++++++++++-
 t/harness  |   21 +++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletions(-)
 create mode 100755 t/harness

diff --git a/t/Makefile b/t/Makefile
index 819b936..d8bf05f 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -49,4 +49,37 @@ full-svn-test:
 valgrind:
 	GIT_TEST_OPTS=--valgrind $(MAKE)
 
-.PHONY: pre-clean $(T) aggregate-results clean valgrind
+# Smoke testing targets
+-include ../GIT-BUILD-OPTIONS
+-include ../GIT-VERSION-FILE
+uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo unknown')
+uname_M := $(shell sh -c 'uname -m 2>/dev/null || echo unknown')
+
+test-results:
+	mkdir -p test-results
+
+test-results/git-smoke.tar.gz:
+	$(PERL_PATH) ./harness \
+		--archive="test-results/git-smoke.tar.gz" \
+		$(T)
+
+smoke: test-results/git-smoke.tar.gz
+
+SMOKE_UPLOAD_FLAGS =
+ifdef SMOKE_USERNAME
+	SMOKE_UPLOAD_FLAGS += -F username="$(SMOKE_USERNAME)" -F password="$(SMOKE_PASSWORD)"
+endif
+
+smoke_report: smoke
+	curl \
+		-H "Expect: " \
+		-F project=Git \
+		-F architecture="$(uname_M)" \
+		-F platform="$(uname_S)" \
+		-F revision="$(GIT_VERSION)" \
+		-F report_file=@test-results/git-smoke.tar.gz \
+		$(SMOKE_UPLOAD_FLAGS) \
+		http://smoke.git.nix.is/app/projects/process_add_report/1 \
+	| grep -v ^Redirecting
+
+.PHONY: pre-clean $(T) aggregate-results clean valgrind smoke smoke_report
diff --git a/t/harness b/t/harness
new file mode 100755
index 0000000..f5c02f4
--- /dev/null
+++ b/t/harness
@@ -0,0 +1,21 @@
+#!/usr/bin/perl
+use strict;
+use warnings;
+use Getopt::Long ();
+use TAP::Harness::Archive;
+
+Getopt::Long::Parser->new(
+	config => [ qw/ pass_through / ],
+)->getoptions(
+	'jobs:1'    => \(my $jobs = $ENV{TEST_JOBS}),
+	'archive=s' => \my $archive,
+) or die "$0: Couldn't getoptions()";
+
+TAP::Harness::Archive->new({
+	jobs             => $jobs,
+	archive          => $archive,
+	($ENV{GIT_TEST_OPTS}
+	 ? (test_args    => [ split /\s+/, $ENV{GIT_TEST_OPTS} ])
+	 : ()),
+	extra_properties => {},
+})->runtests(@ARGV);
-- 
1.7.1

Re: [PATCH v2 1/4] tests: Infrastructure for Git smoke testing

[Sverre Rabbelier]

Heya,

On Sun, Aug 8, 2010 at 09:49, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> +smoke_report: smoke
> +       curl \
> +               -H "Expect: " \
> +               -F project=Git \
> +               -F architecture="$(uname_M)" \
> +               -F platform="$(uname_S)" \
> +               -F revision="$(GIT_VERSION)" \
> +               -F report_file=@test-results/git-smoke.tar.gz \
> +               $(SMOKE_UPLOAD_FLAGS) \
> +               http://smoke.git.nix.is/app/projects/process_add_report/1 \

I think at the very least this URL should be given it's own variable
so that it can be overridden easily.

-- 
Cheers,

Sverre Rabbelier

[PATCH v2 2/4] t/README: Document the Smoke testing

[Ævar Arnfjörð Bjarmason]

Git now has a smoke testing service at http://smoke.git.nix.is that
anyone can send reports to. Change the t/README file to mention this.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/README |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 66 insertions(+), 0 deletions(-)

diff --git a/t/README b/t/README
index e957d4f..69af05f 100644
--- a/t/README
+++ b/t/README
@@ -574,3 +574,69 @@ Then, at the top-level:
 That'll generate a detailed cover report in the "cover_db_html"
 directory, which you can then copy to a webserver, or inspect locally
 in a browser.
+
+Smoke testing
+-------------
+
+The Git test suite has support for smoke testing. Smoke testing is
+when you submit the results of a test run to a central server for
+analysis and aggregation.
+
+Running a smoke tester is an easy and valuable way of contributing to
+Git development, particularly if you have access to an uncommon OS on
+obscure hardware.
+
+After building Git you can generate a smoke report like this in the
+"t" directory:
+
+    make clean smoke
+
+You can also pass arguments via the environment. This should make it
+faster:
+
+    GIT_TEST_OPTS='--root=/dev/shm' TEST_JOBS=10 make clean smoke
+
+The "smoke" target will run the Git test suite with Perl's
+"TAP::Harness" module, and package up the results in a .tar.gz archive
+with "TAP::Harness::Archive". The former is included with Perl v5.10.1
+or later, but you'll need to install the latter from the CPAN. See the
+"Test coverage" section above for how you might do that.
+
+Once the "smoke" target finishes you'll see a message like this:
+
+    TAP Archive created at <path to git>/t/test-results/git-smoke.tar.gz
+
+To upload the smoke report you need to have curl(1) installed, then
+do:
+
+    make smoke_report
+
+To upload the report anonymously. Hopefully that'll return something
+like "Reported #7 added.".
+
+If you're going to be uploading reports frequently please request a
+user account by E-Mailing gitsmoke@v.nix.is. Once you have a username
+and password you'll be able to do:
+
+    SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> make smoke_report
+
+Once the report is uploaded it'll be made available at
+http://smoke.git.nix.is, here's an overview of Recent Smoke Reports
+for Git:
+
+    http://smoke.git.nix.is/app/projects/smoke_reports/1
+
+The reports will also be mirrored to GitHub every few hours:
+
+    http://github.com/gitsmoke/smoke-reports
+
+The Smolder SQLite database is also mirrored and made available for
+download:
+
+    http://github.com/gitsmoke/smoke-database
+
+Note that the database includes hashed (with crypt()) user passwords
+and E-Mail addresses. Don't use a valuable password for the smoke
+service if you have an account, or an E-Mail address you don't want to
+be publicly known. The user accounts are just meant to be convenient
+labels, they're not meant to be secure.
-- 
1.7.1

[PATCH v2 3/4] t/Makefile: Can't include GIT-BUILD-OPTIONS, it's a .sh

[Ævar Arnfjörð Bjarmason]

Change the smoke testing portion of t/Makefile not to include
GIT-BUILD-OPTIONS. It's a shellscript, not a Makefile snippet, so it
had the nasty side-effect of sneaking e.g. SHELL_PATH = '/bin/sh'
(with quotes) everywhere.

Just add our own PERL_PATH variable as a workaround. The t/Makefile
already has e.g. an equivalent SHELL_PATH and TAR option which
duplicate the definitions in GIT-BUILD-OPTIONS.

Reported-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/Makefile |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/t/Makefile b/t/Makefile
index d8bf05f..00f6482 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -8,6 +8,7 @@
 
 #GIT_TEST_OPTS=--verbose --debug
 SHELL_PATH ?= $(SHELL)
+PERL_PATH ?= /usr/bin/perl
 TAR ?= $(TAR)
 RM ?= rm -f
 
@@ -50,7 +51,6 @@ valgrind:
 	GIT_TEST_OPTS=--valgrind $(MAKE)
 
 # Smoke testing targets
--include ../GIT-BUILD-OPTIONS
 -include ../GIT-VERSION-FILE
 uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo unknown')
 uname_M := $(shell sh -c 'uname -m 2>/dev/null || echo unknown')
-- 
1.7.1

[PATCH v2 4/4] t/README: Add SMOKE_{COMMENT,TAGS}= to smoke_report target

[Ævar Arnfjörð Bjarmason]

The smoke server supports a free form text field with comments about a
report, and a comma delimited list of tags. Change the smoke_report
target to expose this functionality. Now smokers can send more data
that explains and categorizes the reports they're submitting.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/Makefile |    6 ++++++
 t/README   |    7 +++++++
 2 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/t/Makefile b/t/Makefile
index 00f6482..b81fb91 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -69,6 +69,12 @@ SMOKE_UPLOAD_FLAGS =
 ifdef SMOKE_USERNAME
 	SMOKE_UPLOAD_FLAGS += -F username="$(SMOKE_USERNAME)" -F password="$(SMOKE_PASSWORD)"
 endif
+ifdef SMOKE_COMMENT
+	SMOKE_UPLOAD_FLAGS += -F comments="$(SMOKE_COMMENT)"
+endif
+ifdef SMOKE_TAGS
+	SMOKE_UPLOAD_FLAGS += -F tags="$(SMOKE_TAGS)"
+endif
 
 smoke_report: smoke
 	curl \
diff --git a/t/README b/t/README
index 69af05f..9f676d7 100644
--- a/t/README
+++ b/t/README
@@ -620,6 +620,13 @@ and password you'll be able to do:
 
     SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> make smoke_report
 
+You can also add an additional comment to attach to the report, and/or
+a comma separated list of tags:
+
+    SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> \
+        SMOKE_COMMENT=<comment> SMOKE_TAGS=<tags> \
+        make smoke_report
+
 Once the report is uploaded it'll be made available at
 http://smoke.git.nix.is, here's an overview of Recent Smoke Reports
 for Git:
-- 
1.7.1