All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>, "Jeff King" <peff@peff.net>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH v2 8/8] t/README: Document the do's and don'ts of tests
Date: Thu,  1 Jul 2010 20:17:58 +0000	[thread overview]
Message-ID: <1278015478-6920-9-git-send-email-avarab@gmail.com> (raw)
In-Reply-To: <1278015478-6920-1-git-send-email-avarab@gmail.com>

Add a "Do's, don'ts & things to keep in mind" subsection to the
"Writing Tests" documentation. Much of this is based on Junio C
Hamano's "Test your stuff" section in
<7vhbkj2kcr.fsf@alter.siamese.dyndns.org>.

I turned it into a list of do's and don'ts to make it easier to skim
it, and integrated my note that a TAP harness will get confused if you
print "ok" or "not ok" at the beginning of a line.

Thad had to be fixed in 335f87871fe5aa6b3fd55b2b4e80f16fe9681483 when
TAP support was introduced.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/README |   78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 78 insertions(+), 0 deletions(-)

diff --git a/t/README b/t/README
index 5a291e1..4a1cfc9 100644
--- a/t/README
+++ b/t/README
@@ -231,6 +231,84 @@ This test harness library does the following things:
    consistently when command line arguments --verbose (or -v),
    --debug (or -d), and --immediate (or -i) is given.
 
+Do's, don'ts & things to keep in mind
+-------------------------------------
+
+Here's a few examples of things you probably should and shouldn't do
+when writing tests.
+
+Do:
+
+ - Put as much code as possible inside test_expect_success and other
+   assertions.
+
+   Even code that isn't a test per se, but merely some setup code
+   should be inside a test assertion if at all possible. Test scripts
+   should only have trivial code outside of their assertions.
+
+ - Chain your test assertions
+
+   Write test code like this:
+
+	git merge foo &&
+	git push bar &&
+	test ...
+
+   Instead of:
+
+	git merge hla
+	git push gh
+	test ...
+
+   That way all of the commands in your tests will succeed or fail. If
+   you must ignore the return value of something (e.g. the return
+   value of export is unportable) it's best to indicate so explicitly
+   with a semicolon:
+
+	export HLAGH;
+	git merge hla &&
+	git push gh &&
+	test ...
+
+Don't:
+
+ - exit() within a <script> part.
+
+   The harness will catch this as a programming error of the test.
+   Use test_done instead if you need to stop the tests early (see
+   "Skipping tests" below).
+
+ - Break the TAP output
+
+   The raw output from your test might be interpreted by a TAP
+   harness. You usually don't have to worry about that. TAP harnesses
+   will ignore everything they don't know about, but don't step on
+   their toes in these areas:
+
+   - Don't print lines like "$x..$y" where $x and $y are integers.
+
+   - Don't print lines that begin with "ok" or "not ok".
+
+   A TAP harness expect a line that begins with either "ok" and "not
+   ok" to signal a test passed or failed (and our harness already
+   produces such lines), so your script shouldn't emit such lines to
+   their output.
+
+   You can gleam some further possible issues from the TAP grammar
+   (see http://search.cpan.org/perldoc?TAP::Parser::Grammar#TAP_Grammar)
+   but the best indication is to just run the tests with prove(1),
+   it'll complain if anything is amiss.
+
+Keep in mind:
+
+ - That what you print to stderr and stdout is usually ignored
+
+   Inside <script> part, the standard output and standard error
+   streams are discarded, and the test harness only reports "ok" or
+   "not ok" to the end user running the tests. Under --verbose, they
+   are shown to help debugging the tests.
+
+
 Skipping tests
 --------------
 
-- 
1.7.1.251.g92a7

  parent reply	other threads:[~2010-07-01 20:19 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-07-01 20:17 [PATCH v2 0/8] Improvements for t/README Ævar Arnfjörð Bjarmason
2010-07-01 20:17 ` [PATCH v2 1/8] t/README: Tests are all +x, ./test, not sh ./test Ævar Arnfjörð Bjarmason
2010-07-01 23:54   ` Junio C Hamano
2010-07-02  1:00     ` Ævar Arnfjörð Bjarmason
2010-07-01 20:17 ` [PATCH v2 2/8] t/README: The trash is in 't/trash directory.$name' Ævar Arnfjörð Bjarmason
2010-07-01 20:17 ` [PATCH v2 3/8] t/README: Typo: paralell -> parallel Ævar Arnfjörð Bjarmason
2010-07-01 20:17 ` [PATCH v2 4/8] t/README: Document the prereq functions, and 3-arg test_* Ævar Arnfjörð Bjarmason
2010-07-01 20:17 ` [PATCH v2 5/8] t/README: Document test_external* Ævar Arnfjörð Bjarmason
2010-07-01 20:52   ` Jakub Narebski
2010-07-01 21:18     ` [PATCH v3 " Ævar Arnfjörð Bjarmason
2010-07-01 20:17 ` [PATCH v2 6/8] t/README: Document test_expect_code Ævar Arnfjörð Bjarmason
2010-07-01 20:17 ` [PATCH v2 7/8] t/README: Add a section about skipping tests Ævar Arnfjörð Bjarmason
2010-07-01 20:17 ` Ævar Arnfjörð Bjarmason [this message]
2010-07-02  7:28 ` [PATCH v2 0/8] Improvements for t/README Jeff King

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1278015478-6920-9-git-send-email-avarab@gmail.com \
    --to=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.