perl/Git.pm: Writing a test suite

perl/Git.pm: Writing a test suite

[Lea Wiemann]

Git.pm doesn't currently have a test suite, and I think it should get one.

Since Git.pm makes calls to the git binary, it would seem sensible to me 
to create mock output for the external calls, like Grit (a Ruby API) 
does it, here: http://github.com/mojombo/grit/tree/master/test/fixtures 
  (Having a complete test repository to test against like Ruby/Git 
(another API) at 
http://repo.or.cz/w/rubygit.git?a=tree;f=tests/files;hb=HEAD is probably 
too brittle and makes it hard to test edge cases.)

I haven't done testing in perl before, so if someone wants to exchange 
ideas and/or help me get this set up, please post here or message me on 
IRC (lea_w/lea_1 on #git at Freenode).

-- Lea

Re: perl/Git.pm: Writing a test suite

[Rafael Garcia-Suarez]

2008/5/30 Lea Wiemann <lewiemann@gmail.com>:
> Git.pm doesn't currently have a test suite, and I think it should get one.
>
> Since Git.pm makes calls to the git binary, it would seem sensible to me to
> create mock output for the external calls, like Grit (a Ruby API) does it,
> here: http://github.com/mojombo/grit/tree/master/test/fixtures  (Having a
> complete test repository to test against like Ruby/Git (another API) at
> http://repo.or.cz/w/rubygit.git?a=tree;f=tests/files;hb=HEAD is probably too
> brittle and makes it hard to test edge cases.)
>
> I haven't done testing in perl before, so if someone wants to exchange ideas
> and/or help me get this set up, please post here or message me on IRC
> (lea_w/lea_1 on #git at Freenode).

Testing is big in the perl world. I suggest to start with reading :
http://www.perlfoundation.org/perl5/index.cgi?testing
I'm not overloaded with tuits at the moment, but I'll try to review a
bit your testing patches when you post them here.

Re: perl/Git.pm: Writing a test suite

[Petr Baudis]

  Hi,

On Fri, May 30, 2008 at 09:43:48AM +0200, Lea Wiemann wrote:
> Git.pm doesn't currently have a test suite, and I think it should get one.
>
> Since Git.pm makes calls to the git binary, it would seem sensible to me to 
> create mock output for the external calls, like Grit (a Ruby API) does it, 
> here: http://github.com/mojombo/grit/tree/master/test/fixtures  (Having a 
> complete test repository to test against like Ruby/Git (another API) at 
> http://repo.or.cz/w/rubygit.git?a=tree;f=tests/files;hb=HEAD is probably 
> too brittle and makes it hard to test edge cases.)

  I think this is bad idea; you should test not just Git.pm
functionality, but equally importantly also its integration with
(current version of) Git. If Git's output/behaviour changes in some
subtle way, you want to see if it breaks Git.pm.

> I haven't done testing in perl before, so if someone wants to exchange 
> ideas and/or help me get this set up, please post here or message me on IRC 
> (lea_w/lea_1 on #git at Freenode).

  There are two possible ways, either using the Git test suite
infrastructure, or the Perl test suite infrastructure. I'm ambivalent
about which one to choose, both have their advantages; I wonder what
others' opinion is. I would personally go with the Perl test suite in
this case but can't support it with much more than just a gut feeling.

-- 
				Petr "Pasky" Baudis
Whatever you can do, or dream you can, begin it.
Boldness has genius, power, and magic in it.	-- J. W. von Goethe

Re: perl/Git.pm: Writing a test suite

[Lea Wiemann]

Petr Baudis wrote:
> Lea Wiemann wrote:
>> it would seem sensible to me to create mock output for the external calls
> 
>  I think this is bad idea; you should test not just Git.pm
> functionality, but equally importantly also its integration with
> (current version of) Git.

Good point.  So I'll work on test repositories with real calls to Git 
where possible.

>   There are two possible ways, either using the Git test suite
> infrastructure, or the Perl test suite infrastructure.

I'd definitely want to use Perl's testing infrastructure, but probably 
it's sensible to use both Git's and Perl's infrastructure (e.g. call the 
Git.pm perl tests from the Git test suite shell scripts, and maybe have 
setup code in the shell scripts).  Here are two questions:

1. Is it OK for the Git.pm tests to be called from the Git test scripts 
but generate different-looking output (since the tests are run in Perl)?

2. I might need external test helpers from CPAN at some point.  Is that 
a bad idea because of dependencies, or are we OK with having additional 
dependencies for running the test suite if necessary?

-- Lea

Re: perl/Git.pm: Writing a test suite

[Junio C Hamano]

Lea Wiemann <lewiemann@gmail.com> writes:

> 1. Is it OK for the Git.pm tests to be called from the Git test
> scripts but generate different-looking output (since the tests are run
> in Perl)?

Cosmetics.  As long as they are not disproportionally chatty nor too
terse, and correctly catch breakages and stop (if told to with -i) on the
first one, I think it is Ok to be different (e.g. no support for colored
output).

> 2. I might need external test helpers from CPAN at some point.  Is
> that a bad idea because of dependencies, or are we OK with having
> additional dependencies for running the test suite if necessary?

As long as you use the ones that is widely used in Perl world I do not
think it would be a problem.