Re: [PATCH] tests: add an optional test to test git-annex

[Brandon Williams <bmwill@google.com>]

On 05/17, Junio C Hamano wrote:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
> 
> >> Well, it is one thing to place git-annex under CI to make sure its
> >> latest and greatest works together well with our latest and greatest
> >> (and it may be something we want to see happen), but driving its
> >> tests from our testsuite sounds like a tail wagging the dog, at
> >> least to me.
> >
> > To me this is just a question of:
> >
> > * Is it the case that git-annex tests for a lot of edge cases we don't
> > test for: Yes, probably. As evidenced by them spotting this
> > regression, and not us.
> 
> And I'd encourage them to keep doing so.
> 
> > * We can (and should) add a test for the specific breakage we caused
> > in 2.13.0, but that's no replacement for other things annex may be
> > covering & we may be missing which'll catch future breakages.
> >
> > * It's a pretty established practice to test a library (git) along
> > with its consumers (e.g. annex) before a major release.
> 
> I am not so sure about the division of labor.  What you are
> advocating would work _ONLY_ if we test with a perfect & bug-free
> version of the consumers.  If they are also a moving target, then
> I do not think it is worth it.  After all, we are *not* in the
> business of testing these consumers.

I agree with this. It makes no sense to test consumers of git, its
downstream's job to do that.  Though I do think that its perfectly
reasonable to test that our API works as advertised such that consumer's
can rely on git.

> 
> Unless I misunderstood you and you were saying that we freeze a
> version, or a set of versions, of customer that is/are known to pass
> their own tests, and test the combination of that frozen version of
> the customer with our daily development.  If that is the case, then
> I would agree that we are using their test to test us, not them.
> But I somehow didn't get that impression, hence my reaction.

-- 
Brandon Williams