Re: [PATCH v4 8/7] wildmatch test: skip file creation tests on Windows proper

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Sat, Jan 06 2018, Johannes Schindelin jotted:

> Hi Junio,
>
> On Fri, 5 Jan 2018, Junio C Hamano wrote:
>
>> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>>
>> > Skip the newly added file creation tests on Windows proper, these
>> > already work under Cygwin, but as that involves a significant
>> > emulation layer the results are different under Windows proper with
>> > MinGW.
>> > ...
>> > diff --git a/t/t3070-wildmatch.sh b/t/t3070-wildmatch.sh
>> > index f606f91acb..50a53e7a62 100755
>> > --- a/t/t3070-wildmatch.sh
>> > +++ b/t/t3070-wildmatch.sh
>> > @@ -7,6 +7,14 @@ test_description='wildmatch tests'
>> >  create_test_file() {
>> >  	file=$1
>> >
>> > +	# These tests limp along under Cygwin, but the failures on
>> > +	# native Windows are still too many. Skip file tests there
>> > +	# until they're sorted out.
>> > +	if test_have_prereq MINGW
>> > +	then
>> > +		return 1
>> > +	fi
>> > +
>>
>> That looks to be a nuclear option.
>
> Indeed:
>
> 	# still have 84 known breakage(s)
> 	# failed 52 among remaining 1428 test(s)
> 	1..1512
>
> Let's just switch it off completely because 52 of those 1512 test cases
> break only on Windows? Pretty heavy-handed.

Can you please provide me with the output of the test under -v -x -d
from github.com:avar/git.git wildmatch-refactor-8 branch?

It has some improvements that should make things a bit faster for you,
but also most of the logic is now in test_expect_success so -x is more
helpful.

I should then be able to fix these 52 remaining cases (and at least 1 of
them should be fixed already).

> But do read on.
>
>> For now it may be suffice, but somehow it feels that it goes directly
>> against Dscho's wish to treat (or pretend to treat) Windows as
>> first-class citizens and/or just like other platforms.
>
> It not only goes against my wish to treat Windows as a normal citizen
> instead of like Rey's parents, it also goes against my wish to have a
> focused and meaningful test suite. Nobody likes to run tests that take too
> long. And look at this:
>
> 	...
> 	ok 1511 - ipathmatch: match 'Z' '[Z-y]'
> 	ok 1512 - ipathmatch(ls): match '[Z-y]' 'Z'
> 	# still have 84 known breakage(s)
> 	# failed 52 among remaining 1428 test(s)
> 	1..1512
>
> 	real    5m51.432s
> 	user    0m33.986s
> 	sys     2m13.162s
>
> Yep. It takes *over eight minutes*.
>
> And this is a *fast* machine.
>
> Why? Because of the completely overboard testing of all kinds of
> *potential* breakages *at some point* in the future.
>
> I understand that Ævar wants to increase test coverage. I am sympathetic
> to this cause.
>
> But I completely disagree that increasing the test coverage beyond reason
> is a good thing. Tests *can* take too long, and they do, in practice, and
> the results are always problematic: every developer who has to deal with
> test suites that take too long... does not run them. As simple as that.
> And then you have *zero* test coverage. Pretty stupid, eh? And contrary to
> your intentions, too.
>
> So yes, I think that it has been proven beyond any doubt that t3070 takes
> *waaaaaaaaaay* too long on Windows, for dubitable benefit.
>
> I could see a reasonable compromise where
>
> - an extensive test matrix is hidden behind an EXPENSIVE_WILDMATCH prereq,
>
> - the test matrix would be written in a much more understandable way, i.e.
>   using English words rather than "1"s. If need be, there could be blocks
>   ("test this with case-sensitive matching", "skip case-sensitive matching",
>   you get the idea),

I take your point with the readability of some of this stuff, and will
get around to fixing that before the next submission.

> - these magic skippings of certain test cases (where the (non-traced)
>   `create_test_file()` function returns 1 for certain things) still would
>   need to go away, in favor probably of prereqs and/or blocks where flags
>   are set accordingly in a preamble,

This is a lot of work for dubious benefit. I'm not going to try to
maintain some exhaustive mapping that's potentially going to be a hash
4-levels deep of:

    os.os_version.fs.vs_version

And that's before we'd get to the potential 6-level crazyness of:

    os.os_version.os_flags.fs.vs_version.fs_flags

It's way easier and more portable (even to OSs or FSs nobody's invented
yet) to just test whether a file can be created, and if not skip the
test.

As I explained in 20180105221222.28867-1-avarab@gmail.com the actual
benefit of this test is that as much as possible is tested
*somewhere*. There's no reason to suspect that e.g. Linux will overnight
make the character ":" illegal in filenames and we'll be the only ones
who notice it.

Since we don't treat ":" or any other character differently for the
purposes of glob matching on Windows, but *do* treat the raw glob
matching differently than calling wildmatch() directly, as the current
tests do, I really don't see what the point of this exercises would be.

> - by default, i.e. without the EXPENSIVE_WILDMATCH prereq, it should test
>   a *tiny* subset that is indicative of the most likely bugs.

> As it is, I like 8/7 in the present form for all the wrong reasons: it
> protects me from the damage a full t3070 would do to me.

This test takes around 8 seconds to run on a 3.4GHz i7 on Linux, and
around 4 seconds on a 2.6GHz i5 on my SSD laptop on Linux.

I agree that there should be some prerequisite to skip this on Windows
by default since 6 minutes is clearly excessive (although I think you'll
find it runs a bit faster in the branch above), but that should be
something like:

    test_lazy_prereq EXPENSIVE_ON_WINDOWS '
            test -n "$GIT_TEST_LONG" || test_have_prereq !MINGW,!CYGWIN
    '

As the long runtime is not inherent to the test, but to excessive
slowness caused by certain operations on certain platforms, or maybe it
should be EXPENSIVE_ON_SLOW_FS or EXPENSIVE_IF_FORKING_IS_SLOW or if
we'd like to generalize it.