Re: scripting speedups [was: [Summit topic] Crazy (and not so crazy) ideas]

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Tue, Oct 26 2021, Eric Wong wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>> * Test suite is slow. Shell scripts and process forking.
>> 
>>    * What if we had a special shell that interpreted the commands in a
>>      single process?
>> 
>>    * Even Git commands like rev-parse and hash-object, as long as that’s
>>      not the command you’re trying to test
>
> This is something I've wanted in a very long time as a scripter.
> fast-import has been great over the years, as is
> "cat-file --batch(-check)", but there's gaps should be filled
> (preferably without fragile linkage of shared libraries into a
> script process)
>
>>    * Dscho wants to slip in a C-based solution
>> 
>>    * Jonathan tan commented: going back to your custom shell for tests
>>      idea, one thing we could do is have a custom command that generates
>>      the repo commits that we want (and that saves process spawns and
>>      might make the tests simpler too)
>
> Perhaps a not-seriously-proposed patch from 2006 could be
> modernized for our now-libified internals:

I think something very short of a "C-based solution" could give us most
of the wins here. Johannes was probably thinking of the scripting being
slow on Windows aspect of it.

But the main benefit of hypothetical C-based testing is that you can
connect it to the dependency tree we have in the Makefile, and only
re-run tests for code you needed to re-compile.

So e.g. we don't need to run tests that invoke "git tag" if the
dependency graph of builtin/tag.c didn't change.

With COMPUTE_HEADER_DEPENDENCIES we've got access to that dependency
information for our C code.

With trace2 we could record an initial test run, and know which built-in
commands are executed by which tests (even down to the sub-test level).

Connecting these two means that we can find all tests that say run "git
fsck", and if builtin/fsck.c is the only thing that changed in an
interactive rebase, that's the only tests we need to run.

Of course changes to things like cache.h or t/test-lib.sh would spoil
that cache entirely, but pretty much the same is true for re-compiling
things now, so would changing say builtin/init-db.c, as almost every
test does a "git init" somewhere.

But I think that approch is viable, and should take us from a huge
hypothetical project like "rewrite all the tests in C" to something
that's a viable weekend hacking project for someone who's interested.