Re: [PATCH 4/8] meson: detect missing tests at configure time

[Patrick Steinhardt <ps@pks.im>]

On Fri, Dec 13, 2024 at 10:58:47AM +0100, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > diff --git a/t/meson.build b/t/meson.build
> > index 9e676e69363ed6311426500d98fe281e30d26bcb..f1fbc6ae179079f4d5d86f9a60956fad84d0495c 100644
> > --- a/t/meson.build
> > +++ b/t/meson.build
> > @@ -1092,6 +1092,42 @@ integration_tests = [
> >    't9903-bash-prompt.sh',
> >  ]
> >  
> > +# Sanity check that we are not missing any tests present in 't/'. This check
> > +# only runs once at configure time and is thus best-effort, only. It is
> > +# sufficient to catch missing test suites in our CI though.
> > +foreach glob, tests : {
> > +  't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,
> > +  'unit-tests/t-*.c': unit_test_programs,
> > +  'unit-tests/u-*.c': clar_test_suites,
> > +}
> > +  actual_tests = run_command(shell, '-c', 'ls ' + glob,
> > +    check: true,
> > +    env: script_environment,
> > +  ).stdout().strip().split('\n')
> > +
> > +  if tests != actual_tests
> > +    missing_tests = [ ]
> > +    foreach actual_test : actual_tests
> > +      if actual_test not in tests
> > +        missing_tests += actual_test
> > +      endif
> > +    endforeach
> > +    if missing_tests.length() > 0
> > +      error('Missing tests:\n\n - ' + '\n - '.join(missing_tests))
> 
> This gives nice output:
> 
>     $ touch t/unit-tests/u-bar.c t/unit-tests/u-foo.c
> 
>     $ meson setup builddir --reconfigure
> 
>     The Meson build system
>     Version: 1.4.1
>     [snip]
>     Configuring update.sample using configuration
>     Configuring exclude using configuration
> 
>     t/meson.build:1116:6: ERROR: Problem encountered: Missing tests:
> 
>      - unit-tests/u-bar.c
>      - unit-tests/u-foo.c
> 
>     A full log can be found at git/builddir/meson-logs/meson-log.txt
> 
> But I think the error message is a little bit confusing. It sounds like
> the test file is missing, but it's the configuration of the test that is
> missing.
> 
> Now I've realized it hard to write a good error message here. But I
> would suggest something like "Tests files found, but not configured".
> 
> > +    endif
> > +
> > +    superfluous_tests = [ ]
> > +    foreach integration_test : tests
> > +      if integration_test not in actual_tests
> > +        superfluous_tests += integration_test
> > +      endif
> > +    endforeach
> > +    if superfluous_tests.length() > 0
> > +      error('Superfluous tests:\n\n - ' + '\n - '.join(superfluous_tests))
> 
> Also here I would suggest different error message, maybe something like
> "Tests configured, but file not found"

Both good suggestions, thanks!

Patrick