Re: The fate of iotest 297

[Kevin Wolf <kwolf@redhat.com>]

Am 18.05.2022 um 20:21 hat John Snow geschrieben:
> To wire it up to "make check" by *default*, I believe I need to expand the
> configure script to poll for certain requisites and then create some
> wrapper script of some kind that only engages the python tests if the
> requisites were met ... and I lose some control over the mypy/pylint
> versioning windows. I have to tolerate a wider versioning, or it'll never
> get run in practice.
> 
> I have some reluctance to doing this, because pylint and mypy change so
> frequently that I don't want "make check" to fail spuriously in the future.
> 
> (In practice, these failures occur 100% of the time when I am on vacation.)

So we seem to agree that it's something that we do expect to fail from
time to time. Maybe this is how I could express my point better: If it's
a hard failure, it should fail as early as possible - i.e. ideally
before the developer sends a patch, but certainly before failing a pull
request.

This allows another option that would work for me (but probably not for
you): Don't make it a hard failure. In practice, it would probably mean
that you end up fixing up things after people like we sometimes have to
do for the non-auto iotests.

> That said ... maybe I can add a controlled venv version of "check-python"
> and just have a --disable-check-python or something that spec files can opt
> into. Maybe that will work well enough?
> 
> i.e. maybe configure can check for the presence of pip, the python venv
> module (debian doesnt ship it standard...), and PyPI connectivity and if
> so, enables the test. Otherwise, we skip it.

I think this should work. If detecting the right environment is hard, I
don't think there is even a requirement to do so. You can make
--enable-check-python the default and if people don't want it, they can
explicitly disable it. (I understand that until you run 'make check', it
doesn't make a difference anyway, so pure users would never have to
change the option, right?)

> Got it. I'll see what I can come up with that checks the boxes for
> everyone, thanks for clarifying yours.
> 
> I want to make everything "just work" but I'm also afraid of writing too
> much magic crap that could break and frustrate people who have no desire to
> understand python packaging junk, so I'm trying to balance that.

Yes, sounds like we need to find some balance there. Test infrastructure
breaking locally for no obvious reason can be quite frustrating. But
sending a patch and getting it queued, only to be notified that it's
dropped again because of a mypy problem two weeks later when the
maintainer sends the pull request, can be equally (if not even more)
frustrating.

Kevin