Re: [PATCH] netmap: support git-submodule build otption

[Markus Armbruster <armbru@redhat.com>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Oct 07, 2019 at 01:39:30PM +0100, Peter Maydell wrote:
>> On Mon, 7 Oct 2019 at 13:36, Markus Armbruster <armbru@redhat.com> wrote:
>> > If CI of QEMU code isn't useful, then I suspect the QEMU code isn't
>> > useful, period.  Giuseppe assures us the netmap QEMU code *is* useful.
>> > It followe we better make sure our CI covers it.
>> 
>> It would be an interesting idea to have a requirement that
>> any new library dependency can't be introduced into QEMU
>> unless one of the systems we do builds on can be set up
>> so the new code is compiled...
>
> Being able to at least compile code successfully is a pretty low bar
> to cross in terms of CI, so I think that's reasonable in general.
>
> I would not stop in terms of libraries though. We should document our
> broader expectations for CI
>
> Compilation
>
>  - All new code must be compiled in one of[1] our CI systems.

I think we should hold old code to the same standard.  Anything that's
not compiled now either gets fixed or deprecated.  If it's still unfixed
at the end of the deprecation grace period, it goes.

>    This implies
>
>     - Libraries must be available in some OS we compile for

How do we track compliance?  It's not obvious to (ignorant) me what
features exactly each CI compile enables.  Without that, it's not
obvious whether any CI run enables use of a certain library.

>     - New host architectures must have cross compilers available

Native tool chain in CI also satisfies "must be compiled in CI".

>     - New OS distro targets must have VM test image support

Non-virtual host in CI also satisfies "must be compiled in CI".

> This is not far off what we unofficially expect already, so
> it shouldn't be too distruptive if we formally adopt it as a
> mandatory rule.

Feels like a no-brainer, to be honest.

> Testing
>
>  - All significant new features should have either unit or
>    functional or integration test coverage
>
>  ... something something something ...

Spot the weasel words!  ;-P

> We've not really set any expectations around CI beyond compile
> testing thus far. We've got a mix of unit testing & functional
> testing in the tests/ dir. We're increasingly getting stuff
> added there when major features are added. Making this mandatory
> is probably too large a step, but it is likely helpful if we
> at least set some recommendations / guidelines to push people
> in the direction we want to go longer term.

We've been pushing, but not evenly.  It's basically up to maintainers,
and their expectations on testing vary.

> Regards,
> Daniel
>
> [1] Having to say "one of" is not especially great. It would be very nice
>     to get to the point where we have either container images or VM images
>     and no matter what CI harness(es) we use, they always run with either
>     a container or VM image[2]. Even if we have bare metal machines available
>     we can still execute actual builds inside containers or VM images so
>     everyone uses a consistent environment for everything related to CI.
>
> [2] macOS is a problem/exception here given that we can't legally distribute
>     VM images, nor can we provide a cross compiler toolchain. For everything
>     else we can make VM/container images though.

That makes Macs a secondary host at best.  If it breaks, it breaks.  If
somebody fixes it, nice, if not, *shrug*.  Don't expect git-bisect to
work.