[RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Thomas Huth]

Our distro support policy has been written with a best-effort
estimation of what users and developers need. However, as we now
know, the support for older long-term distributions can get really
troublesome for upstream development, since it is for example close
to impossible to keep the code for all Python versions maintained,
especially if upstream projects dropped support a longer time ago
already (Python 3.6 has been EOL at the end of 2021) while it is
still the main version of some long-term distros (CentOS/RHEL 8 and
openSUSE/SLES 15).
The QEMU project only has a limited amount of people working on
the development, so we just cannot afford of supporting both, very
old and the latest versions of our dependencies without burning
the few people who are working on those topics. So we *have* to
refine our support statement instead:

1) Once a new major version has been released, it should be enough
to limit the support for the previous major versions to one year
instead of two. One year should be enough time to get all people
who are interested in following the development of QEMU and who would
like to use the latest and greatest version of QEMU to upgrade their
system to the next major release of their distribution. All others
are likely happy with the old version of QEMU that is provided by
their distributor anyway and thus likely won't try to compile the
latest and greatest version of QEMU on their system.

2) For long-term distributions that release a new version only very
seldom, we limit the support to four years after the initial release.

Note: These changes mean that openSUSE is not considered as supported
anymore (since version 15.0 has been released in May 2018), and
RHEL/CentOS 8 will not be supported anymore in 3 months (since version
8.0 has been released in May 2019).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 docs/about/build-platforms.rst | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
index 1c1e7b9e11..cdc38f16a4 100644
--- a/docs/about/build-platforms.rst
+++ b/docs/about/build-platforms.rst
@@ -67,10 +67,11 @@ Non-supported architectures may be removed in the future following the
 Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
 -----------------------------------------
 
-The project aims to support the most recent major version at all times. Support
-for the previous major version will be dropped 2 years after the new major
-version is released or when the vendor itself drops support, whichever comes
-first. In this context, third-party efforts to extend the lifetime of a distro
+The project aims to support the most recent major version at all times for
+up to four years after its initial release. Support for the previous major
+version will be dropped one years after the new major version is released
+or when the vendor itself drops support, whichever comes first.
+In this context, third-party efforts to extend the lifetime of a distro
 are not considered, even when they are endorsed by the vendor (eg. Debian LTS);
 the same is true of repositories that contain packages backported from later
 releases (e.g. Debian backports). Within each major release, only the most
-- 
2.31.1

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Paolo Bonzini]

On 2/17/23 14:26, Thomas Huth wrote:
> Note: These changes mean that openSUSE is not considered as supported
> anymore (since version 15.0 has been released in May 2018), and
> RHEL/CentOS 8 will not be supported anymore in 3 months (since version
> 8.0 has been released in May 2019).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>

This has the advantage of being a very simple change to the support 
policy.  However, it has the disadvantage that at this point both SLE15 
and RHEL8 are not hard to support _at run-time_, only the build is a bit 
problematic; so, it kinda throws away the baby with the bathwater.

I have posted another RFC at 
https://lore.kernel.org/qemu-devel/20230217124150.205012-1-pbonzini@redhat.com; 
they share the 4 year deadline but it only applies to non-native 
dependencies (which means Python right now).

Thanks for posting this, it's useful to have two different possibilities 
to compare.

Paolo

> ---
>   docs/about/build-platforms.rst | 9 +++++----
>   1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 1c1e7b9e11..cdc38f16a4 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -67,10 +67,11 @@ Non-supported architectures may be removed in the future following the
>   Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
>   -----------------------------------------
>   
> -The project aims to support the most recent major version at all times. Support
> -for the previous major version will be dropped 2 years after the new major
> -version is released or when the vendor itself drops support, whichever comes
> -first. In this context, third-party efforts to extend the lifetime of a distro
> +The project aims to support the most recent major version at all times for
> +up to four years after its initial release. Support for the previous major
> +version will be dropped one years after the new major version is released
> +or when the vendor itself drops support, whichever comes first.
> +In this context, third-party efforts to extend the lifetime of a distro
>   are not considered, even when they are endorsed by the vendor (eg. Debian LTS);
>   the same is true of repositories that contain packages backported from later
>   releases (e.g. Debian backports). Within each major release, only the most

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Daniel P. Berrangé]

On Fri, Feb 17, 2023 at 03:44:23PM +0100, Paolo Bonzini wrote:
> On 2/17/23 14:26, Thomas Huth wrote:
> > Note: These changes mean that openSUSE is not considered as supported
> > anymore (since version 15.0 has been released in May 2018), and
> > RHEL/CentOS 8 will not be supported anymore in 3 months (since version
> > 8.0 has been released in May 2019).
> > 
> > Signed-off-by: Thomas Huth <thuth@redhat.com>
> 
> This has the advantage of being a very simple change to the support policy.
> However, it has the disadvantage that at this point both SLE15 and RHEL8 are
> not hard to support _at run-time_, only the build is a bit problematic; so,
> it kinda throws away the baby with the bathwater.

I think it could be a little fuzzy. I agree if thinking about
QEMU.git in isolation.

If we consider that python-qemu-qmp.git is conceptually positioned
as a general purpose QEMU python client, that would extend to runtime
support.

We could define a separate support policy for python-qemu-qmp.git,
but if we're intending to delete the in-tree python QMP code and
use python-qemu-qmp.git directly, then its support policy does
interact with qemu.git.

> I have posted another RFC at https://lore.kernel.org/qemu-devel/20230217124150.205012-1-pbonzini@redhat.com;
> they share the 4 year deadline but it only applies to non-native
> dependencies (which means Python right now).
> 
> Thanks for posting this, it's useful to have two different possibilities to
> compare.
> 
> Paolo
> 
> > ---
> >   docs/about/build-platforms.rst | 9 +++++----
> >   1 file changed, 5 insertions(+), 4 deletions(-)
> > 
> > diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> > index 1c1e7b9e11..cdc38f16a4 100644
> > --- a/docs/about/build-platforms.rst
> > +++ b/docs/about/build-platforms.rst
> > @@ -67,10 +67,11 @@ Non-supported architectures may be removed in the future following the
> >   Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
> >   -----------------------------------------
> > -The project aims to support the most recent major version at all times. Support
> > -for the previous major version will be dropped 2 years after the new major
> > -version is released or when the vendor itself drops support, whichever comes
> > -first. In this context, third-party efforts to extend the lifetime of a distro
> > +The project aims to support the most recent major version at all times for
> > +up to four years after its initial release. Support for the previous major
> > +version will be dropped one years after the new major version is released
> > +or when the vendor itself drops support, whichever comes first.
> > +In this context, third-party efforts to extend the lifetime of a distro
> >   are not considered, even when they are endorsed by the vendor (eg. Debian LTS);
> >   the same is true of repositories that contain packages backported from later
> >   releases (e.g. Debian backports). Within each major release, only the most
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Daniel P. Berrangé]

On Fri, Feb 17, 2023 at 02:26:31PM +0100, Thomas Huth wrote:
> Our distro support policy has been written with a best-effort
> estimation of what users and developers need. However, as we now
> know, the support for older long-term distributions can get really
> troublesome for upstream development, since it is for example close
> to impossible to keep the code for all Python versions maintained,
> especially if upstream projects dropped support a longer time ago
> already (Python 3.6 has been EOL at the end of 2021) while it is
> still the main version of some long-term distros (CentOS/RHEL 8 and
> openSUSE/SLES 15).
> The QEMU project only has a limited amount of people working on
> the development, so we just cannot afford of supporting both, very
> old and the latest versions of our dependencies without burning
> the few people who are working on those topics. So we *have* to
> refine our support statement instead:
> 
> 1) Once a new major version has been released, it should be enough
> to limit the support for the previous major versions to one year
> instead of two. One year should be enough time to get all people
> who are interested in following the development of QEMU and who would
> like to use the latest and greatest version of QEMU to upgrade their
> system to the next major release of their distribution. All others
> are likely happy with the old version of QEMU that is provided by
> their distributor anyway and thus likely won't try to compile the
> latest and greatest version of QEMU on their system.
> 
> 2) For long-term distributions that release a new version only very
> seldom, we limit the support to four years after the initial release.
> 
> Note: These changes mean that openSUSE is not considered as supported
> anymore (since version 15.0 has been released in May 2018), and
> RHEL/CentOS 8 will not be supported anymore in 3 months (since version
> 8.0 has been released in May 2019).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  docs/about/build-platforms.rst | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 1c1e7b9e11..cdc38f16a4 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -67,10 +67,11 @@ Non-supported architectures may be removed in the future following the
>  Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
>  -----------------------------------------
>  
> -The project aims to support the most recent major version at all times. Support
> -for the previous major version will be dropped 2 years after the new major
> -version is released or when the vendor itself drops support, whichever comes
> -first. In this context, third-party efforts to extend the lifetime of a distro
> +The project aims to support the most recent major version at all times for
> +up to four years after its initial release. Support for the previous major
> +version will be dropped one years after the new major version is released
> +or when the vendor itself drops support, whichever comes first.


So this change alters the support policy for every dependancy we care
about. IOW, it opens the door to dropping compilers, and many native
libraries too, in addition to the python modules which are the pressing
problem.

I'm not so comfortable with doing the broader approach. RHEL-8 deployments
are still overwhealmingly dominant over RHEL-9, because the kind of people
who deploy enterprise distros don't rush to upgrade to new versions, but
they are interesting as a source of contributors to QEMU.

I'm also not so comfortable dropping the only version of SLES that we
explicitly target, when we don't know when their new major release
will arrive.

If we allow compilers, libraries to be bumped, then someone stuck on
RHEL-8 has a significant task to build their own toolchain/libraries
in order to work with QEMU still. If we only allow python modules to
be bumped, the solution is just a pip install / virtualenv away.

IOW, the cost/benefit tradeoff for bumping python components is
much more favourable that the tradeoff for bumping native components.

> +In this context, third-party efforts to extend the lifetime of a distro
>  are not considered, even when they are endorsed by the vendor (eg. Debian LTS);
>  the same is true of repositories that contain packages backported from later
>  releases (e.g. Debian backports). Within each major release, only the most

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Thomas Huth]

On 17/02/2023 16.06, Daniel P. Berrangé wrote:
> On Fri, Feb 17, 2023 at 02:26:31PM +0100, Thomas Huth wrote:
...
> I'm also not so comfortable dropping the only version of SLES that we
> explicitly target, when we don't know when their new major release
> will arrive.

Let's hope that the next major version will show up at least five years 
after the previous one ... but what if it takes many more years? Do we want 
to support very old long term distros for "almost forever"?

Also, should we maybe at least limit the time to 5 years? Otherwise, if 
openSUSE 16 gets released 5 years after v15, we have to support v15 for 7 
years in total due to the "two more years" rule...

> If we allow compilers, libraries to be bumped, then someone stuck on
> RHEL-8 has a significant task to build their own toolchain/libraries
> in order to work with QEMU still. If we only allow python modules to
> be bumped, the solution is just a pip install / virtualenv away.

Honestly, being a Python ignorant, I'm more comfortable with "./configure && 
make && make install" instead of messing up my system with pip like I did in 
the past ... but I guess it's ok if it is properly done automatically under 
the hood with a venv ... I'll get used to it ;-)

  Thomas

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Markus Armbruster]

Thomas Huth <thuth@redhat.com> writes:

> On 17/02/2023 16.06, Daniel P. Berrangé wrote:
>> On Fri, Feb 17, 2023 at 02:26:31PM +0100, Thomas Huth wrote:
> ...
>> I'm also not so comfortable dropping the only version of SLES that we
>> explicitly target, when we don't know when their new major release
>> will arrive.
>
> Let's hope that the next major version will show up at least five
> years after the previous one ... but what if it takes many more years?
> Do we want to support very old long term distros for "almost forever"?
>
> Also, should we maybe at least limit the time to 5 years? Otherwise,
> if openSUSE 16 gets released 5 years after v15, we have to support v15
> for 7 years in total due to the "two more years" rule...

Let's keep in mind that "2 years after the new major version" isn't the
law, it's a rule we give ourselves so we don't have to debate from first
principles every time.  If supporting a major version of SLES (or
anything for that matter) according to the rule becomes too expensive,
we can and should change the rule.

>> If we allow compilers, libraries to be bumped, then someone stuck on
>> RHEL-8 has a significant task to build their own toolchain/libraries
>> in order to work with QEMU still. If we only allow python modules to
>> be bumped, the solution is just a pip install / virtualenv away.
>
> Honestly, being a Python ignorant, I'm more comfortable with
> "./configure && make && make install" instead of messing up my system
> with pip like I did in the past ... but I guess it's ok if it is
> properly done automatically under the hood with a venv ... I'll get
> used to it ;-)

"pip in venv" should not mess up your system.

Automation is of course welcome, and likely to reduce mess-ups through
incorrect use / non-use of venvs.

Optional dependencies default to off when not satisfied.  Without
automation, satisfying them is up to the developer.  When you're not
hacking on Python code yourself, there is no need for you to satisfy the
mypy dependency, be it with dnf, pip, or whatever.

I believe the one Python-related dependency of general interest right
now is Sphinx, which is required for building documentation.

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Markus Armbruster]

Thomas Huth <thuth@redhat.com> writes:

> Our distro support policy has been written with a best-effort
> estimation of what users and developers need. However, as we now
> know, the support for older long-term distributions can get really
> troublesome for upstream development, since it is for example close
> to impossible to keep the code for all Python versions maintained,
> especially if upstream projects dropped support a longer time ago
> already (Python 3.6 has been EOL at the end of 2021) while it is
> still the main version of some long-term distros (CentOS/RHEL 8 and
> openSUSE/SLES 15).
> The QEMU project only has a limited amount of people working on
> the development, so we just cannot afford of supporting both, very
> old and the latest versions of our dependencies without burning
> the few people who are working on those topics. So we *have* to
> refine our support statement instead:
>
> 1) Once a new major version has been released, it should be enough
> to limit the support for the previous major versions to one year
> instead of two. One year should be enough time to get all people
> who are interested in following the development of QEMU and who would
> like to use the latest and greatest version of QEMU to upgrade their
> system to the next major release of their distribution. All others
> are likely happy with the old version of QEMU that is provided by
> their distributor anyway and thus likely won't try to compile the
> latest and greatest version of QEMU on their system.
>
> 2) For long-term distributions that release a new version only very
> seldom, we limit the support to four years after the initial release.
>
> Note: These changes mean that openSUSE is not considered as supported
> anymore (since version 15.0 has been released in May 2018), and
> RHEL/CentOS 8 will not be supported anymore in 3 months (since version
> 8.0 has been released in May 2019).
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  docs/about/build-platforms.rst | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 1c1e7b9e11..cdc38f16a4 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -67,10 +67,11 @@ Non-supported architectures may be removed in the future following the
>  Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
>  -----------------------------------------
>  
> -The project aims to support the most recent major version at all times. Support
> -for the previous major version will be dropped 2 years after the new major
> -version is released or when the vendor itself drops support, whichever comes
> -first. In this context, third-party efforts to extend the lifetime of a distro
> +The project aims to support the most recent major version at all times for
> +up to four years after its initial release. Support for the previous major
> +version will be dropped one years after the new major version is released

"one year" (singular)

Suggest "the next major version".

> +or when the vendor itself drops support, whichever comes first.
> +In this context, third-party efforts to extend the lifetime of a distro
>  are not considered, even when they are endorsed by the vendor (eg. Debian LTS);
>  the same is true of repositories that contain packages backported from later
>  releases (e.g. Debian backports). Within each major release, only the most

If we take Paolo's "[RFC PATCH] docs: build-platforms: refine
requirements on Python build dependencies", then the rationale for this
one becomes weaker.  But I believe it's well worth serious consideration
all the same.

Why would we *not* want to do what Thomas proposes?

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Daniel P. Berrangé]

On Fri, Feb 17, 2023 at 04:55:49PM +0100, Markus Armbruster wrote:
> Thomas Huth <thuth@redhat.com> writes:
> 
> > Our distro support policy has been written with a best-effort
> > estimation of what users and developers need. However, as we now
> > know, the support for older long-term distributions can get really
> > troublesome for upstream development, since it is for example close
> > to impossible to keep the code for all Python versions maintained,
> > especially if upstream projects dropped support a longer time ago
> > already (Python 3.6 has been EOL at the end of 2021) while it is
> > still the main version of some long-term distros (CentOS/RHEL 8 and
> > openSUSE/SLES 15).
> > The QEMU project only has a limited amount of people working on
> > the development, so we just cannot afford of supporting both, very
> > old and the latest versions of our dependencies without burning
> > the few people who are working on those topics. So we *have* to
> > refine our support statement instead:
> >
> > 1) Once a new major version has been released, it should be enough
> > to limit the support for the previous major versions to one year
> > instead of two. One year should be enough time to get all people
> > who are interested in following the development of QEMU and who would
> > like to use the latest and greatest version of QEMU to upgrade their
> > system to the next major release of their distribution. All others
> > are likely happy with the old version of QEMU that is provided by
> > their distributor anyway and thus likely won't try to compile the
> > latest and greatest version of QEMU on their system.
> >
> > 2) For long-term distributions that release a new version only very
> > seldom, we limit the support to four years after the initial release.
> >
> > Note: These changes mean that openSUSE is not considered as supported
> > anymore (since version 15.0 has been released in May 2018), and
> > RHEL/CentOS 8 will not be supported anymore in 3 months (since version
> > 8.0 has been released in May 2019).
> >
> > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > ---
> >  docs/about/build-platforms.rst | 9 +++++----
> >  1 file changed, 5 insertions(+), 4 deletions(-)
> >
> > diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> > index 1c1e7b9e11..cdc38f16a4 100644
> > --- a/docs/about/build-platforms.rst
> > +++ b/docs/about/build-platforms.rst
> > @@ -67,10 +67,11 @@ Non-supported architectures may be removed in the future following the
> >  Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
> >  -----------------------------------------
> >  
> > -The project aims to support the most recent major version at all times. Support
> > -for the previous major version will be dropped 2 years after the new major
> > -version is released or when the vendor itself drops support, whichever comes
> > -first. In this context, third-party efforts to extend the lifetime of a distro
> > +The project aims to support the most recent major version at all times for
> > +up to four years after its initial release. Support for the previous major
> > +version will be dropped one years after the new major version is released
> 
> "one year" (singular)
> 
> Suggest "the next major version".
> 
> > +or when the vendor itself drops support, whichever comes first.
> > +In this context, third-party efforts to extend the lifetime of a distro
> >  are not considered, even when they are endorsed by the vendor (eg. Debian LTS);
> >  the same is true of repositories that contain packages backported from later
> >  releases (e.g. Debian backports). Within each major release, only the most
> 
> If we take Paolo's "[RFC PATCH] docs: build-platforms: refine
> requirements on Python build dependencies", then the rationale for this
> one becomes weaker.  But I believe it's well worth serious consideration
> all the same.
> 
> Why would we *not* want to do what Thomas proposes?

I think my response covers the reasons why not. In summary

The cost/benefit tradeoff of mandating a new python runtime is
quite favourable, as the the distro vendor still provids the
python runtime, and so only cost is pip installation which is
largely eliminated if we have QEMU auto-create a virtualenv.

The cost/benefit tradeoff of dropping the platforms entirely
is not obviously favourable when we don't have clear demand
to bump the min versions of native packages, and the cost to
users stuck on these platforms to build their own toolchain
or libraries is very high.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Thomas Huth]

On 17/02/2023 16.59, Daniel P. Berrangé wrote:
> On Fri, Feb 17, 2023 at 04:55:49PM +0100, Markus Armbruster wrote:
....
> The cost/benefit tradeoff of dropping the platforms entirely
> is not obviously favourable when we don't have clear demand
> to bump the min versions of native packages, and the cost to
> users stuck on these platforms to build their own toolchain
> or libraries is very high.

There's another urgent point which I completely forget to mention in my 
patch description (not sure how I managed that, since it's bugging me quite 
badly in the past weeks): We're struggling heavily with CI minutes. If we 
have to support multiple major releases for a long time in parallel, there 
will always be the desire to have all major releases also tested in the CI 
... and honestly, we're really struggling quite badly there right now - as 
you know, we've already run out of CI minutes in January in the main 
project, and also in my forked repo I'm struggling each month. Additionally, 
it's of course additional effort to keep everything in the "green" state the 
more you have to support.

We're currently "lucky" in a sense that we're only testing one version of 
CentOS, Debian and Ubuntu right now, but there have been voices in the past 
weeks asking for more already (like we also did in the past already). I'd 
really appreciate if we could have a clearer policy here to support less at 
the same time. It would help with the pressure on the CI and the effort and 
time it takes to maintain all that stuff.

  Thomas

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 2262 bytes --]

Il ven 17 feb 2023, 19:47 Thomas Huth <thuth@redhat.com> ha scritto:

> On 17/02/2023 16.59, Daniel P. Berrangé wrote:
> > On Fri, Feb 17, 2023 at 04:55:49PM +0100, Markus Armbruster wrote:
> ....
> > The cost/benefit tradeoff of dropping the platforms entirely
> > is not obviously favourable when we don't have clear demand
> > to bump the min versions of native packages, and the cost to
> > users stuck on these platforms to build their own toolchain
> > or libraries is very high.
>
> There's another urgent point which I completely forget to mention in my
> patch description (not sure how I managed that, since it's bugging me
> quite
> badly in the past weeks): We're struggling heavily with CI minutes.


The only viable solution for CI minutes is going to be private runners,
it's not easy to cut 30% of the jobs.

We're using less than half of our Azure sponsorship budget, and could also
find other sources; either Azure Kubernetes or AWS Fargate are pretty cheap
for running CI because unlike VM instances you pay for just the time that
CI is running (at least with Azure you still have VMs but they scale out
dynamically).

The complicated part is setting up the kubernetes executor for
gitlab-runner, but we'll find someone. :)

Paolo


If we
> have to support multiple major releases for a long time in parallel, there
> will always be the desire to have all major releases also tested in the CI
> ... and honestly, we're really struggling quite badly there right now - as
> you know, we've already run out of CI minutes in January in the main
> project, and also in my forked repo I'm struggling each month.
> Additionally,
> it's of course additional effort to keep everything in the "green" state
> the
> more you have to support.
>
> We're currently "lucky" in a sense that we're only testing one version of
> CentOS, Debian and Ubuntu right now, but there have been voices in the
> past
> weeks asking for more already (like we also did in the past already). I'd
> really appreciate if we could have a clearer policy here to support less
> at
> the same time. It would help with the pressure on the CI and the effort
> and
> time it takes to maintain all that stuff.
>
>   Thomas
>
>

[-- Attachment #2: Type: text/html, Size: 3068 bytes --]

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Il ven 17 feb 2023, 19:47 Thomas Huth <thuth@redhat.com> ha scritto:
>
>> On 17/02/2023 16.59, Daniel P. Berrangé wrote:
>> > On Fri, Feb 17, 2023 at 04:55:49PM +0100, Markus Armbruster wrote:
>> ....
>> > The cost/benefit tradeoff of dropping the platforms entirely
>> > is not obviously favourable when we don't have clear demand
>> > to bump the min versions of native packages, and the cost to
>> > users stuck on these platforms to build their own toolchain
>> > or libraries is very high.
>>
>> There's another urgent point which I completely forget to mention in my
>> patch description (not sure how I managed that, since it's bugging me
>> quite
>> badly in the past weeks): We're struggling heavily with CI minutes.
>
>
> The only viable solution for CI minutes is going to be private runners,
> it's not easy to cut 30% of the jobs.
>
> We're using less than half of our Azure sponsorship budget, and could also
> find other sources; either Azure Kubernetes or AWS Fargate are pretty cheap
> for running CI because unlike VM instances you pay for just the time that
> CI is running (at least with Azure you still have VMs but they scale out
> dynamically).

To anyone arguing for support of yet another host architecture / target
architecture / configuration / whatever: sponsoring its CI would
demonstrate seriousness :)

> The complicated part is setting up the kubernetes executor for
> gitlab-runner, but we'll find someone. :)

Re: [RFC PATCH] docs/about/build-platforms: Refine the distro support policy

[Daniel P. Berrangé]

On Fri, Feb 17, 2023 at 07:47:51PM +0100, Thomas Huth wrote:
> On 17/02/2023 16.59, Daniel P. Berrangé wrote:
> > On Fri, Feb 17, 2023 at 04:55:49PM +0100, Markus Armbruster wrote:
> ....
> > The cost/benefit tradeoff of dropping the platforms entirely
> > is not obviously favourable when we don't have clear demand
> > to bump the min versions of native packages, and the cost to
> > users stuck on these platforms to build their own toolchain
> > or libraries is very high.
> 
> There's another urgent point which I completely forget to mention in my
> patch description (not sure how I managed that, since it's bugging me quite
> badly in the past weeks): We're struggling heavily with CI minutes. If we
> have to support multiple major releases for a long time in parallel, there
> will always be the desire to have all major releases also tested in the CI
> ... and honestly, we're really struggling quite badly there right now - as
> you know, we've already run out of CI minutes in January in the main
> project, and also in my forked repo I'm struggling each month. Additionally,
> it's of course additional effort to keep everything in the "green" state the
> more you have to support.
> 
> We're currently "lucky" in a sense that we're only testing one version of
> CentOS, Debian and Ubuntu right now, but there have been voices in the past
> weeks asking for more already (like we also did in the past already). I'd
> really appreciate if we could have a clearer policy here to support less at
> the same time. It would help with the pressure on the CI and the effort and
> time it takes to maintain all that stuff.

Note, we have never equated CI coverage for a platform with the
technical support for a platform. We have sooooo many combinations
which are not tested and are supported from a technical POV. It
would be nice to have comprehensive testing, but we're unlikely
to ever achieve it. IOW, lack automated CI of testing is not a
reason to delete support for a platform, it just means that we'll
rely on manual testing, and users can't have such high confidence
in the platform.


One thing we've not explored is multiple levels of CI coverage.
We could degrade certain platforms to have periodic post-merge
testing, rather than pre-merge. eg cut down gating platform
combos, but then once a week run a broader CI pipeline. This
would allow regressions to creep in periodically for more obscure
combinations, but still give us an alert so we can fix them before
release.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|