[PATCH] ci: fix Python dependency on Ubuntu 24.04

[PATCH] ci: fix Python dependency on Ubuntu 24.04

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2019 bytes --]

Newer versions of Ubuntu have dropped Python 2 starting with Ubuntu
23.04. By default though, our CI setups will try to use that Python
version on all Ubuntu-based jobs except for the "linux-gcc" one.

We didn't notice this issue due to two reasons:

  - The "ubuntu:latest" tag always points to the latest LTS release.
    Until a few weeks ago this was Ubuntu 22.04, which still had Python
    2.

  - Our Docker-based CI jobs had their own script to install
    dependencies until 9cdeb34b96 (ci: merge scripts which install
    dependencies, 2024-04-12), where we didn't even try to install
    Python at all for many of them.

Since the CI refactorings have originally been implemented, Ubuntu
24.04 was released, and it being an LTS versions means that the "latest"
tag now points to that Python-2-less version. Consequently, those jobs
that use "ubuntu:latest" broke.

Address this by using Python 2 on Ubuntu 20.04, only, whereas we use
Python 3 on all other Ubuntu jobs. Eventually, we should think about
dropping support for Python 2 completely.

Reported-by: Justin Tobler <jtobler@gmail.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---

Note: this topic depends on ps/ci-test-with-jgit at 70b81fbf3c (t0612:
add tests to exercise Git/JGit reftable compatibility, 2024-04-12).

 ci/lib.sh | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/ci/lib.sh b/ci/lib.sh
index 473a2d0348..273f3540a6 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -325,9 +325,13 @@ ubuntu-*)
 		break
 	fi
 
-	PYTHON_PACKAGE=python2
-	if test "$jobname" = linux-gcc
+	# Python 2 is end of life, and Ubuntu 23.04 and newer don't actually
+	# have it anymore. We thus only test with Python 2 on older LTS
+	# releases.
+	if "$distro" = "ubuntu-20.04"
 	then
+		PYTHON_PACKAGE=python2
+	else
 		PYTHON_PACKAGE=python3
 	fi
 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/$PYTHON_PACKAGE"

base-commit: b6db6b1598946fbf777e55ff0d187b11ff3bd21f
-- 
2.45.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] ci: fix Python dependency on Ubuntu 24.04

[Justin Tobler]

On 24/05/06 07:35AM, Patrick Steinhardt wrote:
> Newer versions of Ubuntu have dropped Python 2 starting with Ubuntu
> 23.04. By default though, our CI setups will try to use that Python
> version on all Ubuntu-based jobs except for the "linux-gcc" one.

Naive question, why were the "linux-gcc" jobs the only ones using
Python 3?

> 
> We didn't notice this issue due to two reasons:
> 
>   - The "ubuntu:latest" tag always points to the latest LTS release.
>     Until a few weeks ago this was Ubuntu 22.04, which still had Python
>     2.
> 
>   - Our Docker-based CI jobs had their own script to install
>     dependencies until 9cdeb34b96 (ci: merge scripts which install
>     dependencies, 2024-04-12), where we didn't even try to install
>     Python at all for many of them.
> 
> Since the CI refactorings have originally been implemented, Ubuntu
> 24.04 was released, and it being an LTS versions means that the "latest"
> tag now points to that Python-2-less version. Consequently, those jobs
> that use "ubuntu:latest" broke.
> 
> Address this by using Python 2 on Ubuntu 20.04, only, whereas we use
> Python 3 on all other Ubuntu jobs. Eventually, we should think about
> dropping support for Python 2 completely.
> 
> Reported-by: Justin Tobler <jtobler@gmail.com>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
> 
> Note: this topic depends on ps/ci-test-with-jgit at 70b81fbf3c (t0612:
> add tests to exercise Git/JGit reftable compatibility, 2024-04-12).
> 
>  ci/lib.sh | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/ci/lib.sh b/ci/lib.sh
> index 473a2d0348..273f3540a6 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -325,9 +325,13 @@ ubuntu-*)
>  		break
>  	fi
>  
> -	PYTHON_PACKAGE=python2
> -	if test "$jobname" = linux-gcc

With this change, "linux-gcc" jobs using ubuntu 20.04 will now use
Python 2. Is that what we want?

-Justin

> +	# Python 2 is end of life, and Ubuntu 23.04 and newer don't actually
> +	# have it anymore. We thus only test with Python 2 on older LTS
> +	# releases.
> +	if "$distro" = "ubuntu-20.04"
>  	then
> +		PYTHON_PACKAGE=python2
> +	else
>  		PYTHON_PACKAGE=python3
>  	fi
>  	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/$PYTHON_PACKAGE"
> 
> base-commit: b6db6b1598946fbf777e55ff0d187b11ff3bd21f
> -- 
> 2.45.0
> 

Re: [PATCH] ci: fix Python dependency on Ubuntu 24.04

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1762 bytes --]

On Mon, May 06, 2024 at 07:55:25AM -0500, Justin Tobler wrote:
> On 24/05/06 07:35AM, Patrick Steinhardt wrote:
> > Newer versions of Ubuntu have dropped Python 2 starting with Ubuntu
> > 23.04. By default though, our CI setups will try to use that Python
> > version on all Ubuntu-based jobs except for the "linux-gcc" one.
> 
> Naive question, why were the "linux-gcc" jobs the only ones using
> Python 3?

This has been introduced in 6bb40ed20a (ci: use python3 in linux-gcc and
osx-gcc and python2 elsewhere, 2020-01-23). It's been a first step
towards exercising Python 3 in our CI pipeline. The jobs were basically
picked at random, there is no inherent reason.

So ultimately, it does not matter which jobs use what. But what does
matter is whether the image even supports Python 2. And as we nowadays
do have a fair split of jobs using "ubuntu:20.04" and "ubuntu:latest",
I'd just use that the the criterium to pick the Python version.

Once we drop support for Ubuntu 20.04, we can then more forward and stop
supporting Python 2 altogether. Or maybe even earlier than that -- I do
not think it is reasonable to ask Git to maintain compatibility with a
project that is end of life already.

[snip]
> > diff --git a/ci/lib.sh b/ci/lib.sh
> > index 473a2d0348..273f3540a6 100755
> > --- a/ci/lib.sh
> > +++ b/ci/lib.sh
> > @@ -325,9 +325,13 @@ ubuntu-*)
> >  		break
> >  	fi
> >  
> > -	PYTHON_PACKAGE=python2
> > -	if test "$jobname" = linux-gcc
> 
> With this change, "linux-gcc" jobs using ubuntu 20.04 will now use
> Python 2. Is that what we want?

Well, as explained above, the selection was arbitrary in the first
place. Now it's less so as the choice is mandated by what is actually
supported.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] ci: fix Python dependency on Ubuntu 24.04

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> So ultimately, it does not matter which jobs use what. But what does
> matter is whether the image even supports Python 2. And as we nowadays
> do have a fair split of jobs using "ubuntu:20.04" and "ubuntu:latest",
> I'd just use that the the criterium to pick the Python version.
>
> Once we drop support for Ubuntu 20.04, we can then more forward and stop
> supporting Python 2 altogether. Or maybe even earlier than that -- I do
> not think it is reasonable to ask Git to maintain compatibility with a
> project that is end of life already.

Yup.  If an distro sticks to Python 2 for whatever reason, they may
want to donate and dedicate CI resources, but otherwise it is not
really our job to keep compatibility on our side.

>> With this change, "linux-gcc" jobs using ubuntu 20.04 will now use
>> Python 2. Is that what we want?
>
> Well, as explained above, the selection was arbitrary in the first
> place. Now it's less so as the choice is mandated by what is actually
> supported.

Yup, as long as there is one that supports Python 2 among the
distros we happen to use for our CIs, we can pick that one.
Otherwise we should drop.

Dropping Python 2 support does not have to wait for Git 3.0, I would
say.

Thanks.

Re: [PATCH] ci: fix Python dependency on Ubuntu 24.04

[Justin Tobler]

On 24/05/06 07:35AM, Patrick Steinhardt wrote:
> Newer versions of Ubuntu have dropped Python 2 starting with Ubuntu
> 23.04. By default though, our CI setups will try to use that Python
> version on all Ubuntu-based jobs except for the "linux-gcc" one.
> 
> We didn't notice this issue due to two reasons:
> 
>   - The "ubuntu:latest" tag always points to the latest LTS release.
>     Until a few weeks ago this was Ubuntu 22.04, which still had Python
>     2.
> 
>   - Our Docker-based CI jobs had their own script to install
>     dependencies until 9cdeb34b96 (ci: merge scripts which install
>     dependencies, 2024-04-12), where we didn't even try to install
>     Python at all for many of them.
> 
> Since the CI refactorings have originally been implemented, Ubuntu
> 24.04 was released, and it being an LTS versions means that the "latest"
> tag now points to that Python-2-less version. Consequently, those jobs
> that use "ubuntu:latest" broke.
> 
> Address this by using Python 2 on Ubuntu 20.04, only, whereas we use
> Python 3 on all other Ubuntu jobs. Eventually, we should think about
> dropping support for Python 2 completely.
> 
> Reported-by: Justin Tobler <jtobler@gmail.com>

Not a big deal, but the email is slightly off. Should be:
<jltobler@gmail.com>

Otherwise this patch looks good to me. :)

-Justin 

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
...

Re: [PATCH] ci: fix Python dependency on Ubuntu 24.04

[Junio C Hamano]

Justin Tobler <jltobler@gmail.com> writes:

>> Reported-by: Justin Tobler <jtobler@gmail.com>
>
> Not a big deal, but the email is slightly off. Should be:
> <jltobler@gmail.com>
>
> Otherwise this patch looks good to me. :)

The patch e-mail also went to that "slight off" address, which is
funny.  I've amended the Reported-by: line locally while queuing.

Thanks, both of you.

Re: [PATCH] ci: fix Python dependency on Ubuntu 24.04

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1653 bytes --]

On Mon, May 06, 2024 at 02:06:04PM -0500, Justin Tobler wrote:
> On 24/05/06 07:35AM, Patrick Steinhardt wrote:
> > Newer versions of Ubuntu have dropped Python 2 starting with Ubuntu
> > 23.04. By default though, our CI setups will try to use that Python
> > version on all Ubuntu-based jobs except for the "linux-gcc" one.
> > 
> > We didn't notice this issue due to two reasons:
> > 
> >   - The "ubuntu:latest" tag always points to the latest LTS release.
> >     Until a few weeks ago this was Ubuntu 22.04, which still had Python
> >     2.
> > 
> >   - Our Docker-based CI jobs had their own script to install
> >     dependencies until 9cdeb34b96 (ci: merge scripts which install
> >     dependencies, 2024-04-12), where we didn't even try to install
> >     Python at all for many of them.
> > 
> > Since the CI refactorings have originally been implemented, Ubuntu
> > 24.04 was released, and it being an LTS versions means that the "latest"
> > tag now points to that Python-2-less version. Consequently, those jobs
> > that use "ubuntu:latest" broke.
> > 
> > Address this by using Python 2 on Ubuntu 20.04, only, whereas we use
> > Python 3 on all other Ubuntu jobs. Eventually, we should think about
> > dropping support for Python 2 completely.
> > 
> > Reported-by: Justin Tobler <jtobler@gmail.com>
> 
> Not a big deal, but the email is slightly off. Should be:
> <jltobler@gmail.com>
> 
> Otherwise this patch looks good to me. :)

You know, every single time I copy over addresses. This one time I
didn't and of course I immediately mistype it. Well, lesson learned,
copy & paste it is.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]