[PATCH] gitlab: force allow use of pip in Cirrus jobs

[PATCH] gitlab: force allow use of pip in Cirrus jobs

[Daniel P. Berrangé]

Python is transitioning to a world where you're not allowed to use 'pip
install' outside of a virutal env by default. The rationale is to stop
use of pip clashing with distro provided python packages, which creates
a major headache on distro upgrades.

All our CI environments, however, are 100% disposable so the upgrade
headaches don't exist. Thus we can undo the python defaults to allow
pip to work.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus/build.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

FYI lcitool recently added logic that purges the EXTERNALLY-MANAGED
marker in all the dockerfiles. QEMU will pick that up on the next
refresh, so Linux containers won't have trouble in future.

The macos/FreeBSD build env isn't fully managed by lcitool though,
so we need this manual addition to purge the marker that prevents
use of 'pip'.

This fixes the CI regression that just started hitting on Cirrus
CI macOS images, likely from homebrew python updates.

diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
index 29d55c4aa3..43dd52dd19 100644
--- a/.gitlab-ci.d/cirrus/build.yml
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -21,7 +21,7 @@ build_task:
   install_script:
     - @UPDATE_COMMAND@
     - @INSTALL_COMMAND@ @PKGS@
-    - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
+    - if test -n "@PYPI_PKGS@" ; then PYLIB=$(@PYTHON@ -c 'import sysconfig; print(sysconfig.get_path("stdlib"))'); rm -f $PYLIB/EXTERNALLY-MANAGED; @PIP3@ install @PYPI_PKGS@ ; fi
   clone_script:
     - git clone --depth 100 "$CI_REPOSITORY_URL" .
     - git fetch origin "$CI_COMMIT_REF_NAME"
-- 
2.43.0

Re: [PATCH] gitlab: force allow use of pip in Cirrus jobs

[Philippe Mathieu-Daudé]

On 22/2/24 12:40, Daniel P. Berrangé wrote:
> Python is transitioning to a world where you're not allowed to use 'pip
> install' outside of a virutal env by default. The rationale is to stop
> use of pip clashing with distro provided python packages, which creates
> a major headache on distro upgrades.
> 
> All our CI environments, however, are 100% disposable so the upgrade
> headaches don't exist. Thus we can undo the python defaults to allow
> pip to work.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus/build.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)

Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH] gitlab: force allow use of pip in Cirrus jobs

[Thomas Huth]

On 22/02/2024 12.40, Daniel P. Berrangé wrote:
> Python is transitioning to a world where you're not allowed to use 'pip
> install' outside of a virutal env by default. The rationale is to stop
> use of pip clashing with distro provided python packages, which creates
> a major headache on distro upgrades.
> 
> All our CI environments, however, are 100% disposable so the upgrade
> headaches don't exist. Thus we can undo the python defaults to allow
> pip to work.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus/build.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> FYI lcitool recently added logic that purges the EXTERNALLY-MANAGED
> marker in all the dockerfiles. QEMU will pick that up on the next
> refresh, so Linux containers won't have trouble in future.
> 
> The macos/FreeBSD build env isn't fully managed by lcitool though,
> so we need this manual addition to purge the marker that prevents
> use of 'pip'.
> 
> This fixes the CI regression that just started hitting on Cirrus
> CI macOS images, likely from homebrew python updates.
> 
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> index 29d55c4aa3..43dd52dd19 100644
> --- a/.gitlab-ci.d/cirrus/build.yml
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -21,7 +21,7 @@ build_task:
>     install_script:
>       - @UPDATE_COMMAND@
>       - @INSTALL_COMMAND@ @PKGS@
> -    - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
> +    - if test -n "@PYPI_PKGS@" ; then PYLIB=$(@PYTHON@ -c 'import sysconfig; print(sysconfig.get_path("stdlib"))'); rm -f $PYLIB/EXTERNALLY-MANAGED; @PIP3@ install @PYPI_PKGS@ ; fi
>     clone_script:
>       - git clone --depth 100 "$CI_REPOSITORY_URL" .
>       - git fetch origin "$CI_COMMIT_REF_NAME"

Thanks, that works for me:

  https://gitlab.com/thuth/qemu/-/jobs/6233183719

Tested-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH] gitlab: force allow use of pip in Cirrus jobs

[Peter Maydell]

On Thu, 22 Feb 2024 at 11:41, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> Python is transitioning to a world where you're not allowed to use 'pip
> install' outside of a virutal env by default. The rationale is to stop
> use of pip clashing with distro provided python packages, which creates
> a major headache on distro upgrades.
>
> All our CI environments, however, are 100% disposable so the upgrade
> headaches don't exist. Thus we can undo the python defaults to allow
> pip to work.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/cirrus/build.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> FYI lcitool recently added logic that purges the EXTERNALLY-MANAGED
> marker in all the dockerfiles. QEMU will pick that up on the next
> refresh, so Linux containers won't have trouble in future.
>
> The macos/FreeBSD build env isn't fully managed by lcitool though,
> so we need this manual addition to purge the marker that prevents
> use of 'pip'.
>
> This fixes the CI regression that just started hitting on Cirrus
> CI macOS images, likely from homebrew python updates.

I've applied this directly to get the CI back to green; thanks.

-- PMM