[RFC PATCH] gitlab-ci: Switch the 'check-patch' & 'check-dco' jobs to use python-container

[RFC PATCH] gitlab-ci: Switch the 'check-patch' & 'check-dco' jobs to use python-container

[Thomas Huth]

The 'check-patch' and 'check-dco' jobs only need Python and git for
checking the patches, so it's not really necessary to use a container
here that has all the other build dependencies installed. By installing
"git" in the python container, we can use this light-weight container
for these jobs instead.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 Mark as RFC since I'm not sure whether we want to have "git" in
 the python container or not?

 .gitlab-ci.d/static_checks.yml         | 8 ++++----
 tests/docker/dockerfiles/python.docker | 1 +
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
index 5e955540d3..0e080bd0a0 100644
--- a/.gitlab-ci.d/static_checks.yml
+++ b/.gitlab-ci.d/static_checks.yml
@@ -1,8 +1,8 @@
 check-patch:
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
   needs:
-    job: amd64-centos8-container
+    job: python-container
   script:
     - .gitlab-ci.d/check-patch.py
   variables:
@@ -15,9 +15,9 @@ check-patch:
 
 check-dco:
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
   needs:
-    job: amd64-centos8-container
+    job: python-container
   script: .gitlab-ci.d/check-dco.py
   variables:
     GIT_DEPTH: 1000
diff --git a/tests/docker/dockerfiles/python.docker b/tests/docker/dockerfiles/python.docker
index 56d88417df..b2fb3a306d 100644
--- a/tests/docker/dockerfiles/python.docker
+++ b/tests/docker/dockerfiles/python.docker
@@ -6,6 +6,7 @@ MAINTAINER John Snow <jsnow@redhat.com>
 # Please keep this list sorted alphabetically
 ENV PACKAGES \
     gcc \
+    git \
     make \
     pipenv \
     python3 \
-- 
2.27.0

Re: [RFC PATCH] gitlab-ci: Switch the 'check-patch' & 'check-dco' jobs to use python-container

[Daniel P. Berrangé]

On Tue, May 03, 2022 at 10:36:21PM +0200, Thomas Huth wrote:
> The 'check-patch' and 'check-dco' jobs only need Python and git for
> checking the patches, so it's not really necessary to use a container
> here that has all the other build dependencies installed. By installing
> "git" in the python container, we can use this light-weight container
> for these jobs instead.

Our python container is far from light-weight....

$ time podman pull registry.gitlab.com/qemu-project/qemu/qemu/python

real	1m52.717s
user	1m32.327s
sys	0m19.453s

vs 

$ time podman pull python:3.8-alpine

real	0m4.509s
user	0m3.780s
sys	0m1.052s


It is quicker to use the alpine python container and then just
install 'git' on every job, than it is to use the pre-built
qemu python container

> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  Mark as RFC since I'm not sure whether we want to have "git" in
>  the python container or not?
> 
>  .gitlab-ci.d/static_checks.yml         | 8 ++++----
>  tests/docker/dockerfiles/python.docker | 1 +
>  2 files changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
> index 5e955540d3..0e080bd0a0 100644
> --- a/.gitlab-ci.d/static_checks.yml
> +++ b/.gitlab-ci.d/static_checks.yml
> @@ -1,8 +1,8 @@
>  check-patch:
>    stage: build
> -  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> +  image: $CI_REGISTRY_IMAGE/qemu/python:latest
>    needs:
> -    job: amd64-centos8-container
> +    job: python-container
>    script:
>      - .gitlab-ci.d/check-patch.py
>    variables:
> @@ -15,9 +15,9 @@ check-patch:
>  
>  check-dco:
>    stage: build
> -  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> +  image: $CI_REGISTRY_IMAGE/qemu/python:latest
>    needs:
> -    job: amd64-centos8-container
> +    job: python-container
>    script: .gitlab-ci.d/check-dco.py
>    variables:
>      GIT_DEPTH: 1000

IOW this is sufficient:

  image: python:3.8-alpine
  needs: []
  before_script:
    - apk update
    - apk add git
  script: ./gitlab-ci.d/check-dco.py


I expect the same would work for check-patch.py container

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] gitlab-ci: Switch the 'check-patch' & 'check-dco' jobs to use python-container

[Thomas Huth]

On 04/05/2022 11.12, Daniel P. Berrangé wrote:
> On Tue, May 03, 2022 at 10:36:21PM +0200, Thomas Huth wrote:
>> The 'check-patch' and 'check-dco' jobs only need Python and git for
>> checking the patches, so it's not really necessary to use a container
>> here that has all the other build dependencies installed. By installing
>> "git" in the python container, we can use this light-weight container
>> for these jobs instead.
> 
> Our python container is far from light-weight....
> 
> $ time podman pull registry.gitlab.com/qemu-project/qemu/qemu/python
> 
> real	1m52.717s
> user	1m32.327s
> sys	0m19.453s
> 
> vs
> 
> $ time podman pull python:3.8-alpine
> 
> real	0m4.509s
> user	0m3.780s
> sys	0m1.052s
> 
> 
> It is quicker to use the alpine python container and then just
> install 'git' on every job, than it is to use the pre-built
> qemu python container
> 
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   Mark as RFC since I'm not sure whether we want to have "git" in
>>   the python container or not?
>>
>>   .gitlab-ci.d/static_checks.yml         | 8 ++++----
>>   tests/docker/dockerfiles/python.docker | 1 +
>>   2 files changed, 5 insertions(+), 4 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
>> index 5e955540d3..0e080bd0a0 100644
>> --- a/.gitlab-ci.d/static_checks.yml
>> +++ b/.gitlab-ci.d/static_checks.yml
>> @@ -1,8 +1,8 @@
>>   check-patch:
>>     stage: build
>> -  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>> +  image: $CI_REGISTRY_IMAGE/qemu/python:latest
>>     needs:
>> -    job: amd64-centos8-container
>> +    job: python-container
>>     script:
>>       - .gitlab-ci.d/check-patch.py
>>     variables:
>> @@ -15,9 +15,9 @@ check-patch:
>>   
>>   check-dco:
>>     stage: build
>> -  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>> +  image: $CI_REGISTRY_IMAGE/qemu/python:latest
>>     needs:
>> -    job: amd64-centos8-container
>> +    job: python-container
>>     script: .gitlab-ci.d/check-dco.py
>>     variables:
>>       GIT_DEPTH: 1000
> 
> IOW this is sufficient:
> 
>    image: python:3.8-alpine
>    needs: []
>    before_script:
>      - apk update
>      - apk add git
>    script: ./gitlab-ci.d/check-dco.py
> 
> 
> I expect the same would work for check-patch.py container

... or would it make sense to switch tests/docker/dockerfiles/python.docker 
to use alpine instead of fedora?

  Thomas

Re: [RFC PATCH] gitlab-ci: Switch the 'check-patch' & 'check-dco' jobs to use python-container

[Daniel P. Berrangé]

On Wed, May 04, 2022 at 11:18:30AM +0200, Thomas Huth wrote:
> On 04/05/2022 11.12, Daniel P. Berrangé wrote:
> > On Tue, May 03, 2022 at 10:36:21PM +0200, Thomas Huth wrote:
> > > The 'check-patch' and 'check-dco' jobs only need Python and git for
> > > checking the patches, so it's not really necessary to use a container
> > > here that has all the other build dependencies installed. By installing
> > > "git" in the python container, we can use this light-weight container
> > > for these jobs instead.
> > 
> > Our python container is far from light-weight....
> > 
> > $ time podman pull registry.gitlab.com/qemu-project/qemu/qemu/python
> > 
> > real	1m52.717s
> > user	1m32.327s
> > sys	0m19.453s
> > 
> > vs
> > 
> > $ time podman pull python:3.8-alpine
> > 
> > real	0m4.509s
> > user	0m3.780s
> > sys	0m1.052s
> > 
> > 
> > It is quicker to use the alpine python container and then just
> > install 'git' on every job, than it is to use the pre-built
> > qemu python container
> > 
> > > 
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > >   Mark as RFC since I'm not sure whether we want to have "git" in
> > >   the python container or not?
> > > 
> > >   .gitlab-ci.d/static_checks.yml         | 8 ++++----
> > >   tests/docker/dockerfiles/python.docker | 1 +
> > >   2 files changed, 5 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
> > > index 5e955540d3..0e080bd0a0 100644
> > > --- a/.gitlab-ci.d/static_checks.yml
> > > +++ b/.gitlab-ci.d/static_checks.yml
> > > @@ -1,8 +1,8 @@
> > >   check-patch:
> > >     stage: build
> > > -  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> > > +  image: $CI_REGISTRY_IMAGE/qemu/python:latest
> > >     needs:
> > > -    job: amd64-centos8-container
> > > +    job: python-container
> > >     script:
> > >       - .gitlab-ci.d/check-patch.py
> > >     variables:
> > > @@ -15,9 +15,9 @@ check-patch:
> > >   check-dco:
> > >     stage: build
> > > -  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> > > +  image: $CI_REGISTRY_IMAGE/qemu/python:latest
> > >     needs:
> > > -    job: amd64-centos8-container
> > > +    job: python-container
> > >     script: .gitlab-ci.d/check-dco.py
> > >     variables:
> > >       GIT_DEPTH: 1000
> > 
> > IOW this is sufficient:
> > 
> >    image: python:3.8-alpine
> >    needs: []
> >    before_script:
> >      - apk update
> >      - apk add git
> >    script: ./gitlab-ci.d/check-dco.py
> > 
> > 
> > I expect the same would work for check-patch.py container
> 
> ... or would it make sense to switch tests/docker/dockerfiles/python.docker
> to use alpine instead of fedora?

That could work too

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] gitlab-ci: Switch the 'check-patch' & 'check-dco' jobs to use python-container

[Thomas Huth]

On 04/05/2022 11.20, Daniel P. Berrangé wrote:
> On Wed, May 04, 2022 at 11:18:30AM +0200, Thomas Huth wrote:
>> On 04/05/2022 11.12, Daniel P. Berrangé wrote:
>>> On Tue, May 03, 2022 at 10:36:21PM +0200, Thomas Huth wrote:
>>>> The 'check-patch' and 'check-dco' jobs only need Python and git for
>>>> checking the patches, so it's not really necessary to use a container
>>>> here that has all the other build dependencies installed. By installing
>>>> "git" in the python container, we can use this light-weight container
>>>> for these jobs instead.
>>>
>>> Our python container is far from light-weight....
>>>
>>> $ time podman pull registry.gitlab.com/qemu-project/qemu/qemu/python
>>>
>>> real	1m52.717s
>>> user	1m32.327s
>>> sys	0m19.453s
>>>
>>> vs
>>>
>>> $ time podman pull python:3.8-alpine
>>>
>>> real	0m4.509s
>>> user	0m3.780s
>>> sys	0m1.052s
>>>
>>>
>>> It is quicker to use the alpine python container and then just
>>> install 'git' on every job, than it is to use the pre-built
>>> qemu python container
>>>
>>>>
>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>> ---
>>>>    Mark as RFC since I'm not sure whether we want to have "git" in
>>>>    the python container or not?
>>>>
>>>>    .gitlab-ci.d/static_checks.yml         | 8 ++++----
>>>>    tests/docker/dockerfiles/python.docker | 1 +
>>>>    2 files changed, 5 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
>>>> index 5e955540d3..0e080bd0a0 100644
>>>> --- a/.gitlab-ci.d/static_checks.yml
>>>> +++ b/.gitlab-ci.d/static_checks.yml
>>>> @@ -1,8 +1,8 @@
>>>>    check-patch:
>>>>      stage: build
>>>> -  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>>>> +  image: $CI_REGISTRY_IMAGE/qemu/python:latest
>>>>      needs:
>>>> -    job: amd64-centos8-container
>>>> +    job: python-container
>>>>      script:
>>>>        - .gitlab-ci.d/check-patch.py
>>>>      variables:
>>>> @@ -15,9 +15,9 @@ check-patch:
>>>>    check-dco:
>>>>      stage: build
>>>> -  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>>>> +  image: $CI_REGISTRY_IMAGE/qemu/python:latest
>>>>      needs:
>>>> -    job: amd64-centos8-container
>>>> +    job: python-container
>>>>      script: .gitlab-ci.d/check-dco.py
>>>>      variables:
>>>>        GIT_DEPTH: 1000
>>>
>>> IOW this is sufficient:
>>>
>>>     image: python:3.8-alpine
>>>     needs: []
>>>     before_script:
>>>       - apk update
>>>       - apk add git
>>>     script: ./gitlab-ci.d/check-dco.py
>>>
>>>
>>> I expect the same would work for check-patch.py container
>>
>> ... or would it make sense to switch tests/docker/dockerfiles/python.docker
>> to use alpine instead of fedora?
> 
> That could work too

I just tried it, and it did not work out as expected: The 
check-python-pipenv job needs a more sophisticated environment than the 
alpine container can provide easily (e.g. it still needs python-3.6), so I 
updated my patch with your idea instead and used the python:alpine container 
as a base there now.

  Thomas