Help adding package to gitlab runners

Help adding package to gitlab runners

[Jag Raman]

Hi,

I'm wondering about the best way to install a dependency package on
GitLab runners.

I'm adding libvfio-user [1] as a submodule to QEMU. libvfio-user depends
on the availability of the "json-c" package. As such, we need to install this
package to build the library as part of QEMU.

I saw that we could add a "before_script" section to the jobs in
".gitlab-ci.d/buildtest.yml" - this section could have commands to install the
"json-c" package before running the job. However, this option is specific to
a job.

I wonder if there is a way to do this for all jobs. I saw that
"scripts/ci/setup/build-environment.yml" has a list of packages needed to
build QEMU - this one seems Ubuntu-specific.

How long does it take for changes made to this file (build-environment.yml)
to propagate? Are all QEMU's GitLab runners Ubuntu-based?

Thank you very much!
--
Jag

[1]: https://github.com/nutanix/libvfio-user

Re: Help adding package to gitlab runners

[Thomas Huth]

On 13/09/2021 03.14, Jag Raman wrote:
> Hi,
> 
> I'm wondering about the best way to install a dependency package on
> GitLab runners.
>
> I'm adding libvfio-user [1] as a submodule to QEMU. libvfio-user depends
> on the availability of the "json-c" package. As such, we need to install this
> package to build the library as part of QEMU.
> 
> I saw that we could add a "before_script" section to the jobs in
> ".gitlab-ci.d/buildtest.yml" - this section could have commands to install the
> "json-c" package before running the job. However, this option is specific to
> a job.
> 
> I wonder if there is a way to do this for all jobs. I saw that
> "scripts/ci/setup/build-environment.yml" has a list of packages needed to
> build QEMU - this one seems Ubuntu-specific.
> 
> How long does it take for changes made to this file (build-environment.yml)
> to propagate? Are all QEMU's GitLab runners Ubuntu-based?

No, this file (build-environment.yml) is only used to prepare the non-shared 
custom runners (see .gitlab-ci.d/custom-runners.yml), which only run on the 
staging branch of the qemu-project (or if you supply your own non-x86 runners).

I think in your case, you likely want to modify the shared runners instead. 
For this, we're using containers that are prepared in the first two stages 
of the CI pipelines. You can modify the container definitions in 
tests/docker/dockerfiles/. Once you add your package there, it should get 
used directly the next time you push a branch with the modification to gitlab.

  HTH,
   Thomas

Re: Help adding package to gitlab runners

[Jag Raman]

> On Sep 13, 2021, at 1:38 AM, Thomas Huth <thuth@redhat.com> wrote:
> 
> On 13/09/2021 03.14, Jag Raman wrote:
>> Hi,
>> I'm wondering about the best way to install a dependency package on
>> GitLab runners.
>> 
>> I'm adding libvfio-user [1] as a submodule to QEMU. libvfio-user depends
>> on the availability of the "json-c" package. As such, we need to install this
>> package to build the library as part of QEMU.
>> I saw that we could add a "before_script" section to the jobs in
>> ".gitlab-ci.d/buildtest.yml" - this section could have commands to install the
>> "json-c" package before running the job. However, this option is specific to
>> a job.
>> I wonder if there is a way to do this for all jobs. I saw that
>> "scripts/ci/setup/build-environment.yml" has a list of packages needed to
>> build QEMU - this one seems Ubuntu-specific.
>> How long does it take for changes made to this file (build-environment.yml)
>> to propagate? Are all QEMU's GitLab runners Ubuntu-based?
> 
> No, this file (build-environment.yml) is only used to prepare the non-shared custom runners (see .gitlab-ci.d/custom-runners.yml), which only run on the staging branch of the qemu-project (or if you supply your own non-x86 runners).
> 
> I think in your case, you likely want to modify the shared runners instead. For this, we're using containers that are prepared in the first two stages of the CI pipelines. You can modify the container definitions in tests/docker/dockerfiles/. Once you add your package there, it should get used directly the next time you push a branch with the modification to gitlab.

Thank you for confirming, Thomas! :)

--
Jag

> 
> HTH,
>  Thomas