Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa <crosa@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 6601 bytes --]

On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote:
> On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote:
> > To have the jobs dispatched to custom runners, gitlab-runner must
> > be installed, active as a service and properly configured.  The
> > variables file and playbook introduced here should help with those
> > steps.
> > 
> > The playbook introduced here covers a number of different Linux
> > distributions and FreeBSD, and are intended to provide a reproducible
> > environment.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> 
> In general, there's been put quite some effort into the playbooks - sorry I'm
> late to the game - is there a plan to introduce QEMU as a project to lcitool?

I think it's becoming quite clear that having so much duplication (in
the dockerfiles, tests/vm, this playbook, etc) is costly and error
prone.  I don't know if anyone has invested time in a PoC to
consolidate those (with lcitool), but I can certainly see the upside
to that.  BTW, are you volunteering (wink wink)? :)

> We've taken care of most of the bits in the playbooks that are being introduced
> and for the remaining ones I think it would be that big of an overhaul to do
> the adjustments. One major re-factor though would IMO be to break the
> dependency lcitool has on the machine naming, kind of restricting it to a
> limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which
> makes it inconvenient to prepare physical hosts.
>

Right... I wasn't aware of that depedency.  And, this may be a nice
project to make sure that lcitool doesn't have any other libvirt
specificities.

> More comments inline...
> 
> >  docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
> >  scripts/ci/setup/.gitignore        |  1 +
> >  scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
> >  scripts/ci/setup/vars.yml.template | 13 ++++++
> >  4 files changed, 149 insertions(+)
> >  create mode 100644 scripts/ci/setup/.gitignore
> >  create mode 100644 scripts/ci/setup/gitlab-runner.yml
> >  create mode 100644 scripts/ci/setup/vars.yml.template
> > 
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 208b5e399b..a234a5e24c 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -84,3 +84,66 @@ To run the playbook, execute::
> >  
> >    cd scripts/ci/setup
> >    ansible-playbook -i inventory build-environment.yml
> > +
> > +gitlab-runner setup and registration
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +The gitlab-runner agent needs to be installed on each machine that
> > +will run jobs.  The association between a machine and a GitLab project
> > +happens with a registration token.  To find the registration token for
> > +your repository/project, navigate on GitLab's web UI to:
> > +
> > + * Settings (the gears like icon), then
> > + * CI/CD, then
> > + * Runners, and click on the "Expand" button, then
> > + * Under "Set up a specific Runner manually", look for the value under
> > +   "Use the following registration token during setup"
> > +
> > +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> > +``scripts/ci/setup/vars.yml``.  Then, set the
> > +``gitlab_runner_registration_token`` variable to the value obtained
> > +earlier.
> > +
> > +.. note:: gitlab-runner is not available from the standard location
> > +          for all OS and architectures combinations.  For some systems,
> > +          a custom build may be necessary.  Some builds are avaiable
> > +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> > +          URI may be used as a value on ``vars.yml``
> 
> Yes, this can be suboptimal...Would it make sense to fall back to build the
> binary of a given version from git as a fallback during this playbook if the
> necessary arch version isn't provided the official way? Just an idea, I'd like
> to avoid the need for you to become the maintainer of the binaries and keep up
> with the releases.
>

Well, building them during the playbook would be a lot more
complex... You can have your own repo with your own builds, and just
tweak your vars.yml.

> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory gitlab-runner.yml
> > +
> > +.. note:: there are currently limitations to gitlab-runner itself when
> > +          setting up a service under FreeBSD systems.  You will need to
> > +          perform steps 4 to 10 manually, as described at
> 
> Which one of them is considered an automation problem? In lcitool we made
> gitlab-runner completely automated on all distros, including FreeBSD:
>

It looks like lcitool went the more practical route.  I was hoping to
not have to treat gitlab-runner in such a special way in any
"supported" OS.  What I mean is, I'd rather write the code within
gitlab-runner (or reespective libraries).  Of course, I did not get to
it, so that's why I just documented the steps here.

I'll take a look at lcitool's playbook to see if I can easily
canibalize some of that.  But, at this time, we don't runners for
FreeBSD anyway, so I guess this is not *that* urgent.

> 4) log file permissions - you're creating the user, you can as well create the
> file with correct permissions
> 
> 5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml
> it will pull a bunch of dependencies which ensure the dir exists
> 
> 6) gitlab_runner service script should IMO provided by this automation as a
> template and install to the correct location
> 
> 7) ensure the service script is executable - template module can do that
> 
> 8) register the runner - you're doing that
> 
> 9) enable the service - Ansible's service module is generic and supports
> init/OpenRC
> 
> 10) I don't see a step 10
>

This was either a mistake or the installation steps got updated.  I'll
adjust that.

> IOW I think there should be as little manual intervention as possible and in
> this case I don't think any manual steps are needed by the user.
>

Agreed.  I was not super happy with the current state of this playbook
wrt FreeBSD, but I decided to pick other battles to fight.  Also,
newer gitlab-runner versions *may* be using the updated service
management lib, which may reduce/remove the need for custom handling.

Anyway, I'll look at what can be improved here, considering the cost.

Thanks!
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]