Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Erik Skultety <eskultet@redhat.com>]

On Mon, Oct 19, 2020 at 04:41:38PM -0400, Cleber Rosa wrote:
> On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote:
> > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote:
> > > To have the jobs dispatched to custom runners, gitlab-runner must
> > > be installed, active as a service and properly configured.  The
> > > variables file and playbook introduced here should help with those
> > > steps.
> > > 
> > > The playbook introduced here covers a number of different Linux
> > > distributions and FreeBSD, and are intended to provide a reproducible
> > > environment.
> > > 
> > > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > > ---
> > 
> > In general, there's been put quite some effort into the playbooks - sorry I'm
> > late to the game - is there a plan to introduce QEMU as a project to lcitool?
> 
> I think it's becoming quite clear that having so much duplication (in
> the dockerfiles, tests/vm, this playbook, etc) is costly and error
> prone.  I don't know if anyone has invested time in a PoC to
> consolidate those (with lcitool), but I can certainly see the upside
> to that.  BTW, are you volunteering (wink wink)? :)

I don't think I was trying to :), but sure, I can dedicate some time to it.
I'll need a bit of guidance in the QEMU world though for sure.

> 
> > We've taken care of most of the bits in the playbooks that are being introduced
> > and for the remaining ones I think it would be that big of an overhaul to do
> > the adjustments. One major re-factor though would IMO be to break the
> > dependency lcitool has on the machine naming, kind of restricting it to a
> > limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which
> > makes it inconvenient to prepare physical hosts.
> >
> 
> Right... I wasn't aware of that depedency.  And, this may be a nice
> project to make sure that lcitool doesn't have any other libvirt
> specificities.

Yeah, that one will be a major change. Thinking about this some more, while
having a QEMU project inside lcitool sounds definitely useful, it would
probably be a wasted effort wrt task priority right now, because without
expanding lcitool's range of applicable hosts I don't see how QEMU would
utilize the tool without using quirks.

> 
> > More comments inline...
> > 
> > >  docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
> > >  scripts/ci/setup/.gitignore        |  1 +
> > >  scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
> > >  scripts/ci/setup/vars.yml.template | 13 ++++++
> > >  4 files changed, 149 insertions(+)
> > >  create mode 100644 scripts/ci/setup/.gitignore
> > >  create mode 100644 scripts/ci/setup/gitlab-runner.yml
> > >  create mode 100644 scripts/ci/setup/vars.yml.template
> > > 
> > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > > index 208b5e399b..a234a5e24c 100644
> > > --- a/docs/devel/ci.rst
> > > +++ b/docs/devel/ci.rst
> > > @@ -84,3 +84,66 @@ To run the playbook, execute::
> > >  
> > >    cd scripts/ci/setup
> > >    ansible-playbook -i inventory build-environment.yml
> > > +
> > > +gitlab-runner setup and registration
> > > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > +
> > > +The gitlab-runner agent needs to be installed on each machine that
> > > +will run jobs.  The association between a machine and a GitLab project
> > > +happens with a registration token.  To find the registration token for
> > > +your repository/project, navigate on GitLab's web UI to:
> > > +
> > > + * Settings (the gears like icon), then
> > > + * CI/CD, then
> > > + * Runners, and click on the "Expand" button, then
> > > + * Under "Set up a specific Runner manually", look for the value under
> > > +   "Use the following registration token during setup"
> > > +
> > > +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> > > +``scripts/ci/setup/vars.yml``.  Then, set the
> > > +``gitlab_runner_registration_token`` variable to the value obtained
> > > +earlier.
> > > +
> > > +.. note:: gitlab-runner is not available from the standard location
> > > +          for all OS and architectures combinations.  For some systems,
> > > +          a custom build may be necessary.  Some builds are avaiable
> > > +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> > > +          URI may be used as a value on ``vars.yml``
> > 
> > Yes, this can be suboptimal...Would it make sense to fall back to build the
> > binary of a given version from git as a fallback during this playbook if the
> > necessary arch version isn't provided the official way? Just an idea, I'd like
> > to avoid the need for you to become the maintainer of the binaries and keep up
> > with the releases.
> >
> 
> Well, building them during the playbook would be a lot more
> complex... You can have your own repo with your own builds, and just
> tweak your vars.yml.

I didn't want to suggest this as a replacement to vars.yml.
What if people wanting to run the playbook don't have their own repo and you
(I mean you personally) are not providing a fresh build of gitlab-runner for
the given architecture?
The automation machinery would surely be more complex, but at the same time,
I can image a dedicated playbook that would handle gitlab-runner building and
installation of the binary to the correct place, you'd then simply include that
playbook if vars.yml don't specify an alternative repo and download from AWS
failed.
Another thing to consider - you currently host the missing arch builds under
your personal namespace on fedorapeople, what if you decide one day you're not
interested in being involved in QEMU CI that much and suddenly you stop
building the binaries, someone has to take over. On the other hand, if you
automate this whole process, having exposed the builds would definitely not
slow down machine provisions, but would also not cause them to fail if nobody
exposes them.

> 
> > > +
> > > +To run the playbook, execute::
> > > +
> > > +  cd scripts/ci/setup
> > > +  ansible-playbook -i inventory gitlab-runner.yml
> > > +
> > > +.. note:: there are currently limitations to gitlab-runner itself when
> > > +          setting up a service under FreeBSD systems.  You will need to
> > > +          perform steps 4 to 10 manually, as described at
> > 
> > Which one of them is considered an automation problem? In lcitool we made
> > gitlab-runner completely automated on all distros, including FreeBSD:
> >
> 
> It looks like lcitool went the more practical route.  I was hoping to
> not have to treat gitlab-runner in such a special way in any
> "supported" OS.  What I mean is, I'd rather write the code within
> gitlab-runner (or reespective libraries).  Of course, I did not get to
> it, so that's why I just documented the steps here.
> 
> I'll take a look at lcitool's playbook to see if I can easily
> canibalize some of that.  But, at this time, we don't runners for
> FreeBSD anyway, so I guess this is not *that* urgent.

Okay, I was simply curious. If it's not something you don't need at the moment,
then let's not focus on it.

> 
> > 4) log file permissions - you're creating the user, you can as well create the
> > file with correct permissions
> > 
> > 5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml
> > it will pull a bunch of dependencies which ensure the dir exists
> > 
> > 6) gitlab_runner service script should IMO provided by this automation as a
> > template and install to the correct location
> > 
> > 7) ensure the service script is executable - template module can do that
> > 
> > 8) register the runner - you're doing that
> > 
> > 9) enable the service - Ansible's service module is generic and supports
> > init/OpenRC
> > 
> > 10) I don't see a step 10
> >
> 
> This was either a mistake or the installation steps got updated.  I'll
> adjust that.
> 
> > IOW I think there should be as little manual intervention as possible and in
> > this case I don't think any manual steps are needed by the user.
> >
> 
> Agreed.  I was not super happy with the current state of this playbook
> wrt FreeBSD, but I decided to pick other battles to fight.  Also,
> newer gitlab-runner versions *may* be using the updated service
> management lib, which may reduce/remove the need for custom handling.

What do you mean by "service management lib"?

Regards,
Erik