Re: [RFC PATCH] scripts/ci: update gitlab-runner playbook to use latest runner

["Alex Bennée" <alex.bennee@linaro.org>]

Alex Bennée <alex.bennee@linaro.org> writes:

> We were using quite and old runner on our machines and running into
> issues with stalling jobs. Gitlab in the meantime now reliably provide
> the latest packaged versions of the runner under a stable URL. This
> update:
>
>   - creates a per-arch subdir for builds
>   - switches from binary tarballs to deb packages
>   - re-uses the same binary for the secondary runner
>   - updates distro check for second to 22.04
>
> So far I've tested on aarch64.ci.qemu.org but I shall do s390x next as
> its having issues with stale runners as well.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  scripts/ci/setup/gitlab-runner.yml | 42 ++++++++++++++++--------------
>  scripts/ci/setup/vars.yml.template |  2 --
>  2 files changed, 22 insertions(+), 22 deletions(-)
>
> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> index 33128be85d..05cdb2ae4b 100644
> --- a/scripts/ci/setup/gitlab-runner.yml
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -30,6 +30,11 @@
>          home: /home/gitlab-runner
>          shell: /bin/bash
>  
> +    - name: Create working directories for build
> +      file:
> +        path: "/home/gitlab-runner/{{ ansible_facts[\"architecture\"] }}"
> +        state: directory
> +
>      - name: Remove the .bash_logout file when on Ubuntu systems
>        file:
>          path: /home/gitlab-runner/.bash_logout
> @@ -50,17 +55,17 @@
>  
>      - name: Download the matching gitlab-runner
>        get_url:
> -        dest: /usr/local/bin/gitlab-runner
> -        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
> -        owner: gitlab-runner
> -        group: gitlab-runner
> -        mode: u=rwx,g=rwx,o=rx
> +        dest: "/root/"
> +        url: "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_{{ gitlab_runner_arch }}.deb"
> +
> +    - name: Install gitlab-runner via package manager
> +      apt: deb="/root/gitlab-runner_{{ gitlab_runner_arch }}.deb"
>  
>      - name: Register the gitlab-runner
> -      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> +      command: "/usr/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
>  
>      - name: Install the gitlab-runner service using its own functionality
> -      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
> +      command: "/usr/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner/{{ ansible_facts[\"architecture\"] }}"
>        register: gitlab_runner_install_service_result
>        failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
>  
> @@ -70,33 +75,30 @@
>          state: started
>          enabled: yes
>  
> -    - name: Download secondary gitlab-runner
> -      get_url:
> -        dest: /usr/local/bin/gitlab-runner-arm
> -        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-arm"
> -        owner: gitlab-runner
> -        group: gitlab-runner
> -        mode: u=rwx,g=rwx,o=rx
> +    - name: Create working directories secondary runner
> +      file:
> +        path: "/home/gitlab-runner/arm"
> +        state: directory
>        when:
>          - ansible_facts['distribution'] == 'Ubuntu'
>          - ansible_facts['architecture'] == 'aarch64'
> -        - ansible_facts['distribution_version'] == '20.04'
> +        - ansible_facts['distribution_version'] == '22.04'
>  
>      - name: Register secondary gitlab-runner
> -      command: "/usr/local/bin/gitlab-runner-arm register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list aarch32,{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> +      command: "/usr/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list aarch32,{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
>        when:
>          - ansible_facts['distribution'] == 'Ubuntu'
>          - ansible_facts['architecture'] == 'aarch64'
> -        - ansible_facts['distribution_version'] == '20.04'
> +        - ansible_facts['distribution_version'] == '22.04'

  I think everything bellow here can be dropped now as both runners seem
  to be showing the same runner ids:

  13:42:48 [root@aarch64:/e/systemd] + systemctl status gitlab-runner-arm
  ● gitlab-runner-arm.service - GitLab Runner
       Loaded: loaded (/etc/systemd/system/gitlab-runner-arm.service; enabled; vendor preset: enabled)
       Active: active (running) since Fri 2023-01-06 13:42:45 UTC; 9s ago
     Main PID: 38362 (gitlab-runner)
        Tasks: 33 (limit: 307985)
       Memory: 16.0M
          CPU: 81ms
       CGroup: /system.slice/gitlab-runner-arm.service
               └─38362 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner/arm --config /etc/gitlab-runner/config.toml --service gitlab-runner-arm --user gitla>

  Jan 06 13:42:45 aarch64.ci.qemu.org gitlab-runner[38362]: Running in system-mode.                           
  Jan 06 13:42:45 aarch64.ci.qemu.org gitlab-runner[38362]:                                                   
  Jan 06 13:42:45 aarch64.ci.qemu.org gitlab-runner[38362]: Configuration loaded                                builds=0
  Jan 06 13:42:45 aarch64.ci.qemu.org gitlab-runner[38362]: listen_address not defined, metrics & debug endpoints disabled  builds=0
  Jan 06 13:42:45 aarch64.ci.qemu.org gitlab-runner[38362]: [session_server].listen_address not defined, session endpoints disabled  builds=0
  Jan 06 13:42:45 aarch64.ci.qemu.org gitlab-runner[38362]: Initializing executor providers                     builds=0
  Jan 06 13:42:45 aarch64.ci.qemu.org gitlab-runner[38362]: Checking for jobs...nothing                         runner=HXc1tXPb
  Jan 06 13:42:46 aarch64.ci.qemu.org gitlab-runner[38362]: Checking for jobs...nothing                         runner=pwrzaBKo
  Jan 06 13:42:47 aarch64.ci.qemu.org gitlab-runner[38362]: Checking for jobs...nothing                         runner=E8PpwMky
  Jan 06 13:42:47 aarch64.ci.qemu.org gitlab-runner[38362]: Checking for jobs...nothing                         runner=syqX1xoo
  13:43:06 [root@aarch64:/e/systemd] + systemctl status gitlab-runner
  ● gitlab-runner.service - GitLab Runner
       Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
       Active: active (running) since Fri 2023-01-06 13:42:42 UTC; 28s ago
     Main PID: 38270 (gitlab-runner)
        Tasks: 27 (limit: 307985)
       Memory: 14.3M
          CPU: 70ms
       CGroup: /system.slice/gitlab-runner.service
               └─38270 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner

  Jan 06 13:42:42 aarch64.ci.qemu.org gitlab-runner[38270]: Running in system-mode.                           
  Jan 06 13:42:42 aarch64.ci.qemu.org gitlab-runner[38270]:                                                   
  Jan 06 13:42:42 aarch64.ci.qemu.org gitlab-runner[38270]: Configuration loaded                                builds=0
  Jan 06 13:42:42 aarch64.ci.qemu.org gitlab-runner[38270]: listen_address not defined, metrics & debug endpoints disabled  builds=0
  Jan 06 13:42:42 aarch64.ci.qemu.org gitlab-runner[38270]: [session_server].listen_address not defined, session endpoints disabled  builds=0
  Jan 06 13:42:42 aarch64.ci.qemu.org gitlab-runner[38270]: Initializing executor providers                     builds=0
  Jan 06 13:42:43 aarch64.ci.qemu.org gitlab-runner[38270]: Checking for jobs...nothing                         runner=HXc1tXPb
  Jan 06 13:42:43 aarch64.ci.qemu.org gitlab-runner[38270]: Checking for jobs...nothing                         runner=pwrzaBKo
  Jan 06 13:42:44 aarch64.ci.qemu.org gitlab-runner[38270]: Checking for jobs...nothing                         runner=E8PpwMky
  Jan 06 13:42:45 aarch64.ci.qemu.org gitlab-runner[38270]: Checking for jobs...nothing                         runner=syqX1xoo



-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro