[PATCH v2 0/5] ci: add GitLab CI definition

[Patrick Steinhardt <ps@pks.im>]

[-- Attachment #1: Type: text/plain, Size: 9177 bytes --]

Hi,

this is the second version of this patch series that adds GitLab CI
definitions to the Git project. Please refer to the cover letter for v1
of this series [1] for the motivation and intent -- I won't repeat it
here as it's a bit on the longer side.

Changes compared to v1:

    - Improved commit messages.

    - The `group ()` function is now generic across all the different CI
      solutions to reduce duplication.

This should hopefully address the feedback from Oswald -- thanks again!

Patrick

[1]: <cover.1698305961.git.ps@pks.im>


Patrick Steinhardt (5):
  ci: reorder definitions for grouping functions
  ci: make grouping setup more generic
  ci: group installation of Docker dependencies
  ci: split out logic to set up failed test artifacts
  ci: add support for GitLab CI

 .gitlab-ci.yml                    |  51 +++++++++++
 ci/install-docker-dependencies.sh |  15 +++-
 ci/lib.sh                         | 139 ++++++++++++++++++++----------
 ci/print-test-failures.sh         |   6 ++
 4 files changed, 166 insertions(+), 45 deletions(-)
 create mode 100644 .gitlab-ci.yml

Range-diff against v1:
1:  586a8d1003b ! 1:  4eb9cfc816b ci: reorder definitions for grouping functions
    @@ Commit message
         output in our CI, where these groups then end up as collapsible sections
         in the respective pipeline platform. The way these functions are defined
         is not easily extensible though as we have an up front check for the CI
    -    _not_ being GitLab Actions, where we define the non-stub logic in the
    +    _not_ being GitHub Actions, where we define the non-stub logic in the
         else branch.
     
    -    Reorder the definitions such that we explicitly handle GitHub Actions.
    +    Reorder the conditional branches such that we explicitly handle GitHub
    +    Actions.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
2:  ec390354f15 ! 2:  85617ef8577 ci: make grouping setup more generic
    @@ Commit message
         or not. This ensures we can more readily add support for additional CI
         platforms.
     
    -    Second, this commit changes `end_group ()` to also accept a parameter
    -    that indicates _which_ group should end. This will be required by a
    -    later commit that introduces support for GitLab CI.
    +    Furthermore, the `group ()` function is made generic so that it is the
    +    same for both GitHub Actions and for other platforms. There is a
    +    semantic conflict here though: GitHub Actions used to call `set +x` in
    +    `group ()` whereas the non-GitHub case unconditionally uses `set -x`.
    +    The latter would get overriden if we kept the `set +x` in the generic
    +    version of `group ()`. To resolve this conflict, we simply drop the `set
    +    +x` in the generic variant of this function. As `begin_group ()` calls
    +    `set -x` anyway this is not much of a change though, as the only
    +    commands that aren't printed anymore now are the ones between the
    +    beginning of `group ()` and the end of `begin_group ()`.
    +
    +    Last, this commit changes `end_group ()` to also accept a parameter that
    +    indicates _which_ group should end. This will be required by a later
    +    commit that introduces support for GitLab CI.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    @@ ci/lib.sh: then
      		echo '::endgroup::' >&2
      	}
     -	trap end_group EXIT
    - 
    - 	group () {
    - 		set +x
    +-
    +-	group () {
    +-		set +x
     -		begin_group "$1"
    -+
    -+		group="$1"
    - 		shift
    -+		begin_group "$group"
    -+
    - 		# work around `dash` not supporting `set -o pipefail`
    - 		(
    - 			"$@" 2>&1
    -@@ ci/lib.sh: then
    - 		sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/'
    - 		res=$(cat exit.status)
    - 		rm exit.status
    +-		shift
    +-		# work around `dash` not supporting `set -o pipefail`
    +-		(
    +-			"$@" 2>&1
    +-			echo $? >exit.status
    +-		) |
    +-		sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/'
    +-		res=$(cat exit.status)
    +-		rm exit.status
     -		end_group
    -+
    -+		end_group "$group"
    - 		return $res
    - 	}
    +-		return $res
    +-	}
     -
     -	begin_group "CI setup"
      else
      	begin_group () { :; }
      	end_group () { :; }
    -@@ ci/lib.sh: else
    + 
    +-	group () {
    +-		shift
    +-		"$@"
    +-	}
      	set -x
      fi
      
    ++group () {
    ++	group="$1"
    ++	shift
    ++	begin_group "$group"
    ++
    ++	# work around `dash` not supporting `set -o pipefail`
    ++	(
    ++		"$@" 2>&1
    ++		echo $? >exit.status
    ++	) |
    ++	sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/'
    ++	res=$(cat exit.status)
    ++	rm exit.status
    ++
    ++	end_group "$group"
    ++	return $res
    ++}
    ++
     +begin_group "CI setup"
     +trap "end_group 'CI setup'" EXIT
     +
3:  a65d235dd3c ! 3:  57bbc50e3dc ci: group installation of Docker dependencies
    @@ Metadata
      ## Commit message ##
         ci: group installation of Docker dependencies
     
    -    Pull in "lib.sh" into "install-docker-dependencies.sh" such that we can
    -    set up proper groups for those dependencise. This allows the reader to
    -    collapse sections in the CI output on GitHub Actions (and later on on
    -    GitLab CI).
    +    The output of CI jobs tends to be quite long-winded and hard to digest.
    +    To help with this, many CI systems provide the ability to group output
    +    into collapsible sections, and we're also doing this in some of our
    +    scripts.
    +
    +    One notable omission is the script to install Docker dependencies.
    +    Address it to bring more structure to the output for Docker-based jobs.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
4:  4a864a1d174 ! 4:  5ab11d5236d ci: split out logic to set up failed test artifacts
    @@ Commit message
         ci: split out logic to set up failed test artifacts
     
         We have some logic in place to create a directory with the output from
    -    failed tests, which will then subsequently be uploaded as CI artifact.
    +    failed tests, which will then subsequently be uploaded as CI artifacts.
         We're about to add support for GitLab CI, which will want to reuse the
         logic.
     
5:  35b07e5378d ! 5:  37a507e9b25 ci: add support for GitLab CI
    @@ Commit message
     
         Part of a problem we hit at GitLab rather frequently is that our own,
         custom CI setup we have is so different to the setup that the Git
    -    project has. More esoteric jobs like "linux-TEST-vars" that also sets a
    +    project has. More esoteric jobs like "linux-TEST-vars" that also set a
         couple of environment variables do not exist in GitLab's custom CI
         setup, and maintaining them to keep up with what Git does feels like
         wasted time. The result is that we regularly send patch series upstream
    -    that would otherwise fail to compile or pass tests in GitHub Workflows.
    -    We would thus like to integrate the GitLab CI configuration into the Git
    -    project to help us ensure to send better patch series upstream and thus
    -    reduce overhead for the maintainer.
    +    that fail to compile or pass tests in GitHub Workflows. We would thus
    +    like to integrate the GitLab CI configuration into the Git project to
    +    help us send better patch series upstream and thus reduce overhead for
    +    the maintainer.
     
         The integration does not necessarily have to be a first-class citizen,
         which would in practice only add to the fallout that pipeline failures
    @@ ci/install-docker-dependencies.sh: linux32)
     
      ## ci/lib.sh ##
     @@ ci/lib.sh: then
    + 		need_to_end_group=
      		echo '::endgroup::' >&2
      	}
    - 
    -+	group () {
    -+		set +x
    -+
    -+		group="$1"
    -+		shift
    -+		begin_group "$group"
    -+
    -+		# work around `dash` not supporting `set -o pipefail`
    -+		(
    -+			"$@" 2>&1
    -+			echo $? >exit.status
    -+		) |
    -+		sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/'
    -+		res=$(cat exit.status)
    -+		rm exit.status
    -+
    -+		end_group "$group"
    -+		return $res
    -+	}
     +elif test true = "$GITLAB_CI"
     +then
     +	begin_group () {
    @@ ci/lib.sh: then
     +		echo -e "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K"
     +		trap - EXIT
     +	}
    -+
    - 	group () {
    - 		set +x
    - 
    + else
    + 	begin_group () { :; }
    + 	end_group () { :; }
     @@ ci/lib.sh: then
      	MAKEFLAGS="$MAKEFLAGS --jobs=10"
      	test windows != "$CI_OS_NAME" ||
-- 
2.42.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]