[PATCH v4 0/8] ci: add GitLab CI definition

[Patrick Steinhardt <ps@pks.im>]

[-- Attachment #1: Type: text/plain, Size: 7520 bytes --]

Hi,

this is the fourth version of my patch series that introduces support
for GitLab CI.

Changes compared to v3:

    - Stopped using nproc(1) to figure out the number of builds jobs for
      GitHub Actions and Azure Pipelines. Instead, we now continue to
      use the hardcoded 10 jobs there, whereas on GitLab CI we now use
      nproc. We can adapt GitHub/Azure at a later point as required, but
      I don't feel comfortable doing changes there.

    - Improved the linux-musl job. Namely, we now also install all
      required Apache modules, which makes the Apache-based test setup
      work. There is a packaging issue with the WebDAV module though, so
      we now skip tests that depend on it on Alpine Linux.

I still didn't move `.gitlab-ci.yml` to `contrib/`. As Taylor argued
(and I don't disagree), moving it to `contrib/` would convey the spirit
that this is _not_ an authoritative CI pipeline setup. But I wanted to
hear other opinions first before moving it into `contrib/`.

Patrick

Patrick Steinhardt (8):
  ci: reorder definitions for grouping functions
  ci: make grouping setup more generic
  ci: group installation of Docker dependencies
  ci: split out logic to set up failed test artifacts
  ci: unify setup of some environment variables
  ci: squelch warnings when testing with unusable Git repo
  ci: install test dependencies for linux-musl
  ci: add support for GitLab CI

 .gitlab-ci.yml                    |  53 +++++++++
 ci/install-docker-dependencies.sh |  23 +++-
 ci/lib.sh                         | 190 +++++++++++++++++++++---------
 ci/print-test-failures.sh         |   6 +
 t/lib-httpd.sh                    |  17 ++-
 5 files changed, 233 insertions(+), 56 deletions(-)
 create mode 100644 .gitlab-ci.yml

Range-diff against v3:
1:  ef44ed5c3b1 = 1:  8595fe5016a ci: reorder definitions for grouping functions
2:  77798fa7a7a = 2:  7358a943392 ci: make grouping setup more generic
3:  4542bd38dc2 = 3:  6d842592c6f ci: group installation of Docker dependencies
4:  5fdda7fd83f = 4:  e15651b3f5d ci: split out logic to set up failed test artifacts
5:  6af0075fd87 ! 5:  a64799b6e25 ci: unify setup of some environment variables
    @@ Commit message
         parts are separated. While at it, we also perform some additional small
         improvements:
     
    -        - We use nproc instead of a hardcoded count of jobs for make and
    -          prove. This ensures that the number of concurrent processes adapts
    -          to the host automatically.
    -
             - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.
               It doesn't hurt on platforms where we don't persist the state, so
               this further reduces boilerplate.
    @@ ci/lib.sh: then
      	exit 1
      fi
      
    -+MAKEFLAGS="$MAKEFLAGS --jobs=$(nproc)"
    -+GIT_PROVE_OPTS="--timer --jobs $(nproc) --state=failed,slow,save"
    ++MAKEFLAGS="$MAKEFLAGS --jobs=10"
    ++GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
     +
     +GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x"
     +if test windows = "$CI_OS_NAME"
6:  78d863bf24e = 6:  f7d2a8666fe ci: squelch warnings when testing with unusable Git repo
7:  f150d61a1ce ! 7:  9b43b0d90e3 ci: install test dependencies for linux-musl
    @@ Commit message
               available, all Subversion-related tests are skipped without the
               SVN::Core Perl library anyway.
     
    -    Furthermore, in order to make the Apache-based tests work, this commit
    -    also adds the Alpine-specific modules path of it to the list of known
    -    paths.
    +    The Apache2-based tests require a bit more care though. For one, the
    +    module path is different on Alpine Linux, which requires us to add it to
    +    the list of known module paths to detect it. But second, the WebDAV
    +    module on Alpine Linux is broken because it does not bundle the default
    +    database backend [1]. We thus need to skip the WebDAV-based tests on
    +    Alpine Linux for now.
    +
    +    [1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    @@ ci/install-docker-dependencies.sh: linux32)
      	apk add --update build-base curl-dev openssl-dev expat-dev gettext \
     -		pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null
     +		pcre2-dev python3 musl-libintl perl-utils ncurses \
    -+		apache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null
    ++		apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \
    ++		bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null
      	;;
      pedantic)
      	dnf -yq update >/dev/null &&
    @@ t/lib-httpd.sh: for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \
      do
      	if test -d "$DEFAULT_HTTPD_MODULE_PATH"
      	then
    +@@ t/lib-httpd.sh: else
    + 		"Could not identify web server at '$LIB_HTTPD_PATH'"
    + fi
    + 
    ++if test -n "$LIB_HTTPD_DAV" && test -f /etc/os-release
    ++then
    ++	case "$(grep "^ID=" /etc/os-release | cut -d= -f2-)" in
    ++	alpine)
    ++		# The WebDAV module in Alpine Linux is broken at least up to
    ++		# Alpine v3.16 as the default DBM driver is missing.
    ++		#
    ++		# https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112
    ++		test_skip_or_die GIT_TEST_HTTPD \
    ++			"Apache WebDAV module does not have default DBM backend driver"
    ++		;;
    ++	esac
    ++fi
    ++
    + install_script () {
    + 	write_script "$HTTPD_ROOT_PATH/$1" <"$TEST_PATH/$1"
    + }
8:  5272d66d9f1 ! 8:  f3f2c98a0dc ci: add support for GitLab CI
    @@ ci/install-docker-dependencies.sh: linux32)
     -	apk add --update build-base curl-dev openssl-dev expat-dev gettext \
     +	apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \
      		pcre2-dev python3 musl-libintl perl-utils ncurses \
    - 		apache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null
    + 		apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \
    + 		bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null
      	;;
     +linux-*)
     +	apt update -q &&
    @@ ci/lib.sh: then
      else
      	begin_group () { :; }
      	end_group () { :; }
    +@@ ci/lib.sh: then
    + 	cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME"
    + 
    + 	GIT_TEST_OPTS="--write-junit-xml"
    ++	JOBS=10
    + elif test true = "$GITHUB_ACTIONS"
    + then
    + 	CI_TYPE=github-actions
     @@ ci/lib.sh: then
      	cache_dir="$HOME/none"
      
      	GIT_TEST_OPTS="--github-workflow-markup"
    ++	JOBS=10
     +elif test true = "$GITLAB_CI"
     +then
     +	CI_TYPE=gitlab-ci
    @@ ci/lib.sh: then
     +	cache_dir="$HOME/none"
     +
     +	runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -)
    ++	JOBS=$(nproc)
      else
      	echo "Could not identify CI type" >&2
      	env >&2
    + 	exit 1
    + fi
    + 
    +-MAKEFLAGS="$MAKEFLAGS --jobs=10"
    +-GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
    ++MAKEFLAGS="$MAKEFLAGS --jobs=${JOBS}"
    ++GIT_PROVE_OPTS="--timer --jobs ${JOBS} --state=failed,slow,save"
    + 
    + GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x"
    + if test windows = "$CI_OS_NAME"
     
      ## ci/print-test-failures.sh ##
     @@ ci/print-test-failures.sh: do
-- 
2.42.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]