Git development
 help / color / mirror / Atom feed
From: Tamir Duberstein <tamird@gmail.com>
To: git@vger.kernel.org
Cc: Patrick Steinhardt <ps@pks.im>,
	Junio C Hamano <gitster@pobox.com>,  Jeff King <peff@peff.net>,
	Tamir Duberstein <tamird@gmail.com>
Subject: [PATCH v3 2/2] ci: use twice the CPU count on both providers
Date: Wed, 30 Sep 2026 10:20:35 -0400	[thread overview]
Message-ID: <20260930-ci-large-test-resources-v3-2-d65ac7c21b5f@gmail.com> (raw)
In-Reply-To: <20260930-ci-large-test-resources-v3-0-d65ac7c21b5f@gmail.com>

GitHub Actions sets JOBS to ten regardless of runner size, while
GitLab CI uses the detected CPU count. Use twice the CPU count for
Make and prove on both providers, doubling GitLab's job count.

Five GitHub Actions attempts per policy, with the long tests enabled,
gave these sums of per-job median successful build/test-step times
(minutes; four or five samples per job) [1-3]:

                      Fixed 10   CPU count   2x CPU count
  Linux Make             278.9       273.9          259.9
  macOS Make              94.5       119.1           99.8
  Windows Make           102.2       103.8          100.8

Workflow overhead is excluded; Windows runner images varied.

Use twice the CPU count to scale concurrency with runner size while
avoiding the larger macOS slowdown observed with one job per CPU.
Compared with ten jobs, this trades a lower Linux total for a higher
macOS total.

Keep GitLab's Linux and Windows CPU queries. On macOS, use the native
sysctl command on both providers so CPU detection does not depend on
GNU coreutils.

Link: https://github.com/tamird/git/actions/runs/36070869894/attempts/1 [1]
Link: https://github.com/tamird/git/actions/runs/36070867524/attempts/1 [2]
Link: https://github.com/tamird/git/actions/runs/36070867647/attempts/1 [3]

Assisted-by: LLM
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 ci/lib.sh | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/ci/lib.sh b/ci/lib.sh
index c6ccbf8c17..f0b7f35850 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -227,7 +227,6 @@ then
 	cache_dir="$HOME/none"
 
 	GIT_TEST_OPTS="--github-workflow-markup"
-	JOBS=10
 
 	distro=$(echo "$CI_JOB_IMAGE" | tr : -)
 elif test true = "$GITLAB_CI"
@@ -250,7 +249,6 @@ then
 	case "$OS,$CI_JOB_IMAGE" in
 	Windows_NT,*)
 		CI_OS_NAME=windows
-		JOBS=$NUMBER_OF_PROCESSORS
 		;;
 	*,macos-*)
 		# GitLab CI has Python installed via multiple package managers,
@@ -260,11 +258,9 @@ then
 		export PATH="$(brew --prefix)/bin:$PATH"
 
 		CI_OS_NAME=osx
-		JOBS=$(nproc)
 		;;
 	*,almalinux:*|*,alpine:*|*,debian:*|*,fedora:*|*,ubuntu:*|*,i386/ubuntu:*)
 		CI_OS_NAME=linux
-		JOBS=$(nproc)
 		;;
 	*)
 		echo "Could not identify OS image" >&2
@@ -291,6 +287,19 @@ else
 	exit 1
 fi
 
+case "$CI_OS_NAME" in
+windows|windows_nt)
+	JOBS=$NUMBER_OF_PROCESSORS
+	;;
+osx)
+	JOBS=$(sysctl -n hw.logicalcpu)
+	;;
+*)
+	JOBS=$(nproc)
+	;;
+esac
+JOBS=$((2 * JOBS))
+
 MAKEFLAGS="$MAKEFLAGS --jobs=$JOBS"
 GIT_PROVE_OPTS="--timer --jobs $JOBS"
 

-- 
2.56.0.rc2.851.g50a151a6e9.frankengit


  parent reply	other threads:[~2026-09-30 14:20 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-09-23 17:13 [PATCH 0/2] ci: reduce pressure from large test fixtures Tamir Duberstein
2026-09-23 17:13 ` [PATCH 1/2] t4205: compare huge output without diff Tamir Duberstein
2026-09-24  6:13   ` Patrick Steinhardt
2026-09-24 20:41     ` Tamir Duberstein
2026-09-25  0:13       ` Jeff King
2026-09-25  6:41         ` Junio C Hamano
2026-09-28  6:36       ` Patrick Steinhardt
2026-09-23 17:13 ` [PATCH 2/2] ci: match Linux jobs to available CPUs Tamir Duberstein
2026-09-24  6:13   ` Patrick Steinhardt
2026-09-25 15:03     ` Tamir Duberstein
2026-09-23 18:36 ` [PATCH 0/2] ci: reduce pressure from large test fixtures Tamir Duberstein
2026-09-25 16:35 ` [PATCH v2 0/2] ci: use cmp and align job-count selection Tamir Duberstein
2026-09-25 16:35   ` [PATCH v2 1/2] t4205: compare huge output without diff Tamir Duberstein
2026-09-28  6:36     ` Patrick Steinhardt
2026-09-25 16:35   ` [PATCH v2 2/2] ci: align job counts across CI providers Tamir Duberstein
2026-09-28  6:36     ` Patrick Steinhardt
2026-09-28 10:16       ` Tamir Duberstein
2026-09-28 11:14         ` Patrick Steinhardt
2026-09-30 14:20   ` [PATCH v3 0/2] ci: use cmp and align job-count selection Tamir Duberstein
2026-09-30 14:20     ` [PATCH v3 1/2] t4205: compare huge output without diff Tamir Duberstein
2026-09-30 14:20     ` Tamir Duberstein [this message]
2026-09-30 14:45       ` [PATCH v3 2/2] ci: use twice the CPU count on both providers Patrick Steinhardt
2026-09-30 14:45     ` [PATCH v3 0/2] ci: use cmp and align job-count selection Patrick Steinhardt
2026-09-30 14:58       ` Tamir Duberstein
2026-09-30 18:04         ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260930-ci-large-test-resources-v3-2-d65ac7c21b5f@gmail.com \
    --to=tamird@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=peff@peff.net \
    --cc=ps@pks.im \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox