All of lore.kernel.org
 help / color / mirror / Atom feed
From: Alan Maguire <almagui@kernel.org>
To: dwarves@vger.kernel.org
Cc: Alan Maguire <almagui@kernel.org>
Subject: [PATCH dwarves 1/6] CI: Make kernel source fetch tolerant of git host failures
Date: Mon, 16 Nov 2026 11:20:38 +0000	[thread overview]
Message-ID: <20261116112043.3831978-2-almagui@kernel.org> (raw)
In-Reply-To: <20261116112043.3831978-1-almagui@kernel.org>

CI jobs have been failing with timeouts fetching bpf-next from
git.kernel.org.  Replace the libbpf/ci/get-linux-source action with
a script that does a shallow fetch, retrying across mirrors:

1. git.kernel.org bpf-next master
2. github.com/kernel-patches/bpf bpf-next_base (github-to-github;
   bpf-next plus BPF CI commits)
3. kernel.googlesource.com bpf-next master

with up to 3 rounds and backoff between them.  Each fetch attempt is
bounded by a timeout, and git's low-speed limits abort stalled
transfers so we move on to the next mirror rather than hanging until
the job times out.  The commit fetched is logged.

Since the source is now a detached shallow checkout, build-kernel.sh
no longer checks out REPO_BRANCH; its fallback clone (for local use)
is now a shallow clone of that branch.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Alan Maguire <almagui@kernel.org>
---
 .github/scripts/build-kernel.sh      |  4 +--
 .github/scripts/get-kernel-source.sh | 46 ++++++++++++++++++++++++++++
 .github/workflows/vmtest.yml         |  7 ++---
 3 files changed, 51 insertions(+), 6 deletions(-)
 create mode 100755 .github/scripts/get-kernel-source.sh

diff --git a/.github/scripts/build-kernel.sh b/.github/scripts/build-kernel.sh
index 63806f2..8326e1d 100755
--- a/.github/scripts/build-kernel.sh
+++ b/.github/scripts/build-kernel.sh
@@ -18,11 +18,11 @@ export PAHOLE=${GITHUB_WORKSPACE}/install/usr/local/bin/pahole
 which pahole
 $PAHOLE --version
 
+# In CI, get-kernel-source.sh has already fetched the source.
 if [[ ! -d $REPO_TARGET ]]; then
-	git clone $REPO $REPO_TARGET
+	git clone --depth=1 --branch $REPO_BRANCH $REPO $REPO_TARGET
 fi
 cd $REPO_TARGET
-git checkout $REPO_BRANCH
 
 buildopts=
 if [[ "$CC" = "clang" ]]; then
diff --git a/.github/scripts/get-kernel-source.sh b/.github/scripts/get-kernel-source.sh
new file mode 100755
index 0000000..2d28215
--- /dev/null
+++ b/.github/scripts/get-kernel-source.sh
@@ -0,0 +1,46 @@
+#!/usr/bin/bash
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Fetch bpf-next kernel source, tolerating failures/stalls of any
+# single git host by retrying across mirrors.
+#
+
+GITHUB_WORKSPACE=${GITHUB_WORKSPACE:-$(dirname $0)/../..}
+DEST=${GITHUB_WORKSPACE}/.kernel
+FETCH_TIMEOUT=${FETCH_TIMEOUT:-15m}
+ATTEMPTS=${ATTEMPTS:-3}
+
+# "URL REF" pairs, tried in order.  kernel-patches/bpf bpf-next_base tracks
+# bpf-next master plus BPF CI commits, and is github-to-github so is the
+# first fallback.
+MIRRORS=(
+	"https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master"
+	"https://github.com/kernel-patches/bpf.git bpf-next_base"
+	"https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git master"
+)
+
+git init -q "$DEST" && cd "$DEST" || exit 1
+# Abort stalled transfers (< 1KB/s for 60s) rather than hanging until
+# the job times out, so we move on to the next mirror.
+git config http.lowSpeedLimit 1000
+git config http.lowSpeedTime 60
+
+for attempt in $(seq 1 $ATTEMPTS); do
+	for mirror in "${MIRRORS[@]}"; do
+		read -r url ref <<< "$mirror"
+		echo "Fetching $ref from $url (attempt $attempt/$ATTEMPTS)"
+		if timeout $FETCH_TIMEOUT git fetch --depth=1 "$url" "$ref"; then
+			git checkout -q FETCH_HEAD
+			echo "Fetched kernel source from $url $ref:"
+			git log -1 --oneline
+			exit 0
+		fi
+		echo "Fetch from $url failed"
+	done
+	if [[ $attempt -lt $ATTEMPTS ]]; then
+		sleep $((attempt * 30))
+	fi
+done
+
+echo "Failed to fetch kernel source from all mirrors" >&2
+exit 1
diff --git a/.github/workflows/vmtest.yml b/.github/workflows/vmtest.yml
index d6f9f79..1e876a0 100644
--- a/.github/workflows/vmtest.yml
+++ b/.github/workflows/vmtest.yml
@@ -55,10 +55,9 @@ jobs:
         run: .github/scripts/build-pahole.sh
 
       - name: Get kernel source
-        uses: libbpf/ci/get-linux-source@v3
-        with:
-          repo: 'https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git'
-          dest: '${{ github.workspace }}/.kernel'
+        shell: bash
+        timeout-minutes: 60
+        run: .github/scripts/get-kernel-source.sh
 
       - name: Configure, build kernel with current pahole
         shell: bash
-- 
2.56.0


  reply	other threads:[~2026-11-16 11:20 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 11:20 [PATCH dwarves 0/6] Add support for LTO kernel builds, other CI fixes Alan Maguire
2026-11-16 11:20 ` Alan Maguire [this message]
2026-11-16 11:20 ` [PATCH dwarves 2/6] tests: Match BTF and DWARF functions with set operations in btf_functions.sh Alan Maguire
2026-11-16 11:20 ` [PATCH dwarves 3/6] CI: Scope workflow concurrency groups to the branch for push runs Alan Maguire
2026-11-16 11:20 ` [PATCH dwarves 4/6] tests: Expect no DWARF loader threads in reproducible_build.sh when CUs are merged Alan Maguire
2026-11-16 11:20 ` [PATCH dwarves 5/6] CI: Fix ondemand workflow inputs and LLVM version in function comparison Alan Maguire
2026-11-16 11:20 ` [PATCH dwarves 6/6] CI: Add clang ThinLTO kernel build to test matrix Alan Maguire

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116112043.3831978-2-almagui@kernel.org \
    --to=almagui@kernel.org \
    --cc=dwarves@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.