From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C8151878 for ; Sat, 30 Mar 2024 02:25:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711765545; cv=none; b=XKUTygH27q6vVPZPXdz0PyEaRfr26XcUy9wcWD52bXGCER60+zEfmS0ExurP71DurEvA8jksRJbY/AkJmtsQc57urYqiALsCB8Sk+m2RA4lIEiwLcd8Q5FY7N+8xN2ccwEoIwGcDcCkq22N/cXFnZG5fDMehUVFFDhc2Tt2ujwQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711765545; c=relaxed/simple; bh=4qPZtsdigQnaXu2vlm4NcniFdbvhsNzZl8PJH0tThZU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bMjIF46sOAtEllHCHlOWtijVMoVegoBHCvj9MERSPwc9DbNsvdP0BhdA3j5ELOdgdvvO9236JoHAz3JJG2yqy83I+wI1vjWD7t0aFPRxx0kGXxXAMTXTDW6GykVTA6gV507HVcoAkgCQtFv8hMQNns/DNx+sHZMaMBjTlZjD0Og= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=rFEQ2ejp; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="rFEQ2ejp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=rh/cKXkypX32+NpKXmN1JYKozh5CwLsQmpVsW6LPKR4=; b=rFEQ2ejptqaWRw30M5QO56hsrC ulRUDHx/e1hbNky1jFAxvpjkqXnMTkVvPD9cDivQY3/fcPZSH6zFnaLAqtdYVTw8c9IzrEZiyeLJ9 AYTgE3MTT7M09BkqrVXNE2o0hbHT89To2ZJT8+HVvla800JBzYPDdkuN0Hir/w7QPusxc+UGQMgp9 FiolS3gWPL5sBvHrHzFUSpBqRkBCKDIlSlqixO2NPEL5+pAlMxqODcxY1MMgeZu0+Ki7nkQOhbvzH 8Lt+6ZwceBQQZ62SkYzCw52oj/AZ3EVjwMbyAZJxO3nswzxdqlEqo2+kvF8tH03HF3KLCunNcw6Kt gbt/eKzA==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rqOPe-00000002ZLW-0Ci7; Sat, 30 Mar 2024 02:25:42 +0000 From: Luis Chamberlain To: kdevops@lists.linux.dev Cc: Luis Chamberlain Subject: [PATCH 08/18] mirrors: add kdevops git alternative for fstests Date: Fri, 29 Mar 2024 19:25:30 -0700 Message-ID: <20240330022540.612487-9-mcgrof@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240330022540.612487-1-mcgrof@kernel.org> References: <20240330022540.612487-1-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain kdevops supports its own CONFIG_GIT_ALTERNATIVES which allows kdevops to mirror upstream projects on its own organization, so that our own fork can host patches which perhaps have not landed upstream on fstests yet, and also to ensure reproducible workflows. If we follow upstream, the branch may be a moving target and sometimes thigns may work sometimes things will not. For deatils refer to our Kconfig documentation of GIT_ALTERNATIVES. This now mirrors the fstests kdevops git alternatives. We will place these mirrored alternatives under a directory representing the alternative source, in this case its kdevops, /mirror/kdevops/fstest.git. Note that we have an alternative for fstests for btrfs but this doesn't yet properly pronounce itself as an alternative itself, its was just caked in as another option. We should fix that so that we can also mirror btrfs's fstests and be able to easily use it. Now with today's defaults you will end up using mirroring for both kdevops and fstests when you enable the fstests workflow if a mirror was found for each one on your system: grep git extra_vars.yaml | grep git kdevops_git: git://192.168.122.1/mirror/kdevops.git kdevops_git_version: HEAD fstests_git: git://192.168.122.1/mirror/kdevops/fstests.git fstests_git_version: HEAD It is important to use --reference here as the kdevops fork and so can share objects on the server with the fstests tree. If you add support for a tree which is a fork, do the same. Signed-off-by: Luis Chamberlain --- Makefile.linux-mirror | 1 + kconfigs/Kconfig.mirror | 30 +++++++++++++++++++ .../linux-mirror-systemd/Makefile | 7 +++++ .../kdevops-fstests-mirror.service | 13 ++++++++ .../kdevops-fstests-mirror.timer | 11 +++++++ 5 files changed, 62 insertions(+) create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer diff --git a/Makefile.linux-mirror b/Makefile.linux-mirror index d6b0c848eaeb..f6cc1b447eb8 100644 --- a/Makefile.linux-mirror +++ b/Makefile.linux-mirror @@ -62,3 +62,4 @@ export QEMU := $(subst ",,$(CONFIG_MIRROR_QEMU_URL)) export QEMU_JIC23 := $(subst ",,$(CONFIG_MIRROR_QEMU_JIC23_URL)) export KDEVOPS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_URL)) export FSTESTS := $(subst ",,$(CONFIG_MIRROR_FSTESTS_URL)) +export KDEVOPS_FSTESTS := $(subst ",,$(CONFIG_MIRROR_KDEVOPS_FSTESTS_URL)) diff --git a/kconfigs/Kconfig.mirror b/kconfigs/Kconfig.mirror index fa5ec8c1333f..076d5d117984 100644 --- a/kconfigs/Kconfig.mirror +++ b/kconfigs/Kconfig.mirror @@ -44,6 +44,7 @@ config INSTALL_LOCAL_LINUX_MIRROR kdevops specific trees and kdevops git alternatives: /mirror/kdevops.git + /mirror/kdevops/fstests.git Tools: /mirror/fstests.git @@ -429,5 +430,34 @@ config MIRROR_FSTESTS_URL default "https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git" if MIRROR_FSTESTS_HTTPS default "https://kernel.googlesource.com/pub/scm/fs/xfs/xfstests-dev.git" if MIRROR_FSTESTS_HTTPS_GOOGLE +choice + prompt "Tools - kdevops/fstests.git mirror protocol/source" + default MIRROR_KDEVOPS_FSTESTS_HTTPS_GITHUB + default MIRROR_KDEOVPS_FSTESTS_HTTPS_GITLAB + depends on INSTALL_LOCAL_LINUX_MIRROR + +config MIRROR_KDEVOPS_FSTESTS_HTTPS_GITHUB + bool "HTTPS (GitHub)" + help + If you enable this option then GitHub HTTPS will be used as the + source of the mirror. The URL is: + + https://github.com/linux-kdevops/fstests.git + +config MIRROR_KDEVOPS_FSTESTS_HTTPS_GITLAB + bool "HTTPS (GitLab)" + help + If you enable this option then GitLab HTTPS will be used as the + source of the mirror. The URL is: + + https://gitlab.com/linux-kdevops/fstests.git + +endchoice + +config MIRROR_KDEVOPS_FSTESTS_URL + string + default "https://github.com/linux-kdevops/fstests.git" if MIRROR_KDEVOPS_FSTESTS_HTTPS_GITHUB + default "https://gitlab.com/linux-kdevops/fstests.git" if MIRROR_KDEVOPS_FSTESTS_HTTPS_GITLAB + endif # ENABLE_LOCAL_LINUX_MIRROR endif # TERRAFORM diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile index 7de8cc4e326d..4a69ee7ca88f 100644 --- a/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile +++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/Makefile @@ -19,7 +19,9 @@ KDEVOPS_SERVICES := kdevops-mirror.service KDEVOPS_TIMERS := kdevops-mirror.timer TOOLS_SERVICES := fstests-mirror.service +TOOLS_SERVICES += kdevops-fstests-mirror.service TOOLS_TIMERS := fstests-mirror.timer +TOOLS_TIMERS += kdevops-fstests-mirror.timer QEMU_SERVICES := qemu-mirror.service QEMU_SERVICES += qemu-jic23-mirror.service @@ -40,6 +42,7 @@ QEMU_TARGET := $(MIRROR_PATH)/qemu.git QEMU_JIC23_TARGET := $(MIRROR_PATH)/qemu-jic23.git KDEVOPS_TARGET := $(MIRROR_PATH)/kdevops.git FSTESTS_TARGET := $(MIRROR_PATH)/fstests.git +KDEVOPS_FSTESTS_TARGET := $(MIRROR_PATH)/kdevops/fstests.git MIRROR_SERVICES := $(LINUX_SERVICES) MIRROR_SERVICES += $(QEMU_SERVICES) @@ -116,6 +119,10 @@ mirror: echo " CLONE fstests" ;\ git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(FSTESTS) $(FSTESTS_TARGET) ;\ fi + $(Q)if [ ! -d $(KDEVOPS_FSTESTS_TARGET) ]; then \ + echo " CLONE kdevops-fstests" ;\ + git -C $(MIRROR_PATH) clone $(GIT_VERBOSE) --progress --mirror $(KDEVOPS_FSTESTS) --reference $(FSTESTS_TARGET) $(KDEVOPS_FSTESTS_TARGET) ;\ + fi install: $(Q)mkdir -p $(USER_SYSTEM) diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service new file mode 100644 index 000000000000..822a62548937 --- /dev/null +++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service @@ -0,0 +1,13 @@ +[Unit] +Description=kdevops-fstests mirror [kdevops/fstests.git] +Documentation=man:git(1) +ConditionPathExists=/mirror/kdevops/fstests.git + +[Service] +Type=oneshot +ExecStartPre=/usr/bin/git -C /mirror/kdevops/fstests.git remote update --prune +ExecStart=/usr/bin/git -C /mirror/kdevops/fstests.git fetch --tags --prune +ExecStartPost=/usr/bin/git -C /mirror/kdevops/fstests.git fetch origin +refs/heads/*:refs/heads/* + +[Install] +WantedBy=multi-user.target diff --git a/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer new file mode 100644 index 000000000000..4ec7a794b488 --- /dev/null +++ b/playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer @@ -0,0 +1,11 @@ +[Unit] +Description=kdevops fstests mirror query timer [kdevops/fstests.git] +ConditionPathExists=/mirror/kdevops/fstests.git + +[Timer] +OnBootSec=10m +# Every 6 hours +OnUnitInactiveSec=360m + +[Install] +WantedBy=default.target -- 2.43.0