From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F38B17F7 for ; Sat, 30 Mar 2024 02:25:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711765544; cv=none; b=Wjlb6t9qkQlRCSJ7XGnI+KgwOPWfrtCR7KpmZau7th7gMdX3IAEEPnbXPVVsO1WUNZ/XpKY7PvEsYmM2DysNyTDfS+liiPuaEW9LsKPQkHTj8cWV9YY1KF3VKVpy4Cz+iyUiz9IVq6Gm/F9wHSKyMLGn7C7BsjKDsTXLrNcvQ3U= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711765544; c=relaxed/simple; bh=ZhGUGYI2zagkmaNGWctTd8uQTBBNs5sa+Dr0vq5Ra8U=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=ENu6y0G0lfcVbNjFcrloadOBtSQnmjta5xVd0h0bIby3hZLMn9p05SkUg9A7OUuI7Shn3mVqNxZONbIxUJZ1xewpCT+sFmSa7Hri173pmuvtnHIRu0SfgfZXH7t34c6e3DAYoFuiU+EKHsxKte5kDNNkSirr5DV6LjidzDt3DkU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=SzmCNmD6; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="SzmCNmD6" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-Type: Content-ID:Content-Description:In-Reply-To:References; bh=UInkMSUmZUeiX88HKeY2QEvSLmt7ucF4TL6nF7B4D1M=; b=SzmCNmD6MQjjDYgcPPsYG2G07X BYwned+LDZLK8Yuxkt6W0PRj4QYlUnHoIiqBsAkWIjZUX/Q3yontF8JBOs1HO4U40dhvZOboMgUpQ 0R1KKU9CtWhIsNvTul7mj/aW4TU4l25Fm2cJT9hsahC/rmbMFPp9CBIRNtS3EqSv99gl/1FsCuFnG 8kG0E4kbUsU6AkEH7pGwfsH9GuC4SqI9P8dGEk2kMs1gvHyj2HAYlHpbEIlAJpRmx83bzfBbTzQd2 G7ND8q2YlcxHFAIodoSkH+s+kx+toaP2/kq7JL8EARm4ggaaz2twoIqB4K+goyK5bA1Ie4naxr/nh YcAFQz+g==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rqOPd-00000002ZL1-2Dgy; Sat, 30 Mar 2024 02:25:41 +0000 From: Luis Chamberlain To: kdevops@lists.linux.dev Cc: Luis Chamberlain Subject: [PATCH 00/18] mirrors: enhance and add tons of mirrors Date: Fri, 29 Mar 2024 19:25:22 -0700 Message-ID: <20240330022540.612487-1-mcgrof@kernel.org> X-Mailer: git-send-email 2.44.0 Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain If you are stuck in a world where you don't have much data for your downloads you have only one option: mirror everything. So the motivation for this is the exand the existing mirror support on kdevops to mirror more things other than just the kernel and qemu. We now can mirror *everything* we clone including kdevops itself. We also enhance this all so that we can *use* the mirrors by *default* if you have them and you are using local virtualization. We differentiate between vagrant and guestfs. Towards the end this just simplifies the process by adding a yaml file for the mirrors so that we can ingest them and process that with a few python scripts. The last commits demonstrate how easy it is now to add new mirrors. There's only few mirrors left to add but we can do that next week. Luis Chamberlain (18): linux-mirror: add kdevops mirroring support Makefile.min_deps: add netcat as a min binary dependency provision: generalize bridge-ip further mirrors: add guestfs mirror options without 9p mirrors: enable kdevops mirror mirrors: add fstests to mirrors fstests: use local primary fstests mirror if present mirrors: add kdevops git alternative for fstests mirrors: use kdevops fstests mirror when available mirrors: add blktests mirror mirrors: use blktests mirror when its available mirrors: increase scope of mirroring in output mirrors: move status check mirrors: move mirror editing into one file mirrors: add new mirrors.yaml and add xfsprogs mirrors: add xfsdump mirror and use it mirrors: add dbench git mirrors: add blktrace mirror and use it .gitignore | 5 + Kconfig | 1 + Makefile | 2 - Makefile.linux-mirror | 55 ++----- Makefile.min_deps | 1 + kconfigs/Kconfig.ansible_provisioning | 4 +- kconfigs/Kconfig.defaults | 18 +++ kconfigs/Kconfig.guestfs | 6 - kconfigs/Kconfig.mirror | 115 +++++++++++++- kconfigs/workflows/Kconfig.shared | 11 +- playbooks/linux-mirror.yml | 4 + playbooks/roles/blktests/defaults/main.yml | 1 + playbooks/roles/blktests/tasks/main.yml | 2 +- playbooks/roles/fstests/defaults/main.yml | 2 + .../tasks/install-deps/debian/main.yml | 2 +- playbooks/roles/linux-mirror/README.md | 9 ++ .../roles/linux-mirror/defaults/main.yml | 8 + .../linux-mirror-systemd/Makefile | 62 +++++++- .../blktests-mirror.service | 13 ++ .../blktests-mirror.timer | 11 ++ .../fstests-mirror.service | 13 ++ .../linux-mirror-systemd/fstests-mirror.timer | 11 ++ .../kdevops-fstests-mirror.service | 13 ++ .../kdevops-fstests-mirror.timer | 11 ++ .../kdevops-mirror.service | 13 ++ .../linux-mirror-systemd/kdevops-mirror.timer | 10 ++ .../linux-mirror/python/gen-mirror-files.py | 149 ++++++++++++++++++ .../linux-mirror/python/start-mirroring.py | 110 +++++++++++++ playbooks/roles/linux-mirror/tasks/main.yml | 91 +++++++++++ .../linux-mirror/templates/mirrors.yaml.j2 | 13 ++ scripts/append-makefile-vars.sh | 10 ++ scripts/check_mirror_present.sh | 11 ++ scripts/get-distro-default-bridge.sh | 15 ++ scripts/get-distro-prefix.sh | 18 +++ scripts/provision.Makefile | 12 ++ workflows/blktests/Kconfig | 29 +++- workflows/blktests/Makefile | 1 + workflows/fstests/Kconfig | 20 ++- workflows/fstests/Makefile.xfs | 2 + workflows/fstests/xfs/Kconfig | 20 ++- workflows/linux/Kconfig.linus | 3 +- workflows/linux/Kconfig.mcgrof | 2 + workflows/linux/Kconfig.next | 3 +- workflows/linux/Kconfig.stable | 3 +- 44 files changed, 852 insertions(+), 63 deletions(-) create mode 100644 kconfigs/Kconfig.defaults create mode 100644 playbooks/linux-mirror.yml create mode 100644 playbooks/roles/linux-mirror/README.md create mode 100644 playbooks/roles/linux-mirror/defaults/main.yml create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.service create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.timer create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.service create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.timer create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.service create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.timer create mode 100755 playbooks/roles/linux-mirror/python/gen-mirror-files.py create mode 100755 playbooks/roles/linux-mirror/python/start-mirroring.py create mode 100644 playbooks/roles/linux-mirror/tasks/main.yml create mode 100644 playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 create mode 100755 scripts/append-makefile-vars.sh create mode 100755 scripts/check_mirror_present.sh create mode 100755 scripts/get-distro-default-bridge.sh create mode 100755 scripts/get-distro-prefix.sh -- 2.43.0