From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A11A6284B5A for ; Tue, 22 Apr 2025 15:49:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745336959; cv=none; b=HwzgTUr1kjDFxQEnVSEMMSamQCyFEN69BYPszfqxp8lD57SzyEL5SIpy7d3X/O/Wh7gkPR2M1l7ZCx1VxNpNYXvn+gl0ZI4/Z4UYXwypSRr/ZFsiRgonTL0OZ2fU6Vc8NPOJLrwEIEc8wTpnZxGVewLIAecn73SsFQr4Ppb19Z0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745336959; c=relaxed/simple; bh=tXFwqTC8vWiM4pf9ozeUeCwVOHmgsiALWlZ/2+9J2qg=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=jfwZneU0c/0dUMcfiXY6Eph2mqrLTw2gBDfo7OVCUDBRvQ3yOdBytbyRZdgVHMdOTShWv9CS+fI7enp9uAdDAVee/eXi7mOHFJZbeCsVAwR0AD6TEaIHJD9ai2tOFPPx6mVmeaNl+7my3E0OvRziKm4Dep/0iVXpaUnXNbvi+BA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CY/mBnoK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CY/mBnoK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E8985C4CEE9; Tue, 22 Apr 2025 15:49:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745336958; bh=tXFwqTC8vWiM4pf9ozeUeCwVOHmgsiALWlZ/2+9J2qg=; h=From:To:Cc:Subject:Date:From; b=CY/mBnoKlLQ+V2Lhn0CWZGgsbOAC6nfbkhmKstsTsxAsAbAwdVdFMwnS3FiEHo7Vp 3bfV+/05k1WGJ/r4+j9ims6EWRCgF/F/90RmA+eHuXB1MSLu9e4k+SBFH3vqp2kxEx Ev/Y0Emh6+JreRbocjcalA2EbJaHCuCOnCVvW68JyqasC1wvLQ9iHx8AMoyO7x8Nec emdLyJhTCogDvDnMUuMVlKaJQ/66ai+9SfHhnN5bmproSTYPX1/J1SEjeC349htyEW fQ/Ta0P81sMdHQR9xPeiK2R6CEj90lTn54ke9Zn0mV5viR+nWnlgvSDaNegndvnFjC KMEouR5DlKmqg== From: cel@kernel.org To: Luis Chamberlain , Daniel Gomez , Scott Mayhew Cc: , Chuck Lever Subject: [RFC PATCH 0/3] Build once, test everywhere Date: Tue, 22 Apr 2025 11:49:03 -0400 Message-ID: <20250422154906.526319-1-cel@kernel.org> X-Mailer: git-send-email 2.49.0 Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Chuck Lever With apologies to the Java folks. I've been looking for ways to make our workflow runs more efficient because my lab resources are limited, and moving to the cloud costs money per CPU second. One thing that seems like an obvious win would be to build the test kernel one time (for example, after a merge/pull request), then re-use that kernel binary for all the workflows we want to run on it. For libvirt, the usual way to do this is use the 9p option. There are a few cases that are important to me where that option does not work: 1. The control host is running late-model Fedora, and the test runners want to run old LTS kernels 2. The control host or test runners run a RHEL-based kernel, which does not support 9p, last I checked 3. Cloud, where there is no way for cloud-based test runners to mount the control host with 9p That leaves me with building the test kernel in each test runner. That's a lot of kernel builds; and in the cloud, the kernel source code has to be cloned onto each runner from kernel.org, because we don't have repo mirroring there. Ted pointed out another use case where build-once is great: when you want to under-provision the test runners to exercise code paths that handle resource exhaustion. Building the kernel on such test runners takes forever. Cloud providers throw another curve: there is typically a per- tenant or per-availability zone limit on the number of CPU cores you can provision at once. So a workflow that starts several instances has to limit each instance to only one or two cores. Building the kernel on those instances also takes forever. I really want something more like KOTD, where test kernels are built once, packaged, and placed somewhere that the runners can find and install them. I'm not real interested in setting up a persistent yum repo and builder, though; I'd like kdevops to manage all that for me. So I got out my whittling knife and built this proof of concept where kdevops brings up a kernel builder node that is tailored for a fast kernel build (eg, it is one guest with 16 vCPUs). The test kernel is packaged (.rpm or .deb) and fetched to the control host, then the builder node is destroyed. I added a top-level "make" target that uploads the test kernel packages to each test runner, and they install it. This is not part of the patch series, but shows how to run it: $ make mrproper defconfig-kernel-builder $ make && make bringup && make linux-packages && make destroy $ make defconfig-workflow-one $ make && make bringup && make linux-artifacts && make destroy $ make defconfig-workflow-two $ make && make bringup && make linux-artifacts && make destroy $ make defconfig-workflow-three $ make && make bringup && make linux-artifacts && make destroy $ make mrproper All three workflows use the same kernel, built just once, for all of their test runners. This is still band-aids and chewing gum, but it seems to work on both libvirt and cloud configurations. There is more than one way to skin this cat, though. Chuck Lever (3): Add a guest/instance for building the test kernel playbooks: Add a build_linux role Experimental: Add a separate install_linux role .gitignore | 2 + playbooks/build_linux.yml | 4 + playbooks/install_linux.yml | 4 + playbooks/roles/build_linux/README.md | 74 +++++ playbooks/roles/build_linux/defaults/main.yml | 38 +++ .../tasks/install-deps/debian/main.yml | 46 +++ .../tasks/install-deps/redhat/main.yml | 102 ++++++ .../tasks/install-deps/suse/main.yml | 31 ++ playbooks/roles/build_linux/tasks/main.yml | 295 ++++++++++++++++++ playbooks/roles/gen_hosts/defaults/main.yml | 2 + playbooks/roles/gen_hosts/tasks/main.yml | 12 + .../roles/gen_hosts/templates/builder.j2 | 13 + playbooks/roles/gen_nodes/defaults/main.yml | 2 + playbooks/roles/gen_nodes/tasks/main.yml | 22 ++ playbooks/roles/install_linux/README.md | 136 ++++++++ .../roles/install_linux/defaults/main.yml | 43 +++ .../tasks/install-deps/debian/main.yml | 44 +++ .../tasks/install-deps/redhat/main.yml | 76 +++++ .../tasks/install-deps/suse/main.yml | 31 ++ playbooks/roles/install_linux/tasks/main.yml | 142 +++++++++ .../tasks/update-grub/debian.yml | 8 + .../tasks/update-grub/install.yml | 196 ++++++++++++ .../install_linux/tasks/update-grub/main.yml | 15 + .../tasks/update-grub/redhat.yml | 36 +++ .../install_linux/tasks/update-grub/suse.yml | 11 + workflows/linux/Kconfig | 10 + workflows/linux/Makefile | 19 ++ 27 files changed, 1414 insertions(+) create mode 100644 playbooks/build_linux.yml create mode 100644 playbooks/install_linux.yml create mode 100644 playbooks/roles/build_linux/README.md create mode 100644 playbooks/roles/build_linux/defaults/main.yml create mode 100644 playbooks/roles/build_linux/tasks/install-deps/debian/main.yml create mode 100644 playbooks/roles/build_linux/tasks/install-deps/redhat/main.yml create mode 100644 playbooks/roles/build_linux/tasks/install-deps/suse/main.yml create mode 100644 playbooks/roles/build_linux/tasks/main.yml create mode 100644 playbooks/roles/gen_hosts/templates/builder.j2 create mode 100644 playbooks/roles/install_linux/README.md create mode 100644 playbooks/roles/install_linux/defaults/main.yml create mode 100644 playbooks/roles/install_linux/tasks/install-deps/debian/main.yml create mode 100644 playbooks/roles/install_linux/tasks/install-deps/redhat/main.yml create mode 100644 playbooks/roles/install_linux/tasks/install-deps/suse/main.yml create mode 100644 playbooks/roles/install_linux/tasks/main.yml create mode 100644 playbooks/roles/install_linux/tasks/update-grub/debian.yml create mode 100644 playbooks/roles/install_linux/tasks/update-grub/install.yml create mode 100644 playbooks/roles/install_linux/tasks/update-grub/main.yml create mode 100644 playbooks/roles/install_linux/tasks/update-grub/redhat.yml create mode 100644 playbooks/roles/install_linux/tasks/update-grub/suse.yml -- 2.49.0