[RFC PATCH 0/3] Build once, test everywhere

[cel@kernel.org]

From: Chuck Lever <chuck.lever@oracle.com>

With apologies to the Java folks.

I've been looking for ways to make our workflow runs more efficient
because my lab resources are limited, and moving to the cloud costs
money per CPU second.

One thing that seems like an obvious win would be to build the test
kernel one time (for example, after a merge/pull request), then
re-use that kernel binary for all the workflows we want to run on
it.

For libvirt, the usual way to do this is use the 9p option. There
are a few cases that are important to me where that option does not
work:

1. The control host is running late-model Fedora, and the test
   runners want to run old LTS kernels
2. The control host or test runners run a RHEL-based kernel, which
   does not support 9p, last I checked
3. Cloud, where there is no way for cloud-based test runners to
   mount the control host with 9p

That leaves me with building the test kernel in each test runner.
That's a lot of kernel builds; and in the cloud, the kernel source
code has to be cloned onto each runner from kernel.org, because we
don't have repo mirroring there.

Ted pointed out another use case where build-once is great: when
you want to under-provision the test runners to exercise code
paths that handle resource exhaustion. Building the kernel on
such test runners takes forever.

Cloud providers throw another curve: there is typically a per-
tenant or per-availability zone limit on the number of CPU cores
you can provision at once. So a workflow that starts several
instances has to limit each instance to only one or two cores.
Building the kernel on those instances also takes forever.

I really want something more like KOTD, where test kernels are
built once, packaged, and placed somewhere that the runners can
find and install them. I'm not real interested in setting up a
persistent yum repo and builder, though; I'd like kdevops to
manage all that for me.

So I got out my whittling knife and built this proof of concept
where kdevops brings up a kernel builder node that is tailored for
a fast kernel build (eg, it is one guest with 16 vCPUs). The test
kernel is packaged (.rpm or .deb) and fetched to the control host,
then the builder node is destroyed.

I added a top-level "make" target that uploads the test kernel
packages to each test runner, and they install it.

This is not part of the patch series, but shows how to run it:

$ make mrproper defconfig-kernel-builder
$ make && make bringup && make linux-packages && make destroy
$ make defconfig-workflow-one
$ make && make bringup && make linux-artifacts && make destroy
$ make defconfig-workflow-two
$ make && make bringup && make linux-artifacts && make destroy
$ make defconfig-workflow-three
$ make && make bringup && make linux-artifacts && make destroy
$ make mrproper

All three workflows use the same kernel, built just once, for all
of their test runners.

This is still band-aids and chewing gum, but it seems to work on
both libvirt and cloud configurations. There is more than one way
to skin this cat, though.

Chuck Lever (3):
  Add a guest/instance for building the test kernel
  playbooks: Add a build_linux role
  Experimental: Add a separate install_linux role

 .gitignore                                    |   2 +
 playbooks/build_linux.yml                     |   4 +
 playbooks/install_linux.yml                   |   4 +
 playbooks/roles/build_linux/README.md         |  74 +++++
 playbooks/roles/build_linux/defaults/main.yml |  38 +++
 .../tasks/install-deps/debian/main.yml        |  46 +++
 .../tasks/install-deps/redhat/main.yml        | 102 ++++++
 .../tasks/install-deps/suse/main.yml          |  31 ++
 playbooks/roles/build_linux/tasks/main.yml    | 295 ++++++++++++++++++
 playbooks/roles/gen_hosts/defaults/main.yml   |   2 +
 playbooks/roles/gen_hosts/tasks/main.yml      |  12 +
 .../roles/gen_hosts/templates/builder.j2      |  13 +
 playbooks/roles/gen_nodes/defaults/main.yml   |   2 +
 playbooks/roles/gen_nodes/tasks/main.yml      |  22 ++
 playbooks/roles/install_linux/README.md       | 136 ++++++++
 .../roles/install_linux/defaults/main.yml     |  43 +++
 .../tasks/install-deps/debian/main.yml        |  44 +++
 .../tasks/install-deps/redhat/main.yml        |  76 +++++
 .../tasks/install-deps/suse/main.yml          |  31 ++
 playbooks/roles/install_linux/tasks/main.yml  | 142 +++++++++
 .../tasks/update-grub/debian.yml              |   8 +
 .../tasks/update-grub/install.yml             | 196 ++++++++++++
 .../install_linux/tasks/update-grub/main.yml  |  15 +
 .../tasks/update-grub/redhat.yml              |  36 +++
 .../install_linux/tasks/update-grub/suse.yml  |  11 +
 workflows/linux/Kconfig                       |  10 +
 workflows/linux/Makefile                      |  19 ++
 27 files changed, 1414 insertions(+)
 create mode 100644 playbooks/build_linux.yml
 create mode 100644 playbooks/install_linux.yml
 create mode 100644 playbooks/roles/build_linux/README.md
 create mode 100644 playbooks/roles/build_linux/defaults/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/debian/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/redhat/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/suse/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/main.yml
 create mode 100644 playbooks/roles/gen_hosts/templates/builder.j2
 create mode 100644 playbooks/roles/install_linux/README.md
 create mode 100644 playbooks/roles/install_linux/defaults/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/debian/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/redhat/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/suse/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/debian.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/install.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/redhat.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/suse.yml

-- 
2.49.0