From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 206F5635
	for <kdevops@lists.linux.dev>; Sat, 26 Jul 2025 18:00:56 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201
ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1753552857; cv=none; b=fz7AkmpRTSDr29153nipvhzVGQe3JgB2fTXfGCKHcvqawhDErThv3vQpon/4EGmosK3ucvmyXHsANafgkYkV9sOdCLDxaagRpoC+tFDLAWwvwh57n6lg3IF+wQTYtNtEPAaFonVxswY4LNBFPft2BE92DLEkaUrrWG6mqajsh5Y=
ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1753552857; c=relaxed/simple;
	bh=0j+YmKQTG92Fei6Ie5EGgrXBv+180YBSzOHuMofdmJI=;
	h=Message-ID:Date:MIME-Version:Subject:To:References:From:Cc:
	 In-Reply-To:Content-Type; b=Og5gToSycUxXZjWgGEfwLZgShpAoSVyiLelUzkBfmE86Ly4q4QTWDdRUmwRChuZY54t27cHWF5H9CItnunau3p4OLXAQe5blBPzPb4OjsmMua/TUWO55XTQ2TomiJR/ujR4onA2QjBxEFMKxuK8BFxEy6XVtMhg6fmKX1j8mfIM=
ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=eiJ6lthl; arc=none smtp.client-ip=10.30.226.201
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="eiJ6lthl"
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 42DF8C4CEED;
	Sat, 26 Jul 2025 18:00:56 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
	s=k20201202; t=1753552856;
	bh=0j+YmKQTG92Fei6Ie5EGgrXBv+180YBSzOHuMofdmJI=;
	h=Date:Subject:To:References:From:Cc:In-Reply-To:From;
	b=eiJ6lthlweS8dm5A+A29Kf2jY9utZo/1I0oaglGJd3J2AoP+Mnrk3rSyH+0HLQ4V4
	 0TYB14gMuxyoFGOus+BaQlqBxzvKyAv6peXPhniGbHHVTfDaY6c01AOqenr74bln83
	 juqY6U4ZFPZ8xoNscdQj3o21CXWHnFfGwKgW5O9N9ddPXdn7od/YYuHhMJqVR1Ng1a
	 vnDc90oO5ZMclEnDuPYMAraWcDKl+TsB2GDAlKdKuDPD543i13UYmUNy2Z/53mJ0s9
	 fFwKrLuL1tHd49UXVZKCxIGXtHvW/AZAjbVd53zvb7bZvOT+tJZPR30c4NO4lfe7LL
	 YXqG8MK/fJ12Q==
Message-ID: <edc97322-0091-4192-90d8-cd56dc660683@kernel.org>
Date: Sat, 26 Jul 2025 14:00:55 -0400
Precedence: bulk
X-Mailing-List: kdevops@lists.linux.dev
List-Id: <kdevops.lists.linux.dev>
List-Subscribe: <mailto:kdevops+subscribe@lists.linux.dev>
List-Unsubscribe: <mailto:kdevops+unsubscribe@lists.linux.dev>
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 4/4] bootlinux: add support for A/B kernel testing
To: Luis Chamberlain <mcgrof@kernel.org>
References: <20250726011653.2622672-1-mcgrof@kernel.org>
 <20250726011653.2622672-5-mcgrof@kernel.org>
Content-Language: en-US
From: Chuck Lever <cel@kernel.org>
Organization: kernel.org
Cc: Daniel Gomez <da.gomez@kruces.com>, kdevops@lists.linux.dev
In-Reply-To: <20250726011653.2622672-5-mcgrof@kernel.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

On 7/25/25 9:16 PM, Luis Chamberlain wrote:
> Right now we use the same kernel for all target nodes. We want to
> compare and contrast different kenrels for different features. We
> add support for A/B testing by leveraging the baseline and dev groups
> provided to us by KDEVOPS_BASELINE_AND_DEV.
> 
> This extends the bootlinux playbook by enabling us to allow a different
> kernel tree / ref to be used for the dev group. This just becomes a
> configuration thing. The targets are intuitive:
> 
>   make linux                 # Handles A/B compilation transparently
>   make linux-baseline        # Build and install baseline kernel only
>   make linux-dev             # Build and install development kernel only

My "build the kernel once and package it" patches are still under test,
but this patch conflicts heavily with that work. I'm not sure how to
reconcile it, but if you feel this patch is ready to merge now, let's
commit it and I will rework my set.

What my set does is change the operation of the "make linux" target so
that, depending on Kconfig settings, it either:

- Works as it does today (builds the kernel on each test runner)
- Builds the kernel and packages it on a separate node, saving the
  package
- Finds the packaged kernel and installs it on the test runners

Along the way there are a number of clean-ups suggested by Daniel,
including improving the selection of kernel .configs, and splitting
apart the tasks for 9p and non-9p builds.

It seems to me that we want to add a lot of complexity and nuance to
the "make linux" target. It might make sense to first split bootlinux
into three playbooks:

1. build this { URL, commit hash } tuple
2. install from the source tree or existing package
3. handle the grub details

Then A/B testing (and other new features) can be built on top of those
smaller plays.

Or, simply add a new "make linux-ab" target for this use case; I'm
guessing that the current set of "make linux" tasks might be utilized
far more frequently than this specialized case.

(Architecting the kdevops UX might be a little beyond the skill set of
AI code generators right at the moment)

More comments below...


> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
>  PROMPTS.md                                  |  48 ++++++
>  docs/kdevops-make-linux.md                  | 158 ++++++++++++++++++++
>  playbooks/roles/bootlinux/defaults/main.yml |  12 ++
>  playbooks/roles/bootlinux/tasks/main.yml    |  99 +++++++++++-
>  scripts/infer_last_stable_kernel.sh         |  35 +++++
>  workflows/linux/Kconfig                     | 116 +++++++++++++-
>  workflows/linux/Makefile                    |  39 +++++
>  7 files changed, 500 insertions(+), 7 deletions(-)
>  create mode 100755 scripts/infer_last_stable_kernel.sh
> 
> diff --git a/PROMPTS.md b/PROMPTS.md
> index a4ecf39..3fde7e9 100644
> --- a/PROMPTS.md
> +++ b/PROMPTS.md
> @@ -123,3 +123,51 @@ source "workflows/mmtests/Kconfig.thpchallenge"
>  source "workflows/mmtests/Kconfig.fs"
>  
>     This separation is preferred as it helps us scale.
> +
> +## Kernel development and A/B testing support
> +
> +### Adding A/B kernel testing support for different kernel versions
> +
> +**Prompt:**
> +We want to add support for when users enable KDEVOPS_BASELINE_AND_DEV we want
> +to extend workflows/linux/Kconfig with the a choise set of options to either a)
> +use the same kernel ref or b) allow the user to specify a different ref tag.
> +This will enable A/B testing with different kernel versions. When a different
> +kernel refs are desirable we will want to extend the compilation step and
> +installation of the Linux kernel in two steps. The first will be for the ref
> +and target of A (baseline tag) and the second will be for the target ref of B
> +(dev tag). However we want to fold these two steps in one for when
> +KDEVOPS_BASELINE_AND_DEV is used and make install is used, it would happen
> +transparently for us. The resulting linux kernel directory would end up with
> +the "dev" ref at the end. In case a user wants to re-compile a target ref for
> +baseline or dev we want to add (if we don't have already) a make linux-baseline
> +and make linux-dev so that we can build and install the target ref tag on the
> +baseline (A) or dev (B). The make linux target then would serially do make
> +linux-baseline and make linux-dev. Extend documentation for all this and also
> +add the respective prompt to PROMPTS.md once done. Avoid adding extra spaces to
> +code or documentation at the end of each line. These end up in red color on
> +diffs and hurt my eyes. Extend CLAUDE.md to understand styling for these rules
> +about not wanting lines ending in white space for styling.
> +
> +**AI:** Claude Code
> +**Commit:** [To be determined]
> +**Result:** Complete A/B kernel testing implementation with comprehensive configuration options.
> +**Grading:** 70%
> +
> +**Notes:**
> +
> +The implementation successfully added:
> +
> +1. **Makefile Implementation**: the AI failed to grasp the value of
> +   output yaml, and made ugly Makefile changes to extract variables.
> +
> +2. **Ansible Integration**: The AI failed to write the required changes on
> +   the ansible playbook at first. A secondary prompt made it just move the
> +   definitions to the ansible playbook but failed to address serially compiling
> +   linux for the baseline group first followed by the dev group after.
> +
> +3. **Documentation**: The AI is not grasping the preference to respect 80
> +   character lengths.
> +
> +4. **Menus**: The AI didn't do a good job at placing menus in a way that
> +   would make more intuitive sense for users.
> diff --git a/docs/kdevops-make-linux.md b/docs/kdevops-make-linux.md
> index e68eee5..8f54372 100644
> --- a/docs/kdevops-make-linux.md
> +++ b/docs/kdevops-make-linux.md
> @@ -13,3 +13,161 @@ To verify the kernel on it:
>  ```bash
>  make uname
>  ```
> +
> +## A/B Kernel Testing
> +
> +kdevops supports A/B testing with different kernel versions when
> +`KDEVOPS_BASELINE_AND_DEV` is enabled. This allows you to compare performance
> +or behavior between different kernel versions across baseline and development nodes.
> +
> +### Configuration Options
> +
> +When A/B testing is enabled, you can choose between two approaches:
> +
> +#### Same Kernel Reference (Default)
> +Use the same kernel tree and reference for both baseline and dev nodes:
> +```
> +A/B kernel testing configuration (BOOTLINUX_AB_SAME_REF) [Y/n/?]
> +```
> +
> +This is useful for testing configuration changes or different test parameters
> +with identical kernels.
> +
> +#### Different Kernel References
> +Use different kernel references for baseline and dev nodes:
> +```
> +A/B kernel testing configuration
> +  1. Use same kernel reference for baseline and dev (BOOTLINUX_AB_SAME_REF)
> +> 2. Use different kernel references for baseline and dev (BOOTLINUX_AB_DIFFERENT_REF)
> +```
> +
> +This enables testing between different kernel versions, commits, or branches.
> +
> +When using different references, configure:
> +- **Development kernel tree URL**: Git repository (defaults to baseline tree)
> +- **Development kernel reference**: Branch, tag, or commit (e.g., "v6.8", "linux-next")
> +- **Development kernel release/local version**: Custom version strings for identification
> +
> +### Make Targets
> +
> +#### Standard Linux Building
> +```bash
> +make linux                 # Build and install kernels for all nodes
> +```
> +
> +When A/B testing with different references is enabled, this automatically:
> +1. Builds and installs baseline kernel on baseline nodes
> +2. Builds and installs development kernel on dev nodes
> +3. Leaves the working directory with the dev kernel checked out
> +
> +#### Individual Node Targeting
> +```bash
> +make linux-baseline        # Build and install kernel for baseline nodes only
> +make linux-dev             # Build and install kernel for dev nodes only
> +```
> +
> +These targets are available when `KDEVOPS_BASELINE_AND_DEV=y` and allow
> +selective building and installation.
> +
> +### Usage Examples
> +
> +#### Testing Kernel Versions
> +Compare v6.7 (baseline) vs v6.8 (development):
> +
> +```bash
> +# Configure baseline kernel
> +menuconfig → Workflows → Linux kernel → Git tree to clone: linus
> +            Reference to use: v6.7
> +
> +# Configure A/B testing
> +menuconfig → Workflows → Linux kernel → A/B kernel testing
> +            → Use different kernel references
> +            → Development kernel reference: v6.8
> +
> +make bringup               # Provision baseline and dev nodes
> +make linux                 # Install v6.7 on baseline, v6.8 on dev
> +make fstests               # Run tests on both kernel versions
> +make fstests-compare       # Compare results between versions
> +```
> +
> +#### Testing Development Branches
> +Compare stable vs linux-next:
> +
> +```bash
> +# Baseline: stable kernel
> +menuconfig → Reference to use: v6.8
> +
> +# Development: linux-next
> +menuconfig → A/B kernel testing → Development kernel reference: linux-next
> +
> +make linux-baseline        # Install stable kernel on baseline nodes
> +make linux-dev             # Install linux-next on dev nodes
> +```
> +
> +#### Bisection Support
> +Test specific commits during bisection:
> +
> +```bash
> +# Update development reference for bisection
> +menuconfig → Development kernel reference: abc123def
> +
> +make linux-dev             # Install bisection commit on dev nodes
> +# Run tests and analyze results
> +```
> +
> +### Working Directory State
> +
> +After running `make linux` with different references:
> +- The Linux source directory contains the **development kernel** checkout
> +- Both baseline and dev nodes have their respective kernels installed
> +- Use `git log --oneline -5` to verify the current checkout
> +
> +To switch the working directory to baseline:
> +```bash
> +git checkout v6.7          # Switch to baseline reference
> +```
> +
> +### Integration with Testing Workflows
> +
> +A/B kernel testing integrates seamlessly with all kdevops testing workflows:
> +
> +```bash
> +# Run fstests with kernel comparison
> +make linux                 # Install different kernels
> +make fstests               # Test both kernel versions
> +make fstests-compare       # Generate comparison analysis
> +
> +# Run fio-tests with kernel comparison
> +make linux                 # Install different kernels
> +make fio-tests             # Performance test both kernels
> +make fio-tests-compare     # Compare performance metrics

I know that Chinner has made claims about how closely his QEMU-based
performance testing approaches the same results from bare metal testing.
Even so ...

This example is a little dangerous. fio results (especially latency)
will depend on there being enough idle CPU horsepower on the hypervisor
system. For cloud, all bets are off, because we have no control over the
other workloads on that physical host unless we have rented a bare metal
instance.

Even in the guestfs world, it's easy to set up a configuration where you
have provisioned more vCPUs than your system has threads / cores. There
can be other workload traffic on the system too, and you can be sure
that running the A and B tests at the same time will interfere with each
other.

"-compare" makes total sense for functional test results. For comparing
automatically generated performance results, there are loads of caveats.

(I rely on fio testing for NFS, so I'm pleased to see the fio-test
workflow -- but still, plenty of care must be taken before the results
can be used for inter-run comparison).


> +
> +# Run sysbench with kernel comparison
> +make linux                 # Install different kernels
> +make sysbench              # Database tests on both kernels
> +```
> +
> +### Best Practices
> +
> +1. **Version Identification**: Use descriptive kernel release versions to distinguish builds

This has been an ongoing problem for my "build once, package, and
install everywhere" set. How to identify which package to install,
in particular when the build step and the install step do not take
place during the same "make linux" run?

The kernel's Kconfig has the CONFIG_LOCALVERSION_AUTO option which
adds the 12-hexit commit hash to the name of the kernel:

   6.16.0-rc6-00005-g6b59765c97a3

Which:
- Gives us an identifier tied to a specific version of the code base
- rpmbuild makes part of the kernel package name, like so:

   kernel-6.12.40_rc1_g596aae841edf-2.x86_64.rpm

But CONFIG_LOCALVERSION_AUTO does not identify the particular .config
used to build the kernel. For instance you might want to A/B the same
code base built with different .configs. Or, maybe you are explicitly
intending not to support multiple .configs here.


> +2. **Sequential Testing**: Install kernels before running test workflows
> +3. **Result Organization**: Use baseline/dev labels in test result analysis
> +4. **Git Management**: Keep track of which reference is currently checked out
> +5. **Systematic Comparison**: Use `*-compare` targets for meaningful analysis
> +
> +### Troubleshooting
> +
> +#### Build Failures
> +- Ensure both kernel references are valid and accessible
> +- Check that build dependencies are installed on all nodes
> +- Verify git repository permissions and network connectivity
> +
> +#### Version Conflicts
> +- Use different `kernelrelease` and `localversion` settings for clear identification
> +- Check `/boot` directory for kernel installation conflicts
> +- Verify GRUB configuration after kernel installation
> +
> +#### Node Targeting Issues
> +- Confirm `KDEVOPS_BASELINE_AND_DEV=y` is enabled
> +- Verify baseline and dev node groups exist in inventory
> +- Check ansible host patterns with `make linux-baseline HOSTS=baseline`
> diff --git a/playbooks/roles/bootlinux/defaults/main.yml b/playbooks/roles/bootlinux/defaults/main.yml
> index fd5674b..4146292 100644
> --- a/playbooks/roles/bootlinux/defaults/main.yml
> +++ b/playbooks/roles/bootlinux/defaults/main.yml
> @@ -52,3 +52,15 @@ kdevops_workflow_enable_cxl: False
>  
>  bootlinux_cxl_test: False
>  bootlinux_tree_set_by_cli: False
> +
> +# A/B testing defaults
> +bootlinux_ab_same_ref: True
> +bootlinux_ab_different_ref: False
> +
> +# Development kernel settings (used when bootlinux_ab_different_ref is True)
> +bootlinux_dev_tree: ""
> +bootlinux_dev_tree_ref: "master"
> +bootlinux_dev_tree_kernelrelease: ""
> +bootlinux_dev_tree_localversion: ""
> +bootlinux_tree_custom_kernelrelease: False
> +bootlinux_tree_custom_localversion: false
> diff --git a/playbooks/roles/bootlinux/tasks/main.yml b/playbooks/roles/bootlinux/tasks/main.yml
> index 7671389..283ac88 100644
> --- a/playbooks/roles/bootlinux/tasks/main.yml
> +++ b/playbooks/roles/bootlinux/tasks/main.yml
> @@ -61,6 +61,74 @@
>    when:
>      - not kdevops_baseline_and_dev|bool
>  
> +- name: Determine if this is a dev node for A/B testing
> +  set_fact:
> +    bootlinux_is_dev_node: "{{ ansible_hostname | regex_search('^.*-dev$') is not none }}"
> +  when:
> +    - kdevops_baseline_and_dev|bool
> +    - bootlinux_ab_different_ref|bool
> +
> +- name: Set development group full custom kernel release
> +  set_fact:
> +    target_linux_kernelrelease: "{{ bootlinux_dev_tree_kernelrelease if bootlinux_dev_tree_kernelrelease != '' else target_linux_kernelrelease }}"
> +  when:
> +    - kdevops_baseline_and_dev|bool
> +    - bootlinux_ab_different_ref|bool
> +    - bootlinux_tree_custom_kernelrelease|bool
> +    - bootlinux_is_dev_node|default(false)|bool
> +
> +- name: Set development group local append version
> +  set_fact:
> +    target_linux_localversion: "{{ bootlinux_dev_tree_localversion if bootlinux_dev_tree_localversion != '' else target_linux_localversion }}"
> +  when:
> +    - kdevops_baseline_and_dev|bool
> +    - bootlinux_ab_different_ref|bool
> +    - bootlinux_tree_custom_localversion|bool
> +    - bootlinux_is_dev_node|default(false)|bool
> +
> +- name: Set development kernel parameters for dev nodes
> +  set_fact:
> +    target_linux_git: "{{ bootlinux_dev_tree if bootlinux_dev_tree != '' else target_linux_git }}"
> +    target_linux_ref: "{{ bootlinux_dev_tree_ref }}"
> +    target_linux_config: "config-{{ bootlinux_dev_tree_ref }}"
> +  when:
> +    - kdevops_baseline_and_dev|bool
> +    - bootlinux_ab_different_ref|bool
> +    - bootlinux_is_dev_node|default(false)|bool
> +
> +- name: Determine active kernel parameters for A/B testing with 9P
> +  set_fact:
> +    active_linux_ref: "{{ bootlinux_dev_tree_ref }}"
> +  when:
> +    - kdevops_baseline_and_dev|bool
> +    - bootlinux_ab_different_ref|bool
> +    - bootlinux_tree_custom_kernelrelease|bool
> +    - bootlinux_9p|bool
> +  run_once: true
> +  delegate_to: localhost
> +
> +- name: Determine full custom kernel release for A/B testing with 9P
> +  set_fact:
> +    active_linux_kernelrelease: "{{ hostvars[groups['dev'][0]]['target_linux_kernelrelease'] if hostvars[groups['dev'][0]]['target_linux_kernelrelease'] is defined else target_linux_kernelrelease }}"
> +  when:
> +    - kdevops_baseline_and_dev|bool
> +    - bootlinux_ab_different_ref|bool
> +    - bootlinux_tree_custom_kernelrelease|bool
> +    - bootlinux_9p|bool
> +  run_once: true
> +  delegate_to: localhost
> +
> +- name: Determine localversion kernel release for A/B testing with 9P
> +  set_fact:
> +    active_linux_localversion: "{{ hostvars[groups['dev'][0]]['target_linux_localversion'] if hostvars[groups['dev'][0]]['target_linux_localversion'] is defined else target_linux_localversion }}"
> +  when:
> +    - kdevops_baseline_and_dev|bool
> +    - bootlinux_ab_different_ref|bool
> +    - bootlinux_tree_custom_localversion|bool
> +    - bootlinux_9p|bool
> +  run_once: true
> +  delegate_to: localhost
> +
>  - include_role:
>      name: create_data_partition
>  
> @@ -412,6 +480,20 @@
>      - not bootlinux_9p|bool
>      - snaik_oil_file.stat.exists
>  
> +- name: Checkout correct git ref for A/B testing with 9P
> +  git:
> +    repo: "{{ target_linux_git }}"
> +    dest: "{{ bootlinux_9p_host_path }}"
> +    version: "{{ active_linux_ref | default(target_linux_ref) }}"
> +    force: yes
> +  tags: [ 'build-linux' ]
> +  when:
> +    - bootlinux_9p|bool
> +    - kdevops_baseline_and_dev|bool
> +    - bootlinux_ab_different_ref|bool
> +  run_once: true
> +  delegate_to: localhost
> +
>  - name: Get nproc on the control node
>    command: "{{ num_jobs }}"
>    tags: [ 'build-linux', 'cxl-build' ]
> @@ -429,21 +511,24 @@
>    tags: [ 'build-linux' ]
>    when:
>      - bootlinux_9p|bool
> -    - target_linux_kernelrelease | length > 0
> +    - (active_linux_kernelrelease | default(target_linux_kernelrelease)) | length > 0
>    run_once: true
>    delegate_to: localhost
>  
> -- name: Generate user kernelrelease {{ target_linux_kernelversion.stdout }}-{{ target_linux_kernelrelease }}
> +- name: Generate user kernelrelease
>    set_fact:
> -    target_user_kernelrelease: "{{ target_linux_kernelversion.stdout }}-{{ target_linux_kernelrelease }}"
> +    target_user_kernelrelease: "{{ target_linux_kernelversion.stdout }}-{{ active_linux_kernelrelease | default(target_linux_kernelrelease) }}"
>    tags: [ 'build-linux' ]
>    when:
>      - bootlinux_9p|bool
> -    - target_linux_kernelrelease | length > 0
> +    - bootlinux_tree_custom_kernelrelease|bool
> +    - (active_linux_kernelrelease | default(target_linux_kernelrelease)) | length > 0
> +    - target_linux_kernelversion is defined
> +    - target_linux_kernelversion.stdout is defined
>    run_once: true
>    delegate_to: localhost
>  
> -- name: Build {{ target_linux_tree }} {{ target_user_kernelrelease }} on the control node using {{ nproc_9p.stdout }} threads
> +- name: Build {{ target_linux_tree }} with custom kernel release on the control node using {{ nproc_9p.stdout }} threads
>    make:
>      jobs: "{{ nproc_9p.stdout }}"
>      chdir: "{{ bootlinux_9p_host_path }}"
> @@ -452,11 +537,13 @@
>    tags: [ 'build-linux' ]
>    when:
>      - bootlinux_9p|bool
> +    - bootlinux_tree_custom_kernelrelease|bool
>      - target_linux_kernelrelease | length > 0
> +    - target_user_kernelrelease is defined
>    run_once: true
>    delegate_to: localhost
>  
> -- name: Build {{ target_linux_tree }} {{ target_user_kernelrelease }} on the control node using {{ nproc_9p.stdout }} threads
> +- name: Build {{ target_linux_tree }} on the control node using {{ nproc_9p.stdout }} threads
>    make:
>      jobs: "{{ nproc_9p.stdout }}"
>      chdir: "{{ bootlinux_9p_host_path }}"
> diff --git a/scripts/infer_last_stable_kernel.sh b/scripts/infer_last_stable_kernel.sh
> new file mode 100755
> index 0000000..9cc19a9
> --- /dev/null
> +++ b/scripts/infer_last_stable_kernel.sh
> @@ -0,0 +1,35 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: copyleft-next-0.3.1
> +
> +# This script infers the last stable kernel version from the git repository.
> +# It looks for the most recent non-rc tag (e.g., v6.14, v6.13) that would
> +# be a good default for A/B testing with different kernel references.
> +
> +GIT_TREE="${1:-/mirror/linux.git}"
> +
> +if [ ! -d "$GIT_TREE" ]; then
> +    echo "v6.12"  # fallback if no git tree available
> +    exit 0
> +fi
> +
> +# Get all v6.x tags, excluding release candidates
> +# Sort them by version and get the last stable release
> +LAST_STABLE=$(git --git-dir="$GIT_TREE" tag --list 'v6.*' | \
> +    grep -v -- '-rc' | \
> +    sort -V | \
> +    tail -1)
> +
> +if [ -z "$LAST_STABLE" ]; then
> +    # If no stable v6.x found, try v5.x as fallback
> +    LAST_STABLE=$(git --git-dir="$GIT_TREE" tag --list 'v5.*' | \
> +        grep -v -- '-rc' | \
> +        sort -V | \
> +        tail -1)
> +fi
> +
> +# Final fallback if nothing found
> +if [ -z "$LAST_STABLE" ]; then
> +    echo "v6.12"
> +else
> +    echo "$LAST_STABLE"
> +fi
> \ No newline at end of file
> diff --git a/workflows/linux/Kconfig b/workflows/linux/Kconfig
> index 183ac77..d08270e 100644
> --- a/workflows/linux/Kconfig
> +++ b/workflows/linux/Kconfig
> @@ -139,6 +139,41 @@ config BOOTLINUX_CUSTOM
>  
>  endchoice
>  
> +if KDEVOPS_BASELINE_AND_DEV
> +
> +choice
> +	prompt "A/B kernel testing configuration"
> +	default BOOTLINUX_AB_DIFFERENT_REF
> +	help
> +	  When A/B testing is enabled, you can choose to use the same
> +	  kernel reference for both baseline and dev nodes, or specify
> +	  different kernel references to test different kernel versions.
> +	  We default to assuming you want to test a different kernel on
> +	  each.
> +
> +config BOOTLINUX_AB_SAME_REF
> +	bool "Use same kernel reference for baseline and dev"
> +	output yaml
> +	help
> +	  Use the same kernel tree and reference for both baseline and
> +	  development nodes. This is useful for testing configuration
> +	  changes or different test parameters with the same kernel.
> +
> +config BOOTLINUX_AB_DIFFERENT_REF
> +	bool "Use different kernel references for baseline and dev"
> +	output yaml
> +	help
> +	  Use different kernel references for baseline and development
> +	  nodes. This enables testing between different kernel versions,
> +	  commits, or branches. The baseline will use the main configured
> +	  kernel reference, while dev uses a separate reference.
> +
> +endchoice
> +
> +endif
> +
> +menu "A -     main    group kernel configuration"
> +
>  source "workflows/linux/Kconfig.linus"
>  source "workflows/linux/Kconfig.stable"
>  source "workflows/linux/Kconfig.dev"
> @@ -180,6 +215,65 @@ config BOOTLINUX_TREE_CUSTOM_REF
>  
>  endif # BOOTLINUX_CUSTOM
>  
> +endmenu
> +
> +if KDEVOPS_BASELINE_AND_DEV
> +
> +if BOOTLINUX_AB_DIFFERENT_REF
> +
> +menu "B - development group kernel configuration"
> +
> +config BOOTLINUX_DEV_TREE
> +	string "B group development kernel tree URL"
> +	output yaml
> +	default BOOTLINUX_TREE
> +	help
> +	  Git tree URL for the development kernel. If left empty or same
> +	  as the baseline tree, the same tree will be used with a different
> +	  reference. This allows testing different branches or forks.
> +
> +config BOOTLINUX_DEV_TREE_REF
> +	string "B group development kernel reference"
> +	output yaml
> +	default $(shell, scripts/infer_last_stable_kernel.sh)
> +	help
> +	  Git reference (branch, tag, or commit) for the development kernel.
> +	  This should be different from the baseline reference to enable
> +	  meaningful A/B comparison between kernel versions.
> +
> +	  The default is automatically inferred as the most recent stable
> +	  kernel version (e.g., v6.15) from the git repository.
> +
> +	  Examples:
> +	  - "v6.8" (stable release)
> +	  - "linux-next" (latest development)
> +	  - "v6.7..v6.8" (range for bisection)
> +	  - commit SHA (specific commit)
> +
> +config BOOTLINUX_DEV_TREE_KERNELRELEASE
> +	string "Development kernel release version"
> +	depends on BOOTLINUX_TREE_CUSTOM_KERNELRELEASE
> +	output yaml
> +	help
> +	  The string here (e.g. 'devel') will be appended to the result of make
> +	  kernelversion. Example: '6.8.0-rc3-devel' but only for the dev group.
> +	  Leave it empty unless you want a custom tag at the end.
> +
> +config BOOTLINUX_DEV_TREE_LOCALVERSION
> +	string "Development kernel local version"
> +	output yaml
> +	depends on BOOTLINUX_TREE_CUSTOM_LOCALVERSION
> +	default BOOTLINUX_TREE_LOCALVERSION
> +	help
> +	  The Linux local version to use for the development kernel (for uname).
> +	  If left empty, will use the same as baseline.
> +
> +endmenu
> +
> +endif # BOOTLINUX_AB_DIFFERENT_REF
> +
> +endif # KDEVOPS_BASELINE_AND_DEV
> +
>  # This ends up being the directory name used for the /data/ partition
>  # where linux is deployed on the nodes.
>  config BOOTLINUX_TREE_NAME
> @@ -264,23 +358,39 @@ config BOOTLINUX_TREE_REF
>  	default BOOTLINUX_TREE_CEL_LINUX_REF if BOOTLINUX_TREE_CEL_LINUX
>  	default BOOTLINUX_TREE_CUSTOM_REF if BOOTLINUX_CUSTOM
>  
> +config BOOTLINUX_TREE_CUSTOM_KERNELRELEASE
> +	bool "Do you want a full custom kernel release name?"
> +	output yaml
> +	help
> +	  Do you want a full custom Linux kernel release which will be output
> +	  through uname?
> +
>  config BOOTLINUX_TREE_KERNELRELEASE
>  	string "Linux kernel release version to use"
> +	depends on BOOTLINUX_TREE_CUSTOM_KERNELRELEASE
>  	help
>  	  The Linux kernel release version to use (for uname).
>  
>  	  The string here (e.g. 'devel') will be appended to the result of make
>  	  kernelversion. Example: '6.8.0-rc3-devel'
>  
> +config BOOTLINUX_TREE_CUSTOM_LOCALVERSION
> +	bool "Do you want to append a custom kernel release tag?"
> +	output yaml
> +	help
> +	  Do you want a full custom Linux kernel release which will be output
> +	  through uname?
>  
>  config BOOTLINUX_TREE_LOCALVERSION
>  	string "Linux local version to use"
> +	depends on BOOTLINUX_TREE_CUSTOM_LOCALVERSION
>  	help
>  	  The Linux local version to use (for uname).
>  
>  config BOOTLINUX_SHALLOW_CLONE
>  	bool "Shallow git clone"
> -	default y
> +	default y if !KDEVOPS_BASELINE_AND_DEV
> +	depends on !BOOTLINUX_AB_DIFFERENT_REF
>  	help
>  	  If enabled the git tree cloned with be cloned using a shallow tree
>  	  with history truncated. You want to enable this if you really don't
> @@ -291,6 +401,10 @@ config BOOTLINUX_SHALLOW_CLONE
>  	  just using the targets as dummy target runners and don't expect to
>  	  be using 'git log' on the target guests.
>  
> +	  This option is automatically disabled when using A/B testing with
> +	  different kernel references, as shallow clones may not contain all
> +	  the required refs for checkout.
> +
>  config BOOTLINUX_SHALLOW_CLONE_DEPTH
>  	int "Shallow git clone depth"
>  	default 30 if BOOTLINUX_TREE_SET_BY_CLI
> diff --git a/workflows/linux/Makefile b/workflows/linux/Makefile
> index f68c090..ab8bcbb 100644
> --- a/workflows/linux/Makefile
> +++ b/workflows/linux/Makefile
> @@ -71,6 +71,10 @@ PHONY +=  linux-help-menu
>  linux-help-menu:
>  	@echo "Linux git kernel development options"
>  	@echo "linux              - Git clones a linux git tree, build Linux, installs and reboots into it"
> +	@if [[ "$(CONFIG_KDEVOPS_BASELINE_AND_DEV)" == "y" ]]; then \
> +		echo "linux-baseline     - Build and install kernel for baseline nodes only" ;\
> +		echo "linux-dev          - Build and install kernel for dev nodes only" ;\
> +	fi
>  	@if [[ "$(CONFIG_BOOTLINUX_9P)" == "y" ]]; then \
>  		echo "linux-mount        - Mounts 9p path on targets" ;\
>  	fi
> @@ -89,10 +93,45 @@ linux-help-end:
>  LINUX_HELP_EXTRA :=
>  
>  PHONY += linux
> +ifeq (y,$(CONFIG_KDEVOPS_BASELINE_AND_DEV))
> +ifeq (y,$(CONFIG_BOOTLINUX_AB_DIFFERENT_REF))
> +linux: linux-baseline linux-dev
> +else
> +linux: $(KDEVOPS_NODES)
> +	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i \
> +		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
> +		--extra-vars="$(BOOTLINUX_ARGS)" $(LIMIT_HOSTS)
> +endif
> +else
>  linux: $(KDEVOPS_NODES)
>  	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i \
>  		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
>  		--extra-vars="$(BOOTLINUX_ARGS)" $(LIMIT_HOSTS)
> +endif
> +
> +PHONY += linux-baseline
> +ifeq (y,$(CONFIG_KDEVOPS_BASELINE_AND_DEV))
> +linux-baseline: $(KDEVOPS_NODES)
> +	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i \
> +		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
> +		--extra-vars="$(BOOTLINUX_ARGS)" --limit baseline
> +else
> +linux-baseline:
> +	@echo "linux-baseline requires KDEVOPS_BASELINE_AND_DEV=y"
> +	@exit 1
> +endif
> +
> +PHONY += linux-dev
> +ifeq (y,$(CONFIG_KDEVOPS_BASELINE_AND_DEV))
> +linux-dev: $(KDEVOPS_NODES)
> +	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i \
> +		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
> +		--extra-vars="$(BOOTLINUX_ARGS)" --limit dev
> +else
> +linux-dev:
> +	@echo "linux-dev requires KDEVOPS_BASELINE_AND_DEV=y"
> +	@exit 1
> +endif
>  
>  PHONY += linux-mount
>  linux-mount:


-- 
Chuck Lever