[PATCH 4/4] bootlinux: add support for A/B kernel testing

[Luis Chamberlain <mcgrof@kernel.org>]

Right now we use the same kernel for all target nodes. We want to
compare and contrast different kenrels for different features. We
add support for A/B testing by leveraging the baseline and dev groups
provided to us by KDEVOPS_BASELINE_AND_DEV.

This extends the bootlinux playbook by enabling us to allow a different
kernel tree / ref to be used for the dev group. This just becomes a
configuration thing. The targets are intuitive:

  make linux                 # Handles A/B compilation transparently
  make linux-baseline        # Build and install baseline kernel only
  make linux-dev             # Build and install development kernel only

Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 PROMPTS.md                                  |  48 ++++++
 docs/kdevops-make-linux.md                  | 158 ++++++++++++++++++++
 playbooks/roles/bootlinux/defaults/main.yml |  12 ++
 playbooks/roles/bootlinux/tasks/main.yml    |  99 +++++++++++-
 scripts/infer_last_stable_kernel.sh         |  35 +++++
 workflows/linux/Kconfig                     | 116 +++++++++++++-
 workflows/linux/Makefile                    |  39 +++++
 7 files changed, 500 insertions(+), 7 deletions(-)
 create mode 100755 scripts/infer_last_stable_kernel.sh

diff --git a/PROMPTS.md b/PROMPTS.md
index a4ecf39..3fde7e9 100644
--- a/PROMPTS.md
+++ b/PROMPTS.md
@@ -123,3 +123,51 @@ source "workflows/mmtests/Kconfig.thpchallenge"
 source "workflows/mmtests/Kconfig.fs"
 
    This separation is preferred as it helps us scale.
+
+## Kernel development and A/B testing support
+
+### Adding A/B kernel testing support for different kernel versions
+
+**Prompt:**
+We want to add support for when users enable KDEVOPS_BASELINE_AND_DEV we want
+to extend workflows/linux/Kconfig with the a choise set of options to either a)
+use the same kernel ref or b) allow the user to specify a different ref tag.
+This will enable A/B testing with different kernel versions. When a different
+kernel refs are desirable we will want to extend the compilation step and
+installation of the Linux kernel in two steps. The first will be for the ref
+and target of A (baseline tag) and the second will be for the target ref of B
+(dev tag). However we want to fold these two steps in one for when
+KDEVOPS_BASELINE_AND_DEV is used and make install is used, it would happen
+transparently for us. The resulting linux kernel directory would end up with
+the "dev" ref at the end. In case a user wants to re-compile a target ref for
+baseline or dev we want to add (if we don't have already) a make linux-baseline
+and make linux-dev so that we can build and install the target ref tag on the
+baseline (A) or dev (B). The make linux target then would serially do make
+linux-baseline and make linux-dev. Extend documentation for all this and also
+add the respective prompt to PROMPTS.md once done. Avoid adding extra spaces to
+code or documentation at the end of each line. These end up in red color on
+diffs and hurt my eyes. Extend CLAUDE.md to understand styling for these rules
+about not wanting lines ending in white space for styling.
+
+**AI:** Claude Code
+**Commit:** [To be determined]
+**Result:** Complete A/B kernel testing implementation with comprehensive configuration options.
+**Grading:** 70%
+
+**Notes:**
+
+The implementation successfully added:
+
+1. **Makefile Implementation**: the AI failed to grasp the value of
+   output yaml, and made ugly Makefile changes to extract variables.
+
+2. **Ansible Integration**: The AI failed to write the required changes on
+   the ansible playbook at first. A secondary prompt made it just move the
+   definitions to the ansible playbook but failed to address serially compiling
+   linux for the baseline group first followed by the dev group after.
+
+3. **Documentation**: The AI is not grasping the preference to respect 80
+   character lengths.
+
+4. **Menus**: The AI didn't do a good job at placing menus in a way that
+   would make more intuitive sense for users.
diff --git a/docs/kdevops-make-linux.md b/docs/kdevops-make-linux.md
index e68eee5..8f54372 100644
--- a/docs/kdevops-make-linux.md
+++ b/docs/kdevops-make-linux.md
@@ -13,3 +13,161 @@ To verify the kernel on it:
 ```bash
 make uname
 ```
+
+## A/B Kernel Testing
+
+kdevops supports A/B testing with different kernel versions when
+`KDEVOPS_BASELINE_AND_DEV` is enabled. This allows you to compare performance
+or behavior between different kernel versions across baseline and development nodes.
+
+### Configuration Options
+
+When A/B testing is enabled, you can choose between two approaches:
+
+#### Same Kernel Reference (Default)
+Use the same kernel tree and reference for both baseline and dev nodes:
+```
+A/B kernel testing configuration (BOOTLINUX_AB_SAME_REF) [Y/n/?]
+```
+
+This is useful for testing configuration changes or different test parameters
+with identical kernels.
+
+#### Different Kernel References
+Use different kernel references for baseline and dev nodes:
+```
+A/B kernel testing configuration
+  1. Use same kernel reference for baseline and dev (BOOTLINUX_AB_SAME_REF)
+> 2. Use different kernel references for baseline and dev (BOOTLINUX_AB_DIFFERENT_REF)
+```
+
+This enables testing between different kernel versions, commits, or branches.
+
+When using different references, configure:
+- **Development kernel tree URL**: Git repository (defaults to baseline tree)
+- **Development kernel reference**: Branch, tag, or commit (e.g., "v6.8", "linux-next")
+- **Development kernel release/local version**: Custom version strings for identification
+
+### Make Targets
+
+#### Standard Linux Building
+```bash
+make linux                 # Build and install kernels for all nodes
+```
+
+When A/B testing with different references is enabled, this automatically:
+1. Builds and installs baseline kernel on baseline nodes
+2. Builds and installs development kernel on dev nodes
+3. Leaves the working directory with the dev kernel checked out
+
+#### Individual Node Targeting
+```bash
+make linux-baseline        # Build and install kernel for baseline nodes only
+make linux-dev             # Build and install kernel for dev nodes only
+```
+
+These targets are available when `KDEVOPS_BASELINE_AND_DEV=y` and allow
+selective building and installation.
+
+### Usage Examples
+
+#### Testing Kernel Versions
+Compare v6.7 (baseline) vs v6.8 (development):
+
+```bash
+# Configure baseline kernel
+menuconfig → Workflows → Linux kernel → Git tree to clone: linus
+            Reference to use: v6.7
+
+# Configure A/B testing
+menuconfig → Workflows → Linux kernel → A/B kernel testing
+            → Use different kernel references
+            → Development kernel reference: v6.8
+
+make bringup               # Provision baseline and dev nodes
+make linux                 # Install v6.7 on baseline, v6.8 on dev
+make fstests               # Run tests on both kernel versions
+make fstests-compare       # Compare results between versions
+```
+
+#### Testing Development Branches
+Compare stable vs linux-next:
+
+```bash
+# Baseline: stable kernel
+menuconfig → Reference to use: v6.8
+
+# Development: linux-next
+menuconfig → A/B kernel testing → Development kernel reference: linux-next
+
+make linux-baseline        # Install stable kernel on baseline nodes
+make linux-dev             # Install linux-next on dev nodes
+```
+
+#### Bisection Support
+Test specific commits during bisection:
+
+```bash
+# Update development reference for bisection
+menuconfig → Development kernel reference: abc123def
+
+make linux-dev             # Install bisection commit on dev nodes
+# Run tests and analyze results
+```
+
+### Working Directory State
+
+After running `make linux` with different references:
+- The Linux source directory contains the **development kernel** checkout
+- Both baseline and dev nodes have their respective kernels installed
+- Use `git log --oneline -5` to verify the current checkout
+
+To switch the working directory to baseline:
+```bash
+git checkout v6.7          # Switch to baseline reference
+```
+
+### Integration with Testing Workflows
+
+A/B kernel testing integrates seamlessly with all kdevops testing workflows:
+
+```bash
+# Run fstests with kernel comparison
+make linux                 # Install different kernels
+make fstests               # Test both kernel versions
+make fstests-compare       # Generate comparison analysis
+
+# Run fio-tests with kernel comparison
+make linux                 # Install different kernels
+make fio-tests             # Performance test both kernels
+make fio-tests-compare     # Compare performance metrics
+
+# Run sysbench with kernel comparison
+make linux                 # Install different kernels
+make sysbench              # Database tests on both kernels
+```
+
+### Best Practices
+
+1. **Version Identification**: Use descriptive kernel release versions to distinguish builds
+2. **Sequential Testing**: Install kernels before running test workflows
+3. **Result Organization**: Use baseline/dev labels in test result analysis
+4. **Git Management**: Keep track of which reference is currently checked out
+5. **Systematic Comparison**: Use `*-compare` targets for meaningful analysis
+
+### Troubleshooting
+
+#### Build Failures
+- Ensure both kernel references are valid and accessible
+- Check that build dependencies are installed on all nodes
+- Verify git repository permissions and network connectivity
+
+#### Version Conflicts
+- Use different `kernelrelease` and `localversion` settings for clear identification
+- Check `/boot` directory for kernel installation conflicts
+- Verify GRUB configuration after kernel installation
+
+#### Node Targeting Issues
+- Confirm `KDEVOPS_BASELINE_AND_DEV=y` is enabled
+- Verify baseline and dev node groups exist in inventory
+- Check ansible host patterns with `make linux-baseline HOSTS=baseline`
diff --git a/playbooks/roles/bootlinux/defaults/main.yml b/playbooks/roles/bootlinux/defaults/main.yml
index fd5674b..4146292 100644
--- a/playbooks/roles/bootlinux/defaults/main.yml
+++ b/playbooks/roles/bootlinux/defaults/main.yml
@@ -52,3 +52,15 @@ kdevops_workflow_enable_cxl: False
 
 bootlinux_cxl_test: False
 bootlinux_tree_set_by_cli: False
+
+# A/B testing defaults
+bootlinux_ab_same_ref: True
+bootlinux_ab_different_ref: False
+
+# Development kernel settings (used when bootlinux_ab_different_ref is True)
+bootlinux_dev_tree: ""
+bootlinux_dev_tree_ref: "master"
+bootlinux_dev_tree_kernelrelease: ""
+bootlinux_dev_tree_localversion: ""
+bootlinux_tree_custom_kernelrelease: False
+bootlinux_tree_custom_localversion: false
diff --git a/playbooks/roles/bootlinux/tasks/main.yml b/playbooks/roles/bootlinux/tasks/main.yml
index 7671389..283ac88 100644
--- a/playbooks/roles/bootlinux/tasks/main.yml
+++ b/playbooks/roles/bootlinux/tasks/main.yml
@@ -61,6 +61,74 @@
   when:
     - not kdevops_baseline_and_dev|bool
 
+- name: Determine if this is a dev node for A/B testing
+  set_fact:
+    bootlinux_is_dev_node: "{{ ansible_hostname | regex_search('^.*-dev$') is not none }}"
+  when:
+    - kdevops_baseline_and_dev|bool
+    - bootlinux_ab_different_ref|bool
+
+- name: Set development group full custom kernel release
+  set_fact:
+    target_linux_kernelrelease: "{{ bootlinux_dev_tree_kernelrelease if bootlinux_dev_tree_kernelrelease != '' else target_linux_kernelrelease }}"
+  when:
+    - kdevops_baseline_and_dev|bool
+    - bootlinux_ab_different_ref|bool
+    - bootlinux_tree_custom_kernelrelease|bool
+    - bootlinux_is_dev_node|default(false)|bool
+
+- name: Set development group local append version
+  set_fact:
+    target_linux_localversion: "{{ bootlinux_dev_tree_localversion if bootlinux_dev_tree_localversion != '' else target_linux_localversion }}"
+  when:
+    - kdevops_baseline_and_dev|bool
+    - bootlinux_ab_different_ref|bool
+    - bootlinux_tree_custom_localversion|bool
+    - bootlinux_is_dev_node|default(false)|bool
+
+- name: Set development kernel parameters for dev nodes
+  set_fact:
+    target_linux_git: "{{ bootlinux_dev_tree if bootlinux_dev_tree != '' else target_linux_git }}"
+    target_linux_ref: "{{ bootlinux_dev_tree_ref }}"
+    target_linux_config: "config-{{ bootlinux_dev_tree_ref }}"
+  when:
+    - kdevops_baseline_and_dev|bool
+    - bootlinux_ab_different_ref|bool
+    - bootlinux_is_dev_node|default(false)|bool
+
+- name: Determine active kernel parameters for A/B testing with 9P
+  set_fact:
+    active_linux_ref: "{{ bootlinux_dev_tree_ref }}"
+  when:
+    - kdevops_baseline_and_dev|bool
+    - bootlinux_ab_different_ref|bool
+    - bootlinux_tree_custom_kernelrelease|bool
+    - bootlinux_9p|bool
+  run_once: true
+  delegate_to: localhost
+
+- name: Determine full custom kernel release for A/B testing with 9P
+  set_fact:
+    active_linux_kernelrelease: "{{ hostvars[groups['dev'][0]]['target_linux_kernelrelease'] if hostvars[groups['dev'][0]]['target_linux_kernelrelease'] is defined else target_linux_kernelrelease }}"
+  when:
+    - kdevops_baseline_and_dev|bool
+    - bootlinux_ab_different_ref|bool
+    - bootlinux_tree_custom_kernelrelease|bool
+    - bootlinux_9p|bool
+  run_once: true
+  delegate_to: localhost
+
+- name: Determine localversion kernel release for A/B testing with 9P
+  set_fact:
+    active_linux_localversion: "{{ hostvars[groups['dev'][0]]['target_linux_localversion'] if hostvars[groups['dev'][0]]['target_linux_localversion'] is defined else target_linux_localversion }}"
+  when:
+    - kdevops_baseline_and_dev|bool
+    - bootlinux_ab_different_ref|bool
+    - bootlinux_tree_custom_localversion|bool
+    - bootlinux_9p|bool
+  run_once: true
+  delegate_to: localhost
+
 - include_role:
     name: create_data_partition
 
@@ -412,6 +480,20 @@
     - not bootlinux_9p|bool
     - snaik_oil_file.stat.exists
 
+- name: Checkout correct git ref for A/B testing with 9P
+  git:
+    repo: "{{ target_linux_git }}"
+    dest: "{{ bootlinux_9p_host_path }}"
+    version: "{{ active_linux_ref | default(target_linux_ref) }}"
+    force: yes
+  tags: [ 'build-linux' ]
+  when:
+    - bootlinux_9p|bool
+    - kdevops_baseline_and_dev|bool
+    - bootlinux_ab_different_ref|bool
+  run_once: true
+  delegate_to: localhost
+
 - name: Get nproc on the control node
   command: "{{ num_jobs }}"
   tags: [ 'build-linux', 'cxl-build' ]
@@ -429,21 +511,24 @@
   tags: [ 'build-linux' ]
   when:
     - bootlinux_9p|bool
-    - target_linux_kernelrelease | length > 0
+    - (active_linux_kernelrelease | default(target_linux_kernelrelease)) | length > 0
   run_once: true
   delegate_to: localhost
 
-- name: Generate user kernelrelease {{ target_linux_kernelversion.stdout }}-{{ target_linux_kernelrelease }}
+- name: Generate user kernelrelease
   set_fact:
-    target_user_kernelrelease: "{{ target_linux_kernelversion.stdout }}-{{ target_linux_kernelrelease }}"
+    target_user_kernelrelease: "{{ target_linux_kernelversion.stdout }}-{{ active_linux_kernelrelease | default(target_linux_kernelrelease) }}"
   tags: [ 'build-linux' ]
   when:
     - bootlinux_9p|bool
-    - target_linux_kernelrelease | length > 0
+    - bootlinux_tree_custom_kernelrelease|bool
+    - (active_linux_kernelrelease | default(target_linux_kernelrelease)) | length > 0
+    - target_linux_kernelversion is defined
+    - target_linux_kernelversion.stdout is defined
   run_once: true
   delegate_to: localhost
 
-- name: Build {{ target_linux_tree }} {{ target_user_kernelrelease }} on the control node using {{ nproc_9p.stdout }} threads
+- name: Build {{ target_linux_tree }} with custom kernel release on the control node using {{ nproc_9p.stdout }} threads
   make:
     jobs: "{{ nproc_9p.stdout }}"
     chdir: "{{ bootlinux_9p_host_path }}"
@@ -452,11 +537,13 @@
   tags: [ 'build-linux' ]
   when:
     - bootlinux_9p|bool
+    - bootlinux_tree_custom_kernelrelease|bool
     - target_linux_kernelrelease | length > 0
+    - target_user_kernelrelease is defined
   run_once: true
   delegate_to: localhost
 
-- name: Build {{ target_linux_tree }} {{ target_user_kernelrelease }} on the control node using {{ nproc_9p.stdout }} threads
+- name: Build {{ target_linux_tree }} on the control node using {{ nproc_9p.stdout }} threads
   make:
     jobs: "{{ nproc_9p.stdout }}"
     chdir: "{{ bootlinux_9p_host_path }}"
diff --git a/scripts/infer_last_stable_kernel.sh b/scripts/infer_last_stable_kernel.sh
new file mode 100755
index 0000000..9cc19a9
--- /dev/null
+++ b/scripts/infer_last_stable_kernel.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+# This script infers the last stable kernel version from the git repository.
+# It looks for the most recent non-rc tag (e.g., v6.14, v6.13) that would
+# be a good default for A/B testing with different kernel references.
+
+GIT_TREE="${1:-/mirror/linux.git}"
+
+if [ ! -d "$GIT_TREE" ]; then
+    echo "v6.12"  # fallback if no git tree available
+    exit 0
+fi
+
+# Get all v6.x tags, excluding release candidates
+# Sort them by version and get the last stable release
+LAST_STABLE=$(git --git-dir="$GIT_TREE" tag --list 'v6.*' | \
+    grep -v -- '-rc' | \
+    sort -V | \
+    tail -1)
+
+if [ -z "$LAST_STABLE" ]; then
+    # If no stable v6.x found, try v5.x as fallback
+    LAST_STABLE=$(git --git-dir="$GIT_TREE" tag --list 'v5.*' | \
+        grep -v -- '-rc' | \
+        sort -V | \
+        tail -1)
+fi
+
+# Final fallback if nothing found
+if [ -z "$LAST_STABLE" ]; then
+    echo "v6.12"
+else
+    echo "$LAST_STABLE"
+fi
\ No newline at end of file
diff --git a/workflows/linux/Kconfig b/workflows/linux/Kconfig
index 183ac77..d08270e 100644
--- a/workflows/linux/Kconfig
+++ b/workflows/linux/Kconfig
@@ -139,6 +139,41 @@ config BOOTLINUX_CUSTOM
 
 endchoice
 
+if KDEVOPS_BASELINE_AND_DEV
+
+choice
+	prompt "A/B kernel testing configuration"
+	default BOOTLINUX_AB_DIFFERENT_REF
+	help
+	  When A/B testing is enabled, you can choose to use the same
+	  kernel reference for both baseline and dev nodes, or specify
+	  different kernel references to test different kernel versions.
+	  We default to assuming you want to test a different kernel on
+	  each.
+
+config BOOTLINUX_AB_SAME_REF
+	bool "Use same kernel reference for baseline and dev"
+	output yaml
+	help
+	  Use the same kernel tree and reference for both baseline and
+	  development nodes. This is useful for testing configuration
+	  changes or different test parameters with the same kernel.
+
+config BOOTLINUX_AB_DIFFERENT_REF
+	bool "Use different kernel references for baseline and dev"
+	output yaml
+	help
+	  Use different kernel references for baseline and development
+	  nodes. This enables testing between different kernel versions,
+	  commits, or branches. The baseline will use the main configured
+	  kernel reference, while dev uses a separate reference.
+
+endchoice
+
+endif
+
+menu "A -     main    group kernel configuration"
+
 source "workflows/linux/Kconfig.linus"
 source "workflows/linux/Kconfig.stable"
 source "workflows/linux/Kconfig.dev"
@@ -180,6 +215,65 @@ config BOOTLINUX_TREE_CUSTOM_REF
 
 endif # BOOTLINUX_CUSTOM
 
+endmenu
+
+if KDEVOPS_BASELINE_AND_DEV
+
+if BOOTLINUX_AB_DIFFERENT_REF
+
+menu "B - development group kernel configuration"
+
+config BOOTLINUX_DEV_TREE
+	string "B group development kernel tree URL"
+	output yaml
+	default BOOTLINUX_TREE
+	help
+	  Git tree URL for the development kernel. If left empty or same
+	  as the baseline tree, the same tree will be used with a different
+	  reference. This allows testing different branches or forks.
+
+config BOOTLINUX_DEV_TREE_REF
+	string "B group development kernel reference"
+	output yaml
+	default $(shell, scripts/infer_last_stable_kernel.sh)
+	help
+	  Git reference (branch, tag, or commit) for the development kernel.
+	  This should be different from the baseline reference to enable
+	  meaningful A/B comparison between kernel versions.
+
+	  The default is automatically inferred as the most recent stable
+	  kernel version (e.g., v6.15) from the git repository.
+
+	  Examples:
+	  - "v6.8" (stable release)
+	  - "linux-next" (latest development)
+	  - "v6.7..v6.8" (range for bisection)
+	  - commit SHA (specific commit)
+
+config BOOTLINUX_DEV_TREE_KERNELRELEASE
+	string "Development kernel release version"
+	depends on BOOTLINUX_TREE_CUSTOM_KERNELRELEASE
+	output yaml
+	help
+	  The string here (e.g. 'devel') will be appended to the result of make
+	  kernelversion. Example: '6.8.0-rc3-devel' but only for the dev group.
+	  Leave it empty unless you want a custom tag at the end.
+
+config BOOTLINUX_DEV_TREE_LOCALVERSION
+	string "Development kernel local version"
+	output yaml
+	depends on BOOTLINUX_TREE_CUSTOM_LOCALVERSION
+	default BOOTLINUX_TREE_LOCALVERSION
+	help
+	  The Linux local version to use for the development kernel (for uname).
+	  If left empty, will use the same as baseline.
+
+endmenu
+
+endif # BOOTLINUX_AB_DIFFERENT_REF
+
+endif # KDEVOPS_BASELINE_AND_DEV
+
 # This ends up being the directory name used for the /data/ partition
 # where linux is deployed on the nodes.
 config BOOTLINUX_TREE_NAME
@@ -264,23 +358,39 @@ config BOOTLINUX_TREE_REF
 	default BOOTLINUX_TREE_CEL_LINUX_REF if BOOTLINUX_TREE_CEL_LINUX
 	default BOOTLINUX_TREE_CUSTOM_REF if BOOTLINUX_CUSTOM
 
+config BOOTLINUX_TREE_CUSTOM_KERNELRELEASE
+	bool "Do you want a full custom kernel release name?"
+	output yaml
+	help
+	  Do you want a full custom Linux kernel release which will be output
+	  through uname?
+
 config BOOTLINUX_TREE_KERNELRELEASE
 	string "Linux kernel release version to use"
+	depends on BOOTLINUX_TREE_CUSTOM_KERNELRELEASE
 	help
 	  The Linux kernel release version to use (for uname).
 
 	  The string here (e.g. 'devel') will be appended to the result of make
 	  kernelversion. Example: '6.8.0-rc3-devel'
 
+config BOOTLINUX_TREE_CUSTOM_LOCALVERSION
+	bool "Do you want to append a custom kernel release tag?"
+	output yaml
+	help
+	  Do you want a full custom Linux kernel release which will be output
+	  through uname?
 
 config BOOTLINUX_TREE_LOCALVERSION
 	string "Linux local version to use"
+	depends on BOOTLINUX_TREE_CUSTOM_LOCALVERSION
 	help
 	  The Linux local version to use (for uname).
 
 config BOOTLINUX_SHALLOW_CLONE
 	bool "Shallow git clone"
-	default y
+	default y if !KDEVOPS_BASELINE_AND_DEV
+	depends on !BOOTLINUX_AB_DIFFERENT_REF
 	help
 	  If enabled the git tree cloned with be cloned using a shallow tree
 	  with history truncated. You want to enable this if you really don't
@@ -291,6 +401,10 @@ config BOOTLINUX_SHALLOW_CLONE
 	  just using the targets as dummy target runners and don't expect to
 	  be using 'git log' on the target guests.
 
+	  This option is automatically disabled when using A/B testing with
+	  different kernel references, as shallow clones may not contain all
+	  the required refs for checkout.
+
 config BOOTLINUX_SHALLOW_CLONE_DEPTH
 	int "Shallow git clone depth"
 	default 30 if BOOTLINUX_TREE_SET_BY_CLI
diff --git a/workflows/linux/Makefile b/workflows/linux/Makefile
index f68c090..ab8bcbb 100644
--- a/workflows/linux/Makefile
+++ b/workflows/linux/Makefile
@@ -71,6 +71,10 @@ PHONY +=  linux-help-menu
 linux-help-menu:
 	@echo "Linux git kernel development options"
 	@echo "linux              - Git clones a linux git tree, build Linux, installs and reboots into it"
+	@if [[ "$(CONFIG_KDEVOPS_BASELINE_AND_DEV)" == "y" ]]; then \
+		echo "linux-baseline     - Build and install kernel for baseline nodes only" ;\
+		echo "linux-dev          - Build and install kernel for dev nodes only" ;\
+	fi
 	@if [[ "$(CONFIG_BOOTLINUX_9P)" == "y" ]]; then \
 		echo "linux-mount        - Mounts 9p path on targets" ;\
 	fi
@@ -89,10 +93,45 @@ linux-help-end:
 LINUX_HELP_EXTRA :=
 
 PHONY += linux
+ifeq (y,$(CONFIG_KDEVOPS_BASELINE_AND_DEV))
+ifeq (y,$(CONFIG_BOOTLINUX_AB_DIFFERENT_REF))
+linux: linux-baseline linux-dev
+else
+linux: $(KDEVOPS_NODES)
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i \
+		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
+		--extra-vars="$(BOOTLINUX_ARGS)" $(LIMIT_HOSTS)
+endif
+else
 linux: $(KDEVOPS_NODES)
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i \
 		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
 		--extra-vars="$(BOOTLINUX_ARGS)" $(LIMIT_HOSTS)
+endif
+
+PHONY += linux-baseline
+ifeq (y,$(CONFIG_KDEVOPS_BASELINE_AND_DEV))
+linux-baseline: $(KDEVOPS_NODES)
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i \
+		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
+		--extra-vars="$(BOOTLINUX_ARGS)" --limit baseline
+else
+linux-baseline:
+	@echo "linux-baseline requires KDEVOPS_BASELINE_AND_DEV=y"
+	@exit 1
+endif
+
+PHONY += linux-dev
+ifeq (y,$(CONFIG_KDEVOPS_BASELINE_AND_DEV))
+linux-dev: $(KDEVOPS_NODES)
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i \
+		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
+		--extra-vars="$(BOOTLINUX_ARGS)" --limit dev
+else
+linux-dev:
+	@echo "linux-dev requires KDEVOPS_BASELINE_AND_DEV=y"
+	@exit 1
+endif
 
 PHONY += linux-mount
 linux-mount:
-- 
2.47.2