From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5073D2E3AEC for ; Mon, 11 Aug 2025 22:24:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754951098; cv=none; b=pLlbTfns7i8SjNuY/mt7SRxK7qXqvaaU/DvriHUm0/8uhqo+yO0iXcpHG+AVF2vYxGLOxdYlZ15q5V4fTAlY+baPljctP9GePBVXGHEIIQzYkA15Fq+EMrdbpVkE91z6l+EDvOCjOtMhr4K990/zEEkJUwfUI3lsPE04BEjneEE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754951098; c=relaxed/simple; bh=wZd0gckWczY8QJQzEv2+DzDcGeeVWgoS+BAkobOowuQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=L9RiwNBmrKY4moMieIGFhhx877pdFO7ff/IcsGa+H1c4g4KfdTbf15ZdZ3mVzneTH0GxMEPrMfGnInY2JvoD435sNhcnKGnyet08CExbu3xW919WonrOOTjyG/EGamtAecdZuWgdtc9wum+UI64TO81QPqfJi939tpBTgqvSN1k= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=0K7Wa4Ju; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="0K7Wa4Ju" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=sWBatoaxfAO75RQ0knJxbx44qoni3jt4t4aTPfNw59o=; b=0K7Wa4Juey7oqeNGOs+c04egF5 6n/aMxXhiOODYjvQ4ho++NgJpwjEdZqiMeFv/GcTiUgXK8bv0mQah4JcHkAnlSdZqOwv6BGwI7deN 6ujdoB+LhpJ+FbrP/njxsvleGJ3/IO3B82H6bIwB/WX8Lo59GgArQ2kbZxdL90NCCORxV6u3KG8sQ 1ZftwgjJde8ALfgSyNHD3HY8u7CfNz9lum6ZtzCCdCkGdMSarT/KczRPj97k9nlNjz/zZgVyIQ7zZ htG5UdjCHG/GF1RXUD76YaRwiix8g7z5U3pYKaxrVg4pKF2OY8nL4ivosKcdW+qfyKFdgL/Xf1RDC 8H7NT6Qg==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulawo-00000009Hkp-06y2; Mon, 11 Aug 2025 22:24:54 +0000 From: Luis Chamberlain To: Chuck Lever , Daniel Gomez , kdevops@lists.linux.dev Cc: Luis Chamberlain , Claude AI Subject: [PATCH 23/23] reboot-limit: add kexec comparison feature Date: Mon, 11 Aug 2025 15:24:50 -0700 Message-ID: <20250811222452.2213071-24-mcgrof@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250811222452.2213071-1-mcgrof@kernel.org> References: <20250811222452.2213071-1-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain Add comprehensive support for comparing regular reboots vs kexec reboots in the reboot-limit workflow. This enables users to test both reboot types sequentially and compare their performance characteristics. New defconfigs: - reboot-limit-kexec: Configure for kexec-only testing - reboot-limit-compare: Configure for comparison mode testing New Kconfig options: - REBOOT_LIMIT_TYPE_COMPARE_BOTH: Test both regular and kexec reboots - REBOOT_LIMIT_DATA_REGULAR/KEXEC: Separate data paths for comparison Workflow enhancements: - Sequential testing: regular reboot first, then kexec - Separate statistics collection for each reboot type - Enhanced data organization with regular/ and kexec/ subdirectories Analysis improvements: - Automatic comparison mode detection - Side-by-side performance analysis - Speedup calculation and visualization - Comprehensive comparative graphs showing both boot types Usage: make defconfig-reboot-limit-compare make bringup make reboot-limit-tests make reboot-limit-graph Generated-by: Claude AI Signed-off-by: Claude AI --- defconfigs/reboot-limit-compare | 36 ++ defconfigs/reboot-limit-kexec | 32 ++ playbooks/roles/gen_hosts/tasks/main.yml | 13 + playbooks/roles/gen_hosts/templates/hosts.j2 | 55 ++- playbooks/roles/gen_nodes/tasks/main.yml | 32 ++ .../reboot-limit/tasks/do-reboot-compare.yml | 126 +++++++ .../roles/reboot-limit/tasks/do-reboot.yml | 28 +- .../reboot-limit/tasks/handle-reboot-data.yml | 94 ++++++ playbooks/roles/reboot-limit/tasks/main.yml | 84 ++++- .../demos/reboot-limit/analyze_results.py | 314 +++++++++++++++++- workflows/demos/reboot-limit/Kconfig | 31 ++ workflows/demos/reboot-limit/Makefile | 13 + 12 files changed, 828 insertions(+), 30 deletions(-) create mode 100644 defconfigs/reboot-limit-compare create mode 100644 defconfigs/reboot-limit-kexec create mode 100644 playbooks/roles/reboot-limit/tasks/do-reboot-compare.yml create mode 100644 playbooks/roles/reboot-limit/tasks/handle-reboot-data.yml diff --git a/defconfigs/reboot-limit-compare b/defconfigs/reboot-limit-compare new file mode 100644 index 00000000..7f88cf02 --- /dev/null +++ b/defconfigs/reboot-limit-compare @@ -0,0 +1,36 @@ +# Demo defconfig for reboot-limit workflow with comparison testing +# +# This enables the reboot-limit workflow to compare regular reboots +# vs kexec reboots with separate performance monitoring and analysis. + +# Enable workflows and reboot-limit workflow +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_TESTS_DEMOS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_WORKFLOWS_REBOOT_LIMIT=y + +# Enable baseline and dev testing +CONFIG_KDEVOPS_BASELINE_AND_DEV=y + +# Enable data collection for analysis +CONFIG_REBOOT_LIMIT_ENABLE_DATA_COLLECTION=y +CONFIG_REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE=y + +# Enable loop testing for continuous operation +CONFIG_REBOOT_LIMIT_ENABLE_LOOP=y +CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL=100 + +# Set a reasonable number of reboots per test run +CONFIG_REBOOT_LIMIT_BOOT_MAX=100 + +# Use comparison mode to test both regular and kexec reboots +CONFIG_REBOOT_LIMIT_TYPE_COMPARE_BOTH=y + +# We'll build our kernel +CONFIG_WORKFLOW_LINUX_CUSTOM=y +CONFIG_BOOTLINUX=y +CONFIG_BOOTLINUX_9P=y + +# Enable A/B testing with different kernel references +CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y diff --git a/defconfigs/reboot-limit-kexec b/defconfigs/reboot-limit-kexec new file mode 100644 index 00000000..e6416a2f --- /dev/null +++ b/defconfigs/reboot-limit-kexec @@ -0,0 +1,32 @@ +# Demo defconfig for reboot-limit workflow with kexec +# +# This enables the reboot-limit workflow for system stability testing +# through continuous kexec reboots with performance monitoring. + +# Enable workflows and reboot-limit workflow +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_TESTS_DEMOS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_WORKFLOWS_REBOOT_LIMIT=y + +# Enable data collection for analysis +CONFIG_REBOOT_LIMIT_ENABLE_DATA_COLLECTION=y +CONFIG_REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE=y + +# Enable loop testing for continuous operation +CONFIG_REBOOT_LIMIT_ENABLE_LOOP=y +CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL=100 + +# Set a reasonable number of reboots per test run +CONFIG_REBOOT_LIMIT_BOOT_MAX=100 + +# Use kexec reboot method for faster rebooting +CONFIG_REBOOT_LIMIT_TYPE_SYSTEMD_KEXEC=y + +# Enable baseline and development nodes for A/B testing +CONFIG_KDEVOPS_BASELINE_AND_DEV=y + +# Storage configuration +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE_SET_CUSTOM=y +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE=20 diff --git a/playbooks/roles/gen_hosts/tasks/main.yml b/playbooks/roles/gen_hosts/tasks/main.yml index 0a742d89..15c1ddbb 100644 --- a/playbooks/roles/gen_hosts/tasks/main.yml +++ b/playbooks/roles/gen_hosts/tasks/main.yml @@ -351,6 +351,19 @@ - kdevops_workflow_enable_mmtests - ansible_hosts_template.stat.exists +- name: Generate the Ansible hosts file for a dedicated reboot-limit setup + tags: [ 'hosts' ] + template: + src: "{{ kdevops_hosts_template }}" + dest: "{{ ansible_cfg_inventory }}" + force: yes + trim_blocks: True + lstrip_blocks: True + when: + - kdevops_workflows_dedicated_workflow + - workflows_reboot_limit + - ansible_hosts_template.stat.exists + - name: Verify if final host file exists stat: path: "{{ ansible_cfg_inventory }}" diff --git a/playbooks/roles/gen_hosts/templates/hosts.j2 b/playbooks/roles/gen_hosts/templates/hosts.j2 index ca6adcfc..f89fae48 100644 --- a/playbooks/roles/gen_hosts/templates/hosts.j2 +++ b/playbooks/roles/gen_hosts/templates/hosts.j2 @@ -6,52 +6,87 @@ Each workflow which has its own custom ansible host file generated should use its own jinja2 template file and define its own ansible task for its generation. #} {% if kdevops_workflows_dedicated_workflow %} +{% if workflows_reboot_limit %} +[all] +localhost ansible_connection=local +{{ kdevops_host_prefix }}-reboot-limit +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-reboot-limit-dev +{% endif %} + +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +[baseline] +{{ kdevops_host_prefix }}-reboot-limit + +[baseline:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% if kdevops_baseline_and_dev %} +[dev] +{{ kdevops_host_prefix }}-reboot-limit-dev + +[dev:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% endif %} +[reboot-limit] +{{ kdevops_host_prefix }}-reboot-limit +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-reboot-limit-dev +{% endif %} + +[reboot-limit:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +{% else %} [all] localhost ansible_connection=local write-your-own-template-for-your-workflow-and-task +{% endif %} {% else %} [all] localhost ansible_connection=local -{{ kdevops_hosts_prefix }} +{{ kdevops_host_prefix }} {% if kdevops_baseline_and_dev == True %} -{{ kdevops_hosts_prefix }}-dev +{{ kdevops_host_prefix }}-dev {% endif %} {% if kdevops_enable_iscsi %} -{{ kdevops_hosts_prefix }}-iscsi +{{ kdevops_host_prefix }}-iscsi {% endif %} {% if kdevops_nfsd_enable %} -{{ kdevops_hosts_prefix }}-nfsd +{{ kdevops_host_prefix }}-nfsd {% endif %} [all:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [baseline] -{{ kdevops_hosts_prefix }} +{{ kdevops_host_prefix }} [baseline:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [dev] {% if kdevops_baseline_and_dev %} -{{ kdevops_hosts_prefix }}-dev +{{ kdevops_host_prefix }}-dev {% endif %} [dev:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" {% if kdevops_enable_iscsi %} [iscsi] -{{ kdevops_hosts_prefix }}-iscsi +{{ kdevops_host_prefix }}-iscsi [iscsi:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" {% endif %} {% if kdevops_nfsd_enable %} [nfsd] -{{ kdevops_hosts_prefix }}-nfsd +{{ kdevops_host_prefix }}-nfsd [nfsd:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" {% endif %} [service] {% if kdevops_enable_iscsi %} -{{ kdevops_hosts_prefix }}-iscsi +{{ kdevops_host_prefix }}-iscsi {% endif %} {% if kdevops_nfsd_enable %} -{{ kdevops_hosts_prefix }}-nfsd +{{ kdevops_host_prefix }}-nfsd {% endif %} [service:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" diff --git a/playbooks/roles/gen_nodes/tasks/main.yml b/playbooks/roles/gen_nodes/tasks/main.yml index 4192ae89..119d262e 100644 --- a/playbooks/roles/gen_nodes/tasks/main.yml +++ b/playbooks/roles/gen_nodes/tasks/main.yml @@ -558,6 +558,38 @@ - kdevops_workflow_enable_mmtests - ansible_nodes_template.stat.exists +- name: Generate the reboot-limit kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template + tags: [ 'hosts' ] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ [kdevops_host_prefix + '-reboot-limit'] }}" + all_generic_nodes: "{{ [kdevops_host_prefix + '-reboot-limit'] }}" + template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: yes + when: + - kdevops_workflows_dedicated_workflow + - workflows_reboot_limit + - ansible_nodes_template.stat.exists + - not kdevops_baseline_and_dev + +- name: Generate the reboot-limit kdevops nodes file with dev hosts using {{ kdevops_nodes_template }} as jinja2 source template + tags: [ 'hosts' ] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ [kdevops_host_prefix + '-reboot-limit', kdevops_host_prefix + '-reboot-limit-dev'] }}" + all_generic_nodes: "{{ [kdevops_host_prefix + '-reboot-limit', kdevops_host_prefix + '-reboot-limit-dev'] }}" + template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: yes + when: + - kdevops_workflows_dedicated_workflow + - workflows_reboot_limit + - ansible_nodes_template.stat.exists + - kdevops_baseline_and_dev + - name: Get the control host's timezone ansible.builtin.command: "timedatectl show -p Timezone --value" register: kdevops_host_timezone diff --git a/playbooks/roles/reboot-limit/tasks/do-reboot-compare.yml b/playbooks/roles/reboot-limit/tasks/do-reboot-compare.yml new file mode 100644 index 00000000..43404dc2 --- /dev/null +++ b/playbooks/roles/reboot-limit/tasks/do-reboot-compare.yml @@ -0,0 +1,126 @@ +--- +# This task performs both regular and kexec reboots sequentially for comparison +- name: Print uname for each host + tags: [ 'run_tests' ] + debug: var=ansible_kernel + +- name: Hint to our watchdog our reboot-limit comparison tests are about to kick off + local_action: file path="{{ reboot_limit_local_results_dir }}/.begin" state=touch + tags: [ 'run_tests' ] + run_once: true + +# Phase 1: Regular reboot test +- name: Starting Phase 1 - Regular reboot test ({{ reboot_num }} of {{ reboot_limit_max }}) + debug: + msg: "Starting regular reboot test - reboot {{ reboot_num }} of {{ reboot_limit_max }}" + tags: [ 'run_tests' ] + +- name: Run the regular reboot test using the ansible reboot module + become: yes + become_method: sudo + reboot: + post_reboot_delay: 10 + tags: [ 'run_tests' ] + +- name: Handle regular reboot count and data collection + include_tasks: handle-reboot-data.yml + vars: + reboot_type: "regular" + data_path: "{{ reboot_limit_data_regular }}" + tags: [ 'run_tests' ] + +# Phase 2: Kexec reboot test +- name: Starting Phase 2 - Kexec reboot test ({{ reboot_num }} of {{ reboot_limit_max }}) + debug: + msg: "Starting kexec reboot test - reboot {{ reboot_num }} of {{ reboot_limit_max }}" + tags: [ 'run_tests' ] + +# Kexec preparation tasks +- name: Get current kernel version for kexec + command: uname -r + register: current_kernel_version + tags: [ 'run_tests' ] + +- name: Check for kernel image locations for kexec + stat: + path: "{{ kernel_path }}" + register: kernel_stat + loop: + - "/boot/vmlinuz-{{ current_kernel_version.stdout }}" + - "/boot/vmlinux-{{ current_kernel_version.stdout }}" + - "/boot/kernel-{{ current_kernel_version.stdout }}" + loop_control: + loop_var: kernel_path + when: + - current_kernel_version is defined + tags: [ 'run_tests' ] + +- name: Set kernel path for kexec + set_fact: + kexec_kernel_path: "{{ kernel_item.stat.path }}" + loop: "{{ kernel_stat.results }}" + loop_control: + loop_var: kernel_item + when: + - kernel_item.stat.exists + tags: [ 'run_tests' ] + +- name: Check for initrd/initramfs locations for kexec + stat: + path: "{{ initrd_path }}" + register: initrd_stat + loop: + - "/boot/initrd.img-{{ current_kernel_version.stdout }}" + - "/boot/initramfs-{{ current_kernel_version.stdout }}.img" + - "/boot/initrd-{{ current_kernel_version.stdout }}" + - "/boot/initrd-{{ current_kernel_version.stdout }}.img" + loop_control: + loop_var: initrd_path + when: + - current_kernel_version is defined + tags: [ 'run_tests' ] + +- name: Set initrd path for kexec + set_fact: + kexec_initrd_path: "{{ initrd_item.stat.path }}" + loop: "{{ initrd_stat.results }}" + loop_control: + loop_var: initrd_item + when: + - initrd_item.stat.exists + tags: [ 'run_tests' ] + +- name: Read current kernel command line for kexec + slurp: + src: /proc/cmdline + register: cmdline_content + tags: [ 'run_tests' ] + +- name: Load kernel into kexec + become: yes + become_method: sudo + command: > + kexec -l {{ kexec_kernel_path }} + --initrd={{ kexec_initrd_path }} + --command-line="{{ cmdline_content.content | b64decode | trim }}" + when: + - kexec_kernel_path is defined + - kexec_initrd_path is defined + tags: [ 'run_tests' ] + +- name: Run the kexec reboot test using systemctl kexec + become: yes + become_method: sudo + reboot: + msg: "Rebooting system via systemctl kexec for reboot-limit comparison test" + reboot_command: "systemctl kexec" + post_reboot_delay: 10 + reboot_timeout: 300 + tags: [ 'run_tests' ] + +- name: Handle kexec reboot count and data collection + include_tasks: handle-reboot-data.yml + vars: + reboot_type: "kexec" + data_path: "{{ reboot_limit_data_kexec }}" + tags: [ 'run_tests' ] diff --git a/playbooks/roles/reboot-limit/tasks/do-reboot.yml b/playbooks/roles/reboot-limit/tasks/do-reboot.yml index ec2c29df..7b38f9dc 100644 --- a/playbooks/roles/reboot-limit/tasks/do-reboot.yml +++ b/playbooks/roles/reboot-limit/tasks/do-reboot.yml @@ -39,12 +39,14 @@ - name: Check for kernel image locations for kexec stat: - path: "{{ item }}" + path: "{{ kernel_path }}" register: kernel_stat - with_items: + loop: - "/boot/vmlinuz-{{ current_kernel_version.stdout }}" - "/boot/vmlinux-{{ current_kernel_version.stdout }}" - "/boot/kernel-{{ current_kernel_version.stdout }}" + loop_control: + loop_var: kernel_path when: - reboot_limit_test_type == "systemctl_kexec" - current_kernel_version is defined @@ -52,22 +54,26 @@ - name: Set kernel path for kexec set_fact: - kexec_kernel_path: "{{ item.stat.path }}" - with_items: "{{ kernel_stat.results }}" + kexec_kernel_path: "{{ kernel_item.stat.path }}" + loop: "{{ kernel_stat.results }}" + loop_control: + loop_var: kernel_item when: - reboot_limit_test_type == "systemctl_kexec" - - item.stat.exists + - kernel_item.stat.exists tags: [ 'run_tests' ] - name: Check for initrd/initramfs locations for kexec stat: - path: "{{ item }}" + path: "{{ initrd_path }}" register: initrd_stat - with_items: + loop: - "/boot/initrd.img-{{ current_kernel_version.stdout }}" - "/boot/initramfs-{{ current_kernel_version.stdout }}.img" - "/boot/initrd-{{ current_kernel_version.stdout }}" - "/boot/initrd-{{ current_kernel_version.stdout }}.img" + loop_control: + loop_var: initrd_path when: - reboot_limit_test_type == "systemctl_kexec" - current_kernel_version is defined @@ -75,11 +81,13 @@ - name: Set initrd path for kexec set_fact: - kexec_initrd_path: "{{ item.stat.path }}" - with_items: "{{ initrd_stat.results }}" + kexec_initrd_path: "{{ initrd_item.stat.path }}" + loop: "{{ initrd_stat.results }}" + loop_control: + loop_var: initrd_item when: - reboot_limit_test_type == "systemctl_kexec" - - item.stat.exists + - initrd_item.stat.exists tags: [ 'run_tests' ] - name: Read current kernel command line for kexec diff --git a/playbooks/roles/reboot-limit/tasks/handle-reboot-data.yml b/playbooks/roles/reboot-limit/tasks/handle-reboot-data.yml new file mode 100644 index 00000000..6bbdbb3c --- /dev/null +++ b/playbooks/roles/reboot-limit/tasks/handle-reboot-data.yml @@ -0,0 +1,94 @@ +--- +# Handles reboot counting and data collection for a specific reboot type +# Variables expected: +# reboot_type: "regular" or "kexec" +# data_path: path where to store data for this reboot type + +- name: Set reboot type specific file paths + set_fact: + reboot_type_analyze_file: "{{ data_path }}/{{ ansible_ssh_host }}/{{ reboot_limits_systemctl_analyze_log }}" + reboot_type_count_file: "{{ data_path }}/{{ ansible_ssh_host }}/{{ reboot_limits_count_log }}" + tags: [ 'run_tests' ] + +- name: Create the data collection directory for {{ reboot_type }} reboot type + become: yes + become_method: sudo + file: + path: "{{ data_path }}/{{ ansible_ssh_host }}" + state: directory + tags: [ 'run_tests' ] + +- name: Check if the {{ reboot_type }} reboot count file exists + become: yes + become_method: sudo + stat: + path: "{{ reboot_type_count_file }}" + register: reboot_type_count_file_stat + tags: [ 'run_tests' ] + +- name: Read last {{ reboot_type }} boot count + become: yes + become_method: sudo + slurp: + src: "{{ reboot_type_count_file }}" + register: reboot_type_last_count + when: + - reboot_type_count_file_stat.stat.exists + tags: [ 'run_tests' ] + +- name: Set the current {{ reboot_type }} boot count into a variable + set_fact: + reboot_type_count: "{{ reboot_type_last_count['content'] | b64decode | int }}" + tags: [ 'run_tests' ] + when: + - reboot_type_count_file_stat.stat.exists + +- name: Adjust the {{ reboot_type }} boot count if we rebooted OK + set_fact: + reboot_type_count: "{{ reboot_type_count | int + 1 }}" + tags: [ 'run_tests' ] + when: + - reboot_type_count_file_stat.stat.exists + +- name: Set the current {{ reboot_type }} boot count when no prior test exists + set_fact: + reboot_type_count: 1 + tags: [ 'run_tests' ] + when: + - not reboot_type_count_file_stat.stat.exists + +- name: Write current {{ reboot_type }} boot count to file ({{ reboot_type_count }}) + become: yes + become_method: sudo + copy: + content: "{{ reboot_type_count }}" + dest: "{{ reboot_type_count_file }}" + tags: [ 'run_tests' ] + +- name: Wait for boot up to complete before running systemd-analyze for {{ reboot_type }} + become: yes + become_method: sudo + command: "systemctl is-system-running --wait" + when: + - reboot_limit_enable_systemd_analyze|bool + tags: [ 'run_tests' ] + +- name: Collect systemctl-analyze results for {{ reboot_type }} + become: yes + become_method: sudo + command: "systemd-analyze" + register: systemd_analyze_cmd + when: + - reboot_limit_enable_systemd_analyze|bool + tags: [ 'run_tests' ] + +- name: Append systemctl-analyze output for {{ reboot_type }} + become: yes + become_method: sudo + tags: [ 'run_tests' ] + lineinfile: + path: "{{ reboot_type_analyze_file }}" + line: "{{ systemd_analyze_cmd.stdout }}" + create: yes + when: + - reboot_limit_enable_systemd_analyze|bool diff --git a/playbooks/roles/reboot-limit/tasks/main.yml b/playbooks/roles/reboot-limit/tasks/main.yml index 7bf0f573..5cd5e088 100644 --- a/playbooks/roles/reboot-limit/tasks/main.yml +++ b/playbooks/roles/reboot-limit/tasks/main.yml @@ -24,6 +24,25 @@ file: path: "{{ reboot_limit_data }}/{{ ansible_ssh_host }}" state: directory + when: not reboot_limit_compare_both_enabled|default(false)|bool + tags: [ 'install', 'first_run' ] + +- name: Create the regular reboot data collection directory for comparison mode + become: yes + become_method: sudo + file: + path: "{{ reboot_limit_data_regular }}/{{ ansible_ssh_host }}" + state: directory + when: reboot_limit_compare_both_enabled|default(false)|bool + tags: [ 'install', 'first_run' ] + +- name: Create the kexec reboot data collection directory for comparison mode + become: yes + become_method: sudo + file: + path: "{{ reboot_limit_data_kexec }}/{{ ansible_ssh_host }}" + state: directory + when: reboot_limit_compare_both_enabled|default(false)|bool tags: [ 'install', 'first_run' ] - name: Set the file to collect systemctl-analyze results @@ -40,7 +59,7 @@ reboot_limit_count_file: "{{ reboot_limit_data}}/{{ ansible_ssh_host }}/{{ reboot_limits_count_log }}" tags: [ 'read_count', 'vars' ] -- name: Delete old results directory files if a reset was called +- name: Delete old results directory files if a reset was called (single mode) become: yes become_method: sudo file: @@ -51,6 +70,25 @@ - "{{ reboot_limit_count_file }}" loop_control: label: "{{ item | regex_replace(reboot_limit_data | regex_escape()) | regex_replace('^/', '') }}" + when: not reboot_limit_compare_both_enabled|default(false)|bool + tags: [ 'reset' ] + +- name: Delete old results directory files if a reset was called (comparison mode - regular) + become: yes + become_method: sudo + file: + path: "{{ reboot_limit_data_regular }}/{{ ansible_ssh_host }}" + state: absent + when: reboot_limit_compare_both_enabled|default(false)|bool + tags: [ 'reset' ] + +- name: Delete old results directory files if a reset was called (comparison mode - kexec) + become: yes + become_method: sudo + file: + path: "{{ reboot_limit_data_kexec }}/{{ ansible_ssh_host }}" + state: absent + when: reboot_limit_compare_both_enabled|default(false)|bool tags: [ 'reset' ] - name: Set the path where we collect our local reboot-limit results @@ -68,12 +106,23 @@ run_once: true tags: [ 'first_run' ] -- name: Run the reboot loop +- name: Run the reboot loop (single mode) include_tasks: do-reboot.yml with_sequence: count={{ reboot_limit_max }} + loop_control: + loop_var: reboot_num + when: not reboot_limit_compare_both_enabled|default(false)|bool + tags: [ 'run_tests' ] + +- name: Run the reboot comparison loop (both regular and kexec) + include_tasks: do-reboot-compare.yml + with_sequence: count={{ reboot_limit_max }} + loop_control: + loop_var: reboot_num + when: reboot_limit_compare_both_enabled|default(false)|bool tags: [ 'run_tests' ] -- name: Copy the latest results over when we're done +- name: Copy the latest results over when we're done (single mode) tags: [ 'copy_results' ] become: yes become_flags: 'su - -c' @@ -87,3 +136,32 @@ - "{{ reboot_limit_count_file }}" loop_control: label: "{{ item | regex_replace(reboot_limit_data | regex_escape()) | regex_replace('^/', '') }}" + when: not reboot_limit_compare_both_enabled|default(false)|bool + +- name: Copy the regular reboot results in comparison mode + tags: [ 'copy_results' ] + become: yes + become_flags: 'su - -c' + become_method: sudo + fetch: + src: "{{ reboot_limit_data_regular }}/{{ ansible_ssh_host }}/{{ item }}" + dest: "{{ reboot_limit_local_results_dir }}/regular/{{ ansible_ssh_host }}/{{ item }}" + flat: yes + with_items: + - "{{ reboot_limits_systemctl_analyze_log }}" + - "{{ reboot_limits_count_log }}" + when: reboot_limit_compare_both_enabled|default(false)|bool + +- name: Copy the kexec reboot results in comparison mode + tags: [ 'copy_results' ] + become: yes + become_flags: 'su - -c' + become_method: sudo + fetch: + src: "{{ reboot_limit_data_kexec }}/{{ ansible_ssh_host }}/{{ item }}" + dest: "{{ reboot_limit_local_results_dir }}/kexec/{{ ansible_ssh_host }}/{{ item }}" + flat: yes + with_items: + - "{{ reboot_limits_systemctl_analyze_log }}" + - "{{ reboot_limits_count_log }}" + when: reboot_limit_compare_both_enabled|default(false)|bool diff --git a/scripts/workflows/demos/reboot-limit/analyze_results.py b/scripts/workflows/demos/reboot-limit/analyze_results.py index bfd6f41a..1f87330e 100755 --- a/scripts/workflows/demos/reboot-limit/analyze_results.py +++ b/scripts/workflows/demos/reboot-limit/analyze_results.py @@ -27,6 +27,9 @@ class RebootLimitAnalyzer: def __init__(self, results_dir: str): self.results_dir = Path(results_dir) self.hosts_data: Dict[str, Dict] = {} + self.comparison_mode = False + self.regular_data: Dict[str, Dict] = {} + self.kexec_data: Dict[str, Dict] = {} def parse_systemd_analyze_line(self, line: str) -> Optional[Dict[str, float]]: """ @@ -88,10 +91,28 @@ class RebootLimitAnalyzer: def load_all_data(self): """Load data for all hosts in the results directory.""" - # Look for host directories - for item in self.results_dir.iterdir(): - if item.is_dir(): - self.hosts_data[item.name] = self.load_host_data(item) + # Check if we're in comparison mode (regular/ and kexec/ subdirs exist) + regular_dir = self.results_dir / "regular" + kexec_dir = self.results_dir / "kexec" + + if regular_dir.exists() and kexec_dir.exists(): + self.comparison_mode = True + print("Detected comparison mode: analyzing both regular and kexec reboots") + + # Load regular reboot data + for item in regular_dir.iterdir(): + if item.is_dir(): + self.regular_data[item.name] = self.load_host_data(item) + + # Load kexec reboot data + for item in kexec_dir.iterdir(): + if item.is_dir(): + self.kexec_data[item.name] = self.load_host_data(item) + else: + # Standard single mode + for item in self.results_dir.iterdir(): + if item.is_dir(): + self.hosts_data[item.name] = self.load_host_data(item) def calculate_statistics(self, times: List[float]) -> Dict[str, float]: """Calculate statistical measures for a list of times.""" @@ -108,6 +129,13 @@ class RebootLimitAnalyzer: def plot_boot_times(self, output_file: str = "reboot_limit_analysis.png"): """Generate plots for boot time analysis.""" + if self.comparison_mode: + self.plot_comparison_analysis(output_file) + else: + self.plot_single_mode_analysis(output_file) + + def plot_single_mode_analysis(self, output_file: str): + """Generate plots for single mode analysis.""" if not self.hosts_data: print("No data to plot") return @@ -221,8 +249,190 @@ class RebootLimitAnalyzer: plt.savefig(output_file, dpi=300, bbox_inches="tight") print(f"Saved plot to {output_file}") + def plot_comparison_analysis(self, output_file: str): + """Generate streamlined comparison plot combining all hosts' regular vs kexec data.""" + if not self.regular_data and not self.kexec_data: + print("No comparison data to plot") + return + + # Ensure the output directory exists + output_path = Path(output_file) + if output_path.parent != Path("."): + output_path.parent.mkdir(parents=True, exist_ok=True) + + # Combine data from all hosts + all_regular_times = [] + all_kexec_times = [] + host_labels = [] + + # Collect all data points from all hosts + all_hosts = sorted(set(self.regular_data.keys()) | set(self.kexec_data.keys())) + + for host in all_hosts: + regular_data = self.regular_data.get(host, {"boot_times": []}) + kexec_data = self.kexec_data.get(host, {"boot_times": []}) + + regular_times = regular_data.get("boot_times", []) + kexec_times = kexec_data.get("boot_times", []) + + # Add host data to combined arrays + if regular_times: + regular_totals = [bt["total"] for bt in regular_times] + all_regular_times.extend(regular_totals) + host_labels.extend([f"{host}-regular"] * len(regular_totals)) + + if kexec_times: + kexec_totals = [bt["total"] for bt in kexec_times] + all_kexec_times.extend(kexec_totals) + + # Create single comprehensive comparison plot + fig, ax = plt.subplots(1, 1, figsize=(12, 8)) + + # Plot combined data + if all_regular_times: + regular_boot_numbers = list(range(1, len(all_regular_times) + 1)) + ax.plot( + regular_boot_numbers, + all_regular_times, + "b-", + linewidth=2, + label="Regular Reboot", + alpha=0.8, + marker="o", + markersize=4, + ) + + if all_kexec_times: + kexec_boot_numbers = list(range(1, len(all_kexec_times) + 1)) + ax.plot( + kexec_boot_numbers, + all_kexec_times, + "g-", + linewidth=2, + label="Kexec Reboot", + alpha=0.8, + marker="s", + markersize=4, + ) + + # Calculate and display combined statistics + if all_regular_times and all_kexec_times: + regular_stats = self.calculate_statistics(all_regular_times) + kexec_stats = self.calculate_statistics(all_kexec_times) + + if regular_stats and kexec_stats: + # Add mean lines + ax.axhline( + y=regular_stats["mean"], + color="blue", + linestyle="--", + alpha=0.7, + linewidth=2, + label=f"Regular Mean: {regular_stats['mean']:.2f}s", + ) + ax.axhline( + y=kexec_stats["mean"], + color="green", + linestyle="--", + alpha=0.7, + linewidth=2, + label=f"Kexec Mean: {kexec_stats['mean']:.2f}s", + ) + + # Add shaded confidence regions + ax.fill_between( + range(1, max(len(all_regular_times), len(all_kexec_times)) + 1), + regular_stats["mean"] - regular_stats["stdev"], + regular_stats["mean"] + regular_stats["stdev"], + alpha=0.2, + color="blue", + label=f"Regular ±1σ: {regular_stats['stdev']:.2f}s", + ) + + ax.fill_between( + range(1, max(len(all_regular_times), len(all_kexec_times)) + 1), + kexec_stats["mean"] - kexec_stats["stdev"], + kexec_stats["mean"] + kexec_stats["stdev"], + alpha=0.2, + color="green", + label=f"Kexec ±1σ: {kexec_stats['stdev']:.2f}s", + ) + + # Calculate performance improvement + speedup = ( + regular_stats["mean"] / kexec_stats["mean"] + if kexec_stats["mean"] > 0 + else 0 + ) + time_saved = regular_stats["mean"] - kexec_stats["mean"] + percent_improvement = ( + (regular_stats["mean"] - kexec_stats["mean"]) + / regular_stats["mean"] + ) * 100 + + # Create comprehensive statistics text + stats_text = f"PERFORMANCE COMPARISON\n" + stats_text += f"{'='*25}\n" + stats_text += f"Regular Boot:\n" + stats_text += f" Mean: {regular_stats['mean']:.2f}s\n" + stats_text += f" Range: {regular_stats['min']:.2f}s - {regular_stats['max']:.2f}s\n" + stats_text += f" Samples: {len(all_regular_times)}\n\n" + stats_text += f"Kexec Boot:\n" + stats_text += f" Mean: {kexec_stats['mean']:.2f}s\n" + stats_text += ( + f" Range: {kexec_stats['min']:.2f}s - {kexec_stats['max']:.2f}s\n" + ) + stats_text += f" Samples: {len(all_kexec_times)}\n\n" + stats_text += f"IMPROVEMENT:\n" + stats_text += f" Speedup: {speedup:.2f}x faster\n" + stats_text += f" Time Saved: {time_saved:.2f}s per boot\n" + stats_text += f" Improvement: {percent_improvement:.1f}%\n\n" + stats_text += f"Combined Hosts: {', '.join(all_hosts)}" + + # Position statistics box + ax.text( + 0.02, + 0.98, + stats_text, + transform=ax.transAxes, + verticalalignment="top", + fontsize=10, + bbox=dict( + boxstyle="round,pad=0.5", facecolor="lightblue", alpha=0.8 + ), + ) + + # Customize plot appearance + ax.set_xlabel("Boot Sequence", fontsize=12, fontweight="bold") + ax.set_ylabel("Boot Time (seconds)", fontsize=12, fontweight="bold") + ax.set_title( + "Regular vs Kexec Boot Performance Comparison\n(Combined Data from All Hosts)", + fontsize=14, + fontweight="bold", + ) + ax.legend(loc="upper right", fontsize=10) + ax.grid(True, alpha=0.3) + ax.xaxis.set_major_locator(ticker.MaxNLocator(integer=True)) + + # Improve visual styling + ax.spines["top"].set_visible(False) + ax.spines["right"].set_visible(False) + ax.spines["left"].set_linewidth(0.5) + ax.spines["bottom"].set_linewidth(0.5) + + plt.tight_layout() + plt.savefig(output_file, dpi=300, bbox_inches="tight", facecolor="white") + print(f"Saved comparison plot to {output_file}") + def print_summary(self): """Print a summary of the analysis to stdout.""" + if self.comparison_mode: + self.print_comparison_summary() + else: + self.print_single_mode_summary() + + def print_single_mode_summary(self): + """Print summary for single mode analysis.""" for host, data in self.hosts_data.items(): print(f"\n{'=' * 60}") print(f"Host: {host}") @@ -255,6 +465,79 @@ class RebootLimitAnalyzer: else: print(" No boot time data available") + def print_comparison_summary(self): + """Print summary for comparison mode analysis.""" + all_hosts = set(self.regular_data.keys()) | set(self.kexec_data.keys()) + + for host in sorted(all_hosts): + print(f"\n{'=' * 80}") + print(f"Host: {host} - COMPARISON ANALYSIS") + print(f"{'=' * 80}") + + regular_data = self.regular_data.get( + host, {"boot_times": [], "boot_count": 0} + ) + kexec_data = self.kexec_data.get(host, {"boot_times": [], "boot_count": 0}) + + print(f"\nREGULAR REBOOT RESULTS:") + print(f" Total boots: {regular_data['boot_count']}") + + if regular_data["boot_times"]: + regular_total_times = [bt["total"] for bt in regular_data["boot_times"]] + regular_stats = self.calculate_statistics(regular_total_times) + + print(f" Samples analyzed: {len(regular_total_times)}") + if regular_stats: + print(f" Mean: {regular_stats['mean']:.2f}s") + print(f" StdDev: {regular_stats['stdev']:.2f}s") + print( + f" Range: {regular_stats['max'] - regular_stats['min']:.2f}s" + ) + else: + print(" No boot time data available") + regular_stats = None + + print(f"\nKEXEC REBOOT RESULTS:") + print(f" Total boots: {kexec_data['boot_count']}") + + if kexec_data["boot_times"]: + kexec_total_times = [bt["total"] for bt in kexec_data["boot_times"]] + kexec_stats = self.calculate_statistics(kexec_total_times) + + print(f" Samples analyzed: {len(kexec_total_times)}") + if kexec_stats: + print(f" Mean: {kexec_stats['mean']:.2f}s") + print(f" StdDev: {kexec_stats['stdev']:.2f}s") + print(f" Range: {kexec_stats['max'] - kexec_stats['min']:.2f}s") + else: + print(" No boot time data available") + kexec_stats = None + + # Comparison analysis + if regular_stats and kexec_stats: + print(f"\nCOMPARISON ANALYSIS:") + speedup = ( + regular_stats["mean"] / kexec_stats["mean"] + if kexec_stats["mean"] > 0 + else 0 + ) + time_saved = regular_stats["mean"] - kexec_stats["mean"] + + print(f" Kexec Speedup: {speedup:.2f}x faster") + print(f" Time Saved per Boot: {time_saved:.2f}s") + print(f" Regular Mean: {regular_stats['mean']:.2f}s") + print(f" Kexec Mean: {kexec_stats['mean']:.2f}s") + + if speedup > 1.1: + print(f" ✓ Kexec provides significant speedup!") + elif speedup > 1.0: + print(f" → Kexec provides minor speedup") + else: + print(f" ⚠ Regular reboot is faster") + else: + print(f"\nCOMPARISON ANALYSIS:") + print(f" Cannot compare - missing data for one or both reboot types") + def main(): parser = argparse.ArgumentParser( @@ -287,10 +570,27 @@ def main(): analyzer = RebootLimitAnalyzer(args.results_dir) analyzer.load_all_data() - if not analyzer.hosts_data: + # Check if we have data (either single mode or comparison mode) + has_data = False + if analyzer.comparison_mode: + if analyzer.regular_data or analyzer.kexec_data: + has_data = True + else: + if analyzer.hosts_data: + has_data = True + + if not has_data: print(f"No host data found in '{args.results_dir}'") - print("Make sure you've run 'make reboot-limit-baseline' first") - sys.exit(1) + if analyzer.comparison_mode: + print( + "Make sure you've run 'make reboot-limit-tests' with comparison mode enabled" + ) + else: + print( + "Make sure you've run 'make reboot-limit-baseline' or 'make reboot-limit-tests' first" + ) + print("This is normal if you haven't run any tests yet.") + sys.exit(0) # Exit cleanly when no data exists # Print summary analyzer.print_summary() diff --git a/workflows/demos/reboot-limit/Kconfig b/workflows/demos/reboot-limit/Kconfig index 91fcd122..ecafe4bd 100644 --- a/workflows/demos/reboot-limit/Kconfig +++ b/workflows/demos/reboot-limit/Kconfig @@ -32,6 +32,14 @@ config REBOOT_LIMIT_TYPE_SYSTEMD_KEXEC help This will try to reboot instead using systemctl kexec. +config REBOOT_LIMIT_TYPE_COMPARE_BOTH + bool "Compare regular reboot vs kexec" + help + This will test both regular reboot (ansible) and kexec reboot + sequentially to compare their performance characteristics. + Regular reboot will be tested first, followed by kexec reboot. + Statistics will be collected separately for each reboot type. + endchoice config REBOOT_LIMIT_TEST_TYPE @@ -39,6 +47,13 @@ config REBOOT_LIMIT_TEST_TYPE default "ansible" if REBOOT_LIMIT_TYPE_ANSIBLE default "systemctl_reboot" if REBOOT_LIMIT_TYPE_SYSTEMD_REBOOT default "systemctl_kexec" if REBOOT_LIMIT_TYPE_SYSTEMD_KEXEC + default "compare_both" if REBOOT_LIMIT_TYPE_COMPARE_BOTH + +config REBOOT_LIMIT_COMPARE_BOTH_ENABLED + bool + default y if REBOOT_LIMIT_TYPE_COMPARE_BOTH + help + Internal configuration variable to track when comparison mode is enabled. config REBOOT_LIMIT_BOOT_MAX int "How many reboots should we do to consider reboots OK?" @@ -119,6 +134,22 @@ config REBOOT_LIMIT_DATA of each node. Note that {{data_path}} corresponds to the location set by the configuration option CONFIG_WORKFLOW_DATA_PATH. +config REBOOT_LIMIT_DATA_REGULAR + string "Where to place regular reboot statistics in comparison mode" + default "{{data_path}}/reboot-limit/regular" + depends on REBOOT_LIMIT_COMPARE_BOTH_ENABLED + help + This is the target location for regular (ansible) reboot statistics + when running in comparison mode. + +config REBOOT_LIMIT_DATA_KEXEC + string "Where to place kexec reboot statistics in comparison mode" + default "{{data_path}}/reboot-limit/kexec" + depends on REBOOT_LIMIT_COMPARE_BOTH_ENABLED + help + This is the target location for kexec reboot statistics + when running in comparison mode. + config REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE bool "Enable data collection of systemd-analyze results" default y diff --git a/workflows/demos/reboot-limit/Makefile b/workflows/demos/reboot-limit/Makefile index 384442f4..680eed88 100644 --- a/workflows/demos/reboot-limit/Makefile +++ b/workflows/demos/reboot-limit/Makefile @@ -47,6 +47,14 @@ ifeq (y,$(CONFIG_REBOOT_LIMIT_ENABLE_DATA_COLLECTION)) REBOOT_LIMIT_DATA :=$(subst ",,$(CONFIG_REBOOT_LIMIT_DATA)) REBOOT_LIMIT_ARGS += reboot_limit_data=\"$(REBOOT_LIMIT_DATA)\" +ifeq (y,$(CONFIG_REBOOT_LIMIT_COMPARE_BOTH_ENABLED)) +REBOOT_LIMIT_DATA_REGULAR :=$(subst ",,$(CONFIG_REBOOT_LIMIT_DATA_REGULAR)) +REBOOT_LIMIT_DATA_KEXEC :=$(subst ",,$(CONFIG_REBOOT_LIMIT_DATA_KEXEC)) +REBOOT_LIMIT_ARGS += reboot_limit_data_regular=\"$(REBOOT_LIMIT_DATA_REGULAR)\" +REBOOT_LIMIT_ARGS += reboot_limit_data_kexec=\"$(REBOOT_LIMIT_DATA_KEXEC)\" +REBOOT_LIMIT_ARGS += reboot_limit_compare_both_enabled=True +endif + # This is an example of how to map a boolean from kconfig into ansible ifeq (y,$(CONFIG_REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE)) REBOOT_LIMIT_ARGS += reboot_limit_enable_systemd_analyze=True @@ -75,6 +83,11 @@ endif # the extra_vars.yaml file. WORKFLOW_ARGS += $(REBOOT_LIMIT_ARGS) +# Export workflow enabled flag for ansible +ifeq (y,$(CONFIG_WORKFLOWS_REBOOT_LIMIT)) +WORKFLOW_ARGS += workflows_reboot_limit=True +endif + # The default is our workflow does not have loop testing enabled REBOOT_LIMIT_LOOP := false REBOOT_LIMIT_LOOP_KOTD := false -- 2.47.2