From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52A5F2E0915 for ; Mon, 11 Aug 2025 22:43:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754952192; cv=none; b=pIjYIrrHbeJNttjwMlr6yB5Sk+yElABcSj5Yiv5IGY6UITTWPVWIGGb41Ix59c2aVW9a6dTCbdgti64VmMdTqa+yzEMbGIIY3SuPmsXsKhdEsAEr5tDihuS0GqWSg8nzCjwgAYb75iScqaKrbdIJoiJDDg/s07mFl7s1mxS5VSA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754952192; c=relaxed/simple; bh=WNIR76VB5fTmDkycyQJDpwyZ5MXTrV7vJIu39tScOJw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OlwxsOYOqaJg9ZyAC6eg+J9UkfxWjr9eH8FvDSZVPECtJB8oyhJeG3F0MRPl7z+gcLj9JcqInBNnxh4yCsA1mrohhtP+MwR+4uMmKA5BZtCm57icj2QAi87sZLIVaOkqXVOzbGuVotC4WvvT05CBt4swULeEruh4HNChg/OAdQ4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=SN2/Cuhf; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="SN2/Cuhf" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=eEddz+M2iykP8f0MOHo/oNAVEN1Vq78NCpam3lcS0HE=; b=SN2/CuhfF/VqfWUX56FMO2lr8b T2WqCRgEp9mLRK5ikM41AkGDne1K9TlyToTykppcC6RluB7+jiipFf+bRL/07u8zpaXPAH4eeDFcv V9Nv2cUQpU2xlz+FE/mNacfWenlea6Sx27+Bf+s2Mwd3kNHpHkA37OHKNEroZsLjy6wnJZEW3aTsx 2zra/ddPmgGFr+FYj5LYrj+pFeb/mP6bl+cTvSXz/fhNr9cpMsImoRU6zPAshhfRcEWebnG0/dXeW JAFB2tVtN3+bpbPgUO7Od+xQavs6nMC+VWT86Ddz4DrGxjlmKVkmApb6z2vVBhDNH+HPshfalP5R8 mKEKOE/g==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulbET-00000009J8R-3m4Z; Mon, 11 Aug 2025 22:43:09 +0000 From: Luis Chamberlain To: Chuck Lever , Daniel Gomez , kdevops@lists.linux.dev Cc: Luis Chamberlain Subject: [PATCH 5/8] bootlinux: add intelligent git repository detection and management Date: Mon, 11 Aug 2025 15:43:04 -0700 Message-ID: <20250811224307.2218478-6-mcgrof@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250811224307.2218478-1-mcgrof@kernel.org> References: <20250811224307.2218478-1-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain Add smart logic to infer when git clone is needed regardless of the bootlinux_tree_set_by_cli setting. This solves the issue where directories exist with files (like .config) but no .git repository, which previously caused the git clone to be skipped. The new logic: 1. Checks if the target directory exists 2. Checks if .git directory exists within it 3. Infers git clone is needed if directory exists but .git doesn't 4. Clones the repository even when bootlinux_tree_set_by_cli is true 5. Ensures the correct ref is checked out if git exists but on wrong ref This handles the common case where: - Directory is pre-created for 9p mount support - Configuration files are copied before git clone - bootlinux_tree_set_by_cli persists from previous runs The implementation also: - Fetches updates if the target ref doesn't exist locally - Switches to the correct ref if repository exists but is on wrong branch - Maintains backward compatibility with existing workflows - Applies consistently across all build methods (9p, targets, builder) This makes the system more robust and user-friendly by intelligently handling partial setups and recovering from incomplete states. Generated-by: Claude AI Signed-off-by: Luis Chamberlain --- playbooks/roles/bootlinux/tasks/build/9p.yml | 119 ++++++++++++++---- .../roles/bootlinux/tasks/build/builder.yml | 98 +++++++++++++-- .../roles/bootlinux/tasks/build/targets.yml | 98 +++++++++++++-- 3 files changed, 273 insertions(+), 42 deletions(-) diff --git a/playbooks/roles/bootlinux/tasks/build/9p.yml b/playbooks/roles/bootlinux/tasks/build/9p.yml index 5c9489c3..d0ae61ad 100644 --- a/playbooks/roles/bootlinux/tasks/build/9p.yml +++ b/playbooks/roles/bootlinux/tasks/build/9p.yml @@ -26,23 +26,47 @@ run_once: true delegate_to: localhost -- name: Check if target directory exists when using 9p and Linux CLI was set +- name: Check if target directory exists stat: path: "{{ bootlinux_9p_host_path }}" register: target_directory_stat run_once: true delegate_to: localhost + +- name: Check if .git directory exists in target path + stat: + path: "{{ bootlinux_9p_host_path }}/.git" + register: git_directory_stat + run_once: true + delegate_to: localhost when: - - bootlinux_tree_set_by_cli|bool + - target_directory_stat.stat.exists -- name: Fail if target directory does not exist when using 9p and Linux CLI was set - fail: - msg: "The target directory {{ bootlinux_9p_host_path }} does not exist." +- name: Infer that git clone is needed when .git doesn't exist + set_fact: + needs_git_clone: true + when: + - target_directory_stat.stat.exists + - not git_directory_stat.stat.exists run_once: true delegate_to: localhost + +- name: Set needs_git_clone when directory doesn't exist + set_fact: + needs_git_clone: true when: - - bootlinux_tree_set_by_cli|bool - not target_directory_stat.stat.exists + run_once: true + delegate_to: localhost + +- name: Set needs_git_clone to false when .git exists + set_fact: + needs_git_clone: false + when: + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + run_once: true + delegate_to: localhost - name: Verify target git ref exists before cloning command: "git ls-remote {{ target_linux_git }} {{ active_linux_ref | default(target_linux_ref) }}" @@ -50,7 +74,7 @@ run_once: true delegate_to: localhost when: - - not bootlinux_tree_set_by_cli|bool + - needs_git_clone|bool tags: [ 'clone'] - name: Fail if git ref does not exist @@ -68,20 +92,11 @@ - If using A/B testing with different refs, ensure shallow cloning is disabled - The repository URL '{{ target_linux_git }}' is correct and accessible when: - - not bootlinux_tree_set_by_cli|bool + - needs_git_clone|bool - ref_check.rc != 0 run_once: true delegate_to: localhost -- name: Check if target directory exists for dirty check - stat: - path: "{{ bootlinux_9p_host_path }}" - register: git_dir_stat - run_once: true - delegate_to: localhost - when: - - not bootlinux_tree_set_by_cli|bool - - name: Check if git tree is dirty command: "git -C {{ bootlinux_9p_host_path }} status --porcelain" register: git_status @@ -90,9 +105,9 @@ run_once: true delegate_to: localhost when: - - not bootlinux_tree_set_by_cli|bool - - git_dir_stat.stat.exists - - git_dir_stat.stat.isdir + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists - name: Fail if git tree has local modifications fail: @@ -109,9 +124,9 @@ Modified files: {{ git_status.stdout }} when: - - not bootlinux_tree_set_by_cli|bool - - git_dir_stat.stat.exists - - git_dir_stat.stat.isdir + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists - git_status.stdout | length > 0 run_once: true delegate_to: localhost @@ -129,9 +144,65 @@ until: not result.failed tags: [ 'clone'] when: - - not bootlinux_tree_set_by_cli|bool + - needs_git_clone|bool + run_once: true + delegate_to: localhost + +- name: Get current git ref when git exists but clone wasn't needed + command: "git -C {{ bootlinux_9p_host_path }} rev-parse HEAD" + register: current_ref + changed_when: false + run_once: true + delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + +- name: Get target ref SHA + command: "git -C {{ bootlinux_9p_host_path }} rev-parse {{ active_linux_ref | default(target_linux_ref) }}" + register: target_ref_sha + changed_when: false + failed_when: false run_once: true delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + +- name: Fetch updates if target ref doesn't exist locally + command: "git -C {{ bootlinux_9p_host_path }} fetch origin" + run_once: true + delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + +- name: Get target ref SHA after fetch + command: "git -C {{ bootlinux_9p_host_path }} rev-parse {{ active_linux_ref | default(target_linux_ref) }}" + register: target_ref_sha_after_fetch + changed_when: false + run_once: true + delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + +- name: Checkout target ref if not on correct ref + command: "git -C {{ bootlinux_9p_host_path }} checkout {{ active_linux_ref | default(target_linux_ref) }}" + run_once: true + delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or + (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0) - name: Copy kernel delta if requested on the control node template: diff --git a/playbooks/roles/bootlinux/tasks/build/builder.yml b/playbooks/roles/bootlinux/tasks/build/builder.yml index d36815ed..1213c56f 100644 --- a/playbooks/roles/bootlinux/tasks/build/builder.yml +++ b/playbooks/roles/bootlinux/tasks/build/builder.yml @@ -10,9 +10,43 @@ - target_linux_install_b4 - ansible_os_family == "Debian" +- name: Check if target directory exists + stat: + path: "{{ target_linux_dir_path }}" + register: target_directory_stat + +- name: Check if .git directory exists in target path + stat: + path: "{{ target_linux_dir_path }}/.git" + register: git_directory_stat + when: + - target_directory_stat.stat.exists + +- name: Infer that git clone is needed when .git doesn't exist + set_fact: + needs_git_clone: true + when: + - target_directory_stat.stat.exists + - not git_directory_stat.stat.exists + +- name: Set needs_git_clone when directory doesn't exist + set_fact: + needs_git_clone: true + when: + - not target_directory_stat.stat.exists + +- name: Set needs_git_clone to false when .git exists + set_fact: + needs_git_clone: false + when: + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - name: Verify target git ref exists before cloning command: "git ls-remote {{ target_linux_git }} {{ target_linux_ref }}" register: ref_check + when: + - needs_git_clone|bool - name: Fail if git ref does not exist fail: @@ -29,21 +63,18 @@ - If using A/B testing with different refs, ensure shallow cloning is disabled - The repository URL '{{ target_linux_git }}' is correct and accessible when: + - needs_git_clone|bool - ref_check.rc != 0 -- name: Check if target directory exists for dirty check - stat: - path: "{{ target_linux_dir_path }}" - register: git_dir_stat - - name: Check if git tree is dirty command: "git -C {{ target_linux_dir_path }} status --porcelain" register: git_status changed_when: false failed_when: false when: - - git_dir_stat.stat.exists - - git_dir_stat.stat.isdir + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists - name: Fail if git tree has local modifications fail: @@ -60,8 +91,9 @@ Modified files: {{ git_status.stdout }} when: - - git_dir_stat.stat.exists - - git_dir_stat.stat.isdir + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists - git_status.stdout | length > 0 - name: Clone {{ target_linux_tree }} @@ -75,6 +107,54 @@ retries: 3 delay: 5 until: result is succeeded + when: + - needs_git_clone|bool + +- name: Get current git ref when git exists but clone wasn't needed + command: "git -C {{ target_linux_dir_path }} rev-parse HEAD" + register: current_ref + changed_when: false + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + +- name: Get target ref SHA + command: "git -C {{ target_linux_dir_path }} rev-parse {{ target_linux_ref }}" + register: target_ref_sha + changed_when: false + failed_when: false + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + +- name: Fetch updates if target ref doesn't exist locally + command: "git -C {{ target_linux_dir_path }} fetch origin" + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + +- name: Get target ref SHA after fetch + command: "git -C {{ target_linux_dir_path }} rev-parse {{ target_linux_ref }}" + register: target_ref_sha_after_fetch + changed_when: false + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + +- name: Checkout target ref if not on correct ref + command: "git -C {{ target_linux_dir_path }} checkout {{ target_linux_ref }}" + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or + (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0) - name: Copy the kernel delta to the builder ansible.builtin.template: diff --git a/playbooks/roles/bootlinux/tasks/build/targets.yml b/playbooks/roles/bootlinux/tasks/build/targets.yml index 414602ab..87393c74 100644 --- a/playbooks/roles/bootlinux/tasks/build/targets.yml +++ b/playbooks/roles/bootlinux/tasks/build/targets.yml @@ -10,10 +10,44 @@ - target_linux_install_b4 - ansible_facts['os_family']|lower != 'debian' +- name: Check if target directory exists + stat: + path: "{{ target_linux_dir_path }}" + register: target_directory_stat + +- name: Check if .git directory exists in target path + stat: + path: "{{ target_linux_dir_path }}/.git" + register: git_directory_stat + when: + - target_directory_stat.stat.exists + +- name: Infer that git clone is needed when .git doesn't exist + set_fact: + needs_git_clone: true + when: + - target_directory_stat.stat.exists + - not git_directory_stat.stat.exists + +- name: Set needs_git_clone when directory doesn't exist + set_fact: + needs_git_clone: true + when: + - not target_directory_stat.stat.exists + +- name: Set needs_git_clone to false when .git exists + set_fact: + needs_git_clone: false + when: + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - name: Verify target git ref exists before cloning command: "git ls-remote {{ target_linux_git }} {{ target_linux_ref }}" register: ref_check tags: [ 'clone'] + when: + - needs_git_clone|bool - name: Fail if git ref does not exist fail: @@ -30,21 +64,18 @@ - If using A/B testing with different refs, ensure shallow cloning is disabled - The repository URL '{{ target_linux_git }}' is correct and accessible when: + - needs_git_clone|bool - ref_check.rc != 0 -- name: Check if target directory exists for dirty check - stat: - path: "{{ target_linux_dir_path }}" - register: git_dir_stat - - name: Check if git tree is dirty command: "git -C {{ target_linux_dir_path }} status --porcelain" register: git_status changed_when: false failed_when: false when: - - git_dir_stat.stat.exists - - git_dir_stat.stat.isdir + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists - name: Fail if git tree has local modifications fail: @@ -61,8 +92,9 @@ Modified files: {{ git_status.stdout }} when: - - git_dir_stat.stat.exists - - git_dir_stat.stat.isdir + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists - git_status.stdout | length > 0 - name: git clone {{ target_linux_tree }} on the target nodes @@ -77,6 +109,54 @@ register: result until: not result.failed tags: [ 'clone'] + when: + - needs_git_clone|bool + +- name: Get current git ref when git exists but clone wasn't needed + command: "git -C {{ target_linux_dir_path }} rev-parse HEAD" + register: current_ref + changed_when: false + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + +- name: Get target ref SHA + command: "git -C {{ target_linux_dir_path }} rev-parse {{ target_linux_ref }}" + register: target_ref_sha + changed_when: false + failed_when: false + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + +- name: Fetch updates if target ref doesn't exist locally + command: "git -C {{ target_linux_dir_path }} fetch origin" + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + +- name: Get target ref SHA after fetch + command: "git -C {{ target_linux_dir_path }} rev-parse {{ target_linux_ref }}" + register: target_ref_sha_after_fetch + changed_when: false + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + +- name: Checkout target ref if not on correct ref + command: "git -C {{ target_linux_dir_path }} checkout {{ target_linux_ref }}" + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or + (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0) - name: Copy kernel delta if requested on the target nodes template: -- 2.47.2