From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7C8C1F37D3 for ; Fri, 1 Aug 2025 19:46:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754077600; cv=none; b=VkgiAl/b7mv4aAZ2khkVdIPI2CtZ0A6UU1zT3WIifAwbePwHwV0NwuqtNsWBQ8AjW63iR3/qRhFAdTGqYRCEceRwfmQ9Oi8lIFWGwptLqbx6P+MAt2r1MDN1TTmQ6XyJOAdo7mqFpIzoqm1yIo5W/GO00FVVC5i00IDbVYVmZVY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754077600; c=relaxed/simple; bh=tmV9jZOX8SCm4HS3bK+QJZ2fhzihwtIHubN3Vv4Ga2I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nFPM9rugzvV3Rv3wDjQ9MJJuFoiQvq4hMnRuFYw0TY89dJGm01pVZcTj1M7OoC0azaWg3DdeZNHFuKGKZBiC8CUl0IwnnYO+PeQ7AkGKXCwUQq1Kj5P4BQSBZRceensnbyymKAAoSZID7wTEskc4QRljzXyv6g1oNdGISDPLkfQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=PSoqlT2I; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="PSoqlT2I" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=dm6v/NYbsIDl4c3qwvjQNFN4JA0EoPJwdKeHGV94c+w=; b=PSoqlT2IHXWTuqFh+y+xoPqjUp AVmgI6igW+CgWleuan1gShHlvfPaQAudk7MsFjBlrzhWuvFluH6/3oFGGdqUUL6NRxHRslwjd7nkW d9vsY8PWdp/Wv3ncHMU3WWIGF9kmscppA6UlwZZVj+Pz93N5e80SsVtlYvzmmtyaTpPS8D7JTmMJ4 mV+KgpIczoYNGyc8tiuRgZAYqtUR2ALP3pAK/q1lPwO8wsoiC0yyGI0iilxt8YFi42NcjgrJ/WvYX e+3XCB7bxWf//fpx+/UX5ndtrn3rnPItXPt+OzmtC9LZWFkAMwnhhltWGz/TjTESMJCEtyuN2yzHu wO/DLBew==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98.2 #2 (Red Hat Linux)) id 1uhvi9-00000006hrl-1PJR; Fri, 01 Aug 2025 19:46:37 +0000 From: Luis Chamberlain To: Chuck Lever , Daniel Gomez , kdevops@lists.linux.dev Cc: Luis Chamberlain Subject: [PATCH v3 10/11] devconfig: enhance hop1 detection to support traditional sources.list Date: Fri, 1 Aug 2025 12:46:34 -0700 Message-ID: <20250801194635.1598544-11-mcgrof@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250801194635.1598544-1-mcgrof@kernel.org> References: <20250801194635.1598544-1-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain Enhance the hop1 mirror detection to work with both DEB822 and traditional sources.list formats. When a hop1 mirror is detected on the control host (regardless of format) and the guest is running Debian testing, automatically convert to DEB822 format for consistency and modern package management. Key improvements: - Detect hop1 servers from both /etc/apt/sources.list.d/debian.sources (DEB822) and /etc/apt/sources.list (traditional format) - Extract mirror host and path correctly from both formats - Automatically convert Debian testing guests to use DEB822 when a hop1 mirror is available and accessible - Preserve the mirror path (e.g., /debian, /debian-mirror) from the control host configuration - Add connectivity checks before using the hop1 mirror - Fall back to official Debian mirrors only if hop1 is unavailable or inaccessible This ensures that local hop1 mirrors are used efficiently even when the control host hasn't migrated to DEB822 format yet, while modernizing the guest configuration for Debian testing systems. Generated-by: Claude AI Signed-off-by: Luis Chamberlain --- .../devconfig/tasks/check-apt-mirrors.yml | 114 ++++++++++++++++++ .../templates/debian-hop1-mirror.sources | 13 ++ 2 files changed, 127 insertions(+) create mode 100644 playbooks/roles/devconfig/templates/debian-hop1-mirror.sources diff --git a/playbooks/roles/devconfig/tasks/check-apt-mirrors.yml b/playbooks/roles/devconfig/tasks/check-apt-mirrors.yml index 96e4048015d1..d86196f15ebe 100644 --- a/playbooks/roles/devconfig/tasks/check-apt-mirrors.yml +++ b/playbooks/roles/devconfig/tasks/check-apt-mirrors.yml @@ -1,5 +1,55 @@ --- # Only run mirror checks for Debian testing (trixie) where mirror issues are common +- name: Check hop count on control host + delegate_to: localhost + shell: | + {{ topdir_path }}/scripts/get-distro-has-hop-count-sources.sh 1 + register: localhost_hop_count + changed_when: false + ignore_errors: yes + +- name: Extract hop1 mirror info from control host (DEB822 format) + delegate_to: localhost + shell: | + if [ -f /etc/apt/sources.list.d/debian.sources ]; then + HOST=$(grep -E "^URIs:" /etc/apt/sources.list.d/debian.sources | head -1 | awk '{print $2}' | sed -E 's|https?://||') + echo "${HOST}" + fi + register: localhost_hop1_mirror_deb822 + changed_when: false + ignore_errors: yes + when: localhost_hop_count.stdout == 'y' + +- name: Extract hop1 mirror info from control host (legacy format) + delegate_to: localhost + shell: | + if [ -f /etc/apt/sources.list ] && [ ! -f /etc/apt/sources.list.d/debian.sources ]; then + LINE=$(grep -v "^#" /etc/apt/sources.list | grep -E "^deb\s+http" | head -1) + if [ -n "$LINE" ]; then + URL=$(echo $LINE | awk '{print $2}') + echo "${URL}" | sed 's|http://||' + fi + fi + register: localhost_hop1_mirror_legacy + changed_when: false + ignore_errors: yes + when: localhost_hop_count.stdout == 'y' + +- name: Set hop1 mirror variables + set_fact: + has_hop1_mirror: "{{ localhost_hop_count.stdout == 'y' }}" + hop1_mirror_full: "{{ localhost_hop1_mirror_deb822.stdout if localhost_hop1_mirror_deb822.stdout != '' else localhost_hop1_mirror_legacy.stdout }}" + when: localhost_hop_count.stdout == 'y' + +- name: Parse hop1 mirror host and path + set_fact: + hop1_mirror_host: "{{ hop1_mirror_full.split('/')[0] }}" + hop1_mirror_path: "/{{ hop1_mirror_full.split('/', 1)[1] if '/' in hop1_mirror_full else 'debian' }}" + when: + - has_hop1_mirror is defined + - has_hop1_mirror | bool + - hop1_mirror_full != '' + - name: Check for DEB822-style sources stat: path: /etc/apt/sources.list.d/debian.sources @@ -42,6 +92,69 @@ Mirror connectivity: {{ 'OK' if mirror_connectivity is not failed else 'FAILED' }} when: apt_mirror_host.stdout != "" +- name: Configure APT sources based on hop1 availability and debian testing + block: + - name: Check connectivity to hop1 mirror if available + wait_for: + host: "{{ hop1_mirror_host }}" + port: 80 + timeout: 10 + register: hop1_mirror_connectivity + ignore_errors: yes + when: + - has_hop1_mirror is defined + - has_hop1_mirror | bool + - hop1_mirror_host is defined + + - name: Use hop1 mirror with DEB822 format for debian testing + block: + - name: Backup current sources + copy: + src: "{{ item }}" + dest: "{{ item }}.backup" + remote_src: yes + become: yes + loop: + - /etc/apt/sources.list + - /etc/apt/sources.list.d/debian.sources + ignore_errors: yes + + - name: Apply hop1 mirror configuration using DEB822 format + template: + src: debian-hop1-mirror.sources + dest: /etc/apt/sources.list.d/debian.sources + owner: root + group: root + mode: '0644' + become: yes + + - name: Remove legacy sources.list if migrating to DEB822 + file: + path: /etc/apt/sources.list + state: absent + become: yes + when: not deb822_sources.stat.exists + + - name: Update APT cache with hop1 mirror + apt: + update_cache: yes + cache_valid_time: 0 + become: yes + + - name: Inform user about hop1 mirror usage + debug: + msg: | + Using local hop1 mirror with DEB822 format: + Mirror: {{ hop1_mirror_host }}{{ hop1_mirror_path }} + + This provides faster package downloads from your local mirror. + Sources have been converted to modern DEB822 format. + when: + - has_hop1_mirror is defined + - has_hop1_mirror | bool + - hop1_mirror_connectivity is not failed + - hop1_mirror_host is defined + - name: Fall back to official Debian mirrors if current mirror fails block: - name: Backup current sources (DEB822 format) @@ -97,3 +210,4 @@ when: - apt_mirror_host.stdout != "" - mirror_connectivity is failed + - not (has_hop1_mirror is defined and has_hop1_mirror | bool and hop1_mirror_connectivity is not failed) diff --git a/playbooks/roles/devconfig/templates/debian-hop1-mirror.sources b/playbooks/roles/devconfig/templates/debian-hop1-mirror.sources new file mode 100644 index 000000000000..eff66706396e --- /dev/null +++ b/playbooks/roles/devconfig/templates/debian-hop1-mirror.sources @@ -0,0 +1,13 @@ +Types: deb deb-src +URIs: http://{{ hop1_mirror_host }}{{ hop1_mirror_path | default('/debian') }} +Suites: testing testing-updates +Components: main contrib non-free non-free-firmware +Enabled: yes +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg + +Types: deb deb-src +URIs: https://security.debian.org/debian-security +Suites: testing-security +Components: main contrib non-free non-free-firmware +Enabled: yes +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg \ No newline at end of file -- 2.47.2