From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D999283FD0 for ; Wed, 27 Aug 2025 09:32:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756287139; cv=none; b=NGcoD2LVLnXN6PPVYuZCh6LmwXXgjE6mC4aEh6afl2aMvjz1C4V7KgxG6lGNDctGDQrxlqncaICfRNqZbnPd7ZPrtg8bojYEDUFQMdJfeFP8HCeoBQkEa9tOBADTqkUtFHjc3g2aurKItrQZFNQKHJnvq3OrIn+L0q4hR0/Mgb4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756287139; c=relaxed/simple; bh=aheAAUzyYkZgaUopDBiq4be3kBFEEAzrvdp14rTpBfw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bE0mxqzguenx36TJVzrSt5KBL7gUR07Zpu7s8RLlJWO6kd6dld1V6b4xBpKRpknRdROE049DFyZui1TsOyVBaMIw8tkgA79B9Ndwp2I6Soze+qEn3VgCORZl3hVo1YcEyXAYXV70elp3IWq+YBIuhDD7QBPllZWseGHZfO4yXYw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=iZRt1DmR; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="iZRt1DmR" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=VkMfILJ7tvsjsxAOA2cNcwWk8hyrn9faCdDX/X/Gofg=; b=iZRt1DmRUtMNkgTu62YSZbiUEm xdAiwtgkFuWdX21v3KoW3DBCNDeZNLJi/YHBzbefEEVlvVnQNCaJZb+Gd5Qd2dViOWSD5QLnIlLI9 FGrGtVtTwGNL72CdcEQOubD/HCXYJc8VKvPTlCReluabbE1AohLfDDeBZYCvJA67msh2PRQmLPp/r vC+SwHyJMGuZ9H9Rn1okC5Lm7qCAl1WUNAsp2oZNkdp96ijrkZZvGYgUY/ghweZMIktm9ojfYFos3 lEz/2yxi9vFF8YlvLJbBe9vuVO01RvOLccjLg2M00sqjPXDDqNM7ckNit7bdRvUhGty6mToyNkz7e Fp91kuPg==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98.2 #2 (Red Hat Linux)) id 1urCVs-0000000Eqw6-3ilB; Wed, 27 Aug 2025 09:32:16 +0000 From: Luis Chamberlain To: Chuck Lever , Daniel Gomez , kdevops@lists.linux.dev Cc: Luis Chamberlain Subject: [PATCH 3/3] mirror: add Nix binary cache mirroring support Date: Wed, 27 Aug 2025 02:32:14 -0700 Message-ID: <20250827093215.3540056-4-mcgrof@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250827093215.3540056-1-mcgrof@kernel.org> References: <20250827093215.3540056-1-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain Add support for a Nix binary cache mirroring infrastructure for enterprise environments to reduce latencies and public bandwidth consumption for CIs. This dramatically reduces NixOS package download times from hundreds of packages to local cache hits. Key features: - nginx-based proxy caching for Nix binary cache - Automatic mirror detection for client-side configuration - systemd service and timer for cache synchronization - Integration with existing defconfig-mirror infrastructure - Client auto-configuration when local mirrors are available Components: - playbooks/roles/nix-cache-mirror/: Complete Ansible role - scripts/check_nix_mirror.sh: Mirror detection and URL discovery - kconfigs/Kconfig.mirror: Mirror configuration options - Integration with defconfig-mirror for one-time setup The mirror setup follows existing kdevops mirror patterns and integrates with the git mirror infrastructure already in place. Generated-by: Claude AI Signed-off-by: Luis Chamberlain --- defconfigs/mirror | 1 + kconfigs/Kconfig.mirror | 68 +++++++- kconfigs/Kconfig.nixos | 29 ++++ playbooks/linux-mirror.yml | 2 + playbooks/nixos.yml | 28 ++- .../linux-mirror/python/start-mirroring.py | 29 +++- .../roles/nix-cache-mirror/defaults/main.yml | 7 + .../roles/nix-cache-mirror/handlers/main.yml | 11 ++ .../roles/nix-cache-mirror/tasks/main.yml | 161 ++++++++++++++++++ .../templates/nix-cache-mirror-cache.conf.j2 | 3 + .../templates/nix-cache-mirror.nginx.j2 | 81 +++++++++ .../templates/nix-cache-sync.service.j2 | 17 ++ .../templates/nix-cache-sync.timer.j2 | 11 ++ scripts/check_nix_mirror.sh | 66 +++++++ 14 files changed, 507 insertions(+), 7 deletions(-) create mode 100644 playbooks/roles/nix-cache-mirror/defaults/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/handlers/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/tasks/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2 create mode 100755 scripts/check_nix_mirror.sh diff --git a/defconfigs/mirror b/defconfigs/mirror index efa0f9ef..05d51c01 100644 --- a/defconfigs/mirror +++ b/defconfigs/mirror @@ -2,3 +2,4 @@ CONFIG_SKIP_BRINGUP=y CONFIG_WORKFLOWS=n CONFIG_INSTALL_LOCAL_LINUX_MIRROR=y CONFIG_LINUX_MIRROR_NFS=y +CONFIG_INSTALL_NIX_CACHE_MIRROR=y diff --git a/kconfigs/Kconfig.mirror b/kconfigs/Kconfig.mirror index 312de47e..067258e5 100644 --- a/kconfigs/Kconfig.mirror +++ b/kconfigs/Kconfig.mirror @@ -379,7 +379,7 @@ choice config MIRROR_STABLE_RC_HTTPS bool "HTTPS (kernel.org)" help - If you enable this option then the mirror will use HTTPS to access + If you enable this option then the mirror will use HTTPS to access the linux-stable-rc repository on git.kernel.org. The full URL is: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git" @@ -387,7 +387,7 @@ config MIRROR_STABLE_RC_HTTPS config MIRROR_STABLE_RC_HTTPS_GOOGLE bool "HTTPS (Google)" help - If you enable this option then the mirror will use HTTPS to access + If you enable this option then the mirror will use HTTPS to access the linux-stable-rc repository on kernel.googlesource.com The full URL is: @@ -676,5 +676,69 @@ config MIRROR_MMTESTS_URL string default DEFAULT_MMTESTS_GITHUB_HTTPS_URL if MIRROR_MMTESTS_HTTPS_GITHUB +config INSTALL_NIX_CACHE_MIRROR + bool "Install Nix binary cache mirror" + output yaml + depends on INSTALL_LOCAL_LINUX_MIRROR + help + Enable this to set up a local Nix binary cache mirror for all guests. + This will significantly speed up NixOS VM builds by caching downloaded + packages locally. + + When enabled, this creates: + - A local Nix binary cache at /mirror/nix-cache/ + - An nginx-based cache server on port 8080 + - Automatic synchronization from cache.nixos.org + + This saves bandwidth and speeds up NixOS builds dramatically as the + initial build requires downloading ~679 packages (685MB). + + The cache will be available to all VMs and systems in your network + at: http://mirror-host:8080/ + +choice + prompt "Nix cache mirror source" + default MIRROR_NIX_CACHE_NIXOS_ORG + depends on INSTALL_NIX_CACHE_MIRROR + +config MIRROR_NIX_CACHE_NIXOS_ORG + bool "cache.nixos.org (official)" + help + Use the official Nix binary cache at cache.nixos.org as the upstream + source for the mirror. + +config MIRROR_NIX_CACHE_NIXOS_ORG_FALLBACK + bool "cache.nixos.org with fastly CDN fallback" + help + Use cache.nixos.org with additional fastly CDN endpoints as fallbacks + for better reliability and performance. + +endchoice + +config MIRROR_NIX_CACHE_UPSTREAM_URL + string + output yaml + default "https://cache.nixos.org" if MIRROR_NIX_CACHE_NIXOS_ORG + default "https://cache.nixos.org https://nixos.cachix.org" if MIRROR_NIX_CACHE_NIXOS_ORG_FALLBACK + +config NIX_CACHE_MIRROR_PORT + int "Nix cache mirror nginx port" + output yaml + default 8080 + depends on INSTALL_NIX_CACHE_MIRROR + help + The port for the local nginx server that will serve the Nix binary cache. + Default is 8080. Ensure this port is available and not blocked by firewall. + +config NIX_CACHE_MIRROR_PATH + string "Nix cache mirror storage path" + output yaml + default "/mirror/nix-cache" + depends on INSTALL_NIX_CACHE_MIRROR + help + Local filesystem path where the Nix binary cache will be stored. + This should be on a filesystem with sufficient space as the cache + can grow to several GB over time. + endif # ENABLE_LOCAL_LINUX_MIRROR endif # TERRAFORM diff --git a/kconfigs/Kconfig.nixos b/kconfigs/Kconfig.nixos index 55361215..3def5372 100644 --- a/kconfigs/Kconfig.nixos +++ b/kconfigs/Kconfig.nixos @@ -91,6 +91,35 @@ config NIXOS_SSH_PORT help SSH port to use for connecting to NixOS VMs. +config NIXOS_USE_LOCAL_MIRROR + bool "Use local Nix binary cache mirror" + output yaml + default $(shell, scripts/check_nix_mirror.sh USE_NIX_CACHE_MIRROR) + help + Enable this to use a local Nix binary cache mirror if available. + This dramatically speeds up NixOS builds by avoiding repeated + downloads of packages from cache.nixos.org. + + When enabled, kdevops will automatically detect and configure + VMs to use the local mirror if available at /mirror/nix-cache + or via HTTP on port 8080. + + This requires that you have set up a mirror server using the + defconfig-mirror configuration. + +config NIXOS_MIRROR_URL + string "Nix binary cache mirror URL" + output yaml + default $(shell, scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL) + depends on NIXOS_USE_LOCAL_MIRROR + help + The URL of the local Nix binary cache mirror. This is + automatically detected based on available mirrors: + - HTTP mirror if running on port 8080 + - File-based mirror at /mirror/nix-cache (for NFS mounts) + + Leave empty to auto-detect at runtime. + config NIXOS_DEBUG_MODE bool "Enable debug mode for NixOS provisioning" default n diff --git a/playbooks/linux-mirror.yml b/playbooks/linux-mirror.yml index 2af0f89d..2ec39d31 100644 --- a/playbooks/linux-mirror.yml +++ b/playbooks/linux-mirror.yml @@ -3,3 +3,5 @@ hosts: localhost roles: - role: linux-mirror + - role: nix-cache-mirror + when: install_nix_cache_mirror | default(false) | bool diff --git a/playbooks/nixos.yml b/playbooks/nixos.yml index eda34586..bdc9b1e8 100644 --- a/playbooks/nixos.yml +++ b/playbooks/nixos.yml @@ -95,6 +95,21 @@ ansible.builtin.set_fact: nixos_ssh_authorized_key: "{{ ssh_public_key['content'] | b64decode | trim }}" + - name: Detect local Nix cache mirror URL if enabled + ansible.builtin.shell: | + bash {{ playbook_dir }}/../scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL + register: detected_mirror_url + when: nixos_use_local_mirror | default(false) | bool and (nixos_mirror_url is not defined or nixos_mirror_url == "") + changed_when: false + + - name: Set detected mirror URL + ansible.builtin.set_fact: + nixos_mirror_url: "{{ detected_mirror_url.stdout | trim }}" + when: + - detected_mirror_url is defined + - detected_mirror_url.stdout is defined + - detected_mirror_url.stdout | trim != "" + - name: Template base NixOS configuration ansible.builtin.template: src: nixos/configuration.nix.j2 @@ -176,9 +191,18 @@ fi # Configure Nix to use local mirror if available - {% if nixos_use_local_mirror is defined and nixos_use_local_mirror and nixos_mirror_url is defined and nixos_mirror_url != "" %} + {% if nixos_use_local_mirror is defined and nixos_use_local_mirror %} + # Dynamically detect mirror URL if not provided + {% if nixos_mirror_url is not defined or nixos_mirror_url == "" %} + DETECTED_MIRROR=$(bash {{ playbook_dir }}/../scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL) + if [ -n "$DETECTED_MIRROR" ]; then + export NIX_CONFIG="substituters = $DETECTED_MIRROR https://cache.nixos.org" + echo "Using detected local Nix cache mirror: $DETECTED_MIRROR" + fi + {% else %} export NIX_CONFIG="substituters = {{ nixos_mirror_url }} https://cache.nixos.org" - echo "Using local Nix cache mirror: {{ nixos_mirror_url }}" + echo "Using configured local Nix cache mirror: {{ nixos_mirror_url }}" + {% endif %} {% endif %} cd {{ nixos_generation_dir }} diff --git a/playbooks/roles/linux-mirror/python/start-mirroring.py b/playbooks/roles/linux-mirror/python/start-mirroring.py index 03ede449..c7c9d81e 100755 --- a/playbooks/roles/linux-mirror/python/start-mirroring.py +++ b/playbooks/roles/linux-mirror/python/start-mirroring.py @@ -54,7 +54,8 @@ def mirror_entry(mirror, args): cmd = cmd + reference_args mirror_target = mirror_path + target if os.path.isdir(mirror_target): - return + sys.stdout.write("Skipping %s - mirror already exists at %s\n" % (short_name, mirror_target)) + return "skipped" sys.stdout.write("Mirroring: %s onto %s\n" % (short_name, mirror_target)) if args.verbose: sys.stdout.write("%s\n" % (cmd)) @@ -74,6 +75,7 @@ def mirror_entry(mirror, args): process.wait() if process.returncode != 0: raise Exception(f"Failed clone with:\n%s" % (" ".join(cmd))) + return "cloned" def main(): @@ -134,15 +136,36 @@ def main(): % (mirror.get("short_name"), args.yaml_mirror, total) ) + # Statistics tracking + stats = {"skipped": 0, "cloned": 0, "failed": 0} # Mirror trees without a reference first for mirror in yaml_vars["mirrors"]: if not mirror.get("reference"): - mirror_entry(mirror, args) + result = mirror_entry(mirror, args) + if result == "skipped": + stats["skipped"] += 1 + elif result == "cloned": + stats["cloned"] += 1 + else: + stats["failed"] += 1 # Mirror trees which need a reference last for mirror in yaml_vars["mirrors"]: if mirror.get("reference"): - mirror_entry(mirror, args) + result = mirror_entry(mirror, args) + if result == "skipped": + stats["skipped"] += 1 + elif result == "cloned": + stats["cloned"] += 1 + else: + stats["failed"] += 1 + # Print summary + sys.stdout.write("\n=== Mirror Summary ===\n") + sys.stdout.write("Total repositories: %d\n" % total) + sys.stdout.write("Skipped (already exist): %d\n" % stats["skipped"]) + sys.stdout.write("Cloned (new): %d\n" % stats["cloned"]) + if stats["failed"] > 0: + sys.stdout.write("Failed: %d\n" % stats["failed"]) if __name__ == "__main__": diff --git a/playbooks/roles/nix-cache-mirror/defaults/main.yml b/playbooks/roles/nix-cache-mirror/defaults/main.yml new file mode 100644 index 00000000..02349b78 --- /dev/null +++ b/playbooks/roles/nix-cache-mirror/defaults/main.yml @@ -0,0 +1,7 @@ +--- +# Default variables for nix-cache-mirror role +nix_cache_mirror_path: "/mirror/nix-cache" +nix_cache_mirror_port: 8080 +nix_cache_upstream_url: "https://cache.nixos.org" +nix_cache_mirror_nginx_conf_path: "/etc/nginx/sites-available/nix-cache-mirror" +nix_cache_mirror_nginx_enabled_path: "/etc/nginx/sites-enabled/nix-cache-mirror" diff --git a/playbooks/roles/nix-cache-mirror/handlers/main.yml b/playbooks/roles/nix-cache-mirror/handlers/main.yml new file mode 100644 index 00000000..d51ca036 --- /dev/null +++ b/playbooks/roles/nix-cache-mirror/handlers/main.yml @@ -0,0 +1,11 @@ +--- +- name: reload nginx + become: true + ansible.builtin.systemd: + name: nginx + state: reloaded + +- name: reload systemd + become: true + ansible.builtin.systemd: + daemon_reload: true diff --git a/playbooks/roles/nix-cache-mirror/tasks/main.yml b/playbooks/roles/nix-cache-mirror/tasks/main.yml new file mode 100644 index 00000000..8c2ef92d --- /dev/null +++ b/playbooks/roles/nix-cache-mirror/tasks/main.yml @@ -0,0 +1,161 @@ +--- +- name: Import optional extra_args file + ansible.builtin.include_vars: "{{ item }}" + ignore_errors: true + with_first_found: + - files: + - "../extra_vars.yml" + - "../extra_vars.yaml" + - "../extra_vars.json" + skip: true + tags: vars + +- name: Fail if nix cache mirror is enabled but user is not root + ansible.builtin.fail: + msg: "Nix cache mirror setup requires root privileges. Please run as root." + when: + - install_nix_cache_mirror | bool + - ansible_user_id != 'root' + tags: ["nix-cache", "mirror"] + +- name: Install nginx for Nix cache mirror + become: true + ansible.builtin.package: + name: + - nginx + - curl + state: present + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Create Nix cache mirror directories + become: true + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + owner: www-data + group: www-data + loop: + - "{{ nix_cache_mirror_path }}" + - "{{ nix_cache_mirror_path }}/nginx-cache" + - "{{ nix_cache_mirror_path }}/tmp" + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Template nginx cache configuration for Nix cache mirror + become: true + ansible.builtin.template: + src: nix-cache-mirror-cache.conf.j2 + dest: /etc/nginx/conf.d/nix-cache-mirror-cache.conf + mode: "0644" + owner: root + group: root + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Template nginx configuration for Nix cache mirror + become: true + ansible.builtin.template: + src: nix-cache-mirror.nginx.j2 + dest: "{{ nix_cache_mirror_nginx_conf_path }}" + mode: "0644" + owner: root + group: root + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Enable nginx site for Nix cache mirror + become: true + ansible.builtin.file: + src: "{{ nix_cache_mirror_nginx_conf_path }}" + dest: "{{ nix_cache_mirror_nginx_enabled_path }}" + state: link + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Remove default nginx site + become: true + ansible.builtin.file: + path: /etc/nginx/sites-enabled/default + state: absent + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Test nginx configuration + become: true + ansible.builtin.command: nginx -t + register: nginx_test + when: install_nix_cache_mirror | bool + changed_when: false + tags: ["nix-cache", "mirror"] + +- name: Start nginx service + become: true + ansible.builtin.systemd: + name: nginx + enabled: true + state: started + daemon_reload: true + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Create systemd service for Nix cache syncer + become: true + ansible.builtin.template: + src: nix-cache-sync.service.j2 + dest: /etc/systemd/system/nix-cache-sync.service + mode: "0644" + when: install_nix_cache_mirror | bool + notify: reload systemd + tags: ["nix-cache", "mirror"] + +- name: Create systemd timer for Nix cache syncer + become: true + ansible.builtin.template: + src: nix-cache-sync.timer.j2 + dest: /etc/systemd/system/nix-cache-sync.timer + mode: "0644" + when: install_nix_cache_mirror | bool + notify: reload systemd + tags: ["nix-cache", "mirror"] + +- name: Reload nginx to pick up new configuration + become: true + ansible.builtin.systemd: + name: nginx + state: reloaded + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Enable and start nix-cache-sync timer + become: true + ansible.builtin.systemd: + name: nix-cache-sync.timer + enabled: true + state: started + daemon_reload: true + when: install_nix_cache_mirror | bool + tags: ["nix-cache", "mirror"] + +- name: Check if firewalld is running + ansible.builtin.command: systemctl is-active firewalld + register: firewalld_status + ignore_errors: true + when: + - install_nix_cache_mirror | bool + - linux_mirror_nfs | bool + tags: ["nix-cache", "mirror"] + +- name: Open firewall for Nix cache mirror HTTP traffic + become: true + ansible.posix.firewalld: + port: "{{ nix_cache_mirror_port }}/tcp" + permanent: true + state: enabled + immediate: true + when: + - install_nix_cache_mirror | bool + - linux_mirror_nfs | bool + - firewalld_status.rc == 0 + tags: ["nix-cache", "mirror"] diff --git a/playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2 b/playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2 new file mode 100644 index 00000000..77b0cda8 --- /dev/null +++ b/playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2 @@ -0,0 +1,3 @@ +# Nix cache mirror proxy cache configuration +# This file must be included at http context level +proxy_cache_path {{ nix_cache_mirror_path }}/nginx-cache levels=1:2 keys_zone=nixcache:100m max_size=50g inactive=7d use_temp_path=off; diff --git a/playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2 b/playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2 new file mode 100644 index 00000000..82c9a297 --- /dev/null +++ b/playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2 @@ -0,0 +1,81 @@ +server { + listen {{ nix_cache_mirror_port }}; + server_name _; + + # Cache location + root {{ nix_cache_mirror_path }}; + + # Enable directory listings + autoindex on; + autoindex_exact_size off; + autoindex_localtime on; + + # Security headers + add_header X-Frame-Options "SAMEORIGIN" always; + add_header X-Content-Type-Options "nosniff" always; + add_header X-XSS-Protection "1; mode=block" always; + + # Logging + access_log /var/log/nginx/nix-cache-mirror.access.log; + error_log /var/log/nginx/nix-cache-mirror.error.log; + + # Main location for cached files + location / { + try_files $uri $uri/ @upstream; + + # Cache headers for clients + expires 1d; + add_header Cache-Control "public, immutable"; + + # Enable range requests for large files + location ~* \.(nar|narinfo)$ { + add_header Accept-Ranges bytes; + } + } + + # Upstream fallback for cache misses + location @upstream { + # Proxy to upstream cache with caching + proxy_pass {{ mirror_nix_cache_upstream_url | default('https://cache.nixos.org') }}; + proxy_set_header Host cache.nixos.org; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Cache configuration (proxy_cache_path is defined in conf.d/) + proxy_cache nixcache; + proxy_cache_valid 200 206 301 302 7d; + proxy_cache_valid 404 1m; + proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504; + proxy_cache_lock on; + proxy_cache_lock_timeout 5s; + proxy_cache_revalidate on; + proxy_cache_min_uses 1; + proxy_cache_background_update on; + + # Buffer settings for better performance + proxy_buffering on; + proxy_buffer_size 4k; + proxy_buffers 8 4k; + proxy_busy_buffers_size 8k; + proxy_temp_path {{ nix_cache_mirror_path }}/tmp; + + # Add headers to indicate cache status + add_header X-Cache-Status $upstream_cache_status; + add_header X-Cache-Date $upstream_http_date; + } + + # Status endpoint + location /status { + access_log off; + return 200 "Nix Cache Mirror OK\n"; + add_header Content-Type text/plain; + } + + # Health check + location /health { + access_log off; + return 200 "healthy\n"; + add_header Content-Type text/plain; + } +} diff --git a/playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2 b/playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2 new file mode 100644 index 00000000..62aee320 --- /dev/null +++ b/playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2 @@ -0,0 +1,17 @@ +[Unit] +Description=Nix Binary Cache Sync Service +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=www-data +Group=www-data +ExecStart=/usr/bin/curl -f -s "{{ mirror_nix_cache_upstream_url | default('https://cache.nixos.org') }}/nix-cache-info" -o "{{ nix_cache_mirror_path }}/nix-cache-info" +ExecStartPost=/bin/bash -c 'find {{ nix_cache_mirror_path }} -name "*.narinfo" -mtime +7 -delete' +ExecStartPost=/bin/bash -c 'find {{ nix_cache_mirror_path }} -name "*.nar*" -mtime +7 -delete' +StandardOutput=journal +StandardError=journal + +[Install] +WantedBy=multi-user.target diff --git a/playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2 b/playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2 new file mode 100644 index 00000000..ef80ad36 --- /dev/null +++ b/playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2 @@ -0,0 +1,11 @@ +[Unit] +Description=Run Nix Binary Cache Sync Service hourly +Requires=nix-cache-sync.service + +[Timer] +OnCalendar=hourly +Persistent=true +RandomizedDelaySec=300 + +[Install] +WantedBy=timers.target diff --git a/scripts/check_nix_mirror.sh b/scripts/check_nix_mirror.sh new file mode 100755 index 00000000..988469c2 --- /dev/null +++ b/scripts/check_nix_mirror.sh @@ -0,0 +1,66 @@ +#!/bin/bash +# SPDX-License-Identifier: copyleft-next-0.3.1 + +# Check for Nix cache mirror availability +NIX_CACHE_PATH="/mirror/nix-cache" +NIX_CACHE_PORT="8080" + +# Function to check if mirror is available via HTTP +check_http_mirror() { + local host=${1:-localhost} + local port=${2:-$NIX_CACHE_PORT} + + # Try to reach the mirror HTTP endpoint + if curl -s --connect-timeout 2 "http://${host}:${port}/status" >/dev/null 2>&1; then + return 0 + fi + return 1 +} + +# Function to check if mirror directory exists and has content +check_local_mirror() { + if [[ -d "$NIX_CACHE_PATH" ]]; then + # Check if directory has some cache files + if find "$NIX_CACHE_PATH" -name "*.narinfo" -o -name "*.nar*" | head -1 | grep -q .; then + return 0 + fi + # Even if empty, the directory exists so mirror is configured + return 0 + fi + return 1 +} + +case "$1" in + "USE_NIX_CACHE_MIRROR") + # Check if we should use the mirror + if check_local_mirror || check_http_mirror; then + echo y + else + echo n + fi + ;; + "NIX_CACHE_MIRROR_AVAILABLE") + # Check if mirror is available (for client-side detection) + if check_http_mirror; then + echo y + else + echo n + fi + ;; + "NIX_CACHE_MIRROR_URL") + # Return the mirror URL if available + if check_http_mirror localhost "$NIX_CACHE_PORT"; then + echo "http://localhost:$NIX_CACHE_PORT" + elif check_http_mirror "$(hostname -I | awk '{print $1}')" "$NIX_CACHE_PORT"; then + echo "http://$(hostname -I | awk '{print $1}'):$NIX_CACHE_PORT" + elif check_local_mirror; then + # Use file:// URL for local directory-based cache + echo "file://$NIX_CACHE_PATH" + else + echo "" + fi + ;; + *) + echo n + ;; +esac -- 2.50.1