public inbox for kdevops@lists.linux.dev
 help / color / mirror / Atom feed
From: Daniel Gomez <da.gomez@kernel.org>
To: Luis Chamberlain <mcgrof@kernel.org>,
	 Chuck Lever <chuck.lever@oracle.com>
Cc: kdevops@lists.linux.dev, Daniel Gomez <da.gomez@samsung.com>
Subject: [PATCH v2 2/4] workflows: bootlinux: add comprehensive ccache support
Date: Fri, 19 Sep 2025 14:25:08 +0200	[thread overview]
Message-ID: <20250919-kernel-fragment-support-v2-2-8d2b7b8cb4e4@samsung.com> (raw)
In-Reply-To: <20250919-kernel-fragment-support-v2-0-8d2b7b8cb4e4@samsung.com>

From: Daniel Gomez <da.gomez@samsung.com>

Add ccache (compiler cache) support to accelerate kernel builds with
both GCC and Clang toolchains. Implementation provides flexible
configuration options while maintaining simplicity for common use cases.

Key features:
- System-wide ccache configuration (default) for immediate use
- kdevops-managed configuration for project isolation
- Automatic package installation across all supported distros
- Modular environment variable composition for clean integration
- LLVM documentation compliance (CC="ccache clang")
- Seamless integration with reproducible builds and existing features

Configuration:
- Cache size specified in GiB (digits only, e.g., 10 = 10 GiB)
- Uses automatic Kconfig→Ansible integration via .extra_vars_auto.yaml
- Template-based configuration management with TOPDIR_PATH isolation
- Organized in dedicated ccache submenu for clean UI

Technical implementation:
- Uses modular bootlinux_make_params for community.general.make module
- Uses modular bootlinux_build_environment for shell tasks
- Automatic compiler detection (ccache gcc vs ccache clang)
- Enhanced debug output for build command validation

Generated-by: Claude AI
Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 playbooks/roles/bootlinux/defaults/main.yml        |  6 ++
 playbooks/roles/bootlinux/tasks/build/9p.yml       |  5 +-
 playbooks/roles/bootlinux/tasks/ccache.yml         | 37 ++++++++++
 .../bootlinux/tasks/install-deps/debian/main.yml   |  1 +
 .../bootlinux/tasks/install-deps/redhat/main.yml   |  1 +
 .../bootlinux/tasks/install-deps/suse/main.yml     |  1 +
 .../tasks/install-minimal-deps/debian/main.yml     |  1 +
 .../tasks/install-minimal-deps/redhat/main.yml     |  2 +
 .../tasks/install-minimal-deps/suse/main.yml       |  1 +
 playbooks/roles/bootlinux/tasks/main.yml           |  9 +++
 playbooks/roles/bootlinux/templates/ccache.conf.j2 | 27 ++++++++
 workflows/linux/Kconfig                            | 78 ++++++++++++++++++++++
 12 files changed, 168 insertions(+), 1 deletion(-)

diff --git a/playbooks/roles/bootlinux/defaults/main.yml b/playbooks/roles/bootlinux/defaults/main.yml
index 2d2eb798..97adec30 100644
--- a/playbooks/roles/bootlinux/defaults/main.yml
+++ b/playbooks/roles/bootlinux/defaults/main.yml
@@ -37,6 +37,8 @@ target_linux_make_cmd: "{{ make }} -j{{ ansible_processor_vcpus }}"
 bootlinux_make_params: >-
   {{
     ({} | combine(
+      {'CC': 'ccache clang' if bootlinux_compiler_clang|default(false)|bool else 'ccache gcc'} if bootlinux_ccache|default(false)|bool else {}
+    ) | combine(
       {
         'KBUILD_BUILD_TIMESTAMP': '',
         'KBUILD_BUILD_USER': 'kdevops',
@@ -49,6 +51,10 @@ bootlinux_make_params: >-
 bootlinux_build_environment: >-
   {{
     ({} | combine(
+      {'CCACHE_CONFIGPATH': topdir_path + '/.ccache/ccache.conf'} if (bootlinux_ccache|default(false)|bool and bootlinux_ccache_kdevops_managed|default(false)|bool) else {}
+    ) | combine(
+      {'CC': 'ccache clang' if bootlinux_compiler_clang|default(false)|bool else 'ccache gcc'} if bootlinux_ccache|default(false)|bool else {}
+    ) | combine(
       {
         'KBUILD_BUILD_TIMESTAMP': '',
         'KBUILD_BUILD_USER': 'kdevops',
diff --git a/playbooks/roles/bootlinux/tasks/build/9p.yml b/playbooks/roles/bootlinux/tasks/build/9p.yml
index 17b19cb3..6f03ed9a 100644
--- a/playbooks/roles/bootlinux/tasks/build/9p.yml
+++ b/playbooks/roles/bootlinux/tasks/build/9p.yml
@@ -164,8 +164,10 @@
       build_jobs: {{ nproc_9p.stdout }}
       bootlinux_make_params: {{ bootlinux_make_params }}
       bootlinux_build_environment: {{ bootlinux_build_environment }}
+      bootlinux_ccache: {{ bootlinux_ccache|default(false) }}
       bootlinux_compiler_clang: {{ bootlinux_compiler_clang|default(false) }}
       bootlinux_reproducible_builds: {{ bootlinux_reproducible_builds|default(false) }}
+      bootlinux_ccache_kdevops_managed: {{ bootlinux_ccache_kdevops_managed|default(false) }}
   run_once: true
   delegate_to: localhost
   vars:
@@ -178,7 +180,8 @@
     echo "CC=${CC:-'not set'}"
     echo "PATH=${PATH}"
     which gcc || echo "gcc not found"
-    env | grep -E '^(CC|PATH)' | sort
+    which ccache || echo "ccache not found"
+    env | grep -E '^(CC|CCACHE|PATH)' | sort
   environment: "{{ bootlinux_build_environment }}"
   register: cc_test_result
   run_once: true
diff --git a/playbooks/roles/bootlinux/tasks/ccache.yml b/playbooks/roles/bootlinux/tasks/ccache.yml
new file mode 100644
index 00000000..38b2c034
--- /dev/null
+++ b/playbooks/roles/bootlinux/tasks/ccache.yml
@@ -0,0 +1,37 @@
+---
+# ccache configuration and setup
+
+- name: Create kdevops ccache directory structure
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    mode: '0755'
+  loop:
+    - "{{ topdir_path }}/.ccache"
+    - "{{ bootlinux_ccache_dir }}"
+  when:
+    - bootlinux_ccache|default(false)|bool
+    - bootlinux_ccache_kdevops_managed|default(false)|bool
+
+- name: Generate kdevops-managed ccache configuration
+  ansible.builtin.template:
+    src: ccache.conf.j2
+    dest: "{{ topdir_path }}/.ccache/ccache.conf"
+    mode: '0644'
+  when:
+    - bootlinux_ccache|default(false)|bool
+    - bootlinux_ccache_kdevops_managed|default(false)|bool
+
+- name: Display ccache configuration info
+  ansible.builtin.debug:
+    msg: |
+      ccache enabled: {{ bootlinux_ccache|default(false)|bool }}
+      ccache mode: {{ 'kdevops-managed' if bootlinux_ccache_kdevops_managed|default(false)|bool else 'system-wide' }}
+      {% if bootlinux_ccache_kdevops_managed|default(false)|bool %}
+      Config file: {{ topdir_path }}/.ccache/ccache.conf
+      Cache directory: {{ bootlinux_ccache_dir }}
+      Max size: {{ bootlinux_ccache_max_size }} GiB
+      {% endif %}
+  when: bootlinux_ccache|default(false)|bool
+  vars:
+    ansible_callback_diy_runner_on_ok_msg: "{{ ansible_callback_diy.result.output.msg }}"
diff --git a/playbooks/roles/bootlinux/tasks/install-deps/debian/main.yml b/playbooks/roles/bootlinux/tasks/install-deps/debian/main.yml
index 4edf8e1d..e5911029 100644
--- a/playbooks/roles/bootlinux/tasks/install-deps/debian/main.yml
+++ b/playbooks/roles/bootlinux/tasks/install-deps/debian/main.yml
@@ -39,4 +39,5 @@
       - zstd
       - libncurses-dev
       - b4
+      - ccache
     state: present
diff --git a/playbooks/roles/bootlinux/tasks/install-deps/redhat/main.yml b/playbooks/roles/bootlinux/tasks/install-deps/redhat/main.yml
index c7b18cd5..7f1955bf 100644
--- a/playbooks/roles/bootlinux/tasks/install-deps/redhat/main.yml
+++ b/playbooks/roles/bootlinux/tasks/install-deps/redhat/main.yml
@@ -52,6 +52,7 @@
       - dwarves
       - userspace-rcu
       - zstd
+      - ccache
 
 - name: Install btrfs-progs
   become: true
diff --git a/playbooks/roles/bootlinux/tasks/install-deps/suse/main.yml b/playbooks/roles/bootlinux/tasks/install-deps/suse/main.yml
index 7f0cf249..5c9f0e3b 100644
--- a/playbooks/roles/bootlinux/tasks/install-deps/suse/main.yml
+++ b/playbooks/roles/bootlinux/tasks/install-deps/suse/main.yml
@@ -28,4 +28,5 @@
       - portmap
       - hwinfo
       - open-iscsi
+      - ccache
     disable_recommends: false
diff --git a/playbooks/roles/bootlinux/tasks/install-minimal-deps/debian/main.yml b/playbooks/roles/bootlinux/tasks/install-minimal-deps/debian/main.yml
index fab4998a..cc05a44a 100644
--- a/playbooks/roles/bootlinux/tasks/install-minimal-deps/debian/main.yml
+++ b/playbooks/roles/bootlinux/tasks/install-minimal-deps/debian/main.yml
@@ -24,4 +24,5 @@
       - make
       - gcc
       - kmod
+      - ccache
     state: present
diff --git a/playbooks/roles/bootlinux/tasks/install-minimal-deps/redhat/main.yml b/playbooks/roles/bootlinux/tasks/install-minimal-deps/redhat/main.yml
index 50651ab1..950dbfe1 100644
--- a/playbooks/roles/bootlinux/tasks/install-minimal-deps/redhat/main.yml
+++ b/playbooks/roles/bootlinux/tasks/install-minimal-deps/redhat/main.yml
@@ -10,6 +10,7 @@
       - make
       - gcc
       - kmod
+      - ccache
     state: present
   when:
     - ansible_facts['distribution_major_version']|int < 8
@@ -22,6 +23,7 @@
       - make
       - gcc
       - kmod
+      - ccache
     state: present
   when:
     - ansible_facts['distribution_major_version']|int >= 8
diff --git a/playbooks/roles/bootlinux/tasks/install-minimal-deps/suse/main.yml b/playbooks/roles/bootlinux/tasks/install-minimal-deps/suse/main.yml
index a4cc27bb..d8f75240 100644
--- a/playbooks/roles/bootlinux/tasks/install-minimal-deps/suse/main.yml
+++ b/playbooks/roles/bootlinux/tasks/install-minimal-deps/suse/main.yml
@@ -10,4 +10,5 @@
       - make
       - gcc
       - kmod-compat
+      - ccache
     state: present
diff --git a/playbooks/roles/bootlinux/tasks/main.yml b/playbooks/roles/bootlinux/tasks/main.yml
index 4110d596..eff6fb6b 100644
--- a/playbooks/roles/bootlinux/tasks/main.yml
+++ b/playbooks/roles/bootlinux/tasks/main.yml
@@ -32,6 +32,15 @@
     - bootlinux_9p|bool
     - not workflow_linux_packaged|bool
 
+# Setup ccache configuration
+- name: Setup ccache configuration
+  ansible.builtin.import_tasks:
+    file: ccache.yml
+  when:
+    - bootlinux_ccache|default(false)|bool
+    - not workflow_linux_packaged|bool
+  tags: ['ccache']
+
 # We do this regardless of what distro you use
 - name: Install b4
   become: true
diff --git a/playbooks/roles/bootlinux/templates/ccache.conf.j2 b/playbooks/roles/bootlinux/templates/ccache.conf.j2
new file mode 100644
index 00000000..8c0a7bf9
--- /dev/null
+++ b/playbooks/roles/bootlinux/templates/ccache.conf.j2
@@ -0,0 +1,27 @@
+# kdevops-managed ccache configuration
+# Generated automatically - do not edit manually
+
+# Cache settings
+cache_dir = {{ bootlinux_ccache_dir }}
+max_size = {{ bootlinux_ccache_max_size }}.0 GiB
+
+# Performance settings
+compression = {{ bootlinux_ccache_compression | lower }}
+stats = true
+
+# Build reproducibility - ensure deterministic compilation
+compiler_check = mtime
+direct_mode = true
+hash_dir = true
+
+# Debugging (disabled by default for performance)
+debug = false
+
+# File handling
+file_clone = false
+hard_link = false
+inode_cache = true
+
+# Security
+umask =
+read_only = false
diff --git a/workflows/linux/Kconfig b/workflows/linux/Kconfig
index 897cb00f..cbfa8a3c 100644
--- a/workflows/linux/Kconfig
+++ b/workflows/linux/Kconfig
@@ -188,6 +188,84 @@ config BOOTLINUX_REPRODUCIBLE_BUILDS
 	  Recommended for CI/CD environments and when build reproducibility
 	  is required. Compatible with both GCC and Clang toolchains.
 
+menu "ccache configuration"
+
+config BOOTLINUX_CCACHE
+	bool "Enable ccache for faster kernel builds"
+	output yaml
+	default n
+	help
+	  Enable ccache (compiler cache) to accelerate repeated kernel
+	  builds by caching compilation results. Particularly effective
+	  for incremental builds and CI/CD environments.
+
+	  When enabled with Clang (LLVM=1), uses CC="ccache clang".
+	  When enabled with GCC, uses CC="ccache gcc".
+
+	  Requires ccache package to be installed on build nodes.
+
+choice
+	prompt "ccache configuration mode"
+	depends on BOOTLINUX_CCACHE
+	default BOOTLINUX_CCACHE_SYSTEM_WIDE
+	help
+	  Choose how ccache configuration is managed.
+
+config BOOTLINUX_CCACHE_SYSTEM_WIDE
+	bool "Use system-wide ccache configuration"
+	output yaml
+	help
+	  Use the system's existing ccache configuration
+	  (~/.cache/ccache or system defaults). Allows manual
+	  tuning via "ccache --set-config" commands.
+
+config BOOTLINUX_CCACHE_KDEVOPS_MANAGED
+	bool "kdevops-managed ccache configuration"
+	output yaml
+	help
+	  kdevops generates and manages a project-specific ccache
+	  configuration file using TOPDIR_PATH. This ensures
+	  isolation from system-wide ccache settings and allows
+	  project-specific tuning.
+
+	  Configuration location: $(TOPDIR_PATH)/.ccache/ccache.conf
+
+endchoice
+
+if BOOTLINUX_CCACHE_KDEVOPS_MANAGED
+
+config BOOTLINUX_CCACHE_MAX_SIZE
+	int "Maximum cache size (GiB)"
+	output yaml
+	default 10
+	help
+	  Maximum size for the ccache in GiB. Common values:
+	  - 5 (default ccache setting)
+	  - 10 (recommended for kernel development)
+	  - 20 (large projects/multiple kernels)
+	  - 50 (high-performance development environments)
+
+config BOOTLINUX_CCACHE_DIR
+	string "Cache directory"
+	output yaml
+	default "$(TOPDIR_PATH)/.ccache/cache"
+	help
+	  Directory where ccache stores cached compilation results.
+	  Using TOPDIR_PATH ensures project isolation.
+
+config BOOTLINUX_CCACHE_COMPRESSION
+	bool "Enable compression"
+	output yaml
+	default y
+	help
+	  Enable compression of cached files to save disk space.
+	  Slightly increases CPU usage but significantly reduces
+	  storage requirements.
+
+endif # BOOTLINUX_CCACHE_KDEVOPS_MANAGED
+
+endmenu
+
 choice
 	prompt "Type of development version of Linux to use"
 	default BOOTLINUX_LINUS if !BOOTLINUX_TREE_SET_BY_CLI && !BOOTLINUX_TREE_REF_SET_BY_CLI

-- 
2.50.1


  parent reply	other threads:[~2025-09-19 12:25 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-09-19 12:25 [PATCH v2 0/4] workflows: bootlinux: enhance kernel configuration fragment support Daniel Gomez
2025-09-19 12:25 ` [PATCH v2 1/4] workflows: bootlinux: add reproducible builds support Daniel Gomez
2025-09-19 12:25 ` Daniel Gomez [this message]
2025-09-19 12:25 ` [PATCH v2 3/4] workflows: bootlinux: add kernel configuration fragments support Daniel Gomez
2025-09-19 12:25 ` [PATCH v2 4/4] workflows: bootlinux: add clean builds configuration option Daniel Gomez
2025-09-19 18:10 ` [PATCH v2 0/4] workflows: bootlinux: enhance kernel configuration fragment support Luis Chamberlain

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250919-kernel-fragment-support-v2-2-8d2b7b8cb4e4@samsung.com \
    --to=da.gomez@kernel.org \
    --cc=chuck.lever@oracle.com \
    --cc=da.gomez@samsung.com \
    --cc=kdevops@lists.linux.dev \
    --cc=mcgrof@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox