From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B41E944F for ; Wed, 23 Jul 2025 01:21:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753233718; cv=none; b=nmEe6GCipCUWUWNe4gQB6lr2jpHIQaZBupRbojgW/xDF67zLRI0NbN4IZ0pgUu+/WzfpnAOVmXWfnIAQYeeA8RPI+4/8JYs5XFkxdzeqvGvly9JEK6PRPm8a/mQnwk200feDi32TZyrEWS4Lfdkg+cMVC5zHN6e5WReAmkr2oRs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753233718; c=relaxed/simple; bh=6188AOLns9N3p6F4JrfnS/ptsAnbGH1yPLuz/9V5FkM=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=PlXd3tlC1vTu3Suo0wCHcsxYmD6iF20hV7bP4oNw2Slq46AFGEAKmyriFX6bM0eIJ47g1UK5ECB/h+bRyPLrV9HJD8z0RroU8Q2ZFCrA0k2jpL+w8DPM5s9w727Y5KgAc4TQ4KhtM29yFSYeSQuiL9w+4gfXQ0UrzaJs6mJfLP8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=2jPMO6v3; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="2jPMO6v3" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-Type: Content-ID:Content-Description:In-Reply-To:References; bh=vsTkSPfoI+GabSgqkibUHzCPEhgF1F9ASdS7wWNBq3k=; b=2jPMO6v354CoMM7aFldlEq4egl iAq+yPTEhVnJoR2DTgbPehoKFmw5SuPFERFRPCR+vp/vz6h8oDzw56nAD2QebARUBZ1IM4cqhZydP B5jmd6VXtRHbumklwDcMAsNxK1XfPO8vhlSSQiuwsNIasfxxEClbh3AGAodbIsuIwKFWNnPQx/7/x UX1LgrnteLivOQh/HCidZ5zxC3umUmicJRvU3O1Uty2OCFKDgY03uuJR+BHdRpUvomTam1sMA5RXJ yKxpyL3YKvKwdwWsh4OMpbAgp8A+W9Ge+oBDpEzrsupKt/539t/BLnVCJ7ulY3HMEOE3JWzsoBhBl s3OO6Vhg==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98.2 #2 (Red Hat Linux)) id 1ueOB5-00000003m4d-3BAi; Wed, 23 Jul 2025 01:21:51 +0000 From: Luis Chamberlain To: Chuck Lever , Daniel Gomez , Viacheslav Dubeyko , Mel Gorman , Jan Kara , David Bueso , Qazi@infradead.org, Shoaib Asif , "Liam R . Howlett" , Lorenzo Stoakes , Vlastimil Babka , Johannes Weiner , kdevops@lists.linux.dev Cc: Luis Chamberlain Subject: [PATCH] docs: declare kdevops a software 3.0 enabler Date: Tue, 22 Jul 2025 18:21:50 -0700 Message-ID: <20250723012150.899773-1-mcgrof@kernel.org> X-Mailer: git-send-email 2.49.0 Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain For a long time we've worked hard to make it easier to extend kdevops with strict rules and clear declarative language. We've been so strict on some of these policies I'm quite sure it discouraged some developers. The projections and hope was that these strict goals would reduce the barrier to the use of generative AI for extending kdevops. Although we've been experimenting with different AI agents on kdevops for a while now, advanced AI agent tools such as Claude Code bring this closer to home, enabling prompts to complete about 70-95% of the tasks. In a short span of about one week we've been able to cut through a huge back log of work which would have otherwise taken us considerable amount of time. This, and our automated tests put in place gives us confidence to declare that the time is here and now and it should be easy for others to also extend kdevops for their needs with generative AI. Extend our documentation with the rationale of the implicit long term goals we had, make these explicit, and also provide guidance for Claude Code. We also provide some recent prompt examples and respective commits merged which should be useful for users wishing to look to extend kdevops. CLAUDE.md [0] is a special file that Claude automatically pulls into context when starting a conversation. This makes it an ideal place for documenting: - Common commands folks should be aware of that we run - Core files and utility functions - Code style guidelines - Testing instructions - Repository etiquette (e.g., branch naming, merge vs. rebase, etc.) - Developer environment setup - Any unexpected behaviors or warnings particular to the project - Other information you want Claude to remember We extend this to ensure the DCO is respected and also give it a few prompts examples and target historical commits with respective grades. Also, extend the video documentation links with the recent kdevops talk in Open Source Summit North America [1] where I mentioned for the first time publicly that kdevops was Software 3.0 enabler for Linux kernel development. Link: https://www.anthropic.com/engineering/claude-code-best-practices # [0] Link: https://www.youtube.com/watch?v=VF-jr_ZE-9Y&list=PLjaT52x3HVboZtdwZnONSHQHM8217ELcc # [1] Signed-off-by: Luis Chamberlain --- CLAUDE.md | 218 +++++++++++++++++++++++++++++++++++++++++++++++++++++ PROMPTS.md | 125 ++++++++++++++++++++++++++++++ README.md | 52 +++++++++++++ 3 files changed, 395 insertions(+) create mode 100644 CLAUDE.md create mode 100644 PROMPTS.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..8bee7c08 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,218 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with +code in this repository. + +## Project Overview + +kdevops is a comprehensive DevOps framework for Linux kernel development and +testing. It provides automation for setting up kernel development environments +and complex testing laboratories for kernel subsystems. + +**Version**: 5.0.2 +**Main Repository**: https://github.com/linux-kdevops/kdevops +**License**: copyleft-next-0.3.1 + +## Core Architecture + +### Build System +- **Configuration**: Uses Linux kernel Kconfig system (`make menuconfig`, + `make nconfig`, `make dynconfig`) +- **Build**: Make-based with modular Makefiles (`Makefile.*`) +- **Automation**: Ansible playbooks in `playbooks/` directory +- **Infrastructure**: Supports virtualization (libguestfs/libvirt), + cloud providers (AWS, Azure, GCE, OCI), and bare metal + +### Key Components +- `workflows/` - Testing workflows (fstests, blktests, selftests, CXL, + mmtests, NFS, etc.) +- `playbooks/` - Ansible automation with 40+ specialized roles +- `kconfigs/` - Modular configuration system. Provides modeling variabiilty + support. This is essentially support for kconfig taken from + the Linux kernel, with support for special "output yaml" + support to allow the project to leverage kconfig symbols on + ansible through the `extra_vars.yaml` file which is always + present. +- `defconfigs/` - Pre-built configurations for common setups. These are + extremely useful for supporting not only easy quick setups + but are also instrumental for our continuous integration + support. +- `scripts/` - Workflow automation and helper scripts + +## Common Development Commands + +### Configuration and Setup +```bash +make menuconfig # Interactive configuration +make dynconfig # Supports dynamically generated kconfig files +make defconfig- # Use predefined configuration (see defconfigs/) +make # Build dependencies and setup +make bringup # Provision and configure systems +make destroy # Destroy provisioned systems +``` + +### Kernel Development +```bash +make linux # Build and install kernel +make linux HOSTS="host1 host2" # Target specific hosts +make linux-uninstall KVER="6.6.0-rc2" # Uninstall specific kernel version +``` + +### Testing Workflows +```bash +# Filesystem testing +make fstests # Run filesystem tests +make fstests-baseline # Establish baseline +make fstests-results # View results + +# Block layer testing +make blktests # Run block layer tests +make blktests-baseline # Establish baseline +make blktests-results # View results + +# Linux kernel selftests +make selftests # Run all selftests +make selftests-firmware # Run specific test suite +make selftests-kmod # Run kernel module tests + +# Other workflows +make pynfs # NFS testing +make gitr # Git regression testing +make ltp # Linux Test Project +make sysbench # Database performance testing +make mmtests # Memory management tests from mmtests +``` + +### Development Utilities +```bash +make help # Show available targets +make V=1 [target] # Verbose build output +make AV=1-6 [target] # Ansible verbose output (levels 0-6) +make dynconfig # Generate dynamic configuration +make mrproper # Clean everything and restart from scratch +``` + +## Key Workflows + +### fstests (Filesystem Testing) +- **Purpose**: Comprehensive filesystem testing for XFS, Btrfs, EXT4, CIFS, NFS, tmpfs +- **Features**: Expunge list management, baseline tracking, regression detection +- **Location**: `workflows/fstests/` +- **Config**: Enable fstests workflow in menuconfig + +### blktests (Block Layer Testing) +- **Purpose**: Block layer subsystem testing +- **Supports**: NVMe, SCSI, loop devices, NBD, ZBD +- **Location**: `workflows/blktests/` +- **Features**: Similar baseline/regression tracking as fstests + +### Linux Kernel Building +- **Source Management**: Multiple git trees (Linus, stable, next, subsystem trees) +- **Features**: 9P filesystem for host-guest development, mirror support +- **Location**: `workflows/linux/` + +### selftests (Kernel Selftests) +- **Purpose**: Parallel execution of Linux kernel selftests +- **Supports**: firmware, kmod, sysctl, and other kernel subsystem tests +- **Location**: `workflows/selftests/` + +## Architecture Highlights + +### Configuration System +- Uses Linux kernel Kconfig for consistent configuration management +- Modular configuration files in `kconfigs/` for different subsystems +- Dynamic configuration generation with `make dynconfig` +- Pre-built configurations in `defconfigs/` directory + +### Workflow System +- Each workflow has dedicated Kconfig and Makefile +- Workflow-specific scripts in `scripts/workflows/` +- Ansible roles for automation in `playbooks/roles/` +- Result collection and baseline management + +### Infrastructure Support +- **Virtualization**: libguestfs with libvirt (recommended), legacy Vagrant +- **Cloud**: AWS, Azure, GCE, OCI, OpenStack support via Terraform +- **PCIe Passthrough**: Real hardware testing in VMs with dynamic device assignment +- **Mirror Support**: For air-gapped environments + +### Kernel CI Features +- Built-in continuous integration support +- Baseline management for known test failures +- Regression detection with git-style diff output +- Watchdog systems for automated recovery from hung tests + +## Important File Locations + +- `Kconfig` - Main configuration entry point +- `workflows/*/Kconfig` - Workflow-specific configuration options +- `workflows/*/Makefile` - Workflow automation targets +- `playbooks/roles/` - Reusable Ansible automation components +- `scripts/workflows/` - Workflow-specific helper scripts +- `docs/` - Comprehensive documentation + +## Development Patterns + +1. **Configuration-Driven**: Everything configurable through Kconfig +2. **Modular Design**: Workflow-specific components included conditionally +3. **Ansible Automation**: Role-based infrastructure and testing automation +4. **Baseline Management**: Comprehensive tracking of known failures and regressions +5. **Template Generation**: Dynamic file generation based on configuration + +## Quick Setup Examples + +### XFS Filesystem Testing +```bash +make defconfig-xfs # Configure for XFS testing +make bringup # Setup test environment +make fstests # Run filesystem tests +``` + +### Kernel Development Environment +```bash +make menuconfig # Configure kernel workflow +make bringup # Setup development VMs +make linux # Build and install kernel +``` + +### Block Layer Testing with NVMe +```bash +make defconfig-blktests_nvme +make bringup +make blktests +``` + +## Testing and Quality Assurance + +- Expunge lists track known test failures in `workflows/*/expunges/` +- Baseline commands establish expected test results +- Results commands show test outcomes and regressions +- Watchdog scripts provide automated monitoring for long-running tests +- Integration with kernel development workflows and patchwork systems + +This framework is designed by kernel developers for kernel developers, +providing production-ready automation for kernel testing and development +workflows. + +## One commit per change + +As with the Linux kernel, this project prefers commits to be atomic and to +the point. We don't want spell fixes to be blended in with code changes. +Spell fixes should go into separate commits. When in doubt, just don't do +any spell fixes unless asked explicitly to do that. + +## Use the Signed-off-by tag + +We want to use the Signed-off-by tag which embodies the application of the +Developer Certificate or Origin. + +## Use Generated-by: Claude AI + +Use this tag for code generated by Claude code AI. Put this before the +Signed-off-by tag. + +## Prompt Examples + +Refer to PROMPTS.md for example set of prompts used to generate code on +kdevops using different AI agents and their respective commits and gradings. +This can be useful for humans but also for generative AI so it can improve. diff --git a/PROMPTS.md b/PROMPTS.md new file mode 100644 index 00000000..a4ecf39f --- /dev/null +++ b/PROMPTS.md @@ -0,0 +1,125 @@ +# kdevops PROMPTS.md + +This file can be used by generative AI agents to learn previous prompts +and example commits and their outcomes, and notes by users of the AI agent +grading. It is also instructive for humans to learn how to use generative +AI to easily extend kdevops for their own needs. + +## Extending existing Linux kernel selftests + +Below are a set of example prompts / result commits of extending existing +kdevops selftests support to add new selftests. We bundle together "selftests" +with a few custom tests which don't fit into the "selftetsts" category but +due to efficiency we simply augment its support to cover them. The special +unicorn workloads are: + + - xarray - has kernel module and userspace tests + - maple tree - has kernel module and userspace tests + - vma - only has userspace tests + +### Adding a new custom Linux kernel userspace test under kdevops selftests + +**Prompt:** +We already support testing the Linux kernel maple tree and radix tree on the +selftests workflow. These are custom tests and each of these are tested on +kdevops in-kernel and then in userspace. The in-kernel testing done requires +using a loadable kernel module for each. + +We have support for bundling these two tests under the +defconfigs/seltests-radix-tree defconfig and we use the +SELFTESTS_TEST_BUNDLE_RADIX_TREE to help us bundle all radix-tree related tests. +We want to extend this with a new userspace only test. Its the vma tests on the +Linux kernel. So it won't have a corresponding Linux kernel module load, +contrary to test_xarray and test_maple_tree. The only thing that needs to be +done to test it is to change directory on the linux kernel directory sources +into tools/testing/vma and as a user run make -j$(nproc) and then just run +sudo ./vma. So add support for this and augment SELFTESTS_TEST_BUNDLE_RADIX_TREE +to select this. + +**AI:** Claude Code +**Commit:** [`191ae5678688`](https://github.com/linux-kdevops/kdevops/commit/191ae5678688) +**Result:** Almost perfect. +**Grading:** 95% + +**Notes:** + +The only issue caught was a complex bug which any human would have also run +into. It created a separate task for running the userspace tests and registered +the output to the same variable name as the task which runs the userspace +tests on the maple tree and xarray. This confuses ansible. And so the +selftests task "Run userspace selftests" needed to be manually fixed to +check if the test was either xarray or maple tree or vma, and use that same +task with variables for the directory where we change into and the command +we run. + + +## Adding completely new workflows to kdevops + +Below are a list of successful prompts used with Claude Code to generate +commits for kdevops. They range from fixing bugs, to adding new workflows +to extending existing selftests support. These are intended to be useful +for humans and Claude Code. Please extend them with more examples. You +can add new types of fields. It just needs to make coherent sense. + +### mmtests integration + +**Prompt:** +mmtests by Mel Gorman is used to help force memory fragmentation. I had +extended mmtests so to ensure it works with debian-testing. And so given +kdevops default are to work with debian-testing adding support for mmtests to +kdevops should be straight forward, it should consist of just looking at its +README and for each distribution adding requirements on dependencies just as we +do for the sysbench workflow on kdevops. The kdevops sysbench workflow also +intelligently used intelligently the kconfig KDEVOPS_BASELINE_AND_DEV option so +to enable A/B testing. So extend kdevops to add support for mmtests. Add +mirror support so that we have it as one of the mirrored git trees we take. +Then add a new mmtests workflow with support for it, and only initially focus +on thpchallenge-fio and thpcompact as possible mmtests workflow targets we can +work. Ensure to use the new output yaml feature so to enable ansible tasks for +mmtests to leverage the kconfig logic passed down automatically. + +**AI:** Claude Code +**Commit:** [`0b829e6a1fb8`](https://github.com/linux-kdevops/kdevops/commit/0b829e6a1fb8) +**Result:** Comprehensive tests generated, with minor manual fixups. +**Grading:** 70% + +**Notes:** +Most of the nuggets were properly implemented except it lacked insight to +review the build component and that: + +a) Upon build the script called may expect user input. This required us to + modify the call to use 'yes yes | ./run-mmtests.sh -b' and that meant + also using and leveraging the ansible shell module. The ansible shell + module is not ideal and not recommended due to the fact that pipes can + easily lead to non deteriministic behaviour. We prefer long term to + use the ansible command module and split task as needed. In this case + in the future to enhance determinism we want to instead add an option + upstream mmtest to the ./run-mmtests.sh command line options to let us + not have to enter any input from users and let it just install depdenencies + for us. + +b) The script ./run-mmtests.sh asks us for user input to install dependencies. + The lack of insight was to realize that other than the README it should have + looked at the script ./run-mmtests.sh to help review if it did try to + install dependencies on its own. What we can do later is instead as a + secondary step use Caude Code to later ask it to analyze the latest + upstream mmtests run-mmtests.sh script and ask it to install augment + the dependency list we have on kdevops for each distribution. + +c) The prompt the user gave lacked sufficient hints to help it understand + where to get more sensible configurations from to be used as templates + for ninja2 conversion. So for example ./bin/autogen-configs generates + configurations for us and these should have been used for the base + templates. + +d) I manually had to enhance styling for variables for the different mmtests + test types. This consisted of for example using variable names like + MMTESTS_THPCOMPACT_THREADS_MIN for thpcompact related knobs. Likewise I + preferred to split configurations for the different mmtests test types + into their own kconfig file. So for example we ended up with: + +source "workflows/mmtests/Kconfig.thpcompact" +source "workflows/mmtests/Kconfig.thpchallenge" +source "workflows/mmtests/Kconfig.fs" + + This separation is preferred as it helps us scale. diff --git a/README.md b/README.md index 4d9be800..15bb7119 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ Table of Contents ================= * [kdevops](#kdevops) + * [Generative AI usage on kdevops](#generative-ai-usage-on-kdevops) + * [Resources for leveraging generative AI on kdevops](#resources-for-leveraging-generative-AI-on-kdevops) * [Quick kdevops demos](#quick-kdevops-demos) * [Start kernel hacking in just 4 commands](#start-kernel-hacking-in-just-4-commands) * [Start running fstests in 2 commands](#start-running-fstests-in-2-commands) @@ -63,6 +65,55 @@ files dynamically based on your system environment. This mechanism will be expanded in the future to make kdevops even more dynamic to support even more features. +## Generative AI usage on kdevops + +The kdeovps project significantly enhances the speed and accuracy of generative +AI for extending its features and adding new workflows. This capability was a +core design principle. While +[generative AI may not yet be optimal for all Linux kernel development](https://neurips.cc/virtual/2024/poster/97426), +we lower the barrier to its use on kdevops by adopting a structured, +declarative approach to defining and implementing Linux workflows. + +Kconfig serves as our clear modeling variability language, providing precise +symbol dependency relationships and encouraging symbol documentation through +straightforward language. + +We've also +[extended kconfig with optional selective YAML output](https://github.com/linux-kdevops/kconfig/commit/2665aeb3b7e0a524e15de31937e8e4344fc131ba) +enabling a direct one-to-one mapping of model variability to Ansible input +variables. Ansible's YAML-based Task Language offers a well-defined structure +for describing tasks, handlers, plays, and roles, which effectively removes +ambiguity regarding goals and objectives. + +We've pushed the limits of Kconfig's leverage, making what would otherwise +be complex functionality such as adding support for new filesystems as simple +as extending Kconfig files. For example, see the +[kdevops mmtests filesystem support Kconfig file](https://github.com/linux-kdevops/kdevops/blob/main/workflows/mmtests/Kconfig.xfs). + +Our long-term hypothesis was that by minimizing actual code implementation and +instead using stricter modeling variability languages -- assisted by +documentation via help menus and clear declarative tasks -- we could more +effectively leverage generative AI to extend kdevops. This hypothesis is +proving true, allowing us to utilize AI agents now like +[Claude Code](https://www.anthropic.com/claude-code) for many kdevops +modifications. + +The kdevops project fundamentally acts as a +[Software 3.0 enabler for Linux kernel development](https://www.latent.space/p/s3). +New workflows are now being added with generative AI, and the entire +[kdevops dashboard](https://kdevops.org/) is fully generated by generative AI. +Recent efforts also enable CI integration without developers needing to touch +any CI files, which was a deliberate design choice given the inherent +complexity and debugging challenges of continuous integration configurations. + +### Resources for leveraging generative AI on kdevops + +To assist both humans and bots with examples on how to leverage generative AI +on kdevops, refer to these documentation files: + + * [CLAUDE.md](CLAUDE.md) - Claude Code guidelines + * [PROMPTS.md](PROMPTS.md) - Example of prompts and respective commits + ## Quick kdevops demos To give you an idea of the power and goals behind kdevops we provide a few @@ -284,6 +335,7 @@ in the future. # Video presentations on kdevops or related + * [March 24, 2025 kdevops: Automating Linux kernel testing](https://www.youtube.com/watch?v=VF-jr_ZE-9Y&list=PLjaT52x3HVboZtdwZnONSHQHM8217ELcc) and [slides](https://docs.google.com/presentation/d/e/2PACX-1vSyM8ol_ZFwmJ6YCI2sYU9xH8MPCLPZT1PfsDHId__xbUq2aSh7mAeMLTGZFGfvjogY6ccxww2GqOTe/pub?start=false&loop=false&delayms=3000) * [May 10, 2023 kdevops: Advances with automation of testing with fstests and blktests](https://www.youtube.com/watch?v=aC4gb0r9Hho&ab_channel=TheLinuxFoundation) * [LWN coverage of this talk](https://lwn.net/Articles/937830/) * A follow up on requests from folks to store failures -- 2.47.2