From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E977433AC for ; Sun, 31 Aug 2025 04:12:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756613530; cv=none; b=UoJsxJBbf6KlpMebet8hKzf/x7ISiJPx5ZPuCf3Wm+GkKZSptPYUAvGK0ha2d/zvT+5A+am6KIyHUHJzffvrBHR2F7plT46aGKyL9iw7MzTH68mqUnAXUCEkLOE+TgeIBcow3S0now6lGKLRD8wT/EQNwaKXqpuNERbGm0K6GS8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756613530; c=relaxed/simple; bh=OunkP/yVS2aC28V3kwi6vtHEdYrVxQYSw+NRHK3l7hI=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=om+AFaEpax9uZsuBbOxbGT2g6N791QfzejcW0EWDYWWAp7TzQ+EPVEZi7i3GrOXo00DHqyk4SmcdA4mdgJdqTVytlPuOb+XDhwBdLmlVocFtKfYpp9lZaQCH/CfVL4jp1yc1HrpjrdBBLp8u6I2bh8yH5tQ3nFFKAugrAnOa/Ek= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=djPVRbrY; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="djPVRbrY" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-Type: Content-ID:Content-Description:In-Reply-To:References; bh=W6BnrDt5lYnT6yWe5T2ulSA29wY4Z9j5xWK8swoelo4=; b=djPVRbrYdOqPPa7MQsnC+rxD5g KFdwfMJ4dACNcIaWYmE0D8oc9peQPewSfdaIbDGYyXoCrlsJrRk72qU570vsQ7iQ+194SIhoZNy2Q Dp3Gd/rMsDoharDbBKQ53d+bX1KRfovuPJKUIprq+v3AkYbOVLYAWmDuRqoQk6BCWICjKVU8hh6jo GmjpjKPbEEFxvJUHZME74lYgqTcritlQvMelaLWmxHHmGcCxJD0lgPWQZhM/Nk/Hm0fwsgr4pVP31 Jp+UHm0R92k6BR3kmL7lm8ALWDHVS1Ed+TI9nfhxas6DD7H2p0GgBC4W4/i+nrJupEhow0KghdJRS CKVZ7bWQ==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98.2 #2 (Red Hat Linux)) id 1usZQE-0000000975I-2tBZ; Sun, 31 Aug 2025 04:12:06 +0000 From: Luis Chamberlain To: Chuck Lever , Daniel Gomez , kdevops@lists.linux.dev Cc: hui81.qi@samsung.com, kundan.kumar@samsung.com, Luis Chamberlain Subject: [PATCH v2 0/4] declared hosts support Date: Sat, 30 Aug 2025 21:11:56 -0700 Message-ID: <20250831041202.2172115-1-mcgrof@kernel.org> X-Mailer: git-send-email 2.49.0 Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain This v2 adresses Chuck's feedback to make the include just use the workflow name. This scales better. While at it, I decided to ensure to only enable workflows which are tested, so far that is none, so they all just need to be disabled. But I'll add one next which goes tested. The data partition stuff is modified so to be skipped as well. We expect the user to have set that up, they can howeer specify a custom path for that. The new workflow is minio WARP testing. Since we already used minio setup via docker, we take that from the milvus ai role, and generalize it to be shared. If folks want to try this: https://github.com/linux-kdevops/kdevops/tree/20250830-lambda-minio And how you can easily test this on an existing server which you already have ssh access to: make defconfig-minio-warp-xfs DECLARE_HOSTS=foo WARP_DEVICE=/dev/nvme4n1 make make minio make minio-warp See the resutls on workflows/minio/results/ That will configure the device with xfs with 4k block size. To test with 16k block size you can use: make defconfig-minio-warp-xfs-16k DECLARE_HOSTS=foo WARP_DEVICE=/dev/nvme4n1 Luis Chamberlain (4): gen_hosts: use kdevops_workflow_name directly for template selection declared_hosts: add support for pre-existing infrastructure Makefile: add missing extra_vars.yaml dependencies minio: add MinIO Warp S3 benchmarking with declared hosts support .gitignore | 2 + Makefile | 12 +- defconfigs/minio-warp | 52 ++ defconfigs/minio-warp-ab | 41 + defconfigs/minio-warp-btrfs | 35 + defconfigs/minio-warp-declared-hosts | 56 ++ defconfigs/minio-warp-multifs | 74 ++ defconfigs/minio-warp-storage | 65 ++ defconfigs/minio-warp-xfs | 35 + defconfigs/minio-warp-xfs-16k | 65 ++ defconfigs/minio-warp-xfs-lbs | 65 ++ defconfigs/xfs_reflink_16k_4ks_declared | 19 + kconfigs/Kconfig.bringup | 9 +- kconfigs/Kconfig.declared_hosts | 81 ++ kconfigs/workflows/Kconfig | 43 + kconfigs/workflows/Kconfig.data_partition | 11 + playbooks/minio.yml | 53 ++ playbooks/roles/ai_setup/tasks/main.yml | 40 +- .../create_data_partition/tasks/main.yml | 23 + playbooks/roles/devconfig/defaults/main.yml | 2 + playbooks/roles/devconfig/tasks/main.yml | 25 + playbooks/roles/gen_hosts/defaults/main.yml | 1 + playbooks/roles/gen_hosts/tasks/main.yml | 319 ++----- playbooks/roles/gen_hosts/templates/hosts.j2 | 242 +---- .../roles/gen_hosts/templates/workflows/ai.j2 | 99 ++ .../gen_hosts/templates/workflows/blktests.j2 | 58 ++ .../gen_hosts/templates/workflows/cxl.j2 | 7 + .../templates/workflows/declared-hosts.j2 | 239 +++++ .../templates/workflows/fio-tests.j2 | 38 + .../gen_hosts/templates/workflows/fstests.j2 | 72 ++ .../gen_hosts/templates/workflows/gitr.j2 | 41 + .../gen_hosts/templates/workflows/ltp.j2 | 41 + .../gen_hosts/templates/workflows/minio.j2 | 173 ++++ .../gen_hosts/templates/workflows/mix.j2 | 62 ++ .../gen_hosts/templates/workflows/mmtests.j2 | 77 ++ .../gen_hosts/templates/workflows/nfstest.j2 | 41 + .../gen_hosts/templates/workflows/pynfs.j2 | 7 + .../templates/workflows/reboot-limit.j2 | 33 + .../templates/workflows/selftests.j2 | 53 ++ .../gen_hosts/templates/workflows/sysbench.j2 | 53 ++ playbooks/roles/gen_nodes/defaults/main.yml | 1 + playbooks/roles/gen_nodes/tasks/main.yml | 132 ++- .../linux-mirror/python/start-mirroring.py | 4 +- playbooks/roles/minio_destroy/tasks/main.yml | 34 + playbooks/roles/minio_install/tasks/main.yml | 61 ++ playbooks/roles/minio_results/tasks/main.yml | 86 ++ playbooks/roles/minio_setup/defaults/main.yml | 16 + playbooks/roles/minio_setup/tasks/main.yml | 100 ++ .../roles/minio_uninstall/tasks/main.yml | 17 + playbooks/roles/minio_warp_run/tasks/main.yml | 249 +++++ .../templates/warp_config.json.j2 | 14 + workflows/Makefile | 4 + workflows/ai/Makefile | 3 - workflows/blktests/Makefile | 3 - workflows/cxl/Makefile | 2 - workflows/demos/reboot-limit/Kconfig | 5 + workflows/fio-tests/Makefile | 3 - workflows/fstests/Makefile | 3 - workflows/gitr/Makefile | 3 - workflows/linux/Makefile | 1 - workflows/ltp/Makefile | 3 - workflows/minio/Kconfig | 23 + workflows/minio/Kconfig.docker | 66 ++ workflows/minio/Kconfig.storage | 364 ++++++++ workflows/minio/Kconfig.warp | 141 +++ workflows/minio/Makefile | 76 ++ .../minio/scripts/analyze_warp_results.py | 858 ++++++++++++++++++ .../minio/scripts/generate_warp_report.py | 404 +++++++++ .../minio/scripts/run_benchmark_suite.sh | 116 +++ workflows/mmtests/Makefile | 3 - workflows/nfstest/Makefile | 3 - workflows/pynfs/Makefile | 3 - workflows/selftests/Makefile | 3 - workflows/sysbench/Makefile | 3 - 74 files changed, 4705 insertions(+), 566 deletions(-) create mode 100644 defconfigs/minio-warp create mode 100644 defconfigs/minio-warp-ab create mode 100644 defconfigs/minio-warp-btrfs create mode 100644 defconfigs/minio-warp-declared-hosts create mode 100644 defconfigs/minio-warp-multifs create mode 100644 defconfigs/minio-warp-storage create mode 100644 defconfigs/minio-warp-xfs create mode 100644 defconfigs/minio-warp-xfs-16k create mode 100644 defconfigs/minio-warp-xfs-lbs create mode 100644 defconfigs/xfs_reflink_16k_4ks_declared create mode 100644 kconfigs/Kconfig.declared_hosts create mode 100644 playbooks/minio.yml create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ai.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/blktests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/cxl.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fstests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/gitr.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ltp.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/minio.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mix.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mmtests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/nfstest.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/pynfs.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/selftests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/sysbench.j2 create mode 100644 playbooks/roles/minio_destroy/tasks/main.yml create mode 100644 playbooks/roles/minio_install/tasks/main.yml create mode 100644 playbooks/roles/minio_results/tasks/main.yml create mode 100644 playbooks/roles/minio_setup/defaults/main.yml create mode 100644 playbooks/roles/minio_setup/tasks/main.yml create mode 100644 playbooks/roles/minio_uninstall/tasks/main.yml create mode 100644 playbooks/roles/minio_warp_run/tasks/main.yml create mode 100644 playbooks/roles/minio_warp_run/templates/warp_config.json.j2 create mode 100644 workflows/minio/Kconfig create mode 100644 workflows/minio/Kconfig.docker create mode 100644 workflows/minio/Kconfig.storage create mode 100644 workflows/minio/Kconfig.warp create mode 100644 workflows/minio/Makefile create mode 100755 workflows/minio/scripts/analyze_warp_results.py create mode 100755 workflows/minio/scripts/generate_warp_report.py create mode 100755 workflows/minio/scripts/run_benchmark_suite.sh -- 2.50.1