[PATCH v2 0/4] declared hosts support

[Luis Chamberlain <mcgrof@kernel.org>]

This v2 adresses Chuck's feedback to make the include just use the
workflow name. This scales better.

While at it, I decided to ensure to only enable workflows which are
tested, so far that is none, so they all just need to be disabled.
But I'll add one next which goes tested.

The data partition stuff is modified so to be skipped as well. We expect
the user to have set that up, they can howeer specify a custom path for
that. The new workflow is minio WARP testing. Since we already used
minio setup via docker, we take that from the milvus ai role, and
generalize it to be shared.

If folks want to try this:

https://github.com/linux-kdevops/kdevops/tree/20250830-lambda-minio

And how you can easily test this on an existing server which
you already have ssh access to:

make defconfig-minio-warp-xfs DECLARE_HOSTS=foo WARP_DEVICE=/dev/nvme4n1
make
make minio
make minio-warp

See the resutls on workflows/minio/results/

That will configure the device with xfs with 4k block size. To test
with 16k block size you can use:

make defconfig-minio-warp-xfs-16k DECLARE_HOSTS=foo WARP_DEVICE=/dev/nvme4n1

Luis Chamberlain (4):
  gen_hosts: use kdevops_workflow_name directly for template selection
  declared_hosts: add support for pre-existing infrastructure
  Makefile: add missing extra_vars.yaml dependencies
  minio: add MinIO Warp S3 benchmarking with declared hosts support

 .gitignore                                    |   2 +
 Makefile                                      |  12 +-
 defconfigs/minio-warp                         |  52 ++
 defconfigs/minio-warp-ab                      |  41 +
 defconfigs/minio-warp-btrfs                   |  35 +
 defconfigs/minio-warp-declared-hosts          |  56 ++
 defconfigs/minio-warp-multifs                 |  74 ++
 defconfigs/minio-warp-storage                 |  65 ++
 defconfigs/minio-warp-xfs                     |  35 +
 defconfigs/minio-warp-xfs-16k                 |  65 ++
 defconfigs/minio-warp-xfs-lbs                 |  65 ++
 defconfigs/xfs_reflink_16k_4ks_declared       |  19 +
 kconfigs/Kconfig.bringup                      |   9 +-
 kconfigs/Kconfig.declared_hosts               |  81 ++
 kconfigs/workflows/Kconfig                    |  43 +
 kconfigs/workflows/Kconfig.data_partition     |  11 +
 playbooks/minio.yml                           |  53 ++
 playbooks/roles/ai_setup/tasks/main.yml       |  40 +-
 .../create_data_partition/tasks/main.yml      |  23 +
 playbooks/roles/devconfig/defaults/main.yml   |   2 +
 playbooks/roles/devconfig/tasks/main.yml      |  25 +
 playbooks/roles/gen_hosts/defaults/main.yml   |   1 +
 playbooks/roles/gen_hosts/tasks/main.yml      | 319 ++-----
 playbooks/roles/gen_hosts/templates/hosts.j2  | 242 +----
 .../roles/gen_hosts/templates/workflows/ai.j2 |  99 ++
 .../gen_hosts/templates/workflows/blktests.j2 |  58 ++
 .../gen_hosts/templates/workflows/cxl.j2      |   7 +
 .../templates/workflows/declared-hosts.j2     | 239 +++++
 .../templates/workflows/fio-tests.j2          |  38 +
 .../gen_hosts/templates/workflows/fstests.j2  |  72 ++
 .../gen_hosts/templates/workflows/gitr.j2     |  41 +
 .../gen_hosts/templates/workflows/ltp.j2      |  41 +
 .../gen_hosts/templates/workflows/minio.j2    | 173 ++++
 .../gen_hosts/templates/workflows/mix.j2      |  62 ++
 .../gen_hosts/templates/workflows/mmtests.j2  |  77 ++
 .../gen_hosts/templates/workflows/nfstest.j2  |  41 +
 .../gen_hosts/templates/workflows/pynfs.j2    |   7 +
 .../templates/workflows/reboot-limit.j2       |  33 +
 .../templates/workflows/selftests.j2          |  53 ++
 .../gen_hosts/templates/workflows/sysbench.j2 |  53 ++
 playbooks/roles/gen_nodes/defaults/main.yml   |   1 +
 playbooks/roles/gen_nodes/tasks/main.yml      | 132 ++-
 .../linux-mirror/python/start-mirroring.py    |   4 +-
 playbooks/roles/minio_destroy/tasks/main.yml  |  34 +
 playbooks/roles/minio_install/tasks/main.yml  |  61 ++
 playbooks/roles/minio_results/tasks/main.yml  |  86 ++
 playbooks/roles/minio_setup/defaults/main.yml |  16 +
 playbooks/roles/minio_setup/tasks/main.yml    | 100 ++
 .../roles/minio_uninstall/tasks/main.yml      |  17 +
 playbooks/roles/minio_warp_run/tasks/main.yml | 249 +++++
 .../templates/warp_config.json.j2             |  14 +
 workflows/Makefile                            |   4 +
 workflows/ai/Makefile                         |   3 -
 workflows/blktests/Makefile                   |   3 -
 workflows/cxl/Makefile                        |   2 -
 workflows/demos/reboot-limit/Kconfig          |   5 +
 workflows/fio-tests/Makefile                  |   3 -
 workflows/fstests/Makefile                    |   3 -
 workflows/gitr/Makefile                       |   3 -
 workflows/linux/Makefile                      |   1 -
 workflows/ltp/Makefile                        |   3 -
 workflows/minio/Kconfig                       |  23 +
 workflows/minio/Kconfig.docker                |  66 ++
 workflows/minio/Kconfig.storage               | 364 ++++++++
 workflows/minio/Kconfig.warp                  | 141 +++
 workflows/minio/Makefile                      |  76 ++
 .../minio/scripts/analyze_warp_results.py     | 858 ++++++++++++++++++
 .../minio/scripts/generate_warp_report.py     | 404 +++++++++
 .../minio/scripts/run_benchmark_suite.sh      | 116 +++
 workflows/mmtests/Makefile                    |   3 -
 workflows/nfstest/Makefile                    |   3 -
 workflows/pynfs/Makefile                      |   3 -
 workflows/selftests/Makefile                  |   3 -
 workflows/sysbench/Makefile                   |   3 -
 74 files changed, 4705 insertions(+), 566 deletions(-)
 create mode 100644 defconfigs/minio-warp
 create mode 100644 defconfigs/minio-warp-ab
 create mode 100644 defconfigs/minio-warp-btrfs
 create mode 100644 defconfigs/minio-warp-declared-hosts
 create mode 100644 defconfigs/minio-warp-multifs
 create mode 100644 defconfigs/minio-warp-storage
 create mode 100644 defconfigs/minio-warp-xfs
 create mode 100644 defconfigs/minio-warp-xfs-16k
 create mode 100644 defconfigs/minio-warp-xfs-lbs
 create mode 100644 defconfigs/xfs_reflink_16k_4ks_declared
 create mode 100644 kconfigs/Kconfig.declared_hosts
 create mode 100644 playbooks/minio.yml
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ai.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/blktests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/cxl.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fstests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/gitr.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ltp.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/minio.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mix.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mmtests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/nfstest.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/pynfs.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/selftests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/sysbench.j2
 create mode 100644 playbooks/roles/minio_destroy/tasks/main.yml
 create mode 100644 playbooks/roles/minio_install/tasks/main.yml
 create mode 100644 playbooks/roles/minio_results/tasks/main.yml
 create mode 100644 playbooks/roles/minio_setup/defaults/main.yml
 create mode 100644 playbooks/roles/minio_setup/tasks/main.yml
 create mode 100644 playbooks/roles/minio_uninstall/tasks/main.yml
 create mode 100644 playbooks/roles/minio_warp_run/tasks/main.yml
 create mode 100644 playbooks/roles/minio_warp_run/templates/warp_config.json.j2
 create mode 100644 workflows/minio/Kconfig
 create mode 100644 workflows/minio/Kconfig.docker
 create mode 100644 workflows/minio/Kconfig.storage
 create mode 100644 workflows/minio/Kconfig.warp
 create mode 100644 workflows/minio/Makefile
 create mode 100755 workflows/minio/scripts/analyze_warp_results.py
 create mode 100755 workflows/minio/scripts/generate_warp_report.py
 create mode 100755 workflows/minio/scripts/run_benchmark_suite.sh

-- 
2.50.1