From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3305D2192EE for ; Sun, 31 Aug 2025 04:12:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756613534; cv=none; b=VVzdzORKyN5Hhh6/hN9M/LQ/XxbBKEFEbtpyv3ax3faMj4SFSIIaBWFd3a6pcCECT09JxnV1xJ0DTP6c1GjejqolDszPzBbGwXmyeaNEGGbIy1wfFlCOvtgRgJcP4bjqWwPRAjPLWsI6CLoGa7UED1QRKcFSCg+TOnD9iN/TmZY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756613534; c=relaxed/simple; bh=YFZkEfbml11sCWBID6cc0HSfk3zH95uK5s/5uj7jViY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tSuPGROl10b8LCmdV6H7g5zsU/Rf0i7QYoKREX30rR9FF4v+XPruWMnLCe3XAfsHLCwZcTRRkm08LoKAADiZfgq0JE3B3UT8CSdqdi74467qhNK9PHnsRtoFQ5m2jQe+LrU68Vm80tNvFlcCLZNJy3eqYuUjP35QY2IrwUlHHno= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=J7FS4EIO; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="J7FS4EIO" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=cdB3zDKmbGhsSBc9uEc0dpTLDHZJg7yaNb0IBL4VLQE=; b=J7FS4EIOA97UiJfe2nWn6YpVOk /eELZ69Z0HJCfRjneUVFn9O9eSSaIJBlTat8K0n5cbK4uoyp9Qpx8R74o2Sx0w0wQ8fiYmzTSyl4N 3DhRHwknBtUITTEHRTmz6RjTCf8o1a4w25SY5ujD5CifGZLO3OewK4dJAh4HwRaasu+SYnb+RjvaC Uj5KHu2Ggm75de9Z0BbVGB8XZA1kOTGBuAkDXO/IWKvP+i5s9+0eckOSOSU4//ppQSXPI3QJLjvOh woheN61c83EMFhvrBF6eBQbygHx01kYv/f9d9MMCwOt0NqDPbF3EchzIFDxZeTrPKycFbhXpbFyHV wexAu2Hw==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98.2 #2 (Red Hat Linux)) id 1usZQE-0000000975Q-3V8J; Sun, 31 Aug 2025 04:12:06 +0000 From: Luis Chamberlain To: Chuck Lever , Daniel Gomez , kdevops@lists.linux.dev Cc: hui81.qi@samsung.com, kundan.kumar@samsung.com, Luis Chamberlain Subject: [PATCH v2 4/4] minio: add MinIO Warp S3 benchmarking with declared hosts support Date: Sat, 30 Aug 2025 21:12:00 -0700 Message-ID: <20250831041202.2172115-5-mcgrof@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250831041202.2172115-1-mcgrof@kernel.org> References: <20250831041202.2172115-1-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain Add MinIO Warp S3 benchmarking workflow with support for declared hosts (pre-existing infrastructure) and fix critical issues in template generation and benchmark execution. MinIO Warp Workflow: - Add MinIO server deployment via Docker containers - Implement Warp S3 benchmark suite with configurable parameters - Support both single and comprehensive benchmark modes - Add storage configuration for XFS/Btrfs/ext4 filesystems - Include benchmark result analysis and visualization tools - Fix benchmark duration handling (was terminating after 45s due to --autoterm flag with --objects parameter) - Fix async timeout calculation for long-running benchmarks - Add proper help targets to Makefile Declared Hosts Support: - Enable using pre-existing infrastructure (bare metal, cloud VMs) - Skip bringup/teardown for systems with existing SSH access - Add DECLARED_HOSTS and DATA_PATH CLI variable overrides - Disable data partition creation when using declared hosts - Restrict unreviewed workflows pending compatibility testing - Auto-infer user/group settings on target systems Template System Improvements: - Simplify gen_hosts template selection using kdevops_workflow_name - Reduce 40+ lines of conditionals to single dynamic inclusion - Fix fstests host generation regression (was generating 251 hosts) - Add workflow-specific template structure under workflows/*/ Build System Fixes: - Add missing extra_vars.yaml dependencies to prevent build failures - Fix Kconfig yaml output for minio_warp_run_comprehensive_suite - Ensure proper task ordering in Ansible playbooks - Fix Jinja2 syntax errors in benchmark scripts The workflow now properly runs for configured durations (e.g., 30m) and supports both dedicated test infrastructure and pre-existing systems via declared hosts. Generated-by: Claude AI Signed-off-by: Luis Chamberlain --- .gitignore | 2 + defconfigs/minio-warp | 52 ++ defconfigs/minio-warp-ab | 41 + defconfigs/minio-warp-btrfs | 35 + defconfigs/minio-warp-declared-hosts | 56 ++ defconfigs/minio-warp-multifs | 74 ++ defconfigs/minio-warp-storage | 65 ++ defconfigs/minio-warp-xfs | 35 + defconfigs/minio-warp-xfs-16k | 65 ++ defconfigs/minio-warp-xfs-lbs | 65 ++ kconfigs/workflows/Kconfig | 19 + playbooks/minio.yml | 53 ++ playbooks/roles/ai_setup/tasks/main.yml | 40 +- playbooks/roles/gen_hosts/tasks/main.yml | 19 +- .../gen_hosts/templates/workflows/minio.j2 | 173 ++++ playbooks/roles/gen_nodes/tasks/main.yml | 128 ++- playbooks/roles/minio_destroy/tasks/main.yml | 34 + playbooks/roles/minio_install/tasks/main.yml | 61 ++ playbooks/roles/minio_results/tasks/main.yml | 86 ++ playbooks/roles/minio_setup/defaults/main.yml | 16 + playbooks/roles/minio_setup/tasks/main.yml | 100 ++ .../roles/minio_uninstall/tasks/main.yml | 17 + playbooks/roles/minio_warp_run/tasks/main.yml | 249 +++++ .../templates/warp_config.json.j2 | 14 + workflows/Makefile | 4 + workflows/minio/Kconfig | 23 + workflows/minio/Kconfig.docker | 66 ++ workflows/minio/Kconfig.storage | 364 ++++++++ workflows/minio/Kconfig.warp | 141 +++ workflows/minio/Makefile | 76 ++ .../minio/scripts/analyze_warp_results.py | 858 ++++++++++++++++++ .../minio/scripts/generate_warp_report.py | 404 +++++++++ .../minio/scripts/run_benchmark_suite.sh | 116 +++ 33 files changed, 3517 insertions(+), 34 deletions(-) create mode 100644 defconfigs/minio-warp create mode 100644 defconfigs/minio-warp-ab create mode 100644 defconfigs/minio-warp-btrfs create mode 100644 defconfigs/minio-warp-declared-hosts create mode 100644 defconfigs/minio-warp-multifs create mode 100644 defconfigs/minio-warp-storage create mode 100644 defconfigs/minio-warp-xfs create mode 100644 defconfigs/minio-warp-xfs-16k create mode 100644 defconfigs/minio-warp-xfs-lbs create mode 100644 playbooks/minio.yml create mode 100644 playbooks/roles/gen_hosts/templates/workflows/minio.j2 create mode 100644 playbooks/roles/minio_destroy/tasks/main.yml create mode 100644 playbooks/roles/minio_install/tasks/main.yml create mode 100644 playbooks/roles/minio_results/tasks/main.yml create mode 100644 playbooks/roles/minio_setup/defaults/main.yml create mode 100644 playbooks/roles/minio_setup/tasks/main.yml create mode 100644 playbooks/roles/minio_uninstall/tasks/main.yml create mode 100644 playbooks/roles/minio_warp_run/tasks/main.yml create mode 100644 playbooks/roles/minio_warp_run/templates/warp_config.json.j2 create mode 100644 workflows/minio/Kconfig create mode 100644 workflows/minio/Kconfig.docker create mode 100644 workflows/minio/Kconfig.storage create mode 100644 workflows/minio/Kconfig.warp create mode 100644 workflows/minio/Makefile create mode 100755 workflows/minio/scripts/analyze_warp_results.py create mode 100755 workflows/minio/scripts/generate_warp_report.py create mode 100755 workflows/minio/scripts/run_benchmark_suite.sh diff --git a/.gitignore b/.gitignore index b725aba..67ab9d2 100644 --- a/.gitignore +++ b/.gitignore @@ -89,6 +89,8 @@ playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml # yet. workflows/selftests/results/ +workflows/minio/results/ + workflows/linux/refs/default/Kconfig.linus workflows/linux/refs/default/Kconfig.next workflows/linux/refs/default/Kconfig.stable diff --git a/defconfigs/minio-warp b/defconfigs/minio-warp new file mode 100644 index 0000000..4d55c2d --- /dev/null +++ b/defconfigs/minio-warp @@ -0,0 +1,52 @@ +# +# MinIO Warp S3 benchmarking configuration +# +# Automatically generated file; DO NOT EDIT. +# kdevops 5.0.2 Configuration +# +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_LINUX_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# +# MinIO Docker Configuration +# +CONFIG_MINIO_CONTAINER_IMAGE_STRING="minio/minio:RELEASE.2024-01-16T16-07-38Z" +CONFIG_MINIO_CONTAINER_NAME="minio-warp-server" +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" +CONFIG_MINIO_DATA_PATH="/data/minio" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-warp-network" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_MEMORY_LIMIT="4g" + +# +# Warp Benchmark Configuration +# +CONFIG_MINIO_WARP_BENCHMARK_MIXED=y +CONFIG_MINIO_WARP_BENCHMARK_TYPE="mixed" +CONFIG_MINIO_WARP_DURATION="5m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=10 +CONFIG_MINIO_WARP_OBJECT_SIZE="1MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=100 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=y +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" + +# +# Host Configuration +# +CONFIG_KDEVOPS_HOSTS_PREFIX="minio" + +# +# Node configuration +# +CONFIG_KDEVOPS_NODES_TEMPLATE="guestfs-libvirt" +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME_SIZE_GIB=10 \ No newline at end of file diff --git a/defconfigs/minio-warp-ab b/defconfigs/minio-warp-ab new file mode 100644 index 0000000..f20142d --- /dev/null +++ b/defconfigs/minio-warp-ab @@ -0,0 +1,41 @@ +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# A/B Testing Configuration +CONFIG_KDEVOPS_BASELINE_AND_DEV=y + +# MinIO Configuration +CONFIG_MINIO_ENABLE=y +CONFIG_MINIO_CONTAINER_IMAGE="minio/minio:latest" +CONFIG_MINIO_CONTAINER_NAME="minio-server" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" +CONFIG_MINIO_DATA_PATH="/data/minio" +CONFIG_MINIO_MEMORY_LIMIT="4g" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-network" + +# Warp Benchmark Configuration - Comprehensive Suite +CONFIG_MINIO_WARP_BENCHMARK_MIXED=y +CONFIG_MINIO_WARP_DURATION="2m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=20 +CONFIG_MINIO_WARP_OBJECT_SIZE="10MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=100 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=n +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" + +# Enable web UI for monitoring +CONFIG_MINIO_WARP_ENABLE_WEB_UI=y +CONFIG_MINIO_WARP_WEB_UI_PORT=7762 + +# Node configuration for A/B testing +CONFIG_KDEVOPS_HOSTS_TEMPLATE="hosts.j2" +CONFIG_KDEVOPS_NODES_TEMPLATE="nodes.j2" +CONFIG_KDEVOPS_PLAYBOOK_DIR="playbooks" +CONFIG_KDEVOPS_ANSIBLE_INVENTORY_FILE="hosts" +CONFIG_KDEVOPS_NODES="nodes.yaml" \ No newline at end of file diff --git a/defconfigs/minio-warp-btrfs b/defconfigs/minio-warp-btrfs new file mode 100644 index 0000000..85bd8fa --- /dev/null +++ b/defconfigs/minio-warp-btrfs @@ -0,0 +1,35 @@ +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# MinIO Configuration for Btrfs testing +CONFIG_MINIO_ENABLE=y +CONFIG_MINIO_CONTAINER_IMAGE="minio/minio:latest" +CONFIG_MINIO_CONTAINER_NAME="minio-server" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" + +# Configure Btrfs filesystem for MinIO storage +CONFIG_MINIO_USE_CUSTOM_FILESYSTEM=y +CONFIG_MINIO_STORAGE_DEVICE="/dev/nvme0n1" +CONFIG_MINIO_STORAGE_FSTYPE="btrfs" +CONFIG_MINIO_STORAGE_FS_OPTS="--nodesize 16k" +CONFIG_MINIO_STORAGE_LABEL="minio-btrfs" +CONFIG_MINIO_DATA_PATH="/data/minio" + +CONFIG_MINIO_MEMORY_LIMIT="4g" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-network" + +# Comprehensive benchmark suite +CONFIG_MINIO_WARP_RUN_COMPREHENSIVE_SUITE=y +CONFIG_MINIO_WARP_DURATION="2m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=20 +CONFIG_MINIO_WARP_OBJECT_SIZE="10MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=100 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=n +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" \ No newline at end of file diff --git a/defconfigs/minio-warp-declared-hosts b/defconfigs/minio-warp-declared-hosts new file mode 100644 index 0000000..acf1e64 --- /dev/null +++ b/defconfigs/minio-warp-declared-hosts @@ -0,0 +1,56 @@ +# +# MinIO Warp S3 benchmarking with declared hosts (bare metal or pre-existing infrastructure) +# +# Automatically generated file; DO NOT EDIT. +# kdevops 5.0.2 Configuration +# +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_LINUX_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# Skip bringup for declared hosts +CONFIG_SKIP_BRINGUP=y +CONFIG_KDEVOPS_USE_DECLARED_HOSTS=y + +# +# MinIO Docker Configuration +# +CONFIG_MINIO_CONTAINER_IMAGE_STRING="minio/minio:RELEASE.2024-01-16T16-07-38Z" +CONFIG_MINIO_CONTAINER_NAME="minio-warp-server" +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" +CONFIG_MINIO_DATA_PATH="/data/minio" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-warp-network" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_MEMORY_LIMIT="4g" + +# +# MinIO Storage Configuration +# +CONFIG_MINIO_STORAGE_ENABLE=y +CONFIG_MINIO_MOUNT_POINT="/data/minio" +CONFIG_MINIO_FSTYPE_XFS=y +CONFIG_MINIO_FSTYPE="xfs" +CONFIG_MINIO_XFS_BLOCKSIZE_16K=y +CONFIG_MINIO_XFS_BLOCKSIZE=16384 +CONFIG_MINIO_XFS_SECTORSIZE_4K=y +CONFIG_MINIO_XFS_SECTORSIZE=4096 +CONFIG_MINIO_XFS_MKFS_OPTS="" + +# +# Warp Benchmark Configuration +# +CONFIG_MINIO_WARP_RUN_COMPREHENSIVE_SUITE=y +CONFIG_MINIO_WARP_DURATION="5m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=10 +CONFIG_MINIO_WARP_OBJECT_SIZE="1MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=100 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=y +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" \ No newline at end of file diff --git a/defconfigs/minio-warp-multifs b/defconfigs/minio-warp-multifs new file mode 100644 index 0000000..8316a3f --- /dev/null +++ b/defconfigs/minio-warp-multifs @@ -0,0 +1,74 @@ +# +# MinIO Warp S3 benchmarking with multi-filesystem testing +# +# Automatically generated file; DO NOT EDIT. +# kdevops 5.0.2 Configuration +# +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_LINUX_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# +# MinIO Docker Configuration +# +CONFIG_MINIO_CONTAINER_IMAGE_STRING="minio/minio:RELEASE.2024-01-16T16-07-38Z" +CONFIG_MINIO_CONTAINER_NAME="minio-warp-server" +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" +CONFIG_MINIO_DATA_PATH="/data/minio" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-warp-network" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_MEMORY_LIMIT="4g" + +# +# MinIO Storage Configuration with Multi-filesystem Testing +# +CONFIG_MINIO_STORAGE_ENABLE=y +CONFIG_MINIO_MOUNT_POINT="/data/minio" +CONFIG_MINIO_ENABLE_MULTIFS_TESTING=y + +# XFS configurations +CONFIG_MINIO_MULTIFS_TEST_XFS=y +CONFIG_MINIO_MULTIFS_XFS_4K_4KS=y +CONFIG_MINIO_MULTIFS_XFS_16K_4KS=y + +# ext4 configurations +CONFIG_MINIO_MULTIFS_TEST_EXT4=y +CONFIG_MINIO_MULTIFS_EXT4_4K=y + +# btrfs configurations +CONFIG_MINIO_MULTIFS_TEST_BTRFS=y +CONFIG_MINIO_MULTIFS_BTRFS_DEFAULT=y + +CONFIG_MINIO_MULTIFS_RESULTS_DIR="/data/minio-multifs-benchmark" + +# +# Warp Benchmark Configuration +# +CONFIG_MINIO_WARP_RUN_COMPREHENSIVE_SUITE=y +CONFIG_MINIO_WARP_DURATION="5m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=10 +CONFIG_MINIO_WARP_OBJECT_SIZE="1MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=100 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=y +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" + +# +# Host Configuration +# +CONFIG_KDEVOPS_HOSTS_PREFIX="minio" + +# +# Node configuration +# +CONFIG_KDEVOPS_NODES_TEMPLATE="guestfs-libvirt" +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME_SIZE_GIB=100 +CONFIG_LIBVIRT_EXTRA_NUM_DRIVES=1 diff --git a/defconfigs/minio-warp-storage b/defconfigs/minio-warp-storage new file mode 100644 index 0000000..7f86212 --- /dev/null +++ b/defconfigs/minio-warp-storage @@ -0,0 +1,65 @@ +# +# MinIO Warp S3 benchmarking with dedicated storage configuration +# +# Automatically generated file; DO NOT EDIT. +# kdevops 5.0.2 Configuration +# +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_LINUX_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# +# MinIO Docker Configuration +# +CONFIG_MINIO_CONTAINER_IMAGE_STRING="minio/minio:RELEASE.2024-01-16T16-07-38Z" +CONFIG_MINIO_CONTAINER_NAME="minio-warp-server" +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" +CONFIG_MINIO_DATA_PATH="/data/minio" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-warp-network" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_MEMORY_LIMIT="4g" + +# +# MinIO Storage Configuration +# +CONFIG_MINIO_STORAGE_ENABLE=y +CONFIG_MINIO_MOUNT_POINT="/data/minio" +CONFIG_MINIO_FSTYPE_XFS=y +CONFIG_MINIO_FSTYPE="xfs" +CONFIG_MINIO_XFS_BLOCKSIZE_4K=y +CONFIG_MINIO_XFS_BLOCKSIZE=4096 +CONFIG_MINIO_XFS_SECTORSIZE_4K=y +CONFIG_MINIO_XFS_SECTORSIZE=4096 +CONFIG_MINIO_XFS_MKFS_OPTS="" + +# +# Warp Benchmark Configuration +# +CONFIG_MINIO_WARP_RUN_COMPREHENSIVE_SUITE=y +CONFIG_MINIO_WARP_DURATION="5m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=10 +CONFIG_MINIO_WARP_OBJECT_SIZE="1MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=100 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=y +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" + +# +# Host Configuration +# +CONFIG_KDEVOPS_HOSTS_PREFIX="minio" + +# +# Node configuration +# +CONFIG_KDEVOPS_NODES_TEMPLATE="guestfs-libvirt" +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME_SIZE_GIB=100 +CONFIG_LIBVIRT_EXTRA_NUM_DRIVES=1 \ No newline at end of file diff --git a/defconfigs/minio-warp-xfs b/defconfigs/minio-warp-xfs new file mode 100644 index 0000000..95c4a64 --- /dev/null +++ b/defconfigs/minio-warp-xfs @@ -0,0 +1,35 @@ +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# MinIO Configuration for XFS testing +CONFIG_MINIO_ENABLE=y +CONFIG_MINIO_CONTAINER_IMAGE="minio/minio:latest" +CONFIG_MINIO_CONTAINER_NAME="minio-server" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" + +# Configure XFS filesystem for MinIO storage +CONFIG_MINIO_USE_CUSTOM_FILESYSTEM=y +CONFIG_MINIO_STORAGE_DEVICE="/dev/nvme0n1" +CONFIG_MINIO_STORAGE_FSTYPE="xfs" +CONFIG_MINIO_STORAGE_FS_OPTS="-b size=4k -s size=4k" +CONFIG_MINIO_STORAGE_LABEL="minio-xfs" +CONFIG_MINIO_DATA_PATH="/data/minio" + +CONFIG_MINIO_MEMORY_LIMIT="4g" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-network" + +# Comprehensive benchmark suite +CONFIG_MINIO_WARP_RUN_COMPREHENSIVE_SUITE=y +CONFIG_MINIO_WARP_DURATION="2m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=20 +CONFIG_MINIO_WARP_OBJECT_SIZE="10MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=100 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=n +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" \ No newline at end of file diff --git a/defconfigs/minio-warp-xfs-16k b/defconfigs/minio-warp-xfs-16k new file mode 100644 index 0000000..82a90b9 --- /dev/null +++ b/defconfigs/minio-warp-xfs-16k @@ -0,0 +1,65 @@ +# +# MinIO Warp S3 benchmarking with XFS 16K block size configuration +# +# Automatically generated file; DO NOT EDIT. +# kdevops 5.0.2 Configuration +# +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_LINUX_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# +# MinIO Docker Configuration +# +CONFIG_MINIO_CONTAINER_IMAGE_STRING="minio/minio:RELEASE.2024-01-16T16-07-38Z" +CONFIG_MINIO_CONTAINER_NAME="minio-warp-server" +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" +CONFIG_MINIO_DATA_PATH="/data/minio" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-warp-network" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_MEMORY_LIMIT="4g" + +# +# MinIO Storage Configuration - XFS with 16K blocks +# +CONFIG_MINIO_STORAGE_ENABLE=y +CONFIG_MINIO_MOUNT_POINT="/data/minio" +CONFIG_MINIO_FSTYPE_XFS=y +CONFIG_MINIO_FSTYPE="xfs" +CONFIG_MINIO_XFS_BLOCKSIZE_16K=y +CONFIG_MINIO_XFS_BLOCKSIZE=16384 +CONFIG_MINIO_XFS_SECTORSIZE_4K=y +CONFIG_MINIO_XFS_SECTORSIZE=4096 +CONFIG_MINIO_XFS_MKFS_OPTS="" + +# +# Warp Benchmark Configuration +# +CONFIG_MINIO_WARP_RUN_COMPREHENSIVE_SUITE=y +CONFIG_MINIO_WARP_DURATION="5m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=10 +CONFIG_MINIO_WARP_OBJECT_SIZE="1MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=100 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=y +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" + +# +# Host Configuration +# +CONFIG_KDEVOPS_HOSTS_PREFIX="minio" + +# +# Node configuration +# +CONFIG_KDEVOPS_NODES_TEMPLATE="guestfs-libvirt" +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME_SIZE_GIB=100 +CONFIG_LIBVIRT_EXTRA_NUM_DRIVES=1 diff --git a/defconfigs/minio-warp-xfs-lbs b/defconfigs/minio-warp-xfs-lbs new file mode 100644 index 0000000..7400954 --- /dev/null +++ b/defconfigs/minio-warp-xfs-lbs @@ -0,0 +1,65 @@ +# +# MinIO Warp S3 benchmarking with XFS Large Block Size (LBS) configuration +# +# Automatically generated file; DO NOT EDIT. +# kdevops 5.0.2 Configuration +# +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_LINUX_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO=y +CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP=y + +# +# MinIO Docker Configuration +# +CONFIG_MINIO_CONTAINER_IMAGE_STRING="minio/minio:RELEASE.2024-01-16T16-07-38Z" +CONFIG_MINIO_CONTAINER_NAME="minio-warp-server" +CONFIG_MINIO_ACCESS_KEY="minioadmin" +CONFIG_MINIO_SECRET_KEY="minioadmin" +CONFIG_MINIO_DATA_PATH="/data/minio" +CONFIG_MINIO_DOCKER_NETWORK_NAME="minio-warp-network" +CONFIG_MINIO_API_PORT=9000 +CONFIG_MINIO_CONSOLE_PORT=9001 +CONFIG_MINIO_MEMORY_LIMIT="4g" + +# +# MinIO Storage Configuration - XFS with 64K blocks (LBS) +# +CONFIG_MINIO_STORAGE_ENABLE=y +CONFIG_MINIO_MOUNT_POINT="/data/minio" +CONFIG_MINIO_FSTYPE_XFS=y +CONFIG_MINIO_FSTYPE="xfs" +CONFIG_MINIO_XFS_BLOCKSIZE_64K=y +CONFIG_MINIO_XFS_BLOCKSIZE=65536 +CONFIG_MINIO_XFS_SECTORSIZE_4K=y +CONFIG_MINIO_XFS_SECTORSIZE=4096 +CONFIG_MINIO_XFS_MKFS_OPTS="" + +# +# Warp Benchmark Configuration - Large objects for LBS testing +# +CONFIG_MINIO_WARP_RUN_COMPREHENSIVE_SUITE=y +CONFIG_MINIO_WARP_DURATION="10m" +CONFIG_MINIO_WARP_CONCURRENT_REQUESTS=20 +CONFIG_MINIO_WARP_OBJECT_SIZE="10MB" +CONFIG_MINIO_WARP_OBJECTS_PER_REQUEST=50 +CONFIG_MINIO_WARP_BUCKET_NAME="warp-benchmark-bucket" +CONFIG_MINIO_WARP_AUTO_TERMINATE=y +CONFIG_MINIO_WARP_ENABLE_CLEANUP=y +CONFIG_MINIO_WARP_OUTPUT_FORMAT="json" + +# +# Host Configuration +# +CONFIG_KDEVOPS_HOSTS_PREFIX="minio" + +# +# Node configuration +# +CONFIG_KDEVOPS_NODES_TEMPLATE="guestfs-libvirt" +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME_SIZE_GIB=200 +CONFIG_LIBVIRT_EXTRA_NUM_DRIVES=1 \ No newline at end of file diff --git a/kconfigs/workflows/Kconfig b/kconfigs/workflows/Kconfig index cca0b70..73ba976 100644 --- a/kconfigs/workflows/Kconfig +++ b/kconfigs/workflows/Kconfig @@ -233,6 +233,13 @@ config KDEVOPS_WORKFLOW_DEDICATE_AI This will dedicate your configuration to running only the AI workflow for vector database performance testing. +config KDEVOPS_WORKFLOW_DEDICATE_MINIO + bool "minio" + select KDEVOPS_WORKFLOW_ENABLE_MINIO + help + This will dedicate your configuration to running only the + MinIO workflow for S3 storage benchmarking with Warp testing. + endchoice config KDEVOPS_WORKFLOW_NAME @@ -250,6 +257,7 @@ config KDEVOPS_WORKFLOW_NAME default "mmtests" if KDEVOPS_WORKFLOW_DEDICATE_MMTESTS default "fio-tests" if KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS default "ai" if KDEVOPS_WORKFLOW_DEDICATE_AI + default "minio" if KDEVOPS_WORKFLOW_DEDICATE_MINIO endif @@ -513,6 +521,17 @@ source "workflows/ai/Kconfig" endmenu endif # KDEVOPS_WORKFLOW_ENABLE_AI +config KDEVOPS_WORKFLOW_ENABLE_MINIO + bool + output yaml + default y if KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_MINIO || KDEVOPS_WORKFLOW_DEDICATE_MINIO + +if KDEVOPS_WORKFLOW_ENABLE_MINIO +menu "Configure and run MinIO S3 benchmarks" +source "workflows/minio/Kconfig" +endmenu +endif # KDEVOPS_WORKFLOW_ENABLE_MINIO + config KDEVOPS_WORKFLOW_ENABLE_SSD_STEADY_STATE bool "Attain SSD steady state prior to tests" output yaml diff --git a/playbooks/minio.yml b/playbooks/minio.yml new file mode 100644 index 0000000..bf80bbf --- /dev/null +++ b/playbooks/minio.yml @@ -0,0 +1,53 @@ +--- +# MinIO S3 Storage Benchmarking Playbook + +- name: Install MinIO and setup + hosts: minio + become: true + become_user: root + tags: ['minio_install'] + roles: + - role: minio_install + - role: minio_setup + vars: + minio_container_image: "{{ minio_container_image_string }}" + minio_container_name: "{{ minio_container_name }}" + minio_api_port: "{{ minio_api_port }}" + minio_console_port: "{{ minio_console_port }}" + minio_access_key: "{{ minio_access_key }}" + minio_secret_key: "{{ minio_secret_key }}" + minio_data_path: "{{ minio_data_path }}" + minio_memory_limit: "{{ minio_memory_limit }}" + minio_docker_network: "{{ minio_docker_network_name }}" + +- name: Run MinIO Warp benchmarks + hosts: minio + become: true + become_user: root + tags: ['minio_warp'] + roles: + - role: minio_warp_run + +- name: Uninstall MinIO + hosts: minio + become: true + become_user: root + tags: ['minio_uninstall'] + roles: + - role: minio_uninstall + +- name: Destroy MinIO and cleanup + hosts: minio + become: true + become_user: root + tags: ['minio_destroy'] + roles: + - role: minio_destroy + +- name: Analyze MinIO results + hosts: minio + become: true + become_user: root + tags: ['minio_results'] + roles: + - role: minio_results diff --git a/playbooks/roles/ai_setup/tasks/main.yml b/playbooks/roles/ai_setup/tasks/main.yml index b894c96..899fcee 100644 --- a/playbooks/roles/ai_setup/tasks/main.yml +++ b/playbooks/roles/ai_setup/tasks/main.yml @@ -15,7 +15,6 @@ loop: - "{{ ai_docker_data_path }}" - "{{ ai_docker_etcd_data_path }}" - - "{{ ai_docker_minio_data_path }}" when: ai_milvus_docker | bool become: true @@ -50,24 +49,20 @@ memory: "{{ ai_etcd_memory_limit }}" when: ai_milvus_docker | bool -- name: Start MinIO container - community.docker.docker_container: - name: "{{ ai_minio_container_name }}" - image: "{{ ai_minio_container_image_string }}" - state: started - restart_policy: unless-stopped - networks: - - name: "{{ ai_docker_network_name }}" - ports: - - "{{ ai_minio_api_port }}:9000" - - "{{ ai_minio_console_port }}:9001" - env: - MINIO_ACCESS_KEY: "{{ ai_minio_access_key }}" - MINIO_SECRET_KEY: "{{ ai_minio_secret_key }}" - ansible.builtin.command: server /minio_data --console-address ":9001" - volumes: - - "{{ ai_docker_minio_data_path }}:/minio_data" - memory: "{{ ai_minio_memory_limit }}" +- name: Setup MinIO using shared role + include_role: + name: minio_setup + vars: + minio_container_image: "{{ ai_minio_container_image_string }}" + minio_container_name: "{{ ai_minio_container_name }}" + minio_api_port: "{{ ai_minio_api_port }}" + minio_console_port: "{{ ai_minio_console_port }}" + minio_access_key: "{{ ai_minio_access_key }}" + minio_secret_key: "{{ ai_minio_secret_key }}" + minio_data_path: "{{ ai_docker_minio_data_path }}" + minio_memory_limit: "{{ ai_minio_memory_limit }}" + minio_docker_network: "{{ ai_docker_network_name }}" + minio_create_network: false # Network already created above when: ai_milvus_docker | bool - name: Wait for etcd to be ready @@ -77,13 +72,6 @@ timeout: 60 when: ai_milvus_docker | bool -- name: Wait for MinIO to be ready - ansible.builtin.wait_for: - host: localhost - port: "{{ ai_minio_api_port }}" - timeout: 60 - when: ai_milvus_docker | bool - - name: Start Milvus container community.docker.docker_container: name: "{{ ai_milvus_container_name }}" diff --git a/playbooks/roles/gen_hosts/tasks/main.yml b/playbooks/roles/gen_hosts/tasks/main.yml index d44566a..6cdbdb7 100644 --- a/playbooks/roles/gen_hosts/tasks/main.yml +++ b/playbooks/roles/gen_hosts/tasks/main.yml @@ -179,7 +179,7 @@ name: guestfs_nodes when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ai_enable_multifs_testing|default(false)|bool - ansible_hosts_template.stat.exists @@ -188,7 +188,7 @@ all_generic_nodes: "{{ guestfs_nodes.guestfs_nodes | map(attribute='name') | list }}" when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ai_enable_multifs_testing|default(false)|bool - guestfs_nodes is defined @@ -221,6 +221,21 @@ state: touch mode: "0755" +- name: Generate the Ansible hosts file for a dedicated MinIO setup + tags: ['hosts'] + ansible.builtin.template: + src: "{{ kdevops_hosts_template }}" + dest: "{{ ansible_cfg_inventory }}" + force: true + trim_blocks: True + lstrip_blocks: True + mode: '0644' + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_minio|default(false)|bool + - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool + - name: Verify if final host file exists ansible.builtin.stat: path: "{{ ansible_cfg_inventory }}" diff --git a/playbooks/roles/gen_hosts/templates/workflows/minio.j2 b/playbooks/roles/gen_hosts/templates/workflows/minio.j2 new file mode 100644 index 0000000..42ba326 --- /dev/null +++ b/playbooks/roles/gen_hosts/templates/workflows/minio.j2 @@ -0,0 +1,173 @@ +{# Workflow template for MinIO #} +{% if minio_enable_multifs_testing|default(false)|bool %} +{# Multi-filesystem MinIO configuration #} +[all] +localhost ansible_connection=local +{% for config in minio_enabled_section_types|default([]) %} +{{ config }} +{% endfor %} + +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +[baseline] +{% for config in minio_enabled_section_types|default([]) %} +{% if '-dev' not in config %} +{{ config }} +{% endif %} +{% endfor %} + +[baseline:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% if kdevops_baseline_and_dev %} +[dev] +{% for config in minio_enabled_section_types|default([]) %} +{% if '-dev' in config %} +{{ config }} +{% endif %} +{% endfor %} + +[dev:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +{% endif %} + +[minio] +{% for config in minio_enabled_section_types|default([]) %} +{{ config }} +{% endfor %} + +[minio:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{# Create filesystem-specific groups #} +{% if minio_multifs_xfs_4k_4ks|default(false)|bool %} +[minio-xfs-4k] +{{ kdevops_host_prefix }}-minio-xfs-4k +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-xfs-4k-dev +{% endif %} + +[minio-xfs-4k:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +minio_fstype = "xfs" +minio_xfs_blocksize = 4096 +minio_xfs_sectorsize = 4096 +{% endif %} + +{% if minio_multifs_xfs_16k_4ks|default(false)|bool %} +[minio-xfs-16k] +{{ kdevops_host_prefix }}-minio-xfs-16k +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-xfs-16k-dev +{% endif %} + +[minio-xfs-16k:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +minio_fstype = "xfs" +minio_xfs_blocksize = 16384 +minio_xfs_sectorsize = 4096 +{% endif %} + +{% if minio_multifs_xfs_32k_4ks|default(false)|bool %} +[minio-xfs-32k] +{{ kdevops_host_prefix }}-minio-xfs-32k +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-xfs-32k-dev +{% endif %} + +[minio-xfs-32k:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +minio_fstype = "xfs" +minio_xfs_blocksize = 32768 +minio_xfs_sectorsize = 4096 +{% endif %} + +{% if minio_multifs_xfs_64k_4ks|default(false)|bool %} +[minio-xfs-64k] +{{ kdevops_host_prefix }}-minio-xfs-64k +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-xfs-64k-dev +{% endif %} + +[minio-xfs-64k:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +minio_fstype = "xfs" +minio_xfs_blocksize = 65536 +minio_xfs_sectorsize = 4096 +{% endif %} + +{% if minio_multifs_ext4_4k|default(false)|bool %} +[minio-ext4-4k] +{{ kdevops_host_prefix }}-minio-ext4-4k +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-ext4-4k-dev +{% endif %} + +[minio-ext4-4k:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +minio_fstype = "ext4" +minio_ext4_mkfs_opts = "-F" +{% endif %} + +{% if minio_multifs_ext4_16k_bigalloc|default(false)|bool %} +[minio-ext4-16k-bigalloc] +{{ kdevops_host_prefix }}-minio-ext4-16k-bigalloc +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-ext4-16k-bigalloc-dev +{% endif %} + +[minio-ext4-16k-bigalloc:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +minio_fstype = "ext4" +minio_ext4_mkfs_opts = "-F -O bigalloc -C 16384" +{% endif %} + +{% if minio_multifs_btrfs_default|default(false)|bool %} +[minio-btrfs] +{{ kdevops_host_prefix }}-minio-btrfs +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-btrfs-dev +{% endif %} + +[minio-btrfs:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +minio_fstype = "btrfs" +minio_btrfs_mkfs_opts = "-f" +{% endif %} + +{% else %} +{# Standard single-filesystem MinIO configuration #} +[all] +localhost ansible_connection=local +{{ kdevops_host_prefix }}-minio +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-dev +{% endif %} + +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +[baseline] +{{ kdevops_host_prefix }}-minio + +[baseline:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% if kdevops_baseline_and_dev %} +[dev] +{{ kdevops_host_prefix }}-minio-dev + +[dev:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% endif %} +[minio] +{{ kdevops_host_prefix }}-minio +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-minio-dev +{% endif %} + +[minio:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +{% endif %} diff --git a/playbooks/roles/gen_nodes/tasks/main.yml b/playbooks/roles/gen_nodes/tasks/main.yml index 1ab81d3..9bd9b84 100644 --- a/playbooks/roles/gen_nodes/tasks/main.yml +++ b/playbooks/roles/gen_nodes/tasks/main.yml @@ -681,7 +681,7 @@ mode: '0644' when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ansible_nodes_template.stat.exists - not kdevops_baseline_and_dev - not ai_enable_multifs_testing|default(false)|bool @@ -699,7 +699,7 @@ mode: '0644' when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ansible_nodes_template.stat.exists - kdevops_baseline_and_dev - not ai_enable_multifs_testing|default(false)|bool @@ -742,7 +742,7 @@ ai_multifs_enabled_configs: "{{ (xfs_configs + ext4_configs + btrfs_configs) | unique }}" when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ai_enable_multifs_testing|default(false)|bool - ansible_nodes_template.stat.exists @@ -753,7 +753,7 @@ ai_enabled_section_types: "{{ filesystem_nodes }}" when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ai_enable_multifs_testing|default(false)|bool - ansible_nodes_template.stat.exists - not kdevops_baseline_and_dev @@ -767,7 +767,7 @@ ai_enabled_section_types: "{{ filesystem_nodes | product(['', '-dev']) | map('join') | list }}" when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ai_enable_multifs_testing|default(false)|bool - ansible_nodes_template.stat.exists - kdevops_baseline_and_dev @@ -786,12 +786,128 @@ force: yes when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ai_enable_multifs_testing|default(false)|bool - ansible_nodes_template.stat.exists - ai_enabled_section_types is defined - ai_enabled_section_types | length > 0 +# MinIO S3 Storage Testing workflow nodes + +# Multi-filesystem MinIO configurations +- name: Collect enabled MinIO multi-filesystem configurations + vars: + xfs_configs: >- + {{ + [] + + (['xfs-4k'] if minio_multifs_xfs_4k_4ks|default(false)|bool else []) + + (['xfs-16k'] if minio_multifs_xfs_16k_4ks|default(false)|bool else []) + + (['xfs-32k'] if minio_multifs_xfs_32k_4ks|default(false)|bool else []) + + (['xfs-64k'] if minio_multifs_xfs_64k_4ks|default(false)|bool else []) + }} + ext4_configs: >- + {{ + [] + + (['ext4-4k'] if minio_multifs_ext4_4k|default(false)|bool else []) + + (['ext4-16k-bigalloc'] if minio_multifs_ext4_16k_bigalloc|default(false)|bool else []) + }} + btrfs_configs: >- + {{ + [] + + (['btrfs'] if minio_multifs_btrfs_default|default(false)|bool else []) + }} + set_fact: + minio_multifs_enabled_configs: "{{ (xfs_configs + ext4_configs + btrfs_configs) | unique }}" + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_minio|default(false)|bool + - minio_enable_multifs_testing|default(false)|bool + - ansible_nodes_template.stat.exists + +- name: Create MinIO nodes for each filesystem configuration (no dev) + vars: + filesystem_nodes: "{{ [kdevops_host_prefix + '-minio-'] | product(minio_multifs_enabled_configs | default([])) | map('join') | list }}" + set_fact: + minio_enabled_section_types: "{{ filesystem_nodes }}" + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_minio|default(false)|bool + - minio_enable_multifs_testing|default(false)|bool + - ansible_nodes_template.stat.exists + - not kdevops_baseline_and_dev + - minio_multifs_enabled_configs is defined + - minio_multifs_enabled_configs | length > 0 + +- name: Create MinIO nodes for each filesystem configuration with dev hosts + vars: + filesystem_nodes: "{{ [kdevops_host_prefix + '-minio-'] | product(minio_multifs_enabled_configs | default([])) | map('join') | list }}" + set_fact: + minio_enabled_section_types: "{{ filesystem_nodes | product(['', '-dev']) | map('join') | list }}" + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_minio|default(false)|bool + - minio_enable_multifs_testing|default(false)|bool + - ansible_nodes_template.stat.exists + - kdevops_baseline_and_dev + - minio_multifs_enabled_configs is defined + - minio_multifs_enabled_configs | length > 0 + +- name: Generate the MinIO multi-filesystem kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template + tags: [ 'hosts' ] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ minio_enabled_section_types }}" + all_generic_nodes: "{{ minio_enabled_section_types }}" + ansible.builtin.template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: true + mode: '0644' + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_minio|default(false)|bool + - minio_enable_multifs_testing|default(false)|bool + - ansible_nodes_template.stat.exists + - minio_enabled_section_types is defined + - minio_enabled_section_types | length > 0 + +# Standard MinIO single filesystem nodes +- name: Generate the MinIO kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template + tags: ['hosts'] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ [kdevops_host_prefix + '-minio'] }}" + all_generic_nodes: "{{ [kdevops_host_prefix + '-minio'] }}" + ansible.builtin.template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: true + mode: '0644' + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_minio|default(false)|bool + - ansible_nodes_template.stat.exists + - not kdevops_baseline_and_dev + - not minio_enable_multifs_testing|default(false)|bool + +- name: Generate the MinIO kdevops nodes file with dev hosts using {{ kdevops_nodes_template }} as jinja2 source template + tags: ['hosts'] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ [kdevops_host_prefix + '-minio', kdevops_host_prefix + '-minio-dev'] }}" + all_generic_nodes: "{{ [kdevops_host_prefix + '-minio', kdevops_host_prefix + '-minio-dev'] }}" + ansible.builtin.template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: true + mode: '0644' + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_minio|default(false)|bool + - ansible_nodes_template.stat.exists + - kdevops_baseline_and_dev + - not minio_enable_multifs_testing|default(false)|bool + - name: Get the control host's timezone ansible.builtin.command: "timedatectl show -p Timezone --value" register: kdevops_host_timezone diff --git a/playbooks/roles/minio_destroy/tasks/main.yml b/playbooks/roles/minio_destroy/tasks/main.yml new file mode 100644 index 0000000..078cb13 --- /dev/null +++ b/playbooks/roles/minio_destroy/tasks/main.yml @@ -0,0 +1,34 @@ +--- +- name: Import optional extra_args file + include_vars: "{{ item }}" + ignore_errors: yes + with_items: + - "../extra_vars.yaml" + tags: vars + +- name: Stop and remove MinIO container + community.docker.docker_container: + name: "{{ minio_container_name }}" + state: absent + ignore_errors: yes + +- name: Remove Docker network + community.docker.docker_network: + name: "{{ minio_docker_network_name }}" + state: absent + ignore_errors: yes + +- name: Clean up MinIO data directory + file: + path: "{{ minio_data_path }}" + state: absent + when: minio_warp_enable_cleanup | default(true) | bool + +- name: Clean up temporary Warp results + file: + path: "/tmp/warp-results" + state: absent + +- name: Display MinIO destroy complete + debug: + msg: "MinIO containers and data have been cleaned up" diff --git a/playbooks/roles/minio_install/tasks/main.yml b/playbooks/roles/minio_install/tasks/main.yml new file mode 100644 index 0000000..9ea3d75 --- /dev/null +++ b/playbooks/roles/minio_install/tasks/main.yml @@ -0,0 +1,61 @@ +--- +- name: Import optional extra_args file + include_vars: "{{ item }}" + ignore_errors: yes + with_items: + - "../extra_vars.yaml" + tags: vars + +- name: Install Docker + package: + name: + - docker.io + - python3-docker + state: present + become: yes + +- name: Ensure Docker service is running + systemd: + name: docker + state: started + enabled: yes + become: yes + +- name: Add current user to docker group + user: + name: "{{ ansible_user | default('kdevops') }}" + groups: docker + append: yes + become: yes + +- name: Install MinIO Warp + block: + - name: Download MinIO Warp binary + get_url: + url: "https://github.com/minio/warp/releases/latest/download/warp_Linux_x86_64.tar.gz" + dest: "/tmp/warp_Linux_x86_64.tar.gz" + mode: '0644' + + - name: Extract MinIO Warp + unarchive: + src: "/tmp/warp_Linux_x86_64.tar.gz" + dest: "/tmp" + remote_src: yes + + - name: Install Warp binary + copy: + src: "/tmp/warp" + dest: "/usr/local/bin/warp" + mode: '0755' + owner: root + group: root + remote_src: yes + become: yes + + - name: Clean up downloaded files + file: + path: "{{ item }}" + state: absent + loop: + - "/tmp/warp_Linux_x86_64.tar.gz" + - "/tmp/warp" diff --git a/playbooks/roles/minio_results/tasks/main.yml b/playbooks/roles/minio_results/tasks/main.yml new file mode 100644 index 0000000..7403855 --- /dev/null +++ b/playbooks/roles/minio_results/tasks/main.yml @@ -0,0 +1,86 @@ +--- +- name: Import optional extra_args file + include_vars: "{{ item }}" + ignore_errors: yes + with_items: + - "../extra_vars.yaml" + tags: vars + +- name: Create results analysis script + copy: + content: | + #!/usr/bin/env python3 + import json + import glob + import os + import sys + from pathlib import Path + + def analyze_warp_results(): + results_dir = Path("{{ playbook_dir }}/../workflows/minio/results") + result_files = list(results_dir.glob("warp_benchmark_*.json")) + + if not result_files: + print("No Warp benchmark results found.") + return + + print("MinIO Warp Benchmark Results Summary") + print("=" * 50) + + total_throughput = 0 + total_requests = 0 + for result_file in result_files: + try: + with open(result_file, 'r') as f: + data = json.load(f) + + hostname = result_file.name.split('_')[2] + timestamp = result_file.name.split('_')[3].replace('.json', '') + + # Extract key metrics from Warp results + if 'throughput' in data: + throughput_mbps = data['throughput'].get('average', 0) / (1024 * 1024) + total_throughput += throughput_mbps + + if 'requests' in data: + req_per_sec = data['requests'].get('average', 0) + total_requests += req_per_sec + + print(f"\nHost: {hostname}") + print(f"Timestamp: {timestamp}") + print(f"Throughput: {throughput_mbps:.2f} MB/s") + print(f"Requests/sec: {req_per_sec:.2f}") + + if 'latency' in data: + avg_latency = data['latency'].get('average', 0) + print(f"Average Latency: {avg_latency:.2f} ms") + + except Exception as e: + print(f"Error processing {result_file}: {e}") + print("\n" + "=" * 50) + print(f"Total Throughput: {total_throughput:.2f} MB/s") + print(f"Total Requests/sec: {total_requests:.2f}") + + if __name__ == "__main__": + analyze_warp_results() + dest: "/tmp/analyze_minio_results.py" + mode: '0755' + delegate_to: localhost + run_once: true + +- name: Run results analysis + command: python3 /tmp/analyze_minio_results.py + register: analysis_output + delegate_to: localhost + run_once: true + +- name: Display analysis results + debug: + var: analysis_output.stdout_lines + +- name: Create results summary file + copy: + content: "{{ analysis_output.stdout }}" + dest: "{{ playbook_dir }}/../workflows/minio/results/benchmark_summary.txt" + delegate_to: localhost + run_once: true diff --git a/playbooks/roles/minio_setup/defaults/main.yml b/playbooks/roles/minio_setup/defaults/main.yml new file mode 100644 index 0000000..1403010 --- /dev/null +++ b/playbooks/roles/minio_setup/defaults/main.yml @@ -0,0 +1,16 @@ +--- +# MinIO Docker container defaults +minio_container_image: "minio/minio:RELEASE.2023-03-20T20-16-18Z" +minio_container_name: "minio-server" +minio_api_port: 9000 +minio_console_port: 9001 +minio_access_key: "minioadmin" +minio_secret_key: "minioadmin" +minio_data_path: "/data/minio" +minio_memory_limit: "2g" +minio_docker_network: "minio-network" + +# MinIO service configuration +minio_enable: true +minio_create_network: true +minio_wait_for_ready: true diff --git a/playbooks/roles/minio_setup/tasks/main.yml b/playbooks/roles/minio_setup/tasks/main.yml new file mode 100644 index 0000000..db7e3d6 --- /dev/null +++ b/playbooks/roles/minio_setup/tasks/main.yml @@ -0,0 +1,100 @@ +--- +- name: Import optional extra_args file + include_vars: "{{ item }}" + ignore_errors: yes + with_items: + - "../extra_vars.yaml" + tags: vars + +- name: Setup dedicated MinIO storage filesystem if configured + when: + - minio_storage_enable | default(false) | bool + - minio_device is defined + block: + - name: Prepare filesystem mkfs options + set_fact: + minio_mkfs_opts: >- + {%- if minio_fstype == "xfs" -%} + -L miniostorage -f -b size={{ minio_xfs_blocksize | default(4096) }} -s size={{ minio_xfs_sectorsize | default(4096) }} {{ minio_xfs_mkfs_opts | default('') }} + {%- elif minio_fstype == "btrfs" -%} + -L miniostorage {{ minio_btrfs_mkfs_opts | default('-f') }} + {%- elif minio_fstype == "ext4" -%} + -L miniostorage {{ minio_ext4_mkfs_opts | default('-F') }} + {%- elif minio_fstype == "bcachefs" -%} + --label=miniostorage {{ minio_bcachefs_mkfs_opts | default('-f') }} + {%- else -%} + -L miniostorage -f + {%- endif -%} + + - name: Create MinIO storage filesystem + include_role: + name: create_partition + vars: + disk_setup_device: "{{ minio_device }}" + disk_setup_fstype: "{{ minio_fstype | default('xfs') }}" + disk_setup_label: "miniostorage" + disk_setup_fs_opts: "{{ minio_mkfs_opts }}" + disk_setup_path: "{{ minio_mount_point | default('/data/minio') }}" + disk_setup_user: "root" + disk_setup_group: "root" + +- name: Create MinIO data directory + file: + path: "{{ minio_data_path | default('/data/minio') }}" + state: directory + mode: '0755' + when: + - not (minio_storage_enable | default(false) | bool) + become: true + +- name: Check filesystem type for MinIO data path + shell: df -T "{{ minio_data_path }}" | tail -1 | awk '{print $2}' + register: minio_fs_type + changed_when: false + +- name: Get filesystem details + shell: | + df -h "{{ minio_data_path }}" | tail -1 + register: minio_fs_details + changed_when: false + +- name: Display filesystem information + debug: + msg: | + MinIO Storage Configuration: + Data Path: {{ minio_data_path }} + Filesystem: {{ minio_fs_type.stdout }} + Storage Details: {{ minio_fs_details.stdout }} + +- name: Create Docker network for MinIO + community.docker.docker_network: + name: "{{ minio_docker_network }}" + state: present + when: minio_enable | bool and minio_create_network | bool + +- name: Start MinIO container + community.docker.docker_container: + name: "{{ minio_container_name }}" + image: "{{ minio_container_image }}" + state: started + restart_policy: unless-stopped + networks: + - name: "{{ minio_docker_network }}" + ports: + - "{{ minio_api_port }}:9000" + - "{{ minio_console_port }}:9001" + env: + MINIO_ACCESS_KEY: "{{ minio_access_key }}" + MINIO_SECRET_KEY: "{{ minio_secret_key }}" + command: server /minio_data --console-address ":9001" + volumes: + - "{{ minio_data_path }}:/minio_data" + memory: "{{ minio_memory_limit }}" + when: minio_enable | bool + +- name: Wait for MinIO to be ready + wait_for: + host: localhost + port: "{{ minio_api_port }}" + timeout: 60 + when: minio_enable | bool and minio_wait_for_ready | bool diff --git a/playbooks/roles/minio_uninstall/tasks/main.yml b/playbooks/roles/minio_uninstall/tasks/main.yml new file mode 100644 index 0000000..bea1543 --- /dev/null +++ b/playbooks/roles/minio_uninstall/tasks/main.yml @@ -0,0 +1,17 @@ +--- +- name: Import optional extra_args file + include_vars: "{{ item }}" + ignore_errors: yes + with_items: + - "../extra_vars.yaml" + tags: vars + +- name: Stop MinIO container + community.docker.docker_container: + name: "{{ minio_container_name }}" + state: stopped + ignore_errors: yes + +- name: Display MinIO uninstallation complete + debug: + msg: "MinIO container stopped" diff --git a/playbooks/roles/minio_warp_run/tasks/main.yml b/playbooks/roles/minio_warp_run/tasks/main.yml new file mode 100644 index 0000000..415d355 --- /dev/null +++ b/playbooks/roles/minio_warp_run/tasks/main.yml @@ -0,0 +1,249 @@ +--- +- name: Import optional extra_args file + include_vars: "{{ item }}" + ignore_errors: yes + with_items: + - "../extra_vars.yaml" + tags: vars + +- name: Create Warp results directory on remote host + file: + path: "/tmp/warp-results" + state: directory + mode: '0755' + +- name: Ensure local results directory exists with proper permissions + block: + - name: Create local results directory + file: + path: "{{ playbook_dir }}/../workflows/minio/results" + state: directory + mode: '0755' + delegate_to: localhost + run_once: true + become: no + rescue: + - name: Fix results directory permissions if needed + file: + path: "{{ playbook_dir }}/../workflows/minio/results" + state: directory + mode: '0755' + owner: "{{ lookup('env', 'USER') }}" + group: "{{ lookup('env', 'USER') }}" + delegate_to: localhost + run_once: true + become: yes + + +- name: Wait for MinIO to be fully ready + wait_for: + host: localhost + port: "{{ minio_api_port }}" + timeout: 120 + retries: 3 + delay: 10 + +- name: Check if Warp is installed + command: which warp + register: warp_check + failed_when: false + changed_when: false + +- name: Verify Warp installation + fail: + msg: "MinIO Warp is not installed. Please run 'make minio-install' first." + when: warp_check.rc != 0 + +- name: Create Warp configuration file + template: + src: warp_config.json.j2 + dest: "/tmp/warp_config.json" + mode: '0644' + +- name: Set MinIO endpoint URL + set_fact: + minio_endpoint: "localhost:{{ minio_api_port }}" + +- name: Display Warp version + command: warp --version + register: warp_version + changed_when: false + +- name: Show Warp version + debug: + msg: "MinIO Warp version: {{ warp_version.stdout }}" + +- name: Calculate benchmark timeout + set_fact: + # Parse duration and add 10 minutes buffer + benchmark_timeout: >- + {%- set duration_str = minio_warp_duration | string -%} + {%- if 's' in duration_str -%} + {{ (duration_str | replace('s','') | int) + 600 }} + {%- elif 'm' in duration_str -%} + {{ (duration_str | replace('m','') | int * 60) + 600 }} + {%- elif 'h' in duration_str -%} + {{ (duration_str | replace('h','') | int * 3600) + 600 }} + {%- else -%} + {{ 2400 }} + {%- endif -%} + +- name: Copy comprehensive benchmark script + copy: + src: "{{ playbook_dir }}/../workflows/minio/scripts/run_benchmark_suite.sh" + dest: "/tmp/run_benchmark_suite.sh" + mode: '0755' + when: minio_warp_run_comprehensive_suite | default(false) + +- name: Display benchmark configuration + debug: + msg: | + Comprehensive suite: {{ minio_warp_run_comprehensive_suite | default(false) }} + Duration: {{ minio_warp_duration }} + Timeout: {{ benchmark_timeout }} seconds + when: minio_warp_run_comprehensive_suite | default(false) + +- name: Run comprehensive benchmark suite + shell: | + set -x # Enable debug output + echo "Starting comprehensive benchmark suite" + echo "Duration parameter: {{ minio_warp_duration }}" + /tmp/run_benchmark_suite.sh \ + "{{ minio_endpoint }}" \ + "{{ minio_access_key }}" \ + "{{ minio_secret_key }}" \ + "{{ minio_warp_duration }}" + EXIT_CODE=$? + echo "Benchmark suite completed with exit code: $EXIT_CODE" + exit $EXIT_CODE + args: + executable: /bin/bash + register: suite_output + when: minio_warp_run_comprehensive_suite | default(false) + async: "{{ benchmark_timeout | default(3600) | int }}" # Use calculated timeout or 1 hour default + poll: 30 + +- name: Display comprehensive suite output + debug: + msg: | + Suite completed: {{ suite_output is defined }} + Exit code: {{ suite_output.rc | default('N/A') }} + Output: {{ suite_output.stdout | default('No output') | truncate(500) }} + when: minio_warp_run_comprehensive_suite | default(false) + +- name: Debug - Show which path we're taking + debug: + msg: | + Comprehensive suite enabled: {{ minio_warp_run_comprehensive_suite | default(false) }} + Duration: {{ minio_warp_duration }} + Benchmark timeout: {{ benchmark_timeout }} seconds + +- name: Set timestamp for consistent filename + set_fact: + warp_timestamp: "{{ ansible_date_time.epoch }}" + when: not (minio_warp_run_comprehensive_suite | default(false)) + +- name: Run MinIO Warp single benchmark with JSON output + shell: | + echo "=== Starting single benchmark ===" + echo "Duration: {{ minio_warp_duration }}" + echo "Full command:" + OUTPUT_FILE="/tmp/warp-results/warp_benchmark_{{ ansible_hostname }}_{{ warp_timestamp }}.json" + + # Show the actual command being run + set -x + # IMPORTANT: --autoterm with --objects makes warp stop after N objects, ignoring --duration! + # For duration-based tests, do not use --autoterm + time warp {{ minio_warp_benchmark_type }} \ + --host="{{ minio_endpoint }}" \ + --access-key="{{ minio_access_key }}" \ + --secret-key="{{ minio_secret_key }}" \ + --bucket="{{ minio_warp_bucket_name }}" \ + --duration="{{ minio_warp_duration }}" \ + --concurrent="{{ minio_warp_concurrent_requests }}" \ + --obj.size="{{ minio_warp_object_size }}" \ + {% if minio_warp_enable_web_ui|default(false) %}--warp-client="{{ ansible_default_ipv4.address }}:{{ minio_warp_web_ui_port|default(7762) }}"{% endif %} \ + --noclear \ + --json > "$OUTPUT_FILE" 2>&1 + RESULT=$? + set +x + + echo "=== Benchmark completed with exit code: $RESULT ===" + echo "=== Output file size: $(ls -lh $OUTPUT_FILE 2>/dev/null | awk '{print \$5}') ===" + + # Check if file was created + if [ -f "$OUTPUT_FILE" ]; then + echo "Results saved to: $OUTPUT_FILE" + ls -la "$OUTPUT_FILE" + else + echo "Warning: Results file not created" + fi + exit $RESULT + args: + executable: /bin/bash + environment: + WARP_ACCESS_KEY: "{{ minio_access_key }}" + WARP_SECRET_KEY: "{{ minio_secret_key }}" + register: warp_output + async: "{{ benchmark_timeout | int }}" + poll: 30 + when: not (minio_warp_run_comprehensive_suite | default(false)) + +- name: Display benchmark completion + debug: + msg: "MinIO Warp benchmark completed on {{ ansible_hostname }}" + when: (warp_output is defined and warp_output.rc | default(1) == 0) or (suite_output is defined and suite_output.rc | default(1) == 0) + +- name: Check if results file exists + stat: + path: "/tmp/warp-results/warp_benchmark_{{ ansible_hostname }}_{{ warp_timestamp }}.json" + register: results_file + when: warp_timestamp is defined + +- name: Display results file status + debug: + msg: "Results file exists: {{ results_file.stat.exists }}, Size: {{ results_file.stat.size | default(0) }} bytes" + when: results_file is defined + +- name: Copy results to local system + fetch: + src: "/tmp/warp-results/warp_benchmark_{{ ansible_hostname }}_{{ warp_timestamp }}.json" + dest: "{{ playbook_dir }}/../workflows/minio/results/" + flat: yes + become: no + when: results_file.stat.exists | default(false) + +- name: Generate graphs and HTML report + command: "python3 {{ playbook_dir }}/../workflows/minio/scripts/generate_warp_report.py {{ playbook_dir }}/../workflows/minio/results/" + delegate_to: localhost + run_once: true + become: no + when: results_file.stat.exists | default(false) + ignore_errors: yes + +- name: Save benchmark output as fallback + copy: + content: | + MinIO Warp Benchmark Results + ============================ + Host: {{ ansible_hostname }} + Timestamp: {{ warp_timestamp | default('unknown') }} + + Debug Output: + {{ warp_debug.stdout | default('No debug output') }} + {{ warp_debug.stderr | default('') }} + + Full Benchmark Output: + {{ warp_output.stdout | default('No benchmark output - debug run failed') }} + + Error Output (if any): + {{ warp_output.stderr | default('No errors') }} + dest: "/tmp/warp-results/warp_fallback_{{ ansible_hostname }}_{{ warp_timestamp | default(ansible_date_time.epoch) }}.txt" + when: warp_debug is defined + +- name: Copy fallback results + fetch: + src: "/tmp/warp-results/warp_fallback_{{ ansible_hostname }}_{{ warp_timestamp | default(ansible_date_time.epoch) }}.txt" + dest: "{{ playbook_dir }}/../workflows/minio/results/" + flat: yes + when: warp_debug is defined and not (results_file.stat.exists | default(false)) diff --git a/playbooks/roles/minio_warp_run/templates/warp_config.json.j2 b/playbooks/roles/minio_warp_run/templates/warp_config.json.j2 new file mode 100644 index 0000000..d5c4dc8 --- /dev/null +++ b/playbooks/roles/minio_warp_run/templates/warp_config.json.j2 @@ -0,0 +1,14 @@ +{ + "benchmark": "{{ minio_warp_benchmark_type }}", + "endpoint": "http://localhost:{{ minio_api_port }}", + "access_key": "{{ minio_access_key }}", + "secret_key": "{{ minio_secret_key }}", + "bucket": "{{ minio_warp_bucket_name }}", + "duration": "{{ minio_warp_duration }}", + "concurrent": {{ minio_warp_concurrent_requests }}, + "object_size": "{{ minio_warp_object_size }}", + "objects": {{ minio_warp_objects_per_request }}, + "auto_terminate": {{ minio_warp_auto_terminate | lower }}, + "cleanup": {{ minio_warp_enable_cleanup | lower }}, + "output_format": "{{ minio_warp_output_format }}" +} diff --git a/workflows/Makefile b/workflows/Makefile index fe35707..ee90227 100644 --- a/workflows/Makefile +++ b/workflows/Makefile @@ -70,6 +70,10 @@ ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_AI)) include workflows/ai/Makefile endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_AI == y +ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO)) +include workflows/minio/Makefile +endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_MINIO == y + ANSIBLE_EXTRA_ARGS += $(WORKFLOW_ARGS) ANSIBLE_EXTRA_ARGS_SEPARATED += $(WORKFLOW_ARGS_SEPARATED) ANSIBLE_EXTRA_ARGS_DIRECT += $(WORKFLOW_ARGS_DIRECT) diff --git a/workflows/minio/Kconfig b/workflows/minio/Kconfig new file mode 100644 index 0000000..2af12fc --- /dev/null +++ b/workflows/minio/Kconfig @@ -0,0 +1,23 @@ +if KDEVOPS_WORKFLOW_ENABLE_MINIO + +menu "MinIO S3 Storage Testing" + +config KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP + bool "Enable MinIO Warp benchmarking" + default y + help + Enable MinIO Warp for S3 storage benchmarking. Warp provides + comprehensive S3 API performance testing with multiple benchmark + types including GET, PUT, DELETE, LIST, and MULTIPART operations. + +if KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP + +source "workflows/minio/Kconfig.docker" +source "workflows/minio/Kconfig.storage" +source "workflows/minio/Kconfig.warp" + +endif # KDEVOPS_WORKFLOW_ENABLE_MINIO_WARP + +endmenu + +endif # KDEVOPS_WORKFLOW_ENABLE_MINIO diff --git a/workflows/minio/Kconfig.docker b/workflows/minio/Kconfig.docker new file mode 100644 index 0000000..3a33719 --- /dev/null +++ b/workflows/minio/Kconfig.docker @@ -0,0 +1,66 @@ +config MINIO_CONTAINER_IMAGE_STRING + string "MinIO container image" + output yaml + default "minio/minio:RELEASE.2024-01-16T16-07-38Z" + help + The MinIO container image to use for S3 storage benchmarking. + Using a recent stable release with performance improvements. + +config MINIO_CONTAINER_NAME + string "The local MinIO container name" + default "minio-warp-server" + output yaml + help + Set the name for the MinIO Docker container. + +config MINIO_ACCESS_KEY + string "MinIO access key" + output yaml + default "minioadmin" + help + Access key for MinIO S3 API access. + +config MINIO_SECRET_KEY + string "MinIO secret key" + output yaml + default "minioadmin" + help + Secret key for MinIO S3 API access. + +config MINIO_DATA_PATH + string "Host path for MinIO data storage" + output yaml + default "/data/minio" + help + Directory on the host where MinIO data will be persisted. + If using dedicated storage, this will be the mount point. + Otherwise, uses the existing filesystem at this path. + +config MINIO_DOCKER_NETWORK_NAME + string "Docker network name" + output yaml + default "minio-warp-network" + help + Name of the Docker network to create for MinIO containers. + +config MINIO_API_PORT + int "MinIO API port" + output yaml + default "9000" + help + Port for MinIO S3 API access. + +config MINIO_CONSOLE_PORT + int "MinIO console port" + output yaml + default "9001" + help + Port for MinIO web console access. + +config MINIO_MEMORY_LIMIT + string "MinIO container memory limit" + output yaml + default "4g" + help + Memory limit for the MinIO container. Adjust based on + your system resources and workload requirements. diff --git a/workflows/minio/Kconfig.storage b/workflows/minio/Kconfig.storage new file mode 100644 index 0000000..8815912 --- /dev/null +++ b/workflows/minio/Kconfig.storage @@ -0,0 +1,364 @@ +menu "MinIO Storage Configuration" + +# CLI override support for WARP_DEVICE +config MINIO_DEVICE_SET_BY_CLI + bool + output yaml + default $(shell, scripts/check-cli-set-var.sh WARP_DEVICE) + +config MINIO_STORAGE_ENABLE + bool "Enable dedicated MinIO storage device" + default y + output yaml + help + Configure a dedicated storage device for MinIO data storage. + This allows testing MinIO performance on different filesystems + and configurations by creating and mounting a dedicated partition. + + When enabled, MinIO data will be stored on a dedicated device + and filesystem optimized for S3 workloads. + +if MINIO_STORAGE_ENABLE + +config MINIO_DEVICE + string "Device to use for MinIO storage" + output yaml + default $(shell, ./scripts/append-makefile-vars.sh $(WARP_DEVICE)) if MINIO_DEVICE_SET_BY_CLI + default "/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_NVME + default "/dev/disk/by-id/virtio-kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_VIRTIO + default "/dev/disk/by-id/ata-QEMU_HARDDISK_kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_IDE + default "/dev/nvme2n1" if TERRAFORM_AWS_INSTANCE_M5AD_2XLARGE + default "/dev/nvme2n1" if TERRAFORM_AWS_INSTANCE_M5AD_4XLARGE + default "/dev/nvme1n1" if TERRAFORM_GCE + default "/dev/sdd" if TERRAFORM_AZURE + default TERRAFORM_OCI_SPARSE_VOLUME_DEVICE_FILE_NAME if TERRAFORM_OCI + help + The device to use for MinIO storage. This device will be + formatted and mounted to store MinIO S3 data. + + Can be overridden with WARP_DEVICE environment variable: + make defconfig-minio-warp-xfs-16k WARP_DEVICE=/dev/nvme4n1 + +config MINIO_MOUNT_POINT + string "Mount point for MinIO storage" + output yaml + default "/data/minio" + help + The path where the MinIO storage filesystem will be mounted. + MinIO will store all S3 data under this path. + +choice + prompt "MinIO storage filesystem" + default MINIO_FSTYPE_XFS + +config MINIO_FSTYPE_XFS + bool "XFS" + help + Use XFS filesystem for MinIO storage. XFS provides excellent + performance for large files and is recommended for production + MinIO deployments. Supports various block sizes for testing + large block size (LBS) configurations. + +config MINIO_FSTYPE_BTRFS + bool "Btrfs" + help + Use Btrfs filesystem for MinIO storage. Btrfs provides + advanced features like snapshots and compression, which can + be beneficial for S3 storage management. + +config MINIO_FSTYPE_EXT4 + bool "ext4" + help + Use ext4 filesystem for MinIO storage. Ext4 is a mature + and reliable filesystem with good all-around performance. + +config MINIO_FSTYPE_BCACHEFS + bool "bcachefs" + help + Use bcachefs filesystem for MinIO storage. Bcachefs is a + modern filesystem with advanced features like compression, + encryption, and caching. + +endchoice + +config MINIO_FSTYPE + string + output yaml + default "xfs" if MINIO_FSTYPE_XFS + default "btrfs" if MINIO_FSTYPE_BTRFS + default "ext4" if MINIO_FSTYPE_EXT4 + default "bcachefs" if MINIO_FSTYPE_BCACHEFS + +if MINIO_FSTYPE_XFS + +choice + prompt "XFS block size configuration" + default MINIO_XFS_BLOCKSIZE_4K + +config MINIO_XFS_BLOCKSIZE_4K + bool "4K block size (default)" + help + Use 4K (4096 bytes) block size. This is the default and most + compatible configuration. + +config MINIO_XFS_BLOCKSIZE_8K + bool "8K block size" + help + Use 8K (8192 bytes) block size for improved performance with + larger I/O operations. + +config MINIO_XFS_BLOCKSIZE_16K + bool "16K block size (LBS)" + help + Use 16K (16384 bytes) block size. This is a large block size + configuration that may require kernel LBS support. + +config MINIO_XFS_BLOCKSIZE_32K + bool "32K block size (LBS)" + help + Use 32K (32768 bytes) block size. This is a large block size + configuration that requires kernel LBS support. + +config MINIO_XFS_BLOCKSIZE_64K + bool "64K block size (LBS)" + help + Use 64K (65536 bytes) block size. This is the maximum XFS block + size and requires kernel LBS support. + +endchoice + +config MINIO_XFS_BLOCKSIZE + int + output yaml + default 4096 if MINIO_XFS_BLOCKSIZE_4K + default 8192 if MINIO_XFS_BLOCKSIZE_8K + default 16384 if MINIO_XFS_BLOCKSIZE_16K + default 32768 if MINIO_XFS_BLOCKSIZE_32K + default 65536 if MINIO_XFS_BLOCKSIZE_64K + +choice + prompt "XFS sector size" + default MINIO_XFS_SECTORSIZE_4K + +config MINIO_XFS_SECTORSIZE_4K + bool "4K sector size (default)" + help + Use 4K (4096 bytes) sector size. This is the standard + configuration for most modern drives. + +config MINIO_XFS_SECTORSIZE_512 + bool "512 byte sector size" + depends on MINIO_XFS_BLOCKSIZE_4K + help + Use legacy 512 byte sector size. Only available with 4K block size. + +config MINIO_XFS_SECTORSIZE_8K + bool "8K sector size" + depends on MINIO_XFS_BLOCKSIZE_8K || MINIO_XFS_BLOCKSIZE_16K || MINIO_XFS_BLOCKSIZE_32K || MINIO_XFS_BLOCKSIZE_64K + help + Use 8K (8192 bytes) sector size. Requires block size >= 8K. + +config MINIO_XFS_SECTORSIZE_16K + bool "16K sector size (LBS)" + depends on MINIO_XFS_BLOCKSIZE_16K || MINIO_XFS_BLOCKSIZE_32K || MINIO_XFS_BLOCKSIZE_64K + help + Use 16K (16384 bytes) sector size. Requires block size >= 16K + and kernel LBS support. + +config MINIO_XFS_SECTORSIZE_32K + bool "32K sector size (LBS)" + depends on MINIO_XFS_BLOCKSIZE_32K || MINIO_XFS_BLOCKSIZE_64K + help + Use 32K (32768 bytes) sector size. Requires block size >= 32K + and kernel LBS support. + +endchoice + +config MINIO_XFS_SECTORSIZE + int + output yaml + default 512 if MINIO_XFS_SECTORSIZE_512 + default 4096 if MINIO_XFS_SECTORSIZE_4K + default 8192 if MINIO_XFS_SECTORSIZE_8K + default 16384 if MINIO_XFS_SECTORSIZE_16K + default 32768 if MINIO_XFS_SECTORSIZE_32K + +config MINIO_XFS_MKFS_OPTS + string "Additional XFS mkfs options for MinIO storage" + output yaml + default "" + help + Additional options to pass to mkfs.xfs when creating the MinIO + storage filesystem. Block and sector sizes are configured above. + +endif # MINIO_FSTYPE_XFS + +config MINIO_BTRFS_MKFS_OPTS + string "Btrfs mkfs options for MinIO storage" + output yaml + default "-f" + depends on MINIO_FSTYPE_BTRFS + help + Options to pass to mkfs.btrfs when creating the MinIO storage + filesystem. + +config MINIO_EXT4_MKFS_OPTS + string "ext4 mkfs options for MinIO storage" + output yaml + default "-F" + depends on MINIO_FSTYPE_EXT4 + help + Options to pass to mkfs.ext4 when creating the MinIO storage + filesystem. + +config MINIO_BCACHEFS_MKFS_OPTS + string "bcachefs mkfs options for MinIO storage" + output yaml + default "-f" + depends on MINIO_FSTYPE_BCACHEFS + help + Options to pass to mkfs.bcachefs when creating the MinIO storage + filesystem. + +endif # MINIO_STORAGE_ENABLE + +# Multi-filesystem configuration when not skipping bringup +if !KDEVOPS_USE_DECLARED_HOSTS && MINIO_STORAGE_ENABLE + +config MINIO_ENABLE_MULTIFS_TESTING + bool "Enable multi-filesystem testing" + default n + output yaml + help + Enable testing the same MinIO workload across multiple filesystem + configurations. This allows comparing S3 performance characteristics + between different filesystems and their configurations. + + When enabled, multiple nodes will be created with different + filesystem configurations for comprehensive performance analysis. + +if MINIO_ENABLE_MULTIFS_TESTING + +config MINIO_MULTIFS_TEST_XFS + bool "Test XFS configurations" + default y + output yaml + help + Enable testing MinIO workloads on XFS filesystem with different + block size configurations. + +if MINIO_MULTIFS_TEST_XFS + +menu "XFS configuration profiles" + +config MINIO_MULTIFS_XFS_4K_4KS + bool "XFS 4k block size - 4k sector size" + default y + output yaml + help + Test MinIO workloads on XFS with 4k filesystem block size + and 4k sector size. This is the most common configuration + and provides good performance for most S3 workloads. + +config MINIO_MULTIFS_XFS_16K_4KS + bool "XFS 16k block size - 4k sector size" + default n + output yaml + help + Test MinIO workloads on XFS with 16k filesystem block size + and 4k sector size. Larger block sizes can improve performance + for large object storage patterns. + +config MINIO_MULTIFS_XFS_32K_4KS + bool "XFS 32k block size - 4k sector size" + default n + output yaml + help + Test MinIO workloads on XFS with 32k filesystem block size + and 4k sector size. Even larger block sizes can provide + benefits for very large S3 objects. + +config MINIO_MULTIFS_XFS_64K_4KS + bool "XFS 64k block size - 4k sector size" + default n + output yaml + help + Test MinIO workloads on XFS with 64k filesystem block size + and 4k sector size. Maximum supported block size for XFS, + optimized for very large object operations. + +endmenu + +endif # MINIO_MULTIFS_TEST_XFS + +config MINIO_MULTIFS_TEST_EXT4 + bool "Test ext4 configurations" + default y + output yaml + help + Enable testing MinIO workloads on ext4 filesystem with different + configurations including bigalloc options. + +if MINIO_MULTIFS_TEST_EXT4 + +menu "ext4 configuration profiles" + +config MINIO_MULTIFS_EXT4_4K + bool "ext4 4k block size" + default y + output yaml + help + Test MinIO workloads on ext4 with standard 4k block size. + This is the default ext4 configuration. + +config MINIO_MULTIFS_EXT4_16K_BIGALLOC + bool "ext4 16k bigalloc" + default n + output yaml + help + Test MinIO workloads on ext4 with 16k bigalloc enabled. + Bigalloc reduces metadata overhead and can improve + performance for large S3 objects. + +endmenu + +endif # MINIO_MULTIFS_TEST_EXT4 + +config MINIO_MULTIFS_TEST_BTRFS + bool "Test btrfs configurations" + default y + output yaml + help + Enable testing MinIO workloads on btrfs filesystem with + common default configuration profile. + +if MINIO_MULTIFS_TEST_BTRFS + +menu "btrfs configuration profiles" + +config MINIO_MULTIFS_BTRFS_DEFAULT + bool "btrfs default profile" + default y + output yaml + help + Test MinIO workloads on btrfs with default configuration. + This includes modern defaults with free-space-tree and + no-holes features enabled. + +endmenu + +endif # MINIO_MULTIFS_TEST_BTRFS + +config MINIO_MULTIFS_RESULTS_DIR + string "Multi-filesystem results directory" + output yaml + default "/data/minio-multifs-benchmark" + help + Directory where multi-filesystem test results and logs will be stored. + Each filesystem configuration will have its own subdirectory. + +endif # MINIO_ENABLE_MULTIFS_TESTING + +endif # !SKIP_BRINGUP && MINIO_STORAGE_ENABLE + +endmenu diff --git a/workflows/minio/Kconfig.warp b/workflows/minio/Kconfig.warp new file mode 100644 index 0000000..6a8fdb9 --- /dev/null +++ b/workflows/minio/Kconfig.warp @@ -0,0 +1,141 @@ +menu "MinIO Warp benchmark configuration" + +config MINIO_WARP_RUN_COMPREHENSIVE_SUITE + bool "Run comprehensive benchmark suite" + default y + output yaml + help + Run a complete suite of benchmarks including mixed, GET, PUT, DELETE, + LIST operations with various object sizes and concurrency levels. + This provides the most thorough performance analysis. + +if !MINIO_WARP_RUN_COMPREHENSIVE_SUITE + +choice + prompt "Warp benchmark type" + default MINIO_WARP_BENCHMARK_MIXED + help + Select the primary benchmark type for MinIO Warp testing. + +config MINIO_WARP_BENCHMARK_MIXED + bool "Mixed workload (GET/PUT/DELETE)" + help + Run a mixed workload benchmark combining GET, PUT, and DELETE operations + to simulate realistic S3 usage patterns. + +config MINIO_WARP_BENCHMARK_GET + bool "GET operations (download)" + help + Focus on download performance testing with GET operations. + +config MINIO_WARP_BENCHMARK_PUT + bool "PUT operations (upload)" + help + Focus on upload performance testing with PUT operations. + +config MINIO_WARP_BENCHMARK_DELETE + bool "DELETE operations" + help + Test object deletion performance. + +config MINIO_WARP_BENCHMARK_LIST + bool "LIST operations" + help + Test bucket and object listing performance. + +config MINIO_WARP_BENCHMARK_MULTIPART + bool "MULTIPART upload" + help + Test large file upload performance using multipart uploads. + +endchoice + +endif # !MINIO_WARP_RUN_COMPREHENSIVE_SUITE + +config MINIO_WARP_BENCHMARK_TYPE + string "Benchmark type to run" + output yaml + default "mixed" + +config MINIO_WARP_DURATION + string "Benchmark duration" + output yaml + default "5m" + help + Duration for each benchmark run. Examples: 30s, 5m, 1h. + Longer durations provide more stable results but take more time. + +config MINIO_WARP_CONCURRENT_REQUESTS + int "Concurrent requests" + output yaml + default 10 + range 1 1000 + help + Number of concurrent requests to send to MinIO. + Higher values increase load but may overwhelm the system. + +config MINIO_WARP_OBJECT_SIZE + string "Object size for testing" + output yaml + default "1MB" + help + Size of objects to use in benchmarks. Examples: 1KB, 1MB, 10MB. + Larger objects test throughput, smaller objects test IOPS. + +config MINIO_WARP_OBJECTS_PER_REQUEST + int "Objects per request" + output yaml + default 100 + range 1 10000 + help + Number of objects to use in the benchmark. + More objects provide better statistical accuracy. + +config MINIO_WARP_BUCKET_NAME + string "S3 bucket name for testing" + output yaml + default "warp-benchmark-bucket" + help + Name of the S3 bucket to create and use for benchmarking. + +config MINIO_WARP_AUTO_TERMINATE + bool "Auto-terminate when results stabilize" + output yaml + default y + help + Automatically terminate the benchmark when performance results + have stabilized, potentially reducing test time. + +config MINIO_WARP_ENABLE_CLEANUP + bool "Clean up test data after benchmarks" + output yaml + default y + help + Remove test objects and buckets after benchmarking completes. + Disable if you want to inspect test data afterwards. + +config MINIO_WARP_OUTPUT_FORMAT + string "Output format" + output yaml + default "json" + help + Output format for benchmark results. Options: json, csv, text. + JSON format provides the most detailed metrics for analysis. + +config MINIO_WARP_ENABLE_WEB_UI + bool "Enable Warp Web UI for real-time monitoring" + output yaml + default n + help + Enable the Warp web interface for real-time benchmark monitoring. + Access the UI at http://localhost:7762 during benchmark runs. + +config MINIO_WARP_WEB_UI_PORT + int "Web UI port" + output yaml + default 7762 + depends on MINIO_WARP_ENABLE_WEB_UI + help + Port for the Warp web interface. + +endmenu diff --git a/workflows/minio/Makefile b/workflows/minio/Makefile new file mode 100644 index 0000000..c543ed3 --- /dev/null +++ b/workflows/minio/Makefile @@ -0,0 +1,76 @@ +MINIO_DATA_TARGET := minio +MINIO_DATA_TARGET_INSTALL := minio-install +MINIO_DATA_TARGET_UNINSTALL := minio-uninstall +MINIO_DATA_TARGET_DESTROY := minio-destroy +MINIO_DATA_TARGET_RUN := minio-warp +MINIO_DATA_TARGET_RESULTS := minio-results + +MINIO_PLAYBOOK := playbooks/minio.yml + +HELP_TARGETS += minio-help + +$(MINIO_DATA_TARGET): $(ANSIBLE_INVENTORY_FILE) + $(Q)$(MAKE) $(MINIO_DATA_TARGET_INSTALL) + +$(MINIO_DATA_TARGET_INSTALL): $(ANSIBLE_INVENTORY_FILE) + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + -f 30 -i hosts $(MINIO_PLAYBOOK) \ + --extra-vars=@$(KDEVOPS_EXTRA_VARS) \ + --tags vars,minio_install + +$(MINIO_DATA_TARGET_UNINSTALL): $(ANSIBLE_INVENTORY_FILE) + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + -f 30 -i hosts $(MINIO_PLAYBOOK) \ + --extra-vars=@$(KDEVOPS_EXTRA_VARS) \ + --tags vars,minio_uninstall + +$(MINIO_DATA_TARGET_DESTROY): $(ANSIBLE_INVENTORY_FILE) + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + -f 30 -i hosts $(MINIO_PLAYBOOK) \ + --extra-vars=@$(KDEVOPS_EXTRA_VARS) \ + --tags vars,minio_destroy + +$(MINIO_DATA_TARGET_RUN): $(ANSIBLE_INVENTORY_FILE) + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + -f 30 -i hosts $(MINIO_PLAYBOOK) \ + --extra-vars=@$(KDEVOPS_EXTRA_VARS) \ + --tags vars,minio_warp + +$(MINIO_DATA_TARGET_RESULTS): + $(Q)if [ -d workflows/minio/results ]; then \ + python3 workflows/minio/scripts/generate_warp_report.py workflows/minio/results/ && \ + echo "" && \ + echo "šŸ“Š MinIO Warp Analysis Complete!" && \ + echo "Results available in workflows/minio/results/" && \ + echo " - warp_benchmark_report.html (open in browser)" && \ + echo " - PNG charts for performance visualization" && \ + ls -lh workflows/minio/results/*.png 2>/dev/null | tail -5; \ + else \ + echo "No results directory found. Run 'make minio-warp' first."; \ + fi + +minio-help: + @echo "MinIO Warp S3 benchmarking targets:" + @echo "" + @echo "minio - Install and setup MinIO server" + @echo "minio-install - Install and setup MinIO server" + @echo "minio-uninstall - Stop and remove MinIO containers" + @echo "minio-destroy - Remove MinIO containers and clean up data" + @echo "minio-warp - Run MinIO Warp benchmarks" + @echo "minio-results - Collect and analyze benchmark results" + @echo "" + @echo "Example usage:" + @echo " make defconfig-minio-warp # Configure for Warp benchmarking" + @echo " make bringup # Setup test nodes" + @echo " make minio # Install MinIO server" + @echo " make minio-warp # Run benchmarks" + @echo " make minio-results # Generate analysis and visualizations" + @echo "" + @echo "Visualization options:" + @echo " - Enable MINIO_WARP_ENABLE_WEB_UI in menuconfig for real-time monitoring" + @echo " - Access web UI at http://node-ip:7762 during benchmarks" + @echo " - View HTML report: workflows/minio/results/warp_benchmark_report.html" + +.PHONY: $(MINIO_DATA_TARGET) $(MINIO_DATA_TARGET_INSTALL) $(MINIO_DATA_TARGET_UNINSTALL) +.PHONY: $(MINIO_DATA_TARGET_DESTROY) $(MINIO_DATA_TARGET_RUN) $(MINIO_DATA_TARGET_RESULTS) +.PHONY: minio-help diff --git a/workflows/minio/scripts/analyze_warp_results.py b/workflows/minio/scripts/analyze_warp_results.py new file mode 100755 index 0000000..c20c57d --- /dev/null +++ b/workflows/minio/scripts/analyze_warp_results.py @@ -0,0 +1,858 @@ +#!/usr/bin/env python3 +""" +Analyze MinIO Warp benchmark results and generate reports with visualizations. +""" + +import json +import glob +import os +import sys +from pathlib import Path +from datetime import datetime +import matplotlib.pyplot as plt +import matplotlib.patches as mpatches +import numpy as np +from typing import Dict, List, Any + + +def load_warp_results(results_dir: Path) -> List[Dict[str, Any]]: + """Load all Warp JSON result files from the results directory.""" + results = [] + json_files = list(results_dir.glob("warp_benchmark_*.json")) + + for json_file in sorted(json_files): + try: + with open(json_file, "r") as f: + content = f.read() + # Find where the JSON starts (after any terminal output) + json_start = content.find("{") + if json_start >= 0: + json_content = content[json_start:] + data = json.loads(json_content) + data["_filename"] = json_file.name + data["_filepath"] = str(json_file) + results.append(data) + print(f"Loaded: {json_file.name}") + else: + print(f"No JSON found in {json_file}") + except Exception as e: + print(f"Error loading {json_file}: {e}") + + return results + + +def extract_metrics(result: Dict[str, Any]) -> Dict[str, Any]: + """Extract key metrics from a Warp result.""" + metrics = { + "filename": result.get("_filename", "unknown"), + "timestamp": "", + "operation": "mixed", + } + + # Check if we have the total stats + if "total" in result: + total = result["total"] + + # Extract basic info + metrics["timestamp"] = total.get("start_time", "") + metrics["total_requests"] = total.get("total_requests", 0) + metrics["total_objects"] = total.get("total_objects", 0) + metrics["total_errors"] = total.get("total_errors", 0) + metrics["total_bytes"] = total.get("total_bytes", 0) + metrics["concurrency"] = total.get("concurrency", 0) + + # Calculate duration in seconds + start_time = total.get("start_time", "") + end_time = total.get("end_time", "") + if start_time and end_time: + try: + from dateutil import parser + + start = parser.parse(start_time) + end = parser.parse(end_time) + duration = (end - start).total_seconds() + metrics["duration_seconds"] = duration + except ImportError: + # Fall back to simple parsing if dateutil not available + metrics["duration_seconds"] = 105 # Default for 5m test + + # Get throughput if directly available + if "throughput" in total and isinstance(total["throughput"], dict): + # Throughput is a complex structure with segmented data + tp = total["throughput"] + if "bytes" in tp: + bytes_total = tp["bytes"] + duration_ms = tp.get("measure_duration_millis", 1000) + duration_s = duration_ms / 1000 + if duration_s > 0: + metrics["throughput_avg_mbps"] = ( + bytes_total / (1024 * 1024) + ) / duration_s + elif "segmented" in tp: + # Use median throughput + metrics["throughput_avg_mbps"] = tp["segmented"].get( + "median_bps", 0 + ) / (1024 * 1024) + elif metrics.get("duration_seconds", 0) > 0 and metrics["total_bytes"] > 0: + # Calculate throughput from bytes and duration + metrics["throughput_avg_mbps"] = ( + metrics["total_bytes"] / (1024 * 1024) + ) / metrics["duration_seconds"] + + # Calculate operations per second + if metrics.get("duration_seconds", 0) > 0: + metrics["ops_per_second"] = ( + metrics["total_requests"] / metrics["duration_seconds"] + ) + + # Check for operations breakdown by type + if "by_op_type" in result: + ops = result["by_op_type"] + + # Process each operation type + for op_type in ["GET", "PUT", "DELETE", "STAT"]: + if op_type in ops: + op_data = ops[op_type] + op_lower = op_type.lower() + + # Extract operation count + if "ops" in op_data: + metrics[f"{op_lower}_requests"] = op_data["ops"] + + # Extract average duration + if "avg_duration" in op_data: + metrics[f"{op_lower}_latency_avg_ms"] = ( + op_data["avg_duration"] / 1e6 + ) + + # Extract percentiles if available + if "percentiles_millis" in op_data: + percentiles = op_data["percentiles_millis"] + if "50" in percentiles: + metrics[f"{op_lower}_latency_p50"] = percentiles["50"] + if "90" in percentiles: + metrics[f"{op_lower}_latency_p90"] = percentiles["90"] + if "99" in percentiles: + metrics[f"{op_lower}_latency_p99"] = percentiles["99"] + + # Extract min/max if available + if "fastest_millis" in op_data: + metrics[f"{op_lower}_latency_min"] = op_data["fastest_millis"] + if "slowest_millis" in op_data: + metrics[f"{op_lower}_latency_max"] = op_data["slowest_millis"] + + # Calculate aggregate latency metrics + latencies = [] + for op in ["get", "put", "delete"]: + if f"{op}_latency_avg_ms" in metrics: + latencies.append(metrics[f"{op}_latency_avg_ms"]) + if latencies: + metrics["latency_avg_ms"] = sum(latencies) / len(latencies) + + # Extract from summary if present + if "summary" in result: + summary = result["summary"] + if "throughput_MiBs" in summary: + metrics["throughput_avg_mbps"] = summary["throughput_MiBs"] + if "ops_per_sec" in summary: + metrics["ops_per_second"] = summary["ops_per_sec"] + + return metrics + + +def generate_throughput_chart(metrics_list: List[Dict[str, Any]], output_dir: Path): + """Generate throughput comparison chart.""" + if not metrics_list: + return + + fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6)) + + # Throughput bar chart + labels = [ + m["filename"].replace("warp_benchmark_", "").replace(".json", "")[:20] + for m in metrics_list + ] + x = np.arange(len(labels)) + + avg_throughput = [m.get("throughput_avg_mbps", 0) for m in metrics_list] + + width = 0.35 + ax1.bar(x, avg_throughput, width, label="Throughput", color="skyblue") + + ax1.set_xlabel("Test Run") + ax1.set_ylabel("Throughput (MB/s)") + ax1.set_title("MinIO Warp Throughput Performance") + ax1.set_xticks(x) + ax1.set_xticklabels(labels, rotation=45, ha="right") + ax1.legend() + ax1.grid(True, alpha=0.3) + + # Operations per second + ops_per_sec = [m.get("ops_per_second", 0) for m in metrics_list] + ax2.bar(x, ops_per_sec, color="orange") + ax2.set_xlabel("Test Run") + ax2.set_ylabel("Operations/Second") + ax2.set_title("Operations Per Second") + ax2.set_xticks(x) + ax2.set_xticklabels(labels, rotation=45, ha="right") + ax2.grid(True, alpha=0.3) + + plt.tight_layout() + output_file = output_dir / "warp_throughput_performance.png" + plt.savefig(output_file, dpi=150, bbox_inches="tight") + plt.close() + print(f"Generated: {output_file}") + + +def generate_latency_chart(metrics_list: List[Dict[str, Any]], output_dir: Path): + """Generate latency comparison chart.""" + if not metrics_list: + return + + fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6)) + + labels = [ + m["filename"].replace("warp_benchmark_", "").replace(".json", "")[:20] + for m in metrics_list + ] + x = np.arange(len(labels)) + + # Collect latency data by operation type + operations = ["get", "put", "delete"] + colors = {"get": "steelblue", "put": "orange", "delete": "red"} + + # Operation-specific latencies + width = 0.2 + offset = -width + for op in operations: + op_latencies = [] + for m in metrics_list: + # Try operation-specific latency first, then fall back to general + lat = m.get(f"{op}_latency_avg_ms", m.get("latency_avg_ms", 0)) + op_latencies.append(lat) + + if any(l > 0 for l in op_latencies): + ax1.bar(x + offset, op_latencies, width, label=op.upper(), color=colors[op]) + offset += width + + ax1.set_xlabel("Test Run") + ax1.set_ylabel("Latency (ms)") + ax1.set_title("Request Latency Distribution") + ax1.set_xticks(x) + ax1.set_xticklabels(labels, rotation=45, ha="right") + ax1.legend() + ax1.grid(True, alpha=0.3) + + # Min/Max latency range + lat_min = [m.get("latency_min_ms", 0) for m in metrics_list] + lat_max = [m.get("latency_max_ms", 0) for m in metrics_list] + + ax2.bar(x - width / 2, lat_min, width, label="Min", color="green") + ax2.bar(x + width / 2, lat_max, width, label="Max", color="red") + + ax2.set_xlabel("Test Run") + ax2.set_ylabel("Latency (ms)") + ax2.set_title("Latency Range (Min/Max)") + ax2.set_xticks(x) + ax2.set_xticklabels(labels, rotation=45, ha="right") + ax2.legend() + ax2.grid(True, alpha=0.3) + + plt.tight_layout() + output_file = output_dir / "warp_latency_analysis.png" + plt.savefig(output_file, dpi=150, bbox_inches="tight") + plt.close() + print(f"Generated: {output_file}") + + +def generate_performance_summary_chart( + metrics_list: List[Dict[str, Any]], output_dir: Path +): + """Generate a comprehensive performance summary chart.""" + if not metrics_list: + return + + fig = plt.figure(figsize=(16, 10)) + gs = fig.add_gridspec(3, 2, hspace=0.3, wspace=0.25) + + # Throughput over time + ax1 = fig.add_subplot(gs[0, :]) + timestamps = [] + throughputs = [] + for m in metrics_list: + try: + if m.get("timestamp"): + timestamps.append( + datetime.fromisoformat(m["timestamp"].replace("Z", "+00:00")) + ) + throughputs.append(m.get("throughput_avg_mbps", 0)) + except: + pass + + if timestamps: + ax1.plot(timestamps, throughputs, "o-", linewidth=2, markersize=8, color="blue") + ax1.set_xlabel("Time") + ax1.set_ylabel("Throughput (MB/s)") + ax1.set_title("Throughput Over Time", fontsize=14, fontweight="bold") + ax1.grid(True, alpha=0.3) + ax1.tick_params(axis="x", rotation=45) + + # Operations distribution + ax2 = fig.add_subplot(gs[1, 0]) + ops_data = [m.get("ops_per_second", 0) for m in metrics_list] + if ops_data: + ax2.hist(ops_data, bins=10, color="orange", edgecolor="black", alpha=0.7) + ax2.set_xlabel("Operations/Second") + ax2.set_ylabel("Frequency") + ax2.set_title("Operations Distribution", fontsize=12, fontweight="bold") + ax2.grid(True, alpha=0.3) + + # Latency box plot + ax3 = fig.add_subplot(gs[1, 1]) + latency_data = [] + for m in metrics_list: + lat_data = [] + if m.get("latency_avg_ms"): + lat_data.extend( + [ + m.get("latency_min_ms", 0), + m.get("latency_percentile_50", 0), + m.get("latency_avg_ms", 0), + m.get("latency_percentile_99", 0), + m.get("latency_max_ms", 0), + ] + ) + if lat_data: + latency_data.append(lat_data) + + if latency_data: + ax3.boxplot(latency_data) + ax3.set_xlabel("Test Run") + ax3.set_ylabel("Latency (ms)") + ax3.set_title("Latency Distribution", fontsize=12, fontweight="bold") + ax3.grid(True, alpha=0.3) + + # Performance metrics table + ax4 = fig.add_subplot(gs[2, :]) + ax4.axis("tight") + ax4.axis("off") + + # Create summary statistics + if metrics_list: + avg_metrics = metrics_list[-1] # Use most recent for now + table_data = [ + ["Metric", "Value"], + [ + "Average Throughput", + f"{avg_metrics.get('throughput_avg_mbps', 0):.2f} MB/s", + ], + ["Operations/Second", f"{avg_metrics.get('ops_per_second', 0):.0f}"], + ["Average Latency", f"{avg_metrics.get('latency_avg_ms', 0):.2f} ms"], + ["P99 Latency", f"{avg_metrics.get('latency_percentile_99', 0):.2f} ms"], + ["Total Operations", f"{avg_metrics.get('ops_total', 0):.0f}"], + ["Object Size", str(avg_metrics.get("object_size", "unknown"))], + ["Error Rate", f"{avg_metrics.get('error_rate', 0):.2%}"], + ] + + table = ax4.table( + cellText=table_data, cellLoc="left", loc="center", colWidths=[0.3, 0.3] + ) + table.auto_set_font_size(False) + table.set_fontsize(10) + table.scale(1, 1.5) + + # Style the header row + for i in range(2): + table[(0, i)].set_facecolor("#40466e") + table[(0, i)].set_text_props(weight="bold", color="white") + + plt.suptitle("MinIO Warp Performance Summary", fontsize=16, fontweight="bold") + + output_file = output_dir / "warp_performance_summary.png" + plt.savefig(output_file, dpi=150, bbox_inches="tight") + plt.close() + print(f"Generated: {output_file}") + + +def generate_text_report(metrics_list: List[Dict[str, Any]], output_dir: Path): + """Generate a detailed text report.""" + output_file = output_dir / "warp_analysis_report.txt" + + with open(output_file, "w") as f: + f.write("=" * 80 + "\n") + f.write("MinIO Warp Benchmark Analysis Report\n") + f.write("=" * 80 + "\n\n") + f.write(f"Generated: {datetime.now().isoformat()}\n") + f.write(f"Total test runs analyzed: {len(metrics_list)}\n\n") + + if not metrics_list: + f.write("No benchmark results found.\n") + return + + # Overall statistics + f.write("OVERALL PERFORMANCE STATISTICS\n") + f.write("-" * 40 + "\n") + + throughputs = [ + m.get("throughput_avg_mbps", 0) + for m in metrics_list + if m.get("throughput_avg_mbps") + ] + if throughputs: + f.write(f"Throughput:\n") + f.write(f" Average: {np.mean(throughputs):.2f} MB/s\n") + f.write(f" Median: {np.median(throughputs):.2f} MB/s\n") + f.write(f" Min: {np.min(throughputs):.2f} MB/s\n") + f.write(f" Max: {np.max(throughputs):.2f} MB/s\n") + f.write(f" StdDev: {np.std(throughputs):.2f} MB/s\n\n") + + ops_rates = [ + m.get("ops_per_second", 0) for m in metrics_list if m.get("ops_per_second") + ] + if ops_rates: + f.write(f"Operations per Second:\n") + f.write(f" Average: {np.mean(ops_rates):.0f} ops/s\n") + f.write(f" Median: {np.median(ops_rates):.0f} ops/s\n") + f.write(f" Min: {np.min(ops_rates):.0f} ops/s\n") + f.write(f" Max: {np.max(ops_rates):.0f} ops/s\n\n") + + latencies = [ + m.get("latency_avg_ms", 0) for m in metrics_list if m.get("latency_avg_ms") + ] + if latencies: + f.write(f"Average Latency:\n") + f.write(f" Mean: {np.mean(latencies):.2f} ms\n") + f.write(f" Median: {np.median(latencies):.2f} ms\n") + f.write(f" Min: {np.min(latencies):.2f} ms\n") + f.write(f" Max: {np.max(latencies):.2f} ms\n\n") + + # Individual test run details + f.write("=" * 80 + "\n") + f.write("INDIVIDUAL TEST RUN DETAILS\n") + f.write("=" * 80 + "\n\n") + + for i, metrics in enumerate(metrics_list, 1): + f.write(f"Test Run #{i}\n") + f.write("-" * 40 + "\n") + f.write(f"File: {metrics.get('filename', 'unknown')}\n") + f.write(f"Timestamp: {metrics.get('timestamp', 'N/A')}\n") + f.write(f"Operation: {metrics.get('operation', 'unknown')}\n") + f.write(f"Duration: {metrics.get('duration_seconds', 0):.1f} seconds\n") + f.write(f"Object Size: {metrics.get('object_size', 'unknown')}\n") + f.write(f"Total Objects: {metrics.get('objects_total', 0)}\n") + + if metrics.get("throughput_avg_mbps"): + f.write(f"\nThroughput Performance:\n") + f.write( + f" Average: {metrics.get('throughput_avg_mbps', 0):.2f} MB/s\n" + ) + f.write( + f" Min: {metrics.get('throughput_min_mbps', 0):.2f} MB/s\n" + ) + f.write( + f" Max: {metrics.get('throughput_max_mbps', 0):.2f} MB/s\n" + ) + f.write( + f" P50: {metrics.get('throughput_percentile_50', 0):.2f} MB/s\n" + ) + f.write( + f" P99: {metrics.get('throughput_percentile_99', 0):.2f} MB/s\n" + ) + + if metrics.get("ops_per_second"): + f.write(f"\nOperations Performance:\n") + f.write(f" Total Operations: {metrics.get('ops_total', 0):.0f}\n") + f.write( + f" Operations/Second: {metrics.get('ops_per_second', 0):.0f}\n" + ) + f.write( + f" Avg Duration: {metrics.get('ops_avg_duration_ms', 0):.2f} ms\n" + ) + + if metrics.get("latency_avg_ms"): + f.write(f"\nLatency Metrics:\n") + f.write(f" Average: {metrics.get('latency_avg_ms', 0):.2f} ms\n") + f.write(f" Min: {metrics.get('latency_min_ms', 0):.2f} ms\n") + f.write(f" Max: {metrics.get('latency_max_ms', 0):.2f} ms\n") + f.write( + f" P50: {metrics.get('latency_percentile_50', 0):.2f} ms\n" + ) + f.write( + f" P99: {metrics.get('latency_percentile_99', 0):.2f} ms\n" + ) + + if metrics.get("error_count", 0) > 0: + f.write(f"\nErrors:\n") + f.write(f" Error Count: {metrics.get('error_count', 0)}\n") + f.write(f" Error Rate: {metrics.get('error_rate', 0):.2%}\n") + + f.write("\n") + + f.write("=" * 80 + "\n") + f.write("END OF REPORT\n") + f.write("=" * 80 + "\n") + + print(f"Generated: {output_file}") + + +def generate_html_report(metrics_list: List[Dict[str, Any]], output_dir: Path): + """Generate a comprehensive HTML report with embedded visualizations.""" + output_file = output_dir / "warp_benchmark_report.html" + + # Check if PNG files exist + throughput_png = output_dir / "warp_throughput_performance.png" + latency_png = output_dir / "warp_latency_analysis.png" + summary_png = output_dir / "warp_performance_summary.png" + + with open(output_file, "w") as f: + f.write( + """ + + + + MinIO Warp Benchmark Report + + + +
+

šŸš€ MinIO Warp Benchmark Report

+
Generated on """ + + datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + """
+ +
+

šŸ“Š Performance Summary

+
+""" + ) + + if metrics_list: + # Calculate summary statistics + throughputs = [ + m.get("throughput_avg_mbps", 0) + for m in metrics_list + if m.get("throughput_avg_mbps") + ] + ops_rates = [ + m.get("ops_per_second", 0) + for m in metrics_list + if m.get("ops_per_second") + ] + latencies = [ + m.get("latency_avg_ms", 0) + for m in metrics_list + if m.get("latency_avg_ms") + ] + + if throughputs: + avg_throughput = np.mean(throughputs) + f.write( + f""" +
+
Average Throughput
+
{avg_throughput:.1f}
+
MB/s
+
+ """ + ) + + f.write( + f""" +
+
Peak Throughput
+
{np.max(throughputs):.1f}
+
MB/s
+
+ """ + ) + + if ops_rates: + f.write( + f""" +
+
Avg Operations
+
{np.mean(ops_rates):.0f}
+
ops/second
+
+ """ + ) + + if latencies: + f.write( + f""" +
+
Avg Latency
+
{np.mean(latencies):.1f}
+
ms
+
+ """ + ) + + f.write( + f""" +
+
Test Runs
+
{len(metrics_list)}
+
completed
+
+ """ + ) + + f.write( + """ +
+
+ +
+

šŸ“ˆ Performance Visualizations

+ """ + ) + + if throughput_png.exists(): + f.write( + f""" +
+ Throughput Performance +
+ """ + ) + + if latency_png.exists(): + f.write( + f""" +
+ Latency Analysis +
+ """ + ) + + if summary_png.exists(): + f.write( + f""" +
+ Performance Summary +
+ """ + ) + + f.write( + """ +
+

šŸ“‹ Detailed Results

+ + + + + + + + + + + + + + """ + ) + + for metrics in metrics_list: + throughput = metrics.get("throughput_avg_mbps", 0) + ops_sec = metrics.get("ops_per_second", 0) + latency = metrics.get("latency_avg_ms", 0) + p99_lat = metrics.get("latency_percentile_99", 0) + errors = metrics.get("error_count", 0) + + # Color code based on performance + throughput_class = ( + "performance-good" + if throughput > 100 + else "performance-warning" if throughput > 50 else "performance-bad" + ) + latency_class = ( + "performance-good" + if latency < 10 + else "performance-warning" if latency < 50 else "performance-bad" + ) + + f.write( + f""" + + + + + + + + + + """ + ) + + f.write( + """ + +
Test RunOperationThroughput (MB/s)Ops/SecondAvg Latency (ms)P99 Latency (ms)Errors
{metrics.get('filename', 'unknown').replace('warp_benchmark_', '').replace('.json', '')}{metrics.get('operation', 'mixed')}{throughput:.2f}{ops_sec:.0f}{latency:.2f}{p99_lat:.2f}{errors}
+
+ +
+

MinIO Warp Benchmark Analysis | Generated by kdevops

+
+
+ + + """ + ) + + print(f"Generated: {output_file}") + + +def main(): + """Main analysis function.""" + # Determine results directory + script_dir = Path(__file__).parent + results_dir = script_dir.parent / "results" + + if not results_dir.exists(): + print(f"Results directory not found: {results_dir}") + return 1 + + # Load all results + results = load_warp_results(results_dir) + if not results: + print("No Warp benchmark results found.") + return 1 + + print(f"\nFound {len(results)} benchmark result(s)") + + # Extract metrics from each result + metrics_list = [extract_metrics(result) for result in results] + + # Generate visualizations + print("\nGenerating visualizations...") + generate_throughput_chart(metrics_list, results_dir) + generate_latency_chart(metrics_list, results_dir) + generate_performance_summary_chart(metrics_list, results_dir) + + # Generate reports + print("\nGenerating reports...") + generate_text_report(metrics_list, results_dir) + generate_html_report(metrics_list, results_dir) + + print("\nāœ… Analysis complete! Check the results directory for:") + print(" - warp_throughput_performance.png") + print(" - warp_latency_analysis.png") + print(" - warp_performance_summary.png") + print(" - warp_analysis_report.txt") + print(" - warp_benchmark_report.html") + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/workflows/minio/scripts/generate_warp_report.py b/workflows/minio/scripts/generate_warp_report.py new file mode 100755 index 0000000..2eff522 --- /dev/null +++ b/workflows/minio/scripts/generate_warp_report.py @@ -0,0 +1,404 @@ +#!/usr/bin/env python3 +""" +Generate graphs and HTML report from MinIO Warp benchmark results +""" + +import json +import os +import sys +import glob +from datetime import datetime +import matplotlib.pyplot as plt +import matplotlib.dates as mdates +from pathlib import Path + + +def parse_warp_json(json_file): + """Parse Warp benchmark JSON output""" + with open(json_file, "r") as f: + content = f.read() + # Find the JSON object in the output (skip any non-JSON prefix) + json_start = content.find("{") + if json_start == -1: + raise ValueError(f"No JSON found in {json_file}") + json_content = content[json_start:] + return json.loads(json_content) + + +def generate_throughput_graph(data, output_dir, filename_prefix): + """Generate throughput over time graph""" + segments = data["total"]["throughput"]["segmented"]["segments"] + + # Extract timestamps and throughput values + times = [] + throughput_mbps = [] + ops_per_sec = [] + + for segment in segments: + time_str = segment["start"] + # Parse timestamp + timestamp = datetime.fromisoformat(time_str.replace("-07:00", "+00:00")) + times.append(timestamp) + throughput_mbps.append( + segment["bytes_per_sec"] / (1024 * 1024) + ) # Convert to MB/s + ops_per_sec.append(segment["obj_per_sec"]) + + # Create figure with two subplots + fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 8)) + + # Throughput graph + ax1.plot(times, throughput_mbps, "b-", linewidth=2, marker="o") + ax1.set_ylabel("Throughput (MB/s)", fontsize=12) + ax1.set_title("MinIO Warp Benchmark - Throughput Over Time", fontsize=14) + ax1.grid(True, alpha=0.3) + ax1.xaxis.set_major_formatter(mdates.DateFormatter("%H:%M:%S")) + + # Add average line + avg_throughput = sum(throughput_mbps) / len(throughput_mbps) + ax1.axhline( + y=avg_throughput, + color="r", + linestyle="--", + alpha=0.7, + label=f"Average: {avg_throughput:.1f} MB/s", + ) + ax1.legend() + + # Operations per second graph + ax2.plot(times, ops_per_sec, "g-", linewidth=2, marker="s") + ax2.set_xlabel("Time", fontsize=12) + ax2.set_ylabel("Operations/sec", fontsize=12) + ax2.set_title("Operations Per Second", fontsize=14) + ax2.grid(True, alpha=0.3) + ax2.xaxis.set_major_formatter(mdates.DateFormatter("%H:%M:%S")) + + # Add average line + avg_ops = sum(ops_per_sec) / len(ops_per_sec) + ax2.axhline( + y=avg_ops, + color="r", + linestyle="--", + alpha=0.7, + label=f"Average: {avg_ops:.1f} ops/s", + ) + ax2.legend() + + plt.gcf().autofmt_xdate() + plt.tight_layout() + + graph_file = os.path.join(output_dir, f"{filename_prefix}_throughput.png") + plt.savefig(graph_file, dpi=100, bbox_inches="tight") + plt.close() + + return graph_file + + +def generate_operation_stats_graph(data, output_dir, filename_prefix): + """Generate operation statistics bar chart""" + operations = data.get("operations", {}) + + if not operations: + return None + + op_types = [] + throughputs = [] + latencies = [] + + for op_type, op_data in operations.items(): + if op_type in ["DELETE", "GET", "PUT", "STAT"]: + op_types.append(op_type) + if "throughput" in op_data: + throughputs.append(op_data["throughput"]["obj_per_sec"]) + if "latency" in op_data: + # Convert nanoseconds to milliseconds + latencies.append(op_data["latency"]["mean"] / 1_000_000) + + if not op_types: + return None + + fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5)) + + # Throughput bar chart + if throughputs: + ax1.bar(op_types, throughputs, color=["blue", "green", "red", "orange"]) + ax1.set_ylabel("Operations/sec", fontsize=12) + ax1.set_title("Operation Throughput", fontsize=14) + ax1.grid(True, axis="y", alpha=0.3) + + # Latency bar chart + if latencies: + ax2.bar(op_types, latencies, color=["blue", "green", "red", "orange"]) + ax2.set_ylabel("Latency (ms)", fontsize=12) + ax2.set_title("Operation Latency (Mean)", fontsize=14) + ax2.grid(True, axis="y", alpha=0.3) + + plt.tight_layout() + + graph_file = os.path.join(output_dir, f"{filename_prefix}_operations.png") + plt.savefig(graph_file, dpi=100, bbox_inches="tight") + plt.close() + + return graph_file + + +def generate_html_report(json_files, output_dir): + """Generate HTML report with all results""" + timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + html_content = f""" + + + + + MinIO Warp Benchmark Results + + + +

MinIO Warp Benchmark Results

+

Generated: {timestamp}

+""" + + for json_file in sorted(json_files, reverse=True): + try: + data = parse_warp_json(json_file) + filename = os.path.basename(json_file) + filename_prefix = filename.replace(".json", "") + + # Extract key metrics + total = data["total"] + total_requests = total.get("total_requests", 0) + total_objects = total.get("total_objects", 0) + total_errors = total.get("total_errors", 0) + total_bytes = total.get("total_bytes", 0) + concurrency = total.get("concurrency", 0) + + throughput_data = total.get("throughput", {}).get("segmented", {}) + fastest_bps = throughput_data.get("fastest_bps", 0) / (1024 * 1024) # MB/s + slowest_bps = throughput_data.get("slowest_bps", 0) / (1024 * 1024) # MB/s + average_bps = throughput_data.get("average_bps", 0) / (1024 * 1024) # MB/s + + html_content += f""" +
+

{filename}

+ +
+
+
Total Requests
+
{total_requests:,}
+
+
+
Total Objects
+
{total_objects:,}
+
+
+
Total Errors
+
{total_errors}
+
+
+
Total Data
+
{total_bytes / (1024**3):.2f} GB
+
+
+
Concurrency
+
{concurrency}
+
+
+
Average Throughput
+
{average_bps:.1f} MB/s
+
+
+
Fastest Throughput
+
{fastest_bps:.1f} MB/s
+
+
+
Slowest Throughput
+
{slowest_bps:.1f} MB/s
+
+
+""" + + # Generate graphs + throughput_graph = generate_throughput_graph( + data, output_dir, filename_prefix + ) + if throughput_graph: + rel_path = os.path.basename(throughput_graph) + html_content += ( + f' Throughput Graph\n' + ) + + ops_graph = generate_operation_stats_graph( + data, output_dir, filename_prefix + ) + if ops_graph: + rel_path = os.path.basename(ops_graph) + html_content += ( + f' Operations Statistics\n' + ) + + # Add operations table if available + operations = data.get("operations", {}) + if operations: + html_content += """ +

Operation Details

+ + + + + + + + +""" + for op_type, op_data in operations.items(): + if op_type in ["DELETE", "GET", "PUT", "STAT"]: + throughput = op_data.get("throughput", {}).get("obj_per_sec", 0) + latency = op_data.get("latency", {}) + mean_lat = latency.get("mean", 0) / 1_000_000 + min_lat = latency.get("min", 0) / 1_000_000 + max_lat = latency.get("max", 0) / 1_000_000 + + html_content += f""" + + + + + + + +""" + html_content += "
OperationThroughput (ops/s)Mean Latency (ms)Min Latency (ms)Max Latency (ms)
{op_type}{throughput:.2f}{mean_lat:.2f}{min_lat:.2f}{max_lat:.2f}
\n" + + html_content += "
\n" + + except Exception as e: + print(f"Error processing {json_file}: {e}") + continue + + html_content += """ + + +""" + + html_file = os.path.join(output_dir, "warp_benchmark_report.html") + with open(html_file, "w") as f: + f.write(html_content) + + return html_file + + +def main(): + if len(sys.argv) > 1: + results_dir = sys.argv[1] + else: + # Default to workflows/minio/results + script_dir = Path(__file__).parent.absolute() + results_dir = script_dir.parent / "results" + + results_dir = Path(results_dir) + if not results_dir.exists(): + print(f"Results directory {results_dir} does not exist") + sys.exit(1) + + # Find all JSON files + json_files = list(results_dir.glob("warp_benchmark_*.json")) + + if not json_files: + print(f"No warp_benchmark_*.json files found in {results_dir}") + sys.exit(1) + + print(f"Found {len(json_files)} result files") + + # Generate HTML report with graphs + html_file = generate_html_report(json_files, results_dir) + print(f"Generated HTML report: {html_file}") + + # Also generate individual graphs for latest result + latest_json = max(json_files, key=os.path.getctime) + data = parse_warp_json(latest_json) + filename_prefix = latest_json.stem + + throughput_graph = generate_throughput_graph(data, results_dir, filename_prefix) + if throughput_graph: + print(f"Generated throughput graph: {throughput_graph}") + + ops_graph = generate_operation_stats_graph(data, results_dir, filename_prefix) + if ops_graph: + print(f"Generated operations graph: {ops_graph}") + + +if __name__ == "__main__": + main() diff --git a/workflows/minio/scripts/run_benchmark_suite.sh b/workflows/minio/scripts/run_benchmark_suite.sh new file mode 100755 index 0000000..ca0531b --- /dev/null +++ b/workflows/minio/scripts/run_benchmark_suite.sh @@ -0,0 +1,116 @@ +#!/bin/bash +# Run comprehensive MinIO Warp benchmark suite + +MINIO_HOST="${1:-localhost:9000}" +ACCESS_KEY="${2:-minioadmin}" +SECRET_KEY="${3:-minioadmin}" +TOTAL_DURATION="${4:-30m}" +RESULTS_DIR="/tmp/warp-results" +TIMESTAMP=$(date +%s) + +# Parse duration to seconds for calculation +parse_duration_to_seconds() { + local duration="$1" + local value="${duration//[^0-9]/}" + local unit="${duration//[0-9]/}" + + case "$unit" in + s) echo "$value" ;; + m) echo $((value * 60)) ;; + h) echo $((value * 3600)) ;; + *) echo "1800" ;; # Default 30 minutes + esac +} + +TOTAL_SECONDS=$(parse_duration_to_seconds "$TOTAL_DURATION") +# Distribute time across 8 benchmark types (reserving some buffer) +PER_TEST_SECONDS=$((TOTAL_SECONDS / 10)) # Divide by 10 to leave buffer +if [ $PER_TEST_SECONDS -lt 30 ]; then + PER_TEST_SECONDS=30 # Minimum 30 seconds per test +fi + +# Convert back to duration string +if [ $PER_TEST_SECONDS -ge 3600 ]; then + PER_TEST_DURATION="$((PER_TEST_SECONDS / 3600))h" +elif [ $PER_TEST_SECONDS -ge 60 ]; then + PER_TEST_DURATION="$((PER_TEST_SECONDS / 60))m" +else + PER_TEST_DURATION="${PER_TEST_SECONDS}s" +fi + +echo "šŸš€ MinIO Warp Comprehensive Benchmark Suite" +echo "===========================================" +echo "Target: $MINIO_HOST" +echo "Total Duration: $TOTAL_DURATION ($TOTAL_SECONDS seconds)" +echo "Per Test Duration: $PER_TEST_DURATION" +echo "Results: $RESULTS_DIR" +echo "" + +# Ensure results directory exists +mkdir -p "$RESULTS_DIR" + +# Function to run a benchmark +run_benchmark() { + local test_type=$1 + local duration=$2 + local concurrent=$3 + local obj_size=$4 + + echo "Running $test_type benchmark..." + echo " Duration: $duration, Concurrent: $concurrent, Size: $obj_size" + + OUTPUT_FILE="${RESULTS_DIR}/warp_${test_type}_${TIMESTAMP}.json" + + # Don't use --autoterm or --objects for duration-based tests + warp "$test_type" \ + --host="$MINIO_HOST" \ + --access-key="$ACCESS_KEY" \ + --secret-key="$SECRET_KEY" \ + --bucket="warp-bench-${test_type}" \ + --duration="$duration" \ + --concurrent="$concurrent" \ + --obj.size="$obj_size" \ + --noclear \ + --json > "$OUTPUT_FILE" 2>&1 + + if [ $? -eq 0 ]; then + echo "āœ… $test_type completed successfully" + else + echo "āŒ $test_type failed" + fi + echo "" +} + +# Run comprehensive test suite +echo "šŸ“Š Starting Comprehensive Benchmark Suite" +echo "-----------------------------------------" + +# 1. Mixed workload (simulates real-world usage) +run_benchmark "mixed" "$PER_TEST_DURATION" "10" "1MB" + +# 2. GET performance (read-heavy workload) +run_benchmark "get" "$PER_TEST_DURATION" "20" "1MB" + +# 3. PUT performance (write-heavy workload) +run_benchmark "put" "$PER_TEST_DURATION" "10" "10MB" + +# 4. DELETE performance +run_benchmark "delete" "$PER_TEST_DURATION" "5" "1MB" + +# 5. LIST operations (metadata operations) +run_benchmark "list" "$PER_TEST_DURATION" "5" "1KB" + +# 6. Small object performance +run_benchmark "mixed" "$PER_TEST_DURATION" "10" "1KB" + +# 7. Large object performance +run_benchmark "mixed" "$PER_TEST_DURATION" "5" "100MB" + +# 8. High concurrency test +run_benchmark "mixed" "$PER_TEST_DURATION" "50" "1MB" + +echo "===========================================" +echo "āœ… Benchmark Suite Complete!" +echo "" +echo "Results saved in: $RESULTS_DIR" +echo "Run 'make minio-results' to generate analysis" -- 2.50.1