From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A779C15667D for ; Wed, 17 Sep 2025 00:34:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758069294; cv=none; b=ZjRU3G9j1uLJGYmXNXeOi5nNvbCzaLcAiqDif/13dOaRHBfiujglufdkQo5B0Dpr0plMqjD0TASleQuviHolsEM/4//SMoOonZ1Q5DuCcQrmkyy8y7hvM9m1J5TbXId1QpRJNvazBylloKB8qVc+B/U4+36tBuCiTeCCOgrwLz0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758069294; c=relaxed/simple; bh=g7EHs6X6Pb1/WArp0XsaU+5yoGjtqElJBVaZs/6qCWQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Y6sDYz7woXwFvG/m+ZvwkKm0Q2j9H4KGkpZHCx/EHPN9aMumK28NNRVkyVZxQv4rlkgi6k9usAkZOsO+gYiCc65YvPak884AfYACsl4IPGlQEC9OciR2vwRNC0EWb0Gdb1Y8H1w/sRnPlsgkWwGsFC6PeuigGcCY7SdcDWbZc44= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=yTtCBg2e; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="yTtCBg2e" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=6JfdxSj4ATYF53XWX6gAAUDqXMmpAvAyTVm1BcvlY+c=; b=yTtCBg2e5xK5EKIyGy/6VWjruU SMKdj9UvX+ZhM5oCFJ1LaLQSIbRZSZq6bSK0GqdS1vjRjGQ6tzd0rg8N0gji2UoT/4/7hj9sK8Le9 2OtRiJS47flpz4yb41v5Pe01swhImpkesNgphuxrDx2vii+LcTeTE0IgyfnuTwkpzCO+RTgrAl/Ra EBlV1SW5kjXREtNbB3o8yqp0cWaXAVZTAZ1ourlbJ9jm7v0ch7O2u/SZFf6aPZ7bgapNJfFB5sCc2 jj1CCZ1gFGcznXlBW3EMb9OtVdr0+RG+nWAC7Zqah27Ns+9Bk+xDaehALApjcI4vlsyIKFGbH1+xr kKLd6ypg==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98.2 #2 (Red Hat Linux)) id 1uyg8K-00000009j5N-05q2; Wed, 17 Sep 2025 00:34:52 +0000 From: Luis Chamberlain To: Chuck Lever , Daniel Gomez , kdevops@lists.linux.dev Cc: Luis Chamberlain Subject: [PATCH v4 4/8] aws: integrate dynamic Kconfig generation with make targets Date: Tue, 16 Sep 2025 17:34:45 -0700 Message-ID: <20250917003451.2318229-5-mcgrof@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250917003451.2318229-1-mcgrof@kernel.org> References: <20250917003451.2318229-1-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain Add Makefile integration for AWS dynamic Kconfig generation using Chuck's scripts with the optimized wrapper. This provides: - make cloud-config-aws: Generate AWS Kconfig files - make cloud-update-aws: Clear cache and regenerate fresh data - make cloud-config: Now includes AWS generation - make cloud-update: Refreshes all cloud provider data - make clean-cloud-config-aws: Clean generated AWS files The integration properly manages AWS Kconfig files alongside Lambda Labs configurations and ensures empty files exist before Kconfig runs to prevent sourcing errors. Also update generate_cloud_configs.py to call the AWS generator and provide summary statistics about available AWS resources. Generated-by: Claude AI Signed-off-by: Luis Chamberlain --- scripts/dynamic-cloud-kconfig.Makefile | 49 ++++++++++++++-- scripts/generate_cloud_configs.py | 78 +++++++++++++++++++++++++- 2 files changed, 120 insertions(+), 7 deletions(-) diff --git a/scripts/dynamic-cloud-kconfig.Makefile b/scripts/dynamic-cloud-kconfig.Makefile index e15651ab..ed2d5366 100644 --- a/scripts/dynamic-cloud-kconfig.Makefile +++ b/scripts/dynamic-cloud-kconfig.Makefile @@ -12,8 +12,18 @@ LAMBDALABS_KCONFIG_IMAGES := $(LAMBDALABS_KCONFIG_DIR)/Kconfig.images.generated LAMBDALABS_KCONFIGS := $(LAMBDALABS_KCONFIG_COMPUTE) $(LAMBDALABS_KCONFIG_LOCATION) $(LAMBDALABS_KCONFIG_IMAGES) -# Add Lambda Labs generated files to mrproper clean list -KDEVOPS_MRPROPER += $(LAMBDALABS_KCONFIGS) +# AWS dynamic configuration +AWS_KCONFIG_DIR := terraform/aws/kconfigs +AWS_KCONFIG_COMPUTE := $(AWS_KCONFIG_DIR)/Kconfig.compute.generated +AWS_KCONFIG_LOCATION := $(AWS_KCONFIG_DIR)/Kconfig.location.generated +AWS_KCONFIG_GPU_AMIS := $(AWS_KCONFIG_DIR)/Kconfig.gpu-amis.generated +AWS_KCONFIG_INSTANCE_TYPES_DIR := $(AWS_KCONFIG_DIR)/instance-types + +# Note: Instance type files are generated dynamically, so we just track the main ones +AWS_KCONFIGS := $(AWS_KCONFIG_COMPUTE) $(AWS_KCONFIG_LOCATION) $(AWS_KCONFIG_GPU_AMIS) + +# Add generated files to mrproper clean list +KDEVOPS_MRPROPER += $(LAMBDALABS_KCONFIGS) $(AWS_KCONFIGS) # Touch Lambda Labs generated files so Kconfig can source them # This ensures the files exist (even if empty) before Kconfig runs @@ -22,20 +32,44 @@ dynamic_lambdalabs_kconfig_touch: DYNAMIC_KCONFIG += dynamic_lambdalabs_kconfig_touch +# Touch AWS generated files so Kconfig can source them +dynamic_aws_kconfig_touch: + $(Q)mkdir -p $(AWS_KCONFIG_INSTANCE_TYPES_DIR) + $(Q)touch $(AWS_KCONFIGS) + +DYNAMIC_KCONFIG += dynamic_aws_kconfig_touch + # Individual Lambda Labs targets are now handled by generate_cloud_configs.py cloud-config-lambdalabs: $(Q)python3 scripts/generate_cloud_configs.py +# AWS targets using Chuck's scripts with caching +cloud-config-aws: + $(Q)python3 terraform/aws/scripts/generate_aws_kconfig.py + +# AWS cache refresh (clears cache and regenerates) +cloud-update-aws: + $(Q)python3 terraform/aws/scripts/generate_aws_kconfig.py clear-cache + $(Q)python3 terraform/aws/scripts/generate_aws_kconfig.py + # Clean Lambda Labs generated files clean-cloud-config-lambdalabs: $(Q)rm -f $(LAMBDALABS_KCONFIGS) -DYNAMIC_CLOUD_KCONFIG += cloud-config-lambdalabs +# Clean AWS generated files +clean-cloud-config-aws: + $(Q)rm -f $(AWS_KCONFIGS) + $(Q)rm -rf $(AWS_KCONFIG_INSTANCE_TYPES_DIR) + +DYNAMIC_CLOUD_KCONFIG += cloud-config-lambdalabs cloud-config-aws cloud-config-help: @echo "Cloud-specific dynamic kconfig targets:" @echo "cloud-config - generates all cloud provider dynamic kconfig content" @echo "cloud-config-lambdalabs - generates Lambda Labs dynamic kconfig content" + @echo "cloud-config-aws - generates AWS dynamic kconfig content" + @echo "cloud-update - refreshes cloud provider data (clears cache)" + @echo "cloud-update-aws - refreshes AWS data (clears cache and regenerates)" @echo "clean-cloud-config - removes all generated cloud kconfig files" @echo "cloud-list-all - list all cloud instances for configured provider" @@ -44,11 +78,16 @@ HELP_TARGETS += cloud-config-help cloud-config: $(Q)python3 scripts/generate_cloud_configs.py -clean-cloud-config: clean-cloud-config-lambdalabs +cloud-update: cloud-update-aws + $(Q)echo "Updated cloud provider configurations." + +clean-cloud-config: clean-cloud-config-lambdalabs clean-cloud-config-aws $(Q)echo "Cleaned all cloud provider dynamic Kconfig files." cloud-list-all: $(Q)chmod +x scripts/cloud_list_all.sh $(Q)scripts/cloud_list_all.sh -PHONY += cloud-config cloud-config-lambdalabs clean-cloud-config clean-cloud-config-lambdalabs cloud-config-help cloud-list-all +PHONY += cloud-config cloud-config-lambdalabs cloud-config-aws cloud-update cloud-update-aws +PHONY += clean-cloud-config clean-cloud-config-lambdalabs clean-cloud-config-aws +PHONY += cloud-config-help cloud-list-all diff --git a/scripts/generate_cloud_configs.py b/scripts/generate_cloud_configs.py index b16294dd..e8251a73 100755 --- a/scripts/generate_cloud_configs.py +++ b/scripts/generate_cloud_configs.py @@ -100,6 +100,68 @@ def get_lambdalabs_summary() -> tuple[bool, str]: return False, "Lambda Labs: Error querying API - using defaults" +def generate_aws_kconfig() -> bool: + """ + Generate AWS Kconfig files using Chuck's scripts. + Returns True on success, False on failure. + """ + script_path = "terraform/aws/scripts/generate_aws_kconfig.py" + + result = subprocess.run( + [sys.executable, script_path], + capture_output=True, + text=True, + check=False, + ) + + return result.returncode == 0 + + +def get_aws_summary() -> tuple[bool, str]: + """ + Get a summary of AWS configurations. + Returns (success, summary_string) + """ + try: + # Check if AWS credentials are configured + result = subprocess.run( + ["aws", "sts", "get-caller-identity"], + capture_output=True, + text=True, + check=False, + ) + + if result.returncode != 0: + return False, "AWS: Credentials not configured - using defaults" + + # Get basic stats from cached data if available + cache_dir = os.path.expanduser("~/.cache/kdevops/aws") + families_cache = os.path.join(cache_dir, "aws_families.json") + + if os.path.exists(families_cache): + with open(families_cache, 'r') as f: + families = json.load(f) + family_count = len(families) + else: + family_count = "72+" # Known minimum + + regions_cache = os.path.join(cache_dir, "aws_regions.json") + if os.path.exists(regions_cache): + with open(regions_cache, 'r') as f: + regions = json.load(f) + region_count = len(regions) + else: + region_count = "30+" # Known minimum + + return ( + True, + f"AWS: {family_count} instance families, {region_count} regions, " + f"900+ instance types available" + ) + except Exception: + return False, "AWS: Error checking configuration" + + def main(): """Main function to generate cloud configurations.""" print("Cloud Provider Configuration Summary") @@ -121,8 +183,20 @@ def main(): print(f"⚠ {summary}") print() - # AWS (placeholder - not implemented) - print("⚠ AWS: Dynamic configuration not yet implemented") + # AWS - Generate Kconfig files + aws_generated = generate_aws_kconfig() + + # AWS - Get summary + success, summary = get_aws_summary() + if success: + print(f"✓ {summary}") + if aws_generated: + print(" Kconfig files generated successfully") + else: + print(" Warning: Failed to generate Kconfig files") + else: + print(f"⚠ {summary}") + print() # Azure (placeholder - not implemented) print("⚠ Azure: Dynamic configuration not yet implemented") -- 2.51.0