* [PATCH v4 0/8] aws: add dynamic kconfig support
@ 2025-09-17 0:34 Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 1/8] aws: prevent SSH key conflicts across multiple kdevops directories Luis Chamberlain
` (7 more replies)
0 siblings, 8 replies; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Luis Chamberlain
Chuck, in hopes to help move forward faster, I've taken your scripts
and tried to base the idea on it instead.
Changes on this v4:
- Adopt Chuck's script
- Make AWS ssh keys unique to avoid clashes
- Use a ~/.cache/ for json data
- Uber optimize for speed
This shaves down generation by *a lot*.
If you want to test:
https://github.com/linux-kdevops/kdevops/tree/mcgrof/aws-dynamic-cloud-config-v4
If you agree to it, please just merge :)
Chuck Lever (1):
terraform/aws: Add scripts to gather provider resource information
Luis Chamberlain (7):
aws: prevent SSH key conflicts across multiple kdevops directories
aws: add optimized Kconfig generator using Chuck's scripts
aws: integrate dynamic Kconfig generation with make targets
aws: add cloud billing support with make cloud-bill
aws: replace static Kconfig files with dynamically generated ones
aws: add GPU instance defconfigs for AI/ML workloads
docs: add documentation for dynamic cloud configuration
defconfigs/aws-gpu-g5-xlarge | 36 +
defconfigs/aws-gpu-p5-48xlarge | 36 +
docs/cloud-dynamic-config.md | 272 ++++++
playbooks/roles/gen_tfvars/defaults/main.yml | 2 +
.../templates/aws/terraform.tfvars.j2 | 3 +-
scripts/aws-costs.sh | 39 +
scripts/aws-parse-costs.py | 98 +++
scripts/aws_ssh_key_name.py | 165 ++++
scripts/dynamic-cloud-kconfig.Makefile | 60 +-
scripts/generate_cloud_configs.py | 78 +-
terraform/Kconfig.ssh | 14 +-
terraform/aws/Kconfig | 5 +-
terraform/aws/kconfigs/Kconfig.compute | 118 ---
terraform/aws/kconfigs/Kconfig.location | 679 ---------------
terraform/aws/kconfigs/Kconfig.ssh | 79 ++
.../aws/kconfigs/instance-types/Kconfig.c7a | 28 -
.../aws/kconfigs/instance-types/Kconfig.i4i | 33 -
.../aws/kconfigs/instance-types/Kconfig.im4gn | 25 -
.../kconfigs/instance-types/Kconfig.is4gen | 25 -
.../aws/kconfigs/instance-types/Kconfig.m5 | 48 --
.../aws/kconfigs/instance-types/Kconfig.m7a | 57 --
terraform/aws/scripts/aws_ami_info.py | 771 ++++++++++++++++++
terraform/aws/scripts/aws_regions_info.py | 371 +++++++++
terraform/aws/scripts/ec2_instance_info.py | 540 ++++++++++++
terraform/aws/scripts/generate_aws_kconfig.py | 474 +++++++++++
25 files changed, 3027 insertions(+), 1029 deletions(-)
create mode 100644 defconfigs/aws-gpu-g5-xlarge
create mode 100644 defconfigs/aws-gpu-p5-48xlarge
create mode 100644 docs/cloud-dynamic-config.md
create mode 100755 scripts/aws-costs.sh
create mode 100755 scripts/aws-parse-costs.py
create mode 100755 scripts/aws_ssh_key_name.py
delete mode 100644 terraform/aws/kconfigs/Kconfig.compute
delete mode 100644 terraform/aws/kconfigs/Kconfig.location
create mode 100644 terraform/aws/kconfigs/Kconfig.ssh
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.c7a
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.i4i
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.im4gn
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.is4gen
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.m5
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.m7a
create mode 100755 terraform/aws/scripts/aws_ami_info.py
create mode 100755 terraform/aws/scripts/aws_regions_info.py
create mode 100755 terraform/aws/scripts/ec2_instance_info.py
create mode 100755 terraform/aws/scripts/generate_aws_kconfig.py
--
2.51.0
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v4 1/8] aws: prevent SSH key conflicts across multiple kdevops directories
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
@ 2025-09-17 0:34 ` Luis Chamberlain
2025-09-17 3:36 ` Chuck Lever
2025-09-17 0:34 ` [PATCH v4 2/8] terraform/aws: Add scripts to gather provider resource information Luis Chamberlain
` (6 subsequent siblings)
7 siblings, 1 reply; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Luis Chamberlain
Problem:
When using kdevops in multiple directories for AWS deployments, all instances
share the same hardcoded SSH key name "kdevops_aws_key" in the AWS backend.
This creates a critical issue: if different directories use different SSH keys
but the same key name, terraform will silently replace the existing key in AWS,
immediately locking users out of any running instances that were using the
previous key.
Root Cause:
The AWS terraform configuration uses a fixed ssh_keyname variable that defaults
to "kdevops_aws_key". When terraform runs with a different public key content
but the same key name, it updates the aws_key_pair resource, replacing the
public key in AWS without warning.
Solution:
Implement directory-based unique SSH key naming for AWS, following the pattern
already established for Lambda Labs. Each kdevops directory now generates a
unique SSH key name based on its absolute path using SHA256 hashing.
Implementation Details:
1. Created scripts/aws_ssh_key_name.py:
- Generates unique key names like "kdevops-aws-projectname-a8999659"
- Uses SHA256 hash of the directory path (8 chars) for uniqueness
- Includes project name for human readability
- Provides options for key name, private key path, or public key path
2. Added terraform/aws/kconfigs/Kconfig.ssh:
- New configuration choice: unique keys (default) vs shared keys
- Unique mode: Each directory gets its own SSH key name automatically
- Shared mode: Legacy behavior with explicit warnings about risks
- Dynamic key name generation using the Python script
3. Updated terraform integration:
- Modified terraform/aws/Kconfig to include new SSH configuration
- Updated playbooks/roles/gen_tfvars/templates/aws/terraform.tfvars.j2
to use dynamic ssh_keyname variable
- Added defaults in playbooks/roles/gen_tfvars/defaults/main.yml
- Integrated with global SSH config in terraform/Kconfig.ssh
Benefits:
- Prevents accidental key overwrites in AWS backend
- No more lockouts from existing instances when using multiple directories
- Backward compatible - users can still use shared keys if needed
- Consistent with Lambda Labs implementation pattern
- Each directory's instances are isolated with their own SSH keys
Example:
Directory: /home/user/projects/kdevops
AWS Key Name: kdevops-aws-projects-kdevops-a8999659
Local Keys: ~/.ssh/kdevops-aws-projects-kdevops-a8999659[.pub]
This ensures that running kdevops from different directories will never
conflict, as each gets its own unique key name in both AWS and locally.
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
playbooks/roles/gen_tfvars/defaults/main.yml | 2 +
.../templates/aws/terraform.tfvars.j2 | 3 +-
scripts/aws_ssh_key_name.py | 165 ++++++++++++++++++
terraform/Kconfig.ssh | 14 +-
terraform/aws/Kconfig | 1 +
terraform/aws/kconfigs/Kconfig.ssh | 79 +++++++++
6 files changed, 257 insertions(+), 7 deletions(-)
create mode 100755 scripts/aws_ssh_key_name.py
create mode 100644 terraform/aws/kconfigs/Kconfig.ssh
diff --git a/playbooks/roles/gen_tfvars/defaults/main.yml b/playbooks/roles/gen_tfvars/defaults/main.yml
index c9e531bc..9957a078 100644
--- a/playbooks/roles/gen_tfvars/defaults/main.yml
+++ b/playbooks/roles/gen_tfvars/defaults/main.yml
@@ -27,6 +27,8 @@ terraform_aws_instance_type: "t2.micro"
terraform_aws_ebs_volumes_per_instance: "0"
terraform_aws_ebs_volume_size: 0
terraform_aws_ebs_volume_type: "gp3"
+terraform_aws_ssh_key_name: "kdevops-aws-fallback"
+terraform_aws_ssh_pubkey_file: "~/.ssh/kdevops_terraform.pub"
terraform_oci_assign_public_ip: false
terraform_oci_use_existing_vcn: false
diff --git a/playbooks/roles/gen_tfvars/templates/aws/terraform.tfvars.j2 b/playbooks/roles/gen_tfvars/templates/aws/terraform.tfvars.j2
index d880254b..4c976f13 100644
--- a/playbooks/roles/gen_tfvars/templates/aws/terraform.tfvars.j2
+++ b/playbooks/roles/gen_tfvars/templates/aws/terraform.tfvars.j2
@@ -14,7 +14,8 @@ aws_ebs_volume_iops = {{ terraform_aws_ebs_volume_iops }}
aws_ebs_volume_throughput = {{ terraform_aws_ebs_volume_throughput }}
{% endif %}
-ssh_config_pubkey_file = "{{ kdevops_terraform_ssh_config_pubkey_file }}"
+ssh_keyname = "{{ terraform_aws_ssh_key_name }}"
+ssh_config_pubkey_file = "{{ terraform_aws_ssh_pubkey_file | default(kdevops_terraform_ssh_config_pubkey_file) }}"
ssh_config_user = "{{ kdevops_terraform_ssh_config_user }}"
ssh_config = "{{ sshconfig }}"
diff --git a/scripts/aws_ssh_key_name.py b/scripts/aws_ssh_key_name.py
new file mode 100755
index 00000000..ff4bc967
--- /dev/null
+++ b/scripts/aws_ssh_key_name.py
@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+"""
+Generate a unique SSH key name for AWS based on the current directory.
+This ensures each kdevops instance uses its own SSH key to prevent conflicts.
+"""
+
+import hashlib
+import os
+import sys
+
+
+def get_directory_hash(path: str, length: int = 8) -> str:
+ """
+ Generate a short hash of the directory path.
+
+ Args:
+ path: Directory path to hash
+ length: Number of hex characters to use (default 8)
+
+ Returns:
+ Hex string of specified length
+ """
+ # Get the absolute path to ensure consistency
+ abs_path = os.path.abspath(path)
+
+ # Create SHA256 hash of the path
+ hash_obj = hashlib.sha256(abs_path.encode("utf-8"))
+
+ # Return first N characters of the hex digest
+ return hash_obj.hexdigest()[:length]
+
+
+def get_project_name(path: str) -> str:
+ """
+ Extract a meaningful project name from the path.
+
+ Args:
+ path: Directory path
+
+ Returns:
+ Project name derived from directory
+ """
+ abs_path = os.path.abspath(path)
+
+ # Get the last two directory components for context
+ # e.g., /home/user/projects/kdevops -> projects-kdevops
+ parts = abs_path.rstrip("/").split("/")
+
+ if len(parts) >= 2:
+ # Use last two directories
+ project_parts = parts[-2:]
+ # Filter out generic names
+ filtered = [
+ p
+ for p in project_parts
+ if p not in ["data", "home", "root", "usr", "var", "tmp", "cloud", "opt"]
+ ]
+ if filtered:
+ return "-".join(filtered)
+
+ # Fallback to just the last directory
+ return parts[-1] if parts else "kdevops"
+
+
+def generate_ssh_key_name(
+ prefix: str = "kdevops-aws", include_project: bool = True
+) -> str:
+ """
+ Generate a unique SSH key name for the current directory.
+
+ Args:
+ prefix: Prefix for the key name (default "kdevops-aws")
+ include_project: Include project name in the key (default True)
+
+ Returns:
+ Unique SSH key name like "kdevops-aws-projectname-a1b2c3d4"
+ """
+ # Find the kdevops root directory
+ script_dir = os.path.dirname(os.path.abspath(__file__))
+ kdevops_root = os.path.dirname(script_dir)
+
+ # Use the kdevops root directory for consistent key naming
+ dir_hash = get_directory_hash(kdevops_root)
+
+ parts = [prefix]
+
+ if include_project:
+ project = get_project_name(kdevops_root)
+ # Limit project name length and sanitize
+ project = project.replace("_", "-").replace(".", "-")[:20]
+ parts.append(project)
+
+ parts.append(dir_hash)
+
+ # Create the key name
+ key_name = "-".join(parts)
+
+ # Ensure it's a valid AWS key name (alphanumeric, hyphens, underscores)
+ key_name = "".join(c if c.isalnum() or c in "-_" else "-" for c in key_name)
+
+ # Remove multiple consecutive hyphens
+ while "--" in key_name:
+ key_name = key_name.replace("--", "-")
+
+ # AWS key pair names have a 255 character limit, but let's be reasonable
+ if len(key_name) > 60:
+ # Keep prefix and full hash for uniqueness
+ key_name = f"{prefix}-{dir_hash}"
+
+ return key_name.strip("-")
+
+
+def get_ssh_key_path(key_type: str = "") -> str:
+ """
+ Generate the local SSH key file path based on directory.
+
+ Args:
+ key_type: Either "" for private key, ".pub" for public key
+
+ Returns:
+ Path to SSH key file
+ """
+ key_name = generate_ssh_key_name()
+ return os.path.expanduser(f"~/.ssh/{key_name}{key_type}")
+
+
+def main():
+ """Main entry point."""
+ if len(sys.argv) > 1:
+ if sys.argv[1] == "--help" or sys.argv[1] == "-h":
+ print("Usage: aws_ssh_key_name.py [--simple|--path|--pubkey-path]")
+ print()
+ print("Generate a unique SSH key name based on current directory.")
+ print()
+ print("Options:")
+ print(" --simple Generate simple name without project context")
+ print(" --path Output full path to private key file")
+ print(" --pubkey-path Output full path to public key file")
+ print(" --help Show this help message")
+ print()
+ print("Examples:")
+ print(" Default: kdevops-aws-projectname-a1b2c3d4")
+ print(" Simple: kdevops-aws-a1b2c3d4")
+ print(" Path: /home/user/.ssh/kdevops-aws-projectname-a1b2c3d4")
+ print(
+ " Pubkey Path: /home/user/.ssh/kdevops-aws-projectname-a1b2c3d4.pub"
+ )
+ sys.exit(0)
+ elif sys.argv[1] == "--simple":
+ print(generate_ssh_key_name(include_project=False))
+ elif sys.argv[1] == "--path":
+ print(get_ssh_key_path())
+ elif sys.argv[1] == "--pubkey-path":
+ print(get_ssh_key_path(".pub"))
+ else:
+ print(f"Unknown option: {sys.argv[1]}", file=sys.stderr)
+ sys.exit(1)
+ else:
+ print(generate_ssh_key_name())
+
+
+if __name__ == "__main__":
+ main()
diff --git a/terraform/Kconfig.ssh b/terraform/Kconfig.ssh
index 8a19d7c5..8f762cde 100644
--- a/terraform/Kconfig.ssh
+++ b/terraform/Kconfig.ssh
@@ -27,26 +27,28 @@ config TERRAFORM_SSH_CONFIG_USER
config TERRAFORM_SSH_CONFIG_PUBKEY_FILE
string "The ssh public key to use to log in"
default "~/.ssh/kdevops_terraform_$(shell, echo $(TOPDIR_PATH) | sha256sum | cut -c1-8).pub" if TERRAFORM_LAMBDALABS
+ default TERRAFORM_AWS_SSH_PUBKEY_FILE if TERRAFORM_AWS && TERRAFORM_AWS_SSH_KEY_UNIQUE
default "~/.ssh/kdevops_terraform.pub"
help
The ssh public key which will be pegged onto the systems's
~/.ssh/authorized_keys file so you can log in.
- For Lambda Labs, the key path is made unique per directory by appending
- the directory checksum to avoid conflicts when running multiple kdevops
- instances.
+ For Lambda Labs and AWS (when using unique keys), the key path is made
+ unique per directory by appending the directory checksum to avoid
+ conflicts when running multiple kdevops instances.
config TERRAFORM_SSH_CONFIG_PRIVKEY_FILE
string "The ssh private key file for authentication"
default "~/.ssh/kdevops_terraform_$(shell, echo $(TOPDIR_PATH) | sha256sum | cut -c1-8)" if TERRAFORM_LAMBDALABS
+ default TERRAFORM_AWS_SSH_PRIVKEY_FILE if TERRAFORM_AWS && TERRAFORM_AWS_SSH_KEY_UNIQUE
default "~/.ssh/kdevops_terraform"
help
The ssh private key file used for authenticating to the systems.
This should correspond to the public key specified above.
- For Lambda Labs, the key path is made unique per directory by appending
- the directory checksum to avoid conflicts when running multiple kdevops
- instances.
+ For Lambda Labs and AWS (when using unique keys), the key path is made
+ unique per directory by appending the directory checksum to avoid
+ conflicts when running multiple kdevops instances.
config TERRAFORM_SSH_CONFIG_GENKEY
bool "Should we create a new random key for you?"
diff --git a/terraform/aws/Kconfig b/terraform/aws/Kconfig
index 557f3a08..d4015cda 100644
--- a/terraform/aws/Kconfig
+++ b/terraform/aws/Kconfig
@@ -11,6 +11,7 @@ source "terraform/aws/kconfigs/Kconfig.storage"
endmenu
menu "Identity & Access"
source "terraform/aws/kconfigs/Kconfig.identity"
+source "terraform/aws/kconfigs/Kconfig.ssh"
endmenu
endif # TERRAFORM_AWS
diff --git a/terraform/aws/kconfigs/Kconfig.ssh b/terraform/aws/kconfigs/Kconfig.ssh
new file mode 100644
index 00000000..3b62be48
--- /dev/null
+++ b/terraform/aws/kconfigs/Kconfig.ssh
@@ -0,0 +1,79 @@
+# AWS SSH Key Management Configuration
+
+choice
+ prompt "AWS SSH key management strategy"
+ default TERRAFORM_AWS_SSH_KEY_UNIQUE
+ help
+ Choose how SSH keys are managed for AWS instances.
+
+ Unique key (recommended): Each kdevops project directory gets its own
+ SSH key, preventing conflicts when using multiple kdevops instances.
+
+ Shared key: Use the same key name across all projects (less secure,
+ risk of overwriting keys and locking out existing instances).
+
+config TERRAFORM_AWS_SSH_KEY_UNIQUE
+ bool "Use unique SSH key per project directory (recommended)"
+ help
+ Generate a unique SSH key name for each kdevops project directory.
+ The key name will be based on the directory path, ensuring no conflicts
+ between different kdevops deployments.
+
+ Example: kdevops-aws-projectname-a1b2c3d4
+
+ This prevents accidentally overwriting SSH keys in AWS when using
+ kdevops in multiple directories, which would lock you out of
+ existing instances.
+
+config TERRAFORM_AWS_SSH_KEY_SHARED
+ bool "Use shared SSH key name (legacy/risky)"
+ help
+ Use a fixed SSH key name that you specify. This is less secure
+ and can lead to accidental key overwrites if you use kdevops in
+ multiple directories with different keys.
+
+ WARNING: Using the same key name with different public keys will
+ replace the key in AWS and lock you out of existing instances!
+
+endchoice
+
+config TERRAFORM_AWS_SSH_KEY_NAME_CUSTOM
+ string "Custom SSH key name (only for shared mode)"
+ default "kdevops_aws_key"
+ depends on TERRAFORM_AWS_SSH_KEY_SHARED
+ help
+ Specify the custom SSH key name to use when in shared mode.
+
+ WARNING: If this key already exists in AWS with a different
+ public key, it will be replaced, potentially locking you out
+ of existing instances!
+
+config TERRAFORM_AWS_SSH_KEY_NAME
+ string
+ output yaml
+ default $(shell, python3 scripts/aws_ssh_key_name.py 2>/dev/null || echo "kdevops-aws-fallback") if TERRAFORM_AWS_SSH_KEY_UNIQUE
+ default TERRAFORM_AWS_SSH_KEY_NAME_CUSTOM if TERRAFORM_AWS_SSH_KEY_SHARED
+ help
+ The actual SSH key name that will be used in AWS.
+ When using unique mode, this is automatically generated based on
+ the current directory path.
+
+config TERRAFORM_AWS_SSH_PUBKEY_FILE
+ string
+ output yaml
+ default $(shell, python3 scripts/aws_ssh_key_name.py --pubkey-path 2>/dev/null || echo "~/.ssh/kdevops_terraform.pub") if TERRAFORM_AWS_SSH_KEY_UNIQUE
+ default "~/.ssh/kdevops_terraform.pub" if TERRAFORM_AWS_SSH_KEY_SHARED
+ help
+ Path to the SSH public key file.
+ When using unique mode, this path includes the directory-specific
+ key name to avoid conflicts.
+
+config TERRAFORM_AWS_SSH_PRIVKEY_FILE
+ string
+ output yaml
+ default $(shell, python3 scripts/aws_ssh_key_name.py --path 2>/dev/null || echo "~/.ssh/kdevops_terraform") if TERRAFORM_AWS_SSH_KEY_UNIQUE
+ default "~/.ssh/kdevops_terraform" if TERRAFORM_AWS_SSH_KEY_SHARED
+ help
+ Path to the SSH private key file.
+ When using unique mode, this path includes the directory-specific
+ key name to avoid conflicts.
--
2.51.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 2/8] terraform/aws: Add scripts to gather provider resource information
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 1/8] aws: prevent SSH key conflicts across multiple kdevops directories Luis Chamberlain
@ 2025-09-17 0:34 ` Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 3/8] aws: add optimized Kconfig generator using Chuck's scripts Luis Chamberlain
` (5 subsequent siblings)
7 siblings, 0 replies; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Chuck Lever, Luis Chamberlain
From: Chuck Lever <chuck.lever@oracle.com>
Luis suggested I try my hand... so here goes.
I prompted Claude for a few ideas about how to grab information
about AMIs, instance types, and regions/AVs. The script output is
either JSON, a human-readable table, or CSV. Additional scripts
(not provided here) would then capture this output and convert it
to Kconfig menus.
This is not at all complete. Just another way to approach these
tasks, for comparison. I learned why the region information is
needed for querying instance types, and that it looks like we
won't get clear and reliable pricing information from AWS, for
various reasons.
I would say that this code works and has a pleasant UX but is
still more complex than we might want to carry in kdevops in the
long run. I'm interested in seeing some human or AI effort to
simplifying these scripts further.
And note that the instance type information is always based on
what's available in the queried region and what the credentialed
user has permission to see. Therefore:
- out of the shrink-wrap, kdevops might provide some sensible
generic default menu selections
- a power user might need to run this before selecting the cloud
resources they want to use
So it better be damn simple and damn reliable. :-) And we probably
need to be very careful before changing the in-tree menus that
are committed to the kdevops repo... They might need to continue
to be hand-rolled. Or we just have some fixed JSON source that
generates the sensible default menus.
Posting this for thoughts and opinions.
Generated-by: Claude Sonnet 4
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
terraform/aws/scripts/aws_ami_info.py | 771 +++++++++++++++++++++
terraform/aws/scripts/aws_regions_info.py | 371 ++++++++++
terraform/aws/scripts/ec2_instance_info.py | 540 +++++++++++++++
3 files changed, 1682 insertions(+)
create mode 100755 terraform/aws/scripts/aws_ami_info.py
create mode 100755 terraform/aws/scripts/aws_regions_info.py
create mode 100755 terraform/aws/scripts/ec2_instance_info.py
diff --git a/terraform/aws/scripts/aws_ami_info.py b/terraform/aws/scripts/aws_ami_info.py
new file mode 100755
index 00000000..d9ea8bea
--- /dev/null
+++ b/terraform/aws/scripts/aws_ami_info.py
@@ -0,0 +1,771 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: MIT
+
+import boto3
+import json
+import sys
+import argparse
+import os
+import re
+from collections import defaultdict
+from configparser import ConfigParser
+from botocore.exceptions import ClientError, NoCredentialsError
+
+
+def get_aws_default_region():
+ """
+ Get the default AWS region from ~/.aws/config file.
+
+ Returns:
+ str: Default region or 'us-east-1' if not found
+ """
+ config_path = os.path.expanduser("~/.aws/config")
+
+ if os.path.exists(config_path):
+ try:
+ config = ConfigParser()
+ config.read(config_path)
+
+ # Check for default profile region
+ if "default" in config:
+ return config["default"].get("region", "us-east-1")
+
+ # Check for profile default section
+ if "profile default" in config:
+ return config["profile default"].get("region", "us-east-1")
+
+ except Exception as e:
+ print(f"Warning: Error reading AWS config file: {e}", file=sys.stderr)
+
+ return "us-east-1"
+
+
+def get_known_ami_owners():
+ """
+ Get dictionary of well-known AMI owners that provide Linux images.
+
+ Returns:
+ dict: Dictionary of owner information
+ """
+ return {
+ "amazon": {
+ "owner_id": "137112412989",
+ "owner_name": "Amazon",
+ "description": "Amazon Linux AMIs",
+ "search_patterns": [
+ r"al2023-ami-.*", # Amazon Linux 2023
+ r"amzn2-ami-.*", # Amazon Linux 2
+ r"amzn-ami-.*", # Amazon Linux 1
+ ],
+ },
+ "ubuntu": {
+ "owner_id": "099720109477",
+ "owner_name": "Canonical",
+ "description": "Ubuntu AMIs",
+ "search_patterns": [
+ r"ubuntu/images/.*ubuntu-.*", # All Ubuntu images
+ ],
+ },
+ "redhat": {
+ "owner_id": "309956199498",
+ "owner_name": "Red Hat",
+ "description": "Red Hat Enterprise Linux AMIs",
+ "search_patterns": [
+ r"RHEL-.*", # All RHEL versions
+ ],
+ },
+ "suse": {
+ "owner_id": "013907871322",
+ "owner_name": "SUSE",
+ "description": "SUSE Linux Enterprise AMIs",
+ "search_patterns": [
+ r"suse-sles-.*",
+ r"suse-sle-.*",
+ ],
+ },
+ "debian": {
+ "owner_id": "136693071363",
+ "owner_name": "Debian",
+ "description": "Debian GNU/Linux AMIs",
+ "search_patterns": [
+ r"debian-.*",
+ ],
+ },
+ "centos": {
+ "owner_id": "125523088429",
+ "owner_name": "CentOS",
+ "description": "CentOS Linux AMIs (Legacy)",
+ "search_patterns": [
+ r"CentOS.*",
+ ],
+ },
+ "rocky": {
+ "owner_id": "792107900819",
+ "owner_name": "Rocky Linux",
+ "description": "Rocky Linux AMIs",
+ "search_patterns": [
+ r"Rocky-.*",
+ ],
+ },
+ "almalinux": {
+ "owner_id": "764336703387",
+ "owner_name": "AlmaLinux",
+ "description": "AlmaLinux AMIs",
+ "search_patterns": [
+ r"AlmaLinux.*",
+ ],
+ },
+ "fedora": {
+ "owner_id": "125523088429",
+ "owner_name": "Fedora Project",
+ "description": "Fedora Linux Cloud AMIs",
+ "search_patterns": [
+ r"Fedora-Cloud-.*",
+ r"Fedora-.*",
+ ],
+ },
+ "oracle": {
+ "owner_id": "131827586825",
+ "owner_name": "Oracle",
+ "description": "Oracle Linux AMIs",
+ "search_patterns": [
+ r"OL.*-.*",
+ ],
+ },
+ }
+
+
+def discover_ami_patterns(
+ owner_id,
+ owner_name,
+ search_patterns,
+ region="us-east-1",
+ quiet=False,
+ max_results=1000,
+):
+ """
+ Dynamically discover AMI patterns by scanning available AMIs for an owner.
+
+ Args:
+ owner_id (str): AWS owner account ID
+ owner_name (str): Human readable owner name
+ search_patterns (list): Regex patterns to filter AMI names
+ region (str): AWS region to query
+ quiet (bool): Suppress debug messages
+ max_results (int): Maximum AMIs to scan
+
+ Returns:
+ dict: Dictionary of discovered AMI patterns and examples
+ """
+ try:
+ if not quiet:
+ print(
+ f"Discovering AMI patterns for {owner_name} in {region}...",
+ file=sys.stderr,
+ )
+
+ ec2_client = boto3.client("ec2", region_name=region)
+
+ # Get all AMIs from this owner
+ all_amis = []
+ paginator = ec2_client.get_paginator("describe_images")
+
+ for page in paginator.paginate(
+ Owners=[owner_id],
+ Filters=[
+ {"Name": "state", "Values": ["available"]},
+ {"Name": "image-type", "Values": ["machine"]},
+ ],
+ ):
+ all_amis.extend(page["Images"])
+ if len(all_amis) >= max_results:
+ all_amis = all_amis[:max_results]
+ break
+
+ if not quiet:
+ print(f"Found {len(all_amis)} total AMIs for {owner_name}", file=sys.stderr)
+
+ # Filter AMIs by search patterns
+ matching_amis = []
+ for ami in all_amis:
+ ami_name = ami.get("Name", "")
+ for pattern in search_patterns:
+ if re.match(pattern, ami_name, re.IGNORECASE):
+ matching_amis.append(ami)
+ break
+
+ if not quiet:
+ print(
+ f"Found {len(matching_amis)} matching AMIs after pattern filtering",
+ file=sys.stderr,
+ )
+
+ # Group AMIs by detected patterns
+ pattern_groups = defaultdict(list)
+
+ for ami in matching_amis:
+ ami_name = ami.get("Name", "")
+ group_key = classify_ami_name(ami_name, owner_name)
+
+ ami_info = {
+ "ami_id": ami["ImageId"],
+ "name": ami_name,
+ "description": ami.get("Description", ""),
+ "creation_date": ami["CreationDate"],
+ "architecture": ami.get("Architecture", "Unknown"),
+ "virtualization_type": ami.get("VirtualizationType", "Unknown"),
+ "root_device_type": ami.get("RootDeviceType", "Unknown"),
+ "platform_details": ami.get("PlatformDetails", "Unknown"),
+ }
+
+ pattern_groups[group_key].append(ami_info)
+
+ # Sort each group by creation date (newest first) and generate patterns
+ discovered_patterns = {}
+ for group_key, amis in pattern_groups.items():
+ # Sort by creation date, newest first
+ sorted_amis = sorted(amis, key=lambda x: x["creation_date"], reverse=True)
+
+ # Generate Terraform-compatible filter pattern
+ terraform_pattern = generate_terraform_pattern(
+ group_key, sorted_amis[:5]
+ ) # Use top 5 for pattern analysis
+
+ discovered_patterns[group_key] = {
+ "display_name": group_key,
+ "ami_count": len(sorted_amis),
+ "latest_ami": sorted_amis[0] if sorted_amis else None,
+ "sample_amis": sorted_amis[:3], # Show 3 most recent
+ "terraform_filter": terraform_pattern,
+ "terraform_example": generate_terraform_example(
+ group_key, terraform_pattern, owner_id
+ ),
+ }
+
+ return discovered_patterns
+
+ except NoCredentialsError:
+ print(
+ "Error: AWS credentials not found. Please configure your credentials.",
+ file=sys.stderr,
+ )
+ return {}
+ except ClientError as e:
+ print(f"AWS API Error: {e}", file=sys.stderr)
+ return {}
+ except Exception as e:
+ print(f"Unexpected error: {e}", file=sys.stderr)
+ return {}
+
+
+def classify_ami_name(ami_name, owner_name):
+ """
+ Classify an AMI name into a logical group for pattern generation.
+
+ Args:
+ ami_name (str): AMI name
+ owner_name (str): Owner name for context
+
+ Returns:
+ str: Classification key
+ """
+ ami_lower = ami_name.lower()
+
+ # Amazon Linux patterns
+ if "al2023" in ami_lower:
+ return "Amazon Linux 2023"
+ elif "amzn2" in ami_lower:
+ return "Amazon Linux 2"
+ elif "amzn-ami" in ami_lower:
+ return "Amazon Linux 1"
+
+ # Ubuntu patterns
+ elif "ubuntu" in ami_lower:
+ if "noble" in ami_lower or "24.04" in ami_lower:
+ return "Ubuntu 24.04 LTS (Noble)"
+ elif "jammy" in ami_lower or "22.04" in ami_lower:
+ return "Ubuntu 22.04 LTS (Jammy)"
+ elif "focal" in ami_lower or "20.04" in ami_lower:
+ return "Ubuntu 20.04 LTS (Focal)"
+ elif "bionic" in ami_lower or "18.04" in ami_lower:
+ return "Ubuntu 18.04 LTS (Bionic)"
+ else:
+ # Extract version number if available
+ version_match = re.search(r"(\d+\.\d+)", ami_name)
+ if version_match:
+ return f"Ubuntu {version_match.group(1)}"
+ return "Ubuntu (Other)"
+
+ # RHEL patterns
+ elif "rhel" in ami_lower:
+ if re.search(r"rhel-?10", ami_lower):
+ return "RHEL 10"
+ elif re.search(r"rhel-?9", ami_lower):
+ return "RHEL 9"
+ elif re.search(r"rhel-?8", ami_lower):
+ return "RHEL 8"
+ elif re.search(r"rhel-?7", ami_lower):
+ return "RHEL 7"
+ else:
+ version_match = re.search(r"rhel-?(\d+)", ami_lower)
+ if version_match:
+ return f"RHEL {version_match.group(1)}"
+ return "RHEL (Other)"
+
+ # Rocky Linux patterns
+ elif "rocky" in ami_lower:
+ version_match = re.search(r"rocky-(\d+)", ami_lower)
+ if version_match:
+ return f"Rocky Linux {version_match.group(1)}"
+ return "Rocky Linux"
+
+ # AlmaLinux patterns
+ elif "almalinux" in ami_lower:
+ version_match = re.search(r"(\d+)", ami_name)
+ if version_match:
+ return f"AlmaLinux {version_match.group(1)}"
+ return "AlmaLinux"
+
+ # Debian patterns
+ elif "debian" in ami_lower:
+ if re.search(r"debian-?12", ami_lower) or "bookworm" in ami_lower:
+ return "Debian 12 (Bookworm)"
+ elif re.search(r"debian-?11", ami_lower) or "bullseye" in ami_lower:
+ return "Debian 11 (Bullseye)"
+ elif re.search(r"debian-?10", ami_lower) or "buster" in ami_lower:
+ return "Debian 10 (Buster)"
+ else:
+ version_match = re.search(r"debian-?(\d+)", ami_lower)
+ if version_match:
+ return f"Debian {version_match.group(1)}"
+ return "Debian (Other)"
+
+ # SUSE patterns
+ elif "suse" in ami_lower or "sles" in ami_lower:
+ version_match = re.search(r"(\d+)", ami_name)
+ if version_match:
+ return f"SUSE Linux Enterprise {version_match.group(1)}"
+ return "SUSE Linux Enterprise"
+
+ # CentOS patterns
+ elif "centos" in ami_lower:
+ version_match = re.search(r"(\d+)", ami_name)
+ if version_match:
+ return f"CentOS {version_match.group(1)}"
+ return "CentOS"
+
+ # Fedora patterns
+ elif "fedora" in ami_lower:
+ version_match = re.search(r"fedora-.*?(\d+)", ami_lower)
+ if version_match:
+ return f"Fedora {version_match.group(1)}"
+ return "Fedora"
+
+ # Oracle Linux patterns
+ elif ami_lower.startswith("ol"):
+ version_match = re.search(r"ol(\d+)", ami_lower)
+ if version_match:
+ return f"Oracle Linux {version_match.group(1)}"
+ return "Oracle Linux"
+
+ # Default: use the owner name
+ return f"{owner_name} (Other)"
+
+
+def generate_terraform_pattern(group_key, sample_amis):
+ """
+ Generate a Terraform-compatible filter pattern from sample AMIs.
+
+ Args:
+ group_key (str): Classification key
+ sample_amis (list): List of sample AMI info
+
+ Returns:
+ str: Terraform filter pattern
+ """
+ if not sample_amis:
+ return ""
+
+ # Analyze common patterns in AMI names
+ names = [ami["name"] for ami in sample_amis]
+
+ # Find the longest common prefix and suffix patterns
+ if len(names) == 1:
+ # Single AMI - create a pattern that matches similar names
+ name = names[0]
+ # Replace specific dates/versions with wildcards
+ pattern = re.sub(r"\d{4}-\d{2}-\d{2}", "*", name) # Replace dates
+ pattern = re.sub(r"-\d+\.\d+\.\d+", "-*", pattern) # Replace version numbers
+ pattern = re.sub(
+ r"_\d+\.\d+\.\d+", "_*", pattern
+ ) # Replace version numbers with underscores
+ return pattern
+
+ # Multiple AMIs - find common pattern
+ common_parts = []
+ min_len = min(len(name) for name in names)
+
+ # Find common prefix
+ prefix_len = 0
+ for i in range(min_len):
+ chars = set(name[i] for name in names)
+ if len(chars) == 1:
+ prefix_len = i + 1
+ else:
+ break
+
+ if prefix_len > 0:
+ prefix = names[0][:prefix_len]
+ return f"{prefix}*"
+
+ # If no common prefix, try to extract the base pattern
+ first_name = names[0]
+ # Replace numbers and dates with wildcards
+ pattern = re.sub(r"\d{8}", "*", first_name) # Replace 8-digit dates
+ pattern = re.sub(r"\d{4}-\d{2}-\d{2}", "*", pattern) # Replace ISO dates
+ pattern = re.sub(r"-\d+\.\d+\.\d+", "-*", pattern) # Replace version numbers
+ pattern = re.sub(
+ r"_\d+\.\d+\.\d+", "_*", pattern
+ ) # Replace version numbers with underscores
+
+ return pattern
+
+
+def generate_terraform_example(group_key, filter_pattern, owner_id):
+ """
+ Generate a complete Terraform example.
+
+ Args:
+ group_key (str): Classification key
+ filter_pattern (str): Filter pattern
+ owner_id (str): AWS owner account ID
+
+ Returns:
+ str: Complete Terraform data source example
+ """
+ # Create a safe resource name
+ resource_name = re.sub(r"[^a-zA-Z0-9_]", "_", group_key.lower())
+ resource_name = re.sub(r"_+", "_", resource_name) # Remove multiple underscores
+ resource_name = resource_name.strip("_") # Remove leading/trailing underscores
+
+ if not filter_pattern:
+ filter_pattern = "*"
+
+ terraform_code = f"""data "aws_ami" "{resource_name}" {{
+ most_recent = true
+ owners = ["{owner_id}"]
+ filter {{
+ name = "name"
+ values = ["{filter_pattern}"]
+ }}
+ filter {{
+ name = "architecture"
+ values = ["x86_64"]
+ }}
+ filter {{
+ name = "virtualization-type"
+ values = ["hvm"]
+ }}
+ filter {{
+ name = "state"
+ values = ["available"]
+ }}
+}}"""
+
+ return terraform_code
+
+
+def get_owner_ami_info(owner_key, region="us-east-1", quiet=False):
+ """
+ Get comprehensive AMI information for a specific owner.
+
+ Args:
+ owner_key (str): Owner key (e.g., 'amazon', 'ubuntu')
+ region (str): AWS region to query
+ quiet (bool): Suppress debug messages
+
+ Returns:
+ dict: Owner information with discovered AMI patterns
+ """
+ known_owners = get_known_ami_owners()
+
+ if owner_key not in known_owners:
+ return None
+
+ owner_info = known_owners[owner_key].copy()
+
+ # Discover actual AMI patterns
+ discovered_patterns = discover_ami_patterns(
+ owner_info["owner_id"],
+ owner_info["owner_name"],
+ owner_info["search_patterns"],
+ region,
+ quiet,
+ )
+
+ owner_info["discovered_patterns"] = discovered_patterns
+ owner_info["total_pattern_count"] = len(discovered_patterns)
+
+ return owner_info
+
+
+def parse_arguments():
+ """Parse command line arguments."""
+ parser = argparse.ArgumentParser(
+ description="Get AWS AMI owner information and Terraform filter examples",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog="""
+Examples:
+ python %(prog)s --owners
+ python %(prog)s amazon
+ python %(prog)s ubuntu --format json
+ python %(prog)s --owners --format csv
+ python %(prog)s redhat --region eu-west-1
+ """,
+ )
+
+ parser.add_argument(
+ "owner_key",
+ nargs="?", # Make owner_key optional when using --owners
+ help="AMI owner key (e.g., amazon, ubuntu, redhat, debian, suse, centos, rocky, almalinux)",
+ )
+
+ parser.add_argument(
+ "--region", "-r", help="AWS region (default: from ~/.aws/config or us-east-1)"
+ )
+
+ parser.add_argument(
+ "--format",
+ "-f",
+ choices=["table", "json", "csv", "terraform"],
+ default="table",
+ help="Output format (default: table)",
+ )
+
+ parser.add_argument(
+ "--quiet", "-q", action="store_true", help="Suppress informational messages"
+ )
+
+ parser.add_argument(
+ "--debug", "-d", action="store_true", help="Enable debug output"
+ )
+
+ parser.add_argument(
+ "--owners", action="store_true", help="List all known AMI owners"
+ )
+
+ parser.add_argument(
+ "--max-results",
+ type=int,
+ default=1000,
+ help="Maximum number of AMIs to scan per owner (default: 1000)",
+ )
+
+ return parser.parse_args()
+
+
+def output_owners_table(owners, quiet=False):
+ """Output AMI owners in table format."""
+ if not quiet:
+ print(f"Known Linux AMI owners ({len(owners)}):\n")
+
+ # Print header
+ print(f"{'Owner Key':<12} {'Owner Name':<15} {'Owner ID':<15} {'Description':<30}")
+ print("-" * 75)
+
+ # Sort owners by key
+ for owner_key in sorted(owners.keys()):
+ owner = owners[owner_key]
+ print(
+ f"{owner_key:<12} "
+ f"{owner['owner_name']:<15} "
+ f"{owner['owner_id']:<15} "
+ f"{owner['description']:<30}"
+ )
+
+
+def output_owners_json(owners):
+ """Output owners in JSON format."""
+ print(json.dumps(owners, indent=2))
+
+
+def output_owners_csv(owners):
+ """Output owners in CSV format."""
+ print("owner_key,owner_name,owner_id,description")
+
+ for owner_key in sorted(owners.keys()):
+ owner = owners[owner_key]
+ description = owner["description"].replace(",", ";") # Avoid CSV issues
+ print(f"{owner_key},{owner['owner_name']},{owner['owner_id']},{description}")
+
+
+def output_owner_table(owner_info, quiet=False):
+ """Output owner AMI information in table format."""
+ if not quiet:
+ print(
+ f"AMI Information for {owner_info['owner_name']} (Owner ID: {owner_info['owner_id']})"
+ )
+ print(f"Description: {owner_info['description']}")
+ print(f"Found {owner_info['total_pattern_count']} AMI pattern groups\n")
+
+ if not owner_info["discovered_patterns"]:
+ print("No AMI patterns discovered for this owner in the specified region.")
+ return
+
+ for pattern_name, pattern_info in sorted(owner_info["discovered_patterns"].items()):
+ print(f"Pattern: {pattern_name}")
+ print(f" AMI Count: {pattern_info['ami_count']}")
+ print(f" Filter Pattern: {pattern_info['terraform_filter']}")
+
+ # Show latest AMI
+ if pattern_info["latest_ami"]:
+ latest = pattern_info["latest_ami"]
+ print(f" Latest AMI: {latest['ami_id']} ({latest['creation_date'][:10]})")
+ print(f" Architecture: {latest['architecture']}")
+
+ # Show sample AMIs
+ if pattern_info["sample_amis"]:
+ print(f" Sample AMIs:")
+ for ami in pattern_info["sample_amis"]:
+ print(
+ f" {ami['ami_id']} - {ami['name'][:60]}{'...' if len(ami['name']) > 60 else ''}"
+ )
+ print(
+ f" Created: {ami['creation_date'][:10]} | Arch: {ami['architecture']} | Virt: {ami['virtualization_type']}"
+ )
+
+ print() # Empty line between patterns
+
+
+def output_owner_json(owner_info):
+ """Output owner information in JSON format."""
+
+ # Convert datetime objects to strings for JSON serialization
+ def json_serializer(obj):
+ if hasattr(obj, "isoformat"):
+ return obj.isoformat()
+ return str(obj)
+
+ print(json.dumps(owner_info, indent=2, default=json_serializer))
+
+
+def output_owner_csv(owner_info):
+ """Output owner information in CSV format."""
+ print(
+ "pattern_name,ami_count,filter_pattern,latest_ami_id,latest_ami_name,creation_date,architecture"
+ )
+
+ for pattern_name, pattern_info in sorted(owner_info["discovered_patterns"].items()):
+ latest = pattern_info.get("latest_ami", {})
+ ami_id = latest.get("ami_id", "")
+ ami_name = latest.get("name", "").replace(",", ";") # Avoid CSV issues
+ creation_date = (
+ latest.get("creation_date", "")[:10] if latest.get("creation_date") else ""
+ )
+ architecture = latest.get("architecture", "")
+
+ print(
+ f"{pattern_name},{pattern_info['ami_count']},{pattern_info['terraform_filter']},{ami_id},{ami_name},{creation_date},{architecture}"
+ )
+
+
+def output_owner_terraform(owner_info):
+ """Output owner information as Terraform examples."""
+ print(f"# Terraform aws_ami data source examples for {owner_info['owner_name']}")
+ print(f"# Owner ID: {owner_info['owner_id']}")
+ print(f"# {owner_info['description']}")
+ print(f"# Found {owner_info['total_pattern_count']} AMI pattern groups")
+ print()
+
+ for pattern_name, pattern_info in sorted(owner_info["discovered_patterns"].items()):
+ print(f"# {pattern_name} ({pattern_info['ami_count']} AMIs available)")
+ if pattern_info["latest_ami"]:
+ print(
+ f"# Latest: {pattern_info['latest_ami']['ami_id']} ({pattern_info['latest_ami']['creation_date'][:10]})"
+ )
+ print(pattern_info["terraform_example"])
+ print()
+
+
+def main():
+ """Main function to run the program."""
+ args = parse_arguments()
+
+ # Determine region
+ if args.region:
+ region = args.region
+ else:
+ region = get_aws_default_region()
+
+ # Handle --owners option
+ if args.owners:
+ owners = get_known_ami_owners()
+ if args.format == "json":
+ output_owners_json(owners)
+ elif args.format == "csv":
+ output_owners_csv(owners)
+ else: # table format (terraform not applicable for owners list)
+ output_owners_table(owners, args.quiet)
+ return
+
+ # Require owner_key if not using --owners
+ if not args.owner_key:
+ print(
+ "Error: owner_key is required unless using --owners option", file=sys.stderr
+ )
+ print("Use --owners to list all available AMI owners", file=sys.stderr)
+ sys.exit(1)
+
+ # Validate owner key
+ known_owners = get_known_ami_owners()
+ if args.owner_key not in known_owners:
+ print(f"Error: Unknown owner key '{args.owner_key}'", file=sys.stderr)
+ print(
+ f"Available owners: {', '.join(sorted(known_owners.keys()))}",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ if not args.quiet:
+ print(
+ f"Discovering AMI patterns for {args.owner_key} in {region}...",
+ file=sys.stderr,
+ )
+ print(f"This may take a moment as we scan available AMIs...", file=sys.stderr)
+
+ # Get owner information with dynamic discovery
+ owner_info = get_owner_ami_info(
+ args.owner_key, region, args.quiet or not args.debug
+ )
+
+ if not owner_info:
+ print(
+ f"Could not retrieve AMI information for owner '{args.owner_key}'.",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ if not owner_info.get("discovered_patterns"):
+ print(
+ f"No AMI patterns discovered for owner '{args.owner_key}' in region {region}.",
+ file=sys.stderr,
+ )
+ print(
+ "This may be because the owner has no AMIs in this region or the search patterns need adjustment.",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ # Output results in specified format
+ if args.format == "json":
+ output_owner_json(owner_info)
+ elif args.format == "csv":
+ output_owner_csv(owner_info)
+ elif args.format == "terraform":
+ output_owner_terraform(owner_info)
+ else: # table format
+ output_owner_table(owner_info, args.quiet)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/terraform/aws/scripts/aws_regions_info.py b/terraform/aws/scripts/aws_regions_info.py
new file mode 100755
index 00000000..00b88e0a
--- /dev/null
+++ b/terraform/aws/scripts/aws_regions_info.py
@@ -0,0 +1,371 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: MIT
+
+import boto3
+import json
+import sys
+import argparse
+import os
+from configparser import ConfigParser
+from botocore.exceptions import ClientError, NoCredentialsError
+
+
+def get_aws_default_region():
+ """
+ Get the default AWS region from ~/.aws/config file.
+
+ Returns:
+ str: Default region or 'us-east-1' if not found
+ """
+ config_path = os.path.expanduser("~/.aws/config")
+
+ if os.path.exists(config_path):
+ try:
+ config = ConfigParser()
+ config.read(config_path)
+
+ # Check for default profile region
+ if "default" in config:
+ return config["default"].get("region", "us-east-1")
+
+ # Check for profile default section
+ if "profile default" in config:
+ return config["profile default"].get("region", "us-east-1")
+
+ except Exception as e:
+ print(f"Warning: Error reading AWS config file: {e}", file=sys.stderr)
+
+ return "us-east-1"
+
+
+def get_all_regions():
+ """
+ Get all available AWS regions with their descriptions.
+
+ Returns:
+ dict: Dictionary of region information
+ """
+ try:
+ # Use a default region to get the list of all regions
+ ec2_client = boto3.client("ec2", region_name="us-east-1")
+ response = ec2_client.describe_regions(AllRegions=True)
+
+ regions = {}
+ for region in response["Regions"]:
+ region_name = region["RegionName"]
+ regions[region_name] = {
+ "region_name": region_name,
+ "region_description": region.get("RegionName", region_name),
+ "opt_in_status": region.get("OptInStatus", "Unknown"),
+ "availability_zones": [],
+ }
+
+ return regions
+
+ except Exception as e:
+ print(f"Error retrieving AWS regions: {e}", file=sys.stderr)
+ return {}
+
+
+def get_region_info(region_name, quiet=False):
+ """
+ Get detailed information about a specific region including availability zones.
+
+ Args:
+ region_name (str): AWS region name (e.g., 'us-east-1', 'eu-west-1')
+ quiet (bool): Suppress debug messages
+
+ Returns:
+ dict: Dictionary containing region information and availability zones
+ """
+ try:
+ if not quiet:
+ print(f"Querying information for region {region_name}...", file=sys.stderr)
+
+ # Initialize EC2 client for the specific region
+ ec2_client = boto3.client("ec2", region_name=region_name)
+
+ # Get region information
+ regions_response = ec2_client.describe_regions(
+ Filters=[{"Name": "region-name", "Values": [region_name]}]
+ )
+
+ if not regions_response["Regions"]:
+ if not quiet:
+ print(f"Region {region_name} not found", file=sys.stderr)
+ return None
+
+ region_info = regions_response["Regions"][0]
+
+ # Get availability zones for the region
+ az_response = ec2_client.describe_availability_zones()
+
+ availability_zones = []
+ for az in az_response["AvailabilityZones"]:
+ zone_info = {
+ "zone_id": az["ZoneId"],
+ "zone_name": az["ZoneName"],
+ "zone_type": az.get("ZoneType", "availability-zone"),
+ "parent_zone_id": az.get("ParentZoneId", ""),
+ "parent_zone_name": az.get("ParentZoneName", ""),
+ "state": az["State"],
+ "messages": [],
+ }
+
+ # Add any messages about the zone
+ if "Messages" in az:
+ zone_info["messages"] = [
+ msg.get("Message", "") for msg in az["Messages"]
+ ]
+
+ availability_zones.append(zone_info)
+
+ # Get network border group information if available
+ try:
+ zone_details = {}
+ for az in az_response["AvailabilityZones"]:
+ if "NetworkBorderGroup" in az:
+ zone_details[az["ZoneName"]] = az["NetworkBorderGroup"]
+ except:
+ zone_details = {}
+
+ result = {
+ "region_name": region_info["RegionName"],
+ "endpoint": region_info.get("Endpoint", f"ec2.{region_name}.amazonaws.com"),
+ "opt_in_status": region_info.get("OptInStatus", "opt-in-not-required"),
+ "availability_zone_count": len(availability_zones),
+ "availability_zones": sorted(
+ availability_zones, key=lambda x: x["zone_name"]
+ ),
+ }
+
+ if not quiet:
+ print(
+ f"Found {len(availability_zones)} availability zones in {region_name}",
+ file=sys.stderr,
+ )
+
+ return result
+
+ except NoCredentialsError:
+ print(
+ "Error: AWS credentials not found. Please configure your credentials.",
+ file=sys.stderr,
+ )
+ return None
+ except ClientError as e:
+ error_code = e.response.get("Error", {}).get("Code", "Unknown")
+ if error_code in ["UnauthorizedOperation", "InvalidRegion"]:
+ print(
+ f"Error: Cannot access region {region_name}. Check region name and permissions.",
+ file=sys.stderr,
+ )
+ else:
+ print(f"AWS API Error: {e}", file=sys.stderr)
+ return None
+ except Exception as e:
+ print(f"Unexpected error: {e}", file=sys.stderr)
+ return None
+
+
+def parse_arguments():
+ """Parse command line arguments."""
+ parser = argparse.ArgumentParser(
+ description="Get AWS region and availability zone information",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog="""
+Examples:
+ python %(prog)s --regions
+ python %(prog)s us-east-1
+ python %(prog)s eu-west-1 --format json
+ python %(prog)s --regions --format csv
+ python %(prog)s ap-southeast-1 --quiet
+ """,
+ )
+
+ parser.add_argument(
+ "region_name",
+ nargs="?", # Make region_name optional when using --regions
+ help="AWS region name (e.g., us-east-1, eu-west-1, ap-southeast-1)",
+ )
+
+ parser.add_argument(
+ "--format",
+ "-f",
+ choices=["table", "json", "csv"],
+ default="table",
+ help="Output format (default: table)",
+ )
+
+ parser.add_argument(
+ "--quiet", "-q", action="store_true", help="Suppress informational messages"
+ )
+
+ parser.add_argument(
+ "--debug", "-d", action="store_true", help="Enable debug output"
+ )
+
+ parser.add_argument(
+ "--regions", action="store_true", help="List all available AWS regions"
+ )
+
+ return parser.parse_args()
+
+
+def output_regions_table(regions, quiet=False):
+ """Output available regions in table format."""
+ if not quiet:
+ print(f"Available AWS regions ({len(regions)}):\n")
+
+ # Print header
+ print(f"{'Region Name':<20} {'Opt-in Status':<20}")
+ print("-" * 42)
+
+ # Sort regions by name
+ sorted_regions = sorted(regions.values(), key=lambda x: x["region_name"])
+
+ for region in sorted_regions:
+ opt_in_status = region.get("opt_in_status", "Unknown")
+ print(f"{region['region_name']:<20} {opt_in_status:<20}")
+
+
+def output_regions_json(regions):
+ """Output regions in JSON format."""
+ # Convert to list for JSON output
+ regions_list = sorted(regions.values(), key=lambda x: x["region_name"])
+ print(json.dumps(regions_list, indent=2))
+
+
+def output_regions_csv(regions):
+ """Output regions in CSV format."""
+ if regions:
+ # Print header
+ print("region_name,opt_in_status")
+
+ # Sort regions by name
+ sorted_regions = sorted(regions.values(), key=lambda x: x["region_name"])
+
+ # Print data
+ for region in sorted_regions:
+ opt_in_status = region.get("opt_in_status", "Unknown")
+ print(f"{region['region_name']},{opt_in_status}")
+
+
+def output_region_table(region_info, quiet=False):
+ """Output region information in table format."""
+ if not quiet:
+ print(f"Region: {region_info['region_name']}\n")
+ print(f"Endpoint: {region_info['endpoint']}")
+ print(f"Opt-in Status: {region_info['opt_in_status']}")
+ print(f"Availability Zones: {region_info['availability_zone_count']}\n")
+
+ # Print availability zones table
+ print(
+ f"{'Zone Name':<15} {'Zone ID':<15} {'Zone Type':<18} {'State':<12} {'Parent Zone':<15}"
+ )
+ print("-" * 80)
+
+ for az in region_info["availability_zones"]:
+ parent_zone = az.get("parent_zone_name", "") or az.get("parent_zone_id", "")
+ zone_type = az.get("zone_type", "availability-zone")
+
+ print(
+ f"{az['zone_name']:<15} "
+ f"{az['zone_id']:<15} "
+ f"{zone_type:<18} "
+ f"{az['state']:<12} "
+ f"{parent_zone:<15}"
+ )
+
+ # Show messages if any zones have them
+ zones_with_messages = [
+ az for az in region_info["availability_zones"] if az.get("messages")
+ ]
+ if zones_with_messages and not quiet:
+ print("\nZone Messages:")
+ for az in zones_with_messages:
+ for message in az["messages"]:
+ print(f" {az['zone_name']}: {message}")
+
+
+def output_region_json(region_info):
+ """Output region information in JSON format."""
+ print(json.dumps(region_info, indent=2))
+
+
+def output_region_csv(region_info):
+ """Output region information in CSV format."""
+ # First output region info
+ print("type,region_name,endpoint,opt_in_status,availability_zone_count")
+ print(
+ f"region,{region_info['region_name']},{region_info['endpoint']},{region_info['opt_in_status']},{region_info['availability_zone_count']}"
+ )
+
+ # Then output availability zones
+ print("\ntype,zone_name,zone_id,zone_type,state,parent_zone_name,parent_zone_id")
+ for az in region_info["availability_zones"]:
+ parent_zone_name = az.get("parent_zone_name", "")
+ parent_zone_id = az.get("parent_zone_id", "")
+ zone_type = az.get("zone_type", "availability-zone")
+
+ print(
+ f"availability_zone,{az['zone_name']},{az['zone_id']},{zone_type},{az['state']},{parent_zone_name},{parent_zone_id}"
+ )
+
+
+def main():
+ """Main function to run the program."""
+ args = parse_arguments()
+
+ # Handle --regions option
+ if args.regions:
+ if not args.quiet:
+ print("Fetching list of all AWS regions...", file=sys.stderr)
+
+ regions = get_all_regions()
+ if regions:
+ if args.format == "json":
+ output_regions_json(regions)
+ elif args.format == "csv":
+ output_regions_csv(regions)
+ else: # table format
+ output_regions_table(regions, args.quiet)
+ else:
+ print("Could not retrieve AWS regions.", file=sys.stderr)
+ sys.exit(1)
+ return
+
+ # Require region_name if not using --regions
+ if not args.region_name:
+ print(
+ "Error: region_name is required unless using --regions option",
+ file=sys.stderr,
+ )
+ print("Use --regions to list all available regions", file=sys.stderr)
+ sys.exit(1)
+
+ if not args.quiet:
+ print(f"Fetching information for region {args.region_name}...", file=sys.stderr)
+
+ # Get region information
+ region_info = get_region_info(args.region_name, args.quiet or not args.debug)
+
+ if not region_info:
+ print(
+ f"Could not retrieve information for region '{args.region_name}'.",
+ file=sys.stderr,
+ )
+ print(f"Try running with --regions to see available regions.", file=sys.stderr)
+ sys.exit(1)
+
+ # Output results in specified format
+ if args.format == "json":
+ output_region_json(region_info)
+ elif args.format == "csv":
+ output_region_csv(region_info)
+ else: # table format
+ output_region_table(region_info, args.quiet)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/terraform/aws/scripts/ec2_instance_info.py b/terraform/aws/scripts/ec2_instance_info.py
new file mode 100755
index 00000000..4dcbc6c9
--- /dev/null
+++ b/terraform/aws/scripts/ec2_instance_info.py
@@ -0,0 +1,540 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: MIT
+
+import boto3
+import json
+import sys
+import argparse
+import os
+from configparser import ConfigParser
+from botocore.exceptions import ClientError, NoCredentialsError
+
+
+def get_aws_default_region():
+ """
+ Get the default AWS region from ~/.aws/config file.
+
+ Returns:
+ str: Default region or 'us-east-1' if not found
+ """
+ config_path = os.path.expanduser("~/.aws/config")
+
+ if os.path.exists(config_path):
+ try:
+ config = ConfigParser()
+ config.read(config_path)
+
+ # Check for default profile region
+ if "default" in config:
+ return config["default"].get("region", "us-east-1")
+
+ # Check for profile default section
+ if "profile default" in config:
+ return config["profile default"].get("region", "us-east-1")
+
+ except Exception as e:
+ print(f"Warning: Error reading AWS config file: {e}", file=sys.stderr)
+
+ return "us-east-1"
+
+
+def get_available_families(region="us-east-1"):
+ """
+ Get all available instance families in the specified region.
+
+ Args:
+ region (str): AWS region to query
+
+ Returns:
+ dict: Dictionary with family info including count of instances per family
+ """
+ try:
+ ec2_client = boto3.client("ec2", region_name=region)
+ response = ec2_client.describe_instance_types()
+
+ families = {}
+ for instance_type in response["InstanceTypes"]:
+ instance_name = instance_type["InstanceType"]
+ family = instance_name.split(".")[0]
+
+ if family not in families:
+ families[family] = {
+ "family_name": family,
+ "instance_count": 0,
+ "has_gpu": False,
+ "architectures": set(),
+ }
+
+ families[family]["instance_count"] += 1
+
+ # Check for GPU
+ if "GpuInfo" in instance_type:
+ families[family]["has_gpu"] = True
+
+ # Get architectures
+ cpu_architectures = instance_type.get("ProcessorInfo", {}).get(
+ "SupportedArchitectures", []
+ )
+ families[family]["architectures"].update(cpu_architectures)
+
+ # Convert architecture sets to sorted lists for JSON serialization
+ for family in families.values():
+ family["architectures"] = sorted(list(family["architectures"]))
+
+ return families
+
+ except Exception as e:
+ print(f"Error retrieving instance families: {e}", file=sys.stderr)
+ return {}
+
+
+def get_gpu_info(instance_type):
+ """
+ Extract GPU information from instance type data.
+
+ Args:
+ instance_type (dict): Instance type data from AWS API
+
+ Returns:
+ str: Formatted GPU information string
+ """
+ if "GpuInfo" not in instance_type:
+ return "None"
+
+ gpu_info = instance_type["GpuInfo"]
+ total_gpu_memory = gpu_info.get("TotalGpuMemoryInMiB", 0)
+ gpus = gpu_info.get("Gpus", [])
+
+ if not gpus:
+ return "GPU present (details unavailable)"
+
+ gpu_details = []
+ for gpu in gpus:
+ gpu_name = gpu.get("Name", "Unknown GPU")
+ gpu_count = gpu.get("Count", 1)
+ gpu_memory = gpu.get("MemoryInfo", {}).get("SizeInMiB", 0)
+
+ if gpu_count > 1:
+ detail = f"{gpu_count}x {gpu_name}"
+ else:
+ detail = gpu_name
+
+ if gpu_memory > 0:
+ detail += f" ({gpu_memory // 1024}GB)"
+
+ gpu_details.append(detail)
+
+ return ", ".join(gpu_details)
+
+
+def get_instance_family_info(family_name, region="us-east-1", quiet=False):
+ """
+ Get instance types, pricing, and hardware info for an AWS instance family.
+
+ Args:
+ family_name (str): Instance family name (e.g., 'm5', 't3', 'c5')
+ region (str): AWS region to query (default: us-east-1)
+ quiet (bool): Suppress debug messages
+
+ Returns:
+ list: List of dictionaries containing instance information
+ """
+ try:
+ # Initialize AWS clients
+ ec2_client = boto3.client("ec2", region_name=region)
+ pricing_client = boto3.client(
+ "pricing", region_name="us-east-1"
+ ) # Pricing API only in us-east-1
+
+ if not quiet:
+ print(
+ f"Querying EC2 API for instances starting with '{family_name}'...",
+ file=sys.stderr,
+ )
+
+ # Get ALL instance types first, then filter
+ response = ec2_client.describe_instance_types()
+
+ # Filter instances that belong to the specified family
+ family_instances = []
+ for instance_type in response["InstanceTypes"]:
+ instance_name = instance_type["InstanceType"]
+ # More flexible matching - check if instance name starts with family name
+ if instance_name.startswith(family_name + ".") or instance_name.startswith(
+ family_name
+ ):
+ family_instances.append(instance_type)
+
+ if not family_instances:
+ if not quiet:
+ print(
+ f"No instances found starting with '{family_name}'. Trying broader search...",
+ file=sys.stderr,
+ )
+
+ # Try a broader search - maybe the family name is part of the instance type
+ family_instances = []
+ for instance_type in response["InstanceTypes"]:
+ instance_name = instance_type["InstanceType"]
+ if family_name.lower() in instance_name.lower():
+ family_instances.append(instance_type)
+
+ if not family_instances:
+ if not quiet:
+ # Show available families to help debug
+ families = get_available_families(region)
+ family_names = sorted([f["family_name"] for f in families.values()])
+ print(f"Available instance families: {family_names}", file=sys.stderr)
+ return []
+
+ if not quiet:
+ print(
+ f"Found {len(family_instances)} instances in family '{family_name}'",
+ file=sys.stderr,
+ )
+
+ instance_info = []
+
+ for instance_type in family_instances:
+ instance_name = instance_type["InstanceType"]
+
+ if not quiet:
+ print(f"Processing {instance_name}...", file=sys.stderr)
+
+ # Extract CPU architecture information
+ cpu_architectures = instance_type.get("ProcessorInfo", {}).get(
+ "SupportedArchitectures", ["Unknown"]
+ )
+ cpu_isa = ", ".join(cpu_architectures) if cpu_architectures else "Unknown"
+
+ # Extract GPU information
+ gpu_info = get_gpu_info(instance_type)
+
+ # Extract hardware specifications
+ hardware_info = {
+ "instance_type": instance_name,
+ "vcpus": instance_type["VCpuInfo"]["DefaultVCpus"],
+ "memory_gb": instance_type["MemoryInfo"]["SizeInMiB"] / 1024,
+ "cpu_isa": cpu_isa,
+ "gpu": gpu_info,
+ "network_performance": instance_type.get("NetworkInfo", {}).get(
+ "NetworkPerformance", "Not specified"
+ ),
+ "storage": "EBS-only",
+ }
+
+ # Check for instance storage
+ if "InstanceStorageInfo" in instance_type:
+ storage_info = instance_type["InstanceStorageInfo"]
+ total_storage = storage_info.get("TotalSizeInGB", 0)
+ storage_type = storage_info.get("Disks", [{}])[0].get("Type", "Unknown")
+ hardware_info["storage"] = f"{total_storage} GB {storage_type}"
+
+ # Get pricing information (note: this often fails due to AWS Pricing API limitations)
+ try:
+ pricing_response = pricing_client.get_products(
+ ServiceCode="AmazonEC2",
+ Filters=[
+ {
+ "Type": "TERM_MATCH",
+ "Field": "instanceType",
+ "Value": instance_name,
+ },
+ {
+ "Type": "TERM_MATCH",
+ "Field": "location",
+ "Value": get_location_name(region),
+ },
+ {"Type": "TERM_MATCH", "Field": "tenancy", "Value": "Shared"},
+ {
+ "Type": "TERM_MATCH",
+ "Field": "operating-system",
+ "Value": "Linux",
+ },
+ {
+ "Type": "TERM_MATCH",
+ "Field": "preInstalledSw",
+ "Value": "NA",
+ },
+ {
+ "Type": "TERM_MATCH",
+ "Field": "capacitystatus",
+ "Value": "Used",
+ },
+ ],
+ )
+
+ if pricing_response["PriceList"]:
+ price_data = json.loads(pricing_response["PriceList"][0])
+ terms = price_data["terms"]["OnDemand"]
+
+ # Extract the hourly price
+ for term_key in terms:
+ price_dimensions = terms[term_key]["priceDimensions"]
+ for price_key in price_dimensions:
+ price_per_hour = price_dimensions[price_key][
+ "pricePerUnit"
+ ]["USD"]
+ hardware_info["price_per_hour_usd"] = f"${price_per_hour}"
+ break
+ break
+ else:
+ hardware_info["price_per_hour_usd"] = "Not available"
+
+ except Exception as e:
+ if not quiet:
+ print(
+ f"Warning: Could not fetch pricing for {instance_name}: {str(e)}",
+ file=sys.stderr,
+ )
+ hardware_info["price_per_hour_usd"] = "Not available"
+
+ instance_info.append(hardware_info)
+
+ return sorted(instance_info, key=lambda x: x["instance_type"])
+
+ except NoCredentialsError:
+ print(
+ "Error: AWS credentials not found. Please configure your credentials.",
+ file=sys.stderr,
+ )
+ return []
+ except ClientError as e:
+ print(f"AWS API Error: {e}", file=sys.stderr)
+ return []
+ except Exception as e:
+ print(f"Unexpected error: {e}", file=sys.stderr)
+ return []
+
+
+def get_location_name(region):
+ """Convert AWS region to location name for pricing API."""
+ region_mapping = {
+ "us-east-1": "US East (N. Virginia)",
+ "us-east-2": "US East (Ohio)",
+ "us-west-1": "US West (N. California)",
+ "us-west-2": "US West (Oregon)",
+ "us-west-2-lax-1": "US West (Los Angeles)",
+ "ca-central-1": "Canada (Central)",
+ "eu-west-1": "Europe (Ireland)",
+ "eu-west-2": "Europe (London)",
+ "eu-west-3": "Europe (Paris)",
+ "eu-central-1": "Europe (Frankfurt)",
+ "eu-north-1": "Europe (Stockholm)",
+ "ap-southeast-1": "Asia Pacific (Singapore)",
+ "ap-southeast-2": "Asia Pacific (Sydney)",
+ "ap-northeast-1": "Asia Pacific (Tokyo)",
+ "ap-northeast-2": "Asia Pacific (Seoul)",
+ "ap-south-1": "Asia Pacific (Mumbai)",
+ "sa-east-1": "South America (Sao Paulo)",
+ }
+ return region_mapping.get(region, "US East (N. Virginia)")
+
+
+def parse_arguments():
+ """Parse command line arguments."""
+ parser = argparse.ArgumentParser(
+ description="Get AWS EC2 instance family information including pricing and hardware specs",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog="""
+Examples:
+ python %(prog)s m5
+ python %(prog)s t3 --region us-west-2
+ python %(prog)s c5 --format json
+ python %(prog)s r5 --quiet
+ python %(prog)s --families
+ python %(prog)s --families --format json
+ """,
+ )
+
+ parser.add_argument(
+ "family_name",
+ nargs="?", # Make family_name optional when using --families
+ help="Instance family name (e.g., m5, t3, c5, r5)",
+ )
+
+ parser.add_argument(
+ "--region", "-r", help="AWS region (default: from ~/.aws/config or us-east-1)"
+ )
+
+ parser.add_argument(
+ "--format",
+ "-f",
+ choices=["table", "json", "csv"],
+ default="table",
+ help="Output format (default: table)",
+ )
+
+ parser.add_argument(
+ "--quiet", "-q", action="store_true", help="Suppress informational messages"
+ )
+
+ parser.add_argument(
+ "--debug", "-d", action="store_true", help="Enable debug output"
+ )
+
+ parser.add_argument(
+ "--families",
+ action="store_true",
+ help="List all available instance families in the region",
+ )
+
+ return parser.parse_args()
+
+
+def output_families_table(families, region, quiet=False):
+ """Output available instance families in table format."""
+ if not quiet:
+ print(f"Available instance families in {region}:\n")
+
+ # Print header
+ print(f"{'Family':<10} {'Count':<6} {'GPU':<5} {'Architectures':<20}")
+ print("-" * 45)
+
+ # Sort families by name
+ sorted_families = sorted(families.values(), key=lambda x: x["family_name"])
+
+ for family in sorted_families:
+ gpu_indicator = "Yes" if family["has_gpu"] else "No"
+ architectures = ", ".join(family["architectures"])
+
+ print(
+ f"{family['family_name']:<10} "
+ f"{family['instance_count']:<6} "
+ f"{gpu_indicator:<5} "
+ f"{architectures:<20}"
+ )
+
+
+def output_families_json(families):
+ """Output families in JSON format."""
+ # Convert to list for JSON output
+ families_list = sorted(families.values(), key=lambda x: x["family_name"])
+ print(json.dumps(families_list, indent=2))
+
+
+def output_families_csv(families):
+ """Output families in CSV format."""
+ if families:
+ # Print header
+ print("family_name,instance_count,has_gpu,architectures")
+
+ # Sort families by name
+ sorted_families = sorted(families.values(), key=lambda x: x["family_name"])
+
+ # Print data
+ for family in sorted_families:
+ architectures = ";".join(
+ family["architectures"]
+ ) # Use semicolon to avoid CSV issues
+ print(
+ f"{family['family_name']},{family['instance_count']},{family['has_gpu']},{architectures}"
+ )
+
+
+def output_table(instances, quiet=False):
+ """Output results in table format."""
+ if not quiet:
+ print(f"Found {len(instances)} instance types:\n")
+
+ # Print header - adjusted for GPU column
+ print(
+ f"{'Instance Type':<15} {'vCPUs':<6} {'Memory (GB)':<12} {'CPU ISA':<10} {'GPU':<25} {'Storage':<20} {'Network':<15} {'Price/Hour':<12}"
+ )
+ print("-" * 130)
+
+ # Print instance details
+ for instance in instances:
+ print(
+ f"{instance['instance_type']:<15} "
+ f"{instance['vcpus']:<6} "
+ f"{instance['memory_gb']:<12.1f} "
+ f"{instance['cpu_isa']:<10} "
+ f"{instance['gpu']:<25} "
+ f"{instance['storage']:<20} "
+ f"{instance['network_performance']:<15} "
+ f"{instance['price_per_hour_usd']:<12}"
+ )
+
+
+def output_json(instances):
+ """Output results in JSON format."""
+ print(json.dumps(instances, indent=2))
+
+
+def output_csv(instances):
+ """Output results in CSV format."""
+ if instances:
+ # Print header
+ headers = instances[0].keys()
+ print(",".join(headers))
+
+ # Print data
+ for instance in instances:
+ values = [str(instance[header]).replace(",", ";") for header in headers]
+ print(",".join(values))
+
+
+def main():
+ """Main function to run the program."""
+ args = parse_arguments()
+
+ # Determine region
+ if args.region:
+ region = args.region
+ else:
+ region = get_aws_default_region()
+
+ # Handle --families option
+ if args.families:
+ families = get_available_families(region)
+ if families:
+ if args.format == "json":
+ output_families_json(families)
+ elif args.format == "csv":
+ output_families_csv(families)
+ else: # table format
+ output_families_table(families, region, args.quiet)
+ else:
+ print("Could not retrieve instance families.", file=sys.stderr)
+ sys.exit(1)
+ return
+
+ # Require family_name if not using --families
+ if not args.family_name:
+ print(
+ "Error: family_name is required unless using --families option",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ if not args.quiet:
+ print(
+ f"Fetching information for {args.family_name} family in {region}...",
+ file=sys.stderr,
+ )
+
+ # Get instance information
+ instances = get_instance_family_info(
+ args.family_name, region, args.quiet or not args.debug
+ )
+
+ if not instances:
+ print(f"No instances found for family '{args.family_name}'.", file=sys.stderr)
+ print(
+ f"Try running with --families to see available instance families.",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ # Output results in specified format
+ if args.format == "json":
+ output_json(instances)
+ elif args.format == "csv":
+ output_csv(instances)
+ else: # table format
+ output_table(instances, args.quiet)
+
+
+if __name__ == "__main__":
+ main()
--
2.51.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 3/8] aws: add optimized Kconfig generator using Chuck's scripts
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 1/8] aws: prevent SSH key conflicts across multiple kdevops directories Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 2/8] terraform/aws: Add scripts to gather provider resource information Luis Chamberlain
@ 2025-09-17 0:34 ` Luis Chamberlain
2025-09-17 3:58 ` Chuck Lever
2025-09-17 0:34 ` [PATCH v4 4/8] aws: integrate dynamic Kconfig generation with make targets Luis Chamberlain
` (4 subsequent siblings)
7 siblings, 1 reply; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Luis Chamberlain
Create a wrapper script that orchestrates Chuck's existing AWS scripts
(ec2_instance_info.py, aws_regions_info.py, aws_ami_info.py) to generate
Kconfig files with JSON caching and parallel processing.
This approach leverages Chuck's already working scripts while adding:
- JSON caching with 24-hour TTL in ~/.cache/kdevops/aws/
- Parallel fetching of instance data (10 concurrent workers)
- Parallel file writing (20 concurrent workers)
- Proper data structure handling for families list
Performance improvements:
- First run: ~21 seconds to fetch all data from AWS
- Cached runs: ~0.04 seconds (525x faster)
- Generates 75 Kconfig files for 72 instance families
The script properly uses Chuck's existing AWS API implementations
rather than reimplementing them, maintaining code reuse and consistency.
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
terraform/aws/scripts/generate_aws_kconfig.py | 462 ++++++++++++++++++
1 file changed, 462 insertions(+)
create mode 100755 terraform/aws/scripts/generate_aws_kconfig.py
diff --git a/terraform/aws/scripts/generate_aws_kconfig.py b/terraform/aws/scripts/generate_aws_kconfig.py
new file mode 100755
index 00000000..c6a60a83
--- /dev/null
+++ b/terraform/aws/scripts/generate_aws_kconfig.py
@@ -0,0 +1,462 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+"""
+AWS Kconfig generator using Chuck's ec2_instance_info.py with JSON caching.
+
+This script orchestrates the generation of Kconfig files for AWS EC2 instances
+using Chuck's existing scripts with added caching and parallelization.
+"""
+
+import os
+import sys
+import json
+import time
+import subprocess
+from pathlib import Path
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from typing import Dict, List, Any, Optional
+
+# Cache configuration
+CACHE_DIR = Path.home() / ".cache" / "kdevops" / "aws"
+CACHE_TTL = 24 * 3600 # 24 hours in seconds
+
+# Scripts directory
+SCRIPTS_DIR = Path(__file__).parent
+EC2_INFO_SCRIPT = SCRIPTS_DIR / "ec2_instance_info.py"
+AMI_INFO_SCRIPT = SCRIPTS_DIR / "aws_ami_info.py"
+REGIONS_INFO_SCRIPT = SCRIPTS_DIR / "aws_regions_info.py"
+
+# Output directories
+KCONFIG_DIR = SCRIPTS_DIR.parent / "kconfigs"
+INSTANCE_TYPES_DIR = KCONFIG_DIR / "instance-types"
+
+
+def ensure_cache_dir():
+ """Create cache directory if it doesn't exist."""
+ CACHE_DIR.mkdir(parents=True, exist_ok=True)
+
+
+def get_cache_file(cache_key: str) -> Path:
+ """Get cache file path for a given key."""
+ return CACHE_DIR / f"{cache_key}.json"
+
+
+def is_cache_valid(cache_file: Path) -> bool:
+ """Check if cache file exists and is still valid."""
+ if not cache_file.exists():
+ return False
+
+ age = time.time() - cache_file.stat().st_mtime
+ return age < CACHE_TTL
+
+
+def load_from_cache(cache_key: str) -> Optional[Any]:
+ """Load data from cache if valid."""
+ cache_file = get_cache_file(cache_key)
+
+ if is_cache_valid(cache_file):
+ try:
+ with cache_file.open('r') as f:
+ return json.load(f)
+ except (json.JSONDecodeError, IOError):
+ pass
+
+ return None
+
+
+def save_to_cache(cache_key: str, data: Any):
+ """Save data to cache."""
+ cache_file = get_cache_file(cache_key)
+
+ try:
+ with cache_file.open('w') as f:
+ json.dump(data, f, indent=2)
+ except IOError as e:
+ print(f"Warning: Failed to save cache: {e}", file=sys.stderr)
+
+
+def run_chuck_script(script: Path, args: List[str]) -> Optional[Any]:
+ """Run one of Chuck's scripts and return JSON output."""
+ cmd = [sys.executable, str(script)] + args + ["--format", "json", "--quiet"]
+
+ try:
+ result = subprocess.run(
+ cmd,
+ capture_output=True,
+ text=True,
+ check=True,
+ env={**os.environ, "AWS_DEFAULT_REGION": os.environ.get("AWS_DEFAULT_REGION", "us-east-1")}
+ )
+
+ if result.stdout:
+ return json.loads(result.stdout)
+ except subprocess.CalledProcessError as e:
+ print(f"Error running {script.name}: {e}", file=sys.stderr)
+ if e.stderr:
+ print(f"stderr: {e.stderr}", file=sys.stderr)
+ except json.JSONDecodeError as e:
+ print(f"Error parsing JSON from {script.name}: {e}", file=sys.stderr)
+
+ return None
+
+
+def fetch_all_families() -> Optional[List[Dict[str, Any]]]:
+ """Fetch all instance families."""
+ cache_key = "aws_families"
+
+ # Check cache first
+ cached = load_from_cache(cache_key)
+ if cached:
+ print("Using cached AWS families data", file=sys.stderr)
+ return cached
+
+ print("Fetching AWS instance families...", file=sys.stderr)
+ families = run_chuck_script(EC2_INFO_SCRIPT, ["--families"])
+
+ if families:
+ save_to_cache(cache_key, families)
+
+ return families
+
+
+def fetch_family_instances(family_name: str) -> Optional[List[Dict]]:
+ """Fetch instances for a specific family."""
+ cache_key = f"aws_family_{family_name}"
+
+ # Check cache first
+ cached = load_from_cache(cache_key)
+ if cached:
+ return cached
+
+ instances = run_chuck_script(EC2_INFO_SCRIPT, [family_name])
+
+ if instances:
+ save_to_cache(cache_key, instances)
+
+ return instances
+
+
+def fetch_all_instances() -> Dict[str, List[Dict]]:
+ """Fetch all instances for all families with parallel processing."""
+ families = fetch_all_families()
+ if not families:
+ print("Error: Could not fetch AWS families", file=sys.stderr)
+ return {}
+
+ # Check if we have a complete cache
+ cache_key = "aws_all_instances"
+ cached = load_from_cache(cache_key)
+ if cached:
+ print("Using cached complete AWS instance data", file=sys.stderr)
+ return cached
+
+ print(f"Fetching instance data for {len(families)} families...", file=sys.stderr)
+ all_instances = {}
+
+ # Extract family names from the list of family dicts
+ family_names = [f['family_name'] for f in families if 'family_name' in f]
+
+ # Use parallel processing to fetch instance data
+ with ThreadPoolExecutor(max_workers=10) as executor:
+ future_to_family = {
+ executor.submit(fetch_family_instances, family): family
+ for family in family_names
+ }
+
+ for future in as_completed(future_to_family):
+ family = future_to_family[future]
+ try:
+ instances = future.result()
+ if instances:
+ all_instances[family] = instances
+ print(f" Fetched {family}: {len(instances)} instances", file=sys.stderr)
+ except Exception as e:
+ print(f" Error fetching {family}: {e}", file=sys.stderr)
+
+ # Save complete dataset to cache
+ if all_instances:
+ save_to_cache(cache_key, all_instances)
+
+ return all_instances
+
+
+def fetch_regions() -> Optional[List[Dict]]:
+ """Fetch AWS regions."""
+ cache_key = "aws_regions"
+
+ cached = load_from_cache(cache_key)
+ if cached:
+ print("Using cached AWS regions data", file=sys.stderr)
+ return cached
+
+ print("Fetching AWS regions...", file=sys.stderr)
+ regions = run_chuck_script(REGIONS_INFO_SCRIPT, ["--regions"])
+
+ if regions:
+ save_to_cache(cache_key, regions)
+
+ return regions
+
+
+def fetch_gpu_amis() -> Optional[Dict]:
+ """Fetch GPU AMI information."""
+ cache_key = "aws_gpu_amis"
+
+ cached = load_from_cache(cache_key)
+ if cached:
+ print("Using cached AWS GPU AMI data", file=sys.stderr)
+ return cached
+
+ print("Fetching AWS GPU AMIs...", file=sys.stderr)
+ amis = run_chuck_script(AMI_INFO_SCRIPT, ["--gpu"])
+
+ if amis:
+ save_to_cache(cache_key, amis)
+
+ return amis
+
+
+def generate_family_kconfig(family: str, instances: List[Dict]) -> str:
+ """Generate Kconfig content for a single family."""
+ content = [f"# AWS {family.upper()} instance sizes (dynamically generated)", ""]
+
+ # Sort instances by a logical order
+ sorted_instances = sorted(instances, key=lambda x: (
+ 'metal' not in x['instance_type'], # metal instances last
+ x.get('vcpus', 0), # then by vCPUs
+ x.get('memory_gb', 0) # then by memory
+ ))
+
+ # Determine default instance (usually the first non-nano/micro)
+ default = sorted_instances[0]['instance_type']
+ for inst in sorted_instances:
+ if not any(size in inst['instance_type'] for size in ['.nano', '.micro']):
+ default = inst['instance_type']
+ break
+
+ # Generate choice block
+ content.append("choice")
+ content.append(f'\tprompt "Instance size for {family.upper()} family"')
+ content.append(f'\tdefault TERRAFORM_AWS_INSTANCE_{default.replace(".", "_").upper()}')
+ content.append("\thelp")
+ content.append(f"\t Select the specific instance size within the {family.upper()} family.")
+ content.append("")
+
+ # Generate config entries
+ for inst in sorted_instances:
+ type_upper = inst['instance_type'].replace('.', '_').upper()
+ content.append(f"config TERRAFORM_AWS_INSTANCE_{type_upper}")
+ content.append(f'\tbool "{inst["instance_type"]}"')
+ content.append("\thelp")
+ content.append(f"\t vCPUs: {inst.get('vcpus', 'N/A')}")
+ content.append(f"\t Memory: {inst.get('memory_gb', 'N/A')} GB")
+ content.append(f"\t Network: {inst.get('network_performance', 'N/A')}")
+ content.append("")
+
+ content.append("endchoice")
+ content.append("")
+
+ # Generate string config
+ content.append(f"config TERRAFORM_AWS_{family.upper()}_SIZE")
+ content.append("\tstring")
+
+ for inst in sorted_instances:
+ type_upper = inst['instance_type'].replace('.', '_').upper()
+ content.append(f'\tdefault "{inst["instance_type"]}" if TERRAFORM_AWS_INSTANCE_{type_upper}')
+
+ content.append(f'\tdefault "{default}"')
+ content.append("")
+
+ return '\n'.join(content)
+
+
+def generate_compute_kconfig(families: Dict[str, Any]) -> str:
+ """Generate main compute Kconfig."""
+ content = ["# AWS EC2 Instance Types (dynamically generated)", ""]
+
+ # Sort families for consistent output
+ sorted_families = sorted(families.keys())
+
+ content.append("choice")
+ content.append('\tprompt "EC2 instance family"')
+ content.append("\tdefault TERRAFORM_AWS_INSTANCE_FAMILY_M5")
+ content.append("\thelp")
+ content.append("\t Select the EC2 instance family to use.")
+ content.append("")
+
+ for family in sorted_families:
+ family_upper = family.upper()
+ family_desc = families[family].get('description', f'{family_upper} instances')
+
+ content.append(f"config TERRAFORM_AWS_INSTANCE_FAMILY_{family_upper}")
+ content.append(f'\tbool "{family_upper} - {family_desc}"')
+ content.append("\thelp")
+ content.append(f"\t {family_desc}")
+ content.append(f"\t Available instances: {families[family].get('count', 0)}")
+ content.append("")
+
+ content.append("endchoice")
+ content.append("")
+
+ # Generate family name config
+ content.append("config TERRAFORM_AWS_INSTANCE_FAMILY")
+ content.append("\tstring")
+
+ for family in sorted_families:
+ family_upper = family.upper()
+ content.append(f'\tdefault "{family}" if TERRAFORM_AWS_INSTANCE_FAMILY_{family_upper}')
+
+ content.append('\tdefault "m5"')
+ content.append("")
+
+ # Include family-specific files
+ for family in sorted_families:
+ content.append(f'if TERRAFORM_AWS_INSTANCE_FAMILY_{family.upper()}')
+ content.append(f'source "terraform/aws/kconfigs/instance-types/Kconfig.{family}.generated"')
+ content.append("endif")
+ content.append("")
+
+ return '\n'.join(content)
+
+
+def generate_location_kconfig(regions: List[Dict]) -> str:
+ """Generate location Kconfig."""
+ content = ["# AWS Regions (dynamically generated)", ""]
+
+ content.append("choice")
+ content.append('\tprompt "AWS region"')
+ content.append("\tdefault TERRAFORM_AWS_REGION_US_EAST_1")
+ content.append("\thelp")
+ content.append("\t Select the AWS region for your infrastructure.")
+ content.append("")
+
+ for region in regions:
+ region_upper = region['region_name'].replace('-', '_').upper()
+ content.append(f"config TERRAFORM_AWS_REGION_{region_upper}")
+ content.append(f'\tbool "{region["region_name"]} - {region.get("location", "")}"')
+ content.append("")
+
+ content.append("endchoice")
+ content.append("")
+
+ # Generate region string config
+ content.append("config TERRAFORM_AWS_REGION")
+ content.append("\tstring")
+
+ for region in regions:
+ region_upper = region['region_name'].replace('-', '_').upper()
+ content.append(f'\tdefault "{region["region_name"]}" if TERRAFORM_AWS_REGION_{region_upper}')
+
+ content.append('\tdefault "us-east-1"')
+ content.append("")
+
+ return '\n'.join(content)
+
+
+def write_kconfig_file(filepath: Path, content: str):
+ """Write Kconfig content to file."""
+ filepath.parent.mkdir(parents=True, exist_ok=True)
+ filepath.write_text(content)
+
+
+def clear_cache():
+ """Clear all cached data."""
+ if CACHE_DIR.exists():
+ for cache_file in CACHE_DIR.glob("*.json"):
+ cache_file.unlink()
+ print("Cache cleared", file=sys.stderr)
+
+
+def main():
+ """Main function."""
+ # Handle cache clearing
+ if len(sys.argv) > 1 and sys.argv[1] == "clear-cache":
+ clear_cache()
+ return
+
+ start_time = time.time()
+
+ # Ensure AWS region is set
+ if "AWS_DEFAULT_REGION" not in os.environ:
+ os.environ["AWS_DEFAULT_REGION"] = "us-east-1"
+ print(f"Set AWS_DEFAULT_REGION to us-east-1", file=sys.stderr)
+
+ ensure_cache_dir()
+
+ # Fetch all data (uses cache if available)
+ print("Generating AWS Kconfig files...", file=sys.stderr)
+
+ # Fetch regions
+ regions = fetch_regions()
+ if not regions:
+ print("Warning: Could not fetch regions", file=sys.stderr)
+ regions = []
+
+ # Fetch all instance data
+ all_instances = fetch_all_instances()
+ if not all_instances:
+ print("Error: Could not fetch instance data", file=sys.stderr)
+ sys.exit(1)
+
+ # Prepare families info for compute Kconfig
+ families_info = {}
+ for family, instances in all_instances.items():
+ families_info[family] = {
+ 'count': len(instances),
+ 'description': f'{family.upper()} instances'
+ }
+
+ print(f"\nGenerating Kconfig files for {len(all_instances)} families...", file=sys.stderr)
+
+ # Generate files in parallel
+ tasks = []
+
+ # Family-specific Kconfig files
+ for family, instances in all_instances.items():
+ filepath = INSTANCE_TYPES_DIR / f"Kconfig.{family}.generated"
+ content = generate_family_kconfig(family, instances)
+ tasks.append((filepath, content))
+
+ # Main Kconfig files
+ tasks.append((KCONFIG_DIR / "Kconfig.compute.generated",
+ generate_compute_kconfig(families_info)))
+
+ if regions:
+ tasks.append((KCONFIG_DIR / "Kconfig.location.generated",
+ generate_location_kconfig(regions)))
+
+ # GPU AMIs (stub for now)
+ tasks.append((KCONFIG_DIR / "Kconfig.gpu-amis.generated",
+ "# AWS GPU AMIs (placeholder)\n"))
+
+ # Write files in parallel
+ with ThreadPoolExecutor(max_workers=20) as executor:
+ futures = []
+ for filepath, content in tasks:
+ future = executor.submit(write_kconfig_file, filepath, content)
+ futures.append((future, filepath))
+
+ for future, filepath in futures:
+ try:
+ future.result()
+ print(f" Generated: {filepath.name}", file=sys.stderr)
+ except Exception as e:
+ print(f" Error writing {filepath}: {e}", file=sys.stderr)
+
+ elapsed = time.time() - start_time
+
+ # Summary
+ print(f"\n✓ Generated {len(tasks)} Kconfig files in {elapsed:.2f} seconds", file=sys.stderr)
+ print(f" • {len(all_instances)} instance families", file=sys.stderr)
+ print(f" • {sum(len(instances) for instances in all_instances.values())} total instance types", file=sys.stderr)
+ print(f" • {len(regions)} regions", file=sys.stderr)
+
+ if elapsed < 1:
+ print(f" • Using cached data (cache valid for 24 hours)", file=sys.stderr)
+ else:
+ print(f" • Fresh data fetched from AWS", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
--
2.51.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 4/8] aws: integrate dynamic Kconfig generation with make targets
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
` (2 preceding siblings ...)
2025-09-17 0:34 ` [PATCH v4 3/8] aws: add optimized Kconfig generator using Chuck's scripts Luis Chamberlain
@ 2025-09-17 0:34 ` Luis Chamberlain
2025-09-17 3:40 ` Chuck Lever
2025-09-17 0:34 ` [PATCH v4 5/8] aws: add cloud billing support with make cloud-bill Luis Chamberlain
` (3 subsequent siblings)
7 siblings, 1 reply; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Luis Chamberlain
Add Makefile integration for AWS dynamic Kconfig generation using
Chuck's scripts with the optimized wrapper. This provides:
- make cloud-config-aws: Generate AWS Kconfig files
- make cloud-update-aws: Clear cache and regenerate fresh data
- make cloud-config: Now includes AWS generation
- make cloud-update: Refreshes all cloud provider data
- make clean-cloud-config-aws: Clean generated AWS files
The integration properly manages AWS Kconfig files alongside Lambda Labs
configurations and ensures empty files exist before Kconfig runs to
prevent sourcing errors.
Also update generate_cloud_configs.py to call the AWS generator and
provide summary statistics about available AWS resources.
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
scripts/dynamic-cloud-kconfig.Makefile | 49 ++++++++++++++--
scripts/generate_cloud_configs.py | 78 +++++++++++++++++++++++++-
2 files changed, 120 insertions(+), 7 deletions(-)
diff --git a/scripts/dynamic-cloud-kconfig.Makefile b/scripts/dynamic-cloud-kconfig.Makefile
index e15651ab..ed2d5366 100644
--- a/scripts/dynamic-cloud-kconfig.Makefile
+++ b/scripts/dynamic-cloud-kconfig.Makefile
@@ -12,8 +12,18 @@ LAMBDALABS_KCONFIG_IMAGES := $(LAMBDALABS_KCONFIG_DIR)/Kconfig.images.generated
LAMBDALABS_KCONFIGS := $(LAMBDALABS_KCONFIG_COMPUTE) $(LAMBDALABS_KCONFIG_LOCATION) $(LAMBDALABS_KCONFIG_IMAGES)
-# Add Lambda Labs generated files to mrproper clean list
-KDEVOPS_MRPROPER += $(LAMBDALABS_KCONFIGS)
+# AWS dynamic configuration
+AWS_KCONFIG_DIR := terraform/aws/kconfigs
+AWS_KCONFIG_COMPUTE := $(AWS_KCONFIG_DIR)/Kconfig.compute.generated
+AWS_KCONFIG_LOCATION := $(AWS_KCONFIG_DIR)/Kconfig.location.generated
+AWS_KCONFIG_GPU_AMIS := $(AWS_KCONFIG_DIR)/Kconfig.gpu-amis.generated
+AWS_KCONFIG_INSTANCE_TYPES_DIR := $(AWS_KCONFIG_DIR)/instance-types
+
+# Note: Instance type files are generated dynamically, so we just track the main ones
+AWS_KCONFIGS := $(AWS_KCONFIG_COMPUTE) $(AWS_KCONFIG_LOCATION) $(AWS_KCONFIG_GPU_AMIS)
+
+# Add generated files to mrproper clean list
+KDEVOPS_MRPROPER += $(LAMBDALABS_KCONFIGS) $(AWS_KCONFIGS)
# Touch Lambda Labs generated files so Kconfig can source them
# This ensures the files exist (even if empty) before Kconfig runs
@@ -22,20 +32,44 @@ dynamic_lambdalabs_kconfig_touch:
DYNAMIC_KCONFIG += dynamic_lambdalabs_kconfig_touch
+# Touch AWS generated files so Kconfig can source them
+dynamic_aws_kconfig_touch:
+ $(Q)mkdir -p $(AWS_KCONFIG_INSTANCE_TYPES_DIR)
+ $(Q)touch $(AWS_KCONFIGS)
+
+DYNAMIC_KCONFIG += dynamic_aws_kconfig_touch
+
# Individual Lambda Labs targets are now handled by generate_cloud_configs.py
cloud-config-lambdalabs:
$(Q)python3 scripts/generate_cloud_configs.py
+# AWS targets using Chuck's scripts with caching
+cloud-config-aws:
+ $(Q)python3 terraform/aws/scripts/generate_aws_kconfig.py
+
+# AWS cache refresh (clears cache and regenerates)
+cloud-update-aws:
+ $(Q)python3 terraform/aws/scripts/generate_aws_kconfig.py clear-cache
+ $(Q)python3 terraform/aws/scripts/generate_aws_kconfig.py
+
# Clean Lambda Labs generated files
clean-cloud-config-lambdalabs:
$(Q)rm -f $(LAMBDALABS_KCONFIGS)
-DYNAMIC_CLOUD_KCONFIG += cloud-config-lambdalabs
+# Clean AWS generated files
+clean-cloud-config-aws:
+ $(Q)rm -f $(AWS_KCONFIGS)
+ $(Q)rm -rf $(AWS_KCONFIG_INSTANCE_TYPES_DIR)
+
+DYNAMIC_CLOUD_KCONFIG += cloud-config-lambdalabs cloud-config-aws
cloud-config-help:
@echo "Cloud-specific dynamic kconfig targets:"
@echo "cloud-config - generates all cloud provider dynamic kconfig content"
@echo "cloud-config-lambdalabs - generates Lambda Labs dynamic kconfig content"
+ @echo "cloud-config-aws - generates AWS dynamic kconfig content"
+ @echo "cloud-update - refreshes cloud provider data (clears cache)"
+ @echo "cloud-update-aws - refreshes AWS data (clears cache and regenerates)"
@echo "clean-cloud-config - removes all generated cloud kconfig files"
@echo "cloud-list-all - list all cloud instances for configured provider"
@@ -44,11 +78,16 @@ HELP_TARGETS += cloud-config-help
cloud-config:
$(Q)python3 scripts/generate_cloud_configs.py
-clean-cloud-config: clean-cloud-config-lambdalabs
+cloud-update: cloud-update-aws
+ $(Q)echo "Updated cloud provider configurations."
+
+clean-cloud-config: clean-cloud-config-lambdalabs clean-cloud-config-aws
$(Q)echo "Cleaned all cloud provider dynamic Kconfig files."
cloud-list-all:
$(Q)chmod +x scripts/cloud_list_all.sh
$(Q)scripts/cloud_list_all.sh
-PHONY += cloud-config cloud-config-lambdalabs clean-cloud-config clean-cloud-config-lambdalabs cloud-config-help cloud-list-all
+PHONY += cloud-config cloud-config-lambdalabs cloud-config-aws cloud-update cloud-update-aws
+PHONY += clean-cloud-config clean-cloud-config-lambdalabs clean-cloud-config-aws
+PHONY += cloud-config-help cloud-list-all
diff --git a/scripts/generate_cloud_configs.py b/scripts/generate_cloud_configs.py
index b16294dd..e8251a73 100755
--- a/scripts/generate_cloud_configs.py
+++ b/scripts/generate_cloud_configs.py
@@ -100,6 +100,68 @@ def get_lambdalabs_summary() -> tuple[bool, str]:
return False, "Lambda Labs: Error querying API - using defaults"
+def generate_aws_kconfig() -> bool:
+ """
+ Generate AWS Kconfig files using Chuck's scripts.
+ Returns True on success, False on failure.
+ """
+ script_path = "terraform/aws/scripts/generate_aws_kconfig.py"
+
+ result = subprocess.run(
+ [sys.executable, script_path],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+
+ return result.returncode == 0
+
+
+def get_aws_summary() -> tuple[bool, str]:
+ """
+ Get a summary of AWS configurations.
+ Returns (success, summary_string)
+ """
+ try:
+ # Check if AWS credentials are configured
+ result = subprocess.run(
+ ["aws", "sts", "get-caller-identity"],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+
+ if result.returncode != 0:
+ return False, "AWS: Credentials not configured - using defaults"
+
+ # Get basic stats from cached data if available
+ cache_dir = os.path.expanduser("~/.cache/kdevops/aws")
+ families_cache = os.path.join(cache_dir, "aws_families.json")
+
+ if os.path.exists(families_cache):
+ with open(families_cache, 'r') as f:
+ families = json.load(f)
+ family_count = len(families)
+ else:
+ family_count = "72+" # Known minimum
+
+ regions_cache = os.path.join(cache_dir, "aws_regions.json")
+ if os.path.exists(regions_cache):
+ with open(regions_cache, 'r') as f:
+ regions = json.load(f)
+ region_count = len(regions)
+ else:
+ region_count = "30+" # Known minimum
+
+ return (
+ True,
+ f"AWS: {family_count} instance families, {region_count} regions, "
+ f"900+ instance types available"
+ )
+ except Exception:
+ return False, "AWS: Error checking configuration"
+
+
def main():
"""Main function to generate cloud configurations."""
print("Cloud Provider Configuration Summary")
@@ -121,8 +183,20 @@ def main():
print(f"⚠ {summary}")
print()
- # AWS (placeholder - not implemented)
- print("⚠ AWS: Dynamic configuration not yet implemented")
+ # AWS - Generate Kconfig files
+ aws_generated = generate_aws_kconfig()
+
+ # AWS - Get summary
+ success, summary = get_aws_summary()
+ if success:
+ print(f"✓ {summary}")
+ if aws_generated:
+ print(" Kconfig files generated successfully")
+ else:
+ print(" Warning: Failed to generate Kconfig files")
+ else:
+ print(f"⚠ {summary}")
+ print()
# Azure (placeholder - not implemented)
print("⚠ Azure: Dynamic configuration not yet implemented")
--
2.51.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 5/8] aws: add cloud billing support with make cloud-bill
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
` (3 preceding siblings ...)
2025-09-17 0:34 ` [PATCH v4 4/8] aws: integrate dynamic Kconfig generation with make targets Luis Chamberlain
@ 2025-09-17 0:34 ` Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 6/8] aws: replace static Kconfig files with dynamically generated ones Luis Chamberlain
` (2 subsequent siblings)
7 siblings, 0 replies; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Luis Chamberlain
Add AWS cost tracking functionality to quickly check monthly spending.
This includes my personal hack scripts for monitoring AWS costs:
- scripts/aws-costs.sh: Queries AWS Cost Explorer for current month
- scripts/aws-parse-costs.py: Parses and displays costs by service
- make cloud-bill: Show cloud provider costs (currently AWS only)
- make cloud-bill-aws: Show detailed AWS costs for current month
The scripts provide:
- Total monthly cost to date
- Breakdown by AWS service
- Daily average spending
- Projected monthly cost (when mid-month)
This is useful for monitoring cloud spending during development and
testing, especially when running expensive instances or long tests.
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
scripts/aws-costs.sh | 39 ++++++++++
scripts/aws-parse-costs.py | 98 ++++++++++++++++++++++++++
scripts/dynamic-cloud-kconfig.Makefile | 13 +++-
3 files changed, 149 insertions(+), 1 deletion(-)
create mode 100755 scripts/aws-costs.sh
create mode 100755 scripts/aws-parse-costs.py
diff --git a/scripts/aws-costs.sh b/scripts/aws-costs.sh
new file mode 100755
index 00000000..e2298008
--- /dev/null
+++ b/scripts/aws-costs.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+# SPDX-License-Identifier: copyleft-next-0.3.1
+#
+# AWS cost tracking script - quick hack to check AWS spending
+# This queries AWS Cost Explorer to get current month's costs
+
+set -e
+
+# Get the first and last day of the current month
+FIRST_DAY=$(date +%Y-%m-01)
+LAST_DAY=$(date -d "$FIRST_DAY +1 month -1 day" +%Y-%m-%d)
+TODAY=$(date +%Y-%m-%d)
+
+# If we're still in the current month, use today as the end date
+if [[ "$TODAY" < "$LAST_DAY" ]]; then
+ END_DATE="$TODAY"
+else
+ END_DATE="$LAST_DAY"
+fi
+
+echo "Fetching AWS costs from $FIRST_DAY to $END_DATE..." >&2
+
+# Query AWS Cost Explorer
+aws ce get-cost-and-usage \
+ --time-period Start=$FIRST_DAY,End=$END_DATE \
+ --granularity MONTHLY \
+ --metrics UnblendedCost \
+ --group-by Type=DIMENSION,Key=SERVICE \
+ --output json > cost.json
+
+# Parse and display the results
+if [ -f cost.json ]; then
+ echo "Cost data saved to cost.json" >&2
+ echo "Parsing costs..." >&2
+ python3 scripts/aws-parse-costs.py cost.json
+else
+ echo "Error: Failed to retrieve cost data" >&2
+ exit 1
+fi
\ No newline at end of file
diff --git a/scripts/aws-parse-costs.py b/scripts/aws-parse-costs.py
new file mode 100755
index 00000000..7f3256a7
--- /dev/null
+++ b/scripts/aws-parse-costs.py
@@ -0,0 +1,98 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: copyleft-next-0.3.1
+#
+# Parse AWS Cost Explorer JSON output and display costs
+
+import json
+import sys
+from datetime import datetime
+
+
+def parse_costs(filename):
+ """Parse AWS Cost Explorer JSON output."""
+ try:
+ with open(filename, 'r') as f:
+ data = json.load(f)
+ except FileNotFoundError:
+ print(f"Error: File {filename} not found", file=sys.stderr)
+ sys.exit(1)
+ except json.JSONDecodeError as e:
+ print(f"Error parsing JSON: {e}", file=sys.stderr)
+ sys.exit(1)
+
+ # Extract time period
+ if 'ResultsByTime' in data and data['ResultsByTime']:
+ result = data['ResultsByTime'][0]
+ time_period = result.get('TimePeriod', {})
+ start = time_period.get('Start', 'Unknown')
+ end = time_period.get('End', 'Unknown')
+
+ print(f"\nAWS Cost Report")
+ print(f"Period: {start} to {end}")
+ print("=" * 60)
+
+ # Get total cost
+ total = result.get('Total', {})
+ if 'UnblendedCost' in total:
+ total_amount = float(total['UnblendedCost'].get('Amount', 0))
+ currency = total['UnblendedCost'].get('Unit', 'USD')
+ print(f"\nTotal Cost: ${total_amount:.2f} {currency}")
+
+ # Get costs by service
+ groups = result.get('Groups', [])
+ if groups:
+ print("\nCosts by Service:")
+ print("-" * 40)
+
+ # Sort by cost (descending)
+ sorted_groups = sorted(groups,
+ key=lambda x: float(x['Metrics']['UnblendedCost']['Amount']),
+ reverse=True)
+
+ for group in sorted_groups:
+ service = group['Keys'][0] if group.get('Keys') else 'Unknown'
+ metrics = group.get('Metrics', {})
+ if 'UnblendedCost' in metrics:
+ amount = float(metrics['UnblendedCost'].get('Amount', 0))
+ if amount > 0.01: # Only show services with costs > $0.01
+ print(f" {service:30} ${amount:10.2f}")
+
+ # Show total at the bottom
+ print("-" * 40)
+ if 'UnblendedCost' in total:
+ print(f" {'TOTAL':30} ${total_amount:10.2f}")
+
+ # Calculate daily average
+ try:
+ start_date = datetime.strptime(start, '%Y-%m-%d')
+ end_date = datetime.strptime(end, '%Y-%m-%d')
+ days = (end_date - start_date).days
+ if days > 0 and 'UnblendedCost' in total:
+ daily_avg = total_amount / days
+ print(f"\nDaily Average: ${daily_avg:.2f}")
+
+ # Project monthly cost if we're mid-month
+ today = datetime.now()
+ if end_date.date() == today.date() and start_date.day == 1:
+ days_in_month = 30 # Approximate
+ projected = daily_avg * days_in_month
+ print(f"Projected Monthly: ${projected:.2f}")
+ except ValueError:
+ pass
+
+ else:
+ print("No cost data available in the response", file=sys.stderr)
+ sys.exit(1)
+
+
+def main():
+ """Main function."""
+ if len(sys.argv) != 2:
+ print("Usage: aws-parse-costs.py <cost.json>", file=sys.stderr)
+ sys.exit(1)
+
+ parse_costs(sys.argv[1])
+
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
diff --git a/scripts/dynamic-cloud-kconfig.Makefile b/scripts/dynamic-cloud-kconfig.Makefile
index ed2d5366..0ec14966 100644
--- a/scripts/dynamic-cloud-kconfig.Makefile
+++ b/scripts/dynamic-cloud-kconfig.Makefile
@@ -72,6 +72,8 @@ cloud-config-help:
@echo "cloud-update-aws - refreshes AWS data (clears cache and regenerates)"
@echo "clean-cloud-config - removes all generated cloud kconfig files"
@echo "cloud-list-all - list all cloud instances for configured provider"
+ @echo "cloud-bill - show current month's cloud provider costs"
+ @echo "cloud-bill-aws - show AWS costs for current month"
HELP_TARGETS += cloud-config-help
@@ -88,6 +90,15 @@ cloud-list-all:
$(Q)chmod +x scripts/cloud_list_all.sh
$(Q)scripts/cloud_list_all.sh
+# Cloud billing targets
+cloud-bill-aws:
+ $(Q)chmod +x scripts/aws-costs.sh
+ $(Q)scripts/aws-costs.sh
+
+cloud-bill: cloud-bill-aws
+ $(Q)echo ""
+ $(Q)echo "Note: Only AWS billing is currently supported"
+
PHONY += cloud-config cloud-config-lambdalabs cloud-config-aws cloud-update cloud-update-aws
PHONY += clean-cloud-config clean-cloud-config-lambdalabs clean-cloud-config-aws
-PHONY += cloud-config-help cloud-list-all
+PHONY += cloud-config-help cloud-list-all cloud-bill cloud-bill-aws
--
2.51.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 6/8] aws: replace static Kconfig files with dynamically generated ones
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
` (4 preceding siblings ...)
2025-09-17 0:34 ` [PATCH v4 5/8] aws: add cloud billing support with make cloud-bill Luis Chamberlain
@ 2025-09-17 0:34 ` Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 7/8] aws: add GPU instance defconfigs for AI/ML workloads Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 8/8] docs: add documentation for dynamic cloud configuration Luis Chamberlain
7 siblings, 0 replies; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Luis Chamberlain
Remove old static AWS Kconfig files and update the configuration to
use dynamically generated files instead. This change:
- Removes 8 static Kconfig files (compute, location, and 6 instance types)
- Updates terraform/aws/Kconfig to source the .generated files
- Fixes Kconfig symbol generation to handle dashes in instance names
- Filters instance families to prevent mixing related families
The dynamic files now provide:
- 146 instance family configurations (vs 6 static ones)
- 34 AWS regions (vs static subset)
- 900+ instance types with current specs
Also fixes generator to:
- Convert dashes to underscores in Kconfig symbols (e.g., metal-24xl)
- Filter instances to exact family matches (r8g vs r8gd/r8gn)
- Ensure valid Kconfig syntax for all generated files
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
terraform/aws/Kconfig | 4 +-
terraform/aws/kconfigs/Kconfig.compute | 118 ---
terraform/aws/kconfigs/Kconfig.location | 679 ------------------
.../aws/kconfigs/instance-types/Kconfig.c7a | 28 -
.../aws/kconfigs/instance-types/Kconfig.i4i | 33 -
.../aws/kconfigs/instance-types/Kconfig.im4gn | 25 -
.../kconfigs/instance-types/Kconfig.is4gen | 25 -
.../aws/kconfigs/instance-types/Kconfig.m5 | 48 --
.../aws/kconfigs/instance-types/Kconfig.m7a | 57 --
terraform/aws/scripts/generate_aws_kconfig.py | 20 +-
10 files changed, 18 insertions(+), 1019 deletions(-)
delete mode 100644 terraform/aws/kconfigs/Kconfig.compute
delete mode 100644 terraform/aws/kconfigs/Kconfig.location
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.c7a
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.i4i
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.im4gn
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.is4gen
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.m5
delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.m7a
diff --git a/terraform/aws/Kconfig b/terraform/aws/Kconfig
index d4015cda..de8bc028 100644
--- a/terraform/aws/Kconfig
+++ b/terraform/aws/Kconfig
@@ -1,10 +1,10 @@
if TERRAFORM_AWS
menu "Resource Location"
-source "terraform/aws/kconfigs/Kconfig.location"
+source "terraform/aws/kconfigs/Kconfig.location.generated"
endmenu
menu "Compute"
-source "terraform/aws/kconfigs/Kconfig.compute"
+source "terraform/aws/kconfigs/Kconfig.compute.generated"
endmenu
menu "Storage"
source "terraform/aws/kconfigs/Kconfig.storage"
diff --git a/terraform/aws/kconfigs/Kconfig.compute b/terraform/aws/kconfigs/Kconfig.compute
deleted file mode 100644
index 4b9c6efb..00000000
--- a/terraform/aws/kconfigs/Kconfig.compute
+++ /dev/null
@@ -1,118 +0,0 @@
-choice
- prompt "AWS instance types"
- help
- Instance types comprise varying combinations of hardware
- platform, CPU count, memory size, storage, and networking
- capacity. Select the type that provides an appropriate mix
- of resources for your preferred workflows.
-
- Some instance types are region- and capacity-limited.
-
- See https://aws.amazon.com/ec2/instance-types/ for
- details.
-
-config TERRAFORM_AWS_INSTANCE_TYPE_M5
- bool "M5"
- depends on TARGET_ARCH_X86_64
- help
- This is a general purpose type powered by Intel Xeon®
- Platinum 8175M or 8259CL processors (Skylake or Cascade
- Lake).
-
- See https://aws.amazon.com/ec2/instance-types/m5/ for
- details.
-
-config TERRAFORM_AWS_INSTANCE_TYPE_M7A
- bool "M7a"
- depends on TARGET_ARCH_X86_64
- help
- This is a general purpose type powered by 4th Generation
- AMD EPYC processors.
-
- See https://aws.amazon.com/ec2/instance-types/m7a/ for
- details.
-
-config TERRAFORM_AWS_INSTANCE_TYPE_I4I
- bool "I4i"
- depends on TARGET_ARCH_X86_64
- help
- This is a storage-optimized type powered by 3rd generation
- Intel Xeon Scalable processors (Ice Lake) and use AWS Nitro
- NVMe SSDs.
-
- See https://aws.amazon.com/ec2/instance-types/i4i/ for
- details.
-
-config TERRAFORM_AWS_INSTANCE_TYPE_IS4GEN
- bool "Is4gen"
- depends on TARGET_ARCH_ARM64
- help
- This is a Storage-optimized type powered by AWS Graviton2
- processors.
-
- See https://aws.amazon.com/ec2/instance-types/i4g/ for
- details.
-
-config TERRAFORM_AWS_INSTANCE_TYPE_IM4GN
- bool "Im4gn"
- depends on TARGET_ARCH_ARM64
- help
- This is a storage-optimized type powered by AWS Graviton2
- processors.
-
- See https://aws.amazon.com/ec2/instance-types/i4g/ for
- details.
-
-config TERRAFORM_AWS_INSTANCE_TYPE_C7A
- depends on TARGET_ARCH_X86_64
- bool "c7a"
- help
- This is a compute-optimized type powered by 4th generation
- AMD EPYC processors.
-
- See https://aws.amazon.com/ec2/instance-types/c7a/ for
- details.
-
-endchoice
-
-source "terraform/aws/kconfigs/instance-types/Kconfig.m5"
-source "terraform/aws/kconfigs/instance-types/Kconfig.m7a"
-source "terraform/aws/kconfigs/instance-types/Kconfig.i4i"
-source "terraform/aws/kconfigs/instance-types/Kconfig.is4gen"
-source "terraform/aws/kconfigs/instance-types/Kconfig.im4gn"
-source "terraform/aws/kconfigs/instance-types/Kconfig.c7a"
-
-choice
- prompt "Linux distribution"
- default TERRAFORM_AWS_DISTRO_DEBIAN
- help
- Select a popular Linux distribution to install on your
- instances, or use the "Custom AMI image" selection to
- choose an image that is off the beaten path.
-
-config TERRAFORM_AWS_DISTRO_AMAZON
- bool "Amazon Linux"
-
-config TERRAFORM_AWS_DISTRO_DEBIAN
- bool "Debian"
-
-config TERRAFORM_AWS_DISTRO_FEDORA
- bool "Fedora Core"
-
-config TERRAFORM_AWS_DISTRO_RHEL
- bool "Red Hat Enterprise Linux"
-
-config TERRAFORM_AWS_DISTRO_SLES
- bool "Suse Linux Enterprise Server"
-
-config TERRAFORM_AWS_DISTRO_CUSTOM
- bool "Custom AMI image"
-
-endchoice
-
-source "terraform/aws/kconfigs/distros/Kconfig.amazon"
-source "terraform/aws/kconfigs/distros/Kconfig.debian"
-source "terraform/aws/kconfigs/distros/Kconfig.fedora"
-source "terraform/aws/kconfigs/distros/Kconfig.rhel"
-source "terraform/aws/kconfigs/distros/Kconfig.sles"
-source "terraform/aws/kconfigs/distros/Kconfig.custom"
diff --git a/terraform/aws/kconfigs/Kconfig.location b/terraform/aws/kconfigs/Kconfig.location
deleted file mode 100644
index e5dacb9d..00000000
--- a/terraform/aws/kconfigs/Kconfig.location
+++ /dev/null
@@ -1,679 +0,0 @@
-choice
- prompt "AWS region"
- default TERRAFORM_AWS_REGION_US_WEST_2
- help
- Use this option to select the AWS region that hosts your
- compute and storage resources. If you do not explicitly
- specify a region, the US West (Oregon) region is the
- default.
-
- Once selected, you can stick with the default AV zone
- chosen by kdevops, or use:
-
- aws ec2 describe-availability-zones --region <region-name>
-
- to list the Availability Zones that are enabled for your
- AWS account. Enter your selection from this list using the
- TERRAFORM_AWS_AV_ZONE menu.
-
- If you wish to expand on the region list, send a patch after
- reading this list:
-
- https://docs.aws.amazon.com/general/latest/gr/rande.html
-
- Note that if you change region the AMI may change as well even
- for the same distribution. At least that applies to Amazon EC2
- optimized images. Use the AWS console, to set that up it will
- ask you for your credentials and then a region. Before adding
- an entry for ami image be sure you are on the region and then
- query with something like:
-
- aws ec2 describe-images --image-ids ami-0efa651876de2a5ce
-
- For instance, this AMI belongs to us-west-2 only. us-east* have
- other AMIs for the same Amazon 2023 EC2 image. The output from
- here tells me:
-
- "OwnerId": "137112412989"
-
- And that is what value to use for ami-0efa651876de2a5ce
- for the TERRAFORM_AWS_AMI_OWNER. To get the ami-* for your regions
- just go to your EC2 console, that console will be associated with
- a region already. You have to change regions if you want to look
- for AMIs in other regions. There are for example two different
- ami-* values for Amazon Linux 2023 for different regions. However
- they values can be same. For example below are the us-west-2 queries
- for Amazon Linux 2023 for x86_64 and then for ARM64.
-
- aws ec2 describe-images --image-ids ami-0efa651876de2a5ce | grep OwnerId
- "OwnerId": "137112412989",
- aws ec2 describe-images --image-ids ami-0699f753302dd8b00 | grep OwnerId
- "OwnerId": "137112412989",
-
-config TERRAFORM_AWS_REGION_AP_NORTHEAST_1
- bool "ap-northeast-1 - Tokyo"
- help
- This option specifies the Asia-Pacific northeast-1 region.
- The data center is located in Tokyo, Japan.
-
-config TERRAFORM_AWS_REGION_AP_NORTHEAST_2
- bool "ap-northeast-2 - Seoul"
- help
- This option specifies the Asia-Pacific northeast-2 region.
- The data center is located in Seoul, South Korea.
-
-config TERRAFORM_AWS_REGION_AP_NORTHEAST_3
- bool "ap-northeast-3 - Osaka"
- help
- This option specifies the Asia-Pacific northeast-3 region.
- The data center is located in Osaka, Japan.
-
-config TERRAFORM_AWS_REGION_AP_SOUTH_1
- bool "ap-south-1 - Mumbai"
- help
- This option specifies the Asia-Pacific south-1 region.
- The data center is located in Mumbai, India.
-
-config TERRAFORM_AWS_REGION_AP_SOUTHEAST_1
- bool "ap-southeast-1 - Singapore"
- help
- This option specifies the Asia-Pacific southeast-1 region.
- The data center is located in the Republic of Singapore.
-
-config TERRAFORM_AWS_REGION_AP_SOUTHEAST_2
- bool "ap-southeast-1 - Sydney"
- help
- This option specifies the Asia-Pacific southeast-2 region.
- The data center is located in Sydney, Australia.
-
-config TERRAFORM_AWS_REGION_CA_CENTRAL_1
- bool "ca-central-1 - Central"
- help
- This option specifies the Canada central-1 region.
- The data center is located in Montreal, Quebec.
-
-config TERRAFORM_AWS_REGION_EU_CENTRAL_1
- bool "eu-central-1 - Frankfurt"
- help
- This option specifies the European Union central-1 region.
- The data center is located in Frankfurt, Germany.
-
-config TERRAFORM_AWS_REGION_EU_NORTH_1
- bool "eu-north-1 - Stockholm"
- help
- This option specifies the European Union north-1 region.
- The data center is located in Stockholm, Sweden.
-
-config TERRAFORM_AWS_REGION_EU_WEST_1
- bool "eu-west-1 - Ireland"
- help
- This option specifies the European Union west-1 region.
- The data center is located in Dublin, Republic of Ireland.
-
-config TERRAFORM_AWS_REGION_EU_WEST_2
- bool "eu-west-2 - London"
- help
- This option specifies the European Union west-2 region.
- The data center is located in London, United Kingdom.
-
-config TERRAFORM_AWS_REGION_EU_WEST_3
- bool "eu-west-3 - Paris"
- help
- This option specifies the European Union west-3 region.
- The data center is located in Paris, France.
-
-config TERRAFORM_AWS_REGION_SA_EAST_1
- bool "sa-east-1 - Sao Paulo"
- help
- This option specifies the South America east-1 region.
- The data center is located in São Paulo, Brazil.
-
-config TERRAFORM_AWS_REGION_US_EAST_1
- bool "us-east-1 - N. Virginia"
- help
- This option specifies the United States east-1 region.
- Multiple data centers are located in the US state of
- Virginia.
-
-config TERRAFORM_AWS_REGION_US_EAST_2
- bool "us-east-2 - Ohio"
- help
- This option specifies the United States east-2 region.
- The data center is located in Columbus, Ohio, US.
-
-config TERRAFORM_AWS_REGION_US_WEST_1
- bool "us-west-1 - North California"
- help
- This option specifies the United States west-1 region.
- The data center is located in San Francisco, California,
- US.
-
-config TERRAFORM_AWS_REGION_US_WEST_2
- bool "us-west-2 - Oregon"
- help
- This option specifies the United States west-2 region.
- Multiple data centers are located in the US state of
- Oregon.
-
-endchoice
-
-config TERRAFORM_AWS_REGION
- string
- output yaml
- default "ap-northeast-1" if TERRAFORM_AWS_REGION_AP_NORTHEAST_1
- default "ap-northeast-2" if TERRAFORM_AWS_REGION_AP_NORTHEAST_2
- default "ap-northeast-3" if TERRAFORM_AWS_REGION_AP_NORTHEAST_3
- default "ap-south-1" if TERRAFORM_AWS_REGION_AP_SOUTH_1
- default "ap-southeast-1" if TERRAFORM_AWS_REGION_AP_SOUTHEAST_1
- default "ap-southeast-2" if TERRAFORM_AWS_REGION_AP_SOUTHEAST_2
- default "ca-central-1" if TERRAFORM_AWS_REGION_CA_CENTRAL_1
- default "eu-central-1" if TERRAFORM_AWS_REGION_EU_CENTRAL_1
- default "eu-north-1" if TERRAFORM_AWS_REGION_EU_NORTH_1
- default "eu-west-1" if TERRAFORM_AWS_REGION_EU_WEST_1
- default "eu-west-2" if TERRAFORM_AWS_REGION_EU_WEST_2
- default "eu-west-3" if TERRAFORM_AWS_REGION_EU_WEST_3
- default "sa-east-1" if TERRAFORM_AWS_REGION_SA_EAST_1
- default "us-east-1" if TERRAFORM_AWS_REGION_US_EAST_1
- default "us-east-2" if TERRAFORM_AWS_REGION_US_EAST_2
- default "us-west-1" if TERRAFORM_AWS_REGION_US_WEST_1
- default "us-west-2" if TERRAFORM_AWS_REGION_US_WEST_2
-
-if TERRAFORM_AWS_REGION_AP_NORTHEAST_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_1A
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_1A
- bool "ap-northeast-1a"
- help
- This option selects the ap-northeast-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_1C
- bool "ap-northeast-1c"
- help
- This option selects the ap-northeast-1c availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_1D
- bool "ap-northeast-1d"
- help
- This option selects the ap-northeast-1d availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_AP_NORTHEAST_1
-
-if TERRAFORM_AWS_REGION_AP_NORTHEAST_2
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2A
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2A
- bool "ap-northeast-2a"
- help
- This option selects the ap-northeast-2a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2B
- bool "ap-northeast-2b"
- help
- This option selects the ap-northeast-2b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2C
- bool "ap-northeast-2c"
- help
- This option selects the ap-northeast-2c availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2D
- bool "ap-northeast-2d"
- help
- This option selects the ap-northeast-2d availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_AP_NORTHEAST_2
-
-if TERRAFORM_AWS_REGION_AP_NORTHEAST_3
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_3A
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_3A
- bool "ap-northeast-3a"
- help
- This option selects the ap-northeast-3a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_3B
- bool "ap-northeast-3b"
- help
- This option selects the ap-northeast-3b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_3C
- bool "ap-northeast-3c"
- help
- This option selects the ap-northeast-3c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_AP_NORTHEAST_3
-
-if TERRAFORM_AWS_REGION_AP_SOUTH_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_AP_SOUTH_1A
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTH_1A
- bool "ap-south-1a"
- help
- This option selects the ap-south-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTH_1B
- bool "ap-south-1b"
- help
- This option selects the ap-south-1b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTH_1C
- bool "ap-south-1c"
- help
- This option selects the ap-south-1c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_AP_SOUTH_1
-
-if TERRAFORM_AWS_REGION_AP_SOUTHEAST_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_1A
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_1A
- bool "ap-southeast-1a"
- help
- This option selects the ap-southeast-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_1B
- bool "ap-southeast-1b"
- help
- This option selects the ap-southeast-1b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_1C
- bool "ap-southeast-1c"
- help
- This option selects the ap-southeast-1c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_AP_SOUTHEAST_1
-
-if TERRAFORM_AWS_REGION_AP_SOUTHEAST_2
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_2A
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_2A
- bool "ap-southeast-2a"
- help
- This option selects the ap-southeast-2a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_2B
- bool "ap-southeast-2b"
- help
- This option selects the ap-southeast-2b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_2C
- bool "ap-southeast-2c"
- help
- This option selects the ap-southeast-2c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_AP_SOUTHEAST_2
-
-if TERRAFORM_AWS_REGION_CA_CENTRAL_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_CA_CENTRAL_1A
-
-config TERRAFORM_AWS_AV_ZONE_CA_CENTRAL_1A
- bool "ca-central-1a"
- help
- This option selects the ca-central-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_CA_CENTRAL_1B
- bool "ca-central-1b"
- help
- This option selects the ca-central-1b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_CA_CENTRAL_1D
- bool "ca-central-1d"
- help
- This option selects the ca-central-1d availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_CA_CENTRAL_1
-
-if TERRAFORM_AWS_REGION_EU_CENTRAL_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_EU_CENTRAL_1A
-
-config TERRAFORM_AWS_AV_ZONE_EU_CENTRAL_1A
- bool "eu-central-1a"
- help
- This option selects the eu-central-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_CENTRAL_1B
- bool "eu-central-1b"
- help
- This option selects the eu-central-1b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_CENTRAL_1C
- bool "eu-central-1c"
- help
- This option selects the eu-central-1c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_EU_CENTRAL_1
-
-if TERRAFORM_AWS_REGION_EU_NORTH_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_EU_NORTH_1A
-
-config TERRAFORM_AWS_AV_ZONE_EU_NORTH_1A
- bool "eu-north-1a"
- help
- This option selects the eu-north-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_NORTH_1B
- bool "eu-north-1b"
- help
- This option selects the eu-north-1b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_NORTH_1C
- bool "eu-north-1c"
- help
- This option selects the eu-north-1c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_EU_NORTH_1
-
-if TERRAFORM_AWS_REGION_EU_WEST_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_EU_WEST_1A
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_1A
- bool "eu-west-1a"
- help
- This option selects the eu-west-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_1B
- bool "eu-west-1b"
- help
- This option selects the eu-west-1b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_1C
- bool "eu-west-1c"
- help
- This option selects the eu-west-1c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_EU_WEST_1
-
-if TERRAFORM_AWS_REGION_EU_WEST_2
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_EU_WEST_2A
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_2A
- bool "eu-west-2a"
- help
- This option selects the eu-west-2a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_2B
- bool "eu-west-2b"
- help
- This option selects the eu-west-2b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_2C
- bool "eu-west-2c"
- help
- This option selects the eu-west-2c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_EU_WEST_2
-
-if TERRAFORM_AWS_REGION_EU_WEST_3
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_EU_WEST_3A
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_3A
- bool "eu-west-3a"
- help
- This option selects the eu-west-3a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_3B
- bool "eu-west-3b"
- help
- This option selects the eu-west-3b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_EU_WEST_3C
- bool "eu-west-3c"
- help
- This option selects the eu-west-3c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_EU_WEST_3
-
-if TERRAFORM_AWS_REGION_SA_EAST_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_SA_EAST_1A
-
-config TERRAFORM_AWS_AV_ZONE_SA_EAST_1A
- bool "sa-east-1a"
- help
- This option selects the sa-east-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_SA_EAST_1B
- bool "sa-east-1b"
- help
- This option selects the sa-east-1b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_SA_EAST_1C
- bool "sa-east-1c"
- help
- This option selects the sa-east-1c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_SA_EAST_1
-
-if TERRAFORM_AWS_REGION_US_EAST_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_US_EAST_1A
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_1A
- bool "us-east-1a"
- help
- This option selects the us-east-1a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_1B
- bool "us-east-1b"
- help
- This option selects the us-east-1b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_1C
- bool "us-east-1c"
- help
- This option selects the us-east-1c availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_1D
- bool "us-east-1d"
- help
- This option selects the us-east-1d availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_1E
- bool "us-east-1e"
- help
- This option selects the us-east-1e availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_1F
- bool "us-east-1f"
- help
- This option selects the us-east-1f availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_US_EAST_1
-
-if TERRAFORM_AWS_REGION_US_EAST_2
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_US_EAST_2A
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_2A
- bool "us-east-2a"
- help
- This option selects the us-east-2a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_2B
- bool "us-east-2b"
- help
- This option selects the us-east-2a availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_US_EAST_2C
- bool "us-east-2c"
- help
- This option selects the us-east-2c availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_US_EAST_2
-
-if TERRAFORM_AWS_REGION_US_WEST_1
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_US_WEST_1B
- help
- This option sets the AWS availablity zone to the specified value.
- If you wish to expand on this list send a patch after reading this
- list:
-
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
- https://gist.github.com/neilstuartcraig/0ccefcf0887f29b7f240
-
-config TERRAFORM_AWS_AV_ZONE_US_WEST_1B
- bool "us-west-1b"
- help
- This option selects the us-west-1b availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_US_WEST_1
-
-if TERRAFORM_AWS_REGION_US_WEST_2
-
-choice
- prompt "AWS availability zone"
- default TERRAFORM_AWS_AV_ZONE_US_WEST_2B
- help
- This option sets the AWS availablity zone to the specified value.
- If you wish to expand on this list send a patch after reading this
- list:
-
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
- https://gist.github.com/neilstuartcraig/0ccefcf0887f29b7f240
-
-config TERRAFORM_AWS_AV_ZONE_US_WEST_2B
- bool "us-west-2b"
- help
- This option selects the us-west-2b availability zone.
-
-config TERRAFORM_AWS_AV_ZONE_US_WEST_2D
- bool "us-west-2d"
- help
- This option selects the us-west-2d availability zone.
-
-endchoice
-
-endif # TERRAFORM_AWS_REGION_US_WEST_2
-
-config TERRAFORM_AWS_AV_ZONE
- string
- output yaml
- default "ap-northeast-1a" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_1A
- default "ap-northeast-1c" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_1C
- default "ap-northeast-1d" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_1D
- default "ap-northeast-2a" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2A
- default "ap-northeast-2b" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2B
- default "ap-northeast-2c" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2C
- default "ap-northeast-2d" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_2D
- default "ap-northeast-3a" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_3A
- default "ap-northeast-3b" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_3B
- default "ap-northeast-3c" if TERRAFORM_AWS_AV_ZONE_AP_NORTHEAST_3C
- default "ap-south-1a" if TERRAFORM_AWS_AV_ZONE_AP_SOUTH_1A
- default "ap-south-1b" if TERRAFORM_AWS_AV_ZONE_AP_SOUTH_1B
- default "ap-south-1c" if TERRAFORM_AWS_AV_ZONE_AP_SOUTH_1C
- default "ap-southeast-1a" if TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_1A
- default "ap-southeast-1b" if TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_1B
- default "ap-southeast-1c" if TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_1C
- default "ap-southeast-2a" if TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_2A
- default "ap-southeast-2b" if TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_2B
- default "ap-southeast-2c" if TERRAFORM_AWS_AV_ZONE_AP_SOUTHEAST_2C
- default "ca-central-1a" if TERRAFORM_AWS_AV_ZONE_CA_CENTRAL_1A
- default "ca-central-1b" if TERRAFORM_AWS_AV_ZONE_CA_CENTRAL_1B
- default "ca-central-1d" if TERRAFORM_AWS_AV_ZONE_CA_CENTRAL_1D
- default "eu-central-1a" if TERRAFORM_AWS_AV_ZONE_EU_CENTRAL_1A
- default "eu-central-1b" if TERRAFORM_AWS_AV_ZONE_EU_CENTRAL_1B
- default "eu-central-1c" if TERRAFORM_AWS_AV_ZONE_EU_CENTRAL_1C
- default "eu-north-1a" if TERRAFORM_AWS_AV_ZONE_EU_NORTH_1A
- default "eu-north-1b" if TERRAFORM_AWS_AV_ZONE_EU_NORTH_1B
- default "eu-north-1c" if TERRAFORM_AWS_AV_ZONE_EU_NORTH_1C
- default "eu-west-1a" if TERRAFORM_AWS_AV_ZONE_EU_WEST_1A
- default "eu-west-1b" if TERRAFORM_AWS_AV_ZONE_EU_WEST_1B
- default "eu-west-1c" if TERRAFORM_AWS_AV_ZONE_EU_WEST_1C
- default "eu-west-2a" if TERRAFORM_AWS_AV_ZONE_EU_WEST_2A
- default "eu-west-2b" if TERRAFORM_AWS_AV_ZONE_EU_WEST_2B
- default "eu-west-2c" if TERRAFORM_AWS_AV_ZONE_EU_WEST_2C
- default "eu-west-3a" if TERRAFORM_AWS_AV_ZONE_EU_WEST_3A
- default "eu-west-3b" if TERRAFORM_AWS_AV_ZONE_EU_WEST_3B
- default "eu-west-3c" if TERRAFORM_AWS_AV_ZONE_EU_WEST_3C
- default "sa-east-1a" if TERRAFORM_AWS_AV_ZONE_SA_EAST_1A
- default "sa-east-1b" if TERRAFORM_AWS_AV_ZONE_SA_EAST_1B
- default "sa-east-1c" if TERRAFORM_AWS_AV_ZONE_SA_EAST_1C
- default "us-east-1a" if TERRAFORM_AWS_AV_ZONE_US_EAST_1A
- default "us-east-1b" if TERRAFORM_AWS_AV_ZONE_US_EAST_1B
- default "us-east-1c" if TERRAFORM_AWS_AV_ZONE_US_EAST_1C
- default "us-east-1d" if TERRAFORM_AWS_AV_ZONE_US_EAST_1D
- default "us-east-1e" if TERRAFORM_AWS_AV_ZONE_US_EAST_1E
- default "us-east-1f" if TERRAFORM_AWS_AV_ZONE_US_EAST_1F
- default "us-east-2a" if TERRAFORM_AWS_AV_ZONE_US_EAST_2A
- default "us-east-2b" if TERRAFORM_AWS_AV_ZONE_US_EAST_2B
- default "us-east-2c" if TERRAFORM_AWS_AV_ZONE_US_EAST_2C
- default "us-west-1b" if TERRAFORM_AWS_AV_ZONE_US_WEST_1B
- default "us-west-2b" if TERRAFORM_AWS_AV_ZONE_US_WEST_2B
- default "us-west-2d" if TERRAFORM_AWS_AV_ZONE_US_WEST_2D
diff --git a/terraform/aws/kconfigs/instance-types/Kconfig.c7a b/terraform/aws/kconfigs/instance-types/Kconfig.c7a
deleted file mode 100644
index 147999d9..00000000
--- a/terraform/aws/kconfigs/instance-types/Kconfig.c7a
+++ /dev/null
@@ -1,28 +0,0 @@
-if TERRAFORM_AWS_INSTANCE_TYPE_C7A
-
-choice
- prompt "AWS instance size"
- default TERRAFORM_AWS_INSTANCE_SIZE_C7A_8XLARGE
- help
- Add storage by increasing the number of EBS volumes per
- instance.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_C7A_8XLARGE
- bool "c7a.8xlarge"
- help
- 32 core, 64 GiB RAM, EBS drives.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_C7A_METAL_48XL
- bool "c7a.metal-48xl"
- help
- 192 core, 384 GiB RAM, EBS drive.
-
-endchoice
-
-config TERRAFORM_AWS_INSTANCE_TYPE
- string
- output yaml
- default "c7a.8xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_C7A_8XLARGE
- default "c7a.metal-48xl" if TERRAFORM_AWS_INSTANCE_SIZE_C7A_METAL_48XL
-
-endif # TERRAFORM_AWS_INSTANCE_TYPE_C7A
diff --git a/terraform/aws/kconfigs/instance-types/Kconfig.i4i b/terraform/aws/kconfigs/instance-types/Kconfig.i4i
deleted file mode 100644
index 3f7b2218..00000000
--- a/terraform/aws/kconfigs/instance-types/Kconfig.i4i
+++ /dev/null
@@ -1,33 +0,0 @@
-if TERRAFORM_AWS_INSTANCE_TYPE_I4I
-
-choice
- prompt "AWS instance type"
- default TERRAFORM_AWS_INSTANCE_SIZE_I4I_4XLARGE
-
-config TERRAFORM_AWS_INSTANCE_SIZE_I4I_LARGE
- bool "i4i.large"
- help
- 16 GiB RAM, 2 vcpus, 1 x 468 AWS Nitro SSD, 10 Gbps Net, 10 Gbps EBS.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_I4I_4XLARGE
- bool "i4i.4xlarge"
- help
- 128 GiB RAM, 16 vcpus, 1 x 3,750 AWS Nitro SSD, 25 Gbps Net,
- 10 Gbps EBS.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_I4I_METAL
- bool "i4i.metal"
- help
- 1024 GiB RAM, 128 real cpus?, 8 x 3,750 AWS Nitro SSD,
- 75 Gbps Net, 40 Gbps EBS. The bees' knees I guess.
-
-endchoice
-
-config TERRAFORM_AWS_INSTANCE_TYPE
- string
- output yaml
- default "i4i.large" if TERRAFORM_AWS_INSTANCE_SIZE_I4I_LARGE
- default "i4i.4xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_I4I_4XLARGE
- default "i4i.metal" if TERRAFORM_AWS_INSTANCE_SIZE_I4I_METAL
-
-endif # TERRAFORM_AWS_INSTANCE_TYPE_I4I
diff --git a/terraform/aws/kconfigs/instance-types/Kconfig.im4gn b/terraform/aws/kconfigs/instance-types/Kconfig.im4gn
deleted file mode 100644
index e224bdca..00000000
--- a/terraform/aws/kconfigs/instance-types/Kconfig.im4gn
+++ /dev/null
@@ -1,25 +0,0 @@
-if TERRAFORM_AWS_INSTANCE_TYPE_IM4GN
-
-choice
- prompt "AWS instance size"
- default TERRAFORM_AWS_INSTANCE_SIZE_IM4GN_LARGE
-
-config TERRAFORM_AWS_INSTANCE_SIZE_IM4GN_LARGE
- bool "im4gn.large"
- help
- 8 GiB RAM, 2vCPUs, 1 x 937 GiB NVMe SSD, 25 Gbps Net, 9.5 Gbps EBS.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_IM4GN_4XLARGE
- bool "im4gn.4xlarge"
- help
- 64 GiB RAM, 16 vcpus, 1 x 7500 NVMe SSD, 25 Gbps Net, 9.5 Gbps EBS.
-
-endchoice
-
-config TERRAFORM_AWS_INSTANCE_TYPE
- string
- output yaml
- default "im4gn.large" if TERRAFORM_AWS_INSTANCE_SIZE_IM4GN_LARGE
- default "im4gn.4xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_IM4GN_4XLARGE
-
-endif # TERRAFORM_AWS_INSTANCE_TYPE_IM4GN
diff --git a/terraform/aws/kconfigs/instance-types/Kconfig.is4gen b/terraform/aws/kconfigs/instance-types/Kconfig.is4gen
deleted file mode 100644
index 4fdca17b..00000000
--- a/terraform/aws/kconfigs/instance-types/Kconfig.is4gen
+++ /dev/null
@@ -1,25 +0,0 @@
-if TERRAFORM_AWS_INSTANCE_TYPE_IS4GEN
-
-choice
- prompt "AWS instance size"
- default TERRAFORM_AWS_INSTANCE_SIZE_IS4GEN_MEDIUM
-
-config TERRAFORM_AWS_INSTANCE_SIZE_IS4GEN_MEDIUM
- bool "is4gen.medium"
- help
- 6 GiB RAM, 1vCPU, 1 x 937 GiB NVMe SSD, 25 Gbps Net, 9.6 Gbps EBS.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_IS4GEN_8XLARGE
- bool "is4gen.8xlarge"
- help
- 192 GiB RAM, 32 vCPUs, 4 x 7500 NVMe SSD, 50 Gbps Net, 19 Gbps EBS.
-
-endchoice
-
-config TERRAFORM_AWS_INSTANCE_TYPE
- string
- output yaml
- default "is4gen.medium" if TERRAFORM_AWS_INSTANCE_SIZE_IS4GEN_MEDIUM
- default "is4gen.8xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_IS4GEN_8XLARGE
-
-endif # TERRAFORM_AWS_INSTANCE_TYPE_IS4GEN
diff --git a/terraform/aws/kconfigs/instance-types/Kconfig.m5 b/terraform/aws/kconfigs/instance-types/Kconfig.m5
deleted file mode 100644
index 534a20e7..00000000
--- a/terraform/aws/kconfigs/instance-types/Kconfig.m5
+++ /dev/null
@@ -1,48 +0,0 @@
-if TERRAFORM_AWS_INSTANCE_TYPE_M5
-
-choice
- prompt "AWS instance type"
- default TERRAFORM_AWS_INSTANCE_SIZE_M5AD_4XLARGE
- help
- Add storage by increasing the number of EBS volumes per
- instance.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M5AD_LARGE
- bool "m5ad.large"
- depends on TARGET_ARCH_X86_64
- help
- 8 GiB RAM, 2 AMD vcpus, 1 10 GiB main drive, up to 10 Gbps
- network speed, and one 75 GiB NVMe drive.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M5AD_XLARGE
- bool "m5ad.xlarge"
- depends on TARGET_ARCH_X86_64
- help
- 16 GiB RAM, 4 AMD vcpus, 1 10 GiB main drive, up to 10 Gbps
- network speed, and one 150 GiB NVMe drive.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M5AD_2XLARGE
- bool "m5ad.2xlarge"
- depends on TARGET_ARCH_X86_64
- help
- 32 GiB RAM, 8 AMD vcpus, 1 10 GiB main drive, up to 10 Gbps
- network speed, and one 300 GiB NVMe drive.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M5AD_4XLARGE
- bool "m5ad.4xlarge"
- depends on TARGET_ARCH_X86_64
- help
- 64 GiB RAM, 16 AMD vcpus, 1 10 GiB main drive, up to 10 Gbps
- and two 300 GiB NVMe drives.
-
-endchoice
-
-config TERRAFORM_AWS_INSTANCE_TYPE
- string
- output yaml
- default "m5ad.large" if TERRAFORM_AWS_INSTANCE_SIZE_M5AD_LARGE
- default "m5ad.xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_M5AD_XLARGE
- default "m5ad.2xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_M5AD_2XLARGE
- default "m5ad.4xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_M5AD_4XLARGE
-
-endif # TERRAFORM_AWS_INSTANCE_TYPE_M5
diff --git a/terraform/aws/kconfigs/instance-types/Kconfig.m7a b/terraform/aws/kconfigs/instance-types/Kconfig.m7a
deleted file mode 100644
index fd7bd6f4..00000000
--- a/terraform/aws/kconfigs/instance-types/Kconfig.m7a
+++ /dev/null
@@ -1,57 +0,0 @@
-if TERRAFORM_AWS_INSTANCE_TYPE_M7A
-
-choice
- prompt "AWS instance type"
- default TERRAFORM_AWS_INSTANCE_SIZE_M7A_XLARGE
- help
- Add storage by increasing the number of EBS volumes per
- instance.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M7A_MEDIUM
- bool "m7a.medium"
- depends on TARGET_ARCH_X86_64
- help
- 4 GiB RAM, 1 AMD Ryzen vcpu, 1 10 GiB main drive, and up to
- 12.5 Gbs network speed.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M7A_LARGE
- bool "m7a.large"
- depends on TARGET_ARCH_X86_64
- help
- 8 GiB RAM, 2 AMD Ryzen vcpus, 1 10 GiB main drive, and up to
- 12.5 Gbs network speed.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M7A_XLARGE
- bool "m7a.xlarge"
- depends on TARGET_ARCH_X86_64
- help
- 16 GiB RAM, 4 AMD Ryzen vcpus, 1 10 GiB main drive, and up to
- 12.5 Gbs network speed.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M7A_2XLARGE
- bool "m7a.2xlarge"
- depends on TARGET_ARCH_X86_64
- help
- 32 GiB RAM, 8 AMD Ryzen vcpus, 1 10 GiB main drive, and up to
- 12.5 Gbs network speed.
-
-config TERRAFORM_AWS_INSTANCE_SIZE_M7A_4XLARGE
- bool "m7a.4xlarge"
- depends on TARGET_ARCH_X86_64
- help
- 64 GiB RAM, 16 AMD Ryzen vcpus, 1 10 GiB main drive, and up to
- 12.5 Gbs network speed.
-
-endchoice
-
-config TERRAFORM_AWS_INSTANCE_TYPE
- string
- output yaml
- default "m7a.medium" if TERRAFORM_AWS_INSTANCE_SIZE_M7A_MEDIUM
- default "m7a.large" if TERRAFORM_AWS_INSTANCE_SIZE_M7A_LARGE
- default "m7a.xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_M7A_XLARGE
- default "m7a.xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_M7A_XLARGE
- default "m7a.2xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_M7A_2XLARGE
- default "m7a.4xlarge" if TERRAFORM_AWS_INSTANCE_SIZE_M7A_4XLARGE
-
-endif # TERRAFORM_AWS_INSTANCE_TYPE_M7A
diff --git a/terraform/aws/scripts/generate_aws_kconfig.py b/terraform/aws/scripts/generate_aws_kconfig.py
index c6a60a83..a8f25b4e 100755
--- a/terraform/aws/scripts/generate_aws_kconfig.py
+++ b/terraform/aws/scripts/generate_aws_kconfig.py
@@ -221,8 +221,19 @@ def generate_family_kconfig(family: str, instances: List[Dict]) -> str:
"""Generate Kconfig content for a single family."""
content = [f"# AWS {family.upper()} instance sizes (dynamically generated)", ""]
+ # Filter instances to only include the exact family (not related families)
+ # e.g., for "r8g" family, exclude "r8gd" and "r8gn" instances
+ filtered_instances = [
+ inst for inst in instances
+ if inst['instance_type'].split('.')[0] == family
+ ]
+
+ if not filtered_instances:
+ # If no exact matches, use all instances (backward compatibility)
+ filtered_instances = instances
+
# Sort instances by a logical order
- sorted_instances = sorted(instances, key=lambda x: (
+ sorted_instances = sorted(filtered_instances, key=lambda x: (
'metal' not in x['instance_type'], # metal instances last
x.get('vcpus', 0), # then by vCPUs
x.get('memory_gb', 0) # then by memory
@@ -238,14 +249,15 @@ def generate_family_kconfig(family: str, instances: List[Dict]) -> str:
# Generate choice block
content.append("choice")
content.append(f'\tprompt "Instance size for {family.upper()} family"')
- content.append(f'\tdefault TERRAFORM_AWS_INSTANCE_{default.replace(".", "_").upper()}')
+ content.append(f'\tdefault TERRAFORM_AWS_INSTANCE_{default.replace(".", "_").replace("-", "_").upper()}')
content.append("\thelp")
content.append(f"\t Select the specific instance size within the {family.upper()} family.")
content.append("")
# Generate config entries
for inst in sorted_instances:
- type_upper = inst['instance_type'].replace('.', '_').upper()
+ # Replace both dots and dashes with underscores for valid Kconfig symbols
+ type_upper = inst['instance_type'].replace('.', '_').replace('-', '_').upper()
content.append(f"config TERRAFORM_AWS_INSTANCE_{type_upper}")
content.append(f'\tbool "{inst["instance_type"]}"')
content.append("\thelp")
@@ -262,7 +274,7 @@ def generate_family_kconfig(family: str, instances: List[Dict]) -> str:
content.append("\tstring")
for inst in sorted_instances:
- type_upper = inst['instance_type'].replace('.', '_').upper()
+ type_upper = inst['instance_type'].replace('.', '_').replace('-', '_').upper()
content.append(f'\tdefault "{inst["instance_type"]}" if TERRAFORM_AWS_INSTANCE_{type_upper}')
content.append(f'\tdefault "{default}"')
--
2.51.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 7/8] aws: add GPU instance defconfigs for AI/ML workloads
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
` (5 preceding siblings ...)
2025-09-17 0:34 ` [PATCH v4 6/8] aws: replace static Kconfig files with dynamically generated ones Luis Chamberlain
@ 2025-09-17 0:34 ` Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 8/8] docs: add documentation for dynamic cloud configuration Luis Chamberlain
7 siblings, 0 replies; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Luis Chamberlain
Add two AWS GPU instance defconfigs to support GPU-accelerated
workloads:
1. aws-gpu-p5-48xlarge: High-end configuration with 8x NVIDIA H100 80GB
GPUs for large-scale AI/ML training and inference. Includes 500GB
storage for models and datasets.
2. aws-gpu-g5-xlarge: Cost-effective single NVIDIA A10G 24GB GPU for
development and smaller workloads. Includes 200GB storage.
Both configurations:
- Use Debian 12 for stability
- Enable kernel development workflows (KOTD)
- Configure 9P filesystem for host-guest development
- Set up in us-east-1 for best GPU availability
- Use GP3 EBS volumes for better performance
These defconfigs leverage the new dynamic Kconfig generation to
automatically include all available P5 and G5 instance sizes.
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
defconfigs/aws-gpu-g5-xlarge | 36 ++++++++++++++++++++++++++++++++++
defconfigs/aws-gpu-p5-48xlarge | 36 ++++++++++++++++++++++++++++++++++
2 files changed, 72 insertions(+)
create mode 100644 defconfigs/aws-gpu-g5-xlarge
create mode 100644 defconfigs/aws-gpu-p5-48xlarge
diff --git a/defconfigs/aws-gpu-g5-xlarge b/defconfigs/aws-gpu-g5-xlarge
new file mode 100644
index 00000000..a2f0a21f
--- /dev/null
+++ b/defconfigs/aws-gpu-g5-xlarge
@@ -0,0 +1,36 @@
+# AWS G5.xlarge GPU instance configuration
+# 1x NVIDIA A10G 24GB GPU for cost-effective GPU development
+
+CONFIG_KDEVOPS_FIRST_RUN=y
+CONFIG_TERRAFORM=y
+CONFIG_TERRAFORM_AWS=y
+CONFIG_TERRAFORM_AWS_DATA_ENABLE=y
+
+# Instance configuration
+CONFIG_TERRAFORM_AWS_INSTANCE_FAMILY_G5=y
+CONFIG_TERRAFORM_AWS_INSTANCE_G5_XLARGE=y
+
+# Moderate root volume
+CONFIG_TERRAFORM_AWS_EBS_SIZE=200
+CONFIG_TERRAFORM_AWS_EBS_VOLUME_TYPE_GP3=y
+
+# US East 1 for availability
+CONFIG_TERRAFORM_AWS_REGION_US_EAST_1=y
+
+# Debian 12 for stability
+CONFIG_VAGRANT_DEBIAN=y
+CONFIG_VAGRANT_DEBIAN12=y
+
+# Use kdevops user
+CONFIG_KDEVOPS_SSH_CONFIG_USER_KDEVOPS=y
+
+# Workflows
+CONFIG_KDEVOPS_WORKFLOW_ENABLE_KOTD=y
+CONFIG_KDEVOPS_WORKFLOW_KOTD_ENABLE_LATEST=y
+
+# Enable git for kernel development
+CONFIG_BOOTLINUX_9P=y
+CONFIG_BOOTLINUX_TREE_LINUS=y
+
+# Networking
+CONFIG_KDEVOPS_ENABLE_TERRAFORM_SSH_COMPLETE_SETUP=y
\ No newline at end of file
diff --git a/defconfigs/aws-gpu-p5-48xlarge b/defconfigs/aws-gpu-p5-48xlarge
new file mode 100644
index 00000000..741b98e3
--- /dev/null
+++ b/defconfigs/aws-gpu-p5-48xlarge
@@ -0,0 +1,36 @@
+# AWS P5.48xlarge GPU instance configuration
+# 8x NVIDIA H100 80GB GPUs for AI/ML workloads
+
+CONFIG_KDEVOPS_FIRST_RUN=y
+CONFIG_TERRAFORM=y
+CONFIG_TERRAFORM_AWS=y
+CONFIG_TERRAFORM_AWS_DATA_ENABLE=y
+
+# Instance configuration
+CONFIG_TERRAFORM_AWS_INSTANCE_FAMILY_P5=y
+CONFIG_TERRAFORM_AWS_INSTANCE_P5_48XLARGE=y
+
+# Large root volume for models and datasets
+CONFIG_TERRAFORM_AWS_EBS_SIZE=500
+CONFIG_TERRAFORM_AWS_EBS_VOLUME_TYPE_GP3=y
+
+# US East 1 typically has best GPU availability
+CONFIG_TERRAFORM_AWS_REGION_US_EAST_1=y
+
+# Debian 12 for stability
+CONFIG_VAGRANT_DEBIAN=y
+CONFIG_VAGRANT_DEBIAN12=y
+
+# Use kdevops user
+CONFIG_KDEVOPS_SSH_CONFIG_USER_KDEVOPS=y
+
+# Workflows
+CONFIG_KDEVOPS_WORKFLOW_ENABLE_KOTD=y
+CONFIG_KDEVOPS_WORKFLOW_KOTD_ENABLE_LATEST=y
+
+# Enable git for kernel development
+CONFIG_BOOTLINUX_9P=y
+CONFIG_BOOTLINUX_TREE_LINUS=y
+
+# Networking
+CONFIG_KDEVOPS_ENABLE_TERRAFORM_SSH_COMPLETE_SETUP=y
\ No newline at end of file
--
2.51.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 8/8] docs: add documentation for dynamic cloud configuration
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
` (6 preceding siblings ...)
2025-09-17 0:34 ` [PATCH v4 7/8] aws: add GPU instance defconfigs for AI/ML workloads Luis Chamberlain
@ 2025-09-17 0:34 ` Luis Chamberlain
7 siblings, 0 replies; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 0:34 UTC (permalink / raw)
To: Chuck Lever, Daniel Gomez, kdevops; +Cc: Luis Chamberlain
Add detailed documentation covering the dynamic cloud configuration
system, including:
- Overview of dynamic configuration benefits
- AWS and Lambda Labs provider details
- Quick start commands for all cloud operations
- Technical implementation details
- Performance optimizations (21s vs 6 minutes)
- Cache management (24-hour TTL)
- Cost tracking with make cloud-bill
- GPU instance configuration examples
- Troubleshooting guide
- Development and debugging instructions
The documentation explains how the system works, from Chuck's AWS
scripts through the caching layer to Kconfig generation, providing
users with a complete understanding of the dynamic configuration
workflow.
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
docs/cloud-dynamic-config.md | 272 +++++++++++++++++++++++++++++++++++
1 file changed, 272 insertions(+)
create mode 100644 docs/cloud-dynamic-config.md
diff --git a/docs/cloud-dynamic-config.md b/docs/cloud-dynamic-config.md
new file mode 100644
index 00000000..c882d799
--- /dev/null
+++ b/docs/cloud-dynamic-config.md
@@ -0,0 +1,272 @@
+# Dynamic Cloud Configuration
+
+kdevops supports dynamic configuration generation for cloud providers, automatically
+querying cloud APIs to provide up-to-date instance types, regions, and pricing
+information.
+
+## Overview
+
+Dynamic cloud configuration ensures your kdevops setup always has access to the
+latest cloud provider offerings without manual updates. This system:
+
+- Queries cloud provider APIs for current instance types and regions
+- Generates Kconfig files with accurate specifications
+- Caches data for performance (24-hour TTL)
+- Supports parallel processing for fast generation
+- Integrates with standard kdevops workflows
+
+## Supported Cloud Providers
+
+### AWS (Amazon Web Services)
+
+AWS dynamic configuration provides:
+- 146+ instance families (vs 6 in static configs)
+- 900+ instance types with current specs
+- 30+ regions with availability zones
+- GPU instance support (P5, G5, etc.)
+- Cost tracking integration
+
+### Lambda Labs
+
+Lambda Labs dynamic configuration provides:
+- GPU-focused instance types
+- Real-time availability checking
+- Automatic region discovery
+- Pricing information
+
+## Quick Start
+
+### Generate Cloud Configurations
+
+```bash
+# Generate all cloud provider configurations
+make cloud-config
+
+# Generate specific provider configurations
+make cloud-config-aws
+make cloud-config-lambdalabs
+```
+
+### Update Cloud Data
+
+To refresh cached data and get the latest information:
+
+```bash
+# Update all providers
+make cloud-update
+
+# Update specific provider
+make cloud-update-aws
+```
+
+### Check Cloud Costs
+
+Monitor your cloud spending:
+
+```bash
+# Show current month's costs
+make cloud-bill
+
+# AWS-specific billing
+make cloud-bill-aws
+```
+
+## AWS Dynamic Configuration
+
+### How It Works
+
+1. **Data Collection**: Uses Chuck's AWS scripts to query EC2 APIs
+ - `terraform/aws/scripts/ec2_instance_info.py`: Instance specifications
+ - `terraform/aws/scripts/aws_regions_info.py`: Region information
+ - `terraform/aws/scripts/aws_ami_info.py`: AMI details
+
+2. **Caching**: JSON data cached in `~/.cache/kdevops/aws/`
+ - 24-hour TTL for cached data
+ - Automatic refresh on cache expiry
+ - Manual refresh with `make cloud-update-aws`
+
+3. **Generation**: Parallel processing creates Kconfig files
+ - Main configs in `terraform/aws/kconfigs/*.generated`
+ - Instance types in `terraform/aws/kconfigs/instance-types/*.generated`
+ - ~21 seconds for fresh generation (vs 6 minutes unoptimized)
+ - ~0.04 seconds when using cache
+
+### Configuration Structure
+
+```
+terraform/aws/kconfigs/
+├── Kconfig.compute.generated # Instance family selection
+├── Kconfig.location.generated # AWS regions
+├── Kconfig.gpu-amis.generated # GPU AMI configurations
+└── instance-types/
+ ├── Kconfig.m5.generated # M5 family sizes
+ ├── Kconfig.p5.generated # P5 GPU instances
+ └── ... (146+ families)
+```
+
+### Using AWS GPU Instances
+
+kdevops includes pre-configured defconfigs for GPU workloads:
+
+```bash
+# High-end: 8x NVIDIA H100 80GB GPUs
+make defconfig-aws-gpu-p5-48xlarge
+
+# Cost-effective: 1x NVIDIA A10G 24GB GPU
+make defconfig-aws-gpu-g5-xlarge
+
+# Then provision
+make bringup
+```
+
+### Cost Management
+
+Track AWS costs with integrated billing support:
+
+```bash
+# Check current month's spending
+make cloud-bill-aws
+```
+
+Output shows:
+- Total monthly cost to date
+- Breakdown by AWS service
+- Daily average spending
+- Projected monthly cost (when mid-month)
+
+## Lambda Labs Dynamic Configuration
+
+Lambda Labs configuration focuses on GPU instances for ML/AI workloads:
+
+```bash
+# Generate Lambda Labs configs
+make cloud-config-lambdalabs
+
+# Use a Lambda Labs defconfig
+make defconfig-lambdalabs-gpu-8x-h100
+```
+
+## Technical Details
+
+### Performance Optimizations
+
+The dynamic configuration system uses several optimizations:
+
+1. **Parallel API Queries**: 10 concurrent workers fetch instance data
+2. **Parallel File Writing**: 20 concurrent workers write Kconfig files
+3. **JSON Caching**: 24-hour cache reduces API calls
+4. **Batch Processing**: Fetches all data in single API call where possible
+
+### Cache Management
+
+Cache location: `~/.cache/kdevops/<provider>/`
+
+Cache files:
+- `aws_families.json`: Instance family list
+- `aws_family_<name>.json`: Per-family instance data
+- `aws_regions.json`: Region information
+- `aws_all_instances.json`: Complete dataset
+
+Clear cache manually:
+```bash
+rm -rf ~/.cache/kdevops/aws/
+make cloud-update-aws
+```
+
+### Adding New Cloud Providers
+
+To add support for a new cloud provider:
+
+1. Create provider-specific scripts in `terraform/<provider>/scripts/`
+2. Add Kconfig directory structure in `terraform/<provider>/kconfigs/`
+3. Update `scripts/dynamic-cloud-kconfig.Makefile` with new targets
+4. Implement generation in `scripts/generate_cloud_configs.py`
+
+## Troubleshooting
+
+### AWS Credentials Not Configured
+
+If you see "AWS: Credentials not configured":
+
+```bash
+# Configure AWS CLI
+aws configure
+
+# Or set environment variables
+export AWS_ACCESS_KEY_ID=your_key
+export AWS_SECRET_ACCESS_KEY=your_secret
+export AWS_DEFAULT_REGION=us-east-1
+```
+
+### Kconfig Errors
+
+If menuconfig shows errors after generation:
+
+1. Clear cache and regenerate:
+ ```bash
+ make cloud-update-aws
+ ```
+
+2. Check for syntax issues:
+ ```bash
+ grep -n "error:" terraform/aws/kconfigs/*.generated
+ ```
+
+### Slow Generation
+
+If generation takes longer than 30 seconds:
+
+1. Check network connectivity to AWS
+2. Verify credentials are valid
+3. Try different AWS region:
+ ```bash
+ export AWS_DEFAULT_REGION=eu-west-1
+ make cloud-update-aws
+ ```
+
+## Development
+
+### Running Scripts Directly
+
+```bash
+# Generate AWS configs with Chuck's scripts
+python3 terraform/aws/scripts/generate_aws_kconfig.py
+
+# Clear cache and regenerate
+python3 terraform/aws/scripts/generate_aws_kconfig.py clear-cache
+
+# Query specific instance family
+python3 terraform/aws/scripts/ec2_instance_info.py m5 --format json
+
+# List all families
+python3 terraform/aws/scripts/ec2_instance_info.py --families --format json
+```
+
+### Debugging
+
+Enable debug output:
+```bash
+# Debug AWS script
+python3 terraform/aws/scripts/ec2_instance_info.py --debug m5
+
+# Verbose Makefile execution
+make V=1 cloud-config-aws
+```
+
+## Best Practices
+
+1. **Regular Updates**: Run `make cloud-update` weekly for latest offerings
+2. **Cost Monitoring**: Check `make cloud-bill` before major deployments
+3. **Cache Management**: Let cache expire naturally unless testing changes
+4. **Region Selection**: Choose regions close to you for lower latency
+5. **Instance Right-Sizing**: Use dynamic configs to find optimal instance sizes
+
+## Future Enhancements
+
+Planned improvements:
+- Azure dynamic configuration support
+- GCE (Google Cloud) dynamic configuration
+- Real-time pricing integration
+- Spot instance availability checking
+- Instance recommendation based on workload
+- Cost optimization suggestions
\ No newline at end of file
--
2.51.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/8] aws: prevent SSH key conflicts across multiple kdevops directories
2025-09-17 0:34 ` [PATCH v4 1/8] aws: prevent SSH key conflicts across multiple kdevops directories Luis Chamberlain
@ 2025-09-17 3:36 ` Chuck Lever
0 siblings, 0 replies; 13+ messages in thread
From: Chuck Lever @ 2025-09-17 3:36 UTC (permalink / raw)
To: Luis Chamberlain, Daniel Gomez, kdevops
On 9/16/25 5:34 PM, Luis Chamberlain wrote:
> Benefits:
> - Prevents accidental key overwrites in AWS backend
> - No more lockouts from existing instances when using multiple directories
> - Backward compatible - users can still use shared keys if needed
> - Consistent with Lambda Labs implementation pattern
> - Each directory's instances are isolated with their own SSH keys
>
> Example:
> Directory: /home/user/projects/kdevops
> AWS Key Name: kdevops-aws-projects-kdevops-a8999659
> Local Keys: ~/.ssh/kdevops-aws-projects-kdevops-a8999659[.pub]
>
> This ensures that running kdevops from different directories will never
> conflict, as each gets its own unique key name in both AWS and locally.
>
> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
I did something like this by hand, using the pathname configured
in the existing terraform ssh config menu. It's definitely a
problem.
But seems like this might be an issue for all the cloud providers, so
maybe this idea could be applied in the generic terraform ssh config
menu instead of making it specific to AWS and Lambda? But I might be
missing something.
--
Chuck Lever
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 4/8] aws: integrate dynamic Kconfig generation with make targets
2025-09-17 0:34 ` [PATCH v4 4/8] aws: integrate dynamic Kconfig generation with make targets Luis Chamberlain
@ 2025-09-17 3:40 ` Chuck Lever
2025-09-17 7:05 ` Luis Chamberlain
0 siblings, 1 reply; 13+ messages in thread
From: Chuck Lever @ 2025-09-17 3:40 UTC (permalink / raw)
To: Luis Chamberlain, Daniel Gomez, kdevops
On 9/16/25 5:34 PM, Luis Chamberlain wrote:
> Add Makefile integration for AWS dynamic Kconfig generation using
> Chuck's scripts with the optimized wrapper. This provides:
>
> - make cloud-config-aws: Generate AWS Kconfig files
> - make cloud-update-aws: Clear cache and regenerate fresh data
> - make cloud-config: Now includes AWS generation
> - make cloud-update: Refreshes all cloud provider data
> - make clean-cloud-config-aws: Clean generated AWS files
>
> The integration properly manages AWS Kconfig files alongside Lambda Labs
> configurations and ensures empty files exist before Kconfig runs to
> prevent sourcing errors.
>
> Also update generate_cloud_configs.py to call the AWS generator and
> provide summary statistics about available AWS resources.
>
> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Yes, this is exactly what I was thinking -- add in the menu generator
scripts first and then construct make targets to drive them.
But I've been working on the scripts themselves and have new smaller
versions of the locations and instance menu generators. Let me flesh
those out a little more (and finish the AMI menu generator) then I will
post here and we can integrate.
--
Chuck Lever
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 3/8] aws: add optimized Kconfig generator using Chuck's scripts
2025-09-17 0:34 ` [PATCH v4 3/8] aws: add optimized Kconfig generator using Chuck's scripts Luis Chamberlain
@ 2025-09-17 3:58 ` Chuck Lever
0 siblings, 0 replies; 13+ messages in thread
From: Chuck Lever @ 2025-09-17 3:58 UTC (permalink / raw)
To: Luis Chamberlain, Daniel Gomez, kdevops
On 9/16/25 5:34 PM, Luis Chamberlain wrote:
> Create a wrapper script that orchestrates Chuck's existing AWS scripts
> (ec2_instance_info.py, aws_regions_info.py, aws_ami_info.py) to generate
> Kconfig files with JSON caching and parallel processing.
>
> This approach leverages Chuck's already working scripts while adding:
> - JSON caching with 24-hour TTL in ~/.cache/kdevops/aws/
> - Parallel fetching of instance data (10 concurrent workers)
> - Parallel file writing (20 concurrent workers)
> - Proper data structure handling for families list
>
> Performance improvements:
> - First run: ~21 seconds to fetch all data from AWS
> - Cached runs: ~0.04 seconds (525x faster)
> - Generates 75 Kconfig files for 72 instance families
>
> The script properly uses Chuck's existing AWS API implementations
> rather than reimplementing them, maintaining code reuse and consistency.
>
> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
I've got a slightly different approach in mind, but same goal: make
these scripts fast. Great minds think alike!
After working with my scripts a bit, it seems like we really don't need
the JSON intermediate step. It made sense when getting output from the
CLI, but the response from the boto3 client is truly enough to assemble
the Kconfig menus directly.
Building the instance types is what seems to take the longest because
the script repeatedly does a describe_instance_types and the response
to that is HUGE. But we can do that once at the top of the script, and
simply use the same response as often as needed.
The use case seems to be "build the Kconfig menus" and then use them
for a while. So we have the 21 second first run, but I'm not yet seeing
a benefit to caching -- why rebuild the menus again if your input data
is going to remain exactly the same for 24 hours? Let me know if I've
misunderstood.
And, I now agree it makes sense to get rid of the menu subdirectories.
Sourcing the little files doesn't seem to add any value. My new scripts
construct one large locations menu and one large compute menu.
--
Chuck Lever
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 4/8] aws: integrate dynamic Kconfig generation with make targets
2025-09-17 3:40 ` Chuck Lever
@ 2025-09-17 7:05 ` Luis Chamberlain
0 siblings, 0 replies; 13+ messages in thread
From: Luis Chamberlain @ 2025-09-17 7:05 UTC (permalink / raw)
To: Chuck Lever; +Cc: Daniel Gomez, kdevops
On Tue, Sep 16, 2025 at 08:40:00PM -0700, Chuck Lever wrote:
> On 9/16/25 5:34 PM, Luis Chamberlain wrote:
> > Add Makefile integration for AWS dynamic Kconfig generation using
> > Chuck's scripts with the optimized wrapper. This provides:
> >
> > - make cloud-config-aws: Generate AWS Kconfig files
> > - make cloud-update-aws: Clear cache and regenerate fresh data
> > - make cloud-config: Now includes AWS generation
> > - make cloud-update: Refreshes all cloud provider data
> > - make clean-cloud-config-aws: Clean generated AWS files
> >
> > The integration properly manages AWS Kconfig files alongside Lambda Labs
> > configurations and ensures empty files exist before Kconfig runs to
> > prevent sourcing errors.
> >
> > Also update generate_cloud_configs.py to call the AWS generator and
> > provide summary statistics about available AWS resources.
> >
> > Generated-by: Claude AI
> > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
>
> Yes, this is exactly what I was thinking -- add in the menu generator
> scripts first and then construct make targets to drive them.
>
> But I've been working on the scripts themselves and have new smaller
> versions of the locations and instance menu generators. Let me flesh
> those out a little more (and finish the AMI menu generator) then I will
> post here and we can integrate.
No need, just merge your changes :) Full steam ahead!
Luis
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-09-17 7:05 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-17 0:34 [PATCH v4 0/8] aws: add dynamic kconfig support Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 1/8] aws: prevent SSH key conflicts across multiple kdevops directories Luis Chamberlain
2025-09-17 3:36 ` Chuck Lever
2025-09-17 0:34 ` [PATCH v4 2/8] terraform/aws: Add scripts to gather provider resource information Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 3/8] aws: add optimized Kconfig generator using Chuck's scripts Luis Chamberlain
2025-09-17 3:58 ` Chuck Lever
2025-09-17 0:34 ` [PATCH v4 4/8] aws: integrate dynamic Kconfig generation with make targets Luis Chamberlain
2025-09-17 3:40 ` Chuck Lever
2025-09-17 7:05 ` Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 5/8] aws: add cloud billing support with make cloud-bill Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 6/8] aws: replace static Kconfig files with dynamically generated ones Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 7/8] aws: add GPU instance defconfigs for AI/ML workloads Luis Chamberlain
2025-09-17 0:34 ` [PATCH v4 8/8] docs: add documentation for dynamic cloud configuration Luis Chamberlain
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox