Re: [PATCH v3 1/3] aws: add dynamic cloud configuration support

[Chuck Lever <cel@kernel.org>]

Hey Luis -

On 9/8/25 8:56 PM, Luis Chamberlain wrote:
> Move away from static AWS kconfig files to dynamic AWS kconfig support.
> We leverage AWS APIs to dynamically to generate Kconfig options for instance
> types, regions, and AMIs. This ensures users always have access to current
> cloud resources without having kdevops admins manually hand craft updates.
> This also makes it less error prone, and lets us scale.
> 
> To generate new AWS dynamic kconfig files, one can use:
> 
> make cloud-config # takes about ~6 minutes

Yeah, the cloud APIs are quite slow. This is why terraform is highly
parallelized!


> Regular users are not expected to have to call this. Only kdevops admins are.
> This will generaed Kconfig *.generated files, which we git-ignore. If we
> want to keep these, and propagate them to kdevops users through a commit,
> kdevops admins can then run:
> 
> make cloud-update
> 
> This will generate kconfig *.static file which we do allow to commit.
> So users are expected to reap benefit of these kconfig *.static files.
> And kdevops admins are the ones who will routinely (once a month or so)
> will run both:
> 
> make cloud-config
> make cloud-update
> 
> The implementation generates configuration files on-demand by querying
> AWS EC2 APIs. If the AWS CLI is unavailable or not configured, the system
> provides hardcoded defaults through fallback functions:
> 
>   * generate_default_instance_families_kconfig()
>   * generate_default_regions_kconfig()
>   * generate_default_gpu_amis_kconfig()
> 
> hese return common instance types like m5, t3 and major regions, ensuring
> kdevops remains functional without AWS CLI access.
> 
> We completely remove all the old AWS static files in favor for the new world
> order of dynamic kconfig for cloud. This commit only removes all the
> static stuff, and adds all the dynamic kconfig scripts for AWS. In the
> subsequent commits we will add static files from a system that does have a
> functional aws cli.
> 
> New files:
> - scripts/aws-cli: CLI wrapper for AWS operations
> - scripts/aws_api.py: AWS API functions with fallback defaults
> - docs/cloud-configuration.md: Setup and usage documentation
> 
> Changes:
> - terraform/aws/kconfigs/: Dynamic Kconfig generation replaces static files
> - scripts/generate_cloud_configs.py: Parallelize cloud kconfig
>   generation support to reduce the time it takes
> - scripts/dynamic-cloud-kconfig.Makefile: Build system integration
> 
> Prerequisites for full functionality:
> - AWS CLI installed (pip install awscli)
> - AWS credentials configured (~/.aws/credentials or environment variables)
> - IAM permissions: ec2:Describe*, pricing:GetProducts (optional)
> 
> Without these prerequisites, kdevops uses built-in fallback functions.
> 
> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
>  .gitignore                                    |    3 +
>  docs/cloud-configuration.md                   |  533 ++++++++
>  docs/kdevops-terraform.md                     |    9 +
>  scripts/aws-cli                               |  436 +++++++
>  scripts/aws_api.py                            | 1161 +++++++++++++++++
>  scripts/dynamic-cloud-kconfig.Makefile        |   88 +-
>  scripts/generate_cloud_configs.py             |  193 ++-
>  terraform/aws/kconfigs/Kconfig.compute        |   93 +-
>  terraform/aws/kconfigs/Kconfig.location       |  690 +---------
>  .../aws/kconfigs/instance-types/Kconfig.c7a   |   28 -
>  .../aws/kconfigs/instance-types/Kconfig.i4i   |   33 -
>  .../aws/kconfigs/instance-types/Kconfig.im4gn |   25 -
>  .../kconfigs/instance-types/Kconfig.is4gen    |   25 -
>  .../aws/kconfigs/instance-types/Kconfig.m5    |   48 -
>  .../aws/kconfigs/instance-types/Kconfig.m7a   |   57 -
>  15 files changed, 2430 insertions(+), 992 deletions(-)
>  create mode 100644 docs/cloud-configuration.md
>  create mode 100755 scripts/aws-cli
>  create mode 100755 scripts/aws_api.py
>  delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.c7a
>  delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.i4i
>  delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.im4gn
>  delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.is4gen
>  delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.m5
>  delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.m7a

My vision was that the scripts would update the files under
instance-types/ , not remove them. This is one reason why I broke
these out into subdirectories -- to make it straightforward for
the files to be machine-generated, since they are all small and
single-purpose.

Should be one file per instance family.

Kconfig.compute (and Kconfig.location) should be updated in the same
way as the instance-types/ directory, and not deleted by the patch,
unless there's something that makes the current directory structure
totally unworkable. (which is totally possible).


When I "make menuconfig" right after pulling these patches, the "Region"
menu is gone. All that is left is a string entry field for an
availability zone. I can debug further if you would like more
details.


> diff --git a/docs/kdevops-terraform.md b/docs/kdevops-terraform.md
> index 3291444d..983dbad5 100644
> --- a/docs/kdevops-terraform.md
> +++ b/docs/kdevops-terraform.md
> @@ -33,6 +33,15 @@ That should let you start configuring your cloud provider options. You can
>  use the same main menu to configure specific workflows supported by kdevops,
>  by defaults no workflows are enabled, and so all you get is the bringup.
>  
> +### Dynamic Cloud Configuration
> +
> +kdevops supports dynamic cloud provider configuration that queries live cloud
> +APIs for up-to-date instance types, regions, and AMI options. This feature
> +ensures you always have access to the latest cloud resources without manual
> +updates. For detailed information about dynamic cloud configuration, including
> +setup, prerequisites, and advanced usage, see the
> +[Cloud Configuration Management documentation](cloud-configuration.md).

This paragraph might mention that dynamic configuration is an advanced
feature (possibly intended for developers rather than users). A new
kdevops user, for example, does not have to do anything with this
feature to start using kdevops' cloud facilities.



> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: MIT
> +"""
> +AWS CLI tool for kdevops
> +
> +A structured CLI tool that wraps AWS CLI commands and provides access to
> +AWS cloud provider functionality for dynamic configuration generation
> +and resource management.
> +"""
> +
> +import argparse
> +import json
> +import sys
> +import os
> +from typing import Dict, List, Any, Optional, Tuple
> +from pathlib import Path

For scripts/aws_cli and scripts/aws_cli.py, an alternative would be to
use boto3 directly, which is the AWS SDK for Python. Just a random idea.


The script [ deleted here ] is much too large to get any kind of handle
on. Really, Claude needs to break these changes up, just as we expect
from any human contributor, so that they can be reviewed and debugged
in a sane fashion.

One way to go about this would be to write just enough in this pass to
handle updating the Kconfig.compute menu, and leave locations alone for
now. It's the new instance families and AMIs that you are most
interested in, yes?

How about one script whose input is an instance family name, and whose
output is one of the submenu files?

Another script can create AMI submenus.

A third can call those two repeatedly to build Kconfig.compute.

Breaking it down like that would make it easier to experiment and
develop with.

I'm wondering if these scripts might be better located under
terraform/aws instead of the top-level scripts/ directory.


> diff --git a/terraform/aws/kconfigs/Kconfig.location b/terraform/aws/kconfigs/Kconfig.location
> index e5dacb9d..fbdf7eb1 100644
> --- a/terraform/aws/kconfigs/Kconfig.location
> +++ b/terraform/aws/kconfigs/Kconfig.location
> @@ -1,679 +1,21 @@
> -choice
> -	prompt "AWS region"
> -	default TERRAFORM_AWS_REGION_US_WEST_2
[ ... ]
> -	default "us-east-2c" if TERRAFORM_AWS_AV_ZONE_US_EAST_2C
> -	default "us-west-1b" if TERRAFORM_AWS_AV_ZONE_US_WEST_1B
> -	default "us-west-2b" if TERRAFORM_AWS_AV_ZONE_US_WEST_2B
> -	default "us-west-2d" if TERRAFORM_AWS_AV_ZONE_US_WEST_2D
> +	  In general, AWS region names are lower-case letters only.
> +	  Sometimes, they include a number suffix (see us-east-2a, for
> +	  example).

This comment is incorrect. "us-east-2a" is an availability zone,
not a region. That makes me a little suspicious -- zones and
regions are definitely not the same thing.


Btw, I'm not using Claude Code -- I'm using a few others, including
Claude Sonnet. I'm not quite prepared to take your prompt and redrive
the code generation.

Have you given any thought to preserving the prompt(s) you are using
(kind of like a commit history) ? Maybe I'm thinking about this
process too deterministically.


-- 
Chuck Lever