From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E8181F03C9 for ; Wed, 10 Sep 2025 20:53:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757537596; cv=none; b=CRtTXZtNV+Kdh2J14PZU22xmks1JzmjZrEC+1CWx5Sy/9qEgPVSsDKsrfStgB2/lpgEhuLQpadORAT7hduME45WIzavR79hRJH22oa9sBqpnq+u4iUUkBYRm7OmjYY4gh/6QUHYzPTo58gfAbNWSVtYrZ6zsesTyUqpUEX7lhoo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757537596; c=relaxed/simple; bh=6LtWeR6njlSQUfOWwNQxVSZ6hXdXMfNfVzcTt/2GVO8=; h=Message-ID:Date:MIME-Version:Subject:To:References:From: In-Reply-To:Content-Type; b=S+E25NOUGvd83NNMFT2cKB5NAWkmjofBTZlYIMNzmDjv5Zmx87KiXJDTv0zqBkpRMRCgu11hHB49lhd8IsCeEWQmnzDitopKzeqbYKqe1JJpq3XelTviF2mZagrUs9443SY9s3/AR0wxPn1E6rje0CfQ8I6CInCqG2nn7ygUphs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZxjxLkrD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZxjxLkrD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C29E0C4CEEB; Wed, 10 Sep 2025 20:53:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1757537596; bh=6LtWeR6njlSQUfOWwNQxVSZ6hXdXMfNfVzcTt/2GVO8=; h=Date:Subject:To:References:From:In-Reply-To:From; b=ZxjxLkrDnO1c8vhfj+mXZxsAk2EuAmLfq7OuGKPeb3s24KtuqjhsvE6gzpODSn4+1 1Cu3117XqnpqCS9Z5Q1aNt1oJ7jOn5J58RtN0v10ig3RxXChO9C4yS1141a5KR/fmJ VT5m3nUyr/faMhkJR+NlAW8NVVQOMEV7aoR4OcWV+5PJK9na/t6mGAJTDfvWU58v0b eCfRd2t3qO+5GbrCVQThqHRzgASlgIMV5KBW9U8Dev2l7LUAlkmzOAxnYAwFllPYbD o7wwDAXt4D5nTh3b5UN7RtS5OaFc3yCknEw6NMLaxSFcUvaquH1cEWyKIsJSkDMvxT 08Fy7pTM6LKmQ== Message-ID: <2d7371b2-23c0-489b-8e20-ea6d8ec30cc2@kernel.org> Date: Wed, 10 Sep 2025 16:53:14 -0400 Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 1/3] aws: add dynamic cloud configuration support To: Luis Chamberlain , Daniel Gomez , kdevops@lists.linux.dev References: <20250909005644.798127-1-mcgrof@kernel.org> <20250909005644.798127-2-mcgrof@kernel.org> Content-Language: en-US From: Chuck Lever Organization: kernel.org In-Reply-To: <20250909005644.798127-2-mcgrof@kernel.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Hey Luis - On 9/8/25 8:56 PM, Luis Chamberlain wrote: > Move away from static AWS kconfig files to dynamic AWS kconfig support. > We leverage AWS APIs to dynamically to generate Kconfig options for instance > types, regions, and AMIs. This ensures users always have access to current > cloud resources without having kdevops admins manually hand craft updates. > This also makes it less error prone, and lets us scale. > > To generate new AWS dynamic kconfig files, one can use: > > make cloud-config # takes about ~6 minutes Yeah, the cloud APIs are quite slow. This is why terraform is highly parallelized! > Regular users are not expected to have to call this. Only kdevops admins are. > This will generaed Kconfig *.generated files, which we git-ignore. If we > want to keep these, and propagate them to kdevops users through a commit, > kdevops admins can then run: > > make cloud-update > > This will generate kconfig *.static file which we do allow to commit. > So users are expected to reap benefit of these kconfig *.static files. > And kdevops admins are the ones who will routinely (once a month or so) > will run both: > > make cloud-config > make cloud-update > > The implementation generates configuration files on-demand by querying > AWS EC2 APIs. If the AWS CLI is unavailable or not configured, the system > provides hardcoded defaults through fallback functions: > > * generate_default_instance_families_kconfig() > * generate_default_regions_kconfig() > * generate_default_gpu_amis_kconfig() > > hese return common instance types like m5, t3 and major regions, ensuring > kdevops remains functional without AWS CLI access. > > We completely remove all the old AWS static files in favor for the new world > order of dynamic kconfig for cloud. This commit only removes all the > static stuff, and adds all the dynamic kconfig scripts for AWS. In the > subsequent commits we will add static files from a system that does have a > functional aws cli. > > New files: > - scripts/aws-cli: CLI wrapper for AWS operations > - scripts/aws_api.py: AWS API functions with fallback defaults > - docs/cloud-configuration.md: Setup and usage documentation > > Changes: > - terraform/aws/kconfigs/: Dynamic Kconfig generation replaces static files > - scripts/generate_cloud_configs.py: Parallelize cloud kconfig > generation support to reduce the time it takes > - scripts/dynamic-cloud-kconfig.Makefile: Build system integration > > Prerequisites for full functionality: > - AWS CLI installed (pip install awscli) > - AWS credentials configured (~/.aws/credentials or environment variables) > - IAM permissions: ec2:Describe*, pricing:GetProducts (optional) > > Without these prerequisites, kdevops uses built-in fallback functions. > > Generated-by: Claude AI > Signed-off-by: Luis Chamberlain > --- > .gitignore | 3 + > docs/cloud-configuration.md | 533 ++++++++ > docs/kdevops-terraform.md | 9 + > scripts/aws-cli | 436 +++++++ > scripts/aws_api.py | 1161 +++++++++++++++++ > scripts/dynamic-cloud-kconfig.Makefile | 88 +- > scripts/generate_cloud_configs.py | 193 ++- > terraform/aws/kconfigs/Kconfig.compute | 93 +- > terraform/aws/kconfigs/Kconfig.location | 690 +--------- > .../aws/kconfigs/instance-types/Kconfig.c7a | 28 - > .../aws/kconfigs/instance-types/Kconfig.i4i | 33 - > .../aws/kconfigs/instance-types/Kconfig.im4gn | 25 - > .../kconfigs/instance-types/Kconfig.is4gen | 25 - > .../aws/kconfigs/instance-types/Kconfig.m5 | 48 - > .../aws/kconfigs/instance-types/Kconfig.m7a | 57 - > 15 files changed, 2430 insertions(+), 992 deletions(-) > create mode 100644 docs/cloud-configuration.md > create mode 100755 scripts/aws-cli > create mode 100755 scripts/aws_api.py > delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.c7a > delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.i4i > delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.im4gn > delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.is4gen > delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.m5 > delete mode 100644 terraform/aws/kconfigs/instance-types/Kconfig.m7a My vision was that the scripts would update the files under instance-types/ , not remove them. This is one reason why I broke these out into subdirectories -- to make it straightforward for the files to be machine-generated, since they are all small and single-purpose. Should be one file per instance family. Kconfig.compute (and Kconfig.location) should be updated in the same way as the instance-types/ directory, and not deleted by the patch, unless there's something that makes the current directory structure totally unworkable. (which is totally possible). When I "make menuconfig" right after pulling these patches, the "Region" menu is gone. All that is left is a string entry field for an availability zone. I can debug further if you would like more details. > diff --git a/docs/kdevops-terraform.md b/docs/kdevops-terraform.md > index 3291444d..983dbad5 100644 > --- a/docs/kdevops-terraform.md > +++ b/docs/kdevops-terraform.md > @@ -33,6 +33,15 @@ That should let you start configuring your cloud provider options. You can > use the same main menu to configure specific workflows supported by kdevops, > by defaults no workflows are enabled, and so all you get is the bringup. > > +### Dynamic Cloud Configuration > + > +kdevops supports dynamic cloud provider configuration that queries live cloud > +APIs for up-to-date instance types, regions, and AMI options. This feature > +ensures you always have access to the latest cloud resources without manual > +updates. For detailed information about dynamic cloud configuration, including > +setup, prerequisites, and advanced usage, see the > +[Cloud Configuration Management documentation](cloud-configuration.md). This paragraph might mention that dynamic configuration is an advanced feature (possibly intended for developers rather than users). A new kdevops user, for example, does not have to do anything with this feature to start using kdevops' cloud facilities. > +#!/usr/bin/env python3 > +# SPDX-License-Identifier: MIT > +""" > +AWS CLI tool for kdevops > + > +A structured CLI tool that wraps AWS CLI commands and provides access to > +AWS cloud provider functionality for dynamic configuration generation > +and resource management. > +""" > + > +import argparse > +import json > +import sys > +import os > +from typing import Dict, List, Any, Optional, Tuple > +from pathlib import Path For scripts/aws_cli and scripts/aws_cli.py, an alternative would be to use boto3 directly, which is the AWS SDK for Python. Just a random idea. The script [ deleted here ] is much too large to get any kind of handle on. Really, Claude needs to break these changes up, just as we expect from any human contributor, so that they can be reviewed and debugged in a sane fashion. One way to go about this would be to write just enough in this pass to handle updating the Kconfig.compute menu, and leave locations alone for now. It's the new instance families and AMIs that you are most interested in, yes? How about one script whose input is an instance family name, and whose output is one of the submenu files? Another script can create AMI submenus. A third can call those two repeatedly to build Kconfig.compute. Breaking it down like that would make it easier to experiment and develop with. I'm wondering if these scripts might be better located under terraform/aws instead of the top-level scripts/ directory. > diff --git a/terraform/aws/kconfigs/Kconfig.location b/terraform/aws/kconfigs/Kconfig.location > index e5dacb9d..fbdf7eb1 100644 > --- a/terraform/aws/kconfigs/Kconfig.location > +++ b/terraform/aws/kconfigs/Kconfig.location > @@ -1,679 +1,21 @@ > -choice > - prompt "AWS region" > - default TERRAFORM_AWS_REGION_US_WEST_2 [ ... ] > - default "us-east-2c" if TERRAFORM_AWS_AV_ZONE_US_EAST_2C > - default "us-west-1b" if TERRAFORM_AWS_AV_ZONE_US_WEST_1B > - default "us-west-2b" if TERRAFORM_AWS_AV_ZONE_US_WEST_2B > - default "us-west-2d" if TERRAFORM_AWS_AV_ZONE_US_WEST_2D > + In general, AWS region names are lower-case letters only. > + Sometimes, they include a number suffix (see us-east-2a, for > + example). This comment is incorrect. "us-east-2a" is an availability zone, not a region. That makes me a little suspicious -- zones and regions are definitely not the same thing. Btw, I'm not using Claude Code -- I'm using a few others, including Claude Sonnet. I'm not quite prepared to take your prompt and redrive the code generation. Have you given any thought to preserving the prompt(s) you are using (kind of like a commit history) ? Maybe I'm thinking about this process too deterministically. -- Chuck Lever