Re: [PATCH 1/2] aws: add dynamic cloud configuration support using AWS CLI

[Chuck Lever <cel@kernel.org>]

On 9/7/25 6:10 PM, Luis Chamberlain wrote:
> On Sun, Sep 07, 2025 at 01:24:43PM -0400, Chuck Lever wrote:
>> On 9/7/25 12:23 AM, Luis Chamberlain wrote:

>>> +### For Regular Users
>>> +
>>> +Regular users benefit from pre-generated static configurations:
>>> +
>>> +1. **Clone or pull the repository**:
>>> +   ```bash
>>> +   git clone https://github.com/linux-kdevops/kdevops
>>> +   cd kdevops
>>> +   ```
>>> +
>>> +2. **Use cloud configurations immediately**:
>>> +   ```bash
>>> +   make menuconfig     # Cloud options load instantly from static files
>>> +   make defconfig-aws-large
>>> +   make
>>> +   ```
>>> +
>>> +No cloud CLI tools or API access required - everything loads from committed static files.
>>
>> I expect that a CLI tool or cloud console access /is/ needed to generate
>> authentication tokens, so this claim ought to be more specific.
> 
> the docs sucked at that, here's an additional patch which expands on
> the requirements, which we can squash:
> 
> From 62ba9c366953ab82ed0de39b44f044a019fe273c Mon Sep 17 00:00:00 2001
> From: Luis Chamberlain <mcgrof@kernel.org>
> Date: Sun, 7 Sep 2025 14:44:41 -0700
> Subject: [PATCH] docs: expand AWS dynamic cloud configuration documentation
> 
> Enhance documentation for AWS dynamic configuration requirements:
> 
> - Detailed prerequisites and AWS CLI requirements
> - AWS credentials configuration methods
> - Required IAM permissions
> - Implementation architecture details
> - Troubleshooting guide for common issues
> - Best practices for administrators
> - Advanced usage scenarios
> - Key design decisions and rationale
> 
> This helps developer and users understand that the system wraps the
> official AWS CLI tool rather than implementing its own API client, and
> requires proper AWS credentials configuration.
> 
> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
>  docs/cloud-configuration.md | 322 +++++++++++++++++++++++++++++++++++-
>  1 file changed, 317 insertions(+), 5 deletions(-)
> 
> diff --git a/docs/cloud-configuration.md b/docs/cloud-configuration.md
> index e8386c82..dfca93dd 100644
> --- a/docs/cloud-configuration.md
> +++ b/docs/cloud-configuration.md
> @@ -11,6 +11,116 @@ The cloud configuration system follows a pattern similar to Linux kernel refs ma
>  - **No dependency on cloud CLI tools** for regular users
>  - **Reduced API calls** to cloud providers
>  
> +## Prerequisites for Cloud Providers
> +
> +### AWS Prerequisites
> +
> +The AWS dynamic configuration system uses the official AWS CLI tool and requires proper authentication to access AWS APIs.
> +
> +#### Requirements
> +
> +1. **AWS CLI Installation**
> +   ```bash
> +   # Using pip
> +   pip install awscli
> +
> +   # On Debian/Ubuntu
> +   sudo apt-get install awscli
> +
> +   # On Fedora/RHEL
> +   sudo dnf install aws-cli
> +
> +   # On macOS
> +   brew install awscli
> +   ```
> +
> +2. **AWS Credentials Configuration**
> +
> +   You need valid AWS credentials configured in one of these ways:
> +
> +   a. **AWS credentials file** (`~/.aws/credentials`):
> +   ```ini
> +   [default]
> +   aws_access_key_id = YOUR_ACCESS_KEY
> +   aws_secret_access_key = YOUR_SECRET_KEY
> +   ```
> +
> +   b. **Environment variables**:
> +   ```bash
> +   export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
> +   export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
> +   export AWS_DEFAULT_REGION=us-east-1  # Optional
> +   ```
> +
> +   c. **IAM Instance Role** (when running on EC2):
> +   - Automatically uses instance metadata service
> +   - No explicit credentials needed

docs/kdevops-terraform.md has similar information, and includes the
other providers. Section 2 could cite that file, or this patch could
modify/update that instead.

Otherwise, for the patch snippet here:

Reviewed-by: Chuck Lever <chuck.lever@oracle.com>


> +
> +3. **Required AWS Permissions**
> +
> +   The IAM user or role needs the following read-only permissions:
> +   ```json
> +   {
> +     "Version": "2012-10-17",
> +     "Statement": [
> +       {
> +         "Effect": "Allow",
> +         "Action": [
> +           "ec2:DescribeRegions",
> +           "ec2:DescribeAvailabilityZones",
> +           "ec2:DescribeInstanceTypes",
> +           "ec2:DescribeImages",
> +           "pricing:GetProducts"
> +         ],
> +         "Resource": "*"
> +       },
> +       {
> +         "Effect": "Allow",
> +         "Action": [
> +           "sts:GetCallerIdentity"
> +         ],
> +         "Resource": "*"
> +       }
> +     ]
> +   }
> +   ```
> +
> +#### Verifying AWS Setup
> +
> +Test your AWS CLI configuration:
> +```bash
> +# Check AWS CLI is installed
> +aws --version
> +
> +# Verify credentials are configured
> +aws sts get-caller-identity
> +
> +# Test EC2 access
> +aws ec2 describe-regions --output table
> +```
> +
> +#### Fallback Behavior
> +
> +If AWS CLI is not available or credentials are not configured:
> +- The system automatically falls back to pre-defined static defaults
> +- Basic instance families (M5, T3, C5, etc.) are still available
> +- Common regions (us-east-1, eu-west-1, etc.) are provided
> +- Default GPU AMI options are included
> +- Users can still use kdevops without AWS API access
> +
> +### Lambda Labs Prerequisites
> +
> +Lambda Labs configuration requires an API key:
> +
> +1. **Obtain API Key**: Sign up at [Lambda Labs](https://lambdalabs.com) and generate an API key
> +
> +2. **Configure API Key**:
> +   ```bash
> +   export LAMBDA_API_KEY=your_api_key_here
> +   ```
> +
> +3. **Fallback Behavior**: Without an API key, default GPU instance types are provided
> +
>  ## Configuration Generation Flow
>  
>  ```
> @@ -133,6 +243,48 @@ No cloud CLI tools or API access required - everything loads from committed stat
>  
>  ## How It Works
>  
> +### Implementation Architecture
> +
> +The cloud configuration system consists of several key components:
> +
> +1. **API Wrapper Scripts** (`scripts/aws-cli`, `scripts/lambda-cli`):
> +   - Provide CLI interfaces to cloud provider APIs
> +   - Handle authentication and error checking
> +   - Format API responses for Kconfig generation
> +
> +2. **API Libraries** (`scripts/aws_api.py`, `scripts/lambdalabs_api.py`):
> +   - Core functions for API interactions
> +   - Generate Kconfig syntax from API data
> +   - Provide fallback defaults when APIs unavailable
> +
> +3. **Generation Orchestrator** (`scripts/generate_cloud_configs.py`):
> +   - Coordinates parallel generation across providers
> +   - Provides summary information
> +   - Handles errors gracefully
> +
> +4. **Makefile Integration** (`scripts/dynamic-cloud-kconfig.Makefile`):
> +   - Defines make targets
> +   - Manages file dependencies
> +   - Handles cleanup and updates
> +
> +### AWS Implementation Details
> +
> +The AWS implementation wraps the official AWS CLI tool rather than implementing its own API client:
> +
> +```python
> +# scripts/aws_api.py
> +def run_aws_command(command: List[str], region: str = None) -> Optional[Any]:
> +    cmd = ["aws"] + command + ["--output", "json"]
> +    # ... executes via subprocess
> +```
> +
> +Key features:
> +- **Parallel Generation**: Uses ThreadPoolExecutor to generate instance family files concurrently
> +- **GPU Detection**: Automatically identifies GPU instances and enables GPU AMI options
> +- **Categorized Instance Types**: Groups instances by use case (general, compute, memory, etc.)
> +- **Pricing Integration**: Queries pricing API when available
> +- **Smart Defaults**: Falls back to well-tested defaults when API unavailable
> +
>  ### Dynamic Configuration Detection
>  
>  kdevops automatically detects whether to use dynamic or static configurations:
> @@ -251,14 +403,173 @@ make cloud-config
>  make cloud-update
>  ```
>  
> +## Troubleshooting
> +
> +### AWS Issues
> +
> +#### "AWS CLI not found" Error
> +```bash
> +# Verify AWS CLI installation
> +which aws
> +aws --version
> +
> +# Install if missing (see Prerequisites section)
> +```
> +
> +#### "Credentials not configured" Error
> +```bash
> +# Check current identity
> +aws sts get-caller-identity
> +
> +# If fails, configure credentials:
> +aws configure
> +# OR
> +export AWS_ACCESS_KEY_ID=your_key
> +export AWS_SECRET_ACCESS_KEY=your_secret
> +```
> +
> +#### "Access Denied" Errors
> +- Verify your IAM user/role has the required permissions (see Prerequisites)
> +- Check if you're in the correct AWS account
> +- Ensure your credentials haven't expired
> +
> +#### Slow Generation Times
> +- Normal for AWS (6+ minutes due to API pagination)
> +- Consider using `make cloud-update` with pre-generated configs
> +- Run generation during off-peak hours
> +
> +#### Missing Instance Types
> +```bash
> +# Force regeneration
> +make clean-cloud-config
> +make cloud-config
> +make cloud-update
> +```
> +
> +### General Issues
> +
> +#### Static Files Not Loading
> +```bash
> +# Verify static files exist
> +ls terraform/aws/kconfigs/*.static
> +
> +# If missing, regenerate:
> +make cloud-config
> +make cloud-update
> +```
> +
> +#### Changes Not Reflected in Menuconfig
> +```bash
> +# Clear Kconfig cache
> +make mrproper
> +make menuconfig
> +```
> +
> +#### Debugging API Calls
> +```bash
> +# Enable debug output
> +export DEBUG=1
> +make cloud-config
> +
> +# Test API directly
> +scripts/aws-cli --output json regions list
> +scripts/aws-cli --output json instance-types list --family m5
> +```
> +
> +## Best Practices
> +
> +1. **Regular Updates**: Administrators should regenerate configurations monthly or when new instance types are announced
> +
> +2. **Commit Messages**: Include generation date and tool versions when committing static files:
> +   ```bash
> +   git commit -m "cloud: update AWS static configurations
> +
> +   Generated with AWS CLI 2.15.0 on 2024-01-15
> +   - Added new G6e instance family
> +   - Updated GPU AMI options
> +   - 127 instance families now available"
> +   ```
> +
> +3. **Testing**: Always test generated configurations before committing:
> +   ```bash
> +   make cloud-config
> +   make cloud-update
> +   make menuconfig  # Verify options appear correctly
> +   ```
> +
> +4. **Partial Generation**: For faster testing, generate only specific providers:
> +   ```bash
> +   make cloud-config-aws      # AWS only
> +   make cloud-config-lambdalabs  # Lambda Labs only
> +   ```
> +
> +5. **CI/CD Integration**: Consider automating configuration updates in CI pipelines
> +
> +## Advanced Usage
> +
> +### Custom AWS Profiles
> +```bash
> +# Use non-default AWS profile
> +export AWS_PROFILE=myprofile
> +make cloud-config
> +```
> +
> +### Specific Region Generation
> +```bash
> +# Generate for specific region (affects default selections)
> +export AWS_DEFAULT_REGION=eu-west-1
> +make cloud-config
> +```
> +
> +### Parallel Generation
> +The system automatically uses parallel processing:
> +- AWS: Up to 10 concurrent instance family generations
> +- Reduces total generation time significantly
> +
> +## File Reference
> +
> +### AWS Files
> +- `terraform/aws/kconfigs/Kconfig.compute.{generated,static}` - Instance families
> +- `terraform/aws/kconfigs/Kconfig.location.{generated,static}` - Regions and zones
> +- `terraform/aws/kconfigs/Kconfig.gpu-amis.{generated,static}` - GPU AMI options
> +- `terraform/aws/kconfigs/instance-types/Kconfig.*.{generated,static}` - Per-family sizes
> +
> +### Marker Files
> +- `.aws_cloud_config_generated` - Enables dynamic AWS config
> +- `.cloud.initialized` - General cloud config marker
> +
> +### Scripts
> +- `scripts/aws-cli` - AWS CLI wrapper with user-friendly commands
> +- `scripts/aws_api.py` - AWS API library and Kconfig generation
> +- `scripts/generate_cloud_configs.py` - Main orchestrator for all providers
> +- `scripts/dynamic-cloud-kconfig.Makefile` - Make targets and integration
> +
>  ## Implementation Details
>  
> -The cloud configuration system is implemented in:
> +The cloud configuration system is implemented using:
> +
> +- **AWS CLI Wrapper**: Uses official AWS CLI via subprocess calls
> +- **Parallel Processing**: ThreadPoolExecutor for concurrent API calls
> +- **Fallback Defaults**: Pre-defined configurations when API unavailable
> +- **Two-tier System**: Generated (dynamic) → Static (committed) files
> +- **Kconfig Integration**: Seamless integration with Linux kernel-style configuration
> +
> +### Key Design Decisions
> +
> +1. **Why wrap AWS CLI instead of using boto3?**
> +   - Reduces dependencies (AWS CLI often already installed)
> +   - Leverages AWS's official tool and authentication methods
> +   - Simpler credential management (uses standard AWS config)
> +
> +2. **Why the two-tier system?**
> +   - Fast loading for regular users (no API calls needed)
> +   - Fresh data when administrators regenerate
> +   - Works offline and in restricted environments
>  
> -- `scripts/dynamic-cloud-kconfig.Makefile` - Make targets and build rules
> -- `scripts/aws_api.py` - AWS configuration generator
> -- `scripts/generate_cloud_configs.py` - Main configuration generator
> -- `terraform/*/kconfigs/` - Provider-specific Kconfig files
> +3. **Why 6 minutes generation time?**
> +   - AWS API pagination limits (100 items per request)
> +   - Comprehensive data collection (all regions, all instance types)
> +   - Parallel processing already optimized
>  
>  ## See Also
>  
> @@ -266,3 +577,4 @@ The cloud configuration system is implemented in:
>  - [Azure VM Sizes](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes)
>  - [GCE Machine Types](https://cloud.google.com/compute/docs/machine-types)
>  - [kdevops Terraform Documentation](terraform.md)
> +- [AWS CLI Documentation](https://docs.aws.amazon.com/cli/)


-- 
Chuck Lever