Re: [PATCH v4 3/8] aws: add optimized Kconfig generator using Chuck's scripts

[Chuck Lever <cel@kernel.org>]

On 9/16/25 5:34 PM, Luis Chamberlain wrote:
> Create a wrapper script that orchestrates Chuck's existing AWS scripts
> (ec2_instance_info.py, aws_regions_info.py, aws_ami_info.py) to generate
> Kconfig files with JSON caching and parallel processing.
> 
> This approach leverages Chuck's already working scripts while adding:
> - JSON caching with 24-hour TTL in ~/.cache/kdevops/aws/
> - Parallel fetching of instance data (10 concurrent workers)
> - Parallel file writing (20 concurrent workers)
> - Proper data structure handling for families list
> 
> Performance improvements:
> - First run: ~21 seconds to fetch all data from AWS
> - Cached runs: ~0.04 seconds (525x faster)
> - Generates 75 Kconfig files for 72 instance families
> 
> The script properly uses Chuck's existing AWS API implementations
> rather than reimplementing them, maintaining code reuse and consistency.
> 
> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>

I've got a slightly different approach in mind, but same goal: make
these scripts fast. Great minds think alike!

After working with my scripts a bit, it seems like we really don't need
the JSON intermediate step. It made sense when getting output from the
CLI, but the response from the boto3 client is truly enough to assemble
the Kconfig menus directly.

Building the instance types is what seems to take the longest because
the script repeatedly does a describe_instance_types and the response
to that is HUGE. But we can do that once at the top of the script, and
simply use the same response as often as needed.

The use case seems to be "build the Kconfig menus" and then use them
for a while. So we have the 21 second first run, but I'm not yet seeing
a benefit to caching -- why rebuild the menus again if your input data
is going to remain exactly the same for 24 hours? Let me know if I've
misunderstood.

And, I now agree it makes sense to get rid of the menu subdirectories.
Sourcing the little files doesn't seem to add any value. My new scripts
construct one large locations menu and one large compute menu.


-- 
Chuck Lever