KernelCI Weekly Newsletter (Week 2)

[Denys Fedoryshchenko <denys.f@collabora.com>]

**Summary**  
This week, we have been working on the following:  

- kci-dev pre-release v0.1.1dev1 (It is now possible to test the pre-
release with pip install kci-dev==0.1.1.dev1)
- First production update of 2025  
- Improvements to the KernelCI API documentation  
- Implementing automated basic cleanup of new KernelCI storage backend
to manage low disk space (currently based on data age)  
- Removal of 4.19 kernel from testing due to its EOL, except for the
CIP project  
- Preparation of an Ansible recipe for setting up a new production (and
storage) server  

---

**Upcoming Discussions**  

1. **Legacy KernelCI Storage Cleanup**  
   As we deprecate legacy storage, we need to discuss the cleanup
process and how to handle the remaining data.  
   While we aim to retain at least the latest rootfs images, decisions
are required on the rest of the data.  
   If you have any important data stored in the legacy storage, please
let us know.  

2. **KernelCI Development Roadmap**  
   Through discussions with the community, we've identified a list of
features and improvements for KernelCI.  
   Please share your ideas or indicate if you'd like to contribute to
any of these items. 
   This is initial draft, and we will continue to refine the list.

   - **Legacy System Deprecation**
   - **Dashboard Improvements and continued development**
   - **KCI-DEV Tool Development**
   - **Reducing Sysadmin Overhead**  
     - Increased automation and observability  
     - Proper IaC, monitoring, and alerting  
     - Self-service onboarding/offboarding processes  
     - Improved documentation, capacity/log management, etc.  
   - **Automated Staging and Production Deployment**  
   - **Kernel Build Infrastructure Improvements**  
     - Load-balancing  
     - Faster artifact upload optimization  
     - Caching, deterministic build scheduling  
     - Support for bare-metal builders  
   - **Storage Infrastructure Improvements**  
     - Data scraping and egress usage protection  
     - Data retention policies, optional tagging  
     - Mini-CDN for traffic redirection to the closest storage  
     - Minimalistic caching  
   - **Maestro API Enhancements**  
     - Unified API endpoints and improved documentation  
     - Additional endpoints for new features  
     - Unified token/user management  
     - WebSocket interface for real-time notifications  
   - **Maestro Data Structure Enhancements**  
     - Timestamps for task queueing, start, and end times  
     - Extended test information beyond pass/fail/miss  
     - Graceful event handling (timeouts/exceptions)  
   - **Generic Pipeline Schedulers**  
     - Generic kbuild and test schedulers  
     - Simple examples for generic test scheduling  
   - **Self-Hosted KernelCI (On-Premises)**  
   - **Cloud VM Testing**  
   - **RISC-V Enablement**  
   - **AOSP VTS Testing**  
   - **KCI-GitLab Development**  
   - **KernelCI Service Reliability for Uptime SLAs**  
   - **Test Framework Modernization and Unification**  
   - **Test Catalog and Integration**  
   - **Test Results Metrics Support**  
     - Discuss storing and visualizing test results metrics  
   - **Tree Registration to Maestro**  
     - Simplify the process for developers to register their trees  
   - **Subsystems and Maintainers Enablement**  
   - **Logspec Tool for KCIDB and Maintenance**  
   - **Issue Editor and KCIDB Backend**  
   - **Web Dashboard Customization for Users**  
   - **AI for Results Analysis**  
   - **Continuous Result Analysis**  
     - Tools and dashboards for quickly identifying infrastructure
errors and bad reporting  
     - Continuous data analysis and automated bug reporting to CI
systems, the Kernel community, or test suite maintainers  
   - **KCIDB Maintenance and Support**  
   - **KernelCI Labs API**  
     - Enable other CI systems to dispatch tests to KernelCI hardware
labs  
     - Define API for test definition submissions and authorization
processes  

---

**Issues**  

- **K8s Build System Congestion:**  
  Due to congestion in the Kubernetes build system, kernel builds have
experienced delays of up to 2 hours.  
  We are actively working to resolve the issue and improve build times.