kernelci-storage update

kernelci-storage update

[Denys Fedoryshchenko]

# KernelCI-Storage Project Update

## Deployment Status  
As mentioned in the last newsletter, we have migrated the production server to a new distribution (Bookworm) and deployed kernelci-storage. The system is also deployed on staging and is currently migrating to files-staging.kernelci.org.

## Implementation Details  
As planned in the first newsletter, we implemented a minimal API sufficient for the KernelCI project to work. Based on feedback, we updated endpoints to make them compatible with the kernelci-core storage backend implementation.

Currently, kernelci-storage is written in Rust. The source code is available at the [kernelci-storage]([https://github.com/nuclearcat/kernelci-storage](https://github.com/nuclearcat/kernelci-storage)) repository but will be moved to the KernelCI organization namespace once pending issues are resolved.

Available endpoints are documented in the [kernelci-storage README]([https://github.com/nuclearcat/kernelci-storage/blob/main/README.md](https://github.com/nuclearcat/kernelci-storage/blob/main/README.md)):  
- `GET|HEAD /path/to/file` - Download artifact (doesn't require authentication) or retrieve file info  
- `POST /upload` - Upload artifact (requires authentication), multipart/form-data  
- field: file0 - file to upload  
- field: path - path to save file to, can be absolute or relative to upload directory (default: upload directory root)

## Implemented Features  
- Heuristics to set correct content-type for uploaded files, while also supporting custom content-type settings  
- Support for uploading large files (resolving issues that existed in the Python backend)  
- Operation as a caching forward proxy to reduce charges from the storage provider per request  
- Support for downloading files with range requests

## Upcoming Features  
- Support for restricting upload directory (by user)  
- Lifecycle management for uploaded files (certain types of files might be deleted earlier)

## Challenges  
We has planned file browsing feature, but during implementation we faced challenge: Azure Blob Storage object list retrieval is very slow, 5000 objects (one request limit) takes 2-5 seconds to complete, and due dtb files we have tens of millions of them. We have solution for this, but it is not implemented yet.


## Future Plans  
We need your feedback! Please share your thoughts and suggestions on the mailing list or send message to our Discord server.

Thanks for reading!