Re: Split archives to new repo - clean kdevops repo

[Jeff Layton <jlayton@kernel.org>]

On Thu, 2024-03-28 at 11:04 -0700, Luis Chamberlain wrote:
> At LSFMM long ago we had a discussion about keeping test results
> around, back of the napkin calcluation then for fstests end up
> with an estimate impact of about 730.40 GiB per year, removing the
> good results and only keeping *.bad and *.dmesg for failed tests
> was about 456.5 MiB per year, and compressing it ~44 KiB per year.
> For 5 filesystems that is about 100 MiB / year.
> 
> Well, kdevops is a big fat pig now over 200 MiB, and even though the
> above math was conservative it turns out that to make results more
> useful for things like an ELK stack, we want to keep good results.
> And in practice now git grep'ing is creating a lot of noise.
> 
> And so it seems best the experiment to keep results for tests has
> come to a crux, and I'd like to suggest we spin off the results into
> an optional archive directory per workflow, and we create a fresh
> new kdevops git tree from scratch, and move the existing one to
> kdevops-history.
> 

We could just git rm the old results, but that does keep the history
around, which will always be huge.

ACK from me on this idea.

> The way I'd see this is, we'd look for ~/kdevops-archive/ and if present
> a new future 'make kdevops-archive' would add directory symlinks to
> 
> workflows/fstests/results/archive/ --> ~/kdevops-archive/workflows/fstests/results/archive/
> 
> That would keep things like scripts/workflows/fstests/copy-results.sh
> working, we'd just have to check if the directory exist first before
> proceeding.
> 
> This:
> 
>   * let's us modify results to contain even *good* results so we can
>     help ELK stacks which want good results too
>   * puts the churn of the archive out to a separate tree
>   * keeps kdevops clean
> 
> If this is agreeable, perhaps we can make the switch Monday? Any
> opposition to this?
> 

I like this idea. Do you intend to keep tracking the results with git,
just in another tree? You could consider using git-lfs to store archive
tarballs. github supports it, but they only give you 1GB by default:

    https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage

Looks like it's only $60 a year for 50G though.
-- 
Jeff Layton <jlayton@kernel.org>