Re: kernel.org mirroring (Re: [GIT PULL] MMC update)

[Jakub Narebski <jnareb@gmail.com>]

Linus Torvalds wrote:
[...] 
> For example, if the git "refs/heads/" (or tags) directory hasn't changed 
> in the last two months, we should probably set any ref-relative gitweb 
> pages to have a caching timeout of a day or two. In contrast, if it's 
> changed in the last hour, maybe we should only cache it for five minutes.
> 
> Jakub: any way to make gitweb set the "expires" fields _much_ more 
> aggressively. I think we should at least have the ability to set a basic 
> rules like
> 
>  - a _minimum_ of five minutes regardless of anything else
> 
>    We might even tweak this based on loadaverage, and it might be 
>    worthwhile to add a randomization, to make sure that you don't get into 
>    situations where everything webpage needs to be recalculated at once.

I think the minimum expires (or minimum _additional_ expires: as of now
giweb only does expires +1d for explicit hash requests) should depend on
how often project changes. How often there are pushes to kernel.org?
 
>  - if refs/ directories are old, raise the minimum by the age of the refs
> 
>    If it's more than an hour old, raise it to ten minutes. If it's more 
>    than a day, raise it to an hour. If it's more than a month old, raise 
>    it to a day. And if it's more than half a year, it's some historical 
>    archive like linux-history, and should probably default to a week or 
>    more.

What about packed refs?

We can certainly raise expires for tags (tags objects), as they should not
usually change.
 
>  - infinite for stuff that isn't ref-related.

As sha1 is not changeable, everything that is accessed by explicit 
sha1 (hash), or by explicit sha1 (hash_base) plus pathname (file_name)
should have effectively infinite expires.


Every caching would need some temporary memory, or temporary disk space.
And perhaps mod_perl specific caching would be useful here...

P.S. I have added Pasky to Cc:, as he manages http://repo.or.cz public
git repository hosting (much smaller than kernel.org and I think under less
load: but also I think withour kernel.org resources).
-- 
Jakub Narebski