Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[demerphq <demerphq@gmail.com>]

2009/8/1 J.H. <warthog19@eaglescrag.net>:
> Mark A Rada wrote:
>>
>>  >Note that for me the above results are not aligned in table.
>>  >This is a cosmetic issue.
>>
>> The table formatting issue was due to a bad habit of mixing tabs and
>> spaces,
>> I decided to go with spaces this time :)
>>
>>
>>  >One thing that would concern me greatly, is not so much the CPU time
>> (though that's a *huge* change in comparison to gz) but the memory usage.
>>  Where gzip and bzip2 are chewing 4M and 13M respectively, xz chews 102M.
>>  >From a 'beefy' server perspective chewing up that much memory per snapshot
>> for that long could be bad.  This is likely something that needs to have
>> some sort of enable/disable switch if it's going to be included.
>>
>> True, and there are two solutions I can think of for this "problem".
>>
>>    1. My tests were at the default compression level, the XZ documentation
>>    says that at lower levels you will get resource usage and compression
>>    ratios that are comparable to BZip2. However, I'm not sure where you
>>    would change the compression level variable for this (globally for the
>>    system, somewhere in $GITWEB_CONFIG, a git config variable). Does
>>    someone know the correct answer here?
>
> Well you can always call xz with -[1-9] to change the compression level
> (same as gzip and bzip2) though I think a full disabling would be 'more'
> preferable, though I'm not sure I like Jakub's suggestion of just deleting
> it after the fact, it would work.
>
>>    2. Implement snapshot caching for Gitweb.
>
> I think it's slightly broken in my version (binary files don't work right
> apparently, it's on my todo list to fix in my upcoming update) but both Lea
> and I have done this long ago (caching layers in Gitweb), which would be an
> acceptable workaround for this, create once and serve many - though this has
> the downside of trading cpu for diskspace.  At least with xz there's less
> diskspace used ;-)
>
> I think more my concern is more what's enabled by default, and since xz is
> still new (as was pointed out) it's probably worth only enabling if the
> admin selects it to be enabled.

FWIW the perl project ripped out all the snapshot generation logic
from gitweb, and replaced it with a tool that generates snapshots
correctly for our requirements (if the build process needs additional
files /currently/ git-archive does not support adding them), this
includes a disk level cache for the snapshots since creating the tar,
adding the additional files, then gziping is quite slow.

If its interesting to people I can post it and the other changes here,
although its not a "nice" change, as I literally ripped out the
existing code.

cheers,
Yves





-- 
perl -Mre=debug -e "/just|another|perl|hacker/"