Re: pack operation is thrashing my server

[Andreas Ericsson <ae@op5.se>]

Thomas Rast wrote:
> Andreas Ericsson wrote:
>> Nicolas Pitre wrote:
>>> |nico@xanadu:linux-2.6> time git rev-list --objects --all > /dev/null
>>> |
>>> |real    0m21.742s
>>> |user    0m21.379s
>>> |sys     0m0.360s
>>>
>>> That's way too long for 1030198 objects (roughly 48k objects/sec).  And 
>>> it gets even worse with the gcc repository:
>>>
>>> |nico@xanadu:gcc> time git rev-list --objects --all > /dev/null
>>> |
>>> |real    1m51.591s
>>> |user    1m50.757s
>>> |sys     0m0.810s
>>>
>>> That's for 1267993 objects, or about 11400 objects/sec.
>>>
>>> Clearly something is not scaling here.
>>>
>> What are the different packing options for the two repositories?
>> A longer deltachain and larger packwindow would increase the
>> enumeration time, wouldn't it?
> 
> For the fun of it, I ran a test without deltas.  Here's my normal
> git.git:
> 
>   $ du -h .git/objects/pack
>   26M     .git/objects/pack
>   $ git rev-list --all | wc -l
>   17638
>   $ git rev-list --all --objects | wc -l
>   82194
> 
> On a hot cache I get about 61800 objects/sec:
> 
>   $ /usr/bin/time git rev-list --all --objects >/dev/null
>   1.33user 0.04system 0:01.39elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
>   0inputs+0outputs (0major+8087minor)pagefaults 0swaps
> 
> I then made a copy of that and repacked it without deltas (remember to
> remove *.keep, I tripped over that twice):
> 
>   $ git repack --depth=0 --window=0 -a -f -d
>   Counting objects: 82906, done.
>   Writing objects: 100% (82906/82906), done.
>   Total 82906 (delta 0), reused 0 (delta 0)
>   $ du -h .git/objects/pack
>   339M    .git/objects/pack
> 
> Which results in only 28739 objects/sec:
> 

Well, if the objects are, on average, >twice the size, would that
explain it? I'd hate to see some of the sharper git minds hop off
on a wild goose chase if it's not necessary.

How does one go about getting the object sizes? rev-list appears
to have no option for it.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231