Does pack v4 do anything to commits?

Does pack v4 do anything to commits?

[Nguyen Thai Ngoc Duy]

Hi Nico,

I had an experiment on speeding up "rev-list --all". If I cache sha-1
of tree and parent, and committer date of single-parent commits, in
binary form, rev-list can be sped up significantly. On linux-2.6.git,
it goes from 14s to 4s (2s to 0.8 for git.git). Profiling shows that
commit parsing (get_sha1_hex, parse_commit_date) dominates rev-list
time.

>From what I remember, pack v4 is mainly about changing tree
representation so that we can traverse object DAG as fast as possible.
Do you do anything to commit representation too? Maybe it's worth
storing the above info along with the compressed commit objects in
pack to shave some more seconds.

By the way, is latest packv4 code available somewhere to fetch?
-- 
Duy

Re: Does pack v4 do anything to commits?

[Nicolas Pitre]

On Sat, 18 Feb 2012, Nguyen Thai Ngoc Duy wrote:

> Hi Nico,
> 
> I had an experiment on speeding up "rev-list --all". If I cache sha-1
> of tree and parent, and committer date of single-parent commits, in
> binary form, rev-list can be sped up significantly. On linux-2.6.git,
> it goes from 14s to 4s (2s to 0.8 for git.git). Profiling shows that
> commit parsing (get_sha1_hex, parse_commit_date) dominates rev-list
> time.
> 
> >From what I remember, pack v4 is mainly about changing tree
> representation so that we can traverse object DAG as fast as possible.
> Do you do anything to commit representation too? Maybe it's worth
> storing the above info along with the compressed commit objects in
> pack to shave some more seconds.

Both the tree and commit object representations are completely changed 
to evacuate SHA1 parsing and searching entirely.  The SHA1 references 
are uncompressed, and replaced by indices into the pack for direct 
lookup without any binary search.  And the commit dates are stored in 
binary form. All path names as well as author/committer names are 
factored out into a table as well. This should make history traversal 
operations almost as fast as walking a linked list in memory, while 
making the actual pack size smaller at the same time.

> By the way, is latest packv4 code available somewhere to fetch?

Well, not yet.  Incidentally, I'm going in the Caribbeans for a week in 
a week, with no kids and only my wife who is going to be busy with scuba 
diving activities.  Like I did last year, I'm going to take some time to 
pursue my work on Pack v4 during that time.  And I intend to publish it 
when I come back, whatever state it is in, so someone else can complete 
the work eventually (I have too much to do to spend significant time on 
Git these days).


Nicolas