caching commit patch-ids for fast git-cherry

caching commit patch-ids for fast git-cherry

[Geoffrey Irving]

I'm planning to use cherry picking to manage long term syncing between
cvs/perforce and git repositories.  This means I'll have scripts
running git-cherry between branches with hundreds of uncommon commits,
and I want git-cherry to be much, much, faster.

It looks like I can do this by caching commit->patch-id pairs from
commit_patch_id() in patch-ids.c to a file, say
$GIT_DIR/commit-patch-id-cache.  The file would be binary and append
only, and could be blown away if .  Any suggestions / concerns before
I write this?  Is there any reusable efficient map code for storing
the commit->patch-id map, or should I just mirror the blocked storage
+ binary search used for struct patch_ids?

Thanks,
Geoffrey

Re: caching commit patch-ids for fast git-cherry

[Johannes Schindelin]

Hi,

On Thu, 29 May 2008, Geoffrey Irving wrote:

> I'm planning to use cherry picking to manage long term syncing between 
> cvs/perforce and git repositories.  This means I'll have scripts running 
> git-cherry between branches with hundreds of uncommon commits, and I 
> want git-cherry to be much, much, faster.
> 
> It looks like I can do this by caching commit->patch-id pairs from
> commit_patch_id() in patch-ids.c to a file, say
> $GIT_DIR/commit-patch-id-cache.  The file would be binary and append
> only, and could be blown away if .  Any suggestions / concerns before
> I write this?  Is there any reusable efficient map code for storing
> the commit->patch-id map, or should I just mirror the blocked storage
> + binary search used for struct patch_ids?

I would store the stuff sorted, so that the lookup is fast, generation 
less so.

For inspiration, you might want to look at the "notes" branch in my 
personal fork:

http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes

Hth,
Dscho

Re: caching commit patch-ids for fast git-cherry

[Geoffrey Irving]

On Thu, May 29, 2008 at 10:13 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Thu, 29 May 2008, Geoffrey Irving wrote:
>
>> I'm planning to use cherry picking to manage long term syncing between
>> cvs/perforce and git repositories.  This means I'll have scripts running
>> git-cherry between branches with hundreds of uncommon commits, and I
>> want git-cherry to be much, much, faster.
>>
>> It looks like I can do this by caching commit->patch-id pairs from
>> commit_patch_id() in patch-ids.c to a file, say
>> $GIT_DIR/commit-patch-id-cache.  The file would be binary and append
>> only, and could be blown away if .  Any suggestions / concerns before
>> I write this?  Is there any reusable efficient map code for storing
>> the commit->patch-id map, or should I just mirror the blocked storage
>> + binary search used for struct patch_ids?
>
> I would store the stuff sorted, so that the lookup is fast, generation
> less so.

The motivation for append-only was robustness, not speed, but I don't
think either concern is very significant.

> For inspiration, you might want to look at the "notes" branch in my
> personal fork:
>
> http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes

Cool.  I'd rather copy just that code entirely rather than use it for
inspiration, since it does exactly what I need.  It would be silly to
have two blocks of code implementing "persistent map from 20 byte hash
to 20 byte hash".

I'll start by just copying the entire nodes-index implementation (with
a few name substitutions), and we (or I) can refactor it later if both
end up in the same respository.

Geoffrey

Re: caching commit patch-ids for fast git-cherry

[Johannes Schindelin]

Hi,

On Thu, 29 May 2008, Geoffrey Irving wrote:

> On Thu, May 29, 2008 at 10:13 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > Hi,
> >
> > On Thu, 29 May 2008, Geoffrey Irving wrote:
> >
> >> I'm planning to use cherry picking to manage long term syncing 
> >> between cvs/perforce and git repositories.  This means I'll have 
> >> scripts running git-cherry between branches with hundreds of uncommon 
> >> commits, and I want git-cherry to be much, much, faster.
> >>
> >> It looks like I can do this by caching commit->patch-id pairs from 
> >> commit_patch_id() in patch-ids.c to a file, say 
> >> $GIT_DIR/commit-patch-id-cache.  The file would be binary and append 
> >> only, and could be blown away if .  Any suggestions / concerns before 
> >> I write this?  Is there any reusable efficient map code for storing 
> >> the commit->patch-id map, or should I just mirror the blocked storage
> >> + binary search used for struct patch_ids?
> >
> > I would store the stuff sorted, so that the lookup is fast, generation 
> > less so.
> 
> The motivation for append-only was robustness, not speed, but I don't 
> think either concern is very significant.

I think that robustness comes from "write new file and rename if all 
succeeded", not from append-only.  Think of the case where you run out of 
disk space; with append-only, it is more complicated to get back to a 
known good state.
 
> > For inspiration, you might want to look at the "notes" branch in my 
> > personal fork:
> >
> > http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes
> 
> Cool.  I'd rather copy just that code entirely rather than use it for 
> inspiration, since it does exactly what I need.  It would be silly to 
> have two blocks of code implementing "persistent map from 20 byte hash 
> to 20 byte hash".

Hehe.  That's what I meant by "inspiration" ;-)

> I'll start by just copying the entire nodes-index implementation (with a 
> few name substitutions), and we (or I) can refactor it later if both end 
> up in the same respository.

Very nice!

Thanks,
Dscho