Re: Git Scaling: What factors most affect Git performance for a large repo?

[Martin Fick <mfick@codeaurora.org>]

On Friday, February 20, 2015 01:29:12 PM David Turner wrote:
>...
> For a more general solution, perhaps a log of ref updates
> could be used. Every time a ref is updated on the server,
> that ref would be written into an append-only log.  Every
> time a client pulls, their pull data includes an index
> into that log.  Then on push, the client could say, "I
> have refs as-of $index", and the server could read the
> log (or do something more-optimized) and send only refs
> updated since that index.

Interesting idea, I like it.

How would you make this reliable?  It relies on updates 
being reliably recorded which would mean that you would have 
to ensure that any tool which touches the repo follows this 
convention.  That is unfortunately a tough thing to enforce 
for most people.

But perhaps, instead of logging updates, the server could 
log snapshots of all refs using an atomically increasing 
sequence number.  Then missed updates do not matter, a 
sequence number is simplly an opaque handle to some full ref 
state that can be diffed against.  The snapshots need not 
even be taken inline with the client connection, or with 
every update for this to work.  It might mean that some 
extra updates are sent when they don't need to be, but at 
least they will be accurate.

I know in the past similar ideas have been passed around, 
but they typically relied on the server keeping track of the 
state of each client.  Instead, here we are talking about 
clients keeping track of state for a particular server.  
Clients already store info about remotes.

A very neat idea indeed, thanks!

-Martin

-- 
The Qualcomm Innovation Center, Inc. is a member of Code 
Aurora Forum, hosted by The Linux Foundation