Re: fetching packs and storing them as packs

[Shawn Pearce <spearce@spearce.org>]

Eran Tromer <git2eran@tromer.org> wrote:
> > Unfortunately I don't have a solution.  I tried to come up with
> > one but didn't.  :-)
> 
> Here's one way to do it.
> Change git-repack to follow references under $GIT_DIR/tmp/refs/ too.
> To receive or fetch a pack:
> 1. Add references to the new heads in
>    `mktemp $GIT_DIR/tmp/refs/XXXXXX`.
> 2. Put the new .pack under $GIT_DIR/objects/pack/.
> 3. Put the new .idx under $GIT_DIR/objects/pack/.
> 4. Update the relevant heads under $GIT_DIR/refs/.
> 5. Delete the references from step 1.
> 
> This is repack-safe and never corrupts the repo. The worst-case failure
> mode is if you die before cleaning the refs from $GIT_DIR/tmp/refs. That
> may mean some packed objects will never be removed by "repack -a -d"
> even if they lose all references from $GIT_DIR/refs, so do "tmpwatch -m
> 240 $GIT_DIR/tmp/refs" to take care of that.

That was actually my (and also Sean's) solution.  Except I would
put the temporary refs as "$GIT_DIR/refs/ref_XXXXXX" as this is
less code to change and its consistent with how temporary loose
objects are created.

Unfortunately it does not completely work.

What happens when the incoming pack (steps #2 and #3) takes 15
minutes to upload (slow ADSL modem, lots of objects) and the
background repack process sees those temporary refs and starts
trying to include those objects?  It can't walk the DAG that those
refs point at because the objects aren't in the current repository.

From what I know of that code the pack-objects process will fail to
find the object pointed at by the ref, rescan the packs directory,
find no new packs, look for the object again, and abort over the
"corruption".

OK so the repository won't get corrupted but the repack would be
forced to abort.


Another issue I just thought about tonight is we may need a
count-packs utility that like count-objects lists the number
of active packs and their total size.  If we start hanging onto
every pack we receive over the wire the pack directory is going to
grow pretty fast and we'll need a way to tell us when its time to
`repack -a -d`.

--