Re: [PATCH] Support 64-bit indexes for pack files.

["Shawn O. Pearce" <spearce@spearce.org>]

Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 26 Feb 2007, Shawn O. Pearce wrote:
> > The latter field is to help pack-objects reuse existing packfile
> > data, as today it needs to sort everything on its own on the fly.
> > Having that last field of data will help avoid that, and will keep
> > the index nicely aligned for 64-bit accesses to the offset.
> 
> Wouldn't that later field help the sliding mmap code as well, knowing in 
> advance what storage size a given object has? (I didn't look at the 
> sliding mmap code so I don't know).

Yes, it probably would.
 
> Actually I've been thinking about another format already.
> 
> What about keeping the pack offset as 32 bits like it is today, but for 
> index v2 if the top bit is set then this become an index into another 
> table containing 64-bit offsets as needed.  This way there is no waste 
> of space for most projects where the pack has yet to reach the 2GB limit 
> for many years to come.

Actually Troy's patch tries to do this by using the current format
and only switching to the new one if the packfile exceeds 4 GiB.
Rather smart.

One thought I had here was to expand the fan-out table from 1<<8
entries to 1<<16 entries, then store only the low 18 bytes of
the SHA-1.  We would have another 2 bytes worth of space to store
the offset, pushing our total offset up to 48 bits.

-- 
Shawn.