Re: GSoC - Designing a faster index format

[elton sky <eltonsky9404@gmail.com>]

Thanks Shawn,

> Or use LevelDb[2]. Its BSD license. Uses an immutable file format, but
> writes updates to new smaller files and eventually collapses
> everything back together into a bigger file. This can be a
> dramatically simpler approach than dealing with your own free block
> system inside of a single file. Its only real downside is needing to
> periodically pay a penalty to rewrite the whole index. But this
> rewrite is going to be faster than the time it takes to rewrite the
> pack files for the same repository, which git gc or git repack
> handles. So I don't think its actually a problem for the index.
>
> You might even be able to take a two level approach to compacting the
> LevelDb database (or something like it). In a minor compaction you
> compact all of the files except the huge base file, leaving you with 2
> files. A huge base file that contains the first tree the user checked
> out, and a second smaller file containing any differences they have
> since the initial checkout (this may just be updated stat data for a
> handful of files that differed across two branches as they switched
> back and forth). During a git gc or git repack, add a new stage to
> collapse the base file and everything else into a single new base file
> as a major compaction.
>
> [2] http://code.google.com/p/leveldb/

I don't know leveldb, but like to have a look.
Just realize this solution is kinda popular. HDFS also uses the
similar image file with edit file format for its file block index.