Re: [PATCH] diff-delta: produce optimal pack data

[Carl Baldwin <cnb@fc.hp.com>]

On Fri, Feb 24, 2006 at 03:02:07PM -0500, Nicolas Pitre wrote:
> Well that is probably a bit tight.  Ideally it should be linear with the 
> size of the data set to process.  If you have 10 files 10MB each it 
> should take about the same time to pack than 10000 files of 10KB each.  
> Of course incrementally packing one additional 10MB file might take more 
> than a second although it is only one file.

Well, I might not have been fair here.  I tried an experiment where I
packed each of the twelve large blob objects explicitly one-by-one using
git-pack-objects.  Incrementally packing each single object was very
fast.  Well under a second per object on my machine.

After the twelve large objects were packed into individual packs the
rest of the packing went very quickly and git v1.2.3's date reuse worked
very well.  This was sort of my attempt at simulating how things would
be if git avoided deltification of each of these large files. I'm sorry
to have been so harsh earlier I just didn't understand that
incrementally packing one-by-one was going to help this much.

This gives me hope that if somehow git were to not attempt to deltify
these objects then performance would be much better than acceptible.

[snip]
> However, if you could let me play with two samples of your big file I'd 
> be grateful.  If so I'd like to make git work well with your data set 
> too which is not that uncommon after all.

I would be happy to do this.  I will probably need to scrub a bit and
make sure that the result shows the same characteristics.  How would you
like me to deliver these files to you?  They are about 25 MB deflated.

Carl

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        RADCAD (R&D CAD)
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -