Re: git annotate runs out of memory

[Davide Libenzi <davidel@xmailserver.org>]

On Tue, 11 Dec 2007, Linus Torvalds wrote:

> On Tue, 11 Dec 2007, Linus Torvalds wrote:
> > 
> > and while I suspect xdiff could be optimized a bit more for the cases 
> > where we have no changes at the end, that's beyond my skills.
> 
> Ok, I lied.
> 
> Nothing is beyond my skills. My mad k0der skillz are unbeatable.
> 
> This speeds up git-blame on ChangeLog-style files by a big amount, by just 
> ignoring the common end that we don't care about, since we don't want any 
> context anyway at that point. So I now get:
> 
> 	[torvalds@woody gcc]$ time git blame gcc/ChangeLog > /dev/null
> 
> 	real    0m7.031s
> 	user    0m6.852s
> 	sys     0m0.180s
> 
> which seems quite reasonable, and is about three times faster than trying 
> to diff those big files.
> 
> Davide: this really _does_ make a huge difference. Maybe xdiff itself 
> should do this optimization on its own, rather than have the caller hack 
> around the fact that xdiff doesn't handle this common case all that well?

I didn't follow the thread, but I can guess from the subject that this is 
about memory, isn't it?
Libxdiff already has a xdl_trim_ends() that strips all the common 
beginning and ending records, but at that point files are already loaded.
Since libxdiff works with memory files in order to keep any sort of 
system dependency out of the window, so the optimization would be 
useless on libxdiff side. This because the user would have to have 
already the file loaded in memory, to pass it to libxdiff.
If this is really about memory, this better be kept on the libxdiff caller 
side, so that it can avoid loading the terminal file sections altogether.
About your code, you may want to have an extend-till-next-eol code after 
the trimming part, since the last line may be used for context in the 
diffs.




- Davide