History surgery with fast-import (Re: Summer of Code project ideas due this Friday)

[Jonathan Nieder <jrnieder@gmail.com>]

Jakub Narebski wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:

>> 4. filter-branch killer: using fast-import's new features to implement
>>    common filter-branch operations (--subdirectory-filter,
>>    --prune-empty, obliterating certain files) faster.
>
> How it would be different from existing reposurgeon tool by ESR
> (cross-VC thanks to using fast-import format), or git_fast_filter by
> Elijah Newren (more of a library than a ready tool)?

Good question.

. reposurgeon loads the whole history into a convenient format,
  manipulates it, then dumps it as a separate step.  So it is even more
  flexible than filter-branch, but with a large history and especially
  on a small machine it can be slower.

. git_fast_filter is what Thomas mentioned --- a filter to go between
  fast-export and fast-import.

Both involve unpacking all trees.  Which is not a huge expense, mind
you, but there is room to go faster.

What I was suggesting is instead a tool that relies on fast-import to
do the heavy lifting.  So, for example, when you ask to delete
path/to/remove.txt from all commits, it would write:

 commit refs/heads/master
 mark :172
 author A U Thor <author@example.com> ...
 committer ...
 data ...
 from ...
 merge ...
 M 040000 [old tree name for that commit] ""
 D path/to/remove.txt

 commit refs/heads/mater
 mark :173
 ...

The idea is inspired by
http://thread.gmane.org/gmane.comp.version-control.git/158375
I am interested in it because I imagine something like this could be
useful for splitting out branches from a naive import of an svn repo
(and similar surgeries).