Importing non-version controlled bits and pieces to Git

Importing non-version controlled bits and pieces to Git

[Peter Karlsson]

Hi!

I am looking at moving a project that has not been version-controlled
in the regular sense into Git. I have found it to currently consist of
40 directories with copies of the set of files in various shapes.

I have reconstructed something that looks like a probable revision tree
for these 40 directories, and I want to put this into Git so that I can
examine what changes have been made where, and merge the various
versions back together so that there can be one version with all the
various fixes.

I was looking for something that could do this for me, but found nothing
mentioned on the Wiki, and my Google karma was not good enough to find
something, either. The closest thing I know of is the import-tars.perl
script that I have used before, when importing something with a
(near-)linear history, but this is too far off from a linear history to
try to tar up the directories and import that way, I believe.

Has anyone written such an import tool, or should I start hacking?

-- 
\\// Peter, digital archeologist

Re: Importing non-version controlled bits and pieces to Git

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Fri, Jun 20, 2008 at 10:42:22AM +0100, Peter Karlsson <peter@softwolves.pp.se> wrote:
> I am looking at moving a project that has not been version-controlled
> in the regular sense into Git. I have found it to currently consist of
> 40 directories with copies of the set of files in various shapes.
> 
> I have reconstructed something that looks like a probable revision tree
> for these 40 directories, and I want to put this into Git so that I can
> examine what changes have been made where, and merge the various
> versions back together so that there can be one version with all the
> various fixes.

Maybe contrib/fast-import/import-tars.perl?

It imports tar.{gz,bz2,Z} files, but you can tar up your trees or it can
just serve as an example and you can write your own script that does the
same with directories.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Importing non-version controlled bits and pieces to Git

[Johannes Schindelin]

Hi,

On Fri, 20 Jun 2008, Miklos Vajna wrote:

> On Fri, Jun 20, 2008 at 10:42:22AM +0100, Peter Karlsson <peter@softwolves.pp.se> wrote:
> > I am looking at moving a project that has not been version-controlled 
> > in the regular sense into Git. I have found it to currently consist of 
> > 40 directories with copies of the set of files in various shapes.
> > 
> > I have reconstructed something that looks like a probable revision 
> > tree for these 40 directories, and I want to put this into Git so that 
> > I can examine what changes have been made where, and merge the various 
> > versions back together so that there can be one version with all the 
> > various fixes.
> 
> Maybe contrib/fast-import/import-tars.perl?

BTW this is a relatively simple script showing how to import commits into 
a Git repository using git-fast-import.

You should be able to use git-fast-import from a simple script yourself; 
if you would like to see an example how to feed fast-import, just export a 
few commits with "git fast-export HEAD~5..".

Hth,
Dscho

Re: Importing non-version controlled bits and pieces to Git

[Peter Karlsson]

Johannes Schindelin:

> On Fri, 20 Jun 2008, Miklos Vajna wrote:
> > Maybe contrib/fast-import/import-tars.perl?

As I noted, I have used that before, and it works best with linear
histories, even if I managed to hack it to create a branch for a
project that had a simple branching structure (an update was created
for version n.x after version n+1.x was released).

> You should be able to use git-fast-import from a simple script
> yourself; if you would like to see an example how to feed
> fast-import, just export a few commits with "git fast-export
> HEAD~5..".

Yeah, that is what I am currently planning on doing. I was just trying
to be lazy and see if someone already had solved the problem for me :-)


One complicating factor is that I am working with DOS/Windows source
code, so all text files have CRLF line-endings, which means converting
before feeding to fast-import, or manually doing "git add" with
core.autocrlf == true. fast-import can't be set up to do CRLF
auto-conversion, can it?

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Importing non-version controlled bits and pieces to Git

[Johannes Sixt]

Peter Karlsson schrieb:
> One complicating factor is that I am working with DOS/Windows source
> code, so all text files have CRLF line-endings, which means converting
> before feeding to fast-import, or manually doing "git add" with
> core.autocrlf == true. fast-import can't be set up to do CRLF
> auto-conversion, can it?

No, but you can put `tr -d '\015'` in the pipeline in front of
fast-import. (Unless, of course, you also have to feed binary data.)

-- Hannes

Re: Importing non-version controlled bits and pieces to Git

[Johannes Schindelin]

Hi,

On Fri, 20 Jun 2008, Peter Karlsson wrote:

> One complicating factor is that I am working with DOS/Windows source 
> code, so all text files have CRLF line-endings, which means converting 
> before feeding to fast-import, or manually doing "git add" with 
> core.autocrlf == true. fast-import can't be set up to do CRLF 
> auto-conversion, can it?

Why not import everything with a CR/LF, and then use filter-branch with a 
really simple tree-filter?  It is slow, alright, but it is safe, too.

Ciao,
Dscho

Re: Importing non-version controlled bits and pieces to Git

[Peter Karlsson]

Johannes Schindelin:

> Why not import everything with a CR/LF, and then use filter-branch
> with a really simple tree-filter? It is slow, alright, but it is
> safe, too.

That might work. My problem is the non-linearity of the data I want to
import. But I might be able to massage the import-tars output before I
feed it to git-fast-import to describe the history I need it to.

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Importing non-version controlled bits and pieces to Git

[Peter Karlsson]

Peter Karlsson:

> That might work. My problem is the non-linearity of the data I want
> to import. But I might be able to massage the import-tars output
> before I feed it to git-fast-import to describe the history I need it
> to.

I ended up making a script that converted a description file that
described the hierarchy of versions and generated a fast-import file
from it. Good to have so that I could move stuff around a bit to make
up a version tree that looked somewhat reasonable (a lot of
cross-directory copying has been going on here, and no-one knows which
version is which).

I will try to make the script available.

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Importing non-version controlled bits and pieces to Git

[Karl Hasselström]

On 2008-06-23 15:46:11 +0100, Peter Karlsson wrote:

> Peter Karlsson:
>
> > That might work. My problem is the non-linearity of the data I
> > want to import. But I might be able to massage the import-tars
> > output before I feed it to git-fast-import to describe the history
> > I need it to.
>
> I ended up making a script that converted a description file that
> described the hierarchy of versions and generated a fast-import file
> from it. Good to have so that I could move stuff around a bit to
> make up a version tree that looked somewhat reasonable (a lot of
> cross-directory copying has been going on here, and no-one knows
> which version is which).

A bit too late now, but ... another thing you could have done is make
a straight linear import à la import-tars, and then modified the
parentage with grafts until it looked good; and then finally use
filter-branch to make the grafts part of the "real" history.

The advantage of this approach is that you can edit the parentage
almost interactively, which should be the best approach given that you
have a small number of commits and have to guess their relationships.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: Importing non-version controlled bits and pieces to Git

[Peter Karlsson]

Karl Hasselström:

> A bit too late now, but ...

Not really, I'm on to the next project to import :-)

> another thing you could have done is make a straight linear import à
> la import-tars, and then modified the parentage with grafts until it
> looked good; and then finally use filter-branch to make the grafts
> part of the "real" history.

Yeah, I was considering this approach, but then I was half-way
designing the script in my head, plus that the import is so quick that
it was easy enough to re-write the configuration file and re-import to
try something else. I love that aspect of Git -- the preparing of the
file to import takes longer than the actual git-fast-import call...

> The advantage of this approach is that you can edit the parentage
> almost interactively, which should be the best approach given that
> you have a small number of commits and have to guess their
> relationships.

"Almost" being the magic word :-) I achieved more or less the same by
re-importing. Plus that the job of preparing tarballs of everything
would have been some work as well, since I had to choose which files to
import from each directory (and do CRLF transform).

-- 
\\// Peter - http://www.softwolves.pp.se/