[RFC] Applying a graft to a tree and "rippling" the changes through the history

[RFC] Applying a graft to a tree and "rippling" the changes through the history

[Ryan Anderson]

[-- Attachment #1: Type: text/plain, Size: 3257 bytes --]


I've written a tool that will take a single commit, add it as a parent
of another commit, and recreate the history above that second commit in
a fully compatible manner.

This is mostly useful for creating a fully merged-up repository of the
Linux Historical tree, and the current working tree.

I run this with /graft-ripple.pl linux-history.tmp/ linus origin

Where "origin" is the branch the historical repository is on, and
"linus" is the branch the current repository is on.

Note: This does not end up fixing up HEAD or any branches, it just pulls
all the objects together and recreates the full history.

GPLv2, but I'll redo with a proper patch, signed-off-by, command line
options and help and docs if anyone else feels this is useful as a
general tool.

========= cut here =============

#!/usr/bin/perl

use strict;
use warnings;

use Data::Dumper;
use IPC::Open2;


sub git_commit_tree {
        my ($tree,$comments,@parents) = @_;

        my @cparents;
        foreach my $p (@parents) {
                push @cparents,"-p",$p;
        }

        my $pid = open2(*Reader, *Writer,
		"git-commit-tree",$tree,@cparents);
        print Writer $comments;
        close(Writer);
        my $commit = <Reader>;

        waitpid $pid, 0;
        close(Reader);

        chomp $commit;

        return $commit;
}

chdir($ARGV[0]);
open(GRL,"-|","git-rev-list","--parents",$ARGV[1])
	or die "Failed to run git-rev-list: " . $!;

my %csets;
my @revs;
while(<GRL>) {
	chomp;
	my ($commit,@parents) = split /\s+/;
	$csets{$commit}{parents} = \@parents;
	push @revs, $commit;

	open(GCF,"-|","git-cat-file","commit",$commit)
		or die "Failed to open git-cat-file: " . $!;

	my $in_comments = 0;
	while(<GCF>) {
		chomp;
		if ($in_comments) {
			$csets{$commit}{comments} .= $_ . "\n";

		} elsif (m/^tree (.+)$/) {
			$csets{$commit}{tree} = $1;
			#printf("tree = %s\n",$1);

		} elsif (m/^parent (.+)$/) {
			# Do nothing, we already got
			# the parents from rev-list.

		} elsif (m/^(author|committer) (.*) <(.*)> (.*)$/) {
			#printf("%s = %s <%s> at %s\n",$1, $2,$3,$4);
			@{$csets{$commit}{$1}}{qw(name email datetime)}
				= ($2,$3,$4);

		} elsif (length == 0) {
			$in_comments = 1;
			$csets{$commit}{comments} = "";
			next;
		}
	}
	close(GCF);

}
close(GRL);

@revs = reverse @revs;
push @{$csets{$revs[0]}{parents}},$ARGV[2];

my %newcsets;
foreach my $old (@revs) {
	printf("Processing commit %s\n",$old);
        $ENV{GIT_AUTHOR_EMAIL} = $csets{$old}{author}{email};
        $ENV{GIT_AUTHOR_NAME} = $csets{$old}{author}{name};
        $ENV{GIT_AUTHOR_DATE} = $csets{$old}{author}{datetime};
        $ENV{GIT_COMMITTER_DATE} = $csets{$old}{committer}{datetime};
        $ENV{GIT_COMMITTER_EMAIL} = $csets{$old}{committer}{email};
        $ENV{GIT_COMMITTER_NAME} = $csets{$old}{committer}{name};

	my @parents = @{$csets{$old}{parents}};
	foreach my $p (@{$csets{$old}{parents}}) {
		if (exists $newcsets{$p}) {
			push @parents, $newcsets{$p}
				if exists $newcsets{$p};
			printf("Found new csetid %s for %s\n",
				$newcsets{$p},$p);
		}
	}
        my $commit = git_commit_tree($csets{$old}{tree},
		$csets{$old}{comments},@parents);
	$newcsets{$old} = $commit;
        printf("Commit for version %s is %s\n",$old,$newcsets{$old});
}


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [RFC] Applying a graft to a tree and "rippling" the changes through the history

[Randal L. Schwartz]

>>>>> "Ryan" == Ryan Anderson <ryan@michonline.com> writes:

Ryan> chdir($ARGV[0]);

That's dangerous without an "or-die".  Being in the wrong directory
before you do a lot of edits is a good way to bust your disk. :)

Ryan> 	my ($commit,@parents) = split /\s+/;

split with no args splits $_ on whitespace, tossing leading whitespace,
just in case they ever put whitespace indentation ahead.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: [RFC] Applying a graft to a tree and "rippling" the changes through the history

[Junio C Hamano]

Ryan Anderson <ryan@michonline.com> writes:

> I've written a tool that will take a single commit, add it as a parent
> of another commit, and recreate the history above that second commit in
> a fully compatible manner.

I think the procedure is reproducible, which is a very nice
property to have for a tool like this, but I am not sure what
you mean by "in a fully compatible manner".  What are you
compatible with?

Also another rhetorical, tongue-in-cheek question.  What is your
plan to ripple the graft through to update signed tags?  ;-)

Re: [RFC] Applying a graft to a tree and "rippling" the changes through the history

[Ryan Anderson]

[-- Attachment #1: Type: text/plain, Size: 1805 bytes --]

Junio C Hamano wrote:
> Ryan Anderson <ryan@michonline.com> writes:
> 
> 
>>I've written a tool that will take a single commit, add it as a parent
>>of another commit, and recreate the history above that second commit in
>>a fully compatible manner.
> 
> 
> I think the procedure is reproducible, which is a very nice
> property to have for a tool like this, but I am not sure what
> you mean by "in a fully compatible manner".  What are you
> compatible with?

Well, what I meant was, "It creates a history that is purely a superset
of the old history, so merges should work cleanly from the pre-graft
subhistory to the fully merged history."

But clearly I was too ... terse.

IOW, this should work perfectly, assuming neither tree has been pulled
into since the history was merged into historical-graft tree:

$ cd linux-head
$ git branch -b ryan-hacking HEAD
$ quilt push -a
$ git commit -a -m "Apply quilt tree"

$ cd ../linux-historical-graft/
$ git pull ../linux-historical-graft/

> Also another rhetorical, tongue-in-cheek question.  What is your
> plan to ripple the graft through to update signed tags?  ;-)

:) Well, since I can't resist answering your rhetorical question:

They signed a specific DAG.  I'm providing a richer, more complete DAG
that is a pure-superset of the one they signed.  It is not, however,
equivalent, so their signature is not related to the superset DAG I have
created.  In practice, however, I don't expect that any tag-signers
would state that there is a meaningful difference between the two DAGs,
from the perspective of their signature.

FYI - I don't think merging the trees like this is a good idea, from the
perspective of something like gitk - gitk took long enough to startup
and display something on my merged tree here that I gave up and killed
it off.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [RFC] Applying a graft to a tree and "rippling" the changes through the history

[Matthias Urlichs]

Hi, Junio C Hamano wrote:

> Ryan Anderson <ryan@michonline.com> writes:
> 
>> I've written a tool that will take a single commit, add it as a parent
>> of another commit, and recreate the history above that second commit in
>> a fully compatible manner.
> 
You're not the only one. My tool is different, however, in that it accepts
a list of old=>new commits. I have used it successfully to re-graft my
changes from a CVS->GIT tree to the corresponding CVS->SVN->GIT tree
(which cannot be identical, because (a) SVN's timestamps are not accurate
enough and (b) SVN's CVS import is more accurate in reproducing CVS
archives than cvsps can be).

> Also another rhetorical, tongue-in-cheek question.  What is your
> plan to ripple the graft through to update signed tags?  ;-)

I'd suggest adding the capability of grafting something onto a tag...


#!/usr/bin/python

# This is a simple script which clones a git subtree to another.

import sys,re,optparse,os,subprocess

parser = optparse.OptionParser("corresponding_file [ old_commit ]", conflict_handler="resolve", description="""\
Transfer a list of commits from one repository to another.

The first argument is a list of SHA1 entries of the form
	old new
It lists which commits are "the same".

old_commit and those of its parents which are not listed in the
corresponding_file are copied, i.e. new commit objects are created.
Their SHA1s are added to the file so that you may repeat the process
with other commits, or run it incrementally.

""")
parser.add_option("-h","--help","-?", action="help",
                    help="Print this help message and exit")
parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
                    help="Report progress")

(options, args) = parser.parse_args()

if len(args) < 1 or len(args) > 3:
	parser.error("requires one to three arguments")


def end(p):
	try:
		retcode = p.wait()
	except OSError,e:
		print >>sys.stderr, "git-rev-list failed:", e
	else:
		if retcode < 0:
			print >>sys.stderr, "git-rev-list was terminated by signal", -retcode
			sys.exit(1)
		elif retcode > 0:
			print >>sys.stderr, "git-rev-list exited with non-zero exit code", retcode
			sys.exit(1)


re_cmt = re.compile(r'\s*#.*')
corr = {}
for l in open(args[0]):
	l = re_cmt.sub("",l).strip()
	if l == "": continue
	try:
		a,b = l.split()
	except ValueError:
		continue
	if len(a) != 40: continue
	if len(b) != 40: continue
	corr[a]=b

corrf=open(args[0],"a")

if len(args) >= 2:
	srctag = args[1]
else:
	srctag = "HEAD"

srcrepo = os.path.curdir

commits=[]
cmd = ["git-rev-list",srctag]
for k in corr.iterkeys():
	cmd.append("^"+k)
if options.verbose:
	print cmd

p=subprocess.Popen(cmd, stdout=subprocess.PIPE)
for l in p.stdout:
	l = l.strip()
	commits.append(l)
end(p)

while len(commits):
	c = commits.pop()
	if c in corr: continue
	if options.verbose:
		print "Processing:",c

	p=subprocess.Popen(["git-cat-file","commit",c], stdout=subprocess.PIPE)
	qf=os.tempnam()
	try:
		q=open(qf,"w")
		nx=False
		for l in p.stdout:
			if nx:
				q.write(l)
				continue
			l = l.strip()
			if l == "":
				nx=True
				q.write("\n")
				continue
			a,b = l.split(" ",1)
			if a == "parent":
				b = corr[b]
			print >>q,a,b
		q.close()
		q=subprocess.Popen(["git-hash-object","-w","-t","commit",qf], stdout=subprocess.PIPE)
		d = q.stdout.read().strip()
		end(q)
	finally:
		os.unlink(qf)
	end(p)
	corr[c] = d
	print >>corrf, c,d
	if options.verbose:
		print c,d,l

# OK, everything is done.
corrf.close()
print d

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
It's not what you know or what you do, it's who you know.