Re: git merging

Re: git merging

[Linus Torvalds]

[ Git mailing list cc'd, since this is a general question that maybe 
  others have wondered about ]

On Fri, 17 Jun 2005, Jens Axboe wrote:
> 
> btw (and forgive me for just asking you), what is the canonical way to
> kill a branch in case you want to redo it? right now I have a block tree
> with a bunch of brances, and I would like to redo one of the branches
> because I forgot to add the signed-off-by entries in there.

You can just simply remove the .git/refs/heads/branch-name file (obviously 
after having made sure that ".git/HEAD" points to something else), and 
then you can run "git prune", which will remove all objects that aren't 
referenced (directly or indirectly) by anything in ".git/refs/**".

Be very very careful, though. "git prune" is a dangerous script, since it
will remove objects from the object store. If you had something that you
don't have a proper reference for, or if you give "git prune" a list of
references manually and it's not complete, "git prune" will happily remove
objects that you didn't mean for it to remove.

The "git prune" script is really trivial, you can see for yourself what 
"git-prune-script" does.

NOTE! It's entirely possible to not actually delete the old branch, but 
just rename it, and use it as a reference while you build up the new 
branch with the same contents (but maybe different commit log messages, or 
maybe just re-ordering the commits in the new version). To rename a 
branch, just rename the ".git/refs/heads/<branch-name>" file to something 
else.

So you may decide to remove (and prune) the old branch only after you've 
re-created a new version of it, for example. That way you can use git 
itself to extract the contents of the branch you want to re-do, which is 
often easier.

Btw, if you have a recent version of "gitk" instealled, you can do

	gitk $(ls .git/revs/heads)

and you'll get a full graphical view of _all_ your branches, which can be 
very instructive.

		Linus

Re: git merging

[Jens Axboe]

On Fri, Jun 17 2005, Linus Torvalds wrote:
> 
> [ Git mailing list cc'd, since this is a general question that maybe 
>   others have wondered about ]
> 
> On Fri, 17 Jun 2005, Jens Axboe wrote:
> > 
> > btw (and forgive me for just asking you), what is the canonical way to
> > kill a branch in case you want to redo it? right now I have a block tree
> > with a bunch of brances, and I would like to redo one of the branches
> > because I forgot to add the signed-off-by entries in there.
> 
> You can just simply remove the .git/refs/heads/branch-name file (obviously 
> after having made sure that ".git/HEAD" points to something else), and 
> then you can run "git prune", which will remove all objects that aren't 
> referenced (directly or indirectly) by anything in ".git/refs/**".
> 
> Be very very careful, though. "git prune" is a dangerous script, since it
> will remove objects from the object store. If you had something that you
> don't have a proper reference for, or if you give "git prune" a list of
> references manually and it's not complete, "git prune" will happily remove
> objects that you didn't mean for it to remove.
> 
> The "git prune" script is really trivial, you can see for yourself what 
> "git-prune-script" does.

That worked! Thanks.

> NOTE! It's entirely possible to not actually delete the old branch, but 
> just rename it, and use it as a reference while you build up the new 
> branch with the same contents (but maybe different commit log messages, or 
> maybe just re-ordering the commits in the new version). To rename a 
> branch, just rename the ".git/refs/heads/<branch-name>" file to something 
> else.
> 
> So you may decide to remove (and prune) the old branch only after you've 
> re-created a new version of it, for example. That way you can use git 
> itself to extract the contents of the branch you want to re-do, which is 
> often easier.

Yeah that's very handy. I love the way the branches work, with just the
sha in the file name. So easy to manipulate.

-- 
Jens Axboe

Re: git merging

[Jeff Garzik]

Linus Torvalds wrote:
> The "git prune" script is really trivial, you can see for yourself what 
> "git-prune-script" does.


I've been too lazy to report this, but...

last time I tried it, git-prune-script ate valid objects, for my 
multi-branch trees.  This was 100% reproducible.  Luckily, the objects 
it ate were all in the main branch, so an rsync-without-delete from 
linux-2.6.git/objects would restore the missing data.

This was verified by repeated runs of
	git-fsck-cache		# OK
	git-prune-script
	git-fsck-cache		# spits out tons of errors
	rsync ...		# restore deleted objects
	git-fsck-cache		# OK

I use hardlinks on the local disk, between several multi-branch repos, 
if that matters.

	Jeff

Re: git merging

[Linus Torvalds]

On Fri, 17 Jun 2005, Jeff Garzik wrote:
> 
> I've been too lazy to report this, but...
> 
> last time I tried it, git-prune-script ate valid objects, for my 
> multi-branch trees.  This was 100% reproducible.

Ok. The most likely reason is that your main branch is not a symlink to 
".git/refs/***" at all, but just a regular ".git/HEAD" file.

"git prune" will by default _only_ look for references in .git/refs/, and
if you have any references anywhere else, you need to do all your refs by
hand.

There might be a bug in the ref-lookup code, of course, but it's more 
likely that your refs just aren't where fsck expects them.

		Linus

Re: git merging

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

Linus Torvalds wrote:
> Ok. The most likely reason is that your main branch is not a symlink to 
> ".git/refs/***" at all, but just a regular ".git/HEAD" file.

This is definitely not the case; my .git/HEAD is _always_ a symlink.

My git-switch-tree script, attached, demonstrates how .git/HEAD symlink 
is retargetted to the specified branch.  My workflow depends on 
.git/HEAD being a symlink.

I'll see if I can reproduce with the latest git.

	Jeff



[-- Attachment #2: git-switch-tree --]
[-- Type: text/plain, Size: 232 bytes --]

#!/bin/sh

if [ "x$1" != "x" ]
then
	if [ ! -f .git/refs/heads/$1 ]
	then
		echo Branch $1 not found.
		exit 1
	fi

	( cd .git && rm -f HEAD && ln -s refs/heads/$1 HEAD )
fi

git-read-tree -m HEAD && git-checkout-cache -q -f -u -a

Re: git merging

[Linus Torvalds]

On Fri, 17 Jun 2005, Jeff Garzik wrote:
> 
> This is definitely not the case; my .git/HEAD is _always_ a symlink.

Ok. Are you sure that you gave the same arguments (or rather, lack of
arguments) to both fsck and "git prune"? The thing is, they are both
really the same thing, so I'm pretty surprised. If git prune says 
something is unreachable, then git-fsck-cache shouldn't complain about it 
being gone, because one just depends on the other..

> My git-switch-tree script, attached, demonstrates how .git/HEAD symlink 
> is retargetted to the specified branch.  My workflow depends on 
> .git/HEAD being a symlink.

Btw, you can now do the same thing more safely and guarantee that it 
doesn't overwrite any old information by using

	git-read-tree -m -u <old-head> <new-head>

which basically switches from "old" to "new", and verifies that all the 
old index contents were valid in "old-head", and that any file that was 
dirty is not different in "new-head".

Your old script would silently overwrite any dirty state in your working
directory, and drop anything that you had done a git-update-cache on but
not committed.

Now, you may have _depended_ on that behaviour as a way to just reset the 
tree to a known state, but if so, I'd suggest using

	git-read-tree --reset HEAD && git-checkout-cache -q -f -u -a

for that instead (which will also throw away any partial merges).

So for the "switch" case, you might make it be something like

	if [ ! -f .git/refs/heads/$1 ]
	then
		echo "Branch '$1' not found"
		exit 1;
	fi
	git-read-tree -m -u HEAD "heads/$1" && ln -sf refs/heads/$1 .git/HEAD

which should do the right thing.

Totally untested, of course ;)

[ But the two-tree read-tree is definitely not untested: this is how we 
  do a safe "fast forward" in the git-resolve-script, which really ends up 
  being the exact same thing: it "switches" from one head to another ].

		Linus

Re: git merging

[Jens Axboe]

On Fri, Jun 17 2005, Linus Torvalds wrote:
> 
> 
> On Fri, 17 Jun 2005, Jeff Garzik wrote:
> > 
> > This is definitely not the case; my .git/HEAD is _always_ a symlink.
> 
> Ok. Are you sure that you gave the same arguments (or rather, lack of
> arguments) to both fsck and "git prune"? The thing is, they are both
> really the same thing, so I'm pretty surprised. If git prune says 
> something is unreachable, then git-fsck-cache shouldn't complain about it 
> being gone, because one just depends on the other..

Seeing something weird here as well, my setup is similar to Jeffs in
that I have branches in refs/heads/* and HEAD a symlink to the active
one.

I committed a bad patch, so wanting to reverse that change I did an

# echo sha_of_previous_commit > .git/HEAD
# git-read-tree -m HEAD && git-checkout-cache -q -f -u -a

which, if I understand correctly, should put me back where I was before.
So continuing commits, checking the tree now gives me:

axboe@nelson:[.]l/git/linux-2.6-block.git $ git prune
error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a
bad sha1 entry '5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c'
axboe@nelson:[.]l/git/linux-2.6-block.git $ git-fsck-cache 
error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a
bad object in tag 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
bad sha1 entry '5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c'

Running git prune again gives me the same output. What is wrong?

-- 
Jens Axboe

Re: git merging

[Matthias Urlichs]

Hi, Jens Axboe wrote:

> error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a
> bad object in tag 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c

What does "git-cat-file -t" say on these two -- are they botth vald commit
objects?

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
He that is giddy thinks the world turns round.
		-- William Shakespeare, "The Taming of the Shrew"

Re: git merging

[Jens Axboe]

On Mon, Jun 20 2005, Matthias Urlichs wrote:
> Hi, Jens Axboe wrote:
> 
> > error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a
> > bad object in tag 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
> 
> What does "git-cat-file -t" say on these two -- are they botth vald commit
> objects?

axboe@nelson:[.]l/git/linux-2.6-block.git $ git-cat-file -t c39ae07f393806ccf406ef966e9a15afc43cc36a
error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a
fatal: git-cat-file c39ae07f393806ccf406ef966e9a15afc43cc36a: bad file
axboe@nelson:[.]l/git/linux-2.6-block.git $ git-cat-file -t 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
tag

-- 
Jens Axboe

Re: git merging

[Linus Torvalds]

[ Daniel put on the To: list to see if he can confirm or deny my theory ]

On Mon, 20 Jun 2005, Jens Axboe wrote:
> 
> axboe@nelson:[.]l/git/linux-2.6-block.git $ git prune
> error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a

That's the 2.6.11 "tree" object.

> bad sha1 entry '5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c'
> axboe@nelson:[.]l/git/linux-2.6-block.git $ git-fsck-cache 
> error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a
> bad object in tag 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
> bad sha1 entry '5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c'
> 
> Running git prune again gives me the same output. What is wrong?

Very interesting. You have my "v2.6.11-tree" tag file pointing the 2.6.11
tree object, but you seem to not have that 2.6.11 tree itself.

Actually, judging from the fact that you got this error _during_ the
prune, maybe you never had it in that repository in the first place?

In particular, if you don't use "rsync", but instead use one of the
"optimized pull" things to create a repository (ie git-http-pull or
git-ssh-pull), I think your newly pulled tree will always miss anything
that isn't a head. And the 2.6.11 tree is a special case: it's a pure
"tree" object without any commit at all pointing to it, just a single tag
that points directly to the tree.

Anyway, the fact that you're missing the original 2.6.11 tree doesn't
really matter, so it's not a huge deal per se. You can re-populate it with
an "rsync -avz --ignore-existing", but clearly something seems to be
wrong.

Daniel: I think git-ssh-pull has problems with any refs that aren't
commits. I also noticed earlier that I can't use git-ssh-push -w to write
a tag - it is apparently unhappy because it would write a "tag" object,
and it only wants to write references to "commit" objects. The same thing
might be true about the special "tag to a tree" object?

Jeff, do you remember (did you check?) if your prune problems might also
have been about the tag to a tree?

		Linus

Re: git merging

[Daniel Barkalow]

On Mon, 20 Jun 2005, Linus Torvalds wrote:

> Daniel: I think git-ssh-pull has problems with any refs that aren't
> commits. I also noticed earlier that I can't use git-ssh-push -w to write
> a tag - it is apparently unhappy because it would write a "tag" object,
> and it only wants to write references to "commit" objects. The same thing
> might be true about the special "tag to a tree" object?

You're correct; it's actually that pull.c has never gotten extended to
look at the object you specify, determine what it is, and then behave
appropriately. Basically, it needs a "process_unknown()" which does the
appropriate thing, and pull() needs to call that instead of
process_commit(). It should also get a "process_tag()" which handles tags
(calling process_unknown() on their objects). I should be able to do it
Tuesday evening or Wednesday night, if nobody else gets it before then.

	-Daniel
*This .sig left intentionally blank*

Re: git merging

[Matthias Urlichs]

Hi, Daniel Barkalow wrote:

> Basically, it needs a "process_unknown()" which does the
> appropriate thing, and pull() needs to call that instead of
> process_commit().

It also needs a request queue.

The beauty of rsync() is that it doesn't depend on a low-latency wire to
work efficiently. Right now, pull() needs one round trip per object, and
that's way too much.

I hope to have some free time for that this week, but no promises. :-/

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
Creativity varies inversely with the number of cooks involved with the
broth.
					-- Bernice Fitz-Gibbon

Re: git merging

[Jens Axboe]

On Mon, Jun 20 2005, Linus Torvalds wrote:
> 
> [ Daniel put on the To: list to see if he can confirm or deny my theory ]
> 
> On Mon, 20 Jun 2005, Jens Axboe wrote:
> > 
> > axboe@nelson:[.]l/git/linux-2.6-block.git $ git prune
> > error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a
> 
> That's the 2.6.11 "tree" object.
> 
> > bad sha1 entry '5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c'
> > axboe@nelson:[.]l/git/linux-2.6-block.git $ git-fsck-cache 
> > error: cannot map sha1 file c39ae07f393806ccf406ef966e9a15afc43cc36a
> > bad object in tag 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
> > bad sha1 entry '5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c'
> > 
> > Running git prune again gives me the same output. What is wrong?
> 
> Very interesting. You have my "v2.6.11-tree" tag file pointing the 2.6.11
> tree object, but you seem to not have that 2.6.11 tree itself.
> 
> Actually, judging from the fact that you got this error _during_ the
> prune, maybe you never had it in that repository in the first place?

I can't tell exactly, but I'm fairly sure this is a new error. I have
two git trees locally - one which is just an rsync of your kernel.org
tree, the other which is based off that (with a clone of the fs dir) and
has multiple branches for developments. The main tree did not have this
problem, while the development tree does/did.

> In particular, if you don't use "rsync", but instead use one of the
> "optimized pull" things to create a repository (ie git-http-pull or
> git-ssh-pull), I think your newly pulled tree will always miss anything
> that isn't a head. And the 2.6.11 tree is a special case: it's a pure
> "tree" object without any commit at all pointing to it, just a single tag
> that points directly to the tree.
> 
> Anyway, the fact that you're missing the original 2.6.11 tree doesn't
> really matter, so it's not a huge deal per se. You can re-populate it with
> an "rsync -avz --ignore-existing", but clearly something seems to be
> wrong.

I pulled with rsync manually from kernel.org, and that did fix things up
for me. The main tree is rsync'ed, but the development tree gets the
changes with /opt/kernel/git/linux-2.6/.git/ as the url given to
git-pull-script.

-- 
Jens Axboe

Re: git merging

[Linus Torvalds]

On Mon, 20 Jun 2005, Jens Axboe wrote:
> 
> I pulled with rsync manually from kernel.org, and that did fix things up
> for me. The main tree is rsync'ed, but the development tree gets the
> changes with /opt/kernel/git/linux-2.6/.git/ as the url given to
> git-pull-script.

Ok, that explains it. Since you're using a regular local filename, the
pull will be using "git-local-pull", which will only fetch the objects
directly needed. And doesn't understand the tag-to-tree thing, so doesn't 
fetch the tree (or possibly you just copied the tags by hand totally 
outside of the regular pull?)

		Linus

Re: git merging

[Jens Axboe]

On Mon, Jun 20 2005, Linus Torvalds wrote:
> 
> 
> On Mon, 20 Jun 2005, Jens Axboe wrote:
> > 
> > I pulled with rsync manually from kernel.org, and that did fix things up
> > for me. The main tree is rsync'ed, but the development tree gets the
> > changes with /opt/kernel/git/linux-2.6/.git/ as the url given to
> > git-pull-script.
> 
> Ok, that explains it. Since you're using a regular local filename, the
> pull will be using "git-local-pull", which will only fetch the objects
> directly needed. And doesn't understand the tag-to-tree thing, so doesn't 
> fetch the tree (or possibly you just copied the tags by hand totally 
> outside of the regular pull?)

Isn't that a little 'end user' confusing from a usability point of view,
that it behaves differently depending on which pull script it ends up
using in the end?

I guess I can just always use rsync even for local trees. And use it
directly, so I always have everything :)

-- 
Jens Axboe

Re: git merging

[Linus Torvalds]

On Tue, 21 Jun 2005, Jens Axboe wrote:
> 
> Isn't that a little 'end user' confusing from a usability point of view,
> that it behaves differently depending on which pull script it ends up
> using in the end?

Yes, I'm not claiming it's good, I'm just claiming it explains what you 
see ;)

So it's a bug, but I'm hoping somebody else will fix it. I've not actually
worked on that "intelligent pull" functionality at all (aside from making
the ssh-push thing work for my limited use).

			Linus