Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
>  On Wed, Apr 16, 2008 at 11:45:39AM +0200, Brian Foster wrote:
>  >  Brian Foster <brian.foster@innova-card.com> writes:
>  >  >  I've recently inherited a bare git repository,
>  >  >  which, as far as I can tell (I'm something of a
>  >  >  newbie with git), seems Ok [ but clones are not ].
>
>  I suspect your original git repository has info/grafts

hi Dmitry,

 bingo!  YES, it does:

	$ cat info/grafts
	8700854c41a40d333e90104971c3abbbcf082e57
34757f56bd7cb007bbb48ce064a2cd7e67c32428
	c87c46fe892211f8aa4fd363ccff4f667a9aaf7d
d560b7a27640cdf9ff1e6a30ae161dc6144618bd
	dd3f3c0636cfd50719c706b030db5473b0270add
e6cfa744e0434f9b700cd8a3780dcf9235ed10e3
	e5a60f1636cceac33777bb8098a0b7a4a136a56c
b38536a9a67b8d6b94e51860195529481703286b
	0ff75b3afff6fb306bef221bf1823ccf5ffc568b
01d4f94e5e5e65058234256241c01a6caf1c5feb
	fb5967688f7b464421cff28f266b64ad2a313a9e
00fe1cce8a053ecf9d8ba160991f28574fb11f63
	2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
	$

 now, showing that I am indeed essentially a git newbie,
 WTF is a "graft"?  I don't recall running into the term.
 I shall, of course, immediately start trying to find some
 info blah blah ..., but pointers/explanations would help.

 the goal is to put things into a sane state so any new
 clones are healthy.  there's only one(?) existing clone,
 which may or may not be(? become?) an issue.

cheers!
	-blf-
-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 01:48:13PM +0200, Brian Foster wrote:
> 
>  now, showing that I am indeed essentially a git newbie,
>  WTF is a "graft"?  I don't recall running into the term.

info/grafts::
	This file records fake commit ancestry information, to
	pretend the set of parents a commit has is different
	from how the commit was actually created.  One record
	per line describes a commit and its fake parents by
	listing their 40-byte hexadecimal object names separated
	by a space and terminated by a newline.

Basically, you use it if you want to modify the history in your
repository. It is often used after importing to Git from some
other repository if you want to make some adjustment to history.
So, you edit info/grafts, then use gitk to verify that you got
what you want, and after that you run git-filter-branch to actually
overwrite the history.

info/grafts should never be used during normal work.


>  the goal is to put things into a sane state so any new
>  clones are healthy.  there's only one(?) existing clone,
>  which may or may not be(? become?) an issue.

If you run git-filter-branch then you'll change "fake" history
to the real one, and after that the repository can be cloned.
Please, make sure that you have a backup before any further changes.

Dmitry

Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Petr Baudis]

On Wed, Apr 16, 2008 at 04:14:24PM +0400, Dmitry Potapov wrote:
> info/grafts should never be used during normal work.

I don't really agree with this advice in general. Grafts can be very
useful especially when migrating to Git - a very good strategy might be
just to start from scratch with your current state, then import your
previous history and graft it. The big advantage is that if you find out
later that the import is screwed up, you can fix that and reimport. The
fact that you don't need to haul 10 years of CVS history around unless
you actually know you need it can be perceived as both advantage and
disadvantage.

The _BAD_ grafts are those that replace list of commit's parents instead
of just appending (typically to an empty one). I didn't actually have
any idea people use grafts in such a twisted and perverse way... ;-)

-- 
				Petr "Pasky" Baudis
Whatever you can do, or dream you can, begin it.
Boldness has genius, power, and magic in it.	-- J. W. von Goethe

Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 04:59:00PM +0200, Petr Baudis wrote:
> On Wed, Apr 16, 2008 at 04:14:24PM +0400, Dmitry Potapov wrote:
> > info/grafts should never be used during normal work.
> 
> I don't really agree with this advice in general. Grafts can be very
> useful especially when migrating to Git

I agree that they can be very useful in the process of migrating, but
I would not recommend to leave them _after_ the migration process is
complete.

> The _BAD_ grafts are those that replace list of commit's parents instead
> of just appending (typically to an empty one). I didn't actually have
> any idea people use grafts in such a twisted and perverse way... ;-)

They are used to replace _BAD_ commits imported from CVS. History in
CVS can be very strange and even with best conversation tools, you may
end up with something not exactly what you want. So, you want to replace
some commits with corrected versions of them. So, you add those commits
to Git repo, "fix" history, and then using gitk check that now you have
exactly what you want. If now the history is okay then you can use
git filter-branch to make those changes real.

Dmitry

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian Foster schrieb:
> Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
>>  I suspect your original git repository has info/grafts
> 
> hi Dmitry,
> 
>  bingo!  YES, it does:
...
>  the goal is to put things into a sane state so any new
>  clones are healthy.  there's only one(?) existing clone,
>  which may or may not be(? become?) an issue.

Just move info/grafts out of the way and you *may* be all set. Don't
delete it - there might be a reason that the file exists.

However!

You might observe one or more of the following:

- Parts of the history are missing. The reason is that the grafts were
used to "splice" histories that are actually independent.

- Previously unkown (to you) parts of the history are now visible. Then
the grafts were used to hide these parts.

- The repository is broken. Then 'git repack -f -d -a' was done while the
grafts were in effect (and obviously the grafts had hidden parts of the
history, which are now no longer available).

Your best bet is to run 'git filter-branch --tag-name-filter cat -- --all'
on *a copy* of your bare repository (with info/grafts in place). (Read the
warning about --tag-name-filter in the man page first.)

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 03:22:04PM +0200, Johannes Sixt wrote:
> Brian Foster schrieb:
> > Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
> >>  I suspect your original git repository has info/grafts
> > 
> > hi Dmitry,
> > 
> >  bingo!  YES, it does:
> ...
> >  the goal is to put things into a sane state so any new
> >  clones are healthy.  there's only one(?) existing clone,
> >  which may or may not be(? become?) an issue.
> 
> Just move info/grafts out of the way and you *may* be all set. Don't
> delete it - there might be a reason that the file exists.

I believe that parts of history hidden by info/grafts are removed now,
otherwise git clone would not have problems to clone this repository.
So the best option now is only to turn the "fake" history to the real
one using git filter-branch as you wrote.


> - The repository is broken. Then 'git repack -f -d -a' was done while the
> grafts were in effect (and obviously the grafts had hidden parts of the
> history, which are now no longer available).

Actually, 'git repack -f -d -a' does not remove hidden parts of the
history (because 'git prune-packed' does not remove hidden parts), but
'git prune' does remove hidden parts. I believe 'git prune' relies on
'git-fsck --unreachable', which outputs hidden parts as unreachable.
Perhaps, git-fsck should be corrected, so it will not treat hidden as
unreachable, because anything what is hidden is still reachable for
anyone who is trying to clone the repository.

Dmitry

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Dmitry Potapov schrieb:
> On Wed, Apr 16, 2008 at 03:22:04PM +0200, Johannes Sixt wrote:
>> Brian Foster schrieb:
>>> Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
>>>>  I suspect your original git repository has info/grafts
>>> hi Dmitry,
>>>
>>>  bingo!  YES, it does:
>> ...
>>>  the goal is to put things into a sane state so any new
>>>  clones are healthy.  there's only one(?) existing clone,
>>>  which may or may not be(? become?) an issue.
>> Just move info/grafts out of the way and you *may* be all set. Don't
>> delete it - there might be a reason that the file exists.
> 
> I believe that parts of history hidden by info/grafts are removed now,
> otherwise git clone would not have problems to clone this repository.

No. The reason why git clone'd repositories have problems is that
git-upload-pack sends a pack that does not contain the hidden objects; but
since the cloned repository doesn't have the info/grafts, it tries to look
up the hidden, now missing, objects, and fails.

>> - The repository is broken. Then 'git repack -f -d -a' was done while the
>> grafts were in effect (and obviously the grafts had hidden parts of the
>> history, which are now no longer available).
> 
> Actually, 'git repack -f -d -a' does not remove hidden parts of the
> history (because 'git prune-packed' does not remove hidden parts), but
> 'git prune' does remove hidden parts. I believe 'git prune' relies on
> 'git-fsck --unreachable', which outputs hidden parts as unreachable.
> Perhaps, git-fsck should be corrected, so it will not treat hidden as
> unreachable, because anything what is hidden is still reachable for
> anyone who is trying to clone the repository.

Actually, no. The trouble is that *all* tools obey grafts. Hence,
git-repack -f -d -a will remove the hidden objects. But a subsequent fsck
won't notice, because it *also* obeys the grafts. git prune will remove
hidden objects only as long as they are loose; if they are already packed,
then only a repack -f will remove them.

Here's a thread with a proposal by Linus how to improve the situation, but
nobody came along and implemented it:

http://thread.gmane.org/gmane.comp.version-control.git/37744/focus=37908

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:
> 
> No. The reason why git clone'd repositories have problems is that
> git-upload-pack sends a pack that does not contain the hidden objects; but
> since the cloned repository doesn't have the info/grafts, it tries to look
> up the hidden, now missing, objects, and fails.

In my experiment, there was no problem with hidden commits and objects
as long as they presented in the original repository. Maybe, it is
because I cloned it as locally using:

   git clone PATH-TO-ORIG-REPO PATH-TO-DEST-REPO

Maybe things would be as you say if I used git://

> 
> Actually, no. The trouble is that *all* tools obey grafts. Hence,
> git-repack -f -d -a will remove the hidden objects. But a subsequent fsck
> won't notice, because it *also* obeys the grafts. git prune will remove
> hidden objects only as long as they are loose; if they are already packed,
> then only a repack -f will remove them.

I use git 1.5.5 and I have a script that creates are repo with a hidden
commit and then it cleans the reflog cleaned (as it would be happen after
expiration of gc.reflogExpire). Then I run 'git-repack -f -d -a' and the
hidden commit still presents, but when I run 'git prune' then the hidden
object disappear. You can try it for yourself. Here is the script, I used:

===
#!/bin/sh

set -e

mkdir t5
cd t5
git init

echo 1 > foo
git add foo
git commit -m 'add foo'
R1=$(git rev-parse HEAD)

echo 2 >> foo
git commit -m 'edit foo' -a
R2=$(git rev-parse HEAD)

echo 3 >> foo
git commit -m 'edit foo again' -a
R3=$(git rev-parse HEAD)

set -x
git log
echo "$R3 $R1" > .git/info/grafts
git log
git-fsck --unreachable
: > .git/logs/HEAD
: > .git/logs/refs/heads/master
git-fsck --unreachable
git gc
git-fsck --unreachable


git-repack -f -d -a
git-fsck --unreachable

git prune
git-fsck --unreachable
===

Dmitry

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Jakub Narebski]

Dmitry Potapov wrote:

> On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:
>> 
>> No. The reason why git clone'd repositories have problems is that
>> git-upload-pack sends a pack that does not contain the hidden objects;
but
>> since the cloned repository doesn't have the info/grafts, it tries to
look
>> up the hidden, now missing, objects, and fails.
> 
> In my experiment, there was no problem with hidden commits and objects
> as long as they presented in the original repository. Maybe, it is
> because I cloned it as locally using:
> 
>    git clone PATH-TO-ORIG-REPO PATH-TO-DEST-REPO
> 
> Maybe things would be as you say if I used git://

file:// should be enough.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Dmitry Potapov schrieb:
> On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:
>> Actually, no. The trouble is that *all* tools obey grafts. Hence,
>> git-repack -f -d -a will remove the hidden objects. But a subsequent fsck
>> won't notice, because it *also* obeys the grafts. git prune will remove
>> hidden objects only as long as they are loose; if they are already packed,
>> then only a repack -f will remove them.
> 
> I use git 1.5.5 and I have a script that creates are repo with a hidden
> commit and then it cleans the reflog cleaned (as it would be happen after
> expiration of gc.reflogExpire). Then I run 'git-repack -f -d -a' and the
> hidden commit still presents, but when I run 'git prune' then the hidden
> object disappear. You can try it for yourself. Here is the script, I used:

And the reason for this is...

> 
> ===
> #!/bin/sh
> 
> set -e
> 
> mkdir t5
> cd t5
> git init
> 
> echo 1 > foo
> git add foo
> git commit -m 'add foo'
> R1=$(git rev-parse HEAD)
> 
> echo 2 >> foo
> git commit -m 'edit foo' -a
> R2=$(git rev-parse HEAD)
> 
> echo 3 >> foo
> git commit -m 'edit foo again' -a
> R3=$(git rev-parse HEAD)
> 
> set -x
> git log
> echo "$R3 $R1" > .git/info/grafts
> git log
> git-fsck --unreachable
> : > .git/logs/HEAD
> : > .git/logs/refs/heads/master
> git-fsck --unreachable
> git gc

... at this point the hidden commit still exists as loose objects (because
the repack inside gc didn't see the commit), and therefore...

> git-fsck --unreachable
> 
> 
> git-repack -f -d -a

... this does not make a difference at all.

> git-fsck --unreachable
> 
> git prune

Here you remove the loose objects (because prune doesn't see the commit,
either), and by doing so you corrupt the repository.

> git-fsck --unreachable
> ===

A repack -a -d makes a difference if info/grafts is installed *after* the
first repack/gc.

-- Hannes