Re: dangling commits

Re: dangling commits

[Andreas Ericsson]

Nick Williams wrote:
> Hi, after cloning the git repo with
> 
> cg-clone http://www.kernel.org/pub/scm/git/git.git git
> 
> and then doing
> 
> git-fsck-objects --full
> 
> I get the following
> 
> dangling commit 42db15448ea3c21ae458d5ea873157449042c07c
> dangling commit 4d04a4022e7f9f3ada3a64e2010ce65e1fcc5c64
> dangling commit a773f5bda1835d739ee7209589e137ddd7199142
> dangling commit ceb90a511add3b362f1384aa6ea35370d12db315
> 
> However if I do cg-clone git://git.kernel.org/pub/scm/git/git.git
> there's no output from git-fsck --full
> 
> git version = 1.1.GIT
> cogito version = cogito-0.17pre.GIT
> 
> did I do something wrong (again)?
> 

Nopes. One clones over http, so you'll get all objects in the object 
database. The other clones over the far more clever git protocol which 
calculates which objects you need. Obviously you don't need dangling 
commits (and their related blobs), so there will be no such items.

That there are on kernel.org at all is because Junio does rebases of the 
pu branch and then pushes them out, which means that the objects from 
the last rebase of that branch are left dangling.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

dangling commits

[Nick Williams]

Hi, after cloning the git repo with

cg-clone http://www.kernel.org/pub/scm/git/git.git git

and then doing

git-fsck-objects --full

I get the following

dangling commit 42db15448ea3c21ae458d5ea873157449042c07c
dangling commit 4d04a4022e7f9f3ada3a64e2010ce65e1fcc5c64
dangling commit a773f5bda1835d739ee7209589e137ddd7199142
dangling commit ceb90a511add3b362f1384aa6ea35370d12db315

However if I do cg-clone git://git.kernel.org/pub/scm/git/git.git
there's no output from git-fsck --full

git version = 1.1.GIT
cogito version = cogito-0.17pre.GIT

did I do something wrong (again)?

Re: dangling commits

[Junio C Hamano]

Nick Williams <njw@jarb.freeserve.co.uk> writes:

> Andreas Ericsson wrote:
>..
>> Nopes. One clones over http, so you'll get all objects in the object
>> database. The other clones over the far more clever git protocol
>> which calculates which objects you need. Obviously you don't need
>> dangling commits (and their related blobs), so there will be no such
>> items.

Note that only because that is these dangling objects are packed
in the past, and when fetching over http, packs are fetched as a
whole.

> So, is there any advantage of using http? Seems like git:// makes more
> sense.

As long as you can go native git:// protocol, I do not see much
reason to use http:// commit walkers.  OTOH, if you are
firewalled and your sysadmins do not let you pass 9418/tcp
outgoing, HTTP might be your only choice.

Re: dangling commits

[Nick Williams]

Andreas Ericsson wrote:
> Nick Williams wrote:
> 
>> Hi, after cloning the git repo with
>>
>> cg-clone http://www.kernel.org/pub/scm/git/git.git git
>>
>> and then doing
>>
>> git-fsck-objects --full
>>
>> I get the following
>>
>> dangling commit 42db15448ea3c21ae458d5ea873157449042c07c
>> dangling commit 4d04a4022e7f9f3ada3a64e2010ce65e1fcc5c64
>> dangling commit a773f5bda1835d739ee7209589e137ddd7199142
>> dangling commit ceb90a511add3b362f1384aa6ea35370d12db315
>>
>> However if I do cg-clone git://git.kernel.org/pub/scm/git/git.git
>> there's no output from git-fsck --full
>>
>> git version = 1.1.GIT
>> cogito version = cogito-0.17pre.GIT
>>
>> did I do something wrong (again)?
>>
> 
> Nopes. One clones over http, so you'll get all objects in the object 
> database. The other clones over the far more clever git protocol which 
> calculates which objects you need. Obviously you don't need dangling 
> commits (and their related blobs), so there will be no such items.

OK, that makes sense - thanks for the explanation.

> 
> That there are on kernel.org at all is because Junio does rebases of the 
> pu branch and then pushes them out, which means that the objects from 
> the last rebase of that branch are left dangling.
> 

So, is there any advantage of using http? Seems like git:// makes more 
sense.

Re: dangling commits

[Wolfgang Denk]

In message <7vslrp2nw0.fsf@assigned-by-dhcp.cox.net> you wrote:
>
> Note that only because that is these dangling objects are packed
> in the past, and when fetching over http, packs are fetched as a
> whole.

Is ther eany way to clean up such a situation and really get  rid  of
the  dangling  commits?  I understand that I'd first need some way to
"unpack" the packs, but how to do this? 

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Heavier than air flying machines are impossible.
                    -- Lord Kelvin, President, Royal Society, c. 1895

Re: dangling commits

[Junio C Hamano]

Wolfgang Denk <wd@denx.de> writes:

> Is ther eany way to clean up such a situation and really get  rid  of
> the  dangling  commits?  I understand that I'd first need some way to
> "unpack" the packs, but how to do this? 

The easiest is to repack into a single big ball of wax:

	$ git repack -a -d

If you know the pack the stale object is in, you can move it out
of objects/pack/ and repack only that one.

	$ mv .git/objects/packs/pack-$badone.{idx,pack} .
	$ git unpack-objects <pack-$badone.pack
        $ git repack

After you are done:

	$ git prune

Re: dangling commits

[Marco Roeland]

On Sunday January 15th 2006 Wolfgang Denk wrote:

> Is ther eany way to clean up such a situation and really get  rid  of
> the  dangling  commits?  I understand that I'd first need some way to
> "unpack" the packs, but how to do this? 

Note that apart from the disk space they use up, dangling commits don't
do any harm.

However you can easily get rid of them by using "git prune".

As far as I know although packs are used in transferring the commits to
your local repository they are stored there as separate objects, so you
certainly don't have to unpack things yourself for using "git prune".
Git is quite smart, fast and safe on its own I find each time! It really
is a wonderful tool by giving you every possibility to work with it
without inflicting policy on you.

If wanted you can use "git repack -a -d" followed by "git prune-packed"
to create a tight packed repository (all commits and blobs in one pack)
but there is no specific need to.
-- 
Marco Roeland

Re: dangling commits

[Junio C Hamano]

Marco Roeland <marco.roeland@xs4all.nl> writes:

> As far as I know although packs are used in transferring the commits to
> your local repository they are stored there as separate objects,...

That is true only when you are using git native protocols
(i.e. git:// and git over ssh).  Some people pull over dumb
transport (http -- some others still use rsync which is even
dumber and has serious limitations), and when you need objects
that are contained in a pack at the upstream, the packfile is
downloaded as a whole, and it is left packed on your end.

Even when you use git native protocol, the objects the initial
clone gives you are kept packed, so when I rewind and rebuild
"pu" to make some of these objects orphaned, they will stay in
the pack the initial clone gave you.  Unpack+repack is needed to
get rid of them.

As you said, they should not hurt much in practice, though.

Re: dangling commits

[Wolfgang Denk]

In message <20060116085238.GA3768@fiberbit.xs4all.nl> you wrote:
> 
> Note that apart from the disk space they use up, dangling commits don't
> do any harm.

I like to have  my  repository  "clean"  so  there  are  no  warnings
normally  from  git-fsck-objects  -  if  you  get used to expect some
"harmless" messages you might easily miss a critical error.

> However you can easily get rid of them by using "git prune".

Tried this, didn't work.

> As far as I know although packs are used in transferring the commits to
> your local repository they are stored there as separate objects, so you

Ummm... please have a look at the  .git/objects/pack/  directory  for
example in your Linux repository.

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
It seems intuitively obvious to me, which  means  that  it  might  be
wrong.                                                 -- Chris Torek

Re: dangling commits

[Marco Roeland]

On Monday January 16th 2006 Wolfgang Denk wrote:

> I like to have  my  repository  "clean"  so  there  are  no  warnings
> normally  from  git-fsck-objects  -  if  you  get used to expect some
> "harmless" messages you might easily miss a critical error.

That's right, yes.

> > However you can easily get rid of them by using "git prune".
> 
> Tried this, didn't work.

Ok, didn't know that it didn't prune directly from packs. I should have
realised that it only can do so efficiently by repacking I suppose.

> Ummm... please have a look at the  .git/objects/pack/  directory  for
> example in your Linux repository.

I did as a matter of fact, but I use "git" as protocol and also
regularly "repack" the repository, when I'm not using the machine for
a while, to make 'gitk' and 'qgit' work faster. It is less apparent
then! Thanks very much for clearing up my knowledge. It only shows the
power of git that even ignorant gits like myself still find it useful
and productive. There's just enough rope to wiggle yourself out of
precarious situations but fortunately most of the time not enough to
hang yourself.
-- 
Marco Roeland

Re: dangling commits

[Marco Roeland]

On Monday January 16th Junio C Hamano wrote:

> Even when you use git native protocol, the objects the initial
> clone gives you are kept packed, so when I rewind and rebuild
> "pu" to make some of these objects orphaned, they will stay in
> the pack the initial clone gave you.  Unpack+repack is needed to
> get rid of them.

Thanks very much for explaining. It makes sense now.

Does it bring many advantages for you to keep rebasing "pu"? I started
out following that branch long ago (well in git reckoning anyway) but
got very scared each time I got a bunch of "errors" on that one. I even
recloned a couple of times to get it "clean" again, until I understood
from the mailing-list that the rebasing was the cause, not something I
did. I since removed it from the "Pull" list, but understand that "+pu"
should do the trick. I'll retry using it one of these days.
-- 
Marco Roeland

Re: dangling commits

[Andreas Ericsson]

Marco Roeland wrote:
> On Monday January 16th Junio C Hamano wrote:
> 
> 
>>Even when you use git native protocol, the objects the initial
>>clone gives you are kept packed, so when I rewind and rebuild
>>"pu" to make some of these objects orphaned, they will stay in
>>the pack the initial clone gave you.  Unpack+repack is needed to
>>get rid of them.
> 
> 
> Thanks very much for explaining. It makes sense now.
> 
> Does it bring many advantages for you to keep rebasing "pu"?


Since "pu" = "proposed updates" it only makes sense to keep it on top of 
the current master, otherwise the effort required for anyone to test it 
in conjunction with the latest master branch would simply be too great.


> I started
> out following that branch long ago (well in git reckoning anyway) but
> got very scared each time I got a bunch of "errors" on that one.
> I since removed it from the "Pull" list, but understand that "+pu"
> should do the trick. I'll retry using it one of these days.


It does. I also remember seeing lots of errors on that one when I first 
started with git (around 0.99b), but that was fixed quite some time ago.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: dangling commits

[Marco Roeland]

On Monday January 16th 2006 Andreas Ericsson wrote:

> Since "pu" = "proposed updates" it only makes sense to keep it on top of 
> the current master, otherwise the effort required for anyone to test it 
> in conjunction with the latest master branch would simply be too great.

Certainly. And it probably is a good testbed for testing the rebasing
routines as well.

But couldn't (in theory) the new "rebased" versions of blobs in the "pu"
branch be first committed as the old not yet rebased version and then
as the new version. Not the fact that the blobs in "pu" are constantly
based on the latest master was the problem if I recollect, but the fact
that blobs sometimes disappeared. In comparing this with Linus' recent
explanation how git-bisect works in terms of light cones this might be
understood as the inherent problems with tachyons I think...

Anyway this is now solved I understand. Thanks.

> >I since removed it from the "Pull" list, but understand that "+pu"
> >should do the trick. I'll retry using it one of these days.
> 
> It does. I also remember seeing lots of errors on that one when I first 
> started with git (around 0.99b), but that was fixed quite some time ago.

Ok, thanks very much for explaining.
-- 
Marco Roeland

Re: dangling commits

[Andreas Ericsson]

Marco Roeland wrote:
> On Monday January 16th 2006 Andreas Ericsson wrote:
> 
> 
>>Since "pu" = "proposed updates" it only makes sense to keep it on top of 
>>the current master, otherwise the effort required for anyone to test it 
>>in conjunction with the latest master branch would simply be too great.
> 
> 
> But couldn't (in theory) the new "rebased" versions of blobs in the "pu"
> branch be first committed as the old not yet rebased version and then
> as the new version.


The blobs are immutable and never change for a rebase, unless the 
file(s) it applies to is changed in master as well. It's the commits 
that do because they get new parents.

Remember that the blob object is just the (deltified?) file that's the 
result of the commit operation. The commit object is an object in its 
own rights, holding author info and commit-time and such. Do

	$ git cat-file commit HEAD

and you'll see what a commit-object looks like.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: dangling commits

[Marco Roeland]

On Monday January 16th 2006 Andreas Ericsson wrote:

> The blobs are immutable and never change for a rebase, unless the 
> file(s) it applies to is changed in master as well. It's the commits 
> that do because they get new parents.

Ah, I need to rebase my mental picture of what a "rebase" is. ;-) And
the fact that each commit _does_ have sort of a blob (well in my mind I
called it so, the file under .git/objects, although its proper name is
indeed "commit") in the repository doesn't make it any easier!

Current documentation about git-rebase(1) is technically correct of
course then: "rebases local commits to the new head of the upstream
tree" but rather sparse for the less initiated. In Dutch we have an
expression for this, to "not be able to see the wood because of the
trees", which is rather appropriate here. Perhaps we can introduce
"liana" as an alternative for commit.

<Nice young men in clean white coats come in to take me away>

Seriously, yours and other peoples comments make the picture much
clearer to me and help out enormously to me and hopefully other lurkers
in working with git and more advanced SCM in general. Thanks,
-- 
Marco Roeland

dangling commits.

[Dave Jones]

What's wrong here?

(tmp)$ mkdir git
(tmp)$ cd git
(git)$ rsync -azq rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/git.git/ .git
(git)$ git-fsck-objects --full
dangling commit fc024e5e1b92fb11d11aa3c15e9e3f92a888acb6
dangling commit be49def3dd524b241e3f14c657e740f2e4d73d39
dangling commit e659abda29b2b25db2771c3e6086f15bc74b1d06
dangling commit 0f5aaf12d411debc2958dfb029ffcf873d980f15
dangling commit 8eed8a5da8330df6901f254eab4b7056855b919c

The version of git on that box is 1.2.6-1 from Debian etch.

		Dave

-- 
http://www.codemonkey.org.uk

Re: dangling commits.

[Sean]

On Sat, 20 May 2006 19:05:31 -0400
Dave Jones <davej@redhat.com> wrote:

> What's wrong here?
> 
> (tmp)$ mkdir git
> (tmp)$ cd git
> (git)$ rsync -azq rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/git.git/ .git
> (git)$ git-fsck-objects --full
> dangling commit fc024e5e1b92fb11d11aa3c15e9e3f92a888acb6
> dangling commit be49def3dd524b241e3f14c657e740f2e4d73d39
> dangling commit e659abda29b2b25db2771c3e6086f15bc74b1d06
> dangling commit 0f5aaf12d411debc2958dfb029ffcf873d980f15
> dangling commit 8eed8a5da8330df6901f254eab4b7056855b919c
> 
> The version of git on that box is 1.2.6-1 from Debian etch.
> 
> 		Dave
> 

Nothing wrong.  Those commits were just dropped from the "pu" branch at
one time or another when it was rewound.  The repository could use a
pruning, but there's no harm done by having those objects remain in
the object database.   If you had used any protocol other than rsync,
they would not have appeared in your cloned copy.

Sean

Re: dangling commits.

[Linus Torvalds]

On Sat, 20 May 2006, Dave Jones wrote:
>
> What's wrong here?

Nothing, except

 - you used rsync to fetch the thing (so you get all objects, regardless 
   of whether they are reachable or not)

 - junio re-bases his "pu" branch, and I just end up following him (I 
   should stop exporting git entirely, here's no point, really). Thus, 
   whenever "pu" gets reset to something else, the old head of pu gets 
   dropped on the floor and ends up as a "dangling" commit (no branch head 
   points to it any more).

The moral(s) of the story being that

 (a) dangling commits are normal if you reset branch heads (which includes 
     things like "git rebase", for example)

 (b) rsync is not a good transfer protocol, since it will just blindly 
     transfer objects that aren't needed - or worse - successfully clone a 
     corrupt repository.

but there's nothing to _worry_ about in this particular schenario.

		Linus

Re: dangling commits.

[Dave Jones]

On Sat, May 20, 2006 at 04:19:06PM -0700, Linus Torvalds wrote:

 > > What's wrong here?
 > 
 > Nothing, except
 > 
 >  - you used rsync to fetch the thing (so you get all objects, regardless 
 >    of whether they are reachable or not)

Ah. I actually noticed this when I poked around the box that does
the nightly snapshots, and wondered for a few minutes why I never
switched it over to git:// last time I poked at that script.
Then it came back to me: rsync -q worked better than redirecting
git to /dev/null
- git-clone's -q was still outputting some stuff, so recloning each
  time the cronjob ran wasn't an option,
- subsequent git pull's were noisy too

When run from a cronjob, unless something fatal happens, I basically
never want to get mail from the snapshotting script.

 >  - junio re-bases his "pu" branch, and I just end up following him (I 
 >    should stop exporting git entirely, here's no point, really).

I just updated the snapshot script to pull from 
http://www.kernel.org/pub/scm/git/git.git/ instead.

thanks,

		Dave

-- 
http://www.codemonkey.org.uk