[PATCH 0/5] More ref logging

[PATCH 0/5] More ref logging

[Shawn Pearce]

These are on top of the other five I just sent:

* [PATCH 1/5] Correct force_write bug in refs.c

	Yea there was a bug in the last bug fix.  Now we
	really don't write the ref unless we need to.

* [PATCH 2/5] Change order of -m option to update-ref.

	Minor documentation nit noticed by Junio.

* [PATCH 3/5] Include ref log detail in commit, reset, etc.

	Enhance some core tools to use the new '-m' switch with
	update-ref.  Trivial but requires the new update-ref.

* [PATCH 4/5] Create/delete branch ref logs.

	This was discussed on #git earlier this morning.
	Automatically create the ref log if -l is given when creating
	a branch and delete the log when deleting the branch.

* [PATCH 5/5] Enable ref log creation in git checkout -b.

	Fix git checkout -b to behave like git branch.

-- 
Shawn.

Re: [PATCH 0/5] More ref logging

[Sean]

On Fri, 19 May 2006 05:14:56 -0400
Shawn Pearce <spearce@spearce.org> wrote:

> * [PATCH 5/5] Enable ref log creation in git checkout -b.
> 
> 	Fix git checkout -b to behave like git branch.

It seems that the ref log is a new way of tagging commits with some
extra meta data.  Conceptually this seems very close to what git tags 
already do.  So... what about using regular git tags rather than
creating a ref log?  All the regular git-rev-list tools could be
used to query the tags and prune would delete them automatically etc.

Just a thought, feel free to ignore it :o)

Sean

Re: [PATCH 0/5] More ref logging

[Sean]

On Fri, 19 May 2006 05:14:56 -0400
Shawn Pearce <spearce@spearce.org> wrote:

> * [PATCH 5/5] Enable ref log creation in git checkout -b.
> 
> 	Fix git checkout -b to behave like git branch.

It seems that the ref log is a new way of tagging commits with some
extra meta data.  Conceptually this seems very close to what git tags 
already do.  So... what about using regular git tags rather than
creating a ref log?  All the regular git-rev-list tools could be
used to query the tags and prune would delete them automatically etc.

Just a thought, feel free to ignore it :o)

Sean

Re: [PATCH 0/5] More ref logging

[Shawn Pearce]

Sean <seanlkml@sympatico.ca> wrote:
> On Fri, 19 May 2006 05:14:56 -0400
> Shawn Pearce <spearce@spearce.org> wrote:
> 
> > * [PATCH 5/5] Enable ref log creation in git checkout -b.
> > 
> > 	Fix git checkout -b to behave like git branch.
> 
> It seems that the ref log is a new way of tagging commits with some
> extra meta data.  Conceptually this seems very close to what git tags 
> already do.  So... what about using regular git tags rather than
> creating a ref log?  All the regular git-rev-list tools could be
> used to query the tags and prune would delete them automatically etc.

It sort of is a new way of tagging commits with extra data.  But its
also sort of a way of versioning your ref `database'.  Using tags
to save the points in time might be useful but it would generate
a lot of temporary files.  A commit every 5 minutes for a typical
working week would generate 480 tags per week.  That's just too much.

I was actually thinking this morning that another way to do this
is to keep a metadata branch within the repository which records
all of the refs in tree objects, then save the root commit under
the special ref `LOG` in GIT_DIR.  Every update to a logged ref
would cause the tree to be updated and a new commit to be built.
The branch would be a relatively simple string of pearls as its
doubtful you would branch it.

There are a number of downsides to this, not the least of which is
I'd like to put a commit or tag SHA1 into the tree object rather than
writing each ref as a blob (saves space).  Currently commits and tags
aren't permitted in a tree object so that would require some effort.
But on the other hand you could pull (and track!) someone elses
ref log through the standard GIT protocol.

But this is starting to head down into the `bind commit` discussion;
how do we record a number of commits as being related and tie them
up into a single super commit?

-- 
Shawn.

Re: [PATCH 0/5] More ref logging

[Sean]

On Sat, 20 May 2006 20:50:09 -0400
Shawn Pearce <spearce@spearce.org> wrote:

> It sort of is a new way of tagging commits with extra data.  But its
> also sort of a way of versioning your ref `database'.  Using tags
> to save the points in time might be useful but it would generate
> a lot of temporary files.  A commit every 5 minutes for a typical
> working week would generate 480 tags per week.  That's just too much.

But isn't that just an implementation detail?  I've actually run
into another situation where tags would be perfect if only they weren't
so expensive (ie. entire repo was in a 50Mb pack including tag objects,
but the .git/refs/tags directory was over 100Mb).

So, if we found a way to store tags more efficiently your 480 tags per
week shouldn't be a problem at all.   The main point being to extend
and optimize the existing infrastructure rather than bolting on a new
class of objects (ie. ref log) which only serves a narrow (albeit
important) purpose.
 
> I was actually thinking this morning that another way to do this
> is to keep a metadata branch within the repository which records
> all of the refs in tree objects, then save the root commit under
> the special ref `LOG` in GIT_DIR.  Every update to a logged ref
> would cause the tree to be updated and a new commit to be built.
> The branch would be a relatively simple string of pearls as its
> doubtful you would branch it.
>
> There are a number of downsides to this, not the least of which is
> I'd like to put a commit or tag SHA1 into the tree object rather than
> writing each ref as a blob (saves space).  Currently commits and tags
> aren't permitted in a tree object so that would require some effort.
> But on the other hand you could pull (and track!) someone elses
> ref log through the standard GIT protocol.
 
Yes, Linus proposed something similar earlier to hold meta data.
But i've come to see tags as a place to store any arbitrary meta
data associated with a commit.   If their implementation was more
efficient you could use them for your project and they could be used
for any number of other purposes as well.

> But this is starting to head down into the `bind commit` discussion;
> how do we record a number of commits as being related and tie them
> up into a single super commit?

Well, a tag that allowed the listing of multiple heads....

Sean

Re: [PATCH 0/5] More ref logging

[Sean]

On Sat, 20 May 2006 20:50:09 -0400
Shawn Pearce <spearce@spearce.org> wrote:

> It sort of is a new way of tagging commits with extra data.  But its
> also sort of a way of versioning your ref `database'.  Using tags
> to save the points in time might be useful but it would generate
> a lot of temporary files.  A commit every 5 minutes for a typical
> working week would generate 480 tags per week.  That's just too much.

But isn't that just an implementation detail?  I've actually run
into another situation where tags would be perfect if only they weren't
so expensive (ie. entire repo was in a 50Mb pack including tag objects,
but the .git/refs/tags directory was over 100Mb).

So, if we found a way to store tags more efficiently your 480 tags per
week shouldn't be a problem at all.   The main point being to extend
and optimize the existing infrastructure rather than bolting on a new
class of objects (ie. ref log) which only serves a narrow (albeit
important) purpose.
 
> I was actually thinking this morning that another way to do this
> is to keep a metadata branch within the repository which records
> all of the refs in tree objects, then save the root commit under
> the special ref `LOG` in GIT_DIR.  Every update to a logged ref
> would cause the tree to be updated and a new commit to be built.
> The branch would be a relatively simple string of pearls as its
> doubtful you would branch it.
>
> There are a number of downsides to this, not the least of which is
> I'd like to put a commit or tag SHA1 into the tree object rather than
> writing each ref as a blob (saves space).  Currently commits and tags
> aren't permitted in a tree object so that would require some effort.
> But on the other hand you could pull (and track!) someone elses
> ref log through the standard GIT protocol.
 
Yes, Linus proposed something similar earlier to hold meta data.
But i've come to see tags as a place to store any arbitrary meta
data associated with a commit.   If their implementation was more
efficient you could use them for your project and they could be used
for any number of other purposes as well.

> But this is starting to head down into the `bind commit` discussion;
> how do we record a number of commits as being related and tie them
> up into a single super commit?

Well, a tag that allowed the listing of multiple heads....

Sean

Re: [PATCH 0/5] More ref logging

[Shawn Pearce]

Sean <seanlkml@sympatico.ca> wrote:
> On Sat, 20 May 2006 20:50:09 -0400
> Shawn Pearce <spearce@spearce.org> wrote:
> 
> > It sort of is a new way of tagging commits with extra data.  But its
> > also sort of a way of versioning your ref `database'.  Using tags
> > to save the points in time might be useful but it would generate
> > a lot of temporary files.  A commit every 5 minutes for a typical
> > working week would generate 480 tags per week.  That's just too much.
> 
> But isn't that just an implementation detail?  I've actually run
> into another situation where tags would be perfect if only they weren't
> so expensive (ie. entire repo was in a 50Mb pack including tag objects,
> but the .git/refs/tags directory was over 100Mb).

Any chance we could get some details on why so many tags were useful?
You have a good argument here but I'm not sure how a better tag
store could be structured.  :-)
 
> So, if we found a way to store tags more efficiently your 480 tags per
> week shouldn't be a problem at all.   The main point being to extend
> and optimize the existing infrastructure rather than bolting on a new
> class of objects (ie. ref log) which only serves a narrow (albeit
> important) purpose.

Yea - despite being the author of ref log I'm still slightly unhappy
with the fact that it doesn't make reuse of existing GIT plumbing.
But I'm sort of OK with that right now as you can't map two indexes
into memory at once currently, nor is there a way to easily update
multiple refs at once if the ref log must serialize access to create
a string of trees and commits.

-- 
Shawn.

Re: [PATCH 0/5] More ref logging

[Sean]

On Sun, 21 May 2006 00:51:46 -0400
Shawn Pearce <spearce@spearce.org> wrote:

> Any chance we could get some details on why so many tags were useful?
> You have a good argument here but I'm not sure how a better tag
> store could be structured.  :-)

It is a conversion from another SCM.  So every one of the ~16K commits
was tagged with the reference number taken from the original SCM.  This
has some very nice benefits in that you can refer to every commit
in git by the original changeset #.  For example, if someone reports a
bug mentioning the original scm's reference id, you can say something
like: "git show p4/1234" without having to go back to the old scm.

Also, qgit, gitk and gitweb display them nicely which can be helpful
during the conversion.  And if/when they're not needed any longer,
you just delete them without having to rewrite the history etc..   
  
> Yea - despite being the author of ref log I'm still slightly unhappy
> with the fact that it doesn't make reuse of existing GIT plumbing.
> But I'm sort of OK with that right now as you can't map two indexes
> into memory at once currently, nor is there a way to easily update
> multiple refs at once if the ref log must serialize access to create
> a string of trees and commits.

Well it's not the end of the world either way, and sometimes it's just
better to implement a workable solution rather than wait for one that's
theoretically cleaner.  It just seemed like it was worth mentioning in
case you saw a way to make it happen without a lot of grief.

Sean

Re: [PATCH 0/5] More ref logging

[Sean]

On Sun, 21 May 2006 00:51:46 -0400
Shawn Pearce <spearce@spearce.org> wrote:

> Any chance we could get some details on why so many tags were useful?
> You have a good argument here but I'm not sure how a better tag
> store could be structured.  :-)

It is a conversion from another SCM.  So every one of the ~16K commits
was tagged with the reference number taken from the original SCM.  This
has some very nice benefits in that you can refer to every commit
in git by the original changeset #.  For example, if someone reports a
bug mentioning the original scm's reference id, you can say something
like: "git show p4/1234" without having to go back to the old scm.

Also, qgit, gitk and gitweb display them nicely which can be helpful
during the conversion.  And if/when they're not needed any longer,
you just delete them without having to rewrite the history etc..   
  
> Yea - despite being the author of ref log I'm still slightly unhappy
> with the fact that it doesn't make reuse of existing GIT plumbing.
> But I'm sort of OK with that right now as you can't map two indexes
> into memory at once currently, nor is there a way to easily update
> multiple refs at once if the ref log must serialize access to create
> a string of trees and commits.

Well it's not the end of the world either way, and sometimes it's just
better to implement a workable solution rather than wait for one that's
theoretically cleaner.  It just seemed like it was worth mentioning in
case you saw a way to make it happen without a lot of grief.

Sean