Archiving tags/branches?

Archiving tags/branches?

[Pete Harlan]

Hi,

I'm looking for a way to manage an ever-growing list of tags.  I've read
some git docs, but am new to git and wonder if the below method doesn't
work or if there's a standard practice I haven't run into.

Most of the tags in my repo are uninteresting to look at, but can't be
deleted.  (Code releases for the most part, or stalled topic branches.)
 If I wanted to archive those, it looks like this would work:

mkdir .git/refs/archived-tags
cp -a .git/refs/tags/* .git/refs/archived-tags
git tag -d <tag-to-hide> # repeat as necessary

I can then maintain a short list of tags that currently interest me, but
am guaranteed not to lose old branches (say) referenced by those tags.

Is there a reason this won't work?

The immediate downsides I see are:

1. The name "archived-tags" might clash someday with a git directory.

2. I have to manually copy this to clones if I want it there too, and
can't manage it from them remotely.

In general, I'm thinking flat tag and branch namespaces must get
unweildy, and short of implementing directory-style namespace management
within git (e.g., hide tags beginning with "." by default, allow tag
subdirectories) I'm looking for a workaround.

Thanks,

--Pete

Re: Archiving tags/branches?

[David Symonds]

On Sat, Oct 18, 2008 at 12:43 PM, Pete Harlan <pgit@pcharlan.com> wrote:
> Hi,
>
> I'm looking for a way to manage an ever-growing list of tags.  I've read
> some git docs, but am new to git and wonder if the below method doesn't
> work or if there's a standard practice I haven't run into.
>
> Most of the tags in my repo are uninteresting to look at, but can't be
> deleted.  (Code releases for the most part, or stalled topic branches.)
>  If I wanted to archive those, it looks like this would work:

Is it really true that they can't be deleted? The only reason to avoid
it might be for preventing Git's GC from cleaning them up, but if all
your branches/tags are reachable via "interesting" branches/tags then
you could just slap the tag name and SHA1 in a text file somewhere.


Dave.

Re: Archiving tags/branches?

[SZEDER Gábor]

Hi Pete,

On Fri, Oct 17, 2008 at 06:43:46PM -0700, Pete Harlan wrote:
>  If I wanted to archive those, it looks like this would work:
> 
> mkdir .git/refs/archived-tags
> cp -a .git/refs/tags/* .git/refs/archived-tags
> git tag -d <tag-to-hide> # repeat as necessary
> 
> I can then maintain a short list of tags that currently interest me, but
> am guaranteed not to lose old branches (say) referenced by those tags.
> 
> Is there a reason this won't work?

Yes:

$ git --version
git version 1.6.0.2.574.g7d0e0
$ git init
Initialized empty Git repository in /home/szeder/tmp/git/archive/.git/
$ echo 1 >foo
$ git add foo
$ git commit -m bar
Created initial commit 0c92489: bar
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 foo
$ git tag t
$ git update-ref refs/archived-tags/t t
$ git tag -d t
Deleted tag 't'
$ cat .git/refs/archived-tags/t
0c92489da6ec6dfd9875eb590d820fcceb01829b
$ git gc
Counting objects: 3, done.
Writing objects: 100% (3/3), done.
Total 3 (delta 0), reused 0 (delta 0)
$ cat .git/refs/archived-tags/t
cat: .git/refs/archived-tags/t: No such file or directory

So, if you put any tags or branches under refs/whatever-non-standard/,
then it gets deleted when you gc (or when gc is run automatically).

I don't know whether this behaviour is intentional or not, but I have
experienced this the hard way recently.

Regards,
Gábor

Re: Archiving tags/branches?

[Johan Herland]

On Saturday 18 October 2008, SZEDER Gábor wrote:
> Hi Pete,
>
> On Fri, Oct 17, 2008 at 06:43:46PM -0700, Pete Harlan wrote:
> >  If I wanted to archive those, it looks like this would work:
> >
> > mkdir .git/refs/archived-tags
> > cp -a .git/refs/tags/* .git/refs/archived-tags
> > git tag -d <tag-to-hide> # repeat as necessary
> >
> > I can then maintain a short list of tags that currently interest me,
> > but am guaranteed not to lose old branches (say) referenced by those
> > tags.
> >
> > Is there a reason this won't work?
>
> Yes:
>
> [...]
>
> So, if you put any tags or branches under refs/whatever-non-standard/,
> then it gets deleted when you gc (or when gc is run automatically).
>
> I don't know whether this behaviour is intentional or not, but I have
> experienced this the hard way recently.

Go have a look in .git/packed-refs. Then have a read through 
git-pack-refs(1).


Have fun! :)

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: Archiving tags/branches?

[SZEDER Gábor]

On Sat, Oct 18, 2008 at 01:15:49PM +0200, Johan Herland wrote:
> Go have a look in .git/packed-refs. Then have a read through 
> git-pack-refs(1).
Oh, indeed, my good old refs are there!  Thanks for the info.

Gábor

Re: Archiving tags/branches?

[Johan Herland]

On Saturday 18 October 2008, SZEDER Gábor wrote:
> On Sat, Oct 18, 2008 at 01:15:49PM +0200, Johan Herland wrote:
> > Go have a look in .git/packed-refs. Then have a read through
> > git-pack-refs(1).
>
> Oh, indeed, my good old refs are there!  Thanks for the info.

BTW, the best way IMHO to archive old refs is to clone your repo (with all 
tags/branches) to a backup disk, and then regularly push (git push --all && 
git push --tags) your new tags/branches to this backup. You are now free to 
delete these tags/branches from your work repo (they will not be deleted 
from the backup unless you use "git push --mirror"). And if you ever need 
to retrieve an old tag/branch, it's just a matter of pulling it from the 
backup repo. Nice, clean, flexible, and requires no changes to git.


Have fun! :)

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: Archiving tags/branches?

[Pete Harlan]

David Symonds wrote:
> On Sat, Oct 18, 2008 at 12:43 PM, Pete Harlan <pgit@pcharlan.com>
> wrote:
>> Hi,
>> 
>> I'm looking for a way to manage an ever-growing list of tags.  I've
>> read some git docs, but am new to git and wonder if the below
>> method doesn't work or if there's a standard practice I haven't run
>> into.
>> 
>> Most of the tags in my repo are uninteresting to look at, but can't
>> be deleted.  (Code releases for the most part, or stalled topic
>> branches.) If I wanted to archive those, it looks like this would
>> work:
> 
> Is it really true that they can't be deleted? The only reason to
> avoid it might be for preventing Git's GC from cleaning them up, but
> if all your branches/tags are reachable via "interesting"
> branches/tags then you could just slap the tag name and SHA1 in a
> text file somewhere.
> 
> 
> Dave.

Thank you for your response.  The tags aren't reachable; they're
dead-end branches.

Our development history looks like this:

o---o---o---o---o---o---o---o---o---o---o---o---o---o master
 \                   \                   \
  o---o---o r1.0      o---o---o r1.1      o---o---o r1.2

with releases branched off the development line and stabilized during
QA.  Fixes into the release branches are cherry-picked out of master,
with no merges.

With a new release every few weeks, the tags pile up.

(There are workflows, such as git.git's, where the release tags form one
long line of development, and when we start using git we may use a
different workflow, but the above was our svn workflow, for the
obvious reason that svn doesn't understand merges.  We're going to
import hundreds of such branches in the conversion to git; most such
names are noise, but we don't want to lose the history.)

I would think a built-in feature for archiving refs would be useful to
other projects, even when the tags/branches are reachable and therefore
one could manually stash them in a file.  Getting the design right is
tricky because there are a lot of different ways to approach it, but the
idea seems generally useful to me.

One direction would be to support directory commands for tags, using
refs/tags and refs/branches as the root directories of trees.  (This was
the solution in svn, which naturally supports a hierarchy of branches.)
 Another would be to have a regexp for hiding tags/branches with a
certain pattern (e.g., leading '.').  What I'll probably do in the short
term is write an alias that lists the most recent 10 tags and use that
most of the time.

--Pete

Re: Archiving tags/branches?

[Pete Harlan]

Johan Herland wrote:
> BTW, the best way IMHO to archive old refs is to clone your repo (with all 
> tags/branches) to a backup disk, and then regularly push (git push --all && 
> git push --tags) your new tags/branches to this backup. You are now free to 
> delete these tags/branches from your work repo (they will not be deleted 
> from the backup unless you use "git push --mirror"). And if you ever need 
> to retrieve an old tag/branch, it's just a matter of pulling it from the 
> backup repo. Nice, clean, flexible, and requires no changes to git.
> 
> 
> Have fun! :)
> 
> ...Johan

Hi,

Thank you; that indeed seems to work and solves the problem of managing
refs/archived-tags manually.

Using a secondary repo solely to overcome a flat tag/branch namespace
feels hackish.  Perhaps git will benefit someday from work in this area,
but until I come up with a patch your suggestion should work fine.  Just
knowing I didn't overlook an existing feature helps a lot.

--Pete

Re: Archiving tags/branches?

[Johan Herland]

On Monday 20 October 2008, Pete Harlan wrote:
> Johan Herland wrote:
> > BTW, the best way IMHO to archive old refs is to clone your repo (with
> > all tags/branches) to a backup disk, and then regularly push (git push
> > --all && git push --tags) your new tags/branches to this backup. You
> > are now free to delete these tags/branches from your work repo (they
> > will not be deleted from the backup unless you use "git push
> > --mirror"). And if you ever need to retrieve an old tag/branch, it's
> > just a matter of pulling it from the backup repo. Nice, clean,
> > flexible, and requires no changes to git.
> >
> >
> > Have fun! :)
> >
> > ...Johan
>
> Hi,
>
> Thank you; that indeed seems to work and solves the problem of managing
> refs/archived-tags manually.
>
> Using a secondary repo solely to overcome a flat tag/branch namespace
> feels hackish.  Perhaps git will benefit someday from work in this area,
> but until I come up with a patch your suggestion should work fine.  Just
> knowing I didn't overlook an existing feature helps a lot.

>From reading your other emails, I get the feeling that I'm in a similar 
situation at $dayjob (i.e. converting ~9 years of development history from 
CVS to Git). We have literally tens of thousands of tags (mostly build and 
release tags) in some of our repos, and keeping all these tags in our daily 
work repos is simply unwieldy and impractical. We therefore plan to have 
official reps which only contain the most important tags, and 
have "archive" repos in a different location that contain all the other 
tags.

You seem to want to keep all your tags in the work repo, but in a 
separate/hidden namespace, so that they don't clutter the default tag 
listings. IMHO, once you get into thousands of tags, cloning and other 
operations where all refs are synchronized become annoyingly slow (although 
things are certainly somewhat better in v1.6). At that point, my only 
advice is to keep the lesser-used tags in separate repos, and pull each ref 
into your work repos on-demand, especially when most of these tags will 
probably never be referenced.


Have fun! :)

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: Archiving tags/branches?

[Jakub Narebski]

Pete Harlan <pgit@pcharlan.com> writes:

> Johan Herland wrote:

> > BTW, the best way IMHO to archive old refs is to clone your repo
> > (with all tags/branches) to a backup disk, and then regularly push
> > (git push --all && git push --tags) your new tags/branches to this
> > backup. You are now free to delete these tags/branches from your
> > work repo (they will not be deleted from the backup unless you use
> > "git push --mirror"). And if you ever need to retrieve an old
> > tag/branch, it's just a matter of pulling it from the backup
> > repo. Nice, clean, flexible, and requires no changes to git.
> 
> Thank you; that indeed seems to work and solves the problem of managing
> refs/archived-tags manually.
> 
> Using a secondary repo solely to overcome a flat tag/branch namespace
> feels hackish.  Perhaps git will benefit someday from work in this area,
> but until I come up with a patch your suggestion should work fine.  Just
> knowing I didn't overlook an existing feature helps a lot.

I don't quite understand what you mean by _flat_ namespace for tags
and branches.

First, it is not unusual to have hierarchical branch names, at least
for short-term topic branches. For example in git.git history (and in
"What's cooking..." announcements on git mailing list) you can find
branch names such as rs/alloc-ref, nd/narrow, tr/workflow-doc.
Additionally remote-tracking branch names have inherently hierarchical
names: refs/remotes/<remote>/<remote branch>.  While tag names usually
are of the type x.y.z, it is not mandated by some technological
limitation.

Second, you can always put your archived refs in another namespace,
beside 'heads', 'tags', and 'remotes'. I for example use
refs/tags/Attic for lightweigth tags to some interesting abandoned
experiments, but it could have been refs/deleted/tags, or
refs/Attic/tags.

Last, please remember that there exists something like packed refs
format (see git-pack-refs(1)... oops, it dies not describe
.git/packed-refs format, unfortunately).

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Archiving tags/branches?

[Pete Harlan]

Johan Herland wrote:
> On Monday 20 October 2008, Pete Harlan wrote:
>> Johan Herland wrote:
>>> BTW, the best way IMHO to archive old refs is to clone your repo (with
>>> all tags/branches) to a backup disk, and then regularly push (git push
>>> --all && git push --tags) your new tags/branches to this backup. You
>>> are now free to delete these tags/branches from your work repo (they
>>> will not be deleted from the backup unless you use "git push
>>> --mirror"). And if you ever need to retrieve an old tag/branch, it's
>>> just a matter of pulling it from the backup repo. Nice, clean,
>>> flexible, and requires no changes to git.
>>>
>>>
>>> Have fun! :)
>>>
>>> ...Johan
>> Hi,
>>
>> Thank you; that indeed seems to work and solves the problem of managing
>> refs/archived-tags manually.
>>
>> Using a secondary repo solely to overcome a flat tag/branch namespace
>> feels hackish.  Perhaps git will benefit someday from work in this area,
>> but until I come up with a patch your suggestion should work fine.  Just
>> knowing I didn't overlook an existing feature helps a lot.
> 
> From reading your other emails, I get the feeling that I'm in a similar 
> situation at $dayjob (i.e. converting ~9 years of development history from 
> CVS to Git). We have literally tens of thousands of tags (mostly build and 
> release tags) in some of our repos, and keeping all these tags in our daily 
> work repos is simply unwieldy and impractical. We therefore plan to have 
> official reps which only contain the most important tags, and 
> have "archive" repos in a different location that contain all the other 
> tags.

Another solution that may work for me is to bind the old lines of
development together using the merge strategy "ours" to link them in a
chain.  When I first read about "ours" I thought it only has evil
applications, but it seems to be created for just this sort of tying
together development that is actually, but was not historically, linked.
 Making those tags reachable from current heads would allow stashing
them in a versioned file somewhere without cluttering up the real
tags/branches or requiring a separate repo.

> You seem to want to keep all your tags in the work repo, but in a 
> separate/hidden namespace, so that they don't clutter the default tag 
> listings. IMHO, once you get into thousands of tags, cloning and other 
> operations where all refs are synchronized become annoyingly slow (although 
> things are certainly somewhat better in v1.6). At that point, my only 
> advice is to keep the lesser-used tags in separate repos, and pull each ref 
> into your work repos on-demand, especially when most of these tags will 
> probably never be referenced.

The efficiency issue is one I hadn't considered; thanks.

--Pete

Re: Archiving tags/branches?

[Pete Harlan]

Jakub Narebski wrote:
> Pete Harlan <pgit@pcharlan.com> writes:
> 
>> Johan Herland wrote:
> 
>>> BTW, the best way IMHO to archive old refs is to clone your repo
>>> (with all tags/branches) to a backup disk, and then regularly push
>>> (git push --all && git push --tags) your new tags/branches to this
>>> backup. You are now free to delete these tags/branches from your
>>> work repo (they will not be deleted from the backup unless you use
>>> "git push --mirror"). And if you ever need to retrieve an old
>>> tag/branch, it's just a matter of pulling it from the backup
>>> repo. Nice, clean, flexible, and requires no changes to git.
>> Thank you; that indeed seems to work and solves the problem of managing
>> refs/archived-tags manually.
>>
>> Using a secondary repo solely to overcome a flat tag/branch namespace
>> feels hackish.  Perhaps git will benefit someday from work in this area,
>> but until I come up with a patch your suggestion should work fine.  Just
>> knowing I didn't overlook an existing feature helps a lot.
> 
> I don't quite understand what you mean by _flat_ namespace for tags
> and branches.
> 
> First, it is not unusual to have hierarchical branch names, at least
> for short-term topic branches. For example in git.git history (and in
> "What's cooking..." announcements on git mailing list) you can find
> branch names such as rs/alloc-ref, nd/narrow, tr/workflow-doc.
> Additionally remote-tracking branch names have inherently hierarchical
> names: refs/remotes/<remote>/<remote branch>.  While tag names usually
> are of the type x.y.z, it is not mandated by some technological
> limitation.

What I mean by "flat" is that "/" is just another character as far as
what git exposes to the user.  Regardless of any semantics the user
chooses to assign to it, and regardless of what advantage git makes use
of "/" internally, unless I can do something like:

% git tag --ls
sometag
someothertag
releases/
% git tag --ls releases/
releases/2008/
releases/2007/
% git tag --ls releases/2008
releases/2008/r3.14
%

"/" is just like any another character in a tag or branch.

(The above notional --ls modifier is probably very easy to write, and if
I do so it may address all of my woes.  Subversion's branching/tagging
can be organized pretty much exactly like this, and importing into git
such a repository is what initially led me to ask about organizing tags
and branches.)

What I'm usually likely to want from a "list tags" command is to see the
most recent few tags, not (say) all 226 tags in git.git.  I'll probably
write a little alias that does that, but even then when looking at the
whole list it would be nice to have the option to navigate it
hierarchically.  (Or in some other manner, and/or possibly with a
configurable directory separator.)

> Second, you can always put your archived refs in another namespace,
> beside 'heads', 'tags', and 'remotes'. I for example use
> refs/tags/Attic for lightweigth tags to some interesting abandoned
> experiments, but it could have been refs/deleted/tags, or
> refs/Attic/tags.

My original question was asking whether this sort of thing would work
(e.g., they would never be automatically pruned), and I'm happy to see
that the answer is yes.  The main downside to it is that you can't
clone/pull/push changes to it using git.

Many thanks to you and everyone for their help.  Git is so flexible that
it can be difficult when starting out to know whether you're missing a
way of attacking a problem.

--Pete

> Last, please remember that there exists something like packed refs
> format (see git-pack-refs(1)... oops, it dies not describe
> .git/packed-refs format, unfortunately).

Re: Archiving tags/branches?

[Jakub Narebski]

On Tue, 21 Oct 2008 06:08, Pete Harlan napisał:
> Jakub Narebski wrote:
>> Pete Harlan <pgit@pcharlan.com> writes:
>>> Johan Herland wrote:
 
>>>
>>> Using a secondary repo solely to overcome a flat tag/branch namespace
>>> feels hackish.  Perhaps git will benefit someday from work in this area,
>>> but until I come up with a patch your suggestion should work fine.  Just
>>> knowing I didn't overlook an existing feature helps a lot.
>> 
>> I don't quite understand what you mean by _flat_ namespace for tags
>> and branches.
>> 
>> First, it is not unusual to have hierarchical branch names, at least
>> for short-term topic branches. For example in git.git history (and in
>> "What's cooking..." announcements on git mailing list) you can find
>> branch names such as rs/alloc-ref, nd/narrow, tr/workflow-doc.
>> Additionally remote-tracking branch names have inherently hierarchical
>> names: refs/remotes/<remote>/<remote branch>.  While tag names usually
>> are of the type x.y.z, it is not mandated by some technological
>> limitation.
> 
> What I mean by "flat" is that "/" is just another character as far as
> what git exposes to the user.  Regardless of any semantics the user
> chooses to assign to it, and regardless of what advantage git makes use
> of "/" internally, unless I can do something like:
> 
> % git tag --ls
> sometag
> someothertag
> releases/
> % git tag --ls releases/
> releases/2008/
> releases/2007/
> % git tag --ls releases/2008
> releases/2008/r3.14
> %

Actually you can have kind of second and third examples; "git tag -l"
takes optional <pattern> argument, so you can do the folowing without
any new features:

  # git tag -l 'releases/2008/*'
  releases/2008/r3.14
 
(the quotes are to protect wildcard characters against expansion by
shell)

> "/" is just like any another character in a tag or branch.
> 
> (The above notional --ls modifier is probably very easy to write, and if
> I do so it may address all of my woes.  Subversion's branching/tagging
> can be organized pretty much exactly like this, and importing into git
> such a repository is what initially led me to ask about organizing tags
> and branches.)

Hmmm... it looks like what you are complaining is not the fact that
tags have flat namespace, but the fact that recursive mode is the
default behavior (something like "ls -R" or "git ls-tree -r").
 
> What I'm usually likely to want from a "list tags" command is to see the
> most recent few tags, not (say) all 226 tags in git.git.  I'll probably
> write a little alias that does that, but even then when looking at the
> whole list it would be nice to have the option to navigate it
> hierarchically.  (Or in some other manner, and/or possibly with a
> configurable directory separator.)

So you would want some '--local' / '--non-recursive' option to listing
all tags (for git-tag) and branches (for git-branch).

As to the "most recent few tags":
  $ git for-each-ref --format='%(refname)' --sort=-taggerdate --count=10 refs/tags/

-- 
Jakub Narebski
Poland

Re: Archiving tags/branches?

[Pete Harlan]

Jakub Narebski wrote:
> > (The above notional --ls modifier is probably very easy to write, and if
> > I do so it may address all of my woes.  Subversion's branching/tagging
> > can be organized pretty much exactly like this, and importing into git
> > such a repository is what initially led me to ask about organizing tags
> > and branches.)
>
> Hmmm... it looks like what you are complaining is not the fact that
> tags have flat namespace, but the fact that recursive mode is the
> default behavior (something like "ls -R" or "git ls-tree -r").
>   

Yes, though I hope it didn't sound like I was complaining, just trying 
to understand how people manage these things.  (And "recursive" mode
being the only mode is precisely what flattens the namespace.)


> > What I'm usually likely to want from a "list tags" command is to see the
> > most recent few tags, not (say) all 226 tags in git.git.  I'll probably
> > write a little alias that does that, but even then when looking at the
> > whole list it would be nice to have the option to navigate it
> > hierarchically.  (Or in some other manner, and/or possibly with a
> > configurable directory separator.)
>
> So you would want some '--local' / '--non-recursive' option to listing
> all tags (for git-tag) and branches (for git-branch).
>
>   
Sure, though I hadn't thought of it in those terms.

> As to the "most recent few tags":
>   $ git for-each-ref --format='%(refname)' --sort=-taggerdate --count=10 refs/tags/
>
>   

Well that's pretty slick, thanks :)  I have aliased that to "lt", and
"lb" to:

for-each-ref --format='%(refname:short)'  --sort=-authordate --count=8
refs/heads/

(which seems less useful, but it was useful as homework...)

Thanks again,

--Pete