Feature request: fetch --prune by default

Feature request: fetch --prune by default

[Alexey Muranov]

Hello,

i would like

`git fetch --prune <remote>`

to be the default behavior of

`git fetch <remote>`

In fact, i think this is the only reasonable behavior.
Keeping copies of deleted remote branches after `fetch` is more confusing than useful.

(Excuse me if this question has already been discussed.)

Thank you.

Alexey Muranov.

Re: Feature request: fetch --prune by default

[Jeff King]

On Thu, Jul 19, 2012 at 09:30:59AM +0200, Alexey Muranov wrote:

> i would like
> 
> `git fetch --prune <remote>`
> 
> to be the default behavior of
> 
> `git fetch <remote>`
> 
> In fact, i think this is the only reasonable behavior.
> Keeping copies of deleted remote branches after `fetch` is more confusing than useful.

I agree it would be much less confusing. However, one downside is that
we do not keep reflogs on deleted branches (and nor did the commits in
remote branches necessarily make it into the HEAD reflog). That makes
"git fetch" a potentially destructive operation (you irrevocably lose
the notion of which remote branches pointed where before the fetch, and
you open up new commits to immediate pruning by "gc --auto".

So I think it would be a lot more palatable if we kept reflogs on
deleted branches. That, in turn, has a few open issues, such as how to
manage namespace conflicts (e.g., the fact that a deleted "foo" branch
can conflict with a new "foo/bar" branch).

-Peff

Re: Feature request: fetch --prune by default

[Dan Johnson]

On Thu, Jul 19, 2012 at 7:55 AM, Jeff King <peff@peff.net> wrote:
> On Thu, Jul 19, 2012 at 09:30:59AM +0200, Alexey Muranov wrote:
>
>> i would like
>>
>> `git fetch --prune <remote>`
>>
>> to be the default behavior of
>>
>> `git fetch <remote>`
>>
>> In fact, i think this is the only reasonable behavior.
>> Keeping copies of deleted remote branches after `fetch` is more confusing than useful.
>
> I agree it would be much less confusing. However, one downside is that
> we do not keep reflogs on deleted branches (and nor did the commits in
> remote branches necessarily make it into the HEAD reflog). That makes
> "git fetch" a potentially destructive operation (you irrevocably lose
> the notion of which remote branches pointed where before the fetch, and
> you open up new commits to immediate pruning by "gc --auto".
>
> So I think it would be a lot more palatable if we kept reflogs on
> deleted branches. That, in turn, has a few open issues, such as how to
> manage namespace conflicts (e.g., the fact that a deleted "foo" branch
> can conflict with a new "foo/bar" branch).

In the meantime, would it make sense to introduce a configuration
variable to request this behavior?

If so, should it be global?

fetch.prune = always

or per-remote?

remote.<name>.prune = always

The global option seems to be more in line with what Alexey is looking
for, but the per-remote one is similar to the tagopt option, which is
a similar idea.

Of course, this might be just a waste of time to introduce a feature
no one would use, in which case we obviously should not introduce such
options.
-- 
-Dan

Re: Feature request: fetch --prune by default

[Stefan Haller]

Dan Johnson <computerdruid@gmail.com> wrote:

> In the meantime, would it make sense to introduce a configuration
> variable to request this behavior?
> 
> fetch.prune = always
> 
> Of course, this might be just a waste of time to introduce a feature
> no one would use, in which case we obviously should not introduce such
> options.

I would use it.


-- 
Stefan Haller
Berlin, Germany
http://www.haller-berlin.de/

Re: Feature request: fetch --prune by default

[Junio C Hamano]

Dan Johnson <computerdruid@gmail.com> writes:

> On Thu, Jul 19, 2012 at 7:55 AM, Jeff King <peff@peff.net> wrote:
> ...
>> So I think it would be a lot more palatable if we kept reflogs on
>> deleted branches. That, in turn, has a few open issues, such as how to
>> manage namespace conflicts (e.g., the fact that a deleted "foo" branch
>> can conflict with a new "foo/bar" branch).
>
> In the meantime, would it make sense to introduce a configuration
> variable to request this behavior?
>
> If so, should it be global?
>
> fetch.prune = always
>
> or per-remote?
>
> remote.<name>.prune = always
>
> The global option seems to be more in line with what Alexey is looking
> for, but the per-remote one is similar to the tagopt option, which is
> a similar idea.
>
> Of course, this might be just a waste of time to introduce a feature
> no one would use, in which case we obviously should not introduce such
> options.

I was reading through the backlog today and noticed that this topic
veered into the "reflog graveyard" tangent.  I wasn't involved in
the main topic, but I think having both configuration variables,
remote.<remote>.prune taking precedence over fetch.prune, as long as
we make sure "fetch --no-prune" will override any configured
default, is not a bad thing per-se.

As long as the users who elect to use this feature are aware of the
pruning of the refs and logs, that is, but "branch [-r] -d" has been
the way to lose both the branch and its log for a long time, so I do
not see a big issue there, either.

The log graveyard is an independently interesting idea, which I may
ping separately, but I consider it pretty much orthogonal to this
particular topic.

Re: Feature request: fetch --prune by default

[Jeff King]

On Thu, Aug 16, 2012 at 04:22:54PM -0700, Junio C Hamano wrote:

> > In the meantime, would it make sense to introduce a configuration
> > variable to request this behavior?
> >
> > If so, should it be global?
> >
> > fetch.prune = always
> >
> > or per-remote?
> >
> > remote.<name>.prune = always
> >
> > The global option seems to be more in line with what Alexey is looking
> > for, but the per-remote one is similar to the tagopt option, which is
> > a similar idea.
> >
> > Of course, this might be just a waste of time to introduce a feature
> > no one would use, in which case we obviously should not introduce such
> > options.
> 
> I was reading through the backlog today and noticed that this topic
> veered into the "reflog graveyard" tangent.  I wasn't involved in
> the main topic, but I think having both configuration variables,
> remote.<remote>.prune taking precedence over fetch.prune, as long as
> we make sure "fetch --no-prune" will override any configured
> default, is not a bad thing per-se.
> 
> As long as the users who elect to use this feature are aware of the
> pruning of the refs and logs, that is, but "branch [-r] -d" has been
> the way to lose both the branch and its log for a long time, so I do
> not see a big issue there, either.
> 
> The log graveyard is an independently interesting idea, which I may
> ping separately, but I consider it pretty much orthogonal to this
> particular topic.

Yeah, I think that is sensible. As long as the _default_ is not to
prune, and people are making a conscious choice to prune, I don't see a
problem at all.

The log graveyard is orthogonal to the proposed option, but I think it
would be a necessary step before flipping the default for that option to
"true".

-Peff

Re: Feature request: fetch --prune by default

[Sam Roberts]

I would use the config feature to turn on --prune for fetch, and was
surprised that it wasn't available - I hit this thread because I figured I
somehow missed it in the config docs.

Having both global and local settings seems nice.



--
View this message in context: http://git.661346.n2.nabble.com/Feature-request-fetch-prune-by-default-tp7563241p7590048.html
Sent from the git mailing list archive at Nabble.com.

Re: Feature request: fetch --prune by default

[Alexey Muranov]

On 19 Jul 2012, at 13:55, Jeff King wrote:

> On Thu, Jul 19, 2012 at 09:30:59AM +0200, Alexey Muranov wrote:
> 
>> i would like
>> 
>> `git fetch --prune <remote>`
>> 
>> to be the default behavior of
>> 
>> `git fetch <remote>`
>> 
>> In fact, i think this is the only reasonable behavior.
>> Keeping copies of deleted remote branches after `fetch` is more confusing than useful.
> 
> I agree it would be much less confusing. However, one downside is that
> we do not keep reflogs on deleted branches (and nor did the commits in
> remote branches necessarily make it into the HEAD reflog). That makes
> "git fetch" a potentially destructive operation (you irrevocably lose
> the notion of which remote branches pointed where before the fetch, and
> you open up new commits to immediate pruning by "gc --auto".

I do not still understand very well some aspects of Git, like the exact purpose of "remote tracking branches" (are they for pull or for push?), so i may be wrong.
However, i thought that a user was not expected to follow the moves of a remote branch of which the user is not an owner: if the user needs to follow the brach and not lose its commits, he/she should create a remote tracking branch.

> So I think it would be a lot more palatable if we kept reflogs on
> deleted branches. That, in turn, has a few open issues, such as how to
> manage namespace conflicts (e.g., the fact that a deleted "foo" branch
> can conflict with a new "foo/bar" branch).

I prefer to think of a remote branch and its local copy as the same thing, which are physically different only because of current real world/hardware/software limitations, which make it necessary to keep a local cache of remote data.  With this approach, reflogs should be deleted with the branch, and there will be no namespace conflicts.

Alexey.

Re: Feature request: fetch --prune by default

[Konstantin Khomoutov]

On Thu, 19 Jul 2012 18:21:21 +0200
Alexey Muranov <alexey.muranov@gmail.com> wrote:

[...]
> I do not still understand very well some aspects of Git, like the
> exact purpose of "remote tracking branches" (are they for pull or for
> push?), so i may be wrong.
This is wery well explained in the Pro Git book, for instance.
And in numerous blog posts etc.

> However, i thought that a user was not
> expected to follow the moves of a remote branch of which the user is
> not an owner: if the user needs to follow the brach and not lose its
> commits, he/she should create a remote tracking branch.
This would present another namespacing issue: how would you name the
branches you're interested in so that they don't clash with your own
personal local branches?  You'd have to invent a scheme which would
encode the remote's name in a branch name.  But remote branches already
do just this.  So you create a remote tracking branch when you intend
to actually *develop* something on that branch with the final intention
to push that work back.

> > So I think it would be a lot more palatable if we kept reflogs on
> > deleted branches. That, in turn, has a few open issues, such as how
> > to manage namespace conflicts (e.g., the fact that a deleted "foo"
> > branch can conflict with a new "foo/bar" branch).
> 
> I prefer to think of a remote branch and its local copy as the same
> thing, which are physically different only because of current real
> world/hardware/software limitations, which make it necessary to keep
> a local cache of remote data.  With this approach, reflogs should be
> deleted with the branch, and there will be no namespace conflicts.
It appears, the distributed nature of a DVCS did not fully sink into
your mindset yet. ;-)
Looks like you mentally treat a Git remote as a thing being used to
access a centralized "reference" server which maintains a master copy
of a repository, of which you happen to also have a local copy.
Then it's quite logically to think that if someone deleted a branch in
the master copy, everyone "downstream" should have the same
remote branch deleted to be in sync with that master copy.
But this is not the only way to organize your work.
You could fetch from someone else's repository and be interested in
their branch "foo", but think what happens when you fetch next time from
that repo and see Git happily deleting your local branch thatremote/foo
simply because someone with push access deleted that branch from the
repo.  This might *not* be what you really want or expect.

Re: Feature request: fetch --prune by default

[Alexey Muranov]

On 19 Jul 2012, at 19:34, Konstantin Khomoutov wrote:

> On Thu, 19 Jul 2012 18:21:21 +0200
> Alexey Muranov <alexey.muranov@gmail.com> wrote:
> 
> [...]
>> I do not still understand very well some aspects of Git, like the
>> exact purpose of "remote tracking branches" (are they for pull or for
>> push?), so i may be wrong.
> This is wery well explained in the Pro Git book, for instance.
> And in numerous blog posts etc.

I have read the Pro Gut book and numerous blog posts, but i keep forgetting the explanation because it does not make much sense to me:

"Tracking branches are local branches that have a direct relationship to a remote branch.  If you’re on a tracking branch and type git push, Git automatically knows which server and branch to push to.  Also, running git pull while on one of these branches fetches all the remote references and then automatically merges in the corresponding remote branch." etc.

Why the same "direct relationship" for push and pull?  What happens if one of the branches was reset (yes, i know, "push -f").  Most importantly, what is the purpose of it? It is natural to expect that you might be pushing to and pulling from different remotes, i can even imagine pulling from more than one.

>> However, i thought that a user was not
>> expected to follow the moves of a remote branch of which the user is
>> not an owner: if the user needs to follow the brach and not lose its
>> commits, he/she should create a remote tracking branch.
> This would present another namespacing issue: how would you name the
> branches you're interested in so that they don't clash with your own
> personal local branches?  You'd have to invent a scheme which would
> encode the remote's name in a branch name.  But remote branches already
> do just this.  So you create a remote tracking branch when you intend
> to actually *develop* something on that branch with the final intention
> to push that work back.

But i am not interested in remote branches, they are just fetched automatically when i do "git fetch".  You cannot commit to a remote branch, and i think it is not common to checkout them without a "-b" option.  If i am interested in them, i name them somehow.  I think this is the only practical way if i do not want to chase reflogs, because the owner of the branch can reset or rebase it anytime.  I do not develop on tracking branches.  In fact, i am not even using "git pull".

>>> So I think it would be a lot more palatable if we kept reflogs on
>>> deleted branches. That, in turn, has a few open issues, such as how
>>> to manage namespace conflicts (e.g., the fact that a deleted "foo"
>>> branch can conflict with a new "foo/bar" branch).
>> 
>> I prefer to think of a remote branch and its local copy as the same
>> thing, which are physically different only because of current real
>> world/hardware/software limitations, which make it necessary to keep
>> a local cache of remote data.  With this approach, reflogs should be
>> deleted with the branch, and there will be no namespace conflicts.
> It appears, the distributed nature of a DVCS did not fully sink into
> your mindset yet. ;-)
> Looks like you mentally treat a Git remote as a thing being used to
> access a centralized "reference" server which maintains a master copy
> of a repository, of which you happen to also have a local copy.
> Then it's quite logically to think that if someone deleted a branch in
> the master copy, everyone "downstream" should have the same
> remote branch deleted to be in sync with that master copy.
> But this is not the only way to organize your work.
> You could fetch from someone else's repository and be interested in
> their branch "foo", but think what happens when you fetch next time from
> that repo and see Git happily deleting your local branch thatremote/foo
> simply because someone with push access deleted that branch from the
> repo.  This might *not* be what you really want or expect.

But this is true that the object store of Git can be viewed as a single centralized repository.  The fact that not everybody has access to every object in Git is a limitation and not a benefit.  These are the branches which are individual, and i do not think it is a good habit to treat every reference that was ever fetched with "git fetch" as your own, and put reflogs of all fetched remote branches under Git version control :D.

If i care about "thatremote/foo" branch, i "track" it, i do not plan to go through reflogs if it is rebased.

Alexey.

Re: Feature request: fetch --prune by default

[Alexey Muranov]

I just want to correct my mistake in what i've just sent:

On 19 Jul 2012, at 23:20, Alexey Muranov wrote:

> because the owner of the branch can reset or rebase it anytime.  I do not develop on tracking branches.  In fact, i am not even using "git pull".

> I do not develop on tracking branches.

Of course i develop on "tracking" branches, i just got confused once again by pull/push thing: i develop on branches that track origin, not upstream.
I think they should be called "remotely tracked branches", so there would be "remote tracking branches" for pull and "remotely tracked branches" for push.

Alexey.

Re: Feature request: fetch --prune by default

[Johannes Sixt]

Am 7/19/2012 23:20, schrieb Alexey Muranov:
> On 19 Jul 2012, at 19:34, Konstantin Khomoutov wrote:
> 
>> On Thu, 19 Jul 2012 18:21:21 +0200 Alexey Muranov
>> <alexey.muranov@gmail.com> wrote:
>> 
>> [...]
>>> I do not still understand very well some aspects of Git, like the 
>>> exact purpose of "remote tracking branches" (are they for pull or
>>> for push?), so i may be wrong.
>> This is wery well explained in the Pro Git book, for instance. And in
>> numerous blog posts etc.
> 
> I have read the Pro Gut book and numerous blog posts, but i keep
> forgetting the explanation because it does not make much sense to me:
> 
> "Tracking branches are local branches that have a direct relationship
> to a remote branch.  If you’re on a tracking branch and type git push,
> Git automatically knows which server and branch to push to.  Also,
> running git pull while on one of these branches fetches all the remote
> references and then automatically merges in the corresponding remote
> branch." etc.

Note the difference between "tracking branch" and "remote tracking
branch"! The "remote tracking branches" are the refs in the refs/remotes/
hierarchy. The "tracking branches" are your own local branches that you
have created with 'git branch topic thatremote/topic' (or perhaps 'git
checkout -b'). The paragraph talks about the latter.

-- Hannes

Re: Feature request: fetch --prune by default

[Alexey Muranov]

On 20 Jul 2012, at 09:11, Johannes Sixt wrote:

> Am 7/19/2012 23:20, schrieb Alexey Muranov:
>> On 19 Jul 2012, at 19:34, Konstantin Khomoutov wrote:
>> 
>>> On Thu, 19 Jul 2012 18:21:21 +0200 Alexey Muranov
>>> <alexey.muranov@gmail.com> wrote:
>>> 
>>> [...]
>>>> I do not still understand very well some aspects of Git, like the 
>>>> exact purpose of "remote tracking branches" (are they for pull or
>>>> for push?), so i may be wrong.
>>> This is wery well explained in the Pro Git book, for instance. And in
>>> numerous blog posts etc.
>> 
>> I have read the Pro Gut book and numerous blog posts, but i keep
>> forgetting the explanation because it does not make much sense to me:
>> 
>> "Tracking branches are local branches that have a direct relationship
>> to a remote branch.  If you’re on a tracking branch and type git push,
>> Git automatically knows which server and branch to push to.  Also,
>> running git pull while on one of these branches fetches all the remote
>> references and then automatically merges in the corresponding remote
>> branch." etc.
> 
> Note the difference between "tracking branch" and "remote tracking
> branch"! The "remote tracking branches" are the refs in the refs/remotes/
> hierarchy. The "tracking branches" are your own local branches that you
> have created with 'git branch topic thatremote/topic' (or perhaps 'git
> checkout -b'). The paragraph talks about the latter.

Hannes, thanks for the explanation, so i was confused once again.

Various blog posts do not make the terminology clear, for example
http://gitready.com/beginner/2009/03/09/remote-tracking-branches.html
sais that there are only "two types of branches: local, and remote-tracking", while i think it depends on perspective.
There are in fact
1. remote,
2. remote-tracking (which are local!),
3. truly local:
  a) which are tracking some remote-tracking(!) branches,
  b) and which are not tracking.

I think i was also misguided by Konstantin, who wrote that "you create a remote tracking branch when you intend to actually *develop* something on that branch" :).

-Alexey.

Re: Feature request: fetch --prune by default

[Junio C Hamano]

Alexey Muranov <alexey.muranov@gmail.com> writes:

> On 20 Jul 2012, at 09:11, Johannes Sixt wrote:
> ...
>> Note the difference between "tracking branch" and "remote tracking
>> branch"! The "remote tracking branches" are the refs in the refs/remotes/
>> hierarchy. The "tracking branches" are your own local branches that you
>> have created with 'git branch topic thatremote/topic' (or perhaps 'git
>> checkout -b'). The paragraph talks about the latter.
>
> Hannes, thanks for the explanation, so i was confused once again.
>
> Various blog posts do not make the terminology clear, for example
> http://gitready.com/beginner/2009/03/09/remote-tracking-branches.html
> sais that there are only "two types of branches: local, and remote-tracking"...
> ...
> I think i was also misguided by Konstantin, who wrote that "you
> create a remote tracking branch when you intend to actually
> *develop* something on that branch" :).

I was re-reading the backlog today, and saw this topic fizzled out.

We obviously cannot fix third-party documentation that teach lies to
people, but is there something we can do to improve our own
documentation with respect to this confusion?

As I wrote it elsewhere, I try to avoid the bareword "tracking" in
general, and call the local branch you build on something like "your
'next' branch that forked from origin/next remote tracking branch"
myself.  Perhaps we can start from checking the documentation with
such a phrasing discipline?

Re: Feature request: fetch --prune by default

[Alexey Muranov]

On 19 Jul 2012, at 13:55, Jeff King wrote:

> I agree it would be much less confusing. However, one downside is that
> we do not keep reflogs on deleted branches (and nor did the commits in
> remote branches necessarily make it into the HEAD reflog). That makes
> "git fetch" a potentially destructive operation (you irrevocably lose
> the notion of which remote branches pointed where before the fetch, and
> you open up new commits to immediate pruning by "gc --auto".

If i understand correctly, existence of a reflog entry will not stop "gc" from removing a commit, will it?
In this case, if a remote branch was rebased or reset, commits can be lost anyway, right?

Alexey.

Re: Feature request: fetch --prune by default

[Dan Johnson]

On Thu, Jul 19, 2012 at 12:40 PM, Alexey Muranov
<alexey.muranov@gmail.com> wrote:
> On 19 Jul 2012, at 13:55, Jeff King wrote:
>
>> I agree it would be much less confusing. However, one downside is that
>> we do not keep reflogs on deleted branches (and nor did the commits in
>> remote branches necessarily make it into the HEAD reflog). That makes
>> "git fetch" a potentially destructive operation (you irrevocably lose
>> the notion of which remote branches pointed where before the fetch, and
>> you open up new commits to immediate pruning by "gc --auto".
>
> If i understand correctly, existence of a reflog entry will not stop "gc" from removing a commit, will it?
> In this case, if a remote branch was rebased or reset, commits can be lost anyway, right?
From the git-gc man page:
git gc tries very hard to be safe about the garbage it collects. In
particular, it will keep not only objects referenced by your current
set of branches and tags, but also objects referenced by the index,
remote-tracking branches, refs saved by git filter-branch in
refs/original/, or reflogs (which may reference commits in branches
that were later amended or rewound).

So yes, a reflog entry does stop gc from removing objects, including
commits. It will expire old reflog entries (90 days by default)
though, so it's not like they will stay around forever.
-- 
-Dan

Re: Feature request: fetch --prune by default

[Alexey Muranov]

On 19 Jul 2012, at 18:48, Dan Johnson wrote:

> From the git-gc man page:
> git gc tries very hard to be safe about the garbage it collects. In
> particular, it will keep not only objects referenced by your current
> set of branches and tags, but also objects referenced by the index,
> remote-tracking branches, refs saved by git filter-branch in
> refs/original/, or reflogs (which may reference commits in branches
> that were later amended or rewound).
> 
> So yes, a reflog entry does stop gc from removing objects, including
> commits. It will expire old reflog entries (90 days by default)
> though, so it's not like they will stay around forever.

Dan, thanks for the explanation.

Alexey.

[RFC/PATCH 0/3] reflog graveyard

[Jeff King]

On Thu, Jul 19, 2012 at 07:55:58AM -0400, Jeff King wrote:

> So I think it would be a lot more palatable if we kept reflogs on
> deleted branches. That, in turn, has a few open issues, such as how to
> manage namespace conflicts (e.g., the fact that a deleted "foo" branch
> can conflict with a new "foo/bar" branch).

Here is a patch series to address that. I think I have smoothed out most
of the rough edges, but I wouldn't be surprised if there are some other
corner cases. One that I notice is that "git log -g" will stop walking
when it hits a null sha1 in the reflog.

  [1/3]: retain reflogs for deleted refs
  [2/3]: teach sha1_name to look in graveyard reflogs
  [3/3]: add tests for reflogs of deleted refs

-Peff

[PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

When a ref is deleted, we completely delete its reflog on
the spot, leaving very little help for the user to reverse
the action. One can sometimes reconstruct the missing
entries based on the HEAD reflog, but not always; the
deleted entries may not have ever been on HEAD (for example,
in the case of a refs/remotes branch that was pruned). That
leaves "git fsck --lost-found", which can be quite tedious.

Instead, let's keep the reflogs for deleted refs around
until their entries naturally expire according to the
regular reflog expiration rules.

This cannot be done by simply leaving the reflog files in
place. The ref namespace does not allow D/F conflicts, so a
ref "foo" would block the creation of another ref "foo/bar",
and vice versa. This limitation is acceptable for two refs
to exist simultaneously, but should not have an impact if
one of the refs is deleted.

This patch moves reflog entries into a special "graveyard"
namespace, and appends a tilde (~) character, which is
not allowed in a valid ref name. This means that the deleted
reflogs of these refs:

   refs/heads/a
   refs/heads/a/b
   refs/heads/a/b/c

will be stored in:

   logs/graveyard/refs/heads/a~
   logs/graveyard/refs/heads/a/b~
   logs/graveyard/refs/heads/a/b/c~

Putting them in the graveyard namespace ensures they will
not conflict with live refs, and the tilde prevents D/F
conflicts within the graveyard namespace.

The implementation is fairly straightforward, but it's worth
noting a few things:

  1. Updates to "logs/graveyard/refs/heads/foo~" happen
     under the ref-lock for "refs/heads/foo". So deletion
     still takes a single lock, and anyone touching the
     reflog directly needs to reverse the transformation to
     find the correct lockfile.

  2. We append entries to the graveyard reflog rather than
     simply renaming the file into place. This means that
     if you create and delete a branch repeatedly, the
     graveyard will contain the concatenation of all
     iterations.

  3. We do not resurrect dead entries when a new ref is
     created with the same name. However, it would be
     possible to build an "undelete" feature on top of this
     if one was so inclined.

  4. The for_each_reflog code has been loosened to allow
     reflogs that do not have a matching ref. In this case,
     the callback is passed the null_sha1, and callers must
     be prepared to handle this case (the only caller that
     cares is the reflog expiration code, which is updated
     here).

Only one test needed to be updated; t7701 tries to create
unreachable objects by deleting branches. Of course that no
longer works, which is the intent of this patch. The test
now works around it by removing the graveyard logs.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/reflog.c                     |  9 +++--
 refs.c                               | 69 +++++++++++++++++++++++++++++++++---
 refs.h                               |  3 ++
 t/t7701-repack-unpack-unreachable.sh |  5 ++-
 4 files changed, 79 insertions(+), 7 deletions(-)

diff --git a/builtin/reflog.c b/builtin/reflog.c
index b3c9e27..e79a2ca 100644
--- a/builtin/reflog.c
+++ b/builtin/reflog.c
@@ -359,6 +359,7 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,
 	struct commit *tip_commit;
 	struct commit_list *tips;
 	int status = 0;
+	int updateref = cmd->updateref && !is_null_sha1(sha1);
 
 	memset(&cb, 0, sizeof(cb));
 
@@ -367,6 +368,10 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,
 	 * getting updated.
 	 */
 	lock = lock_any_ref_for_update(ref, sha1, 0);
+	if (!lock && is_null_sha1(sha1))
+		lock = lock_any_ref_for_update(
+				graveyard_reflog_to_refname(ref),
+				sha1, 0);
 	if (!lock)
 		return error("cannot lock ref '%s'", ref);
 	log_file = git_pathdup("logs/%s", ref);
@@ -426,7 +431,7 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,
 			status |= error("%s: %s", strerror(errno),
 					newlog_path);
 			unlink(newlog_path);
-		} else if (cmd->updateref &&
+		} else if (updateref &&
 			(write_in_full(lock->lock_fd,
 				sha1_to_hex(cb.last_kept_sha1), 40) != 40 ||
 			 write_str_in_full(lock->lock_fd, "\n") != 1 ||
@@ -438,7 +443,7 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,
 			status |= error("cannot rename %s to %s",
 					newlog_path, log_file);
 			unlink(newlog_path);
-		} else if (cmd->updateref && commit_ref(lock)) {
+		} else if (updateref && commit_ref(lock)) {
 			status |= error("Couldn't set %s", lock->ref_name);
 		} else {
 			adjust_shared_perm(log_file);
diff --git a/refs.c b/refs.c
index da74a2b..553de77 100644
--- a/refs.c
+++ b/refs.c
@@ -4,6 +4,8 @@
 #include "tag.h"
 #include "dir.h"
 
+static void mark_reflog_deleted(struct ref_lock *lock);
+
 /*
  * Make sure "ref" is something reasonable to have under ".git/refs/";
  * We do not like it if:
@@ -1780,7 +1782,7 @@ int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
 	 */
 	ret |= repack_without_ref(refname);
 
-	unlink_or_warn(git_path("logs/%s", lock->ref_name));
+	mark_reflog_deleted(lock);
 	invalidate_ref_cache(NULL);
 	unlock_ref(lock);
 	return ret;
@@ -2385,9 +2387,8 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data
 			} else {
 				unsigned char sha1[20];
 				if (read_ref_full(name->buf, sha1, 0, NULL))
-					retval = error("bad ref for %s", name->buf);
-				else
-					retval = fn(name->buf, sha1, 0, cb_data);
+					hashcpy(sha1, null_sha1);
+				retval = fn(name->buf, sha1, 0, cb_data);
 			}
 			if (retval)
 				break;
@@ -2552,3 +2553,63 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 	free(short_name);
 	return xstrdup(refname);
 }
+
+char *refname_to_graveyard_reflog(const char *ref)
+{
+	return git_path("logs/graveyard/%s~", ref);
+}
+
+char *graveyard_reflog_to_refname(const char *log)
+{
+	static struct strbuf buf = STRBUF_INIT;
+
+	if (!prefixcmp(log, "graveyard/"))
+		log += 10;
+
+	strbuf_reset(&buf);
+	strbuf_addstr(&buf, log);
+	if (buf.len > 0 && buf.buf[buf.len-1] == '~')
+		strbuf_setlen(&buf, buf.len - 1);
+
+	return buf.buf;
+}
+
+static int copy_reflog_entries(const char *dst, const char *src)
+{
+	int fdi, fdo, status;
+
+	fdi = open(src, O_RDONLY);
+	if (fdi < 0)
+		return errno == ENOENT ? 0 : -1;
+
+	fdo = open(dst, O_WRONLY | O_APPEND | O_CREAT, 0666);
+	if (fdo < 0) {
+		close(fdi);
+		return -1;
+	}
+
+	status = copy_fd(fdi, fdo);
+	if (close(fdo) < 0)
+		return -1;
+	if (status < 0 || adjust_shared_perm(dst) < 0)
+		return -1;
+	return 0;
+}
+
+static void mark_reflog_deleted(struct ref_lock *lock)
+{
+	static const char msg[] = "ref deleted";
+	const char *log = git_path("logs/%s", lock->ref_name);
+	char *grave = refname_to_graveyard_reflog(lock->ref_name);
+
+	if (log_ref_write(lock->ref_name, lock->old_sha1, null_sha1, msg) < 0)
+		warning("unable to update reflog for %s: %s",
+			lock->ref_name, strerror(errno));
+
+	if (safe_create_leading_directories(grave) < 0 ||
+	    copy_reflog_entries(grave, log) < 0)
+		warning("unable to copy reflog entries to graveyard: %s",
+			strerror(errno));
+
+	unlink_or_warn(log);
+}
diff --git a/refs.h b/refs.h
index d6c2fe2..9d14558 100644
--- a/refs.h
+++ b/refs.h
@@ -111,6 +111,9 @@ int for_each_recent_reflog_ent(const char *refname, each_reflog_ent_fn fn, long,
  */
 extern int for_each_reflog(each_ref_fn, void *);
 
+char *refname_to_graveyard_reflog(const char *ref);
+char *graveyard_reflog_to_refname(const char *log);
+
 #define REFNAME_ALLOW_ONELEVEL 1
 #define REFNAME_REFSPEC_PATTERN 2
 #define REFNAME_DOT_COMPONENT 4
diff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh
index b8d4cde..c06b715 100755
--- a/t/t7701-repack-unpack-unreachable.sh
+++ b/t/t7701-repack-unpack-unreachable.sh
@@ -38,7 +38,9 @@ test_expect_success '-A with -d option leaves unreachable objects unpacked' '
 	git show $csha1 &&
 	git show $tsha1 &&
 	# now expire the reflog, while keeping reachable ones but expiring
-	# unreachables immediately
+	# unreachables immediately; also remove any graveyard reflogs
+	# from deleted branches that would keep things reachable
+	rm -rf .git/logs/graveyard &&
 	test_tick &&
 	sometimeago=$(( $test_tick - 10000 )) &&
 	git reflog expire --expire=$sometimeago --expire-unreachable=$test_tick --all &&
@@ -76,6 +78,7 @@ test_expect_success '-A without -d option leaves unreachable objects packed' '
 	test 1 = $(ls -1 .git/objects/pack/pack-*.pack | wc -l) &&
 	packfile=$(ls .git/objects/pack/pack-*.pack) &&
 	git branch -D transient_branch &&
+	rm -rf .git/logs/graveyard &&
 	test_tick &&
 	git repack -A -l &&
 	test ! -f "$fsha1path" &&
-- 
1.7.10.5.40.g059818d

Re: [PATCH 1/3] retain reflogs for deleted refs

[Alexey Muranov]

Jeff,

i have no idea about Git source and little idea of how it is working internally, but reading through your message i wonder: wouldn't it be a good idea to timestamp the dead reflogs ?

Alexey.

On 19 Jul 2012, at 23:33, Jeff King wrote:

> When a ref is deleted, we completely delete its reflog on
> the spot, leaving very little help for the user to reverse
> the action. One can sometimes reconstruct the missing
> entries based on the HEAD reflog, but not always; the
> deleted entries may not have ever been on HEAD (for example,
> in the case of a refs/remotes branch that was pruned). That
> leaves "git fsck --lost-found", which can be quite tedious.
> 
> Instead, let's keep the reflogs for deleted refs around
> until their entries naturally expire according to the
> regular reflog expiration rules.
> 
> This cannot be done by simply leaving the reflog files in
> place. The ref namespace does not allow D/F conflicts, so a
> ref "foo" would block the creation of another ref "foo/bar",
> and vice versa. This limitation is acceptable for two refs
> to exist simultaneously, but should not have an impact if
> one of the refs is deleted.
> 
> This patch moves reflog entries into a special "graveyard"
> namespace, and appends a tilde (~) character, which is
> not allowed in a valid ref name. This means that the deleted
> reflogs of these refs:
> 
>   refs/heads/a
>   refs/heads/a/b
>   refs/heads/a/b/c
> 
> will be stored in:
> 
>   logs/graveyard/refs/heads/a~
>   logs/graveyard/refs/heads/a/b~
>   logs/graveyard/refs/heads/a/b/c~
> 
> Putting them in the graveyard namespace ensures they will
> not conflict with live refs, and the tilde prevents D/F
> conflicts within the graveyard namespace.
> 
> The implementation is fairly straightforward, but it's worth
> noting a few things:
> 
>  1. Updates to "logs/graveyard/refs/heads/foo~" happen
>     under the ref-lock for "refs/heads/foo". So deletion
>     still takes a single lock, and anyone touching the
>     reflog directly needs to reverse the transformation to
>     find the correct lockfile.
> 
>  2. We append entries to the graveyard reflog rather than
>     simply renaming the file into place. This means that
>     if you create and delete a branch repeatedly, the
>     graveyard will contain the concatenation of all
>     iterations.
> 
>  3. We do not resurrect dead entries when a new ref is
>     created with the same name. However, it would be
>     possible to build an "undelete" feature on top of this
>     if one was so inclined.
> 
>  4. The for_each_reflog code has been loosened to allow
>     reflogs that do not have a matching ref. In this case,
>     the callback is passed the null_sha1, and callers must
>     be prepared to handle this case (the only caller that
>     cares is the reflog expiration code, which is updated
>     here).
> 
> Only one test needed to be updated; t7701 tries to create
> unreachable objects by deleting branches. Of course that no
> longer works, which is the intent of this patch. The test
> now works around it by removing the graveyard logs.
> 
> Signed-off-by: Jeff King <peff@peff.net>
> ---
> builtin/reflog.c                     |  9 +++--
> refs.c                               | 69 +++++++++++++++++++++++++++++++++---
> refs.h                               |  3 ++
> t/t7701-repack-unpack-unreachable.sh |  5 ++-
> 4 files changed, 79 insertions(+), 7 deletions(-)
> 
> diff --git a/builtin/reflog.c b/builtin/reflog.c
> index b3c9e27..e79a2ca 100644
> --- a/builtin/reflog.c
> +++ b/builtin/reflog.c
> @@ -359,6 +359,7 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,
> 	struct commit *tip_commit;
> 	struct commit_list *tips;
> 	int status = 0;
> +	int updateref = cmd->updateref && !is_null_sha1(sha1);
> 
> 	memset(&cb, 0, sizeof(cb));
> 
> @@ -367,6 +368,10 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,
> 	 * getting updated.
> 	 */
> 	lock = lock_any_ref_for_update(ref, sha1, 0);
> +	if (!lock && is_null_sha1(sha1))
> +		lock = lock_any_ref_for_update(
> +				graveyard_reflog_to_refname(ref),
> +				sha1, 0);
> 	if (!lock)
> 		return error("cannot lock ref '%s'", ref);
> 	log_file = git_pathdup("logs/%s", ref);
> @@ -426,7 +431,7 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,
> 			status |= error("%s: %s", strerror(errno),
> 					newlog_path);
> 			unlink(newlog_path);
> -		} else if (cmd->updateref &&
> +		} else if (updateref &&
> 			(write_in_full(lock->lock_fd,
> 				sha1_to_hex(cb.last_kept_sha1), 40) != 40 ||
> 			 write_str_in_full(lock->lock_fd, "\n") != 1 ||
> @@ -438,7 +443,7 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,
> 			status |= error("cannot rename %s to %s",
> 					newlog_path, log_file);
> 			unlink(newlog_path);
> -		} else if (cmd->updateref && commit_ref(lock)) {
> +		} else if (updateref && commit_ref(lock)) {
> 			status |= error("Couldn't set %s", lock->ref_name);
> 		} else {
> 			adjust_shared_perm(log_file);
> diff --git a/refs.c b/refs.c
> index da74a2b..553de77 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -4,6 +4,8 @@
> #include "tag.h"
> #include "dir.h"
> 
> +static void mark_reflog_deleted(struct ref_lock *lock);
> +
> /*
>  * Make sure "ref" is something reasonable to have under ".git/refs/";
>  * We do not like it if:
> @@ -1780,7 +1782,7 @@ int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
> 	 */
> 	ret |= repack_without_ref(refname);
> 
> -	unlink_or_warn(git_path("logs/%s", lock->ref_name));
> +	mark_reflog_deleted(lock);
> 	invalidate_ref_cache(NULL);
> 	unlock_ref(lock);
> 	return ret;
> @@ -2385,9 +2387,8 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data
> 			} else {
> 				unsigned char sha1[20];
> 				if (read_ref_full(name->buf, sha1, 0, NULL))
> -					retval = error("bad ref for %s", name->buf);
> -				else
> -					retval = fn(name->buf, sha1, 0, cb_data);
> +					hashcpy(sha1, null_sha1);
> +				retval = fn(name->buf, sha1, 0, cb_data);
> 			}
> 			if (retval)
> 				break;
> @@ -2552,3 +2553,63 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
> 	free(short_name);
> 	return xstrdup(refname);
> }
> +
> +char *refname_to_graveyard_reflog(const char *ref)
> +{
> +	return git_path("logs/graveyard/%s~", ref);
> +}
> +
> +char *graveyard_reflog_to_refname(const char *log)
> +{
> +	static struct strbuf buf = STRBUF_INIT;
> +
> +	if (!prefixcmp(log, "graveyard/"))
> +		log += 10;
> +
> +	strbuf_reset(&buf);
> +	strbuf_addstr(&buf, log);
> +	if (buf.len > 0 && buf.buf[buf.len-1] == '~')
> +		strbuf_setlen(&buf, buf.len - 1);
> +
> +	return buf.buf;
> +}
> +
> +static int copy_reflog_entries(const char *dst, const char *src)
> +{
> +	int fdi, fdo, status;
> +
> +	fdi = open(src, O_RDONLY);
> +	if (fdi < 0)
> +		return errno == ENOENT ? 0 : -1;
> +
> +	fdo = open(dst, O_WRONLY | O_APPEND | O_CREAT, 0666);
> +	if (fdo < 0) {
> +		close(fdi);
> +		return -1;
> +	}
> +
> +	status = copy_fd(fdi, fdo);
> +	if (close(fdo) < 0)
> +		return -1;
> +	if (status < 0 || adjust_shared_perm(dst) < 0)
> +		return -1;
> +	return 0;
> +}
> +
> +static void mark_reflog_deleted(struct ref_lock *lock)
> +{
> +	static const char msg[] = "ref deleted";
> +	const char *log = git_path("logs/%s", lock->ref_name);
> +	char *grave = refname_to_graveyard_reflog(lock->ref_name);
> +
> +	if (log_ref_write(lock->ref_name, lock->old_sha1, null_sha1, msg) < 0)
> +		warning("unable to update reflog for %s: %s",
> +			lock->ref_name, strerror(errno));
> +
> +	if (safe_create_leading_directories(grave) < 0 ||
> +	    copy_reflog_entries(grave, log) < 0)
> +		warning("unable to copy reflog entries to graveyard: %s",
> +			strerror(errno));
> +
> +	unlink_or_warn(log);
> +}
> diff --git a/refs.h b/refs.h
> index d6c2fe2..9d14558 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -111,6 +111,9 @@ int for_each_recent_reflog_ent(const char *refname, each_reflog_ent_fn fn, long,
>  */
> extern int for_each_reflog(each_ref_fn, void *);
> 
> +char *refname_to_graveyard_reflog(const char *ref);
> +char *graveyard_reflog_to_refname(const char *log);
> +
> #define REFNAME_ALLOW_ONELEVEL 1
> #define REFNAME_REFSPEC_PATTERN 2
> #define REFNAME_DOT_COMPONENT 4
> diff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh
> index b8d4cde..c06b715 100755
> --- a/t/t7701-repack-unpack-unreachable.sh
> +++ b/t/t7701-repack-unpack-unreachable.sh
> @@ -38,7 +38,9 @@ test_expect_success '-A with -d option leaves unreachable objects unpacked' '
> 	git show $csha1 &&
> 	git show $tsha1 &&
> 	# now expire the reflog, while keeping reachable ones but expiring
> -	# unreachables immediately
> +	# unreachables immediately; also remove any graveyard reflogs
> +	# from deleted branches that would keep things reachable
> +	rm -rf .git/logs/graveyard &&
> 	test_tick &&
> 	sometimeago=$(( $test_tick - 10000 )) &&
> 	git reflog expire --expire=$sometimeago --expire-unreachable=$test_tick --all &&
> @@ -76,6 +78,7 @@ test_expect_success '-A without -d option leaves unreachable objects packed' '
> 	test 1 = $(ls -1 .git/objects/pack/pack-*.pack | wc -l) &&
> 	packfile=$(ls .git/objects/pack/pack-*.pack) &&
> 	git branch -D transient_branch &&
> +	rm -rf .git/logs/graveyard &&
> 	test_tick &&
> 	git repack -A -l &&
> 	test ! -f "$fsha1path" &&
> -- 
> 1.7.10.5.40.g059818d
> 

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Fri, Jul 20, 2012 at 12:23:12AM +0200, Alexey Muranov wrote:

> i have no idea about Git source and little idea of how it is working
> internally, but reading through your message i wonder: wouldn't it be
> a good idea to timestamp the dead reflogs ?

Each individual entry in the reflog has its own timestamp, and the
entries are expired individually over time as "git gc" is run. Or did
you mean something else?

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Alexey Muranov]

On 20 Jul 2012, at 16:26, Jeff King wrote:

> On Fri, Jul 20, 2012 at 12:23:12AM +0200, Alexey Muranov wrote:
> 
>> i have no idea about Git source and little idea of how it is working
>> internally, but reading through your message i wonder: wouldn't it be
>> a good idea to timestamp the dead reflogs ?
> 
> Each individual entry in the reflog has its own timestamp, and the
> entries are expired individually over time as "git gc" is run. Or did
> you mean something else?

Yes, sorry, i was not clear, i meant to put dead reflogs into subdirectories yyyy-mm-dd, or maybe yyyy-mm-dd-hhmmss.

-Alexey.

Re: [PATCH 1/3] retain reflogs for deleted refs

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Only one test needed to be updated; t7701 tries to create
> unreachable objects by deleting branches. Of course that no
> longer works, which is the intent of this patch. The test
> now works around it by removing the graveyard logs.

I think the work-around indicates the need for regular users to be
able to also discover, prune and delete these logs.  Do we have
"prune reflog for _this_ ref (or these refs), removing entries that
are older than this threshold"?  If so the codepath would need to
know about the graveyard and the implementation detail of the tilde
suffix so that the end users do not need to know about them.

I like the general direction.  Perhaps a long distant future
direction could be to also use the same trick in the ref namespace
so that we can have 'next' branch itself, and 'next/foo', 'next/bar'
forks that are based on the 'next' branch at the same time (it
obviously is a totally unrelated topic)?

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Thu, Jul 19, 2012 at 03:36:09PM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > Only one test needed to be updated; t7701 tries to create
> > unreachable objects by deleting branches. Of course that no
> > longer works, which is the intent of this patch. The test
> > now works around it by removing the graveyard logs.
> 
> I think the work-around indicates the need for regular users to be
> able to also discover, prune and delete these logs.  Do we have
> "prune reflog for _this_ ref (or these refs), removing entries that
> are older than this threshold"?  If so the codepath would need to
> know about the graveyard and the implementation detail of the tilde
> suffix so that the end users do not need to know about them.

We do have it: "git reflog expire --expire=now deleted-branch" is the
right way to do it. Unfortunately, it does not work with my patch. The
dwim_log correctly notes that a reflog exists (because it checks that
the "graveyard" version of the ref exists), but then expire_reflog does
not correctly fallback when opening the log (it usually has to do the
_reverse_ translation, because it gets the graveyard log name from
for_each_reflog, and has to find the correct lock).

I'll fix it in my re-roll, and then have t7701 use it.

> I like the general direction.  Perhaps a long distant future
> direction could be to also use the same trick in the ref namespace
> so that we can have 'next' branch itself, and 'next/foo', 'next/bar'
> forks that are based on the 'next' branch at the same time (it
> obviously is a totally unrelated topic)?

I would love that, as it would mean we could simply leave the reflogs in
place without having a separate graveyard namespace. Which means there
wouldn't need to be any reflog-specific translation at all, and bugs
like the one above wouldn't exist.

But it would mean that you cannot naively run

  echo $sha1 >.git/refs/heads/foo

anymore. I suspect that the packed-refs conversion rooted out many
scripts that did not use update-ref and rev-parse to access refs, but
the above does still work today. So I suspect there would be some
fallout. Not to mention that older versions of git would be completely
broken, which would mean we need a lengthy deprecation period while
everybody upgrades to versions of git that support the reading side.

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Fri, Jul 20, 2012 at 10:43:37AM -0400, Jeff King wrote:

> > I think the work-around indicates the need for regular users to be
> > able to also discover, prune and delete these logs.  Do we have
> > "prune reflog for _this_ ref (or these refs), removing entries that
> > are older than this threshold"?  If so the codepath would need to
> > know about the graveyard and the implementation detail of the tilde
> > suffix so that the end users do not need to know about them.
> 
> We do have it: "git reflog expire --expire=now deleted-branch" is the
> right way to do it. Unfortunately, it does not work with my patch. The
> dwim_log correctly notes that a reflog exists (because it checks that
> the "graveyard" version of the ref exists), but then expire_reflog does
> not correctly fallback when opening the log (it usually has to do the
> _reverse_ translation, because it gets the graveyard log name from
> for_each_reflog, and has to find the correct lock).
> 
> I'll fix it in my re-roll, and then have t7701 use it.

I noticed I ignored the "discover" and "delete" parts of your paragraph.
As far as deletion goes, I think we can ignore it; expiring all entries
is equivalent.

Discovery is harder. Certainly these should not show up in normal
ref-listing output. I'd be content to leave them slightly hidden as a
first step, and people who know they are looking for the pre-deletion
contents of the "foo" branch can access it by name. Probably a second
step would be a fancier interface to help with listing and resurrecting
dead branches, possibly including branch config.

In other words, I want to focus on getting the ref-level plumbing right,
and then we can care about the porcelain later.

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I noticed I ignored the "discover" and "delete" parts of your paragraph.
> As far as deletion goes, I think we can ignore it; expiring all entries
> is equivalent.
> ...
> In other words, I want to focus on getting the ref-level plumbing right,
> and then we can care about the porcelain later.

Yeah, I agree that is a reasonable way forward.  for-each-ref with a
new option (--include-dead or something) can wait.

Re: [PATCH 1/3] retain reflogs for deleted refs

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> But it would mean that you cannot naively run
>
>   echo $sha1 >.git/refs/heads/foo
>
> anymore. I suspect that the packed-refs conversion rooted out many
> scripts that did not use update-ref and rev-parse to access refs, but
> the above does still work today. So I suspect there would be some
> fallout. Not to mention that older versions of git would be completely
> broken, which would mean we need a lengthy deprecation period while
> everybody upgrades to versions of git that support the reading side.

We have that "core.repositoryversion" thing, so we could treat it
just like "update-index --index-version 4" to make it a "flag day
event for each repository, on the day of end-user's choice".

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Fri, Jul 20, 2012 at 08:42:57AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > But it would mean that you cannot naively run
> >
> >   echo $sha1 >.git/refs/heads/foo
> >
> > anymore. I suspect that the packed-refs conversion rooted out many
> > scripts that did not use update-ref and rev-parse to access refs, but
> > the above does still work today. So I suspect there would be some
> > fallout. Not to mention that older versions of git would be completely
> > broken, which would mean we need a lengthy deprecation period while
> > everybody upgrades to versions of git that support the reading side.
> 
> We have that "core.repositoryversion" thing, so we could treat it
> just like "update-index --index-version 4" to make it a "flag day
> event for each repository, on the day of end-user's choice".

True. The code to handle both cases would be pretty nasty, though,
mostly because we do not isolate the filesystem calls at all right now
(i.e., there are a lot of calls to git_path("logs/%s", refname) in the
code. Which is probably not too bad, but there are a lot of implicit
reverse-conversions (e.g., walking the hierarchy and assuming that the
path you find is a refname).

If we are seriously considering doing this for the full refs namespace
anytime soon, then I'd be tempted to hold off the reflog graveyard until
then.  The code would be a lot simpler and less error-prone if we
didn't have to convert between the namespaces (you would simply not get
the reflog retention behavior in the old repositoryformatversion).

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> I like the general direction.  Perhaps a long distant future
> direction could be to also use the same trick in the ref namespace
> so that we can have 'next' branch itself, and 'next/foo', 'next/bar'
> forks that are based on the 'next' branch at the same time (it
> obviously is a totally unrelated topic)?

I notice that I was responsible for making this topic veer in the
wrong direction by bringing up a new feature "having 'next' and
'next/bar' at the same time" which nobody asked.  Perhaps we can
drop that for now to simplify the scope of the topic, to bring the
log graveyard back on track?

Re: [PATCH 1/3] retain reflogs for deleted refs

[Michael Haggerty]

On 07/19/2012 11:33 PM, Jeff King wrote:
> [...]
> This cannot be done by simply leaving the reflog files in
> place. The ref namespace does not allow D/F conflicts, so a
> ref "foo" would block the creation of another ref "foo/bar",
> and vice versa. This limitation is acceptable for two refs
> to exist simultaneously, but should not have an impact if
> one of the refs is deleted.

This is a great feature.

> This patch moves reflog entries into a special "graveyard"
> namespace, and appends a tilde (~) character, which is
> not allowed in a valid ref name. This means that the deleted
> reflogs of these refs:
>
>     refs/heads/a
>     refs/heads/a/b
>     refs/heads/a/b/c
>
> will be stored in:
>
>     logs/graveyard/refs/heads/a~
>     logs/graveyard/refs/heads/a/b~
>     logs/graveyard/refs/heads/a/b/c~
>
> Putting them in the graveyard namespace ensures they will
> not conflict with live refs, and the tilde prevents D/F
> conflicts within the graveyard namespace.

I agree with Junio that long-term, it would be nice to allow references 
"foo" and "foo/bar" to exist simultaneously.  To get there, we would 
have to redesign the mapping between reference names and the filenames 
used for the references and for the reflogs.

The easiest thing would be to mark files and directories differently; 
something like

     $GIT_DIR/{,logs/}refs/heads/a/b/c~

or

     $GIT_DIR/{,logs/}refs/heads~/a~/b~/c

i.e., munging either directory or file names to strings that are illegal 
in refnames such that it is unambiguous from the name whether a path is 
a file or directory.

And *if* we did that, then we wouldn't need a separate "graveyard" 
namespace, would we?  The reflogs for dead references could live among 
those for living references.

Therefore, I think it would be good if we would choose a convention now 
for dead reflogs that is compatible with this hoped-for future.

The first convention, "logs/refs/heads/a/b/c~" is not usable because a 
reflog for a dead reference with this name would conflict with a reflog 
for a live reference "heads/a" or "heads/a/b" that uses the current 
filename convention.

But the second convention, "logs/refs/heads~/a~/b~/c, cannot conflict 
with current reflog files.  And it would be a step towards allowing 
"foo" and "foo/bar" at the same time.  What do you think about using a 
convention like this instead of the one that you proposed?


Another minor concern is the choice of trailing tilde in the file or 
directory names.  Given that emacs creates backup files by appending a 
tilde to the filename, (1) it would be easy to inadvertently create such 
files, which git might try to interpret as reflogs and (2) there might 
be tools that innately "know" to skip such files in their processing. 
ack-grep, a replacement for grep, is an example that springs to mind.  I 
know that I have written backup scripts that ignore files matching "*~", 
and a garbage-removal script that removes files matching "*~".  Probably 
it is less precarious to name directories rather than files with 
trailing tildes, but either one could be a surprise for sysadmins.

Other possibilities (according to git-check-ref-format(1)):

     refs/.heads/.a/.b/c
     refs/heads./a./b./c (problematic on some Windows filesystems?)
     refs/heads../a../b../c
     refs/heads~dir/a~dir/b~dir/c (or some other suffix)
     refs/heads..a..b..c (not recommended because it flattens directory 
hierarchy)

> The implementation is fairly straightforward, but it's worth
> noting a few things:
>
>    1. Updates to "logs/graveyard/refs/heads/foo~" happen
>       under the ref-lock for "refs/heads/foo". So deletion
>       still takes a single lock, and anyone touching the
>       reflog directly needs to reverse the transformation to
>       find the correct lockfile.

This should be documented in the code.

>    2. We append entries to the graveyard reflog rather than
>       simply renaming the file into place. This means that
>       if you create and delete a branch repeatedly, the
>       graveyard will contain the concatenation of all
>       iterations.

Good.

>    3. We do not resurrect dead entries when a new ref is
>       created with the same name. However, it would be
>       possible to build an "undelete" feature on top of this
>       if one was so inclined.

Nice prospect.

> [...]> diff --git a/refs.c b/refs.c
> index da74a2b..553de77 100644
> --- a/refs.c
> +++ b/refs.c
> [...]
> @@ -2552,3 +2553,63 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
>   	free(short_name);
>   	return xstrdup(refname);
>   }
> +
> +char *refname_to_graveyard_reflog(const char *ref)
> +{
> +	return git_path("logs/graveyard/%s~", ref);
> +}
> +
> +char *graveyard_reflog_to_refname(const char *log)
> +{
> +	static struct strbuf buf = STRBUF_INIT;
> +
> +	if (!prefixcmp(log, "graveyard/"))
> +		log += 10;
> +
> +	strbuf_reset(&buf);
> +	strbuf_addstr(&buf, log);
> +	if (buf.len > 0 && buf.buf[buf.len-1] == '~')
> +		strbuf_setlen(&buf, buf.len - 1);
> +
> +	return buf.buf;
> +}

Given the names of these two functions, I was surprised that they aren't 
inverses of each other.

Function comments would be nice, too, especially for the latter.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Fri, Jul 20, 2012 at 11:49:07AM +0200, Michael Haggerty wrote:

> >This patch moves reflog entries into a special "graveyard"
> >namespace, and appends a tilde (~) character, which is
> >not allowed in a valid ref name. This means that the deleted
> >reflogs of these refs:
> >
> >    refs/heads/a
> >    refs/heads/a/b
> >    refs/heads/a/b/c
> >
> >will be stored in:
> >
> >    logs/graveyard/refs/heads/a~
> >    logs/graveyard/refs/heads/a/b~
> >    logs/graveyard/refs/heads/a/b/c~
> >
> >Putting them in the graveyard namespace ensures they will
> >not conflict with live refs, and the tilde prevents D/F
> >conflicts within the graveyard namespace.
> 
> I agree with Junio that long-term, it would be nice to allow
> references "foo" and "foo/bar" to exist simultaneously.  To get
> there, we would have to redesign the mapping between reference names
> and the filenames used for the references and for the reflogs.

Yes, I would really like that, as it could make the alternate namespace
go away, which is the source of about half the code in my patches (i.e.,
we would only need to loosen the reflog reading code to handle reflogs
that do not have a matching ref).

But I fear that the fallouts from that will be much, much larger. Even
with just this change, older versions of git will be slightly unhappy
(e.g., you will get some extra warnings during fsck and reflog
expiration about these reflogs). But changing the on-disk representation
of the refs namespace will mean a totally new representation of locking.
That's going to break old versions of git completely, and possibly even
some user scripts.

> The easiest thing would be to mark files and directories differently;
> something like
> 
>     $GIT_DIR/{,logs/}refs/heads/a/b/c~
> [...]
> The first convention, "logs/refs/heads/a/b/c~" is not usable because
> a reflog for a dead reference with this name would conflict with a
> reflog for a live reference "heads/a" or "heads/a/b" that uses the
> current filename convention.

Right. That's what I started with, then created the graveyard hierarchy
to avoid conflicts between the "old" namespace (that cannot handle D/F
conflicts) and the "new" one (that can, because it represents files and
directories differently).

> or
> 
>     $GIT_DIR/{,logs/}refs/heads~/a~/b~/c
> 
> i.e., munging either directory or file names to strings that are
> illegal in refnames such that it is unambiguous from the name whether
> a path is a file or directory.

This one can have conflicts in the opposite direction if you don't have
any directories. E.g., you have $GIT_DIR/foo, a deleted ref, which has
no tildes because it has no directories in the path. But you want to
create foo/bar under the "old" system, which cannot happen (under the
new system, it is fine, but the point of this exercise is to overlay the
old and new systems).

That may be an OK tradeoff. We are restrictive in what goes into the
top-level. Although I notice that you did not mark "refs" in the above
example. So you could have the same problem with "refs/stash", for
example. Again, though, we don't tend to have arbitrary data at the
top-level (and I think refs/stash gets special cased in a couple places
already). So it might be an acceptable limitation.

If we want to be pedantic, my patch causes conflicts for top-level refs
called "graveyard" (although I know we have talked about restricting
top-level refs to [A-Z_-], I don't recall if that has actually
happened).

> And *if* we did that, then we wouldn't need a separate "graveyard"
> namespace, would we?  The reflogs for dead references could live
> among those for living references.

Right, assuming the limitation above is OK. But note that it doesn't
really save us any code. We still have to convert between refnames and
graveyard versions. _Eventually_ if the refnames were all converted,
that code could go away.

> But the second convention, "logs/refs/heads~/a~/b~/c, cannot conflict
> with current reflog files.  And it would be a step towards allowing
> "foo" and "foo/bar" at the same time.  What do you think about using
> a convention like this instead of the one that you proposed?

I think it's reasonable. As I said, it doesn't save any code _now_, but since
I am pulling a convention out of thin air, it might as well be one that
has a possibility of converging in the future (all other things being
equal, of course; I do find marking the directories a little uglier to
read, but that is mostly because of the tilde).

> Another minor concern is the choice of trailing tilde in the file or
> directory names.  Given that emacs creates backup files by appending
> a tilde to the filename, (1) it would be easy to inadvertently create
> such files, which git might try to interpret as reflogs and (2) there
> might be tools that innately "know" to skip such files in their
> processing. ack-grep, a replacement for grep, is an example that
> springs to mind.

The use of "~" for backup files was actually something that made me
choose it, since these are, after all, backups of the reflog. But they
are probably more precious than editor backup files, so the special
treatment they're given by other programs is probably not desirable.

> Other possibilities (according to git-check-ref-format(1)):
> 
>     refs/.heads/.a/.b/c
>     refs/heads./a./b./c (problematic on some Windows filesystems?)
>     refs/heads../a../b../c
>     refs/heads~dir/a~dir/b~dir/c (or some other suffix)
>     refs/heads..a..b..c (not recommended because it flattens
> directory hierarchy)

I don't like leading-dot, because those files are also often skipped by
directory traversal of some programs (and certainly they are confusing
to work with if you try to use "ls" to debug your $GIT_DIR/logs
directory). Trailing dot is less ugly to me, but I do wonder about its
special meaning as an extension separator. Double-dots just look gross.

Note that we have a few other magic characters available, too. Colon is
probably the least offensive (metacharacters like *, ?, and [ just make
things unnecessarily painful for shell users).

So I think a suffix like ":d" is probably the least horrible.

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Johannes Sixt]

Am 20.07.2012 17:44, schrieb Jeff King:
> So I think a suffix like ":d" is probably the least horrible.

Not so. It does not work on Windows :-( in the expected way. Trying to
open a file with a colon-separated suffix either opens a resource fork
on NTFS or fails with "invalid path".

-- Hannes

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Fri, Jul 20, 2012 at 06:37:02PM +0200, Johannes Sixt wrote:

> Am 20.07.2012 17:44, schrieb Jeff King:
> > So I think a suffix like ":d" is probably the least horrible.
> 
> Not so. It does not work on Windows :-( in the expected way. Trying to
> open a file with a colon-separated suffix either opens a resource fork
> on NTFS or fails with "invalid path".

Bleh. It seems that we did too good a job in coming up with a list of
disallowed ref characters; they really are things you don't want in your
filenames at all. :)

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Alexey Muranov]

On 20 Jul 2012, at 19:09, Jeff King wrote:

> On Fri, Jul 20, 2012 at 06:37:02PM +0200, Johannes Sixt wrote:
> 
>> Am 20.07.2012 17:44, schrieb Jeff King:
>>> So I think a suffix like ":d" is probably the least horrible.
>> 
>> Not so. It does not work on Windows :-( in the expected way. Trying to
>> open a file with a colon-separated suffix either opens a resource fork
>> on NTFS or fails with "invalid path".
> 
> Bleh. It seems that we did too good a job in coming up with a list of
> disallowed ref characters; they really are things you don't want in your
> filenames at all. :)

How about using '@' as an escape character ?

-Alexey.

Re: [PATCH 1/3] retain reflogs for deleted refs

[Nguyen Thai Ngoc Duy]

On Sat, Jul 21, 2012 at 12:09 AM, Jeff King <peff@peff.net> wrote:
> On Fri, Jul 20, 2012 at 06:37:02PM +0200, Johannes Sixt wrote:
>
>> Am 20.07.2012 17:44, schrieb Jeff King:
>> > So I think a suffix like ":d" is probably the least horrible.
>>
>> Not so. It does not work on Windows :-( in the expected way. Trying to
>> open a file with a colon-separated suffix either opens a resource fork
>> on NTFS or fails with "invalid path".
>
> Bleh. It seems that we did too good a job in coming up with a list of
> disallowed ref characters; they really are things you don't want in your
> filenames at all. :)

So we haven't found any way to present both branches "foo" and
"foo/bar" on file system at the same time. How about when we a new
branch introduces such a conflict, we push the new branch directly to
packed-refs? If we need either of them on a separate file, for fast
update for example, then we unpack just one and repack all refs that
conflict with it. Attempting to update two conflict branches in
parallel may impact performance, but I don't think that happens often.
-- 
Duy

Re: [PATCH 1/3] retain reflogs for deleted refs

[Alexey Muranov]

On 26 Jul 2012, at 14:47, Nguyen Thai Ngoc Duy wrote:

> So we haven't found any way to present both branches "foo" and
> "foo/bar" on file system at the same time. How about when we a new
> branch introduces such a conflict, we push the new branch directly to
> packed-refs? If we need either of them on a separate file, for fast
> update for example, then we unpack just one and repack all refs that
> conflict with it. Attempting to update two conflict branches in
> parallel may impact performance, but I don't think that happens often.
> -- 
> Duy

How about simply deprecating "/" in branch name?

-Alexey.

Re: [PATCH 1/3] retain reflogs for deleted refs

[Matthieu Moy]

Alexey Muranov <alexey.muranov@gmail.com> writes:

> On 26 Jul 2012, at 14:47, Nguyen Thai Ngoc Duy wrote:
>
>> So we haven't found any way to present both branches "foo" and
>> "foo/bar" on file system at the same time. How about when we a new
>> branch introduces such a conflict, we push the new branch directly to
>> packed-refs? If we need either of them on a separate file, for fast
>> update for example, then we unpack just one and repack all refs that
>> conflict with it. Attempting to update two conflict branches in
>> parallel may impact performance, but I don't think that happens often.
>> -- 
>> Duy
>
> How about simply deprecating "/" in branch name?

Err, it's not like nobody's using this feature (Junio does a heavy use
of it in particular) ...

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Thu, Jul 26, 2012 at 06:41:09PM +0200, Matthieu Moy wrote:

> > How about simply deprecating "/" in branch name?
> 
> Err, it's not like nobody's using this feature (Junio does a heavy use
> of it in particular) ...

Not to mention git itself, as it splits up the refs/remotes hierarchy
into subdirectories. I think deprecating "/" is out of the question.

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Alexey Muranov]

On 26 Jul 2012, at 18:59, Jeff King wrote:

> Not to mention git itself, as it splits up the refs/remotes hierarchy
> into subdirectories. I think deprecating "/" is out of the question.
> 
> -Peff

Ok, i guess you know better than me, my vision of Git is probably still too simplistic.

-Alexey.

Re: [PATCH 1/3] retain reflogs for deleted refs

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Fri, Jul 20, 2012 at 06:37:02PM +0200, Johannes Sixt wrote:
>
>> Am 20.07.2012 17:44, schrieb Jeff King:
>> > So I think a suffix like ":d" is probably the least horrible.
>> 
>> Not so. It does not work on Windows :-( in the expected way. Trying to
>> open a file with a colon-separated suffix either opens a resource fork
>> on NTFS or fails with "invalid path".
>
> Bleh. It seems that we did too good a job in coming up with a list of
> disallowed ref characters; they really are things you don't want in your
> filenames at all. :)

Why do no need to even worry about ~ vs : vs whatever in the first
place?

With a flag-day per repository "core.repositoryformatversion = 1",
you do not have to worry about mixture of old-style refs and new
ones, so refs/heads/next-d/log could be a topic branch 'next/log'
that is based on an integration branch 'next' branch that physically
resides at refs/heads/next-f or an entry refs/heads/next in packed
refs.  Only the API functions in refs.c should care, no?

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Thu, Jul 26, 2012 at 10:46:01AM -0700, Junio C Hamano wrote:

> > Bleh. It seems that we did too good a job in coming up with a list of
> > disallowed ref characters; they really are things you don't want in your
> > filenames at all. :)
> 
> Why do no need to even worry about ~ vs : vs whatever in the first
> place?
> 
> With a flag-day per repository "core.repositoryformatversion = 1",
> you do not have to worry about mixture of old-style refs and new
> ones, so refs/heads/next-d/log could be a topic branch 'next/log'
> that is based on an integration branch 'next' branch that physically
> resides at refs/heads/next-f or an entry refs/heads/next in packed
> refs.  Only the API functions in refs.c should care, no?

I think the point was that Michael wanted to select a standard that
could be used for graveyard reflogs _now_, but which would eventually
match the format we use for active refs. And that requires a character
that is not valid in a refname.

Given that the change of format for actives refs would require a flag
day, keeping the graveyard scheme mixable with the current ref rules may
not be worth caring about, though.

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Alexey Muranov]

On 20 Jul 2012, at 17:44, Jeff King wrote:

> On Fri, Jul 20, 2012 at 11:49:07AM +0200, Michael Haggerty wrote:
> 
>>> This patch moves reflog entries into a special "graveyard"
>>> namespace, and appends a tilde (~) character, which is
>>> not allowed in a valid ref name. This means that the deleted
>>> reflogs of these refs:
>>> 
>>>   refs/heads/a
>>>   refs/heads/a/b
>>>   refs/heads/a/b/c
>>> 
>>> will be stored in:
>>> 
>>>   logs/graveyard/refs/heads/a~
>>>   logs/graveyard/refs/heads/a/b~
>>>   logs/graveyard/refs/heads/a/b/c~
>>> 
>>> Putting them in the graveyard namespace ensures they will
>>> not conflict with live refs, and the tilde prevents D/F
>>> conflicts within the graveyard namespace.

Sorry if this idea is stupid or if i miss something, but how about putting deleted reflogs for

refs/heads/a
refs/heads/a/b
refs/heads/a/b/c

to

refs/heads/a@yyyy-mm-dd-hhmmss
refs/heads/a/b@yyyy-mm-dd-hhmmss
refs/heads/a/b/c@yyyy-mm-dd-hhmmss

with the time they were deleted?

-Alexey.

Re: [PATCH 1/3] retain reflogs for deleted refs

[Alexey Muranov]

On 22 Jul 2012, at 13:10, Alexey Muranov wrote:

> Sorry if this idea is stupid or if i miss something, but how about putting deleted reflogs for
> 
> refs/heads/a
> refs/heads/a/b
> refs/heads/a/b/c
> 
> to
> 
> refs/heads/a@yyyy-mm-dd-hhmmss
> refs/heads/a/b@yyyy-mm-dd-hhmmss
> refs/heads/a/b/c@yyyy-mm-dd-hhmmss
> 
> with the time they were deleted?
> 
> -Alexey.

Sorry, i meant to:

logs/refs/heads/a@yyyy-mm-dd-hhmmss
logs/refs/heads/a/b@yyyy-mm-dd-hhmmss
logs/refs/heads/a/b/c@yyyy-mm-dd-hhmmss

-Alexey.

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Sun, Jul 22, 2012 at 01:10:55PM +0200, Alexey Muranov wrote:

> >>>   refs/heads/a
> >>>   refs/heads/a/b
> >>>   refs/heads/a/b/c
> >>> 
> >>> will be stored in:
> >>> 
> >>>   logs/graveyard/refs/heads/a~
> >>>   logs/graveyard/refs/heads/a/b~
> >>>   logs/graveyard/refs/heads/a/b/c~
> >>> 
> >>> Putting them in the graveyard namespace ensures they will
> >>> not conflict with live refs, and the tilde prevents D/F
> >>> conflicts within the graveyard namespace.
> 
> Sorry if this idea is stupid or if i miss something, but how about putting deleted reflogs for
> 
> refs/heads/a
> refs/heads/a/b
> refs/heads/a/b/c
> 
> to
> 
> refs/heads/a@yyyy-mm-dd-hhmmss
> refs/heads/a/b@yyyy-mm-dd-hhmmss
> refs/heads/a/b/c@yyyy-mm-dd-hhmmss
> 
> with the time they were deleted?

I like the readability of the resulting file names, but it has three
problems:

  1. "@" is allowed in ref names, so you may be conflicting with
     existing refs. You could fix that by using "@{...}", which is
     disallowed. E.g., refs/heads/a@{yyyy-mm-dd-hhmmss}.

  2. It makes lookup slightly more expensive, because to find a reflog
     for "refs/heads/a", I have to scan "logs/refs/heads" looking for
     any matching entries of the form "a@{.*}". This is probably not a
     huge deal in practice, though it does make the code more complex.

  3. Most importantly, it does not resolve D/F conflicts (it has the
     same problem as "logs/refs/heads/a~"). If you delete "foo/bar", you
     will end up with "logs/refs/heads/foo/bar@{...}". That will prevent
     D/F conflicts with a new branch "foo/bar/baz", but will still have
     a problem with just "foo".

     You need to either mark each directory to avoid the conflict
     (Michael suggested something like "refs/heads~/foo~/bar"), or
     you need to put the deleted logs into a separate hierarchy (I used
     "logs/graveyard" in my patch).

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Alexey Muranov]

On 22 Jul 2012, at 15:14, Jeff King wrote:

>  3. Most importantly, it does not resolve D/F conflicts (it has the
>     same problem as "logs/refs/heads/a~"). If you delete "foo/bar", you
>     will end up with "logs/refs/heads/foo/bar@{...}". That will prevent
>     D/F conflicts with a new branch "foo/bar/baz", but will still have
>     a problem with just "foo".

Unfortunately i do not really follow this, because i have not seen any directories in "logs/refs/heads/", i only saw files named after local branches there. I do not know how directories are used there.

-Alexey.

Re: [PATCH 1/3] retain reflogs for deleted refs

[Jeff King]

On Sun, Jul 22, 2012 at 04:40:14PM +0200, Alexey Muranov wrote:

> >  3. Most importantly, it does not resolve D/F conflicts (it has the
> >     same problem as "logs/refs/heads/a~"). If you delete "foo/bar", you
> >     will end up with "logs/refs/heads/foo/bar@{...}". That will prevent
> >     D/F conflicts with a new branch "foo/bar/baz", but will still have
> >     a problem with just "foo".
> 
> Unfortunately i do not really follow this, because i have not seen any
> directories in "logs/refs/heads/", i only saw files named after local
> branches there. I do not know how directories are used there.

The user is free to have branch names with slashes, in which case they
are represented in the filesystem as directories. Even without using
slashes in your branch names, you already have subdirectories in
refs/remotes.

-Peff

Re: [PATCH 1/3] retain reflogs for deleted refs

[Johannes Sixt]

Am 20.07.2012 11:49, schrieb Michael Haggerty:
> Other possibilities (according to git-check-ref-format(1)):
> 
>     refs/.heads/.a/.b/c
>     refs/heads./a./b./c (problematic on some Windows filesystems?)

Yes. Probably all filesystems.

>     refs/heads../a../b../c

Same here.

>     refs/heads~dir/a~dir/b~dir/c (or some other suffix)
>     refs/heads..a..b..c (not recommended because it flattens directory
> hierarchy)

-- Hannes

[RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[mhagger]

From: Michael Haggerty <mhagger@alum.mit.edu>

On 08/17/2012 01:29 AM, Junio C Hamano wrote:> Junio C Hamano <gitster@pobox.com> writes:
>> I like the general direction.  Perhaps a long distant future
>> direction could be to also use the same trick in the ref namespace
>> so that we can have 'next' branch itself, and 'next/foo', 'next/bar'
>> forks that are based on the 'next' branch at the same time (it
>> obviously is a totally unrelated topic)?
> 
> I notice that I was responsible for making this topic veer in the
> wrong direction by bringing up a new feature "having 'next' and
> 'next/bar' at the same time" which nobody asked.  Perhaps we can
> drop that for now to simplify the scope of the topic, to bring the
> log graveyard back on track?

Given that a flag day would anyway be required to add a d/f-tolerant
system, I could live with a separate "graveyard" namespace as
originally proposed by Jeff.

However, I still think that as long as we are making a jump, we could
try to land closer to the ultimate destination.  So here are some
patches that apply on top of Jeff's to show what I mean.  (Please also
note that I made some technical comments about Jeff's patches in an
earlier email.)

The first two patches fix a breakage that I see when I apply Jeff's
patches to master.

The third changes the implementation of refname_to_graveyard_reflog()
and graveyard_reflog_to_refname() and touches up some test cases.  It
changes the naming convention for dead references to

    "$GIT_DIR/logs/refs~d/heads~d/foo~d/bar~d/baz~f"

I.e., the dead reflogs are stored closer to the living.  It is not
obvious whether the "refs" part of the name should be munged to
"refs~d" as I have done, or left unmunged.  The argument in favor of
munging is that the algorithm is more uniform.  On the other hand,
extending the same scheme to loose references would produce filenames
like

    "$GIT_DIR/refs~d/heads~d/foo~d/bar~d/baz~f"

or maybe they should be nested inside of the refs directory like

    "$GIT_DIR/refs/refs~d/heads~d/foo~d/bar~d/baz~f"

(which would also give a better place to store top-level reference
names).

I structured the patches to apply on top of Jeff's for presentation
purposes, but if they are desired it would of course make more sense
to squash his and mine together in the obvious way.

I am a little bit worried that there are other test cases that use
"git prune" in the belief that it will remove all commits that were
referred to by deleted references.  The test suite runs cleanly for me
with these patches, but before they are integrated we should audit the
places where the test suite calls to "git prune" to make sure that
they are still testing what they think.

Michael Haggerty (3):
  t9300: format test in modern style prior to modifying it
  Delete reflogs for dead references to allow pruning
  Change naming convention for the reflog graveyard

 refs.c                               | 31 ++++++++++++++++++++++++-------
 t/t7701-repack-unpack-unreachable.sh |  4 ++--
 t/t9300-fast-import.sh               | 13 +++++++------
 3 files changed, 33 insertions(+), 15 deletions(-)

-- 
1.7.11.3

[RFC 1/3] t9300: format test in modern style prior to modifying it

[mhagger]

From: Michael Haggerty <mhagger@alum.mit.edu>


Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 t/t9300-fast-import.sh | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 2fcf269..266ae30 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1698,12 +1698,12 @@ M 160000 $SUBLAST sub
 
 INPUT_END
 
-test_expect_success \
-	'P: verbatim SHA gitlinks' \
-	'git branch -D sub &&
-	 git gc && git prune &&
-	 git fast-import <input &&
-	 test $(git rev-parse --verify subuse2) = $(git rev-parse --verify subuse1)'
+test_expect_success 'P: verbatim SHA gitlinks' '
+	git branch -D sub &&
+	git gc && git prune &&
+	git fast-import <input &&
+	test $(git rev-parse --verify subuse2) = $(git rev-parse --verify subuse1)
+'
 
 test_tick
 cat >input <<INPUT_END
-- 
1.7.11.3

[RFC 2/3] Delete reflogs for dead references to allow pruning

[mhagger]

From: Michael Haggerty <mhagger@alum.mit.edu>

This test is broken by "retain reflogs for deleted refs".  Explicitly
delete the reflogs in the graveyard to allow the corresponding commits
to be pruned.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---

Probably there should be a "git reflog" subcommand to do this.

 t/t9300-fast-import.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 266ae30..dc6c67d 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1700,6 +1700,7 @@ INPUT_END
 
 test_expect_success 'P: verbatim SHA gitlinks' '
 	git branch -D sub &&
+	rm -rf .git/logs/graveyard &&
 	git gc && git prune &&
 	git fast-import <input &&
 	test $(git rev-parse --verify subuse2) = $(git rev-parse --verify subuse1)
-- 
1.7.11.3

Re: [RFC 2/3] Delete reflogs for dead references to allow pruning

[Jeff King]

On Sat, Aug 18, 2012 at 07:14:45PM +0200, mhagger@alum.mit.edu wrote:

> From: Michael Haggerty <mhagger@alum.mit.edu>
> 
> This test is broken by "retain reflogs for deleted refs".  Explicitly
> delete the reflogs in the graveyard to allow the corresponding commits
> to be pruned.
> 
> Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
> ---
> 
> Probably there should be a "git reflog" subcommand to do this.
> 
>  t/t9300-fast-import.sh | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
> index 266ae30..dc6c67d 100755
> --- a/t/t9300-fast-import.sh
> +++ b/t/t9300-fast-import.sh
> @@ -1700,6 +1700,7 @@ INPUT_END
>  
>  test_expect_success 'P: verbatim SHA gitlinks' '
>  	git branch -D sub &&
> +	rm -rf .git/logs/graveyard &&

I think "git reflog expire --expire=now refs/heads/sub" would be less
leaky. That didn't work in my initial version, but it obviously should.
I'll include a tweaked version of your fix when I re-roll.

-Peff

[RFC 3/3] Change naming convention for the reflog graveyard

[mhagger]

From: Michael Haggerty <mhagger@alum.mit.edu>

Store reflogs for dead references among those for live references, in
a scheme that could in the future be extended to prevent
file/directory conflicts for live reference names, as well.  Store the
reflog for a dead reference "refs/foo/bar/baz" in file
"$GIT_DIR/logs/refs~d/heads~d/foo~d/bar~d/baz~f", where the suffix
"~d" is appended to "directory" names and "~f" is append to "file"
names.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
 refs.c                               | 31 ++++++++++++++++++++++++-------
 t/t7701-repack-unpack-unreachable.sh |  4 ++--
 t/t9300-fast-import.sh               |  2 +-
 3 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/refs.c b/refs.c
index 551a0f9..4797b20 100644
--- a/refs.c
+++ b/refs.c
@@ -2580,20 +2580,37 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 
 char *refname_to_graveyard_reflog(const char *ref)
 {
-	return git_path("logs/graveyard/%s~", ref);
+	static struct strbuf buf = STRBUF_INIT;
+
+	const char *p, *slash;
+	strbuf_reset(&buf);
+	for (p = ref; (slash = strchr(p, '/')); p = slash + 1) {
+		strbuf_add(&buf, p, slash - p);
+		strbuf_addstr(&buf, "~d/");
+	}
+	strbuf_addstr(&buf, p);
+	strbuf_addstr(&buf, "~f");
+
+	return git_path("logs/%s", buf.buf);
 }
 
 char *graveyard_reflog_to_refname(const char *log)
 {
 	static struct strbuf buf = STRBUF_INIT;
 
-	if (!prefixcmp(log, "graveyard/"))
-		log += 10;
-
+	const char *p, *slash;
 	strbuf_reset(&buf);
-	strbuf_addstr(&buf, log);
-	if (buf.len > 0 && buf.buf[buf.len-1] == '~')
-		strbuf_setlen(&buf, buf.len - 1);
+	for (p = log; (slash = strchr(p, '/')); p = slash + 1) {
+		if (slash - p > 2 && slash[-2] == '~' && slash[-1] == 'd')
+			strbuf_add(&buf, p, slash - p - 2);
+		else
+			strbuf_add(&buf, p, slash - p);
+		strbuf_addch(&buf, '/');
+	}
+
+	strbuf_addstr(&buf, p);
+	if (buf.len > 2 && !strcmp(buf.buf + buf.len - 2, "~f"))
+		strbuf_setlen(&buf, buf.len - 2);
 
 	return buf.buf;
 }
diff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh
index c06b715..f8d02db 100755
--- a/t/t7701-repack-unpack-unreachable.sh
+++ b/t/t7701-repack-unpack-unreachable.sh
@@ -40,7 +40,7 @@ test_expect_success '-A with -d option leaves unreachable objects unpacked' '
 	# now expire the reflog, while keeping reachable ones but expiring
 	# unreachables immediately; also remove any graveyard reflogs
 	# from deleted branches that would keep things reachable
-	rm -rf .git/logs/graveyard &&
+	rm -rf .git/logs/*~? &&
 	test_tick &&
 	sometimeago=$(( $test_tick - 10000 )) &&
 	git reflog expire --expire=$sometimeago --expire-unreachable=$test_tick --all &&
@@ -78,7 +78,7 @@ test_expect_success '-A without -d option leaves unreachable objects packed' '
 	test 1 = $(ls -1 .git/objects/pack/pack-*.pack | wc -l) &&
 	packfile=$(ls .git/objects/pack/pack-*.pack) &&
 	git branch -D transient_branch &&
-	rm -rf .git/logs/graveyard &&
+	rm -rf .git/logs/*~? &&
 	test_tick &&
 	git repack -A -l &&
 	test ! -f "$fsha1path" &&
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index dc6c67d..8d88f5f 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1700,7 +1700,7 @@ INPUT_END
 
 test_expect_success 'P: verbatim SHA gitlinks' '
 	git branch -D sub &&
-	rm -rf .git/logs/graveyard &&
+	rm -rf .git/logs/*~? &&
 	git gc && git prune &&
 	git fast-import <input &&
 	test $(git rev-parse --verify subuse2) = $(git rev-parse --verify subuse1)
-- 
1.7.11.3

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Junio C Hamano]

mhagger@alum.mit.edu writes:

> Given that a flag day would anyway be required to add a d/f-tolerant
> system, I could live with a separate "graveyard" namespace as
> originally proposed by Jeff.
>
> However, I still think that as long as we are making a jump, we could
> try to land closer to the ultimate destination.

Do we _know_ already what the "ultimate destination" looks like?  

If the answer is yes, then I agree, but otherwise, I doubt it is a
good idea to introduce unnecessary complexity to the system that may
have to be ripped out and redone.

I didn't get the impression that we know the "ultimate destination"
from the previous discussion, especially if we discount the tangent
around "having next and next/foo at the same time" which was on
nobody's wish, but I may be misremembering things.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Alexey Muranov]

On 18 Aug 2012, at 22:39, Junio C Hamano wrote:

> Do we _know_ already what the "ultimate destination" looks like?  
> 
> If the answer is yes, then I agree, but otherwise, I doubt it is a
> good idea to introduce unnecessary complexity to the system that may
> have to be ripped out and redone.
> 
> I didn't get the impression that we know the "ultimate destination"
> from the previous discussion, especially if we discount the tangent
> around "having next and next/foo at the same time" which was on
> nobody's wish, but I may be misremembering things.

Excuse me if i miss something again, but i might be willing to discuss the "ultimate destination".  Could you possibly state in simple terms what the problem with determining the "ultimate destination" is?  I hope my opinion might be useful because i do not know anything about the actual implementation of Git, but for a while i thought i was understanding it's intended mathematical model, until i ran into unexpected for me default behavior of not pruning when fetching.

To just give a quick idea of my ideas, i thought that 'fetching' in Git was an inevitable evil that stands apart from other operations and is necessary only because the computer communication on Earth is not sufficiently developed to keep all Git repositories constantly in sync, and because one might prefer to work with a somewhat dated snapshot of a remote than with the constantly changing current version. I thought "snapshot" could be a good alternative name for "fetch".

-Alexey.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Junio C Hamano]

Alexey Muranov <alexey.muranov@gmail.com> writes:

> On 18 Aug 2012, at 22:39, Junio C Hamano wrote:
>
>> Do we _know_ already what the "ultimate destination" looks like?  
>> 
>> If the answer is yes, then I agree, but otherwise, I doubt it is a
>> good idea to introduce unnecessary complexity to the system that may
>> have to be ripped out and redone.
>> 
>> I didn't get the impression that we know the "ultimate destination"
>> from the previous discussion, especially if we discount the tangent
>> around "having next and next/foo at the same time" which was on
>> nobody's wish, but I may be misremembering things.
>
> Excuse me if i miss something again, but i might be willing to
> discuss the "ultimate destination".  Could you possibly state in
> simple terms what the problem with determining the "ultimate
> destination" is?

Decide if it makes sense to break backward compatibility of loose
ref representation merely to support having a branch "next" and
another branch "next/foo" in the same repository, and if it does,
what the new loose ref representation looks like.

> I hope my opinion might be useful because i do not know anything
> about the actual implementation of Git,...

That sounds like contradiction.

> To just give a quick idea of my ideas, i thought that 'fetching'
> in Git was an inevitable evil that stands apart from other
> operations and is necessary only because the computer
> communication on Earth is not sufficiently developed to keep all
> Git repositories constantly in sync,...

It is a feature, not a symptom of an insufficiently developed
technology, that I do not have to know what random tweaks and
experiments are done in repositories of 47 thousands people who
clone from me, and I can sync with any one of them only when I know
there is something worth looking at when I say "git fetch".

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Alexey Muranov]

On 19 Aug 2012, at 02:02, Junio C Hamano wrote:

> Alexey Muranov <alexey.muranov@gmail.com> writes:
> 
>> I hope my opinion might be useful because i do not know anything
>> about the actual implementation of Git,...
> 
> That sounds like contradiction.

I think that the implementation (the code), the model, and the interface are independent.
On the top level, for example, one does not need to know how commit storage is optimized, it is enough to understand that each commit is a snapshot of a subtree in a file directory.

>> To just give a quick idea of my ideas, i thought that 'fetching'
>> in Git was an inevitable evil that stands apart from other
>> operations and is necessary only because the computer
>> communication on Earth is not sufficiently developed to keep all
>> Git repositories constantly in sync,...
> 
> It is a feature, not a symptom of an insufficiently developed
> technology, that I do not have to know what random tweaks and
> experiments are done in repositories of 47 thousands people who
> clone from me, and I can sync with any one of them only when I know
> there is something worth looking at when I say "git fetch".

Currently, one of the main functions of 'fetch', apart from changing the remote tracking branches, is downloading the remote objects.  This is necessary because of an insufficiently developed technology.

The other main function is changing the local copies of remote branches (changing the remote tracking branches), this is what i described as "taking a snapshot".

I did not understand what you meant by

  "I do not have to know what random tweaks and experiments are done in repositories of 47 thousands people who clone from me, and I can sync with any one of them only when I know there is something worth looking at when I say "git fetch"."

How is it possible to know and not to know what is going on in the remote repository in the same time?

-Alexey.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Alexey Muranov]

On 19 Aug 2012, at 02:02, Junio C Hamano wrote:

> Alexey Muranov <alexey.muranov@gmail.com> writes:
> 
>> I hope my opinion might be useful because i do not know anything
>> about the actual implementation of Git,...
> 
> That sounds like contradiction.

I meant that i am psychologically not attached to the current behavior, and may provide a naïve view point, if you like.

-Alexey.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Alexey Muranov]

On 19 Aug 2012, at 02:02, Junio C Hamano wrote:

> Alexey Muranov <alexey.muranov@gmail.com> writes:
> 
>> Excuse me if i miss something again, but i might be willing to
>> discuss the "ultimate destination".  Could you possibly state in
>> simple terms what the problem with determining the "ultimate
>> destination" is?
> 
> Decide if it makes sense to break backward compatibility of loose
> ref representation merely to support having a branch "next" and
> another branch "next/foo" in the same repository, and if it does,
> what the new loose ref representation looks like.

I looked again through this thread and tried to understand better the issues.

1. I vote for moving dead reflogs to "logs/graveyard" (or to "logs/deadlogs").

2. I think that allowing both "next" and "next/foo" complicates the mapping from branch names to file paths, and it does not seem necessary if dead reflogs are moved away to "graveyard" anyway.

3. There remains the question what to do with dead reflogs for different branches having the same name.  Maybe, keep the death date and time under the graveyard directory and not allow the user to delete 2 times in less than 1 second?

/logs/graveyard/yyyy-mm-dd-hhmmss/refs/heads/next/foo

In a sense this is similar to the git storage model: an "atomic" destructive operation creates a timestamped "commit" in logs/graveyard directory.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Junio C Hamano]

Alexey Muranov <alexey.muranov@gmail.com> writes:

> 2. I think that allowing both "next" and "next/foo" complicates
> the mapping from branch names to file paths, and it does not seem
> necessary if dead reflogs are moved away to "graveyard" anyway.

It is unclear why the first two lines above leads to the conclusion
"it does not seem necessary" (but honestly, I do not particularly
care).

> 3. There remains the question what to do with dead reflogs for
> different branches having the same name.  Maybe, keep the death
> date and time under the graveyard directory and not allow the user
> to delete 2 times in less than 1 second?
>
> /logs/graveyard/yyyy-mm-dd-hhmmss/refs/heads/next/foo

How would that help us in what way?

When I ask "git log -g next/foo" for the "next/foo" branch that
currently exists, I want to see the update history of its tip since
I created it for the last time, and then an entry that says I
created it at such and such time.  If I used to have the branch
before but deleted, then the output should be followed by another
entry that says I deleted it at such and such time, followed by the
history of the tip updates.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Alexey Muranov]

On 19 Aug 2012, at 19:38, Junio C Hamano wrote:

> Alexey Muranov <alexey.muranov@gmail.com> writes:
> 
>> 2. I think that allowing both "next" and "next/foo" complicates
>> the mapping from branch names to file paths, and it does not seem
>> necessary if dead reflogs are moved away to "graveyard" anyway.
> 
> It is unclear why the first two lines above leads to the conclusion
> "it does not seem necessary" (but honestly, I do not particularly
> care).

I thought that the first reason that allowing "next" and "next/foo" seemed necessary was avoiding conflicts with dead reflogs or between dead reflogs.  If dead reflog for "next/foo" is moved away, it will not conflict with a new one for "next".  There remains a problem with a conflict between dead "next/foo" and dead "next".  This can be solved as Jeff suggested by adding special "escape" symbols, or as i suggested below, by keeping reflogs deleted on different occasions in different "timestamp" directories.

>> 3. There remains the question what to do with dead reflogs for
>> different branches having the same name.  Maybe, keep the death
>> date and time under the graveyard directory and not allow the user
>> to delete 2 times in less than 1 second?
>> 
>> /logs/graveyard/yyyy-mm-dd-hhmmss/refs/heads/next/foo
> 
> How would that help us in what way?
> 
> When I ask "git log -g next/foo" for the "next/foo" branch that
> currently exists, I want to see the update history of its tip since
> I created it for the last time, and then an entry that says I
> created it at such and such time.  If I used to have the branch
> before but deleted, then the output should be followed by another
> entry that says I deleted it at such and such time, followed by the
> history of the tip updates.

I only suggested how to resolve conflicts between dead reflogs in graveyard if "next" and "next/foo" cannot coexist.
For example, if first "next/foo" was created and deleted, and then "next" was created and deleted.  It also seems nice to me to have dead reflogs for different identically named branches (created and deleted independently) in separate files.

It is possible to collect the information for "git log -g next/foo" by looking through all "timestamp" subdirectories in graveyard.

-Alexey.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Junio C Hamano]

Alexey Muranov <alexey.muranov@gmail.com> writes:

> I only suggested how to resolve conflicts between dead reflogs in
> graveyard if "next" and "next/foo" cannot coexist.

But Jeff's patch series already has the support for a case where you
delete next (graveyard gets 'next'), create next/foo and then delete
that (graveyard gets 'next/foo', too) anyway (check the list archive
before posting).  It is a solved problem.

> It is possible to collect the information for "git log -g
> next/foo" by looking through all "timestamp" subdirectories in
> graveyard.

It is possible if you wrote a new file every time you add one entry
to reflog, or if you created a directory with timestamp in its name
and wrote a new file there, too.

We are not particularly interested in "it is possible" when many
implementations can all trivially allow it to be "possible"; the
question is what a sensible solution is among them, and I didn't
find "a directory with timestamp in its name" a particularly
sensible way to go.

Either Jeff's "refname $name's log goes to logs/graveyard/$name~" or
Michael's "append ~d to each directory component, append ~f to the
leaf component" that are already proposed will keep "one file per
name" property to allow us to open once and efficiently read the
file through.  Why would we want to see an inferiour alternative
added to the discussion?

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Either Jeff's "refname $name's log goes to logs/graveyard/$name~" or
> Michael's "append ~d to each directory component, append ~f to the
> leaf component" that are already proposed will keep "one file per
> name" property to allow us to open once and efficiently read the
> file through.  Why would we want to see an inferiour alternative
> added to the discussion?

Note that there may be some "other" advantage I am not seeing in the
"directory with timestamp in its name"; if it is a big enough
advantage over what have already been proposed, then that would be a
valid reason why we may want to see it as an alternative (and at
that point, it is no longer inferior).  That is the reason why I
asked "How would that help us in what way?"

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Alexey Muranov]

Dear Junio,

On 20 Aug 2012, at 02:26, Junio C Hamano wrote:

> We are not particularly interested in "it is possible" when many
> implementations can all trivially allow it to be "possible"; the
> question is what a sensible solution is among them, and I didn't
> find "a directory with timestamp in its name" a particularly
> sensible way to go.
> 
> Either Jeff's "refname $name's log goes to logs/graveyard/$name~" or
> Michael's "append ~d to each directory component, append ~f to the
> leaf component" that are already proposed will keep "one file per
> name" property to allow us to open once and efficiently read the
> file through.  Why would we want to see an inferiour alternative
> added to the discussion?

I let the others decide if my idea with "timestamp" directories has a significant advantage over other proposed solutions or not.  It seemed different, so i wanted to add it to the discussion.  I cannot clearly formulate an advantage, but i will try to explain why i proposed it.

I would like also to propose another solution for allowing both "next" and "next/foo" branches, and to try to explain how it is different from the other proposed solutions (unless i missed something).

I would like that the solutions introduce as little new as possible to the existing solutions used in similar situation.

The problem of mapping branch names to file paths looks to me very similar to the problem of mapping URLs to file paths for static web sites, so i would propose to use the same solution: add a special extension to distinguish a file from a directory, for example ".branch" and ".tag" (like ".html" in the case of URL).  This would allow having both branches "next" and "next/foo" with refs stored in files "next.branch" and "next/foo.branch".  This will look very clear and familiar to people not specialist in Git, but familiar with static web sites.  The only limitation this would introduces is that branch names "foo.branch" would need to be forbidden.  If the extension is optional, this makes the new rule almost compatible with the current one, except if somebody is currently using branches n
 amed like "foo.branch" or "next.branch/foo".

For the reflogs of deleted branches, if both "next/foo" and "next" are allowed and it is decided to append to the reflogs when a new branch with the same name is deleted, then of course "timestamp" directories are useless.  However, i do not think that if a branch "tmp" was created and deleted multiple times, all its reflogs have to be concatenated into a single file.  So i viewed the problem of deleting identically named branches as the problem of deleting files under an operating system environment that uses a Trash Bin.  In this case, adding a timestamp usually solves the problem.

-Alexey.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Alexey Muranov]

On 20 Aug 2012, at 13:32, Alexey Muranov wrote:

> The problem of mapping branch names to file paths looks to me very similar to the problem of mapping URLs to file paths for static web sites, so i would propose to use the same solution: add a special extension to distinguish a file from a directory, for example ".branch" and ".tag" (like ".html" in the case of URL).  This would allow having both branches "next" and "next/foo" with refs stored in files "next.branch" and "next/foo.branch".  This will look very clear and familiar to people not specialist in Git, but familiar with static web sites.  The only limitation this would introduces is that branch names "foo.branch" would need to be forbidden.  If the extension is optional, this makes the new rule almost compatible with the current one, except if somebody is currently using branches
  named like "foo.branch" or "next.branch/foo".

Another possible choice for the extensions: ".~br" and ".~tg" (to keep readability of file names and allow all currently allowed branch names).

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Michael Haggerty]

On 08/18/2012 10:39 PM, Junio C Hamano wrote:
> mhagger@alum.mit.edu writes:
> 
>> Given that a flag day would anyway be required to add a d/f-tolerant
>> system, I could live with a separate "graveyard" namespace as
>> originally proposed by Jeff.
>>
>> However, I still think that as long as we are making a jump, we could
>> try to land closer to the ultimate destination.
> 
> Do we _know_ already what the "ultimate destination" looks like?  

No; we can only guess.  I just wanted to submit some code so that the
existence/absence of code would not prejudice the decision.

> If the answer is yes, then I agree, but otherwise, I doubt it is a
> good idea to introduce unnecessary complexity to the system that may
> have to be ripped out and redone.
> 
> I didn't get the impression that we know the "ultimate destination"
> from the previous discussion, especially if we discount the tangent
> around "having next and next/foo at the same time" which was on
> nobody's wish, but I may be misremembering things.

It's been a wish of mine, but it's pretty low priority.  I've also
brainstormed about some other changes that could be connected with a new
repo format:

* Allow "deleted" loose references (for example denoted by value 0{40})
that override packed references with the same name.  This would remove
the need to rewrite the packed-refs file when a reference is deleted.
(A prerequisite for this change would be to allow next and next/foo at
the same time.)

* Push HEAD and its friends down out of $GIT_DIR into a
reference-specific directory.

* Rename lock files to look less like reference names (e.g., something
like "refs/foo~lock" instead of "refs/foo.lock").

* Somehow munge reference names in a way to avoid other filesystem
limitations -- e.g., case insensitivity, filenames like "com" and "prn"
or with multiple dots under Windows.

* ...or maybe a packed-refs file that can (usually) be updated in-place,
and get rid of loose references entirely.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Junio C Hamano]

Michael Haggerty <mhagger@alum.mit.edu> writes:

> It's been a wish of mine, but it's pretty low priority.  I've also
> brainstormed about some other changes that could be connected with a new
> repo format:
>
> * Allow "deleted" loose references (for example denoted by value 0{40})
> that override packed references with the same name.  This would remove
> the need to rewrite the packed-refs file when a reference is deleted.
> (A prerequisite for this change would be to allow next and next/foo at
> the same time.)

We would need to think the performance implications through of the
approach; it tempts us to accumulate the loose "removed" markers in
the hope that it would be an improvement than having to rewrite the
packed-refs over and over, and without numbers to back that theory
up, we may be worsening the system without knowing.

Having said that, it is an interesting idea. I wouldn't use 0{40} as
the sentinel value but rather use letters outside [0-9a-f], though.

> * Push HEAD and its friends down out of $GIT_DIR into a
> reference-specific directory.

Not going to happen for several years, I am afraid, as I think many
casual tools do an equivalent of "test -f $DIR/HEAD" to see if $DIR
is a repository; even our own gitweb does so.

We should advertise an easy way for scripted Porcelains to directly
ask is_git_directory().

> * Rename lock files to look less like reference names (e.g., something
> like "refs/foo~lock" instead of "refs/foo.lock").

If you do the ~d/~f thing, foo.lock becomes a non-issue, no?

> * Somehow munge reference names in a way to avoid other filesystem
> limitations -- e.g., case insensitivity, filenames like "com" and "prn"
> or with multiple dots under Windows.

Very interesting.  I however am afraid that the users and the
projects will learn to avoid the problematic names a lot sooner than
such a change will be implemented to make the issue go away (or they
have already learned long time ago), and the end result may end up
solving a non-issue only to make the output from "find .git/refs"
even more unreadable.

> * ...or maybe a packed-refs file that can (usually) be updated in-place,
> and get rid of loose references entirely.

I find this equally intriguing as your "deleted" one above.

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Jeff King]

On Sat, Aug 18, 2012 at 01:39:41PM -0700, Junio C Hamano wrote:

> mhagger@alum.mit.edu writes:
> 
> > Given that a flag day would anyway be required to add a d/f-tolerant
> > system, I could live with a separate "graveyard" namespace as
> > originally proposed by Jeff.
> >
> > However, I still think that as long as we are making a jump, we could
> > try to land closer to the ultimate destination.
> 
> Do we _know_ already what the "ultimate destination" looks like?  
> 
> If the answer is yes, then I agree, but otherwise, I doubt it is a
> good idea to introduce unnecessary complexity to the system that may
> have to be ripped out and redone.
> 
> I didn't get the impression that we know the "ultimate destination"
> from the previous discussion, especially if we discount the tangent
> around "having next and next/foo at the same time" which was on
> nobody's wish, but I may be misremembering things.

Sorry for the slow response on this topic; I was traveling all last week
and am still catching up with emails.

I don't think we know what the ultimate destination looks like. If I had
to choose, it would probably be something like:

  refs/heads/next.ref
  refs/heads/next/foo.ref

which is easy to read and manipulate. But this is not compatible with
the current system, because:

  1. It cannot use ".ref", as that is allowed in ref names currently.

  2. This can't co-exist with existing, non-tweaked refs, since
     "refs/heads/next" would still conflict (you'd have to instead do
     "refs/heads.dir/next.dir/foo".

But since making a change like this would involve bumping the
repositoryformatversion flag _anyway_, so at that point we don't really
have to care about compatibility, and we are free to design what looks
good.

So in other words, I do not think any ultimate destination that I find
palatable would be achievable without making the full format jump
anyway. If all things were equal, I'd say there is no reason not to get
as close as we can. But I find some of the proposals significantly less
readable (in particular, the directory-munging is IMHO much harder to
read). And it is not as if it is buying us anything; you still have to
make a magic translation between a dead log and a live one.

Another option I've considered is simply holding back the graveyard
topic, working on the d/f tolerant storage, and then implementing the
graveyards on top (which is basically free at that point). But as you
note, it is not really a commonly-requested feature. If it were easy,
I'd say let's do it. But the idea of bumping repositoryformatversion for
the first time in git's history just to add a feature nobody wants is
not very appealing to me.

-Peff

Re: [RFC 0/3] Reflogs for deleted refs: fix breakage and suggest namespace change

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> So in other words, I do not think any ultimate destination that I find
> palatable would be achievable without making the full format jump
> anyway. If all things were equal, I'd say there is no reason not to get
> as close as we can. But I find some of the proposals significantly less
> readable (in particular, the directory-munging is IMHO much harder to
> read). And it is not as if it is buying us anything; you still have to
> make a magic translation between a dead log and a live one.

Yes, that is where the earlier comment of mine on this topic came from.

> Another option I've considered is simply holding back the graveyard
> topic, working on the d/f tolerant storage, and then implementing the
> graveyards on top (which is basically free at that point). But as you
> note, it is not really a commonly-requested feature. If it were easy,
> I'd say let's do it. But the idea of bumping repositoryformatversion for
> the first time in git's history just to add a feature nobody wants is
> not very appealing to me.

Amen.

[PATCH 2/3] teach sha1_name to look in graveyard reflogs

[Jeff King]

The previous commit introduced graveyard reflogs, where the
reflog for a deleted branch "foo" appears in
"logs/graveyard/refs/heads/foo~".

This patch teaches dwim_log to search for these logs if the
ref does not exist, and teaches read_ref_at to fall back to
them when the literal reflog does not exist.  This allows
"deleted@{1}" to refer to the final commit of a deleted
branch (either to view or to re-create the branch).  You can
also go further back, or refer to the deleted reflog entries
by time. Accessing deleted@{0} will yield the null sha1.

Similarly, for_each_reflog_ent learns to fallback to
graveyard refs, which allows the reflog walker to work.
However, this is slightly less friendly, as the revision
parser expects the matching ref to exist before it realizes
that we are interested in the reflog. Therefore you must use
"git log -g deleted@{1}" insted of "git log -g deleted" to
walk a deleted reflog.

In both cases, we also tighten up the mode-checking when
opening the reflogs. dwim_log checks that the entry we found
is a regular file (not a directory) to avoid D/F confusion
(e.g., you ask for "foo" but "foo/bar" exists and we find
the "foo" but it is a directory).

However, read_ref_at and for_each_reflog_ent did not do this
check, and relied on earlier parts of the code to have
verified the log they are about to open. This meant that
even before this patch, a race condition in changing refs
between dwim_log and the actual read could cause bizarre
errors (e.g., read_ref_at would open and try to mmap a
directory). This patch makes it even easier to trigger those
conditions (because the ref namespace and the fallback
graveyard namespace can have D/F ambiguity for a certain
path). To solve this, we check the mode of the file we open
and treat it as if it did not exist if it is not a regular
file (this is the same way dwim_log handles it).

Signed-off-by: Jeff King <peff@peff.net>
---
 refs.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)

diff --git a/refs.c b/refs.c
index 553de77..551a0f9 100644
--- a/refs.c
+++ b/refs.c
@@ -1590,9 +1590,16 @@ int dwim_log(const char *str, int len, unsigned char *sha1, char **log)
 
 		mksnpath(path, sizeof(path), *p, len, str);
 		ref = resolve_ref_unsafe(path, hash, 1, NULL);
-		if (!ref)
-			continue;
-		if (!stat(git_path("logs/%s", path), &st) &&
+		if (!ref) {
+			if (!stat(refname_to_graveyard_reflog(path), &st) &&
+			    S_ISREG(st.st_mode)) {
+				it = path;
+				hashcpy(hash, null_sha1);
+			}
+			else
+				continue;
+		}
+		else if (!stat(git_path("logs/%s", path), &st) &&
 		    S_ISREG(st.st_mode))
 			it = path;
 		else if (strcmp(ref, path) &&
@@ -2201,9 +2208,16 @@ int read_ref_at(const char *refname, unsigned long at_time, int cnt,
 
 	logfile = git_path("logs/%s", refname);
 	logfd = open(logfile, O_RDONLY, 0);
-	if (logfd < 0)
-		die_errno("Unable to read log '%s'", logfile);
-	fstat(logfd, &st);
+	if (logfd < 0 || fstat(logfd, &st) < 0 || !S_ISREG(st.st_mode)) {
+		const char *deleted_log = refname_to_graveyard_reflog(refname);
+
+		if (logfd >= 0)
+			close(logfd);
+		logfd = open(deleted_log, O_RDONLY);
+		if (logfd < 0 || fstat(logfd, &st) < 0 || !S_ISREG(st.st_mode))
+			die_errno("Unable to read log '%s'", logfile);
+		logfile = deleted_log;
+	}
 	if (!st.st_size)
 		die("Log %s is empty.", logfile);
 	mapsz = xsize_t(st.st_size);
@@ -2296,18 +2310,28 @@ int for_each_recent_reflog_ent(const char *refname, each_reflog_ent_fn fn, long
 {
 	const char *logfile;
 	FILE *logfp;
+	struct stat st;
 	struct strbuf sb = STRBUF_INIT;
 	int ret = 0;
 
 	logfile = git_path("logs/%s", refname);
 	logfp = fopen(logfile, "r");
-	if (!logfp)
-		return -1;
+	if (!logfp || fstat(fileno(logfp), &st) < 0 || !S_ISREG(st.st_mode)) {
+		logfile = refname_to_graveyard_reflog(refname);
+
+		if (logfp)
+			fclose(logfp);
+		logfp = fopen(logfile, "r");
+		if (!logfp)
+			return -1;
+		if (fstat(fileno(logfp), &st) < 0 || !S_ISREG(st.st_mode)) {
+			fclose(logfp);
+			return -1;
+		}
+	}
 
 	if (ofs) {
-		struct stat statbuf;
-		if (fstat(fileno(logfp), &statbuf) ||
-		    statbuf.st_size < ofs ||
+		if (st.st_size < ofs ||
 		    fseek(logfp, -ofs, SEEK_END) ||
 		    strbuf_getwholeline(&sb, logfp, '\n')) {
 			fclose(logfp);
-- 
1.7.10.5.40.g059818d

Re: [PATCH 2/3] teach sha1_name to look in graveyard reflogs

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> The previous commit introduced graveyard reflogs, where the
> reflog for a deleted branch "foo" appears in
> "logs/graveyard/refs/heads/foo~".
>
> This patch teaches dwim_log to search for these logs if the
> ref does not exist, and teaches read_ref_at to fall back to
> them when the literal reflog does not exist.  This allows
> "deleted@{1}" to refer to the final commit of a deleted
> branch (either to view or to re-create the branch).  You can
> also go further back, or refer to the deleted reflog entries
> by time. Accessing deleted@{0} will yield the null sha1.
>
> Similarly, for_each_reflog_ent learns to fallback to
> graveyard refs, which allows the reflog walker to work.
> However, this is slightly less friendly, as the revision
> parser expects the matching ref to exist before it realizes
> that we are interested in the reflog. Therefore you must use
> "git log -g deleted@{1}" insted of "git log -g deleted" to
> walk a deleted reflog.
>
> In both cases, we also tighten up the mode-checking when
> opening the reflogs. dwim_log checks that the entry we found
> is a regular file (not a directory) to avoid D/F confusion
> (e.g., you ask for "foo" but "foo/bar" exists and we find
> the "foo" but it is a directory).
>
> However, read_ref_at and for_each_reflog_ent did not do this
> check, and relied on earlier parts of the code to have
> verified the log they are about to open. This meant that
> even before this patch, a race condition in changing refs
> between dwim_log and the actual read could cause bizarre
> errors (e.g., read_ref_at would open and try to mmap a
> directory). This patch makes it even easier to trigger those
> conditions (because the ref namespace and the fallback
> graveyard namespace can have D/F ambiguity for a certain
> path). To solve this, we check the mode of the file we open
> and treat it as if it did not exist if it is not a regular
> file (this is the same way dwim_log handles it).
>
> Signed-off-by: Jeff King <peff@peff.net>

This may or may not be related, but I vaguely recall that "log -g"
traversal hack had a corner case where the walking stops prematurely
upon seeing a gap (or creation/deletion that has 0{40})?  Do you
recall if we have ever dealt with that?

The patch seems fine from a cursory look.  Thanks.

Re: [PATCH 2/3] teach sha1_name to look in graveyard reflogs

[Jeff King]

On Thu, Jul 19, 2012 at 03:39:24PM -0700, Junio C Hamano wrote:

> > Similarly, for_each_reflog_ent learns to fallback to
> > graveyard refs, which allows the reflog walker to work.
> > However, this is slightly less friendly, as the revision
> > parser expects the matching ref to exist before it realizes
> > that we are interested in the reflog. Therefore you must use
> > "git log -g deleted@{1}" insted of "git log -g deleted" to
> > walk a deleted reflog.
> 
> This may or may not be related, but I vaguely recall that "log -g"
> traversal hack had a corner case where the walking stops prematurely
> upon seeing a gap (or creation/deletion that has 0{40})?  Do you
> recall if we have ever dealt with that?

>From my tests, I think it is probably still broken (if you do a delete,
create, delete sequence on a branch and then walk the reflog, it stops
prematurely at the 0{40} sha1).

But what _should_ it show for such an entry? There is no commit to show
in the reflog walker, but it would still be nice to say "BTW, there was
a deletion even here". Obviously just skipping it and showing the next
entry would be better than the current behavior of stopping the
traversal, but I feel like there must be some better behavior.

-Peff

Re: [PATCH 2/3] teach sha1_name to look in graveyard reflogs

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> But what _should_ it show for such an entry? There is no commit to show
> in the reflog walker, but it would still be nice to say "BTW, there was
> a deletion even here". Obviously just skipping it and showing the next
> entry would be better than the current behavior of stopping the
> traversal, but I feel like there must be some better behavior.

Like showing an entry that says "Ref deleted here", which should be
easy to do by creating a phoney commit object and inserting it to
the queue the reflog walker uses, I would guess.

[PATCH 3/3] add tests for reflogs of deleted refs

[Jeff King]

These tests cover the basic functionality of retaining
reflogs for deleted refs.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t1413-reflog-deletion.sh | 74 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)
 create mode 100755 t/t1413-reflog-deletion.sh

diff --git a/t/t1413-reflog-deletion.sh b/t/t1413-reflog-deletion.sh
new file mode 100755
index 0000000..e00d038
--- /dev/null
+++ b/t/t1413-reflog-deletion.sh
@@ -0,0 +1,74 @@
+#!/bin/sh
+
+test_description='test retention of reflog after ref deletion'
+. ./test-lib.sh
+
+test_expect_success 'setup deleted branch' '
+	test_tick && echo one >file && git add file && git commit -m one &&
+	test_tick && echo two >file && git add file && git commit -m two &&
+	git checkout -b foo/bar &&
+	test_tick && echo three >file && git add file && git commit -m three &&
+	git checkout master &&
+	git branch -D foo/bar &&
+	rm -f .git/logs/HEAD
+'
+
+test_expect_success 'branch is no longer accessible' '
+	test_must_fail git rev-parse --verify foo/bar
+'
+
+test_expect_success 'final reflog is null sha1' '
+	echo $_z40 >expect &&
+	git rev-parse --verify foo/bar@{0} >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'deleted reflog entries are accessible' '
+	cat >expect <<-\EOF &&
+	three
+	two
+	EOF
+	{
+		git log -1 --format=%s foo/bar@{1}
+		git log -1 --format=%s foo/bar@{2}
+	} >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'reflog walker can find deleted entries' '
+	cat >expect <<-\EOF &&
+	three
+	two
+	EOF
+	git log -g --format=%s foo/bar@{1} >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'can still create/delete same ref' '
+	git branch foo/bar &&
+	git branch -D foo/bar
+'
+
+test_expect_success 'can still create/delete parent ref' '
+	git branch foo &&
+	git branch -D foo
+'
+
+test_expect_success 'can still create/delete child ref' '
+	git branch foo/bar/baz &&
+	git branch -D foo/bar/baz
+'
+
+test_expect_success 'deleted reflog entries are still reachable' '
+	>expect &&
+	git fsck --unreachable >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'deleted reflog entries are expired normally' '
+	git reflog expire --all --expire=now &&
+	git fsck --unreachable >actual &&
+	test_line_count = 3 actual
+'
+
+test_done
-- 
1.7.10.5.40.g059818d