git-svn and huge data and modifying the git-svn-HEAD branch directly

git-svn and huge data and modifying the git-svn-HEAD branch directly

[Nicolas Vilz 'niv']

hi everyone,

as i mentioned, i do experimental work with git and svn... and i
experienced some problems with git when pulling much data from svn.

Actually that happens after i commit a revision with many and big files.
After that i cannot do a git-svn fetch anymore because git-svn
complains...

fatal: Ref refs/heads/svn-git-HEAD is at
504721bf4b2702d3e56cef69950f42a43568e846 but expected
504721bf4b2702d3e56cef69950f42a43568e846

now i am a little confused about that... oh, i actually modified the
svn-git directly instead of a private working branch... perhaps that was
not intended.

now i am still on rev 2 on this branch but i updated it to rev 5 on the
svn-side...

any hints?

Sincerly
Nicolas

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Eric Wong]

Nicolas Vilz 'niv' <niv@iaglans.de> wrote:
> hi everyone,
> 
> as i mentioned, i do experimental work with git and svn... and i
> experienced some problems with git when pulling much data from svn.
> 
> Actually that happens after i commit a revision with many and big files.
> After that i cannot do a git-svn fetch anymore because git-svn
> complains...
> 
> fatal: Ref refs/heads/svn-git-HEAD is at
> 504721bf4b2702d3e56cef69950f42a43568e846 but expected
> 504721bf4b2702d3e56cef69950f42a43568e846

Those messages are from git-update-ref.  What were some of the messages
from git-svn leading up to that point?

> now i am a little confused about that... oh, i actually modified the
> svn-git directly instead of a private working branch... perhaps that was
> not intended.

You should never, ever modify the git-svn-HEAD branch yourself.
Interface branches should never be modified.  It's the golden rule of
interfacing between different SCM interfaces.  Sorry, I've been doing
things like this this for a while now I guess I didn't make it
abundantly clear in the documentation.

> now i am still on rev 2 on this branch but i updated it to rev 5 on the
> svn-side...
> 
> any hints?

Save your current work in git-svn-HEAD to a private branch

	git branch -b private git-svn-HEAD

then reset git-svn-HEAD to the last revision where it was managed by
git-svn fetch:

	git-checkout git-svn-HEAD
	git-log (look for the last commit with 'git-svn-id:' in it)
	git-reset --hard <last commit with 'git-svn-id:' in it>

Now go to your private branch:

	git checkout private

And continue working on your private branch as usual.

-- 
Eric Wong

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Jan Harkes]

On Mon, Feb 27, 2006 at 10:46:41AM -0800, Eric Wong wrote:
> > now i am a little confused about that... oh, i actually modified the
> > svn-git directly instead of a private working branch... perhaps that was
> > not intended.
> 
> You should never, ever modify the git-svn-HEAD branch yourself.
> Interface branches should never be modified.  It's the golden rule of
> interfacing between different SCM interfaces.  Sorry, I've been doing
> things like this this for a while now I guess I didn't make it
> abundantly clear in the documentation.

If it is not supposed to be changed by the user, maybe it could be
stored as a tag.

Or maybe another type of reference can be introduced. refs/remote/, for
branches we are tracking, but which should not be modified locally.

Jan

[PATCH] contrib/git-svn: tell the user to not modify git-svn-HEAD directly

[Eric Wong]

As a rule, interface branches to different SCMs should never be modified
directly by the user.  They are used exclusively for talking to the
foreign SCM.

Signed-off-by: Eric Wong <normalperson@yhbt.net>

---

 contrib/git-svn/git-svn.txt |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)

4676a850ad5a9e4a88fa5dfba1ac231a58bffda1
diff --git a/contrib/git-svn/git-svn.txt b/contrib/git-svn/git-svn.txt
index b4b7789..b588a2a 100644
--- a/contrib/git-svn/git-svn.txt
+++ b/contrib/git-svn/git-svn.txt
@@ -43,6 +43,11 @@ fetch::
 	Fetch unfetched revisions from the SVN_URL we are tracking.
 	refs/heads/git-svn-HEAD will be updated to the latest revision.
 
+	Note: You should never attempt to modify the git-svn-HEAD branch
+	outside of git-svn.  Instead, create a branch from git-svn-HEAD
+	and work on that branch.  Use the 'commit' command (see below)
+	to write git commits back to git-svn-HEAD.
+
 commit::
 	Commit specified commit or tree objects to SVN.  This relies on
 	your imported fetch data being up-to-date.  This makes
@@ -179,7 +184,9 @@ SVN repositories via one git repository.
 environment variable to a name other other than "git-svn" (the default)
 and git-svn will ignore the contents of the $GIT_DIR/git-svn directory
 and instead do all of its work in $GIT_DIR/$GIT_SVN_ID for that
-invocation.
+invocation.  The interface branch will be $GIT_SVN_ID-HEAD, instead of
+git-svn-HEAD.  Any $GIT_SVN_ID-HEAD branch should never be modified
+by the user outside of git-svn commands.
 
 ADDITIONAL FETCH ARGUMENTS
 --------------------------
-- 
1.2.3.gfc24dc-dirty

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Eric Wong]

Jan Harkes <jaharkes@cs.cmu.edu> wrote:
> On Mon, Feb 27, 2006 at 10:46:41AM -0800, Eric Wong wrote:
> > > now i am a little confused about that... oh, i actually modified the
> > > svn-git directly instead of a private working branch... perhaps that was
> > > not intended.
> > 
> > You should never, ever modify the git-svn-HEAD branch yourself.
> > Interface branches should never be modified.  It's the golden rule of
> > interfacing between different SCM interfaces.  Sorry, I've been doing
> > things like this this for a while now I guess I didn't make it
> > abundantly clear in the documentation.
> 
> If it is not supposed to be changed by the user, maybe it could be
> stored as a tag.
> 
> Or maybe another type of reference can be introduced. refs/remote/, for
> branches we are tracking, but which should not be modified locally.

Either of those could work for me.  Changing git-svn-HEAD to become a
tag would probably be easier (not having to update other tools, such as
git-fetch), but refs/remote may make more sense.

-- 
Eric Wong

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Nicolas Vilz 'niv']

> Nicolas Vilz 'niv' <niv@iaglans.de> wrote:
>> hi everyone,
>>
>> as i mentioned, i do experimental work with git and svn... and i
>> experienced some problems with git when pulling much data from svn.
>>
>> Actually that happens after i commit a revision with many and big files.
>> After that i cannot do a git-svn fetch anymore because git-svn
>> complains...
>>
>> fatal: Ref refs/heads/svn-git-HEAD is at
>> 504721bf4b2702d3e56cef69950f42a43568e846 but expected
>> 504721bf4b2702d3e56cef69950f42a43568e846
>
> Those messages are from git-update-ref.  What were some of the messages
> from git-svn leading up to that point?
>
>> now i am a little confused about that... oh, i actually modified the
>> svn-git directly instead of a private working branch... perhaps that was
>> not intended.
>
> You should never, ever modify the git-svn-HEAD branch yourself.
> Interface branches should never be modified.  It's the golden rule of
> interfacing between different SCM interfaces.  Sorry, I've been doing
> things like this this for a while now I guess I didn't make it
> abundantly clear in the documentation.

ok, i experienced that on little modifications on the git-svn-HEAD branch
either... so its really about modifying and not about the huge data
ammount...


>> now i am still on rev 2 on this branch but i updated it to rev 5 on the
>> svn-side...
>>
>> any hints?
>
> Save your current work in git-svn-HEAD to a private branch
>
> 	git branch -b private git-svn-HEAD
>
> then reset git-svn-HEAD to the last revision where it was managed by
> git-svn fetch:
>
> 	git-checkout git-svn-HEAD
> 	git-log (look for the last commit with 'git-svn-id:' in it)
> 	git-reset --hard <last commit with 'git-svn-id:' in it>
>
> Now go to your private branch:
>
> 	git checkout private
>
> And continue working on your private branch as usual.

I will keep that in mind for the future. Fortunatelly i am still testing
and i saved the git repository before experimenting with git-svn.

Have you any suggestions howto migrate a git-repository to svn and then
work with git-svn on both of it? I tried cg-merge -j to merge my git
branch with the private git svn branch, i am allowed to modify safely.

that does work actually... now i can start getting this automated.

perhaps i will write a patch with that automated script, when it is
finished, just to contribute git.


Sincerly
Nicolas

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Eric Wong]

Nicolas Vilz 'niv' <niv@iaglans.de> wrote:
> ok, i experienced that on little modifications on the git-svn-HEAD branch
> either... so its really about modifying and not about the huge data
> ammount...
 
Huge data should not have anything to do with it.  Well, besides
increasing the chance of somebody committing a conflicting commit while
you're in the middle of your commit.  But hey, that's the nature of
centralized SCMs.

> >> now i am still on rev 2 on this branch but i updated it to rev 5 on the
> >> svn-side...
> >>
> >> any hints?
> >
> > Save your current work in git-svn-HEAD to a private branch
> >
> > 	git branch -b private git-svn-HEAD
> >
> > then reset git-svn-HEAD to the last revision where it was managed by
> > git-svn fetch:
> >
> > 	git-checkout git-svn-HEAD
> > 	git-log (look for the last commit with 'git-svn-id:' in it)
> > 	git-reset --hard <last commit with 'git-svn-id:' in it>
> >
> > Now go to your private branch:
> >
> > 	git checkout private
> >
> > And continue working on your private branch as usual.
> 
> I will keep that in mind for the future. Fortunatelly i am still testing
> and i saved the git repository before experimenting with git-svn.

:)

> Have you any suggestions howto migrate a git-repository to svn and then
> work with git-svn on both of it? I tried cg-merge -j to merge my git
> branch with the private git svn branch, i am allowed to modify safely.
> 
> that does work actually... now i can start getting this automated.
> 
> perhaps i will write a patch with that automated script, when it is
> finished, just to contribute git.

Cool.  I don't know much about cg-*, but I think I did more or less the
same thing (joining branches, but did the join on git-svn-HEAD instead
of a git-only branch) using <revision>=<commit> arguments[1] to git-svn
fetch.

[1] - See the 'Additional Fetch Arguments' section of the manpage for
more info on this.  I'll freely admit that the UI for this was an
accident, but it works fairly well for me.

-- 
Eric Wong

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Nicolas Vilz 'niv']

Eric Wong wrote:
> [1] - See the 'Additional Fetch Arguments' section of the manpage for
> more info on this.  I'll freely admit that the UI for this was an
> accident, but it works fairly well for me.

btw, i think you have a typo in your man-page:
---
# Commit only the git commits you want to SVN::
        git-svn commit <tree-ish> [<tree-ish_2> ...]
# Commit all the git commits from my-branch that don't exist in SVN::
        git commit git-svn-HEAD..my-branch
---
i think the second one is intended to be "git-svn commit
git-svn-HEAD..my-branch", because you want to sync the SVN tree again, not
another git-tree i think...

Nicolas

[PATCH] contrib/git-svn: correct commit example in manpage

[Eric Wong]

Thanks to Nicolas Vilz <niv@iaglans.de> for noticing this.

Signed-off-by: Eric Wong <normalperson@yhbt.net>

---

 contrib/git-svn/git-svn.txt |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

125c2e90f26d8980f415f8066dcc84d02d95f03a
diff --git a/contrib/git-svn/git-svn.txt b/contrib/git-svn/git-svn.txt
index b588a2a..b290739 100644
--- a/contrib/git-svn/git-svn.txt
+++ b/contrib/git-svn/git-svn.txt
@@ -159,7 +159,7 @@ Tracking and contributing to an Subversi
 # Commit only the git commits you want to SVN::
 	git-svn commit <tree-ish> [<tree-ish_2> ...]
 # Commit all the git commits from my-branch that don't exist in SVN::
-	git commit git-svn-HEAD..my-branch
+	git-svn commit git-svn-HEAD..my-branch
 # Something is committed to SVN, pull the latest into your branch::
 	git-svn fetch && git pull . git-svn-HEAD
 # Append svn:ignore settings to the default git exclude file:
-- 
1.2.3.g4676

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Martin Langhoff]

On 2/28/06, Eric Wong <normalperson@yhbt.net> wrote:
> > If it is not supposed to be changed by the user, maybe it could be
> > stored as a tag.
> >
> > Or maybe another type of reference can be introduced. refs/remote/, for
> > branches we are tracking, but which should not be modified locally.
>
> Either of those could work for me.  Changing git-svn-HEAD to become a
> tag would probably be easier (not having to update other tools, such as
> git-fetch), but refs/remote may make more sense.

git-svn-HEAD "moves" so it's really a bad idea to have it as a tag.
Nothing within core git prevents it from moving, but I think that
porcelains will start breaking. Tags and heads are the same thing,
except that heads are expected to change (specifically, to move
forward), and tags are expected to stand still.

Something else is needed -- a convention to mark a head as 'readonly'
so that git-commit/cg-commit refuse to commit to it. cg-commit already
does that for any head matching the name of a branch.

cheers,


martin

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Linus Torvalds]

On Tue, 28 Feb 2006, Martin Langhoff wrote:
> 
> git-svn-HEAD "moves" so it's really a bad idea to have it as a tag.
> Nothing within core git prevents it from moving, but I think that
> porcelains will start breaking. Tags and heads are the same thing,
> except that heads are expected to change (specifically, to move
> forward), and tags are expected to stand still.

Well, I wouldn't say that tags are expected to stand still. Some kinds of 
tags are expected to move: a "this is the last tested version" tag would 
be expected to move with testing. 

That said, the movement is _different_ from a branch. A branch is expected 
to move _with_ development, while a tag is expected to either stay the 
same, or move _after_ development.

However, in many ways git really doesn't care much. The "refs/heads" 
directory is the only one that is really special, in that "git checkout" 
refuses to check out a moving branch in anything but that subdirectory. 
The "tags" subdirectory is slightly special to some helpers (like "git 
pull"), which have flags to pull everythying in that subdirectory. 

But other than those two pretty trivial issues, any ref under "refs/" 
should work perfectly fine. I would argue that a specialized tracking tool 
might well be better off without using either "refs/heads" _or_ 
"refs/tags", since those have accepted meaning outside of tracking.

Using a "refs/remotes" subdirectory makes tons of sense for something like 
this. Or something even more specific, like "refs/svn-tracking/". Git 
shouldn't care - all the tools _should_ work fine with any subdirectory 
structure.

		Linus

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Martin Langhoff]

On 2/28/06, Linus Torvalds <torvalds@osdl.org> wrote:
> On Tue, 28 Feb 2006, Martin Langhoff wrote:
> > git-svn-HEAD "moves" so it's really a bad idea to have it as a tag.
> > Nothing within core git prevents it from moving, but I think that
> > porcelains will start breaking. Tags and heads are the same thing,
> > except that heads are expected to change (specifically, to move
> > forward), and tags are expected to stand still.
>
> Well, I wouldn't say that tags are expected to stand still. Some kinds of
> tags are expected to move: a "this is the last tested version" tag would
> be expected to move with testing.

Alrighty... in my git projects where things like these matter, my
"latest tested" and "current in production" refs are actually in
refs/heads.

> That said, the movement is _different_ from a branch. A branch is expected
> to move _with_ development, while a tag is expected to either stay the
> same, or move _after_ development.

Grumble. I'd say a head is expected to reliably move _forward_...
"with" development, yes, but definitely forward. In my book a tag
wouldn't move, but if I take your word for it, then a tag can perhaps
change arbitrarily?

I'm not sure how much support we have in porcelains for "tracking" a
tag if it starts changing. Right now I think we'd find all sorts of
problems, we'd need to think carefully what moving tags means for
porcelains.

> Or something even more specific, like "refs/svn-tracking/". Git
> shouldn't care - all the tools _should_ work fine with any subdirectory
> structure.

I think the moving-forward (therefore is trackable) vs stays reliably
in place distinction *is* useful. "Moves randomly" may also be useful,
but it should get a different treatment, because it's not "trackable".

Not that git and porcelains can't deal with all this stuff. But if
there is a clear convention then porcelains can be smart and refuse to
commit to the wrong place... it'd be a bit of a UI enhancement
perhaps?


martin

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Eric Wong]

Linus Torvalds <torvalds@osdl.org> wrote:
> 
> 
> On Tue, 28 Feb 2006, Martin Langhoff wrote:
> > 
> > git-svn-HEAD "moves" so it's really a bad idea to have it as a tag.
> > Nothing within core git prevents it from moving, but I think that
> > porcelains will start breaking. Tags and heads are the same thing,
> > except that heads are expected to change (specifically, to move
> > forward), and tags are expected to stand still.
> <snipped>
> Using a "refs/remotes" subdirectory makes tons of sense for something like 
> this. Or something even more specific, like "refs/svn-tracking/". Git 
> shouldn't care - all the tools _should_ work fine with any subdirectory 
> structure.

Git tools only work as long as the 'refs/{remotes,svn-tracking,...}/'
prefix is specified.  git-svn-HEAD (or any $GIT_SVN_ID-HEAD) does get
specified from the command-line quite often:
	
	git checkout -b mine git-svn-HEAD
	git-log git-svn-HEAD..head
	git-svn commit git-svn-HEAD..mine
	git-log mine..git-svn-HEAD

Should rev-parse be taught to be less strict and look for basenames
that can't be found in heads/ and tags/ in other directories?

-- 
Eric Wong

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Andreas Ericsson]

Eric Wong wrote:
> Linus Torvalds <torvalds@osdl.org> wrote:
> 
>>
>>On Tue, 28 Feb 2006, Martin Langhoff wrote:
>>
>>>git-svn-HEAD "moves" so it's really a bad idea to have it as a tag.
>>>Nothing within core git prevents it from moving, but I think that
>>>porcelains will start breaking. Tags and heads are the same thing,
>>>except that heads are expected to change (specifically, to move
>>>forward), and tags are expected to stand still.
>>
>><snipped>
>>Using a "refs/remotes" subdirectory makes tons of sense for something like 
>>this. Or something even more specific, like "refs/svn-tracking/". Git 
>>shouldn't care - all the tools _should_ work fine with any subdirectory 
>>structure.
> 
> 
> Git tools only work as long as the 'refs/{remotes,svn-tracking,...}/'
> prefix is specified.  git-svn-HEAD (or any $GIT_SVN_ID-HEAD) does get
> specified from the command-line quite often:
> 	
> 	git checkout -b mine git-svn-HEAD
> 	git-log git-svn-HEAD..head
> 	git-svn commit git-svn-HEAD..mine
> 	git-log mine..git-svn-HEAD
> 
> Should rev-parse be taught to be less strict and look for basenames
> that can't be found in heads/ and tags/ in other directories?
> 

It already does. The search order is this, for a ref named 'foo':
	$GIT_DIR/foo
	$GIT_DIR/refs/foo
	$GIT_DIR/refs/tags/foo
	$GIT_DIR/refs/heads/foo

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Linus Torvalds]

On Wed, 1 Mar 2006, Andreas Ericsson wrote:
>
> Eric Wong wrote:
> > 
> > Should rev-parse be taught to be less strict and look for basenames
> > that can't be found in heads/ and tags/ in other directories?
> 
> It already does. The search order is this, for a ref named 'foo':
> 	$GIT_DIR/foo
> 	$GIT_DIR/refs/foo
> 	$GIT_DIR/refs/tags/foo
> 	$GIT_DIR/refs/heads/foo

Yes, but I think Eric wanted to avoid having to write the prefix part, 
which git won't let you do right now.

If you have a ref in .git/refs/svn-tracker/git-svn-HEAD, you would have to 
write out all of "svn-tracker/git-svn-HEAD", because unlike a "real 
branch", get_sha1() won't look into the "svn-tracker" without it being 
explicitly mentioned.

Now, some tools will actually do "for_each_ref()" and check the ref-name 
against each of them (so if you pass in "foo", it will check them afainst 
_any_ ref-subdirectory that contains "foo"). But get_sha1() won't.

We could fix get_sha1(), but part of the logic was that other 
subdirectories are special, and as such they _should_ be mentioned, so 
that a file in such a special directory isn't ever confused with a real 
branch.

But if you were to use for example .git/refs/git-svn/tracking as the 
svn-tracking reference head, and then you'd be perfectly able to use

	git log git-svn/tracking..

to see what you've done since the last svn import?

(or use HEAD, if you prefer that over "tracking")

		Linus

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Andreas Ericsson]

Linus Torvalds wrote:
> 
> On Wed, 1 Mar 2006, Andreas Ericsson wrote:
> 
>>Eric Wong wrote:
>>
>>>Should rev-parse be taught to be less strict and look for basenames
>>>that can't be found in heads/ and tags/ in other directories?
>>
>>It already does. The search order is this, for a ref named 'foo':
>>	$GIT_DIR/foo
>>	$GIT_DIR/refs/foo
>>	$GIT_DIR/refs/tags/foo
>>	$GIT_DIR/refs/heads/foo
> 
> 
> Yes, but I think Eric wanted to avoid having to write the prefix part, 
> which git won't let you do right now.
> 
> If you have a ref in .git/refs/svn-tracker/git-svn-HEAD, you would have to 
> write out all of "svn-tracker/git-svn-HEAD", because unlike a "real 
> branch", get_sha1() won't look into the "svn-tracker" without it being 
> explicitly mentioned.
> 
> Now, some tools will actually do "for_each_ref()" and check the ref-name 
> against each of them (so if you pass in "foo", it will check them afainst 
> _any_ ref-subdirectory that contains "foo"). But get_sha1() won't.
> 

Didn't know that. The day is not a complete waste then.


> We could fix get_sha1(), but part of the logic was that other 
> subdirectories are special, and as such they _should_ be mentioned, so 
> that a file in such a special directory isn't ever confused with a real 
> branch.
> 
> But if you were to use for example .git/refs/git-svn/tracking as the 
> svn-tracking reference head, and then you'd be perfectly able to use
> 
> 	git log git-svn/tracking..
> 
> to see what you've done since the last svn import?
> 

Personally I'm all for namespace separation. I'm assuming the script has 
the tracker-branch hardcoded anyway, so I don't really understand why it 
would be necessary to keep other refs in a separate directory and, if it 
*is* necessary, why that subdirectory can't be .git/refs/heads/svn.

Eric mentioned earlier that the tracking-branch can't be committed to 
(ever), so the user convenience for searching other directories should 
be nearly non-existant.

Perhaps I'm missing something obvious. Perhaps I'm just stupid. Perhaps 
the pub just opened and I don't feel like reading it twice to make sure 
I understood. ;)

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Linus Torvalds]

On Wed, 1 Mar 2006, Andreas Ericsson wrote:
> 
> Personally I'm all for namespace separation. I'm assuming the script has the
> tracker-branch hardcoded anyway, so I don't really understand why it would be
> necessary to keep other refs in a separate directory and, if it *is*
> necessary, why that subdirectory can't be .git/refs/heads/svn.
> 
> Eric mentioned earlier that the tracking-branch can't be committed to (ever),
> so the user convenience for searching other directories should be nearly
> non-existant.

The thing about it being .git/refs/heads/svn/xyzzy is that then you can do

	git checkout svn/xyzzy

and start modifying it. Which is exactly against the point: the thing is 
_not_ a branch and you must _not_ commit to it.

It's much more like a tag: it's a pointer to the last point of an 
svn-import.

So I think it should either _be_ a tag (although Dscho worries about some 
broken porcelain being confused by tags changing) or it should be in a 
namespace all it's own. Not under .git/refs/heads/ at any point, because 
it is _not_ a head of development.

		Linus

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Josef Weidendorfer]

On Wednesday 01 March 2006 17:24, Linus Torvalds wrote:
> The thing about it being .git/refs/heads/svn/xyzzy is that then you can do
> 
> 	git checkout svn/xyzzy
> 
> and start modifying it. Which is exactly against the point: the thing is 
> _not_ a branch and you must _not_ commit to it.
> 
> It's much more like a tag: it's a pointer to the last point of an 
> svn-import.

Isn't it the same with tracked branches of a remote git repo?
With this reasoning, all heads that git-clone clones aside from the
special "master" should not be under .git/refs/heads, but better
under .git/refs/remotes/<remoteRepoName>/ ?

<remoteRepoName> is "origin" in the case of git-clone, so .git/remotes/origin
would contain
 URL: http://host/repo.git
 Pull: master:remotes/origin/master

Then there would not be the need for the confusing special branch "origin"
after cloning, as namespaces are separate.

Josef

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Shawn Pearce]

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> On Wednesday 01 March 2006 17:24, Linus Torvalds wrote:
> > The thing about it being .git/refs/heads/svn/xyzzy is that then you can do
> > 
> > 	git checkout svn/xyzzy
> > 
> > and start modifying it. Which is exactly against the point: the thing is 
> > _not_ a branch and you must _not_ commit to it.
> > 
> > It's much more like a tag: it's a pointer to the last point of an 
> > svn-import.
> 
> Isn't it the same with tracked branches of a remote git repo?
> With this reasoning, all heads that git-clone clones aside from the
> special "master" should not be under .git/refs/heads, but better
> under .git/refs/remotes/<remoteRepoName>/ ?
> 
> <remoteRepoName> is "origin" in the case of git-clone, so .git/remotes/origin
> would contain
>  URL: http://host/repo.git
>  Pull: master:remotes/origin/master
> 
> Then there would not be the need for the confusing special branch "origin"
> after cloning, as namespaces are separate.

This is a really good idea.  It certainly would prevent polluting the
heads namespace.  And its a lot easier to explain to someone than the
mapping in the Pull line usually is.

-- 
Shawn.

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Linus Torvalds]

On Wed, 1 Mar 2006, Josef Weidendorfer wrote:
>
> On Wednesday 01 March 2006 17:24, Linus Torvalds wrote:
> > The thing about it being .git/refs/heads/svn/xyzzy is that then you can do
> > 
> > 	git checkout svn/xyzzy
> > 
> > and start modifying it. Which is exactly against the point: the thing is 
> > _not_ a branch and you must _not_ commit to it.
> > 
> > It's much more like a tag: it's a pointer to the last point of an 
> > svn-import.
> 
> Isn't it the same with tracked branches of a remote git repo?
> With this reasoning, all heads that git-clone clones aside from the
> special "master" should not be under .git/refs/heads, but better
> under .git/refs/remotes/<remoteRepoName>/ ?

Yes, I think that would make tons of sense.

> <remoteRepoName> is "origin" in the case of git-clone, so .git/remotes/origin
> would contain
>  URL: http://host/repo.git
>  Pull: master:remotes/origin/master
> 
> Then there would not be the need for the confusing special branch "origin"
> after cloning, as namespaces are separate.

I think that would make things a lot more flexible, and yes, it sounds 
like a good idea.

HOWEVER.

I think it's not only very common, but quite useful, to do what we do now, 
ie

	git log origin..

to see "what is in origin but not in HEAD".

So there's a big usability issue: I don't think it's good to have to say

	git log remotes/origin/master..

to do the same.

So from a usability standpoint, we'd have to teach "get_sha1()" about 
parsing .git/remotes/* files if it cannot find a branch or a tag with that 
name (which it wouldn't be able to, since even if it were to walk the 
directories udner .git/refs/ recursively, it would be named "master" 
there).

But if somebody does the get_sha1() magic, and Junio agrees, then I think 
it would be a great thing to do.

			Linus

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Josef Weidendorfer]

On Wednesday 01 March 2006 18:40, Linus Torvalds wrote:
> But if somebody does the get_sha1() magic, and Junio agrees, then I think 
> it would be a great thing to do.

Yes.

	git log origin/master..

is really not that bad. And if somebody complains about typing, git-clone
could get an option "--remote-name=o" to allow for

	git log o/master..

Josef

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Linus Torvalds]

On Wed, 1 Mar 2006, Josef Weidendorfer wrote:
>
> On Wednesday 01 March 2006 18:40, Linus Torvalds wrote:
> > But if somebody does the get_sha1() magic, and Junio agrees, then I think 
> > it would be a great thing to do.
> 
> Yes.
> 
> 	git log origin/master..
> 
> is really not that bad

It really is.

Think like a user. If I pull from "origin", then the name of that thing is 
"origin", not "origin/master" or "o/master". A user doesn't care what the 
remote branch name is - the whole _point_ of the .git/remotes/xyzzy file 
is to give a short description that includes the names of the branches you 
pull from.

The good news is that "get_sha1()" shouldn't be that hard to extend on. 
Just add a case at the end that says "do we have a .git/remotes/%s file, 
and if so, parse it".

				Linus

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> But if somebody does the get_sha1() magic, and Junio agrees, then I think 
> it would be a great thing to do.

I am inclined to agree here.

Some caveats upfront, though.

Since I was bitten at least once by attempting get_sha1() to
deal with ambiguous names (the issue was between heads and tags
but I think there are similar issues here) I am really reluctant
to have the function look at anywhere other than heads/ and
tags/ without explicit prefix.

Currently get_sha1_basic() says:

	* look in $GIT_DIR with these prefixes in turn and take
          the first match: "", "refs", "refs/tags", "refs/heads".

The extended one _would_ in addition say one of these things:

	* if none of the above prefixes work, try other
          directories under refs/ as prefixes and take the first
          match.

	or

	* if none of the above prefixes work, try other
          directories under refs/ as prefixes and if there is a
          unique match take it.  If there are more than one
          match, do not take either.

In the context of get_sha1(), get_sha1_basic() is used like
this:

	* if get_sha1_basic() finds an answer, use it.
          Otherwise see if it is an abbreviated object name.

The behaviour of a naive implementation of the former would
depend on readdir() and traversal order, which makes (from the
end user's point of view) a hard to understand confusion that is
not reproducible.  Another repository cloned from such would
even give you different answers.

The latter at first sounds sane, but it has a subtle issue,
which was what bitten me previously between heads/ and tags/.
In that broken version, if you have a head called "dead" and a
tag with the same name, neither was taken ("they are not unique,
so do not take either!") and we ended up finding an object whose
SHA1 name began with those two bytes 0xDE 0xAD.  I do not think
this has happened in the field, fortunately, but it would have
been quite hard to diagnose.

So if we were to do it, I would say do the latter, but be very
careful to make sure you fail the whole get_sha1() when you bail
out of the "try possible prefixes" codepath because of
ambiguity.  There may be other issues involved, but I wouldn't
know -- I reverted the "do not take either if they are
ambiguous between heads/ and tags/" patch primarily because of
the reason from the above paragraph, but also did not want to
deal with any other potential issues to keep my sanity ;-).

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Josef Weidendorfer]

On Wednesday 01 March 2006 19:25, Linus Torvalds wrote:
> > 	git log origin/master..
> > 
> > is really not that bad
> 
> It really is.
> 
> Think like a user. If I pull from "origin", then the name of that thing is 
> "origin", not "origin/master" or "o/master". A user doesn't care what the  
> remote branch name is - the whole _point_ of the .git/remotes/xyzzy file 
> is to give a short description that includes the names of the branches you 
> pull from.

So the get_sha1() magic should map "origin" to "remote/origin/master" (or instead
hardcoded master the remote branch from the first "Pull:" line) ?
The ambiguity here would be that shortcut names of remote repositories should not be
used as tag or head names...

I think a big plus of this would be that gitk can show branches tracking remote ones
with another color.
 
> The good news is that "get_sha1()" shouldn't be thse at hard to extend on. 
> Just add a case at the end that says "do we have a .git/remotes/%s file, 
> and if so, parse it".

To be able to say "git log origin.." you need the above magic, too.

Josef

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Josef Weidendorfer]

On Wednesday 01 March 2006 20:11, Junio C Hamano wrote:
> The latter at first sounds sane, but it has a subtle issue,
> which was what bitten me previously between heads/ and tags/.
> In that broken version, if you have a head called "dead" and a
> tag with the same name, neither was taken ("they are not unique,
> so do not take either!") and we ended up finding an object whose
> SHA1 name began with those two bytes 0xDE 0xAD.  I do not think
> this has happened in the field, fortunately, but it would have
> been quite hard to diagnose.
> 
> So if we were to do it, I would say do the latter, but be very
> careful to make sure you fail the whole get_sha1() when you bail
> out of the "try possible prefixes" codepath because of
> ambiguity.

Yes.
Any ambiguity is a source of confusion and user error. Better
bail out. If it is not a performance problem, it would be better
to integrate the check for abbreviated object name into the
ambiguity analysis, and not have 2 stages of searching.
It probably would be a good idea to print out the ambigous names
with the error message, so that you can copy&paste the correct
full name afterwards.

If we go for the .git/refs/remotes/... and have an ambiguity becaues
of remote shortcut names, a error message pointing at a "git-rename-remote"
command would be handy, allowing the user to cleanup the namespace.

> There may be other issues involved, but I wouldn't 
> know -- I reverted the "do not take either if they are
> ambiguous between heads/ and tags/" patch primarily because of
> the reason from the above paragraph, but also did not want to
> deal with any other potential issues to keep my sanity ;-).

I think the real problem here is that names like "dead" can be interpreted
as abbreviated object name. When you introduce such a name as head or tag,
you have a potential ambiguity which can get real at any time.
Perhaps it would be good to print out a warning when the user is about to
create a head or tag name which can be interpreted as abbreviated object name?

Josef

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Johannes Schindelin]

Hi,

On Wed, 1 Mar 2006, Linus Torvalds wrote:

> On Wed, 1 Mar 2006, Andreas Ericsson wrote:
> > 
> > Personally I'm all for namespace separation. I'm assuming the script 
> > has the tracker-branch hardcoded anyway, so I don't really understand 
> > why it would be necessary to keep other refs in a separate directory 
> > and, if it *is* necessary, why that subdirectory can't be 
> > .git/refs/heads/svn.
> > 
> > Eric mentioned earlier that the tracking-branch can't be committed to 
> > (ever), so the user convenience for searching other directories should 
> > be nearly non-existant.
> 
> The thing about it being .git/refs/heads/svn/xyzzy is that then you can 
> do
> 
> 	git checkout svn/xyzzy
> 
> _not_ a branch and you must _not_ commit to it.
> 
> It's much more like a tag: it's a pointer to the last point of an 
> svn-import.
> 
> So I think it should either _be_ a tag (although Dscho worries about some 
> broken porcelain being confused by tags changing) or it should be in a 
> namespace all it's own. Not under .git/refs/heads/ at any point, because 
> it is _not_ a head of development.

I almost missed that you reference me in the email (often, I just delete 
the email if the Subject is of no interest to me).

I did not worry about broken porcelain. I saw broken porcelain. But that 
is more a broken concept than broken porcelain: in a distributed 
environment, there is no way to have a reliable tag. Think about it: 
whenever you have two different versions of a tag, you cannot know which 
one is the correct one.

But my worries do not matter at all for local tags.

Conceptually, however, the last point of a svnimport should *never* be a 
tag, but *always* a head.

Ciao,
Dscho

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Linus Torvalds]

On Wed, 1 Mar 2006, Josef Weidendorfer wrote:
> 
> So the get_sha1() magic should map "origin" to "remote/origin/master" (or instead
> hardcoded master the remote branch from the first "Pull:" line) ?

Right.

> The ambiguity here would be that shortcut names of remote repositories should not be
> used as tag or head names...

Well, it's not so much an ambiguity, since we'd always try tags and heads 
first. So it's just a fallback, the same way the short SHA1 hash is a 
fallback.

> I think a big plus of this would be that gitk can show branches tracking remote ones
> with another color.

Yes. And with a meaningful name.

> To be able to say "git log origin.." you need the above magic, too.

It would all come automagically from just extending get_sha1().

(Actually, technically you'd put it at the end of "get_sha1_basic()")

		Linus

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Martin Langhoff]

On 3/2/06, Junio C Hamano <junkio@cox.net> wrote:
> Linus Torvalds <torvalds@osdl.org> writes:
>
> > But if somebody does the get_sha1() magic, and Junio agrees, then I think
> > it would be a great thing to do.
>
> I am inclined to agree here.

Aren't we doing a lot of work (changes in core git, and corresponding
changes in the porcelain) when simple changes in porcelain would
suffice? Let's imagine that

 - git-commit refuses to commit to a head that has a corresponding
remote (cg-commit does this already with heads that match something in
'branches')
 - git-$SCMimport scripts generate a semi-bogus remotes/headname entry
 - git-pull/push can spot and ignore the semi-bogus remotes/headname entry
 - this means that `touch remotes/foo` is now a cheap way of making
the head readonly
 - depending on the git-$SCMimport script, the remotes/headname file
can perhaps contain useful configuration data for the import, so
git-$SCMimport headname does the right thing.

cheers,


martin

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

On Thu, 2 Mar 2006 10:40:41 +1300, "Martin Langhoff" wrote:
> 
> Aren't we doing a lot of work (changes in core git, and corresponding
> changes in the porcelain) when simple changes in porcelain would
> suffice? Let's imagine that

There might be a simpler change to solve the git-svn-HEAD issue. But I
was about to independently bring up the issue that I wanted to hide
away the "remote-tracking" branches.

I currently get a lot of noise in "git branch" output that are
remote-tracking branches that I will never commit to. (I use a -origin
suffix to help me filter them out, but I'd prefer not to see them at
all here.)

Meanwhile, as I've been teaching new git users, I've had to carefully
teach:

1) Never commit to a branch name that appears on the right side of ':'
   in a Pull: refspec.

2) BTW, that ':' might be only implicit. A refspec of "branch" is
   equivalent to "branch:branch" so don't commit to those either.

That's pretty painful, so I really think these remote-tracking refs
belong outside of refs/heads.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Linus Torvalds]

On Wed, 1 Mar 2006, Carl Worth wrote:

> Meanwhile, as I've been teaching new git users, I've had to carefully
> teach:
> 
> 1) Never commit to a branch name that appears on the right side of ':'
>    in a Pull: refspec.
> 
> 2) BTW, that ':' might be only implicit. A refspec of "branch" is
>    equivalent to "branch:branch" so don't commit to those either.
> 
> That's pretty painful, so I really think these remote-tracking refs
> belong outside of refs/heads.

In the same vein, I think the refs/remotes/<remotename>/<branchname> 
naming will make it possible for people who track multiple remotes to 
sanely work with the fact that they might track 10 separate branches from 
Jeff, one branch from me, and a couple of branches from Greg in the same 
tree, without just going crazy.

I agree that we could solve the "don't touch that branch" issue another 
way, by just making them read-only. But the reason I like the separate 
namespace is that it just seems to organize the branches really well, and 
in an unambiguous - and logical - manner.

I too find myself looking at "git branch" output, and a lot of it is stuff 
I don't really care about - much of it is just the branches I got for just 
tracking Junio's git repo.

		Linus

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Petr Baudis]

Dear diary, on Wed, Mar 01, 2006 at 10:40:01AM CET, I got a letter
where Andreas Ericsson <ae@op5.se> said that...
> It already does. The search order is this, for a ref named 'foo':
> 	$GIT_DIR/foo
> 	$GIT_DIR/refs/foo
> 	$GIT_DIR/refs/tags/foo
> 	$GIT_DIR/refs/heads/foo

Actually, I've hit this recently when supporting an unhappy user on
#git, and I didn't manage to find anything in the archives (but perhaps
I missed it). Is there a particular reason why tags are checked first
than branches?

Why not:

(i) I _think_ that it would be less of a surprise if a branch would be
checked first.

(ii) E.g. Cogito output (cg-status -g) is very confusing when you have a
naming clash - cg-object-id foo will show tag commit ID, but cg-status -g
will say that the "foo" branch has a different commit ID (and it is
_right_).

(iii) Many operations will stop making sense (cg-merge foo, and even
cg-fetch foo will be confused), while in case of the opposite way I can't
think of any command still not making sense.

(iv) A security hole when you auto-fetch tags from remote repositories
- you could then be misled to merge something totally different when the
attacker will introduce a naming clash to your refs hierarchy.

Actually, I'm almost inclined to suggest making Git fail violently in
case of an ambiguous name.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time.  I think
I have forgotten this before.

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Linus Torvalds]

On Sun, 19 Mar 2006, Petr Baudis wrote:
> 
> (i) I _think_ that it would be less of a surprise if a branch would be
> checked first.

Yeah, I guess that's true.

> Actually, I'm almost inclined to suggest making Git fail violently in
> case of an ambiguous name.

Maybe not fail, but at least warn very loudly.

		Linus

Re: git-svn and huge data and modifying the git-svn-HEAD branch directly

[Junio C Hamano]

Petr Baudis <pasky@suse.cz> writes:

> Actually, I'm almost inclined to suggest making Git fail violently in
> case of an ambiguous name.

I am also inclined to suggest that or alternatively making it
warn, but having been almost burned by a bug or two coming from
the complexity of implementing it, I am not so enthused to start
hacking away again on this right now myself.