Re: 2.6.17-rc6-mm2

Re: 2.6.17-rc6-mm2

[Goo GGooo]

On 6/15/06, Goo GGooo <googgooo@gmail.com> wrote:
> Andrew Morton wrote:
>
> > - To fetch an -mm tree using git, use (for example)
> >
> >  git fetch git://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git
> > v2.6.16-rc2-mm1
>
> I'm not able to get -mm tree from GIT. In
> http://git.kernel.org/.../smurf/linux-trees.git/refs/tags/ I can see
> the most recent tags like v2.6.17-rc6-mm2 but cg-clone
> http://git.kernel.org/.../smurf/linux-trees.git gives me only
> 2.6.16-rc3 :(
>
> I tried "cg-fetch v2.6.17-rc6-mm2" which seemed to fetch some more
> tags, then played with git-checkout & friends but still can't get the
> most recent source tree.

All right, finally this worked out:
git pull rsync://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git \
      tag v2.6.17-rc6-mm2

Strange enough with http:// instead of rsync:// I got some message
about nonexistent tag.

Now when I try git pull with http:// again it says the tree is up to
date. However with git:// it started downloading more things and tags.

That's confusing - I believed all protocols should behave the same way...?

Goo

Re: 2.6.17-rc6-mm2

[Linus Torvalds]

On Fri, 16 Jun 2006, Goo GGooo wrote:
> 
> That's confusing - I believed all protocols should behave the same way...?

Not really. The primary protocol is the native git one, and the others try 
to do a best effort, but the http protocol really can't do a very good 
job unless the server side has run "git update-server-info" to help the 
http client along.

I suspect that the -mm git tree simply doesn't do that. In fact, even the 
main tree didn't use to do it, but I finally just broke down and added the 
proper hook to make it always do it automatically when I push.

(In case Andrew wants to do that, the way to do it is:

	echo -e "#!/bin/sh\nexec git-update-server-info" > hooks/post-update
	chmod +x hooks/post-update

inside the git repository - all it will do is always execute that script, 
and this "git-update-server-info", after you've updated the repo).

Finally, the rsync protocol just copies all objects over, and since it 
doesn't even know _which_ objects it is getting, it doesn't do the normal 
tag following that the native git protocol does.

So to recap:
 - http is fundamentally weaker, and needs some server-side help to work
 - rsync is fine for the initial clone, but doesn't actually know what 
   it's doing, so the end result can actually even be a corrupted 
   repository, because you happened to rsync just as it was updating.
 - the native git protocol generally should be considered the golden 
   standard, where the other ones are just fallbacks in case of problems 
   (like firewalls that don't let git:// through, or more commonly hosted 
   servers that don't do the git protocol at all).

Which hopefully clarifies the issue a bit.

		Linus

Re: 2.6.17-rc6-mm2

[Goo GGooo]

On 6/16/06, Linus Torvalds <torvalds@osdl.org> wrote:

> So to recap:
>  - http is fundamentally weaker, and needs some server-side help to work
>  - rsync is fine for the initial clone, but doesn't actually know what
>    it's doing, so the end result can actually even be a corrupted
>    repository, because you happened to rsync just as it was updating.
>  - the native git protocol generally should be considered the golden
>    standard, where the other ones are just fallbacks in case of problems
>    (like firewalls that don't let git:// through, or more commonly hosted
>    servers that don't do the git protocol at all).
>
> Which hopefully clarifies the issue a bit.

Thanks for explanation. Unfortunately I can't use git:// with "git
pull" (at least in git-1.3.2). First it does some traffic, that
suddenly stops - I guess the server starts doing *something*, perhaps
preparing the update for me or whatnot. After a pretty long while it
sends some more data but in the meanwhile my ADSL router dropped the
NAT entry and git sits on my side waiting for data forever. Recently I
tried the same on a system with direct Inet connection and that worked
just fine.

I suggest adding SO_KEEPALIVE option on the git socket.

Goo

Re: 2.6.17-rc6-mm2

[Linus Torvalds]

On Fri, 16 Jun 2006, Goo GGooo wrote:
> 
> Thanks for explanation. Unfortunately I can't use git:// with "git
> pull" (at least in git-1.3.2). First it does some traffic, that
> suddenly stops - I guess the server starts doing *something*, perhaps
> preparing the update for me or whatnot.

Yeah, for a big pull, the server will have to think about the objects it 
is going to send you.

> I suggest adding SO_KEEPALIVE option on the git socket.

Actually, the really irritating thing is that we actually generate all 
these nice status updates, which just makes pulling and cloning a lot more 
comfortable, because you actually see what is going on, and what to 
expect. 

Except they only work over ssh, where we have a separate channel (for 
stderr), and with the native git protocol all that nice status work just 
gets flushed to /dev/null :(

Dang. It's literally the most irritating part of the thing: the protocol 
itself is exactly the same whether you go over ssh:// or over git://, but 
that visual information about what is going on is missing, and it's 
surprisingly important from a usability standpoint.

And in your case, the usability downside actually turned into a real 
accessibility bug.

Oh, well.

		Linus

Re: 2.6.17-rc6-mm2

[Uwe Zeisberger]

Hello,

> I suggest adding SO_KEEPALIVE option on the git socket.
I suggest to do this "manually", that is send an dummy (or status)
package every x seconds.  Then the server could detect if a cloning
client disconnected and stop generating the pack file.

(Currently I see from time to time a git server process (IIRC
git-pack-objects) that creates a packfile and only when it's done fails
to send it.)

Best regards
Uwe

-- 
Uwe Zeisberger

http://www.google.com/search?q=30+hours+and+4+days+in+seconds

Re: 2.6.17-rc6-mm2

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> Actually, the really irritating thing is that we actually generate all 
> these nice status updates, which just makes pulling and cloning a lot more 
> comfortable, because you actually see what is going on, and what to 
> expect. 
> 
> Except they only work over ssh, where we have a separate channel (for 
> stderr), and with the native git protocol all that nice status work just 
> gets flushed to /dev/null :(
> 
> Dang. It's literally the most irritating part of the thing: the protocol 
> itself is exactly the same whether you go over ssh:// or over git://, but 
> that visual information about what is going on is missing, and it's 
> surprisingly important from a usability standpoint.
> 

Perhaps we shouldn't rely on stderr, and instead have a backchannel as part of the 
protocol itself.  After all, the protocol already does packetization, so all it needs is a 
reliable way to pick out the error/status packets; we could even combine that with a 
machine-readable code (like SMTP et al) that could get interpreted by the other side as 
needed.

	-hpa

Re: 2.6.17-rc6-mm2

[bert hubert]

On Thu, Jun 15, 2006 at 11:39:35PM -0700, Linus Torvalds wrote:

> Except they only work over ssh, where we have a separate channel (for 
> stderr), and with the native git protocol all that nice status work just 
> gets flushed to /dev/null :(

It won't help passing firewalls one bit, but you might consider using SCTP
with multiple datastreams for this - theoretically :-)

	Bert

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services

Re: 2.6.17-rc6-mm2

[Junio C Hamano]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Perhaps we shouldn't rely on stderr, and instead have a backchannel as
> part of the protocol itself.

Concurred.  This was one of the thing I was planning to do
anyway.

Re: 2.6.17-rc6-mm2

[Linus Torvalds]

On Fri, 16 Jun 2006, H. Peter Anvin wrote:
> 
> Perhaps we shouldn't rely on stderr, and instead have a backchannel as part of
> the protocol itself.

Absolutely. I'm just irritated at myself for not going that way in the 
first place, but when I originally wrote it, I had my eyes on other 
issues, and the nice status updates got added later..

		Linus

Re: 2.6.17-rc6-mm2

[Michal Ludvig]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

Linus Torvalds wrote:
> 
> On Fri, 16 Jun 2006, Goo GGooo wrote:
> 
>> I suggest adding SO_KEEPALIVE option on the git socket.
> 
> Actually, the really irritating thing is that we actually generate all 
> these nice status updates, which just makes pulling and cloning a lot more 
> comfortable, because you actually see what is going on, and what to 
> expect. 
> 
> Except they only work over ssh, where we have a separate channel (for 
> stderr), and with the native git protocol all that nice status work just 
> gets flushed to /dev/null :(

OpenBSD has CVS access to their repos over SSH even for anonymous users.
Could something similar be set up on git.kernel.org as well?

> And in your case, the usability downside actually turned into a real 
> accessibility bug.

Same issue here. Thanks for the hint. Attached is a patch against git
1.4.0 that solves it perfectly in my case.

Sysctl settings (for keepalive every 10 sec):
net.ipv4.tcp_keepalive_intvl=10
net.ipv4.tcp_keepalive_time=10

Michal

[-- Attachment #2: keepalive.diff --]
[-- Type: text/x-patch, Size: 1375 bytes --]

Set SO_KEEPALIVE option on native git:// sockets.

Signed-off-by: Michal Ludvig <michal@logix.cz>

Index: git-1.4.0/connect.c
===================================================================
--- git-1.4.0.orig/connect.c
+++ git-1.4.0/connect.c
@@ -331,7 +331,7 @@ static int git_tcp_connect_sock(char *ho
 	char *colon, *end;
 	char *port = STR(DEFAULT_GIT_PORT);
 	struct addrinfo hints, *ai0, *ai;
-	int gai;
+	int gai, option;
 
 	if (host[0] == '[') {
 		end = strchr(host + 1, ']');
@@ -363,6 +363,10 @@ static int git_tcp_connect_sock(char *ho
 				ai->ai_socktype, ai->ai_protocol);
 		if (sockfd < 0)
 			continue;
+
+		option = 1;
+		setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &option, sizeof(option));
+
 		if (connect(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
 			close(sockfd);
 			sockfd = -1;
@@ -392,7 +396,7 @@ static int git_tcp_connect_sock(char *ho
 	struct hostent *he;
 	struct sockaddr_in sa;
 	char **ap;
-	unsigned int nport;
+	unsigned int nport, option;
 
 	if (host[0] == '[') {
 		end = strchr(host + 1, ']');
@@ -433,6 +437,9 @@ static int git_tcp_connect_sock(char *ho
 		sa.sin_port = htons(nport);
 		memcpy(&sa.sin_addr, *ap, he->h_length);
 
+		option = 1;
+		setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &option, sizeof(option));
+
 		if (connect(sockfd, (struct sockaddr *)&sa, sizeof sa) < 0) {
 			close(sockfd);
 			sockfd = -1;

Re: 2.6.17-rc6-mm2

[Linus Torvalds]

On Tue, 20 Jun 2006, Michal Ludvig wrote:
> 
> OpenBSD has CVS access to their repos over SSH even for anonymous users.
> Could something similar be set up on git.kernel.org as well?

I suspect the kernel.org people would prefer not to. And I'm almost 
certain that others don't want to. It would really be much better if the 
git protocol itself just had a sideband channel. Oh, well.

			Linus