[PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Ramkumar Ramachandra]

Rewrite the description section to describe what exactly remote
helpers are and the need for them. Also mention the curl family of
remote helpers as an example.
---
 The previous version got corrupted and didn't apply cleanly for some reason.
 Additionally split it into two parts now.

 Documentation/git-remote-helpers.txt |   32 ++++++++++++++++++++++++++------
 1 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/Documentation/git-remote-helpers.txt
b/Documentation/git-remote-helpers.txt
index 1b5f61a..1304813 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -3,7 +3,7 @@ git-remote-helpers(1)

 NAME
 ----
-git-remote-helpers - Helper programs for interoperation with remote git
+git-remote-helpers - Helper programs to interact with remote repositories

 SYNOPSIS
 --------
@@ -12,11 +12,31 @@ SYNOPSIS
 DESCRIPTION
 -----------

-These programs are normally not used directly by end users, but are
-invoked by various git programs that interact with remote repositories
-when the repository they would operate on will be accessed using
-transport code not linked into the main git binary. Various particular
-helper programs will behave as documented here.
+Remote helper programs are normally not used directly by end users,
+but are invoked by git when it needs to interact with remote
+repositories. They implement a subset of the capabilities documented
+here, and conform to the "remote helper protocol". When git needs
+needs to interact with a repository served by a remote helper, it
+spawns the helper as an independent process and interacts with it over
+the specified protocol. Essentially, git sends commands to the helper
+over standard input, and receives the result written to standard
+output by the helper over a pipe. Also, since remote helpers often
+need to link to the libraries required to interact with the remote
+repository, they avoid linking to the main git programs to evade
+licensing issues.
+
+All the capabilities of remote helpers have to do with discovering and
+updating remote refs, transporting objects between local and remote,
+and updating the local object store. Using the 'fetch' capability,
+they can discover refs on the remote, transfer objects from the remote
+reachable via those refs to local, and update the local object
+store. Using the 'push' capability, they can transfer objects from
+local to remote, and update the corresponding refs as necessary.
+
+Git comes with a "curl" family of remote helpers, specifically
+'git-remote-http', 'git-remote-https', 'git-remote-ftp' and
+'git-remote-ftps'. They implement the capabilities 'fetch', 'option',
+and 'push'.

 COMMANDS
 --------
-- 
1.7.0.3

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Gabriel Filion]

Hello,

On 2010-03-28 14:03, Ramkumar Ramachandra wrote:
[...]
> diff --git a/Documentation/git-remote-helpers.txt
> b/Documentation/git-remote-helpers.txt
> index 1b5f61a..1304813 100644
> --- a/Documentation/git-remote-helpers.txt
> +++ b/Documentation/git-remote-helpers.txt
[...]
> @@ -12,11 +12,31 @@ SYNOPSIS
[...]
> +repositories. They implement a subset of the capabilities documented
> +here, and conform to the "remote helper protocol". When git needs
> +needs to interact with a repository served by a remote helper, it

Oops, "needs" is repeated twice here...

Apart from that, it seems nice and helps to better understand what they
should be used for.

It doesn't really concern this patch but If I'm not wrong, there still
is no documentation written on how exactly git invokes the remote
helpers. A good description was given previously by Ilari Liusvaara (so
I've added you, Ilari in the cc list). The description can be seen here:

http://lists.zerezo.com/git/msg712892.html

Should we add this information on this man page?

-- 
Gabriel Filion

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Ramkumar Ramachandra]

Hi,

> Oops, "needs" is repeated twice here...

Oops. Sorry about the silly mistake.
Junio: Could you please make this correction and apply the patch? I'm
writing another patch documenting invocation which depends on this
patch.

--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -15,7 +15,7 @@ DESCRIPTION
 Remote helper programs are normally not used directly by end users,
 but are invoked by git when it needs to interact with remote
 repositories. They implement a subset of the capabilities documented
-here, and conform to the "remote helper protocol". When git needs
+here, and conform to the "remote helper protocol". When git
 needs to interact with a repository served by a remote helper, it
 spawns the helper as an independent process and interacts with it over
 the specified protocol. Essentially, git sends commands to the helper
@@ -38,6 +38,51 @@ Git comes with a "curl" family of remote helpers,
specifically
 'git-remote-ftps'. They implement the capabilities 'fetch', 'option',
 and 'push'.

> http://lists.zerezo.com/git/msg712892.html
>
> Should we add this information on this man page?

Definitely. Thanks for pointing this out- I've already started
preparing a patch for this.

-- Ram

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Sverre Rabbelier]

Heya,

On Sun, Mar 28, 2010 at 19:17, Gabriel Filion <lelutin@gmail.com> wrote:
> Should we add this information on this man page?

Yes we should. Also, I have almost finished a 'testgit' helper, which
simply invokes git-fast-import and git-fast-export behind the scenes.
So it's really a clumsy way to communicate with another git repository
but using the remote-helper import/export API. I've also written tests
for it, so it should be pretty clear from the tests and the simple
helper how to implement your own.

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Daniel Barkalow]

On Sun, 28 Mar 2010, Gabriel Filion wrote:

> Hello,
> 
> On 2010-03-28 14:03, Ramkumar Ramachandra wrote:
> [...]
> > diff --git a/Documentation/git-remote-helpers.txt
> > b/Documentation/git-remote-helpers.txt
> > index 1b5f61a..1304813 100644
> > --- a/Documentation/git-remote-helpers.txt
> > +++ b/Documentation/git-remote-helpers.txt
> [...]
> > @@ -12,11 +12,31 @@ SYNOPSIS
> [...]
> > +repositories. They implement a subset of the capabilities documented
> > +here, and conform to the "remote helper protocol". When git needs
> > +needs to interact with a repository served by a remote helper, it
> 
> Oops, "needs" is repeated twice here...
> 
> Apart from that, it seems nice and helps to better understand what they
> should be used for.
> 
> It doesn't really concern this patch but If I'm not wrong, there still
> is no documentation written on how exactly git invokes the remote
> helpers. A good description was given previously by Ilari Liusvaara (so
> I've added you, Ilari in the cc list). The description can be seen here:
> 
> http://lists.zerezo.com/git/msg712892.html
> 
> Should we add this information on this man page?

That information ought to be in the documentation, but possibly not on 
this man page in particular. I think it would be better to document that 
part in the documentation of the code and programs that call the helper, 
not in the helper documentation. In general, I'd like the helpers to make 
as few assumptions about their caller as possible, and similarly keep the 
information about the caller out of their documentation.

Also, I think it would be more helpful in general to get that information 
into documentation of the transport and remote code and configuration of 
these things, because users are going to come at this from that side ("I 
want to access an SVN repository, how does that work?") rather than from 
the helper documentation side ("I want to exercise git-remote-svn's export 
support, just to run that code").

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Ramkumar Ramachandra]

Hi,

> That information ought to be in the documentation, but possibly not on
> this man page in particular. I think it would be better to document that
> part in the documentation of the code and programs that call the helper,
> not in the helper documentation.

I agree that the callers need to document the subset of the
invocations they make to remote helpers. I think we can defer this
until we have a real remote helper in `git.git` that actually
interfaces with a foreign versioning system.

I've thought about documenting the full set of invocations in the code
for the developer, but there's a problem. Here's an excerpt from
remote-curl.c, showing how it parses its command line arguments:

	remote = remote_get(argv[1]);

	if (argc > 2) {
		url = argv[2];
	} else {
		url = remote->url[0];
	}

Unfortunately, I don't see where else this documentation can fit in:
if it were to go into a specific remote helper's code, then it'll have
to be duplicated for all the remote helpers, since all of them parse
options similarly. It certainly cannot go into remote.c or
transport-helper.c, because they have little/ nothing to do with the
actual argument parsing.

I could try modifying the documentation I've written to serve more to
specify "how remote helpers are invoked" and less about "how callers
invoke remote helpers", and try to fit it in this manpage. It's more
of a developer manpage and less of an end-user manpage as it is. Or we
could create another page about remote helpers intended to be read
exclusively by developers. What are your thoughts on this?

-- Ram

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Jonathan Nieder]

Ramkumar Ramachandra wrote:

>> I think it would be better to document that
>> part in the documentation of the code and programs that call the helper,
>> not in the helper documentation.
[...]
> Unfortunately, I don't see where else this documentation can fit in:
> if it were to go into a specific remote helper's code, then it'll have
> to be duplicated for all the remote helpers, since all of them parse
> options similarly.

One possibility: new manpage, called giturl(7) or something, with:

 - the information from your patch, reformatted a little to be from
   the caller’s perspective;

 - the information currently in the GIT URLS and REMOTES sections 
   of git-pull(1) and and other urls-remotes.txt includers;

 - pointers to appropriate high-level and low-level documentation
   for more information.

This would at least avoid some duplication of text in explaining how
the [remote "<name>"] setups work.

> It certainly cannot go into remote.c or
> transport-helper.c, because they have little/ nothing to do with the
> actual argument parsing.

One possibility would be to put it in Documentation/technical/transport.txt
or some similarly named new file.  Later that file could expand to an
overview of the transport layer, which would be nice to have.

Files in Documentation/technical do not get installed as manpages,
which would make this less convenient when writing a new helper
without a full documentation tree available.

More importantly, the “how to configure access to a foreign
repository” aspect of what you are writing is really more pertinent to
users than remote helper developers.  Remote helper developers only
need to know “first argument is a remote nickname or some nonsense
with a colon; second argument is a transport-native address
identifying the remote repository; second argument can be omitted only
if a remote nickname was used”.

HTH,
Jonathan

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Daniel Barkalow]

On Mon, 29 Mar 2010, Ramkumar Ramachandra wrote:

> Hi,
> 
> > That information ought to be in the documentation, but possibly not on
> > this man page in particular. I think it would be better to document that
> > part in the documentation of the code and programs that call the helper,
> > not in the helper documentation.
> 
> I agree that the callers need to document the subset of the
> invocations they make to remote helpers. I think we can defer this
> until we have a real remote helper in `git.git` that actually
> interfaces with a foreign versioning system.
> 
> I've thought about documenting the full set of invocations in the code
> for the developer, but there's a problem. Here's an excerpt from
> remote-curl.c, showing how it parses its command line arguments:
> 
> 	remote = remote_get(argv[1]);
> 
> 	if (argc > 2) {
> 		url = argv[2];
> 	} else {
> 		url = remote->url[0];
> 	}
> 
> Unfortunately, I don't see where else this documentation can fit in:
> if it were to go into a specific remote helper's code, then it'll have
> to be duplicated for all the remote helpers, since all of them parse
> options similarly. It certainly cannot go into remote.c or
> transport-helper.c, because they have little/ nothing to do with the
> actual argument parsing.
> 
> I could try modifying the documentation I've written to serve more to
> specify "how remote helpers are invoked" and less about "how callers
> invoke remote helpers", and try to fit it in this manpage. It's more
> of a developer manpage and less of an end-user manpage as it is. Or we
> could create another page about remote helpers intended to be read
> exclusively by developers. What are your thoughts on this?

I think getting information on what the helper is supposed to do with its 
command-line arguments into this man page would be good, and the 
appropriate focus for the man page.

The current answer is this:
 argv[1] is the name of a remote, which may be a nickname or some more 
  direct name; in any case, remote_get() will produce the available data 
  for it.
 argv[2] is the URL, if there is one. Since a remote could have more than 
  one URL, the helper gets the URL it should be handling. Some systems 
  don't use URLs, and these will just look at the remote or something like 
  that.

The rest of what you wrote is true, but it's really more information on 
how the attributes of remotes are determined than what the helper should 
be worrying about.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Ramkumar Ramachandra]

On Mon, Mar 29, 2010 at 9:09 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
> The current answer is this:
>  argv[1] is the name of a remote, which may be a nickname or some more
>  direct name; in any case, remote_get() will produce the available data
>  for it.
>  argv[2] is the URL, if there is one. Since a remote could have more than
>  one URL, the helper gets the URL it should be handling. Some systems
>  don't use URLs, and these will just look at the remote or something like
>  that.

Got it. I'll prepare a patch in the evening.

-- Ram

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Tay Ray Chuan]

Hi,

regarding...

On Mon, Mar 29, 2010 at 2:03 AM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:
> [snip] Also mention the curl family of
> remote helpers as an example.

and...

> +Git comes with a "curl" family of remote helpers, specifically
> +'git-remote-http', 'git-remote-https', 'git-remote-ftp' and
> +'git-remote-ftps'. They implement the capabilities 'fetch', 'option',
> +and 'push'.

I think "family" is confusing. They're all symlinks of
git-remote-curl, if I remember correctly.

But then, I suspect you'll be rewriting this portion, since Ilari's
notes has a thing or two why helper names have the protocol in them.

-- 
Cheers,
Ray Chuan

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Ramkumar Ramachandra]

Hi,

> I think "family" is confusing. They're all symlinks of
> git-remote-curl, if I remember correctly.

Actually, there's no Makefile target called git-remote-curl. See
Makefile:1122 to Makefile:1126. They're copies in my build.

	REMOTE_CURL_PRIMARY = git-remote-http$X
	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
	PROGRAM_OBJS += http-fetch.o
	PROGRAMS += $(REMOTE_CURL_NAMES)

> But then, I suspect you'll be rewriting this portion, since Ilari's
> notes has a thing or two why helper names have the protocol in them.

Right. Thanks for pointing this out.

-- Ram

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Ramkumar Ramachandra]

Hi,

On Sun, Mar 28, 2010 at 11:33 PM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:
> Rewrite the description section to describe what exactly remote
> helpers are and the need for them. Also mention the curl family of
> remote helpers as an example.

In accordance with Junio's 4th April "What's Cooking" report, could
some remote helper experts kindly Ack this patch?

Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>

Thanks!
-- Ram

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Junio C Hamano]

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> On Sun, Mar 28, 2010 at 11:33 PM, Ramkumar Ramachandra
> <artagnon@gmail.com> wrote:
>> Rewrite the description section to describe what exactly remote
>> helpers are and the need for them. Also mention the curl family of
>> remote helpers as an example.
>
> In accordance with Junio's 4th April "What's Cooking" report, could
> some remote helper experts kindly Ack this patch?
>
> Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>

I'll copy and paste this, but please make it a habit to sign off your
patch when committing if you plan to be a regular contributor to the
project.

I've re-read all the discussions that led to v7 patch and then re-read
the patch [1/2].  I'll queue it with a minor tweak.

Thanks.

Re: [PATCH v7 1/2] Documentation/remote-helpers: Rewrite description

[Ramkumar Ramachandra]

> I'll copy and paste this, but please make it a habit to sign off your
> patch when committing if you plan to be a regular contributor to the
> project.

Sorry about that. I'll auto sign-off and pay extra attention next time.

-- Ram