credential-helpers + remote-helper, starting point?

credential-helpers + remote-helper, starting point?

[roucherj]

Hello,

I want to know if anyone can help me with git-credential-helpers we are 
trying to use git-credential-helpers in the git-mediawiki (implemented 
as a remote-helper).
We need to ask for the login/pass of the wiki and it would be nice if 
we can use credential-helpers to manage this credentials.

Anyone can send me a starting point, like a url with the documentation 
of git-credential-helpers?
To see how we can import credential-helpers for used with mediawiki.

Re: credential-helpers + remote-helper, starting point?

[Jeff King]

On Thu, May 24, 2012 at 05:14:35PM +0200, roucherj wrote:

> I want to know if anyone can help me with git-credential-helpers we
> are trying to use git-credential-helpers in the git-mediawiki
> (implemented as a remote-helper).
> We need to ask for the login/pass of the wiki and it would be nice if
> we can use credential-helpers to manage this credentials.

Yeah, I think it makes sense to use the credential-helpers.

> Anyone can send me a starting point, like a url with the
> documentation of git-credential-helpers?

Try:

  https://github.com/git/git/blob/master/Documentation/technical/api-credentials.txt

But that is the C API, and I assume you are building on the existing
mediawiki helper that is written in perl. So I think what you really
want is a "git credential" command that will let scripts hook into the
credential API. Something like:

  $ git credential get https://example.com
  username=bob
  password=secret

  $ cat <<\EOF | git credential store https://example.com
  username=bob
  password=secret
  EOF

  $ cat <<\EOF | git credential erase https://example.com
  username=bob
  password=secret
  EOF

I had planned eventually to do something like this for git-svn, but
realized that it was more sane to just let svn library handle the
credential storage.

Do you guys want to try writing "git credential" as above? It might be a
fun side project, but I know you are also on a limited timeframe for
your project. I can work on it if you don't have time.

-Peff

Re: credential-helpers + remote-helper, starting point?

[Matthieu Moy]

Jeff King <peff@peff.net> writes:

>   https://github.com/git/git/blob/master/Documentation/technical/api-credentials.txt

I just re-read that document (I had a quick glance only before), and
started to understand. I think the document really lacks the "big
picture". I took time to understand whether the API was to call
credential helpers, or to write a new one. Actually, there are two
"API": the C one, and the specification of what may flow on the pipe
between the two processes.

Perhaps something like this should be added:

Subject: [PATCH] credential-helper documentation: show the big picture first

---
 Documentation/technical/api-credentials.txt |   47 +++++++++++++++++++++++++--
 1 file changed, 44 insertions(+), 3 deletions(-)

diff --git a/Documentation/technical/api-credentials.txt b/Documentation/technical/api-credentials.txt
index 21ca6a2..5a872c0 100644
--- a/Documentation/technical/api-credentials.txt
+++ b/Documentation/technical/api-credentials.txt
@@ -6,8 +6,40 @@ password credentials from the user (even though credentials in the wider
 world can take many forms, in this document the word "credential" always
 refers to a username and password pair).
 
+When a function in Git or one of its remote-helpers needs to obtain
+credentials (either by asking the user or by fetching from a store),
+it can call the functions in the C API. These functions will fork a
+new process, and communicate with it by passing command-line arguments
+and then communicating through a pipe (see 'Credential Helpers'
+below). The credential helper process will be in charge of actually
+prompting the user and/or storing and fetching the credentials.
+
+For example, the execution of a command connecting to an HTTP server
+and using the credential helper "cache" will have the following
+structure:
+
+------------
++-----+ -----> +-----------------+
+| git |  pipe  | git remote-http | --- to HTTP server --->
++-----+ <----- +-----------------+
+                    ^      |
+                    | pipe |
+                    |      v
+             +----------------------+
+             | git credential-cache |
+             +----------------------+
+------------
+
+git remote-http will take care of contacting the HTTP server, do the
+actual authentication and see if it's accepted by the server. The
+credential helper will deal with the credential store (which can be
+done by contacting a keyring daemon) and the prompting if needed.
+
+C API
+-----
+
 Data Structures
----------------
+~~~~~~~~~~~~~~~
 
 `struct credential`::
 
@@ -28,7 +60,7 @@ This struct should always be initialized with `CREDENTIAL_INIT` or
 
 
 Functions
----------
+~~~~~~~~~
 
 `credential_init`::
 
@@ -72,7 +104,7 @@ Functions
 	Parse a URL into broken-down credential fields.
 
 Example
--------
+~~~~~~~
 
 The example below shows how the functions of the credential API could be
 used to login to a fictitious "foo" service on a remote host:
@@ -130,6 +162,9 @@ int foo_login(struct foo_connection *f)
 Credential Helpers
 ------------------
 
+Choosing the credential helper command
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 Credential helpers are programs executed by git to fetch or save
 credentials from and to long-term storage (where "long-term" is simply
 longer than a single git process; e.g., credentials may be stored
@@ -176,6 +211,9 @@ users by naming their program "git-credential-$NAME", and putting it in
 the $PATH or $GIT_EXEC_PATH during installation, which will allow a user
 to enable it with `git config credential.helper $NAME`.
 
+Credential helper command-line arguments
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 When a helper is executed, it will have one "operation" argument
 appended to its command line, which is one of:
 
@@ -191,6 +229,9 @@ appended to its command line, which is one of:
 
 	Remove a matching credential, if any, from the helper's storage.
 
+Credential helper protocol
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 The details of the credential will be provided on the helper's stdin
 stream. The credential is split into a set of named attributes.
 Attributes are provided to the helper, one per line. Each attribute is
-- 
1.7.10.363.g7fcd3d.dirty

Also, shouldn't the documentation about choosing the command name be
moved to git-config.txt, to document credential.helper? It really seems
to be a user documentation, not a technical one meant for Git
developers.

> But that is the C API, and I assume you are building on the existing
> mediawiki helper that is written in perl.

Right.

> So I think what you really want is a "git credential" command that
> will let scripts hook into the credential API. Something like:
>
>   $ git credential get https://example.com
>   username=bob
>   password=secret
> [...]

This is almost already done by test-credential.c indeed. But that's
probably the simplest way to expose the C API to a perl program.

> Do you guys want to try writing "git credential" as above? It might be a
> fun side project, but I know you are also on a limited timeframe for
> your project. I can work on it if you don't have time.

I leave it up to the students.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: credential-helpers + remote-helper, starting point?

[Jeff King]

On Thu, May 24, 2012 at 09:20:08PM +0200, Matthieu Moy wrote:

> Jeff King <peff@peff.net> writes:
> 
> >   https://github.com/git/git/blob/master/Documentation/technical/api-credentials.txt
> 
> I just re-read that document (I had a quick glance only before), and
> started to understand. I think the document really lacks the "big
> picture". I took time to understand whether the API was to call
> credential helpers, or to write a new one. Actually, there are two
> "API": the C one, and the specification of what may flow on the pipe
> between the two processes.

Right. There are really two audiences for this document: people who are
writing git code that wants to use credentials, and people who want to
write a helper. And it probably makes sense to say that up front, and
point people to the right section.

> diff --git a/Documentation/technical/api-credentials.txt b/Documentation/technical/api-credentials.txt
> index 21ca6a2..5a872c0 100644
> --- a/Documentation/technical/api-credentials.txt
> +++ b/Documentation/technical/api-credentials.txt
> @@ -6,8 +6,40 @@ password credentials from the user (even though credentials in the wider
>  world can take many forms, in this document the word "credential" always
>  refers to a username and password pair).
>  
> +When a function in Git or one of its remote-helpers needs to obtain
> +credentials (either by asking the user or by fetching from a store),
> +it can call the functions in the C API. These functions will fork a
> +new process, and communicate with it by passing command-line arguments
> +and then communicating through a pipe (see 'Credential Helpers'
> +below).

We might or might not fork a new process. It depends on which helpers
are defined.

> The credential helper process will be in charge of actually
> +prompting the user and/or storing and fetching the credentials.

That's not exactly true; the C credential code will actually prompt the
user if no helper provides it (in early versions, the helpers were
responsible for prompting, but that is no longer the case).

> +For example, the execution of a command connecting to an HTTP server
> +and using the credential helper "cache" will have the following
> +structure:
> +
> +------------
> ++-----+ -----> +-----------------+
> +| git |  pipe  | git remote-http | --- to HTTP server --->
> ++-----+ <----- +-----------------+
> +                    ^      |
> +                    | pipe |
> +                    |      v
> +             +----------------------+
> +             | git credential-cache |
> +             +----------------------+
> +------------
> +
> +git remote-http will take care of contacting the HTTP server, do the
> +actual authentication and see if it's accepted by the server. The
> +credential helper will deal with the credential store (which can be
> +done by contacting a keyring daemon) and the prompting if needed.

I feel like adding remote-helpers into the mix just makes the situation
more complex (and necessitates the diagram). I also don't want to get
too much into "how it works inside" and would rather stay at the level
of "here's how you use it".

What do you think of the patch below? It tries to route the reader to
the most useful spot, and it would naturally extend to mentioning a "git
credential" command when one exists (which would be separately
documented in git-credential.txt).

---
diff --git a/Documentation/technical/api-credentials.txt b/Documentation/technical/api-credentials.txt
index 21ca6a2..f6fa203 100644
--- a/Documentation/technical/api-credentials.txt
+++ b/Documentation/technical/api-credentials.txt
@@ -6,8 +6,27 @@ password credentials from the user (even though credentials in the wider
 world can take many forms, in this document the word "credential" always
 refers to a username and password pair).
 
+This document describes two interfaces: the C API that the credential
+subsystem provides to the rest of git, and the protocol that git uses to
+communicate with system-specific "credential helpers". If you are
+writing git code that wants to look up or prompt for credentials, see
+the section "C API" below. If you want to write your own helper, see
+the section on "Credential Helpers" below.
+
+
+C API
+-----
+
+The credential C API is meant to be called by git code which needs to
+acquire or store a credential. It is centered around an object
+representing a single credential and provides three basic operations:
+fill (acquire credentials by calling helpers and/or prompting the user),
+approve (mark a credential as successfully used so that it can be stored
+for later use), and reject (mark a credential as unsuccessful so that it
+can be erased from any persistent storage).
+
 Data Structures
----------------
+~~~~~~~~~~~~~~~
 
 `struct credential`::
 
@@ -28,7 +47,7 @@ This struct should always be initialized with `CREDENTIAL_INIT` or
 
 
 Functions
----------
+~~~~~~~~~
 
 `credential_init`::
 
@@ -72,7 +91,7 @@ Functions
 	Parse a URL into broken-down credential fields.
 
 Example
--------
+~~~~~~~
 
 The example below shows how the functions of the credential API could be
 used to login to a fictitious "foo" service on a remote host:

Re: credential-helpers + remote-helper, starting point?

[Matthieu Moy]

Jeff King <peff@peff.net> writes:

>> +When a function in Git or one of its remote-helpers needs to obtain
>> +credentials (either by asking the user or by fetching from a store),
>> +it can call the functions in the C API. These functions will fork a
>> +new process, and communicate with it by passing command-line arguments
>> +and then communicating through a pipe (see 'Credential Helpers'
>> +below).
>
> We might or might not fork a new process. It depends on which helpers
> are defined.

OK, so s/will fork/may fork/ may be more appropriate.

>> +For example, the execution of a command connecting to an HTTP server
>> +and using the credential helper "cache" will have the following
>> +structure:
>> +
>> +------------
>> ++-----+ -----> +-----------------+
>> +| git |  pipe  | git remote-http | --- to HTTP server --->
>> ++-----+ <----- +-----------------+
>> +                    ^      |
>> +                    | pipe |
>> +                    |      v
>> +             +----------------------+
>> +             | git credential-cache |
>> +             +----------------------+
>> +------------
>> +
>> +git remote-http will take care of contacting the HTTP server, do the
>> +actual authentication and see if it's accepted by the server. The
>> +credential helper will deal with the credential store (which can be
>> +done by contacting a keyring daemon) and the prompting if needed.
>
> I feel like adding remote-helpers into the mix just makes the situation
> more complex (and necessitates the diagram).

The helper is not strictly needed, but that seems to be a common
situation, so I thought describing the common case made sense.

I think it's important to show the difference between the actual
authentication process (e.g. HTTP) and the credential helper. I took
time to understand the example later:

	credential_fill(&c);
        /* [...] */
	status = send_foo_login(f, c.username, c.password);
	switch (status) {

then I understood that send_foo_login was the one doing the actual
authentication, and that further calls to the credential API was needed
to say how it went. Making explicit the fact that the Git code talks
both the helper and the server on which we are authenticating would have
helped me to understand at least.

I think mentionning the fact that a credential helper may connect to a
keyring daemon was important also. I first read some discussions about
credential helpers thinking that we would have to re-implement the
storing mechanism, and really understood what it was all about when I
saw patches for KDE support for example.

Other than that, I do like your wording. If it was just me, I'd take
your introduction, but keep my example + diagram.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: credential-helpers + remote-helper, starting point?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, May 24, 2012 at 09:20:08PM +0200, Matthieu Moy wrote:
>
>> +------------
>> ++-----+ -----> +-----------------+
>> +| git |  pipe  | git remote-http | --- to HTTP server --->
>> ++-----+ <----- +-----------------+
>> +                    ^      |
>> +                    | pipe |
>> +                    |      v
>> +             +----------------------+
>> +             | git credential-cache |
>> +             +----------------------+
>> +------------
>> +
>> +git remote-http will take care of contacting the HTTP server, do the
>> +actual authentication and see if it's accepted by the server. The
>> +credential helper will deal with the credential store (which can be
>> +done by contacting a keyring daemon) and the prompting if needed.
>
> I feel like adding remote-helpers into the mix just makes the situation
> more complex (and necessitates the diagram).

I would also prefer to see an example _without_ the remote helper, but at
the same time it still helps to have an illustration.

> +This document describes two interfaces: the C API that the credential
> +subsystem provides to the rest of git, and the protocol that git uses to
> +communicate with system-specific "credential helpers". If you are
> +writing git code that wants to look up or prompt for credentials, see

Don't you "prompt" yourself?  The above sounds as if you are delegating
both looking up and prompting to the helper.

> +the section "C API" below. If you want to write your own helper, see
> +the section on "Credential Helpers" below.

Nice.

Re: credential-helpers + remote-helper, starting point?

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Jeff King <peff@peff.net> writes:
>
>> On Thu, May 24, 2012 at 09:20:08PM +0200, Matthieu Moy wrote:
>>
>> +This document describes two interfaces: the C API that the credential
>> +subsystem provides to the rest of git, and the protocol that git uses to
>> +communicate with system-specific "credential helpers". If you are
>> +writing git code that wants to look up or prompt for credentials, see
>
> Don't you "prompt" yourself?  The above sounds as if you are delegating
> both looking up and prompting to the helper.

It seems I misunderstood how it worked, indeed. I thought the prompting
could also be done by the helper (it seems sensible if you're going to
use your desktop's keyring to also allow the desktop's GUI for prompting
for example).

Looking closely at the code, credential_fill either delegates fetching
data to the remote helper, or does the prompting. IOW, the prompting is
done behind the C API anyway, but most likely from credential.c than
from the external helper. Is that understanding correct?

Thanks,

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: credential-helpers + remote-helper, starting point?

[roucherj]

On Thu, 24 May 2012 14:21:10 -0400, Jeff King wrote:
> On Thu, May 24, 2012 at 05:14:35PM +0200, roucherj wrote:
>
>> I want to know if anyone can help me with git-credential-helpers we
>> are trying to use git-credential-helpers in the git-mediawiki
>> (implemented as a remote-helper).
>> We need to ask for the login/pass of the wiki and it would be nice 
>> if
>> we can use credential-helpers to manage this credentials.
>
> Yeah, I think it makes sense to use the credential-helpers.
>
>> Anyone can send me a starting point, like a url with the
>> documentation of git-credential-helpers?
>
> Try:
>
>
> 
> https://github.com/git/git/blob/master/Documentation/technical/api-credentials.txt
>
> But that is the C API, and I assume you are building on the existing
> mediawiki helper that is written in perl. So I think what you really
> want is a "git credential" command that will let scripts hook into 
> the
> credential API. Something like:
>
>   $ git credential get https://example.com
>   username=bob
>   password=secret
>
>   $ cat <<\EOF | git credential store https://example.com
>   username=bob
>   password=secret
>   EOF
>
>   $ cat <<\EOF | git credential erase https://example.com
>   username=bob
>   password=secret
>   EOF
>
> I had planned eventually to do something like this for git-svn, but
> realized that it was more sane to just let svn library handle the
> credential storage.
>
> Do you guys want to try writing "git credential" as above? It might 
> be a
> fun side project, but I know you are also on a limited timeframe for
> your project. I can work on it if you don't have time.
>
> -Peff

Jeff thanks you for helping me with use credentials-helpers with 
git-remote
I will try to code your proposed solution.
making something like that:

+-----+ -----> +----------------------+
| git |  pipe  | git-remote-mediawiki |
+-----+ <----- +----------------------+
                      /\      ||
                      || pipe ||
                      ||      \/
                  +----------------+ ----->  O
                  | git-credential | <----- -|-
                  +----------------+        / \
                      /\      ||            User
                      || pipe ||
                      ||      \/
                  +-------------------------+
                  | git-credentials-helpers |
                  +-------------------------+

Re: credential-helpers + remote-helper, starting point?

[Jeff King]

On Fri, May 25, 2012 at 03:28:39PM +0200, roucherj wrote:

> Jeff thanks you for helping me with use credentials-helpers with
> git-remote
> I will try to code your proposed solution.
> making something like that:
> 
> +-----+ -----> +----------------------+
> | git |  pipe  | git-remote-mediawiki |
> +-----+ <----- +----------------------+
>                      /\      ||
>                      || pipe ||
>                      ||      \/
>                  +----------------+ ----->  O
>                  | git-credential | <----- -|-
>                  +----------------+        / \
>                      /\      ||            User
>                      || pipe ||
>                      ||      \/
>                  +-------------------------+
>                  | git-credentials-helpers |
>                  +-------------------------+

Exactly. Let me know if you run into any problems.

-Peff