Remembering which http remotes require authentication

Remembering which http remotes require authentication

[M Hickford]

Most http remotes require authentication to push. Private repositories
also require authentication to fetch.

If I understand correctly, http_request_reauth [1] first makes an
unauthenticated http request. If this fails with error 401
Unauthorized, then Git retries with authentication. This avoids
potentially expensive `credential fill` unless necessary. (`credential
fill` may be fast, slow or even require user input)

Could Git remember which remotes require authentication for which
operations? On high-latency networks such as 3G or satellite
connections [2], skipping a doomed unauthenticated request could
potentially halve the time taken for some commands.

A complication: if a repository changed from private to public, Git
would continue to authenticate when no longer necessary. A solution
could be to remember to skip the unauthenticated request for a limited
amount of time, or a maximum number of requests, before rechecking.

[1] https://github.com/git/git/blob/fe86abd7511a9a6862d5706c6fa1d9b57a63ba09/http.c#L2123-L2169
[2] https://hpbn.co/primer-on-latency-and-bandwidth/

Re: Remembering which http remotes require authentication

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]

On 2023-06-12 at 06:47:19, M Hickford wrote:
> Most http remotes require authentication to push. Private repositories
> also require authentication to fetch.
> 
> If I understand correctly, http_request_reauth [1] first makes an
> unauthenticated http request. If this fails with error 401
> Unauthorized, then Git retries with authentication. This avoids
> potentially expensive `credential fill` unless necessary. (`credential
> fill` may be fast, slow or even require user input)
> 
> Could Git remember which remotes require authentication for which
> operations? On high-latency networks such as 3G or satellite
> connections [2], skipping a doomed unauthenticated request could
> potentially halve the time taken for some commands.
> 
> A complication: if a repository changed from private to public, Git
> would continue to authenticate when no longer necessary. A solution
> could be to remember to skip the unauthenticated request for a limited
> amount of time, or a maximum number of requests, before rechecking.

What I would rather see instead is a way to indicate to Git that it
should always authenticate via a config option (probably something like
`http.*.alwaysAuth`).  I think this would work for your use case, but it
would also work for things like giving authenticated users higher rate
limits for public repositories.

I intend to add this feature at some point relatively soon, but I need
to sneak in a feature first to specify the authentication scheme as part
of the credential protocol, since we won't have it as part of the HTTP
401 response.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Remembering which http remotes require authentication

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

On 2023-06-21 at 06:30:33, M Hickford wrote:
> On Mon, 12 Jun 2023 at 22:22, brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> >
> > What I would rather see instead is a way to indicate to Git that it
> > should always authenticate via a config option (probably something like
> > `http.*.alwaysAuth`).  I think this would work for your use case, but it
> > would also work for things like giving authenticated users higher rate
> > limits for public repositories.
> 
> Interesting. Do you know any such hosts that rate limit unauthenticated
> users? Do their users prefer SSH which always authenticates?

At GitHub, we do this with the API, and because Git LFS uses Git's
credentials and is technically implemented as part of the API, it
applies there as well.  I don't know that we have any intention of doing
it for Git operations themselves, but the reason we typically prefer
authentication is that if there's a problem, it's much easier and a
better experience to contact user XYZ and ask them to be more gentle
than to block their IP.

This is particularly useful for internal users at GitHub (like the
automatic dependency updating service, Dependabot), where we'd like to
account for who's performing lots of operations in terms of rate
limiting, and the alternative is that people use http.extraHeader or a
token in the URL, neither of which we want to encourage.  I'd much
rather rate-limit an internal service which is using too many resources
than impact unrelated users, but if they're all anonymous, that's hard.

> > I intend to add this feature at some point relatively soon, but I need
> > to sneak in a feature first to specify the authentication scheme as part
> > of the credential protocol, since we won't have it as part of the HTTP
> > 401 response.
> 
> Would it suffice to ask the user to specify the authentication scheme
> in the configuration? eg. `http.alwaysAuth=basic` or
> `http.alwaysAuth=digest`

That will probably work for most cases, but we may need to allow either
a scheme or a full WWW-Authenticate header, since in some cases there
may be parameters that are needed.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Remembering which http remotes require authentication

[Junio C Hamano]

M Hickford <mirth.hickford@gmail.com> writes:

> Could Git remember which remotes require authentication for which
> operations?

Interesting idea.

Credential helpers may know which URL needed authentication before,
but that is not per operation.  I wonder if it is sufficient as the
zeroth approximation to always try authenticated access if helpers
have any auth material to the URL regardless of the operation?

There needs a way to force unautheticated access if we were to do
anything of that sort, of course.

Re: Remembering which http remotes require authentication

[M Hickford]

On Mon, 12 Jun 2023 at 23:12, Junio C Hamano <gitster@pobox.com> wrote:
>
> M Hickford <mirth.hickford@gmail.com> writes:
>
> > Could Git remember which remotes require authentication for which
> > operations?
>
> Interesting idea.
>
> Credential helpers may know which URL needed authentication before,
> but that is not per operation.  I wonder if it is sufficient as the
> zeroth approximation to always try authenticated access if helpers
> have any auth material to the URL regardless of the operation?

The simplicity appeals. A challenge is that `helper get` may be slow
or even block requiring user interaction (to unlock a keyring or
complete OAuth). A solution could be to introduce a new action
`get-existing` or `fast-get` to the helper protocol. Storage helpers
would handle this similarly to `get` (unless a keyring is locked).
Credential-generating helpers would refrain from generating fresh
credentials.

However as Brian pointed out, Git doesn't know which authentication
scheme to use until it sees the WWW-Authenticate header in the
response from the initial unauthenticated request, so I think
config or state would be necessary.



>
> There needs a way to force unautheticated access if we were to do
> anything of that sort, of course.
>
>
>