[RFC] Deal with HTTP 401 by requesting credentials.

[RFC] Deal with HTTP 401 by requesting credentials.

[Kevin Stange]

Request credentials from the user if none are already defined when a
HTTP 401 is received on a restricted repository.  Then, resubmit the
request and return the final result.

This allows all webdav transactions to obtain credentials without having
to individually handle the case in each request.  Having push working
with HTTP auth is needed for a use case I have where storing the
credentials in .netrc or using SSH keys is inappropriate.

Apologies for anything wrong I might have done here.  I'm not used to
procedures for this sort of patch submission, or terribly familiar with
the code base.  I'm seeking advice on whether this approach is sane or
completely crazy, and I'm willing to adjust it to make it suitable for
inclusion.

Signed-off-by: Kevin <kevin@steadfast.net>
---
 http.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/http.c b/http.c
index 5cb87f1..e1c9e65 100644
--- a/http.c
+++ b/http.c
@@ -668,6 +668,21 @@ void finish_active_slot(struct active_request_slot
*slot)
 	closedown_active_slot(slot);
 	curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CODE, &slot->http_code);

+	if (slot->http_code == 401) {
+		if(!http_auth.username && !http_auth.password) {
+			active_requests++;
+			credential_fill(&http_auth);
+			init_curl_http_auth(slot->curl);
+			(*slot->finished) = 1;
+			if (start_active_slot(slot)) {
+				run_active_slot(slot);
+				return;
+			}
+		} else {
+			fprintf(stderr,"Authentication Failed!\n");
+		}
+	}
+
 	if (slot->finished != NULL)
 		(*slot->finished) = 1;

-- 
1.7.11.rc0.55.gb2478aa

-- 
Kevin Stange
Chief Technology Officer
Steadfast Networks
http://steadfast.net
Phone: 312-602-2689 ext. 203 | Fax: 312-602-2688 | Cell: 312-320-5867

Re: [RFC] Deal with HTTP 401 by requesting credentials.

[Kevin Stange]

On 05/31/2012 05:24 PM, Kevin Stange wrote:
> Apologies for anything wrong I might have done here.  I'm not used to
> procedures for this sort of patch submission, or terribly familiar with
> the code base.  I'm seeking advice on whether this approach is sane or
> completely crazy, and I'm willing to adjust it to make it suitable for
> inclusion.

Of course, I failed to get my email client to send the patch unwrapped.  Only one line appears to have been affected, but I don't think this is a final patch.  I'll be sure to make a proper submission for any future patches I send.  Sorry!

-- 
Kevin Stange
Chief Technology Officer
Steadfast Networks
http://steadfast.net
Phone: 312-602-2689 ext. 203 | Fax: 312-602-2688 | Cell: 312-320-5867

Re: [RFC] Deal with HTTP 401 by requesting credentials.

[Jeff King]

On Thu, May 31, 2012 at 05:24:55PM -0500, Kevin Stange wrote:

> Request credentials from the user if none are already defined when a
> HTTP 401 is received on a restricted repository.  Then, resubmit the
> request and return the final result.
> 
> This allows all webdav transactions to obtain credentials without having
> to individually handle the case in each request.  Having push working
> with HTTP auth is needed for a use case I have where storing the
> credentials in .netrc or using SSH keys is inappropriate.

We already do this at a higher level in http_request, which in turns
calls into finish_active_slot. So if we were going to go this route,
wouldn't we also want to remove the 401 handling in http_request?

The dumb-http push code is the only thing that does not go through
http_request these days. So another option would be to refactor it to go
through that central point. I took a brief look at this when I was
updating the credential code a few months ago, but didn't consider it a
priority, as most people should be using smart http these days. Is there
a reason you can't use smart-http? It's significantly more efficient.

You also don't necessarily need to handle 401 in every code path of
http-push; once we see the credential once, we will use it everywhere,
so you really only need to handle it on the initial request (assuming
that all requests will have the same authorization requirements).

> Apologies for anything wrong I might have done here.  I'm not used to
> procedures for this sort of patch submission, or terribly familiar with
> the code base.  I'm seeking advice on whether this approach is sane or
> completely crazy, and I'm willing to adjust it to make it suitable for
> inclusion.
> 
> Signed-off-by: Kevin <kevin@steadfast.net>
> ---

Cover letter material (i.e., anything that would not go into the commit
message of the final commit) should go below the "---".

> diff --git a/http.c b/http.c
> index 5cb87f1..e1c9e65 100644
> --- a/http.c
> +++ b/http.c
> @@ -668,6 +668,21 @@ void finish_active_slot(struct active_request_slot
> *slot)
>  	closedown_active_slot(slot);
>  	curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CODE, &slot->http_code);
> 
> +	if (slot->http_code == 401) {
> +		if(!http_auth.username && !http_auth.password) {
> +			active_requests++;
> +			credential_fill(&http_auth);
> +			init_curl_http_auth(slot->curl);
> +			(*slot->finished) = 1;
> +			if (start_active_slot(slot)) {
> +				run_active_slot(slot);
> +				return;
> +			}
> +		} else {
> +			fprintf(stderr,"Authentication Failed!\n");
> +		}
> +	}

Is it safe to just run start_active_slot again without reinitializing
the request? The 401-handling code in http_request actually restarts a
new request. I don't immediately see any state that would need to be
reset; we might have written some data to the output file if curl gave
us any body data, but presumably it would not have done so for a 401.

In the "else" clause you add, I don't think there's any point in
printing an error. The 401 should get propagated back to the caller, who
will produce an error. However, you _should_ call credential_reject,
since you know that the credential you have doesn't work.

Similarly, you would want to call credential_accept after a successful
request, so that helpers can store it.

-Peff

Re: [RFC] Deal with HTTP 401 by requesting credentials.

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, May 31, 2012 at 05:24:55PM -0500, Kevin Stange wrote:
>
>> Request credentials from the user if none are already defined when a
>> HTTP 401 is received on a restricted repository.  Then, resubmit the
>> request and return the final result.
>> 
>> This allows all webdav transactions to obtain credentials without having
>> to individually handle the case in each request.  Having push working
>> with HTTP auth is needed for a use case I have where storing the
>> credentials in .netrc or using SSH keys is inappropriate.
>
> We already do this at a higher level in http_request, which in turns
> calls into finish_active_slot. So if we were going to go this route,
> wouldn't we also want to remove the 401 handling in http_request?

Wouldn't the higher levels know a lot more about the context this
request was made, though?  What problem does the patch try to solve?
Some higher level callers missing the "if we got 401, then reset and
retry" logic?  Wouldn't it be saner to fix the breakage there?

> The dumb-http push code is the only thing that does not go through
> http_request these days. So another option would be to refactor it to go
> through that central point.

It does sound like the right approach to take.

Thanks.

Re: [RFC] Deal with HTTP 401 by requesting credentials.

[Kevin Stange]

[-- Attachment #1: Type: text/plain, Size: 4511 bytes --]

On 06/01/2012 03:35 AM, Jeff King wrote:
> On Thu, May 31, 2012 at 05:24:55PM -0500, Kevin Stange wrote:
> 
>> Request credentials from the user if none are already defined when a
>> HTTP 401 is received on a restricted repository.  Then, resubmit the
>> request and return the final result.
>>
>> This allows all webdav transactions to obtain credentials without having
>> to individually handle the case in each request.  Having push working
>> with HTTP auth is needed for a use case I have where storing the
>> credentials in .netrc or using SSH keys is inappropriate.
> 
> We already do this at a higher level in http_request, which in turns
> calls into finish_active_slot. So if we were going to go this route,
> wouldn't we also want to remove the 401 handling in http_request?

I did see the work being done there, and considered removing it.  In fact, I
used it as a reference for working out what was going on in the authentication
process.  I decided not to do anything there until soliciting feedback and
deciding whether my approach was reasonable.

> The dumb-http push code is the only thing that does not go through
> http_request these days. So another option would be to refactor it to go
> through that central point. I took a brief look at this when I was
> updating the credential code a few months ago, but didn't consider it a
> priority, as most people should be using smart http these days. Is there
> a reason you can't use smart-http? It's significantly more efficient.

Smart HTTP didn't come up in any of my Google searches.  With that as an
option, I might just drop this work now.  I'd rather see incomplete methods
that aren't recommended go away than further facilitate their use, personally.

> You also don't necessarily need to handle 401 in every code path of
> http-push; once we see the credential once, we will use it everywhere,
> so you really only need to handle it on the initial request (assuming
> that all requests will have the same authorization requirements).

I made the change where I did because I wasn't sure if the push code was
avoiding using http_request intentionally, and wasn't sure whether new code
would be written that avoid it as well.

If that's not the case, then I gather http-push would be better rewritten to
just use http_request, if anything.

>> Apologies for anything wrong I might have done here.  I'm not used to
>> procedures for this sort of patch submission, or terribly familiar with
>> the code base.  I'm seeking advice on whether this approach is sane or
>> completely crazy, and I'm willing to adjust it to make it suitable for
>> inclusion.
>>
>> Signed-off-by: Kevin <kevin@steadfast.net>
>> ---
> 
> Cover letter material (i.e., anything that would not go into the commit
> message of the final commit) should go below the "---".

Thanks, will remember this for future reference.

> Is it safe to just run start_active_slot again without reinitializing
> the request? The 401-handling code in http_request actually restarts a
> new request. I don't immediately see any state that would need to be
> reset; we might have written some data to the output file if curl gave
> us any body data, but presumably it would not have done so for a 401.

In my tests, this particular code flow never returns anything to the original
request call because I interrupt it and start the request over.  Now that I
look at it again, there's a chance it leaks the curl response, but it doesn't
return that response and the new request works fine, replacing the original.

> In the "else" clause you add, I don't think there's any point in
> printing an error. The 401 should get propagated back to the caller, who
> will produce an error. However, you _should_ call credential_reject,
> since you know that the credential you have doesn't work.
> 
> Similarly, you would want to call credential_accept after a successful
> request, so that helpers can store it.

If I decide to continue working on this, I will keep these in mind.  I'm
pretty sure that if I can get smart HTTP working, there's no reason to even
bother with this from my perspective, unless you think there's substantial
value in it.

Thanks for the detailed feedback on the proposed change and suggestions on
alternative options.

-- 
Kevin Stange
Chief Technology Officer
Steadfast Networks
http://steadfast.net
Phone: 312-602-2689 ext. 203 | Fax: 312-602-2688 | Cell: 312-320-5867


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [RFC] Deal with HTTP 401 by requesting credentials.

[Jeff King]

On Fri, Jun 01, 2012 at 10:01:58AM -0700, Junio C Hamano wrote:

> >> Request credentials from the user if none are already defined when a
> >> HTTP 401 is received on a restricted repository.  Then, resubmit the
> >> request and return the final result.
> >> 
> >> This allows all webdav transactions to obtain credentials without having
> >> to individually handle the case in each request.  Having push working
> >> with HTTP auth is needed for a use case I have where storing the
> >> credentials in .netrc or using SSH keys is inappropriate.
> >
> > We already do this at a higher level in http_request, which in turns
> > calls into finish_active_slot. So if we were going to go this route,
> > wouldn't we also want to remove the 401 handling in http_request?
> 
> Wouldn't the higher levels know a lot more about the context this
> request was made, though?  What problem does the patch try to solve?
> Some higher level callers missing the "if we got 401, then reset and
> retry" logic?  Wouldn't it be saner to fix the breakage there?

I don't know that the higher level really has much more context; the URL
is really the only context we use. I think the problem is adequately
solved in http_request; it is simply that the http-push code paths do
not use it. The patch solves that by pushing it to a lower level.

-Peff

Re: [RFC] Deal with HTTP 401 by requesting credentials.

[Jeff King]

On Fri, Jun 01, 2012 at 12:02:08PM -0500, Kevin Stange wrote:

> > The dumb-http push code is the only thing that does not go through
> > http_request these days. So another option would be to refactor it to go
> > through that central point. I took a brief look at this when I was
> > updating the credential code a few months ago, but didn't consider it a
> > priority, as most people should be using smart http these days. Is there
> > a reason you can't use smart-http? It's significantly more efficient.
> 
> Smart HTTP didn't come up in any of my Google searches.  With that as an
> option, I might just drop this work now.  I'd rather see incomplete methods
> that aren't recommended go away than further facilitate their use, personally.

Me too. I would love it if dumb http push just went away. It's extremely
neglected, and has very few advantages over smart http (really, the only
advantage is that the server does not need to run git). However, we do
get bug reports on it occasionally, so I think people are still using
it.

So far our approach has mostly been to prevent any serious regressions,
and otherwise not worry too much about dragging it along with new
features.

> I made the change where I did because I wasn't sure if the push code was
> avoiding using http_request intentionally, and wasn't sure whether new code
> would be written that avoid it as well.
> 
> If that's not the case, then I gather http-push would be better rewritten to
> just use http_request, if anything.

Yes, I think so. It's purely historical and accidental that the
http-push code does not use it (and that the 401-handling happened to be
put at that layer).  The push and fetch sides grew up independently and
organically; I think if somebody were writing it now, they would
organize http.c quite differently.

> If I decide to continue working on this, I will keep these in mind.  I'm
> pretty sure that if I can get smart HTTP working, there's no reason to even
> bother with this from my perspective, unless you think there's substantial
> value in it.

No, I don't think there's substantial value. If you can move to smart
http, you are much better off.

-Peff

Re: [RFC] Deal with HTTP 401 by requesting credentials.

[Kevin Stange]

[-- Attachment #1: Type: text/plain, Size: 2145 bytes --]

On 06/05/2012 11:28 AM, Jeff King wrote:
> On Fri, Jun 01, 2012 at 12:02:08PM -0500, Kevin Stange wrote:
> 
>>> The dumb-http push code is the only thing that does not go through
>>> http_request these days. So another option would be to refactor it to go
>>> through that central point. I took a brief look at this when I was
>>> updating the credential code a few months ago, but didn't consider it a
>>> priority, as most people should be using smart http these days. Is there
>>> a reason you can't use smart-http? It's significantly more efficient.
>>
>> Smart HTTP didn't come up in any of my Google searches.  With that as an
>> option, I might just drop this work now.  I'd rather see incomplete methods
>> that aren't recommended go away than further facilitate their use, personally.
> 
> Me too. I would love it if dumb http push just went away. It's extremely
> neglected, and has very few advantages over smart http (really, the only
> advantage is that the server does not need to run git). However, we do
> get bug reports on it occasionally, so I think people are still using
> it.
> 
> So far our approach has mostly been to prevent any serious regressions,
> and otherwise not worry too much about dragging it along with new
> features.

It seemed a lot of "guides" that I found while searching suggested using the
"dumb" system.  That may be why it remains in use.  I can't imagine too many
people have issues running a CGI to ensure everything works right.

>> If I decide to continue working on this, I will keep these in mind.  I'm
>> pretty sure that if I can get smart HTTP working, there's no reason to even
>> bother with this from my perspective, unless you think there's substantial
>> value in it.
> 
> No, I don't think there's substantial value. If you can move to smart
> http, you are much better off.

I've gotten smart HTTP working perfectly.  Thank you for pointing me to it.  I
am considering this resolved.

-- 
Kevin Stange
Chief Technology Officer
Steadfast Networks
http://steadfast.net
Phone: 312-602-2689 ext. 203 | Fax: 312-602-2688 | Cell: 312-320-5867


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]