GSS-Negotiate authentication requires that all data fit into postbuffer

GSS-Negotiate authentication requires that all data fit into postbuffer

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1639 bytes --]

One thing I just noticed is that if git is using GSS-Negotiate
authentication, the entire POST contents have to fit into however much
memory is specified by http.postbuffer:

  vauxhall ok % git push https://bmc@git.crustytoothpaste.net/git/bmc/test.git development
  Counting objects: 37994, done.
  Delta compression using up to 4 threads.
  Compressing objects: 100% (10683/10683), done.
  Writing objects: 100% (37994/37994), 9.15 MiB | 4.45 MiB/s, done.
  Total 37994 (delta 26760), reused 37633 (delta 26467)
  Unable to rewind rpc post data - try increasing http.postBuffer
  Password for 'https://bmc@git.crustytoothpaste.net': 

GSS-Negotiate authentication always requires a rewind with CURL.

The remote in question only supports Negotiate authentication, so
prompting for a password in this case isn't going to help.  I'm probably
going to look into this over the next couple of days, but two things
need to be done here: 1) do not prompt for a password if only Negotiate
authentication is requested, since it just won't work, and 2) recreate
the data as needed if we have to rewind, since otherwise pushing a fresh
copy of the Linux kernel repo simply isn't going to work as the buffer
will have to be too large.  An alternative is to send a small amount of
data, smaller than the postbuffer, in the first chunk and only fail to
rewind if the second or subsequent chunks need rewinding.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: GSS-Negotiate authentication requires that all data fit into postbuffer

[Ilari Liusvaara]

On Sun, Oct 06, 2013 at 12:42:36AM +0000, brian m. carlson wrote:
> 
> GSS-Negotiate authentication always requires a rewind with CURL.
> 
> The remote in question only supports Negotiate authentication, so
> prompting for a password in this case isn't going to help.  I'm probably
> going to look into this over the next couple of days, but two things
> need to be done here: 1) do not prompt for a password if only Negotiate
> authentication is requested, since it just won't work, and 2) recreate
> the data as needed if we have to rewind, since otherwise pushing a fresh
> copy of the Linux kernel repo simply isn't going to work as the buffer
> will have to be too large.  An alternative is to send a small amount of
> data, smaller than the postbuffer, in the first chunk and only fail to
> rewind if the second or subsequent chunks need rewinding.

Isn't 'Expect: 100-Continue' meant for stuff like this (not that it is
always supported properly)?

-Ilari

Re: GSS-Negotiate authentication requires that all data fit into postbuffer

[Daniel Stenberg]

On Sun, 6 Oct 2013, Ilari Liusvaara wrote:

>> GSS-Negotiate authentication always requires a rewind with CURL.

> Isn't 'Expect: 100-Continue' meant for stuff like this (not that it is 
> always supported properly)?

Yes it is and libcurl uses 100-Continue by default for that purpose. But the 
harsh reality is that lots of (most?) servers just don't care and aren't setup 
to respond properly and instead we end up having to send data multiple times 
in vain.

-- 

  / daniel.haxx.se

Re: GSS-Negotiate authentication requires that all data fit into postbuffer

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

On Sun, Oct 06, 2013 at 05:00:02PM +0200, Daniel Stenberg wrote:
> On Sun, 6 Oct 2013, Ilari Liusvaara wrote:
> 
> >>GSS-Negotiate authentication always requires a rewind with CURL.
> 
> >Isn't 'Expect: 100-Continue' meant for stuff like this (not that
> >it is always supported properly)?
> 
> Yes it is and libcurl uses 100-Continue by default for that purpose.
> But the harsh reality is that lots of (most?) servers just don't
> care and aren't setup to respond properly and instead we end up
> having to send data multiple times in vain.

If there's a way to make Apache with mod_auth_kerb do that with curl,
then it doesn't require a change to git, and I'm happy to make it on my
end.  But using the curl command line client, I don't see an Expect:
100-continue anywhere during the connection using Debian's curl
7.32.0-1.  Do I need to send a certain amount of data to see that
behavior?

The command line I used was

  curl -v -d '0000' -H'Transfer-Encoding: chunked' -H'Content-Type: application/x-git-receive-pack-request' --negotiate -u bmc: https://git.crustytoothpaste.net/git/bmc/test.git/git-receive-pack

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: GSS-Negotiate authentication requires that all data fit into postbuffer

[Daniel Stenberg]

On Sun, 6 Oct 2013, brian m. carlson wrote:

> If there's a way to make Apache with mod_auth_kerb do that with curl, then 
> it doesn't require a change to git, and I'm happy to make it on my end. 
> But using the curl command line client, I don't see an Expect: 100-continue 
> anywhere during the connection using Debian's curl 7.32.0-1.  Do I need to 
> send a certain amount of data to see that behavior?

Correct, curl will enable "Expect: 100-continue" if the post size is > 1024 
bytes.

-- 

  / daniel.haxx.se

Re: GSS-Negotiate authentication requires that all data fit into postbuffer

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

On Sun, Oct 06, 2013 at 05:38:24PM +0200, Daniel Stenberg wrote:
> On Sun, 6 Oct 2013, brian m. carlson wrote:
> 
> >If there's a way to make Apache with mod_auth_kerb do that with
> >curl, then it doesn't require a change to git, and I'm happy to
> >make it on my end. But using the curl command line client, I don't
> >see an Expect: 100-continue anywhere during the connection using
> >Debian's curl 7.32.0-1.  Do I need to send a certain amount of
> >data to see that behavior?
> 
> Correct, curl will enable "Expect: 100-continue" if the post size is
> > 1024 bytes.

I've been able to reproduce this behavior with the curl command line
client, but it looks like we disable Expect: 100-continue in git since
some proxy servers are too stupid to understand it, according to commit
959dfcf.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: GSS-Negotiate authentication requires that all data fit into postbuffer

[Jeff King]

On Sun, Oct 06, 2013 at 05:50:00PM +0000, brian m. carlson wrote:

> On Sun, Oct 06, 2013 at 05:38:24PM +0200, Daniel Stenberg wrote:
> > On Sun, 6 Oct 2013, brian m. carlson wrote:
> > 
> > >If there's a way to make Apache with mod_auth_kerb do that with
> > >curl, then it doesn't require a change to git, and I'm happy to
> > >make it on my end. But using the curl command line client, I don't
> > >see an Expect: 100-continue anywhere during the connection using
> > >Debian's curl 7.32.0-1.  Do I need to send a certain amount of
> > >data to see that behavior?
> > 
> > Correct, curl will enable "Expect: 100-continue" if the post size is
> > > 1024 bytes.
> 
> I've been able to reproduce this behavior with the curl command line
> client, but it looks like we disable Expect: 100-continue in git since
> some proxy servers are too stupid to understand it, according to commit
> 959dfcf.

Yeah, instead we try to make two separate requests, and assume that the
first one clears the path for any further requests. Of course that
doesn't work for auth methods that actually negotiate for each request.

We should probably make the "Expect" suppression optional for people who
know they have working systems. It would be nice to trigger it
automatically when people are using something like GSS, but that
decision happens at the curl layer.

-Peff

Re: GSS-Negotiate authentication requires that all data fit into postbuffer

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

On Mon, Oct 07, 2013 at 08:02:41AM -0400, Jeff King wrote:
> Yeah, instead we try to make two separate requests, and assume that the
> first one clears the path for any further requests. Of course that
> doesn't work for auth methods that actually negotiate for each request.
> 
> We should probably make the "Expect" suppression optional for people who
> know they have working systems. It would be nice to trigger it
> automatically when people are using something like GSS, but that
> decision happens at the curl layer.

Yeah, that's what I ultimately determined would be the best choice.
Technically, we could respawn the process, but that would be a
logistical nightmare and extremely inefficient for large transfers.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]