Re: [PATCH/RFC] Allow curl to rewind the RPC read buffer at any time

["Shawn O. Pearce" <spearce@spearce.org>]

Martin Storsj? <martin@martin.st> wrote:
> When using multi-pass authentication methods, the curl library may
> need to rewind the read buffers used for providing data to HTTP POST,
> if data has been output before a 401 error is received.
> 
> This solution buffers all data read by the curl library, in order to allow
> it to rewind the reading buffer at any time later.

NAK.


In the case of git-upload-pack requests, we should fit into 1 MiB
almost all of the time, and thus not need to grow the http.postBuffer
to support a rewind.  The state data plus current have list isn't
all that large.  A 1 MiB request means we have over 20,900 commits
in common with the remote and still haven't been able to find a
sufficient cut point.  Or the remote has 20,000 active, unrelated
branches we are trying to fetch.  Either way, this is a really sick
and twisted situation.

In the case of git-receive-pack requests, we might be uploading an
entire project to an empty repository on the remote side.  This could
be 8 GiB worth of data if the project was something huge like KDE.
We can't assume that we should malloc 8 GiB of memory to buffer
the payload.

The *correct* way to support an arbitrary rewind is to modify the
outgoing channel from remote-curl to its protocol engine (client.in
within the rpc_service method) to somehow request the protocol engine
(aka git-send-pack or git-fetch-pack) to stop and regenerate the
current request.


Another approach would be to modify http-backend (and the protocol)
to support an "auth ping" request prior to spooling out the entire
payload if its more than an http.postBuffer size.  Basically we
do what the "Expect: 100-continue" protocol is supposed to do,
but in the application layer rather than the HTTP/1.1 layer, so
our CGI actually gets invoked.

This unfortunately still relies on the underlying libcurl to not
discard the authentication data after that initial "auth ping".
But to be honest, I think that is a reasonable expectation.  The
#@!*@!* library should be able to generate two requests back-to-back
to the same URL without needing to rewind the 2nd request.

-- 
Shawn.