[PATCH] Enable info/refs gzip decompression in HTTP client

[PATCH] Enable info/refs gzip decompression in HTTP client

[Shawn O. Pearce]

From: "Shawn O. Pearce" <spearce@spearce.org>

Some HTTP servers try to use gzip compression on the /info/refs
request to save transfer bandwidth. Repositories with many tags
may find the /info/refs request can be gzipped to be 50% of the
original size due to the few but often repeated bytes used (hex
SHA-1 and commonly digits in tag names).

For most HTTP requests enable "Accept-Encoding: gzip" ensuring
the /info/refs payload can use this encoding format.

Disable the Accept-Encoding header on probe RPCs as response bodies
are supposed to be exactly 4 bytes long, "0000". The HTTP headers
requesting and indicating compression use more space than the data
transferred in the body.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 http.c                | 1 +
 remote-curl.c         | 4 ++--
 t/t5551-http-fetch.sh | 3 ++-
 3 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/http.c b/http.c
index 9bac1d8..345c171 100644
--- a/http.c
+++ b/http.c
@@ -818,6 +818,7 @@ static int http_request(const char *url, void *result, int target, int options)
 
 	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
+	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "gzip");
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
diff --git a/remote-curl.c b/remote-curl.c
index 3ec474f..4a0927e 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -393,7 +393,7 @@ static int probe_rpc(struct rpc_state *rpc)
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
 	curl_easy_setopt(slot->curl, CURLOPT_URL, rpc->service_url);
-	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
+	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL);
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, "0000");
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
@@ -449,7 +449,7 @@ static int post_rpc(struct rpc_state *rpc)
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
 	curl_easy_setopt(slot->curl, CURLOPT_URL, rpc->service_url);
-	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
+	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "gzip");
 
 	headers = curl_slist_append(headers, rpc->hdr_content_type);
 	headers = curl_slist_append(headers, rpc->hdr_accept);
diff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh
index 2db5c35..380c175 100755
--- a/t/t5551-http-fetch.sh
+++ b/t/t5551-http-fetch.sh
@@ -32,13 +32,14 @@ setup_askpass_helper
 cat >exp <<EOF
 > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1
 > Accept: */*
+> Accept-Encoding: gzip
 > Pragma: no-cache
 < HTTP/1.1 200 OK
 < Pragma: no-cache
 < Cache-Control: no-cache, max-age=0, must-revalidate
 < Content-Type: application/x-git-upload-pack-advertisement
 > POST /smart/repo.git/git-upload-pack HTTP/1.1
-> Accept-Encoding: deflate, gzip
+> Accept-Encoding: gzip
 > Content-Type: application/x-git-upload-pack-request
 > Accept: application/x-git-upload-pack-result
 > Content-Length: xxx
-- 
1.7.12.1.510.g5dd77d8

Re: [PATCH] Enable info/refs gzip decompression in HTTP client

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> From: "Shawn O. Pearce" <spearce@spearce.org>
>
> Some HTTP servers try to use gzip compression on the /info/refs
> request to save transfer bandwidth. Repositories with many tags
> may find the /info/refs request can be gzipped to be 50% of the
> original size due to the few but often repeated bytes used (hex
> SHA-1 and commonly digits in tag names).
>
> For most HTTP requests enable "Accept-Encoding: gzip" ensuring
> the /info/refs payload can use this encoding format.
>
> Disable the Accept-Encoding header on probe RPCs as response bodies
> are supposed to be exactly 4 bytes long, "0000". The HTTP headers
> requesting and indicating compression use more space than the data
> transferred in the body.

All of the above sounds very convincing, but ...

> diff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh
> index 2db5c35..380c175 100755
> --- a/t/t5551-http-fetch.sh
> +++ b/t/t5551-http-fetch.sh
> @@ -32,13 +32,14 @@ setup_askpass_helper
>  cat >exp <<EOF
>  > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1
>  > Accept: */*
> +> Accept-Encoding: gzip
>  > Pragma: no-cache
>  < HTTP/1.1 200 OK
>  < Pragma: no-cache
>  < Cache-Control: no-cache, max-age=0, must-revalidate
>  < Content-Type: application/x-git-upload-pack-advertisement
>  > POST /smart/repo.git/git-upload-pack HTTP/1.1
> -> Accept-Encoding: deflate, gzip
> +> Accept-Encoding: gzip

... was loss of "deflate" intended?  If so why?  Could you explain
it in the log message?

>  > Content-Type: application/x-git-upload-pack-request
>  > Accept: application/x-git-upload-pack-result
>  > Content-Length: xxx

Re: [PATCH] Enable info/refs gzip decompression in HTTP client

[Shawn Pearce]

On Wed, Sep 19, 2012 at 5:43 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> -> Accept-Encoding: deflate, gzip
>> +> Accept-Encoding: gzip
>
> ... was loss of "deflate" intended?  If so why?  Could you explain
> it in the log message?

Yes. I would add the following to the end of the commit message as a
new paragraph, please amend this for me:

--8<--
Only request gzip encoding from servers. Although deflate is
supported by libcurl, most servers have standardized on gzip
encoding for compression as that is what most browsers support.
Asking for deflate increases request sizes by a few bytes, but
is unlikely to ever be used by a server.
-->8--