[PATCH v2 0/4] http: exponential backoff, configurable status codes, and post_rpc() retries

["Ravi Mistry via GitGitGadget" <gitgitgadget@gmail.com>]

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries,
2026-03-17) added HTTP 429 retry support using http.maxRetries,
http.retryAfter, and http.maxRetryTime.

In practice, when running Git against hosting backends and reverse proxies
under high concurrency, four gaps remain:

 1. Off-by-one retry count and Retry-After Trace2 timing: Because
    http_request_recoverable() pre-decrements the retry counter
    (--rate_limit_retries) in its loop condition, setting http.maxRetries=1
    performs 0 retries (and N configured retries performs N-1 retries). In
    addition, the Retry-After header is extracted after Trace2 logging has
    already run, so the http/429-retry-after Trace2 event always logs -1.

 2. No exponential backoff when Retry-After is absent: When a server does
    not send a Retry-After header, Git waits the same fixed http.retryAfter
    delay on every retry. If that delay is short (e.g., 1s), Git burns
    through all of its retries in just a few seconds before a busy server
    can recover. If that delay is long, even a brief glitch has to wait a
    long time on the first retry. Doubling the delay on each attempt (1s,
    2s, 4s, ...) keeps the first retry fast while giving the server more
    time to recover on later attempts, matching curl's own --retry behavior.

 3. Transient 5xx errors cannot be retried: Reverse proxies, load balancers,
    and Git backends frequently surface transient overload or connection
    resets as 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway
    Timeout. Because 5xx errors are not universally transient across all
    environments, 429 should remain the default, with an opt-in config to
    specify additional status codes.

 4. HTTP POST requests are not retried: Git's existing 429 retry logic only
    covers HTTP GET requests (such as the initial /info/refs handshake).
    However, the actual data transfer during fetch and clone, as well as
    on-demand blob downloads in partial clones (during checkout, diff, or
    blame), are sent as HTTP POST requests (/git-upload-pack) via
    post_rpc(), which only retries authentication and immediately aborts on
    429 or 5xx.

This 4-patch series addresses all four gaps:

 * Patch 1/4 fixes the off-by-one retry count bug and extracts Retry-After
   in finish_active_slot() before Trace2 logging runs.
 * Patch 2/4 doubles http.retryAfter on each subsequent retry attempt (1s,
   2s, 4s, capped at http.maxRetryTime) when Retry-After is absent.
 * Patch 3/4 adds http.retryStatusCodes (and GIT_HTTP_RETRY_STATUS_CODES),
   defaulting to "429", so environments can opt into retrying codes like
   "429,502,503,504".
 * Patch 4/4 extracts a shared http_retry_wait() helper and extends retry
   support to HTTP POST requests in probe_rpc() and post_rpc() (while fixing
   a missing Content-Encoding: gzip header on retry).

Prior discussions:

 * https://lore.kernel.org/git/20201012184806.166251-3-smcallis@google.com/
 * https://lore.kernel.org/git/pull.2008.v6.git.1773752435.gitgitgadget@gmail.com/

Changes since v1:

 * Patch 1/4: Clarified in the commit message why the http_max_retries <= 0
   check was moved after the retry loop.
 * Patch 2/4: Passed http_max_retries - rate_limit_retries instead of adding
   a separate attempt counter, and clarified the http.retryAfter and
   http.maxRetryTime docs.
 * Patch 3/4: Restored (Too Many Requests) and clarified
   http.retryStatusCodes in the config docs, and moved the 400..599 range
   check to the top of is_retryable_http_code().
 * Patch 4/4: Added the attempt counter here when replacing
   rate_limit_retries with http_retry_wait().

Ravi Mistry (4):
  http: fix retry count and Retry-After timing
  http: add exponential backoff without Retry-After
  http: add http.retryStatusCodes config option
  remote-curl: retry transient HTTP errors in post_rpc()

 Documentation/config/http.adoc |  52 ++++++++---
 http.c                         | 125 ++++++++++++++++++--------
 http.h                         |  12 ++-
 remote-curl.c                  |  75 ++++++++--------
 t/lib-httpd/http-429.sh        |  63 ++++++++++++--
 t/t5584-http-429-retry.sh      | 155 +++++++++++++++++++++++++++++++++
 6 files changed, 385 insertions(+), 97 deletions(-)


base-commit: 8d1b215c01512b9658d4525c54ff9b9245f9a7e8
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2287%2Frmistry%2Fhttp-exponential-retries-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2287/rmistry/http-exponential-retries-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2287

Range-diff vs v1:

 1:  80b7f207f4 ! 1:  d53778dd46 http: fix retry count and Retry-After timing
     @@ Commit message
             --rate_limit_retries wraps from 0 to -1 (truthy in C).
      
             Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0
     -       in the loop condition and returning HTTP_ERROR when
     -       http_max_retries <= 0.
     +       in the loop condition and checking http_max_retries <= 0 after the
     +       loop (which handles both an initial 429 and a 429 after 401 re-auth
     +       without logging "retries-exhausted", making the pre-loop check
     +       redundant).
      
          2. In http_request(), CURLINFO_RETRY_AFTER is queried after
             run_one_slot() has already called handle_curl_result(), which logs
 2:  36cef34668 ! 2:  fe1e847ded http: add exponential backoff without Retry-After
     @@ Commit message
          http.retryAfter * 2^(attempt - 1)), clamping the delay at
          http.maxRetryTime. This keeps the first retry fast while giving the
          server more time to recover on later attempts, matching curl's own
     -    --retry behavior. When the server provides an explicit Retry-After
     -    header, continue to honor the server-provided delay (or fail fast if it
     -    exceeds http.maxRetryTime).
     +    --retry behavior (while keeping http.retryAfter's existing default of 0
     +    for immediate retries when unconfigured). When the server provides an
     +    explicit Retry-After header, continue to honor the server-provided delay
     +    (or fail fast if it exceeds http.maxRetryTime).
      
          Signed-off-by: Ravi Mistry <rmistry@google.com>
      
     @@ Documentation/config/http.adoc: http.keepAliveCount::
      +	Initial wait time in seconds before retrying when a server returns
       	HTTP 429 (Too Many Requests) without a Retry-After header.
      -	Defaults to 0 (retry immediately). When a Retry-After header is
     -+	Defaults to 0 (retry immediately). When set to a positive value and
     -+	no Retry-After header is present, the delay doubles on each
     -+	subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
     ++	Defaults to 0, which retries immediately without waiting on every
     ++	attempt (unless a Retry-After header is present). When set to a
     ++	positive value and no Retry-After header is present, the first
     ++	retry waits `http.retryAfter` seconds and the delay doubles on
     ++	each subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
      +	capped at `http.maxRetryTime`. When a Retry-After header is
       	present, its value takes precedence over this setting; however,
       	automatic use of the server-provided `Retry-After` header requires
       	libcurl 7.66.0 or later. On older versions, configure this setting
     -@@ Documentation/config/http.adoc: http.maxRetryTime::
     - 	handling HTTP 429 (Too Many Requests) responses. If the server
     - 	requests a delay (via Retry-After header) or if `http.retryAfter`
     - 	is configured with a value that exceeds this maximum, Git will fail
     +@@ Documentation/config/http.adoc: http.maxRetries::
     + 
     + http.maxRetryTime::
     + 	Maximum time in seconds to wait for a single retry attempt when
     +-	handling HTTP 429 (Too Many Requests) responses. If the server
     +-	requests a delay (via Retry-After header) or if `http.retryAfter`
     +-	is configured with a value that exceeds this maximum, Git will fail
      -	immediately rather than waiting. Default is 300 seconds (5 minutes).
     -+	immediately rather than waiting. Exponentially backed-off delays
     -+	computed from `http.retryAfter` are capped at this maximum. Default
     -+	is 300 seconds (5 minutes).
     ++	handling HTTP 429 (Too Many Requests) responses.
     ++	If the server requests a delay (via Retry-After header) or if the
     ++	configured `http.retryAfter` value itself exceeds this maximum,
     ++	Git will fail immediately rather than waiting. Exponentially
     ++	backed-off delays computed from `http.retryAfter`, however, are
     ++	capped at this maximum rather than causing an immediate failure.
     ++	Default is 300 seconds (5 minutes).
       	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
       	variable. See also `http.retryAfter` and `http.maxRetries`.
       
     @@ http.c: static long handle_rate_limit_retry(long slot_retry_after)
       	}
       }
       
     -@@ http.c: static int http_request_recoverable(const char *url,
     - 	int i = 3;
     - 	int ret;
     - 	int rate_limit_retries = http_max_retries;
     -+	int attempt = 0;
     - 
     - 	if (!options)
     - 		options = &empty_opts;
      @@ http.c: static int http_request_recoverable(const char *url,
       			BUG("Unknown http_request target");
       		}
       		if (ret == HTTP_RATE_LIMITED) {
      -			retry_delay = handle_rate_limit_retry(options->retry_after);
     -+			retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt);
     ++			retry_delay = handle_rate_limit_retry(
     ++				options->retry_after,
     ++				http_max_retries - rate_limit_retries);
       			if (retry_delay < 0)
       				return HTTP_ERROR;
       
 3:  f56ab2afaa ! 3:  6510dddc39 http: add http.retryStatusCodes config option
     @@ Documentation/config/http.adoc: http.keepAliveCount::
       http.retryAfter::
       	Initial wait time in seconds before retrying when a server returns
      -	HTTP 429 (Too Many Requests) without a Retry-After header.
     -+	a retryable HTTP status code (HTTP 429 by default) without a
     -+	Retry-After header.
     - 	Defaults to 0 (retry immediately). When set to a positive value and
     - 	no Retry-After header is present, the delay doubles on each
     - 	subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
     ++	a retryable HTTP status code without a Retry-After header. The
     ++	set of retryable status codes is configured by
     ++	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
     ++	Requests) by default.
     + 	Defaults to 0, which retries immediately without waiting on every
     + 	attempt (unless a Retry-After header is present). When set to a
     + 	positive value and no Retry-After header is present, the first
      @@ Documentation/config/http.adoc: http.retryAfter::
       	libcurl 7.66.0 or later. On older versions, configure this setting
       	manually to control the retry delay. Can be overridden by the
     @@ Documentation/config/http.adoc: http.retryAfter::
      -	Maximum number of times to retry after receiving HTTP 429 (Too Many
      -	Requests) responses. Set to 0 (the default) to disable retries.
      +	Maximum number of times to retry after receiving a retryable HTTP
     -+	status code (HTTP 429 by default). Set to 0 (the default) to disable
     -+	retries.
     ++	status code. The set of retryable status codes is configured by
     ++	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
     ++	Requests) by default. Set to 0 (the default) to disable retries.
       	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
      -	See also `http.retryAfter` and `http.maxRetryTime`.
      +	See also `http.retryAfter`, `http.maxRetryTime`, and
     @@ Documentation/config/http.adoc: http.retryAfter::
       
       http.maxRetryTime::
       	Maximum time in seconds to wait for a single retry attempt when
     --	handling HTTP 429 (Too Many Requests) responses. If the server
     --	requests a delay (via Retry-After header) or if `http.retryAfter`
     -+	handling a retryable HTTP status code (HTTP 429 by default). If the
     -+	server requests a delay (via Retry-After header) or if `http.retryAfter`
     - 	is configured with a value that exceeds this maximum, Git will fail
     - 	immediately rather than waiting. Exponentially backed-off delays
     - 	computed from `http.retryAfter` are capped at this maximum. Default
     - 	is 300 seconds (5 minutes).
     +-	handling HTTP 429 (Too Many Requests) responses.
     ++	handling a retryable HTTP status code. The set of retryable
     ++	status codes is configured by `http.retryStatusCodes`, which
     ++	includes only HTTP 429 (Too Many Requests) by default.
     + 	If the server requests a delay (via Retry-After header) or if the
     + 	configured `http.retryAfter` value itself exceeds this maximum,
     + 	Git will fail immediately rather than waiting. Exponentially
     +@@ Documentation/config/http.adoc: http.maxRetryTime::
     + 	capped at this maximum rather than causing an immediate failure.
     + 	Default is 300 seconds (5 minutes).
       	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
      -	variable. See also `http.retryAfter` and `http.maxRetries`.
      +	variable. See also `http.retryAfter`, `http.maxRetries`, and
     @@ Documentation/config/http.adoc: http.retryAfter::
      +http.retryStatusCodes::
      +	Comma- or whitespace-separated list of HTTP status codes (in the
      +	400..599 range) that should be retried up to `http.maxRetries`
     -+	times. Defaults to `429`. For example, setting this to
     -+	`429,502,503,504` also retries transient `5xx` gateway and
     ++	times. Defaults to `429` (Too Many Requests). For example, setting
     ++	this to `429,502,503,504` also retries transient `5xx` gateway and
      +	service-unavailable errors. Can be overridden by the
      +	`GIT_HTTP_RETRY_STATUS_CODES` environment variable.
      +	See also `http.maxRetries`, `http.retryAfter`, and
     @@ http.c: void normalize_curl_result(CURLcode *result, long http_code,
      +	const char *p = http_retry_status_codes ? http_retry_status_codes : "429";
      +	char *end;
      +
     -+	for (; code >= 400 && code < 600 && *p; p = end) {
     ++	if (code < 400 || code >= 600)
     ++		return 0;
     ++
     ++	for (; *p; p = end) {
      +		while (*p == ',' || isspace(*p))
      +			p++;
      +		if (strtol(p, &end, 10) == code && end > p &&
 4:  52819accc8 ! 4:  786964ff00 remote-curl: retry transient HTTP errors in post_rpc()
     @@ http.c: static int http_request_recoverable(const char *url,
       	int i = 3;
       	int ret;
      -	int rate_limit_retries = http_max_retries;
     - 	int attempt = 0;
     ++	int attempt = 0;
       
       	if (!options)
     + 		options = &empty_opts;
      @@ http.c: static int http_request_recoverable(const char *url,
       	}
       
     @@ http.c: static int http_request_recoverable(const char *url,
       			BUG("Unknown http_request target");
       		}
      -		if (ret == HTTP_RATE_LIMITED) {
     --			retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt);
     +-			retry_delay = handle_rate_limit_retry(
     +-				options->retry_after,
     +-				http_max_retries - rate_limit_retries);
      -			if (retry_delay < 0)
      -				return HTTP_ERROR;
      -

-- 
gitgitgadget