All of lore.kernel.org
 help / color / mirror / Atom feed
From: Patrick Steinhardt <ps@pks.im>
To: Ravi Mistry via GitGitGadget <gitgitgadget@gmail.com>
Cc: git@vger.kernel.org,
	Vaidas Pilkauskas <vaidas.pilkauskas@shopify.com>,
	"brian m. carlson" <sandals@crustytoothpaste.net>,
	Ravi Mistry <rmistry@google.com>
Subject: Re: [PATCH v3 1/5] http: fix retry count and Retry-After timing
Date: Wed, 18 Nov 2026 13:00:24 +0100	[thread overview]
Message-ID: <av2TTPCkiA-49K_I@pks.im> (raw)
In-Reply-To: <d53778dd4635bd7dfb5b115a23f3642ea3ae40e8.1794022836.git.gitgitgadget@gmail.com>

On Sat, Nov 07, 2026 at 03:40:32AM +0000, Ravi Mistry via GitGitGadget wrote:
> From: Ravi Mistry <rmistry@google.com>
> 
> Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries,
> 2026-03-17) introduced two small bugs in HTTP 429 retry handling:
> 
> 1. In http_request_recoverable(), rate_limit_retries is initialized to
>    http_max_retries (the number of retries to perform after the initial
>    failed request) and decremented with pre-decrement
>    (--rate_limit_retries) in the while condition. Because of the
>    pre-decrement, setting http.maxRetries=1 performs 0 retries, and
>    more generally N configured retries only retries N-1 times. In
>    addition, if the first request returns 401 (HTTP_REAUTH) and the
>    re-authenticated request returns 429 when http.maxRetries is 0,
>    --rate_limit_retries wraps from 0 to -1 (truthy in C).
> 
>    Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0
>    in the loop condition and checking http_max_retries <= 0 after the
>    loop (which handles both an initial 429 and a 429 after 401 re-auth
>    without logging "retries-exhausted", making the pre-loop check
>    redundant).
> 
> 2. In http_request(), CURLINFO_RETRY_AFTER is queried after
>    run_one_slot() has already called handle_curl_result(), which logs
>    results->retry_after to Trace2 ("http/429-retry-after") while it is
>    still -1. Move the CURLINFO_RETRY_AFTER extraction into
>    finish_active_slot() (initializing slot->results->retry_after = -1
>    first) so that both handle_curl_result() and callers of
>    run_one_slot() observe the parsed Retry-After value.

Nit: feels like two separate bugs that should have been fixed in two
separate commits.

> diff --git a/http.c b/http.c
> index d79678d01f..0308f973a6 100644
> --- a/http.c
> +++ b/http.c
> @@ -345,11 +345,20 @@ static void finish_active_slot(struct active_request_slot *slot)
>  	if (slot->results) {
>  		slot->results->curl_result = slot->curl_result;
>  		slot->results->http_code = slot->http_code;
> +		slot->results->retry_after = -1;
>  		curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL,
>  				  &slot->results->auth_avail);
>  
>  		curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE,
>  			&slot->results->http_connectcode);
> +#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
> +		{
> +			curl_off_t retry_after;
> +			if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
> +					      &retry_after) == CURLE_OK && retry_after > 0)
> +				slot->results->retry_after = (long)retry_after;
> +		}

Makes me wonder whether `results->retry_after` should be declared as
`curl_off_t`. But that's not a problem introduced by this patch series,
as you simply move code around anyway.

> @@ -2474,7 +2470,8 @@ static int http_request_recoverable(const char *url,
>  	}
>  
>  	while ((ret == HTTP_REAUTH && --i) ||
> -	       (ret == HTTP_RATE_LIMITED && --rate_limit_retries)) {
> +	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
> +		rate_limit_retries-- > 0)) {
>  		long retry_delay = -1;
>  		/*
>  		 * The previous request may have put cruft into our output stream; we

Right, this here fixes the pre- vs post-decrement issue. Makes sense.

Honestly, every time I see a pre-decrement I'm already wondering whether
it is correct in the first place. It always feels a bit too neat in my
opinion.

> @@ -2518,6 +2515,8 @@ static int http_request_recoverable(const char *url,
>  		ret = http_request(url, result, target, options);
>  	}
>  	if (ret == HTTP_RATE_LIMITED) {
> +		if (http_max_retries <= 0)
> +			return HTTP_ERROR;
>  		trace2_data_string("http", the_repository,
>  				   "http/429-error", "retries-exhausted");
>  		return HTTP_RATE_LIMITED;

Right. If we never had retries configured in the first place then it
doesn't make sense to log "retries-exhausted", because there weren't
any.

It's a bit curious that we return `HTTP_ERROR` instead of
`HTTP_RATE_LIMITED`, as we're essentially still rate-limited. I would
have expected us to only skip loggint the "retries-exhausted" message.

> diff --git a/t/lib-httpd/http-429.sh b/t/lib-httpd/http-429.sh
> index 1a5d7987db..5537fb7ac4 100644
> --- a/t/lib-httpd/http-429.sh
> +++ b/t/lib-httpd/http-429.sh
> @@ -32,12 +32,33 @@ repo_name="${repo_path%%/*}"
>  safe_name=$(echo "${test_context}-${retry_after}-${repo_name}" | tr '/' '_' | tr -cd 'a-zA-Z0-9_-')
>  state="http-429-state-${safe_name}"
>  
> -# Check if this is the first call (no state directory exists), or if
> -# the retry-after-value is "permanent", which indicates a 429 must be
> -# returned for every request (even if the state directory exists).
> -if test "$retry_after" != permanent && ! mkdir "$state" 2>/dev/null
> +max_fails=1
> +case "$test_context" in
> +	*-fail-2*) max_fails=2 ;;
> +	*-fail-3*) max_fails=3 ;;
> +esac

Nit: coding style, arms of a `case` statement are typically on the same
level as `case`.

Patrick

  reply	other threads:[~2026-11-18 12:00 UTC|newest]

Thread overview: 49+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-31 15:57 [PATCH 0/4] http: exponential backoff, configurable status codes, and post_rpc() retries Ravi Mistry via GitGitGadget
2026-10-31 15:57 ` [PATCH 1/4] http: fix retry count and Retry-After timing Ravi Mistry via GitGitGadget
2026-11-03  1:03   ` Junio C Hamano
2026-11-03 19:05     ` Ravi Mistry
2026-10-31 15:57 ` [PATCH 2/4] http: add exponential backoff without Retry-After Ravi Mistry via GitGitGadget
2026-11-03 13:21   ` Junio C Hamano
2026-11-03 19:05     ` Ravi Mistry
2026-10-31 15:57 ` [PATCH 3/4] http: add http.retryStatusCodes config option Ravi Mistry via GitGitGadget
2026-11-03 13:41   ` Junio C Hamano
2026-11-03 19:06     ` Ravi Mistry
2026-10-31 15:57 ` [PATCH 4/4] remote-curl: retry transient HTTP errors in post_rpc() Ravi Mistry via GitGitGadget
2026-11-03 14:47   ` Junio C Hamano
2026-11-03 18:57 ` [PATCH v2 0/4] http: exponential backoff, configurable status codes, and post_rpc() retries Ravi Mistry via GitGitGadget
2026-11-03 18:57   ` [PATCH v2 1/4] http: fix retry count and Retry-After timing Ravi Mistry via GitGitGadget
2026-11-03 18:57   ` [PATCH v2 2/4] http: add exponential backoff without Retry-After Ravi Mistry via GitGitGadget
2026-11-04 22:05     ` brian m. carlson
2026-11-05  0:10       ` Ravi Mistry
2026-11-05 17:45         ` Junio C Hamano
2026-11-05 22:32         ` brian m. carlson
2026-11-06 16:27           ` Ravi Mistry
2026-11-03 18:57   ` [PATCH v2 3/4] http: add http.retryStatusCodes config option Ravi Mistry via GitGitGadget
2026-11-03 20:56     ` Junio C Hamano
2026-11-03 18:57   ` [PATCH v2 4/4] remote-curl: retry transient HTTP errors in post_rpc() Ravi Mistry via GitGitGadget
2026-11-07  3:40 ` [PATCH v3 0/5] http: exponential backoff, configurable status codes, and post_rpc() retries Ravi Mistry via GitGitGadget
2026-11-07  3:40   ` [PATCH v3 1/5] http: fix retry count and Retry-After timing Ravi Mistry via GitGitGadget
2026-11-18 12:00     ` Patrick Steinhardt [this message]
2026-11-18 21:52       ` Ravi Mistry
2026-11-07  3:40   ` [PATCH v3 2/5] http: add exponential backoff without Retry-After Ravi Mistry via GitGitGadget
2026-11-18 12:00     ` Patrick Steinhardt
2026-11-18 21:53       ` Ravi Mistry
2026-11-07  3:40   ` [PATCH v3 3/5] http: add http.retryStatusCodes config option Ravi Mistry via GitGitGadget
2026-11-18 12:00     ` Patrick Steinhardt
2026-11-18 21:53       ` Ravi Mistry
2026-11-19 11:28         ` Patrick Steinhardt
2026-11-19 12:23           ` Ravi Mistry
2026-11-07  3:40   ` [PATCH v3 4/5] remote-curl: retry transient HTTP errors in post_rpc() Ravi Mistry via GitGitGadget
2026-11-18 12:01     ` Patrick Steinhardt
2026-11-18 21:53       ` Ravi Mistry
2026-11-07  3:40   ` [PATCH v3 5/5] http: default http.retryAfter to 10 seconds Ravi Mistry via GitGitGadget
2026-11-18 12:01     ` Patrick Steinhardt
2026-11-18 21:54       ` Ravi Mistry
2026-11-07  4:37   ` [PATCH v3 0/5] http: exponential backoff, configurable status codes, and post_rpc() retries Junio C Hamano
2026-11-10  3:58     ` Ravi Mistry
2026-11-19 15:57 ` [PATCH v4 " Ravi Mistry via GitGitGadget
2026-11-19 15:57   ` [PATCH v4 1/5] http: fix retry count and Retry-After timing Ravi Mistry via GitGitGadget
2026-11-19 15:58   ` [PATCH v4 2/5] http: add exponential backoff without Retry-After Ravi Mistry via GitGitGadget
2026-11-19 15:58   ` [PATCH v4 3/5] http: add http.retryStatusCodes config option Ravi Mistry via GitGitGadget
2026-11-19 15:58   ` [PATCH v4 4/5] remote-curl: retry transient HTTP errors in post_rpc() Ravi Mistry via GitGitGadget
2026-11-19 15:58   ` [PATCH v4 5/5] http: default http.retryAfter to 10 seconds Ravi Mistry via GitGitGadget

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av2TTPCkiA-49K_I@pks.im \
    --to=ps@pks.im \
    --cc=git@vger.kernel.org \
    --cc=gitgitgadget@gmail.com \
    --cc=rmistry@google.com \
    --cc=sandals@crustytoothpaste.net \
    --cc=vaidas.pilkauskas@shopify.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.