From: Patrick Steinhardt <ps@pks.im>
To: Ravi Mistry via GitGitGadget <gitgitgadget@gmail.com>
Cc: git@vger.kernel.org,
Vaidas Pilkauskas <vaidas.pilkauskas@shopify.com>,
"brian m. carlson" <sandals@crustytoothpaste.net>,
Ravi Mistry <rmistry@google.com>
Subject: Re: [PATCH v3 1/5] http: fix retry count and Retry-After timing
Date: Wed, 18 Nov 2026 13:00:24 +0100 [thread overview]
Message-ID: <av2TTPCkiA-49K_I@pks.im> (raw)
In-Reply-To: <d53778dd4635bd7dfb5b115a23f3642ea3ae40e8.1794022836.git.gitgitgadget@gmail.com>
On Sat, Nov 07, 2026 at 03:40:32AM +0000, Ravi Mistry via GitGitGadget wrote:
> From: Ravi Mistry <rmistry@google.com>
>
> Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries,
> 2026-03-17) introduced two small bugs in HTTP 429 retry handling:
>
> 1. In http_request_recoverable(), rate_limit_retries is initialized to
> http_max_retries (the number of retries to perform after the initial
> failed request) and decremented with pre-decrement
> (--rate_limit_retries) in the while condition. Because of the
> pre-decrement, setting http.maxRetries=1 performs 0 retries, and
> more generally N configured retries only retries N-1 times. In
> addition, if the first request returns 401 (HTTP_REAUTH) and the
> re-authenticated request returns 429 when http.maxRetries is 0,
> --rate_limit_retries wraps from 0 to -1 (truthy in C).
>
> Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0
> in the loop condition and checking http_max_retries <= 0 after the
> loop (which handles both an initial 429 and a 429 after 401 re-auth
> without logging "retries-exhausted", making the pre-loop check
> redundant).
>
> 2. In http_request(), CURLINFO_RETRY_AFTER is queried after
> run_one_slot() has already called handle_curl_result(), which logs
> results->retry_after to Trace2 ("http/429-retry-after") while it is
> still -1. Move the CURLINFO_RETRY_AFTER extraction into
> finish_active_slot() (initializing slot->results->retry_after = -1
> first) so that both handle_curl_result() and callers of
> run_one_slot() observe the parsed Retry-After value.
Nit: feels like two separate bugs that should have been fixed in two
separate commits.
> diff --git a/http.c b/http.c
> index d79678d01f..0308f973a6 100644
> --- a/http.c
> +++ b/http.c
> @@ -345,11 +345,20 @@ static void finish_active_slot(struct active_request_slot *slot)
> if (slot->results) {
> slot->results->curl_result = slot->curl_result;
> slot->results->http_code = slot->http_code;
> + slot->results->retry_after = -1;
> curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL,
> &slot->results->auth_avail);
>
> curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE,
> &slot->results->http_connectcode);
> +#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
> + {
> + curl_off_t retry_after;
> + if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
> + &retry_after) == CURLE_OK && retry_after > 0)
> + slot->results->retry_after = (long)retry_after;
> + }
Makes me wonder whether `results->retry_after` should be declared as
`curl_off_t`. But that's not a problem introduced by this patch series,
as you simply move code around anyway.
> @@ -2474,7 +2470,8 @@ static int http_request_recoverable(const char *url,
> }
>
> while ((ret == HTTP_REAUTH && --i) ||
> - (ret == HTTP_RATE_LIMITED && --rate_limit_retries)) {
> + (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
> + rate_limit_retries-- > 0)) {
> long retry_delay = -1;
> /*
> * The previous request may have put cruft into our output stream; we
Right, this here fixes the pre- vs post-decrement issue. Makes sense.
Honestly, every time I see a pre-decrement I'm already wondering whether
it is correct in the first place. It always feels a bit too neat in my
opinion.
> @@ -2518,6 +2515,8 @@ static int http_request_recoverable(const char *url,
> ret = http_request(url, result, target, options);
> }
> if (ret == HTTP_RATE_LIMITED) {
> + if (http_max_retries <= 0)
> + return HTTP_ERROR;
> trace2_data_string("http", the_repository,
> "http/429-error", "retries-exhausted");
> return HTTP_RATE_LIMITED;
Right. If we never had retries configured in the first place then it
doesn't make sense to log "retries-exhausted", because there weren't
any.
It's a bit curious that we return `HTTP_ERROR` instead of
`HTTP_RATE_LIMITED`, as we're essentially still rate-limited. I would
have expected us to only skip loggint the "retries-exhausted" message.
> diff --git a/t/lib-httpd/http-429.sh b/t/lib-httpd/http-429.sh
> index 1a5d7987db..5537fb7ac4 100644
> --- a/t/lib-httpd/http-429.sh
> +++ b/t/lib-httpd/http-429.sh
> @@ -32,12 +32,33 @@ repo_name="${repo_path%%/*}"
> safe_name=$(echo "${test_context}-${retry_after}-${repo_name}" | tr '/' '_' | tr -cd 'a-zA-Z0-9_-')
> state="http-429-state-${safe_name}"
>
> -# Check if this is the first call (no state directory exists), or if
> -# the retry-after-value is "permanent", which indicates a 429 must be
> -# returned for every request (even if the state directory exists).
> -if test "$retry_after" != permanent && ! mkdir "$state" 2>/dev/null
> +max_fails=1
> +case "$test_context" in
> + *-fail-2*) max_fails=2 ;;
> + *-fail-3*) max_fails=3 ;;
> +esac
Nit: coding style, arms of a `case` statement are typically on the same
level as `case`.
Patrick
next prev parent reply other threads:[~2026-11-18 12:00 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-31 15:57 [PATCH 0/4] http: exponential backoff, configurable status codes, and post_rpc() retries Ravi Mistry via GitGitGadget
2026-10-31 15:57 ` [PATCH 1/4] http: fix retry count and Retry-After timing Ravi Mistry via GitGitGadget
2026-11-03 1:03 ` Junio C Hamano
2026-11-03 19:05 ` Ravi Mistry
2026-10-31 15:57 ` [PATCH 2/4] http: add exponential backoff without Retry-After Ravi Mistry via GitGitGadget
2026-11-03 13:21 ` Junio C Hamano
2026-11-03 19:05 ` Ravi Mistry
2026-10-31 15:57 ` [PATCH 3/4] http: add http.retryStatusCodes config option Ravi Mistry via GitGitGadget
2026-11-03 13:41 ` Junio C Hamano
2026-11-03 19:06 ` Ravi Mistry
2026-10-31 15:57 ` [PATCH 4/4] remote-curl: retry transient HTTP errors in post_rpc() Ravi Mistry via GitGitGadget
2026-11-03 14:47 ` Junio C Hamano
2026-11-03 18:57 ` [PATCH v2 0/4] http: exponential backoff, configurable status codes, and post_rpc() retries Ravi Mistry via GitGitGadget
2026-11-03 18:57 ` [PATCH v2 1/4] http: fix retry count and Retry-After timing Ravi Mistry via GitGitGadget
2026-11-03 18:57 ` [PATCH v2 2/4] http: add exponential backoff without Retry-After Ravi Mistry via GitGitGadget
2026-11-04 22:05 ` brian m. carlson
2026-11-05 0:10 ` Ravi Mistry
2026-11-05 17:45 ` Junio C Hamano
2026-11-05 22:32 ` brian m. carlson
2026-11-06 16:27 ` Ravi Mistry
2026-11-03 18:57 ` [PATCH v2 3/4] http: add http.retryStatusCodes config option Ravi Mistry via GitGitGadget
2026-11-03 20:56 ` Junio C Hamano
2026-11-03 18:57 ` [PATCH v2 4/4] remote-curl: retry transient HTTP errors in post_rpc() Ravi Mistry via GitGitGadget
2026-11-07 3:40 ` [PATCH v3 0/5] http: exponential backoff, configurable status codes, and post_rpc() retries Ravi Mistry via GitGitGadget
2026-11-07 3:40 ` [PATCH v3 1/5] http: fix retry count and Retry-After timing Ravi Mistry via GitGitGadget
2026-11-18 12:00 ` Patrick Steinhardt [this message]
2026-11-07 3:40 ` [PATCH v3 2/5] http: add exponential backoff without Retry-After Ravi Mistry via GitGitGadget
2026-11-18 12:00 ` Patrick Steinhardt
2026-11-07 3:40 ` [PATCH v3 3/5] http: add http.retryStatusCodes config option Ravi Mistry via GitGitGadget
2026-11-18 12:00 ` Patrick Steinhardt
2026-11-07 3:40 ` [PATCH v3 4/5] remote-curl: retry transient HTTP errors in post_rpc() Ravi Mistry via GitGitGadget
2026-11-18 12:01 ` Patrick Steinhardt
2026-11-07 3:40 ` [PATCH v3 5/5] http: default http.retryAfter to 10 seconds Ravi Mistry via GitGitGadget
2026-11-18 12:01 ` Patrick Steinhardt
2026-11-07 4:37 ` [PATCH v3 0/5] http: exponential backoff, configurable status codes, and post_rpc() retries Junio C Hamano
2026-11-10 3:58 ` Ravi Mistry
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av2TTPCkiA-49K_I@pks.im \
--to=ps@pks.im \
--cc=git@vger.kernel.org \
--cc=gitgitgadget@gmail.com \
--cc=rmistry@google.com \
--cc=sandals@crustytoothpaste.net \
--cc=vaidas.pilkauskas@shopify.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox