All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Vaidas Pilkauskas via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: Vaidas Pilkauskas <vaidas.pilkauskas@shopify.com>,
	Vaidas Pilkauskas <vaidas.pilkauskas@shopify.com>
Subject: [PATCH 1/3] http: add support for HTTP 429 rate limit retries
Date: Wed, 26 Nov 2025 12:30:25 +0000	[thread overview]
Message-ID: <ae0087cd1c7fbb6b748d6767b476c1bd1a19996f.1764160227.git.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.2008.git.1764160227.gitgitgadget@gmail.com>

From: Vaidas Pilkauskas <vaidas.pilkauskas@shopify.com>

Add retry logic for HTTP 429 (Too Many Requests) responses to handle
server-side rate limiting gracefully. When Git's HTTP client receives
a 429 response, it can now automatically retry the request after an
appropriate delay, respecting the server's rate limits.

The implementation supports the RFC-compliant Retry-After header in
both delay-seconds (integer) and HTTP-date (RFC 2822) formats. If a
past date is provided, Git retries immediately without waiting.

Retry behavior is controlled by three new configuration options:

  * http.maxRetries: Maximum number of retry attempts (default: 0,
    meaning retries are disabled by default). Users must explicitly
    opt-in to retry behavior.

  * http.retryAfter: Default delay in seconds when the server doesn't
    provide a Retry-After header (default: -1, meaning fail if no
    header is provided). This serves as a fallback mechanism.

  * http.maxRetryTime: Maximum delay in seconds for a single retry
    (default: 300). If the server requests a delay exceeding this
    limit, Git fails immediately rather than waiting. This prevents
    indefinite blocking on unreasonable server requests.

All three options can be overridden via environment variables:
GIT_HTTP_MAX_RETRIES, GIT_HTTP_RETRY_AFTER, and
GIT_HTTP_MAX_RETRY_TIME.

The retry logic implements a fail-fast approach: if any delay
(whether from server header or configuration) exceeds maxRetryTime,
Git fails immediately with a clear error message rather than capping
the delay. This provides better visibility into rate limiting issues.

The implementation includes extensive test coverage for basic retry
behavior, Retry-After header formats (integer and HTTP-date),
configuration combinations, maxRetryTime limits, invalid header
handling, environment variable overrides, and edge cases.

Signed-off-by: Vaidas Pilkauskas <vaidas.pilkauskas@shopify.com>
---
 Documentation/config/http.adoc |  24 ++
 http-push.c                    |   8 +
 http-walker.c                  |   5 +
 http.c                         | 149 +++++++++++-
 http.h                         |   2 +
 remote-curl.c                  |   4 +
 t/meson.build                  |   1 +
 t/t5584-http-429-retry.sh      | 429 +++++++++++++++++++++++++++++++++
 8 files changed, 618 insertions(+), 4 deletions(-)
 create mode 100755 t/t5584-http-429-retry.sh

diff --git a/Documentation/config/http.adoc b/Documentation/config/http.adoc
index 9da5c298cc..9e3c888df4 100644
--- a/Documentation/config/http.adoc
+++ b/Documentation/config/http.adoc
@@ -315,6 +315,30 @@ http.keepAliveCount::
 	unset, curl's default value is used. Can be overridden by the
 	`GIT_HTTP_KEEPALIVE_COUNT` environment variable.
 
+http.retryAfter::
+	Default wait time in seconds before retrying when a server returns
+	HTTP 429 (Too Many Requests) without a Retry-After header. If set
+	to -1 (the default), Git will fail immediately when encountering
+	a 429 response without a Retry-After header. When a Retry-After
+	header is present, its value takes precedence over this setting.
+	Can be overridden by the `GIT_HTTP_RETRY_AFTER` environment variable.
+	See also `http.maxRetries` and `http.maxRetryTime`.
+
+http.maxRetries::
+	Maximum number of times to retry after receiving HTTP 429 (Too Many
+	Requests) responses. Set to 0 (the default) to disable retries.
+	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
+	See also `http.retryAfter` and `http.maxRetryTime`.
+
+http.maxRetryTime::
+	Maximum time in seconds to wait for a single retry attempt when
+	handling HTTP 429 (Too Many Requests) responses. If the server
+	requests a delay (via Retry-After header) or if `http.retryAfter`
+	is configured with a value that exceeds this maximum, Git will fail
+	immediately rather than waiting. Default is 300 seconds (5 minutes).
+	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
+	variable. See also `http.retryAfter` and `http.maxRetries`.
+
 http.noEPSV::
 	A boolean which disables using of EPSV ftp command by curl.
 	This can be helpful with some "poor" ftp servers which don't
diff --git a/http-push.c b/http-push.c
index d86ce77119..a602a302ec 100644
--- a/http-push.c
+++ b/http-push.c
@@ -716,6 +716,10 @@ static int fetch_indices(void)
 	case HTTP_MISSING_TARGET:
 		ret = 0;
 		break;
+	case HTTP_RATE_LIMITED:
+		error("rate limited by '%s', please try again later", repo->url);
+		ret = -1;
+		break;
 	default:
 		ret = -1;
 	}
@@ -1548,6 +1552,10 @@ static int remote_exists(const char *path)
 	case HTTP_MISSING_TARGET:
 		ret = 0;
 		break;
+	case HTTP_RATE_LIMITED:
+		error("rate limited by '%s', please try again later", url);
+		ret = -1;
+		break;
 	case HTTP_ERROR:
 		error("unable to access '%s': %s", url, curl_errorstr);
 		/* fallthrough */
diff --git a/http-walker.c b/http-walker.c
index e886e64866..9f06f47de1 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -414,6 +414,11 @@ static int fetch_indices(struct walker *walker, struct alt_base *repo)
 		repo->got_indices = 1;
 		ret = 0;
 		break;
+	case HTTP_RATE_LIMITED:
+		error("rate limited by '%s', please try again later", repo->base);
+		repo->got_indices = 0;
+		ret = -1;
+		break;
 	default:
 		repo->got_indices = 0;
 		ret = -1;
diff --git a/http.c b/http.c
index 41f850db16..212805cad5 100644
--- a/http.c
+++ b/http.c
@@ -22,6 +22,7 @@
 #include "object-file.h"
 #include "odb.h"
 #include "tempfile.h"
+#include "date.h"
 
 static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
 static int trace_curl_data = 1;
@@ -149,6 +150,14 @@ static char *cached_accept_language;
 static char *http_ssl_backend;
 
 static int http_schannel_check_revoke = 1;
+
+/* Retry configuration */
+static long http_retry_after = -1; /* Default retry-after in seconds when header is missing (-1 means not set, exit with 128) */
+static long http_max_retries = 0; /* Maximum number of retry attempts (0 means retries are disabled) */
+static long http_max_retry_time = 300; /* Maximum time to wait for a single retry (default 5 minutes) */
+
+/* Store retry_after value from 429 responses for retry logic (-1 = not set, 0 = retry immediately, >0 = delay in seconds) */
+static long last_retry_after = -1;
 /*
  * With the backend being set to `schannel`, setting sslCAinfo would override
  * the Certificate Store in cURL v7.60.0 and later, which is not what we want
@@ -209,13 +218,14 @@ static inline int is_hdr_continuation(const char *ptr, const size_t size)
 	return size && (*ptr == ' ' || *ptr == '\t');
 }
 
-static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p UNUSED)
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
 {
 	size_t size = eltsize * nmemb;
 	struct strvec *values = &http_auth.wwwauth_headers;
 	struct strbuf buf = STRBUF_INIT;
 	const char *val;
 	size_t val_len;
+	struct active_request_slot *slot = (struct active_request_slot *)p;
 
 	/*
 	 * Header lines may not come NULL-terminated from libcurl so we must
@@ -257,6 +267,47 @@ static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p UN
 		goto exit;
 	}
 
+	/* Parse Retry-After header for rate limiting */
+	if (skip_iprefix_mem(ptr, size, "retry-after:", &val, &val_len)) {
+		strbuf_add(&buf, val, val_len);
+		strbuf_trim(&buf);
+
+		if (slot && slot->results) {
+			/* Parse the retry-after value (delay-seconds or HTTP-date) */
+			char *endptr;
+			long retry_after;
+
+			errno = 0;
+			retry_after = strtol(buf.buf, &endptr, 10);
+
+			/* Check if it's a valid integer (delay-seconds format) */
+			if (endptr != buf.buf && *endptr == '\0' &&
+			    errno != ERANGE && retry_after > 0) {
+				slot->results->retry_after = retry_after;
+			} else {
+				/* Try parsing as HTTP-date format */
+				timestamp_t timestamp;
+				int offset;
+				if (!parse_date_basic(buf.buf, &timestamp, &offset)) {
+					/* Successfully parsed as date, calculate delay from now */
+					timestamp_t now = time(NULL);
+					if (timestamp > now) {
+						slot->results->retry_after = (long)(timestamp - now);
+					} else {
+						/* Past date means retry immediately */
+						slot->results->retry_after = 0;
+					}
+				} else {
+					/* Failed to parse as either delay-seconds or HTTP-date */
+					warning(_("unable to parse Retry-After header value: '%s'"), buf.buf);
+				}
+			}
+		}
+
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
 	/*
 	 * This line could be a continuation of the previously matched header
 	 * field. If this is the case then we should append this value to the
@@ -575,6 +626,21 @@ static int http_options(const char *var, const char *value,
 		return 0;
 	}
 
+	if (!strcmp("http.retryafter", var)) {
+		http_retry_after = git_config_int(var, value, ctx->kvi);
+		return 0;
+	}
+
+	if (!strcmp("http.maxretries", var)) {
+		http_max_retries = git_config_int(var, value, ctx->kvi);
+		return 0;
+	}
+
+	if (!strcmp("http.maxretrytime", var)) {
+		http_max_retry_time = git_config_int(var, value, ctx->kvi);
+		return 0;
+	}
+
 	/* Fall back on the default ones */
 	return git_default_config(var, value, ctx, data);
 }
@@ -1422,6 +1488,10 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
 	set_long_from_env(&curl_tcp_keepintvl, "GIT_TCP_KEEPINTVL");
 	set_long_from_env(&curl_tcp_keepcnt, "GIT_TCP_KEEPCNT");
 
+	set_long_from_env(&http_retry_after, "GIT_HTTP_RETRY_AFTER");
+	set_long_from_env(&http_max_retries, "GIT_HTTP_MAX_RETRIES");
+	set_long_from_env(&http_max_retry_time, "GIT_HTTP_MAX_RETRY_TIME");
+
 	curl_default = get_curl_handle();
 }
 
@@ -1871,6 +1941,10 @@ static int handle_curl_result(struct slot_results *results)
 			}
 			return HTTP_REAUTH;
 		}
+	} else if (results->http_code == 429) {
+		/* Store the retry_after value for use in retry logic */
+		last_retry_after = results->retry_after;
+		return HTTP_RATE_LIMITED;
 	} else {
 		if (results->http_connectcode == 407)
 			credential_reject(the_repository, &proxy_auth);
@@ -1886,6 +1960,8 @@ int run_one_slot(struct active_request_slot *slot,
 		 struct slot_results *results)
 {
 	slot->results = results;
+	/* Initialize retry_after to -1 (not set) */
+	results->retry_after = -1;
 	if (!start_active_slot(slot)) {
 		xsnprintf(curl_errorstr, sizeof(curl_errorstr),
 			  "failed to start HTTP request");
@@ -2149,6 +2225,7 @@ static int http_request(const char *url,
 	}
 
 	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERDATA, slot);
 
 	accept_language = http_get_accept_language_header();
 
@@ -2253,19 +2330,36 @@ static int update_url_from_redirect(struct strbuf *base,
 	return 1;
 }
 
+/*
+ * Sleep for the specified number of seconds before retrying.
+ */
+static void sleep_for_retry(long retry_after)
+{
+	if (retry_after > 0) {
+		unsigned int remaining;
+		warning(_("rate limited, waiting %ld seconds before retry"), retry_after);
+		remaining = sleep(retry_after);
+		while (remaining > 0) {
+			/* Sleep was interrupted, continue sleeping */
+			remaining = sleep(remaining);
+		}
+	}
+}
+
 static int http_request_reauth(const char *url,
 			       void *result, int target,
 			       struct http_get_options *options)
 {
 	int i = 3;
 	int ret;
+	int rate_limit_retries = http_max_retries;
 
 	if (always_auth_proactively())
 		credential_fill(the_repository, &http_auth, 1);
 
 	ret = http_request(url, result, target, options);
 
-	if (ret != HTTP_OK && ret != HTTP_REAUTH)
+	if (ret != HTTP_OK && ret != HTTP_REAUTH && ret != HTTP_RATE_LIMITED)
 		return ret;
 
 	if (options && options->effective_url && options->base_url) {
@@ -2276,7 +2370,7 @@ static int http_request_reauth(const char *url,
 		}
 	}
 
-	while (ret == HTTP_REAUTH && --i) {
+	while ((ret == HTTP_REAUTH || ret == HTTP_RATE_LIMITED) && --i) {
 		/*
 		 * The previous request may have put cruft into our output stream; we
 		 * should clear it out before making our next request.
@@ -2302,7 +2396,54 @@ static int http_request_reauth(const char *url,
 			BUG("Unknown http_request target");
 		}
 
-		credential_fill(the_repository, &http_auth, 1);
+		if (ret == HTTP_RATE_LIMITED) {
+			/* Handle rate limiting with retry logic */
+			int retry_attempt = http_max_retries - rate_limit_retries + 1;
+
+			if (rate_limit_retries <= 0) {
+				/* Retries are disabled or exhausted */
+				if (http_max_retries > 0) {
+					error(_("too many rate limit retries, giving up"));
+				}
+				return HTTP_ERROR;
+			}
+
+			/* Decrement retries counter */
+			rate_limit_retries--;
+
+			/* Use the stored retry_after value or configured default */
+			if (last_retry_after >= 0) {
+				/* Check if retry delay exceeds maximum allowed */
+				if (last_retry_after > http_max_retry_time) {
+					error(_("rate limited (HTTP 429) requested %ld second delay, "
+						"exceeds http.maxRetryTime of %ld seconds"),
+					      last_retry_after, http_max_retry_time);
+					last_retry_after = -1; /* Reset after use */
+					return HTTP_ERROR;
+				}
+				sleep_for_retry(last_retry_after);
+				last_retry_after = -1; /* Reset after use */
+			} else {
+				/* No Retry-After header provided */
+				if (http_retry_after < 0) {
+					/* Not configured - exit with error */
+					error(_("rate limited (HTTP 429) and no Retry-After header provided. "
+						"Configure http.retryAfter or set GIT_HTTP_RETRY_AFTER."));
+					return HTTP_ERROR;
+				}
+				/* Check if configured default exceeds maximum allowed */
+				if (http_retry_after > http_max_retry_time) {
+					error(_("configured http.retryAfter (%ld seconds) exceeds "
+						"http.maxRetryTime (%ld seconds)"),
+					      http_retry_after, http_max_retry_time);
+					return HTTP_ERROR;
+				}
+				/* Use configured default retry-after value */
+				sleep_for_retry(http_retry_after);
+			}
+		} else if (ret == HTTP_REAUTH) {
+			credential_fill(the_repository, &http_auth, 1);
+		}
 
 		ret = http_request(url, result, target, options);
 	}
diff --git a/http.h b/http.h
index f9d4593404..eb40456450 100644
--- a/http.h
+++ b/http.h
@@ -20,6 +20,7 @@ struct slot_results {
 	long http_code;
 	long auth_avail;
 	long http_connectcode;
+	long retry_after;
 };
 
 struct active_request_slot {
@@ -167,6 +168,7 @@ struct http_get_options {
 #define HTTP_REAUTH	4
 #define HTTP_NOAUTH	5
 #define HTTP_NOMATCHPUBLICKEY	6
+#define HTTP_RATE_LIMITED	7
 
 /*
  * Requests a URL and stores the result in a strbuf.
diff --git a/remote-curl.c b/remote-curl.c
index 69f919454a..5959461cd3 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -529,6 +529,10 @@ static struct discovery *discover_refs(const char *service, int for_push)
 		show_http_message(&type, &charset, &buffer);
 		die(_("unable to access '%s' with http.pinnedPubkey configuration: %s"),
 		    transport_anonymize_url(url.buf), curl_errorstr);
+	case HTTP_RATE_LIMITED:
+		show_http_message(&type, &charset, &buffer);
+		die(_("rate limited by '%s', please try again later"),
+		    transport_anonymize_url(url.buf));
 	default:
 		show_http_message(&type, &charset, &buffer);
 		die(_("unable to access '%s': %s"),
diff --git a/t/meson.build b/t/meson.build
index dc43d69636..98bd6949e6 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -698,6 +698,7 @@ integration_tests = [
   't5581-http-curl-verbose.sh',
   't5582-fetch-negative-refspec.sh',
   't5583-push-branches.sh',
+  't5584-http-429-retry.sh',
   't5600-clone-fail-cleanup.sh',
   't5601-clone.sh',
   't5602-clone-remote-exec.sh',
diff --git a/t/t5584-http-429-retry.sh b/t/t5584-http-429-retry.sh
new file mode 100755
index 0000000000..8bcc382763
--- /dev/null
+++ b/t/t5584-http-429-retry.sh
@@ -0,0 +1,429 @@
+#!/bin/sh
+
+test_description='test HTTP 429 Too Many Requests retry logic'
+
+. ./test-lib.sh
+
+. "$TEST_DIRECTORY"/lib-httpd.sh
+
+start_httpd
+
+test_expect_success 'setup test repository' '
+	test_commit initial &&
+	git clone --bare . "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git --git-dir="$HTTPD_DOCUMENT_ROOT_PATH/repo.git" config http.receivepack true
+'
+
+test_expect_success 'HTTP 429 with retries disabled (maxRetries=0) fails immediately' '
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: 1\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Set maxRetries to 0 (disabled)
+	test_config http.maxRetries 0 &&
+	test_config http.retryAfter 1 &&
+
+	# Should fail immediately without any retry attempt
+	test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+
+	# Verify no retry happened (no "waiting" message in stderr)
+	! grep -i "waiting.*retry" err &&
+
+	# The one-time script will be consumed on first request (not a retry)
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 permanent should fail after max retries' '
+	# Install a permanent error script to prove retries are limited
+	write_script "$HTTPD_ROOT_PATH/http-429-permanent.sh" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: 1\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Permanently rate limited\n"
+	EOF
+
+	# Enable retries with a limit
+	test_config http.maxRetries 2 &&
+
+	# Git should retry but eventually fail when 429 persists
+	test_must_fail git ls-remote "$HTTPD_URL/error/http-429-permanent.sh/repo.git" 2>err
+'
+
+test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '
+	# Create a one-time script that returns 429 with Retry-After header
+	# on the first request. Subsequent requests will succeed.
+	# This contrasts with the permanent 429 above - proving retry works
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	# Return HTTP 429 response instead of git response
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: 1\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited - please retry after 1 second\n"
+	# Output something different from input so the script gets removed
+	cat "$1" >/dev/null
+	EOF
+
+	# Enable retries
+	test_config http.maxRetries 3 &&
+
+	# Git should retry after receiving 429 and eventually succeed
+	git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+
+	# The one-time script should have been consumed (proving retry happened)
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 without Retry-After uses configured default' '
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited - no retry info\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Enable retries and configure default delay
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+
+	# Git should retry using configured default and succeed
+	git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 retry delays are respected' '
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: 2\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Enable retries
+	test_config http.maxRetries 3 &&
+
+	# Time the operation - it should take at least 2 seconds due to retry delay
+	start=$(date +%s) &&
+	git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Verify it took at least 2 seconds (allowing some tolerance)
+	test "$duration" -ge 1 &&
+	test_grep "refs/heads/" output &&
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 fails immediately if Retry-After exceeds http.maxRetryTime' '
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: 100\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited with long delay\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Configure max retry time to 3 seconds (much less than requested 100)
+	test_config http.maxRetries 3 &&
+	test_config http.maxRetryTime 3 &&
+
+	# Should fail immediately without waiting
+	start=$(date +%s) &&
+	test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Should fail quickly (less than 2 seconds, no 100 second wait)
+	test "$duration" -lt 2 &&
+	test_grep "exceeds http.maxRetryTime" err &&
+
+	# The one-time script will be consumed on first request
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 fails if configured http.retryAfter exceeds http.maxRetryTime' '
+	# Test misconfiguration: retryAfter > maxRetryTime
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited without header\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Configure retryAfter larger than maxRetryTime
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 100 &&
+	test_config http.maxRetryTime 5 &&
+
+	# Should fail immediately with configuration error
+	start=$(date +%s) &&
+	test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Should fail quickly
+	test "$duration" -lt 2 &&
+	test_grep "configured http.retryAfter.*exceeds.*http.maxRetryTime" err
+'
+
+test_expect_success 'HTTP 429 with Retry-After HTTP-date format' '
+	# Test HTTP-date format (RFC 2822) in Retry-After header
+	# Generate a date 2 seconds in the future
+	future_date=$(TZ=GMT date -d "+2 seconds" "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+		      TZ=GMT date -v+2S "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+		      echo "skip") &&
+
+	if test "$future_date" = "skip"
+	then
+		skip_all="date command does not support required format" &&
+		test_done
+	fi &&
+
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-EOF &&
+	printf "Status: 429 Too Many Requests\\r\\n"
+	printf "Retry-After: $future_date\\r\\n"
+	printf "Content-Type: text/plain\\r\\n"
+	printf "\\r\\n"
+	printf "Rate limited with HTTP-date\\n"
+	cat "\$1" >/dev/null
+	EOF
+
+	# Enable retries
+	test_config http.maxRetries 3 &&
+
+	# Git should parse the HTTP-date and retry after the delay
+	start=$(date +%s) &&
+	git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Should take at least 1 second (allowing tolerance for processing time)
+	test "$duration" -ge 1 &&
+	test_grep "refs/heads/" output &&
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 with HTTP-date exceeding maxRetryTime fails immediately' '
+	# Generate a date 200 seconds in the future
+	future_date=$(TZ=GMT date -d "+200 seconds" "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+		      TZ=GMT date -v+200S "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+		      echo "skip") &&
+
+	if test "$future_date" = "skip"
+	then
+		skip_all="date command does not support required format" &&
+		test_done
+	fi &&
+
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-EOF &&
+	printf "Status: 429 Too Many Requests\\r\\n"
+	printf "Retry-After: $future_date\\r\\n"
+	printf "Content-Type: text/plain\\r\\n"
+	printf "\\r\\n"
+	printf "Rate limited with long HTTP-date\\n"
+	cat "\$1" >/dev/null
+	EOF
+
+	# Configure max retry time much less than the 200 second delay
+	test_config http.maxRetries 3 &&
+	test_config http.maxRetryTime 10 &&
+
+	# Should fail immediately without waiting 200 seconds
+	start=$(date +%s) &&
+	test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Should fail quickly (not wait 200 seconds)
+	test "$duration" -lt 2 &&
+	test_grep "exceeds http.maxRetryTime" err &&
+
+	# The one-time script will be consumed on first request
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 with past HTTP-date should not wait' '
+	past_date=$(TZ=GMT date -d "-10 seconds" "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+		    TZ=GMT date -v-10S "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+		    echo "skip") &&
+
+	if test "$past_date" = "skip"
+	then
+		skip_all="date command does not support required format" &&
+		test_done
+	fi &&
+
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-EOF &&
+	printf "Status: 429 Too Many Requests\\r\\n"
+	printf "Retry-After: $past_date\\r\\n"
+	printf "Content-Type: text/plain\\r\\n"
+	printf "\\r\\n"
+	printf "Rate limited with past date\\n"
+	cat "\$1" >/dev/null
+	EOF
+
+	# Enable retries
+	test_config http.maxRetries 3 &&
+
+	# Git should retry immediately without waiting
+	start=$(date +%s) &&
+	git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Should complete quickly (less than 2 seconds)
+	test "$duration" -lt 2 &&
+	test_grep "refs/heads/" output &&
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 with invalid Retry-After format uses configured default' '
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: invalid-format-123abc\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited with malformed header\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Configure default retry-after
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+
+	# Should use configured default (1 second) since header is invalid
+	start=$(date +%s) &&
+	git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Should take at least 1 second (the configured default)
+	test "$duration" -ge 1 &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting.*retry" err &&
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 will not be retried without config' '
+	# Default config means http.maxRetries=0 (retries disabled)
+	# When 429 is received, it should fail immediately without retry
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: 1\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Do NOT configure anything - use defaults (http.maxRetries defaults to 0)
+
+	# Should fail immediately without retry
+	test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+
+	# Verify no retry happened (no "waiting" message)
+	! grep -i "waiting.*retry" err &&
+
+	# Should get 429 error
+	test_grep "429" err &&
+
+	# The one-time script should be consumed on first request
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'GIT_HTTP_RETRY_AFTER overrides http.retryAfter config' '
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited - no Retry-After header\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Configure retryAfter to 10 seconds
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 10 &&
+
+	# Override with environment variable to 1 second
+	start=$(date +%s) &&
+	GIT_HTTP_RETRY_AFTER=1 git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Should use env var (1 second), not config (10 seconds)
+	test "$duration" -ge 1 &&
+	test "$duration" -lt 5 &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting.*retry" err &&
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'GIT_HTTP_MAX_RETRIES overrides http.maxRetries config' '
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: 1\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Configure maxRetries to 0 (disabled)
+	test_config http.maxRetries 0 &&
+	test_config http.retryAfter 1 &&
+
+	# Override with environment variable to enable retries
+	GIT_HTTP_MAX_RETRIES=3 git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+
+	# Should retry (env var enables it despite config saying disabled)
+	test_grep "refs/heads/" output &&
+	test_grep "waiting.*retry" err &&
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'GIT_HTTP_MAX_RETRY_TIME overrides http.maxRetryTime config' '
+	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+	printf "Status: 429 Too Many Requests\r\n"
+	printf "Retry-After: 50\r\n"
+	printf "Content-Type: text/plain\r\n"
+	printf "\r\n"
+	printf "Rate limited with long delay\n"
+	cat "$1" >/dev/null
+	EOF
+
+	# Configure maxRetryTime to 100 seconds (would accept 50 second delay)
+	test_config http.maxRetries 3 &&
+	test_config http.maxRetryTime 100 &&
+
+	# Override with environment variable to 10 seconds (should reject 50 second delay)
+	start=$(date +%s) &&
+	test_must_fail env GIT_HTTP_MAX_RETRY_TIME=10 \
+		git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+	end=$(date +%s) &&
+	duration=$((end - start)) &&
+
+	# Should fail quickly (not wait 50 seconds) because env var limits to 10
+	test "$duration" -lt 5 &&
+	test_grep "exceeds http.maxRetryTime" err &&
+	test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'verify normal repository access still works' '
+	git ls-remote "$HTTPD_URL/smart/repo.git" >output &&
+	test_grep "refs/heads/" output
+'
+
+test_done
-- 
gitgitgadget


  reply	other threads:[~2025-11-26 12:30 UTC|newest]

Thread overview: 49+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-11-26 12:30 [PATCH 0/3] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2025-11-26 12:30 ` Vaidas Pilkauskas via GitGitGadget [this message]
2025-12-09 23:15   ` [PATCH 1/3] " Taylor Blau
2025-12-12 12:36     ` Vaidas Pilkauskas
2025-11-26 12:30 ` [PATCH 2/3] remote-curl: fix memory leak in show_http_message() Vaidas Pilkauskas via GitGitGadget
2025-12-09 23:52   ` Taylor Blau
2025-11-26 12:30 ` [PATCH 3/3] http: add trace2 logging for retry operations Vaidas Pilkauskas via GitGitGadget
2025-12-18 14:44 ` [PATCH v2 0/2] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2025-12-18 14:44   ` [PATCH v2 1/2] " Vaidas Pilkauskas via GitGitGadget
2026-02-11  1:05     ` Taylor Blau
2026-02-11  9:13       ` Jeff King
2026-02-13 13:41         ` Vaidas Pilkauskas
2026-02-15  9:13           ` Jeff King
2026-02-13 13:30       ` Vaidas Pilkauskas
2025-12-18 14:44   ` [PATCH v2 2/2] http: add trace2 logging for retry operations Vaidas Pilkauskas via GitGitGadget
2026-02-11  1:06     ` Taylor Blau
2026-02-17 11:08   ` [PATCH v3 0/3] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-02-17 11:08     ` [PATCH v3 1/3] strbuf: fix incorrect alloc size in strbuf_reencode() Vaidas Pilkauskas via GitGitGadget
2026-02-17 20:51       ` Junio C Hamano
2026-02-18 13:43         ` Vaidas Pilkauskas
2026-02-17 11:08     ` [PATCH v3 2/3] remote-curl: introduce show_http_message_fatal() helper Vaidas Pilkauskas via GitGitGadget
2026-02-17 11:08     ` [PATCH v3 3/3] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09     ` [PATCH v4 0/5] " Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09       ` [PATCH v4 1/5] strbuf: pass correct alloc to strbuf_attach() in strbuf_reencode() Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09       ` [PATCH v4 2/5] strbuf_attach: fix all call sites to pass correct alloc Vaidas Pilkauskas via GitGitGadget
2026-02-20 22:55         ` Junio C Hamano
2026-02-23 12:49           ` Vaidas Pilkauskas
2026-02-18 14:09       ` [PATCH v4 3/5] strbuf: replace strbuf_grow() in strbuf_attach() with BUG() check Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09       ` [PATCH v4 4/5] remote-curl: introduce show_http_message_fatal() helper Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09       ` [PATCH v4 5/5] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-02-23 14:20       ` [PATCH v5 0/4] " Vaidas Pilkauskas via GitGitGadget
2026-02-23 14:20         ` [PATCH v5 1/4] strbuf: pass correct alloc to strbuf_attach() in strbuf_reencode() Vaidas Pilkauskas via GitGitGadget
2026-02-23 14:20         ` [PATCH v5 2/4] strbuf_attach: fix call sites to pass correct alloc Vaidas Pilkauskas via GitGitGadget
2026-02-23 14:20         ` [PATCH v5 3/4] remote-curl: introduce show_http_message_fatal() helper Vaidas Pilkauskas via GitGitGadget
2026-03-10 17:44           ` Jeff King
2026-02-23 14:20         ` [PATCH v5 4/4] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-03-10 19:07           ` Jeff King
2026-02-24  0:07         ` [PATCH v5 0/4] " Junio C Hamano
2026-03-09 23:34           ` Junio C Hamano
2026-03-10 19:10             ` Jeff King
2026-03-10 19:19               ` Junio C Hamano
2026-03-17 13:00         ` [PATCH v6 0/3] " Vaidas Pilkauskas via GitGitGadget
2026-03-17 13:00           ` [PATCH v6 1/3] strbuf: pass correct alloc to strbuf_attach() in strbuf_reencode() Vaidas Pilkauskas via GitGitGadget
2026-03-17 13:00           ` [PATCH v6 2/3] strbuf_attach: fix call sites to pass correct alloc Vaidas Pilkauskas via GitGitGadget
2026-03-17 13:00           ` [PATCH v6 3/3] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-03-21  3:30             ` Taylor Blau
2026-03-21  3:31           ` [PATCH v6 0/3] " Taylor Blau
2026-03-21  4:57             ` Junio C Hamano
2026-03-23  6:58             ` Vaidas Pilkauskas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ae0087cd1c7fbb6b748d6767b476c1bd1a19996f.1764160227.git.gitgitgadget@gmail.com \
    --to=gitgitgadget@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=vaidas.pilkauskas@shopify.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.