[PATCH 0/3] maintenance: configure credentials to be silent

[PATCH 0/3] maintenance: configure credentials to be silent

[Derrick Stolee via GitGitGadget]

When background maintenance attempts to perform a prefetch to remote
servers, this may trigger authentication requirements. If the credentials
are expired, then the credential helper may need user input in order to get
refreshed credentials. It is not a good experience for users to get
credential pop-ups when not directly interacting with Git.

Add a new configuration value, 'credential.interactive', to specify to the
credential helper that it should not prompt for user interaction. This
option has been respected by Git Credential Manager since 2020 [1], so this
is now presenting it as an official Git config value.

These changes were first merged into the microsoft/git fork in August 2023
[2] but were not upstreamed immediately. The change has been a positive one
for users of that fork, as they no longer get pop-ups and they also are not
getting maintenance.lock file blocks when the prefetch task waits for
credentials. This has become even more important recently as credential
lifetimes have been restricted significantly, leading to a higher likelihood
that this will happen during a background prefetch.

I was reminded of these changes when liuzhongbo started a discussion [3]
about maintenance.lock files and requesting that they are removed if they
are stale. This does not address that issue directly, but is an important
way to reduce the lifetime of maintenance.lock files when blocked on
credential prompts.

[1] https://github.com/git-ecosystem/git-credential-manager/pull/91

[2] https://github.com/microsoft/git/pull/598

[3]
https://lore.kernel.org/git/cce1d054-911e-407e-bc26-1c0bac4dd8e4@gmail.com/T/#t

Thanks, -Stolee

Derrick Stolee (3):
  credential: add new interactive config option
  maintenance: add custom config to background jobs
  scalar: configure maintenance during 'reconfigure'

 Documentation/config/credential.txt |  8 +++++
 builtin/gc.c                        | 53 +++++++++++++++++++++++++----
 credential.c                        | 30 ++++++++++++++--
 scalar.c                            |  3 ++
 t/t5551-http-fetch-smart.sh         | 22 ++++++++++++
 t/t7900-maintenance.sh              |  3 ++
 t/t9210-scalar.sh                   |  7 ++--
 7 files changed, 114 insertions(+), 12 deletions(-)


base-commit: 6531f31ef3bead57a3255fa08efa6e7553c5a9a7
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1798%2Fderrickstolee%2Fbackground-quiet-credentials-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1798/derrickstolee/background-quiet-credentials-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1798
-- 
gitgitgadget

[PATCH 1/3] credential: add new interactive config option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When scripts or background maintenance wish to perform HTTP(S) requests,
there is a risk that our stored credentials might be invalid. At the
moment, this causes the credential helper to ping the user and block the
process. Even if the credential helper does not ping the user, Git falls
back to the 'askpass' method, which includes a direct ping to the user
via the terminal.

Even setting the 'core.askPass' config as something like 'echo' will
causes Git to fallback to a terminal prompt. It uses
git_terminal_prompt(), which finds the terminal from the environment and
ignores whether stdin has been redirected. This can also block the
process awaiting input.

Create a new config option to prevent user interaction, favoring a
failure to a blocked process.

The chosen name, 'credential.interactive', is taken from the config
option used by Git Credential Manager to already avoid user
interactivity, so there is already one credential helper that integrates
with this option. However, older versions of Git Credential Manager also
accepted other string values, including 'auto', 'never', and 'always'.
The modern use is to use a boolean value, but we should still be
careful that some users could have these non-booleans. Further, we
should respect 'never' the same as 'false'. This is respected by the
implementation and test, but not mentioned in the documentation.

The implementation for the Git interactions takes place within
credential_getpass(). The method prototype is modified to return an
'int' instead of 'void'. This allows us to detect that no attempt was
made to fill the given credential, changing the single caller slightly.

Also, a new trace2 region is added around the interactive portion of the
credential request. This provides a way to measure the amount of time
spent in that region for commands that _are_ interactive. It also makes
a conventient way to test that the config option works with
'test_region'.

Signed-off-by: Derrick Stolee <stolee@gmail.com>
---
 Documentation/config/credential.txt |  8 ++++++++
 credential.c                        | 30 ++++++++++++++++++++++++++---
 t/t5551-http-fetch-smart.sh         | 22 +++++++++++++++++++++
 3 files changed, 57 insertions(+), 3 deletions(-)

diff --git a/Documentation/config/credential.txt b/Documentation/config/credential.txt
index 0221c3e620d..470482ff4c2 100644
--- a/Documentation/config/credential.txt
+++ b/Documentation/config/credential.txt
@@ -9,6 +9,14 @@ credential.helper::
 Note that multiple helpers may be defined. See linkgit:gitcredentials[7]
 for details and examples.
 
+credential.interactive::
+	By default, Git and any configured credential helpers will ask for
+	user input when new credentials are required. Many of these helpers
+	will succeed based on stored credentials if those credentials are
+	still valid. To avoid the possibility of user interactivity from
+	Git, set `credential.interactive=false`. Some credential helpers
+	respect this option as well.
+
 credential.useHttpPath::
 	When acquiring credentials, consider the "path" component of an http
 	or https URL to be important. Defaults to false. See
diff --git a/credential.c b/credential.c
index ee46351ce01..6dea3859ece 100644
--- a/credential.c
+++ b/credential.c
@@ -13,6 +13,8 @@
 #include "strbuf.h"
 #include "urlmatch.h"
 #include "git-compat-util.h"
+#include "trace2.h"
+#include "repository.h"
 
 void credential_init(struct credential *c)
 {
@@ -251,14 +253,36 @@ static char *credential_ask_one(const char *what, struct credential *c,
 	return xstrdup(r);
 }
 
-static void credential_getpass(struct credential *c)
+static int credential_getpass(struct credential *c)
 {
+	int interactive;
+	char *value;
+	if (!git_config_get_maybe_bool("credential.interactive", &interactive) &&
+	    !interactive) {
+		trace2_data_intmax("credential", the_repository,
+				   "interactive/skipped", 1);
+		return -1;
+	}
+	if (!git_config_get_string("credential.interactive", &value)) {
+		int same = !strcmp(value, "never");
+		free(value);
+		if (same) {
+			trace2_data_intmax("credential", the_repository,
+					   "interactive/skipped", 1);
+			return -1;
+		}
+	}
+
+	trace2_region_enter("credential", "interactive", the_repository);
 	if (!c->username)
 		c->username = credential_ask_one("Username", c,
 						 PROMPT_ASKPASS|PROMPT_ECHO);
 	if (!c->password)
 		c->password = credential_ask_one("Password", c,
 						 PROMPT_ASKPASS);
+	trace2_region_leave("credential", "interactive", the_repository);
+
+	return 0;
 }
 
 int credential_has_capability(const struct credential_capability *capa,
@@ -501,8 +525,8 @@ void credential_fill(struct credential *c, int all_capabilities)
 			    c->helpers.items[i].string);
 	}
 
-	credential_getpass(c);
-	if (!c->username && !c->password && !c->credential)
+	if (credential_getpass(c) ||
+	    (!c->username && !c->password && !c->credential))
 		die("unable to get password from user");
 }
 
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 7b5ab0eae16..ceb3336a5c4 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -186,6 +186,28 @@ test_expect_success 'clone from password-protected repository' '
 	test_cmp expect actual
 '
 
+test_expect_success 'credential.interactive=false skips askpass' '
+	set_askpass bogus nonsense &&
+	(
+		GIT_TRACE2_EVENT="$(pwd)/interactive-true" &&
+		export GIT_TRACE2_EVENT &&
+		test_must_fail git clone --bare "$HTTPD_URL/auth/smart/repo.git" interactive-true-dir &&
+		test_region credential interactive interactive-true &&
+
+		GIT_TRACE2_EVENT="$(pwd)/interactive-false" &&
+		export GIT_TRACE2_EVENT &&
+		test_must_fail git -c credential.interactive=false \
+			clone --bare "$HTTPD_URL/auth/smart/repo.git" interactive-false-dir &&
+		test_region ! credential interactive interactive-false &&
+
+		GIT_TRACE2_EVENT="$(pwd)/interactive-never" &&
+		export GIT_TRACE2_EVENT &&
+		test_must_fail git -c credential.interactive=never \
+			clone --bare "$HTTPD_URL/auth/smart/repo.git" interactive-never-dir &&
+		test_region ! credential interactive interactive-never
+	)
+'
+
 test_expect_success 'clone from auth-only-for-push repository' '
 	echo two >expect &&
 	set_askpass wrong &&
-- 
gitgitgadget

Re: [PATCH 1/3] credential: add new interactive config option

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> @@ -501,8 +525,8 @@ void credential_fill(struct credential *c, int all_capabilities)
>  			    c->helpers.items[i].string);
>  	}
>  
> -	credential_getpass(c);
> -	if (!c->username && !c->password && !c->credential)
> +	if (credential_getpass(c) ||
> +	    (!c->username && !c->password && !c->credential))
>  		die("unable to get password from user");
>  }

This is a fallback mode after credential helpers have failed to fill
and return.  Unless these helpers pay attention to the "interactive"
configuration, they may still get stuck.  So it would be #leftoverbits
to update each credential helpers to do the right thing.

The sample credential-store backend does not have to be updated I
guess ;-)

> diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
> index 7b5ab0eae16..ceb3336a5c4 100755
> --- a/t/t5551-http-fetch-smart.sh
> +++ b/t/t5551-http-fetch-smart.sh
> @@ -186,6 +186,28 @@ test_expect_success 'clone from password-protected repository' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 'credential.interactive=false skips askpass' '
> +	set_askpass bogus nonsense &&
> +	(
> +		GIT_TRACE2_EVENT="$(pwd)/interactive-true" &&
> +		export GIT_TRACE2_EVENT &&
> +		test_must_fail git clone --bare "$HTTPD_URL/auth/smart/repo.git" interactive-true-dir &&
> +		test_region credential interactive interactive-true &&
> +
> +		GIT_TRACE2_EVENT="$(pwd)/interactive-false" &&
> +		export GIT_TRACE2_EVENT &&
> +		test_must_fail git -c credential.interactive=false \
> +			clone --bare "$HTTPD_URL/auth/smart/repo.git" interactive-false-dir &&
> +		test_region ! credential interactive interactive-false &&
> +
> +		GIT_TRACE2_EVENT="$(pwd)/interactive-never" &&
> +		export GIT_TRACE2_EVENT &&
> +		test_must_fail git -c credential.interactive=never \
> +			clone --bare "$HTTPD_URL/auth/smart/repo.git" interactive-never-dir &&
> +		test_region ! credential interactive interactive-never
> +	)
> +'
> +
>  test_expect_success 'clone from auth-only-for-push repository' '
>  	echo two >expect &&
>  	set_askpass wrong &&

[PATCH 2/3] maintenance: add custom config to background jobs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

At the moment, some background jobs are getting blocked on credentials
during the 'prefetch' task. This leads to other tasks, such as
incremental repacks, getting blocked. Further, if a user manages to fix
their credentials, then they still need to cancel the background process
before their background maintenance can continue working.

Update the background schedules for our four scheduler integrations to
include these config options via '-c' options:

 * 'credential.interactive=false' will stop Git and some credential
   helpers from prompting in the UI (assuming the '-c' parameters are
   carried through and respected by GCM).

 * 'core.askPass=true' will replace the text fallback for a username
   and password into the 'true' command, which will return a success in
   its exit code, but Git will treat the empty string returned as an
   invalid password and move on.

We can do some testing that the credentials are passed, at least in the
systemd case due to writing the service files.

Signed-off-by: Derrick Stolee <stolee@gmail.com>
---
 builtin/gc.c           | 53 ++++++++++++++++++++++++++++++++++++------
 t/t7900-maintenance.sh |  3 +++
 2 files changed, 49 insertions(+), 7 deletions(-)

diff --git a/builtin/gc.c b/builtin/gc.c
index 7dac9714054..fb1be542e06 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -1766,6 +1766,42 @@ static const char *get_frequency(enum schedule_priority schedule)
 	}
 }
 
+static const char *extraconfig[] = {
+	"credential.interactive=false",
+	"core.askPass=true", /* 'true' returns success, but no output. */
+	NULL
+};
+
+static const char *get_extra_config_parameters(void) {
+	static const char *result = NULL;
+	struct strbuf builder = STRBUF_INIT;
+
+	if (result)
+		return result;
+
+	for (const char **s = extraconfig; s && *s; s++)
+		strbuf_addf(&builder, "-c %s ", *s);
+
+	result = strbuf_detach(&builder, NULL);
+	return result;
+}
+
+static const char *get_extra_launchctl_strings(void) {
+	static const char *result = NULL;
+	struct strbuf builder = STRBUF_INIT;
+
+	if (result)
+		return result;
+
+	for (const char **s = extraconfig; s && *s; s++) {
+		strbuf_addstr(&builder, "<string>-c</string>\n");
+		strbuf_addf(&builder, "<string>%s</string>\n", *s);
+	}
+
+	result = strbuf_detach(&builder, NULL);
+	return result;
+}
+
 /*
  * get_schedule_cmd` reads the GIT_TEST_MAINT_SCHEDULER environment variable
  * to mock the schedulers that `git maintenance start` rely on.
@@ -1972,6 +2008,7 @@ static int launchctl_schedule_plist(const char *exec_path, enum schedule_priorit
 		   "<array>\n"
 		   "<string>%s/git</string>\n"
 		   "<string>--exec-path=%s</string>\n"
+		   "%s" /* For extra config parameters. */
 		   "<string>for-each-repo</string>\n"
 		   "<string>--keep-going</string>\n"
 		   "<string>--config=maintenance.repo</string>\n"
@@ -1981,7 +2018,8 @@ static int launchctl_schedule_plist(const char *exec_path, enum schedule_priorit
 		   "</array>\n"
 		   "<key>StartCalendarInterval</key>\n"
 		   "<array>\n";
-	strbuf_addf(&plist, preamble, name, exec_path, exec_path, frequency);
+	strbuf_addf(&plist, preamble, name, exec_path, exec_path,
+		    get_extra_launchctl_strings(), frequency);
 
 	switch (schedule) {
 	case SCHEDULE_HOURLY:
@@ -2216,11 +2254,12 @@ static int schtasks_schedule_task(const char *exec_path, enum schedule_priority
 	      "<Actions Context=\"Author\">\n"
 	      "<Exec>\n"
 	      "<Command>\"%s\\headless-git.exe\"</Command>\n"
-	      "<Arguments>--exec-path=\"%s\" for-each-repo --keep-going --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n"
+	      "<Arguments>--exec-path=\"%s\" %s for-each-repo --keep-going --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n"
 	      "</Exec>\n"
 	      "</Actions>\n"
 	      "</Task>\n";
-	fprintf(tfile->fp, xml, exec_path, exec_path, frequency);
+	fprintf(tfile->fp, xml, exec_path, exec_path,
+		get_extra_config_parameters(), frequency);
 	strvec_split(&child.args, cmd);
 	strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml",
 				  get_tempfile_path(tfile), NULL);
@@ -2361,8 +2400,8 @@ static int crontab_update_schedule(int run_maintenance, int fd)
 			"# replaced in the future by a Git command.\n\n");
 
 		strbuf_addf(&line_format,
-			    "%%d %%s * * %%s \"%s/git\" --exec-path=\"%s\" for-each-repo --keep-going --config=maintenance.repo maintenance run --schedule=%%s\n",
-			    exec_path, exec_path);
+			    "%%d %%s * * %%s \"%s/git\" --exec-path=\"%s\" %s for-each-repo --keep-going --config=maintenance.repo maintenance run --schedule=%%s\n",
+			    exec_path, exec_path, get_extra_config_parameters());
 		fprintf(cron_in, line_format.buf, minute, "1-23", "*", "hourly");
 		fprintf(cron_in, line_format.buf, minute, "0", "1-6", "daily");
 		fprintf(cron_in, line_format.buf, minute, "0", "0", "weekly");
@@ -2562,7 +2601,7 @@ static int systemd_timer_write_service_template(const char *exec_path)
 	       "\n"
 	       "[Service]\n"
 	       "Type=oneshot\n"
-	       "ExecStart=\"%s/git\" --exec-path=\"%s\" for-each-repo --keep-going --config=maintenance.repo maintenance run --schedule=%%i\n"
+	       "ExecStart=\"%s/git\" --exec-path=\"%s\" %s for-each-repo --keep-going --config=maintenance.repo maintenance run --schedule=%%i\n"
 	       "LockPersonality=yes\n"
 	       "MemoryDenyWriteExecute=yes\n"
 	       "NoNewPrivileges=yes\n"
@@ -2572,7 +2611,7 @@ static int systemd_timer_write_service_template(const char *exec_path)
 	       "RestrictSUIDSGID=yes\n"
 	       "SystemCallArchitectures=native\n"
 	       "SystemCallFilter=@system-service\n";
-	if (fprintf(file, unit, exec_path, exec_path) < 0) {
+	if (fprintf(file, unit, exec_path, exec_path, get_extra_config_parameters()) < 0) {
 		error(_("failed to write to '%s'"), filename);
 		fclose(file);
 		goto error;
diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
index abae7a97546..3cd7e1fcacb 100755
--- a/t/t7900-maintenance.sh
+++ b/t/t7900-maintenance.sh
@@ -825,6 +825,9 @@ test_expect_success 'start and stop Linux/systemd maintenance' '
 	test_systemd_analyze_verify "systemd/user/git-maintenance@daily.service" &&
 	test_systemd_analyze_verify "systemd/user/git-maintenance@weekly.service" &&
 
+	grep "core.askPass=true" "systemd/user/git-maintenance@.service" &&
+	grep "credential.interactive=false" "systemd/user/git-maintenance@.service" &&
+
 	printf -- "--user enable --now git-maintenance@%s.timer\n" hourly daily weekly >expect &&
 	test_cmp expect args &&
 
-- 
gitgitgadget

[PATCH 3/3] scalar: configure maintenance during 'reconfigure'

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The 'scalar reconfigure' command is intended to update registered repos
with the latest settings available. However, up to now we were not
reregistering the repos with background maintenance.

In particular, this meant that the background maintenance schedule would
not be updated if there are improvements between versions.

Be sure to register repos for maintenance during the reconfigure step.

Signed-off-by: Derrick Stolee <stolee@gmail.com>
---
 scalar.c          | 3 +++
 t/t9210-scalar.sh | 7 +++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/scalar.c b/scalar.c
index 09560aeab54..73b79a5d4c9 100644
--- a/scalar.c
+++ b/scalar.c
@@ -733,6 +733,9 @@ static int cmd_reconfigure(int argc, const char **argv)
 
 		the_repository = old_repo;
 
+		if (toggle_maintenance(1) >= 0)
+			succeeded = 1;
+
 loop_end:
 		if (!succeeded) {
 			res = -1;
diff --git a/t/t9210-scalar.sh b/t/t9210-scalar.sh
index e8613990e13..027235d61aa 100755
--- a/t/t9210-scalar.sh
+++ b/t/t9210-scalar.sh
@@ -194,8 +194,11 @@ test_expect_success 'scalar reconfigure' '
 	scalar reconfigure one &&
 	test true = "$(git -C one/src config core.preloadIndex)" &&
 	git -C one/src config core.preloadIndex false &&
-	scalar reconfigure -a &&
-	test true = "$(git -C one/src config core.preloadIndex)"
+	rm one/src/cron.txt &&
+	GIT_TRACE2_EVENT="$(pwd)/reconfigure" scalar reconfigure -a &&
+	test_path_is_file one/src/cron.txt &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	test_subcommand git maintenance start <reconfigure
 '
 
 test_expect_success 'scalar reconfigure --all with includeIf.onbranch' '
-- 
gitgitgadget

Re: [PATCH 0/3] maintenance: configure credentials to be silent

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Add a new configuration value, 'credential.interactive', to specify to the
> credential helper that it should not prompt for user interaction. This
> option has been respected by Git Credential Manager since 2020 [1], so this
> is now presenting it as an official Git config value.

So, the other helpers are also supposed to check for the variable
and fail when it has to go interactive now.

> These changes were first merged into the microsoft/git fork in August 2023
> [2] but were not upstreamed immediately. The change has been a positive one
> for users of that fork, as they no longer get pop-ups and they also are not
> getting maintenance.lock file blocks when the prefetch task waits for
> credentials. This has become even more important recently as credential
> lifetimes have been restricted significantly, leading to a higher likelihood
> that this will happen during a background prefetch.

Sounds good.

Re: [PATCH 0/3] maintenance: configure credentials to be silent

[Derrick Stolee]

On 9/20/24 5:56 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> Add a new configuration value, 'credential.interactive', to specify to the
>> credential helper that it should not prompt for user interaction. This
>> option has been respected by Git Credential Manager since 2020 [1], so this
>> is now presenting it as an official Git config value.
> 
> So, the other helpers are also supposed to check for the variable
> and fail when it has to go interactive now.

I would hold off from saying "supposed to" but Git is definitely hinting
towards that behavior.

Perhaps I'm just hung up on the idea that we are not adding a new wrinkle
to the "contract" but recommending a good thing that was previously not part
of the interaction.

Thanks,
-Stolee

Re: [PATCH 0/3] maintenance: configure credentials to be silent

[Junio C Hamano]

Derrick Stolee <stolee@gmail.com> writes:

> On 9/20/24 5:56 PM, Junio C Hamano wrote:
>> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>> 
>>> Add a new configuration value, 'credential.interactive', to specify to the
>>> credential helper that it should not prompt for user interaction. This
>>> option has been respected by Git Credential Manager since 2020 [1], so this
>>> is now presenting it as an official Git config value.
>> So, the other helpers are also supposed to check for the variable
>> and fail when it has to go interactive now.
>
> I would hold off from saying "supposed to" but Git is definitely hinting
> towards that behavior.

I would too.  I didn't mean "they were behaving correctly, but we
changed the rules from under them and they need to be fixed".  With
or without your patch, they would try to go interactive and make the
process get stuck, until they start to check if they should refrain
from going interactive.  With your patch, they have a way to do that
check in a documented way.

> Perhaps I'm just hung up on the idea that we are not adding a new wrinkle
> to the "contract" but recommending a good thing that was previously not part
> of the interaction.
>
> Thanks,
> -Stolee