Re: [GSoC PATCH] backfill: add --[no-]progress option

[Derrick Stolee <stolee@gmail.com>]

On 3/29/2026 11:24 AM, Trieu Huynh wrote:
> 'git backfill' is silent when downloading missing objects, giving
> no feedback during potentially long-running operations on large
> repositories. By contrast, 'git fetch', 'git gc', and
> 'git index-pack' all support --[no-]progress.

I wouldn't use the word "silent" because the output is actually
quite verbose by default. Each batch has progress output with the
remote. For example, this is the output I get when running 'git
backfill' on a blobless partial clone of the Git repo:

$ git backfill
remote: Enumerating objects: 50083, done.
remote: Counting objects: 100% (865/865), done.
remote: Compressing objects: 100% (177/177), done.
remote: Total 50083 (delta 760), reused 688 (delta 688), pack-reused 49218 (from 1)
Receiving objects: 100% (50083/50083), 37.13 MiB | 27.75 MiB/s, done.
Resolving deltas: 100% (47710/47710), done.
remote: Enumerating objects: 50393, done.
remote: Counting objects: 100% (1559/1559), done.
remote: Compressing objects: 100% (366/366), done.
remote: Total 50393 (delta 1366), reused 1193 (delta 1193), pack-reused 48834 (from 2)
Receiving objects: 100% (50393/50393), 44.56 MiB | 31.56 MiB/s, done.
Resolving deltas: 100% (47261/47261), done.
remote: Enumerating objects: 50000, done.
remote: Counting objects: 100% (2313/2313), done.
remote: Compressing objects: 100% (592/592), done.
remote: Total 50000 (delta 1982), reused 1721 (delta 1721), pack-reused 47687 (from 2)
Receiving objects: 100% (50000/50000), 90.49 MiB | 17.85 MiB/s, done.
Resolving deltas: 100% (45321/45321), done.
remote: Enumerating objects: 2155, done.
remote: Counting objects: 100% (27/27), done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 2155 (delta 6), reused 1 (delta 1), pack-reused 2128 (from 1)
Receiving objects: 100% (2155/2155), 891.74 KiB | 3.75 MiB/s, done.
Resolving deltas: 100% (1717/1717), done.

With your patch, I think there would be some extra progress
indicators between these batched fetch requests.

Before moving forward with review of this patch, I think that
it would be valuable to demonstrate the full output with and
without your change.

In addition, I think there would be value in a progress indicator
_instead_ of these verbose outputs from the remote. That would
require a change to how we initialize the fetches in a quiet mode.

(I also understand that this output would probably not be the same
if we have a filesystem protocol for fetching from a local repo,
like we frequently do in the test suite.)

>  static void backfill_context_clear(struct backfill_context *ctx)
> @@ -54,6 +57,7 @@ static void download_batch(struct backfill_context *ctx)
>  	 * avoid possible duplicate downloads of the same objects.
>  	 */
>  	odb_reprepare(ctx->repo->objects);
> +	display_progress(ctx->progress, ++ctx->batches_requested);

This looks correct. My preference is to not use prefix operators
like this on struct members (it reads like you are incrementing
'ctx' and not 'batches_requested', even though it is correct).

However, I'm not sure that we want the progress to indicate the
number of _batches_ but instead should be the number of _objects_.
  
>  static int fill_missing_blobs(const char *path UNUSED,
> @@ -120,12 +124,15 @@ int cmd_backfill(int argc, const char **argv, const char *prefix, struct reposit
>  		.current_batch = OID_ARRAY_INIT,
>  		.min_batch_size = 50000,
>  		.sparse = 0,
> +		.show_progress = -1,
>  	};
>  	struct option options[] = {
>  		OPT_UNSIGNED(0, "min-batch-size", &ctx.min_batch_size,
>  			     N_("Minimum number of objects to request at a time")),
>  		OPT_BOOL(0, "sparse", &ctx.sparse,
>  			 N_("Restrict the missing objects to the current sparse-checkout")),
> +		OPT_BOOL(0, "progress", &ctx.show_progress,
> +			 N_("show progress while downloading missing objects")),
>  		OPT_END(),
>  	};

I hope that this does not cause any issues with the recent changes
to include the rev-list options in git-backfill. Worth checking.

> +test_expect_success 'backfill --progress shows progress' '
> +	git clone --no-checkout --filter=blob:none \
> +		--single-branch --branch=main \
> +		"file://$(pwd)/srv.bare" clone-progress &&
> +	git -C clone-progress backfill --progress 2>err &&
> +	test_grep "Downloading batches" err
> +'
> +
> +test_expect_success 'backfill --no-progress is silent' '
> +	git clone --no-checkout --filter=blob:none \
> +		--single-branch --branch=main \
> +		"file://$(pwd)/srv.bare" clone-no-progress &&
> +	git -C clone-no-progress backfill --no-progress 2>err &&
> +	test_grep ! "Downloading batches" err
> +'
> +
> +test_expect_success 'backfill no flag on non-TTY is silent' '
> +	git clone --no-checkout --filter=blob:none \
> +		--single-branch --branch=main \
> +		"file://$(pwd)/srv.bare" clone-notty &&
> +	git -C clone-notty backfill 2>err &&
> +	test_grep ! "Downloading batches" err
> +'

What you are missing here is that the progress isn't silent when
a TTY is present. There are several tests in the test suite that
use the TTY prerequisite for this kind of behavior, such as this
one from t9211-scalar-clone.sh:

test_expect_success TTY 'progress with tty' '
	enlistment=progress1 &&

	test_config -C to-clone uploadpack.allowfilter true &&
	test_config -C to-clone uploadpack.allowanysha1inwant true &&

	test_terminal env GIT_PROGRESS_DELAY=0 \
		scalar clone "file://$(pwd)/to-clone" "$enlistment" 2>stderr &&
	grep "Enumerating objects" stderr >actual &&
	test_line_count = 2 actual &&
	cleanup_clone $enlistment
'

Thanks,
-Stolee