[PATCH v2 0/2] diff-files: integrate with sparse index

[PATCH v2 0/2] diff-files: integrate with sparse index

[Shuqi Liang]

Turn on sparse-index feature within `git diff-files` command.
Add necessary modifications and test them.

Changes since v1:

1.Add an empty line between the previous test's closing quote
and the start of new test.

2.Use "git add --sparse newdirectory/testfile" instead of 
'git sparse-checkout set' to have staged changes outside 
of the sparse-checkout cone

3.Edit commit message to be more informative


Shuqi Liang (2):
  t1092: add tests for `git diff-files`
  diff-files: integrate with sparse index

 builtin/diff-files.c                     |  4 ++
 t/perf/p2000-sparse-operations.sh        |  2 +
 t/t1092-sparse-checkout-compatibility.sh | 52 ++++++++++++++++++++++++
 3 files changed, 58 insertions(+)


base-commit: a38d39a4c50d1275833aba54c4dbdfce9e2e9ca1
-- 
2.39.0

[PATCH v2 1/2] t1092: add tests for `git diff-files`

[Shuqi Liang]

Before integrating the 'git diff-files' builtin
with the sparse index feature, add tests to
t1092-sparse-checkout-compatibility.sh to ensure it currently works
with sparse-checkout and will still work with sparse index
after that integration.

When adding tests against a sparse-checkout
definition, we test two modes: all changes are
within the sparse-checkout cone and some changes are outside
the sparse-checkout cone.

In order to have staged changes outside of
the sparse-checkout cone, create a 'newdirectory/testfile' and
add it to the index, while leaving it outside of
the sparse-checkout definition.Test 'newdirectory/testfile'
being present on-disk without modifications, then change content inside
'newdirectory/testfile' in order to test 'newdirectory/testfile'
being present on-disk with modifications.

Signed-off-by: Shuqi Liang <cheskaqiqi@gmail.com>
---
 t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 801919009e..9382428352 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -2055,4 +2055,42 @@ test_expect_success 'grep sparse directory within submodules' '
 	test_cmp actual expect
 '
 
+test_expect_success 'diff-files with pathspec inside sparse definition' '
+	init_repos &&
+
+	write_script edit-contents <<-\EOF &&
+	echo text >>$1
+	EOF
+
+	run_on_all ../edit-contents deep/a &&
+
+	test_all_match git diff-files  &&
+	test_all_match git diff-files deep/a &&
+	test_all_match git diff-files --find-object=HEAD:a
+'
+
+test_expect_success 'diff-files with pathspec outside sparse definition' '
+	init_repos &&
+
+	write_script edit-contents <<-\EOF &&
+	echo text >>$1
+	EOF
+
+	#add file to the index but outside of cone
+	run_on_sparse mkdir newdirectory &&
+	run_on_sparse ../edit-contents newdirectory/testfile &&
+	test_sparse_match git add --sparse newdirectory/testfile &&
+
+	#file present on-disk without modifications
+	test_sparse_match git diff-files &&
+	test_sparse_match git diff-files newdirectory/testfile &&
+	test_sparse_match test_must_fail git diff-files --find-object=HEAD:testfile &&
+
+	#file present on-disk with modifications
+	run_on_sparse ../edit-contents newdirectory/testfile &&
+	test_sparse_match git diff-files &&
+	test_sparse_match git diff-files newdirectory/testfile &&
+	test_sparse_match test_must_fail git diff-files --find-object=HEAD:testfile
+'
+
 test_done
-- 
2.39.0

[PATCH v2 2/2] diff-files: integrate with sparse index

[Shuqi Liang]

Remove full index requirement for `git diff-files`
and test to ensure the index is not expanded in `git diff-files`.

The `p2000` tests demonstrate a ~96% execution time reduction for 'git
diff-files' and a ~97% execution time reduction for 'git diff-files'
for a file using a sparse index:

Test                                           before  after
-----------------------------------------------------------------
2000.78: git diff-files (full-v3)              0.09    0.08 -11.1%
2000.79: git diff-files (full-v4)              0.09    0.09 +0.0%
2000.80: git diff-files (sparse-v3)            0.52    0.02 -96.2%
2000.81: git diff-files (sparse-v4)            0.51    0.02 -96.1%
2000.82: git diff-files f2/f4/a (full-v3)      0.06    0.07 +16.7%
2000.83: git diff-files f2/f4/a (full-v4)      0.08    0.08 +0.0%
2000.84: git diff-files f2/f4/a (sparse-v3)    0.46    0.01 -97.8%
2000.85: git diff-files f2/f4/a (sparse-v4)    0.51    0.02 -96.1%

Signed-off-by: Shuqi Liang <cheskaqiqi@gmail.com>
---
 builtin/diff-files.c                     |  4 ++++
 t/perf/p2000-sparse-operations.sh        |  2 ++
 t/t1092-sparse-checkout-compatibility.sh | 14 ++++++++++++++
 3 files changed, 20 insertions(+)

diff --git a/builtin/diff-files.c b/builtin/diff-files.c
index dc991f753b..360464e6ef 100644
--- a/builtin/diff-files.c
+++ b/builtin/diff-files.c
@@ -27,6 +27,10 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix)
 		usage(diff_files_usage);
 
 	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
+
+	prepare_repo_settings(the_repository);
+	the_repository->settings.command_requires_full_index = 0;
+
 	repo_init_revisions(the_repository, &rev, prefix);
 	rev.abbrev = 0;
 
diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh
index 3242cfe91a..82751f2ca3 100755
--- a/t/perf/p2000-sparse-operations.sh
+++ b/t/perf/p2000-sparse-operations.sh
@@ -125,5 +125,7 @@ test_perf_on_all git checkout-index -f --all
 test_perf_on_all git update-index --add --remove $SPARSE_CONE/a
 test_perf_on_all "git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a"
 test_perf_on_all git grep --cached --sparse bogus -- "f2/f1/f1/*"
+test_perf_on_all git diff-files
+test_perf_on_all git diff-files $SPARSE_CONE/a
 
 test_done
diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 9382428352..7cc02b882b 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -2093,4 +2093,18 @@ test_expect_success 'diff-files with pathspec outside sparse definition' '
 	test_sparse_match test_must_fail git diff-files --find-object=HEAD:testfile
 '
 
+test_expect_success 'sparse index is not expanded: diff-files' '
+	init_repos &&
+
+	write_script edit-contents <<-\EOF &&
+	echo text >>$1
+	EOF
+
+	run_on_all ../edit-contents deep/a &&
+
+	ensure_not_expanded diff-files  &&
+	ensure_not_expanded diff-files deep/a &&
+	ensure_not_expanded diff-files --find-object=HEAD:a
+'
+
 test_done
-- 
2.39.0

[PATCH v2 1/2] t1092: add tests for `git diff-files`

[Shuqi Liang]

Before integrating the 'git diff-files' builtin
with the sparse index feature, add tests to
t1092-sparse-checkout-compatibility.sh to ensure it currently works
with sparse-checkout and will still work with sparse index
after that integration.

When adding tests against a sparse-checkout
definition, we test two modes: all changes are
within the sparse-checkout cone and some changes are outside
the sparse-checkout cone.

In order to have staged changes outside of
the sparse-checkout cone, create a 'newdirectory/testfile' and
add it to the index, while leaving it outside of
the sparse-checkout definition.Test 'newdirectory/testfile'
being present on-disk without modifications, then change content inside
'newdirectory/testfile' in order to test 'newdirectory/testfile'
being present on-disk with modifications.

Signed-off-by: Shuqi Liang <cheskaqiqi@gmail.com>
---
 t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 801919009e..9382428352 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -2055,4 +2055,42 @@ test_expect_success 'grep sparse directory within submodules' '
 	test_cmp actual expect
 '
 
+test_expect_success 'diff-files with pathspec inside sparse definition' '
+	init_repos &&
+
+	write_script edit-contents <<-\EOF &&
+	echo text >>$1
+	EOF
+
+	run_on_all ../edit-contents deep/a &&
+
+	test_all_match git diff-files  &&
+	test_all_match git diff-files deep/a &&
+	test_all_match git diff-files --find-object=HEAD:a
+'
+
+test_expect_success 'diff-files with pathspec outside sparse definition' '
+	init_repos &&
+
+	write_script edit-contents <<-\EOF &&
+	echo text >>$1
+	EOF
+
+	#add file to the index but outside of cone
+	run_on_sparse mkdir newdirectory &&
+	run_on_sparse ../edit-contents newdirectory/testfile &&
+	test_sparse_match git add --sparse newdirectory/testfile &&
+
+	#file present on-disk without modifications
+	test_sparse_match git diff-files &&
+	test_sparse_match git diff-files newdirectory/testfile &&
+	test_sparse_match test_must_fail git diff-files --find-object=HEAD:testfile &&
+
+	#file present on-disk with modifications
+	run_on_sparse ../edit-contents newdirectory/testfile &&
+	test_sparse_match git diff-files &&
+	test_sparse_match git diff-files newdirectory/testfile &&
+	test_sparse_match test_must_fail git diff-files --find-object=HEAD:testfile
+'
+
 test_done
-- 
2.39.0

Re: [PATCH v2 1/2] t1092: add tests for `git diff-files`

[Junio C Hamano]

Shuqi Liang <cheskaqiqi@gmail.com> writes:

> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
> index 801919009e..9382428352 100755
> --- a/t/t1092-sparse-checkout-compatibility.sh
> +++ b/t/t1092-sparse-checkout-compatibility.sh
> @@ -2055,4 +2055,42 @@ test_expect_success 'grep sparse directory within submodules' '
>  	test_cmp actual expect
>  '
>  
> +test_expect_success 'diff-files with pathspec inside sparse definition' '
> +	init_repos &&
> +
> +	write_script edit-contents <<-\EOF &&
> +	echo text >>$1
> +	EOF

(Documentation/CodingGuidelines)

 - Redirection operators should be written with space before, but no
   space after them.  In other words, write 'echo test >"$file"'
   instead of 'echo test> $file' or 'echo test > $file'.  Note that
   even though it is not required by POSIX to double-quote the
   redirection target in a variable (as shown above), our code does so
   because some versions of bash issue a warning without the quotes.

> +	#add file to the index but outside of cone

Can you have a SP after "#" here to make it more readable?

> +	run_on_sparse mkdir newdirectory &&
> +	run_on_sparse ../edit-contents newdirectory/testfile &&
> +	test_sparse_match git add --sparse newdirectory/testfile &&

We create a new directory that is outside the cone, with or without
using the sparse-index feature.  We know we are violating the cone,
and have to override the safety with the "--sparse" option.  OK.

What output do we expect out of "git add" to match in the two cases?

> +	#file present on-disk without modifications
> +	test_sparse_match git diff-files &&
> +	test_sparse_match git diff-files newdirectory/testfile &&

As "diff-files" is about comparing between the index and the working
tree, the new path should not appear in the output when the sparse
checkout feature with or without the sparse-index feature is NOT in
use.  Does the picture get different when we are sparse?  IOW, would
we notice that we now have newdirectory/testfile that is supposed to
be missing in the index and show that in the output?

> +	test_sparse_match test_must_fail git diff-files --find-object=HEAD:testfile &&

What does HEAD:testfile refer to in this test?  This expects "diff-files"
invocation to fail, and perhaps in your test it failed in both test
repositories the same way, but are they failing for the right reason?

In a non-sparse repository whose HEAD commit does not have
'testfile' (e.g. "git" source tree), I get

    $ git diff-files --find-object=HEAD:testfile
    error: unable to resolve 'HEAD:testfile'

without sparse checkout or sparse index.  It is unclear what value
we get out of having this test here.

> +	#file present on-disk with modifications
> +	run_on_sparse ../edit-contents newdirectory/testfile &&
> +	test_sparse_match git diff-files &&
> +	test_sparse_match git diff-files newdirectory/testfile &&
> +	test_sparse_match test_must_fail git diff-files --find-object=HEAD:testfile

Ditto.

> +'
> +
>  test_done

Thanks.

Re: [PATCH v2 1/2] t1092: add tests for `git diff-files`

[Shuqi Liang]

Hi Junio

On Tue, Mar 7, 2023 at 1:53 PM Junio C Hamano <gitster@pobox.com> wrote:

> (Documentation/CodingGuidelines)
>
>  - Redirection operators should be written with space before, but no
>    space after them.  In other words, write 'echo test >"$file"'
>    instead of 'echo test> $file' or 'echo test > $file'.  Note that
>    even though it is not required by POSIX to double-quote the
>    redirection target in a variable (as shown above), our code does so
>    because some versions of bash issue a warning without the quotes.


Thanks for the styling reminders! I should go back and reread CodingGuidelines
more often.


> > +     #add file to the index but outside of cone
>
> Can you have a SP after "#" here to make it more readable?

Will do!


> We create a new directory that is outside the cone, with or without
> using the sparse-index feature.  We know we are violating the cone,
> and have to override the safety with the "--sparse" option.  OK.
>
> What output do we expect out of "git add" to match in the two cases?
>
> > +     #file present on-disk without modifications
> > +     test_sparse_match git diff-files &&
> > +     test_sparse_match git diff-files newdirectory/testfile &&
>
> As "diff-files" is about comparing between the index and the working
> tree, the new path should not appear in the output when the sparse
> checkout feature with or without the sparse-index feature is NOT in
> use.  Does the picture get different when we are sparse?  IOW, would
> we notice that we now have newdirectory/testfile that is supposed to
> be missing in the index and show that in the output?

I'm a bit caught up here.
Do you mean I need to add a test for "git add" also?

when we use "git add" instead of "git add --sparse ", we will get different.
Cause newdirectory/testfile is missing in the index so diff-files will not
work in these cases.


> In a non-sparse repository whose HEAD commit does not have
> 'testfile' (e.g. "git" source tree), I get
>
>     $ git diff-files --find-object=HEAD:testfile
>     error: unable to resolve 'HEAD:testfile'
>
> without sparse checkout or sparse index.  It is unclear what value
> we get out of having this test here.

Thanks for pointing out the error. HEAD:testfile is useless for the test here.

-----------------------------------
Thanks
Shuqi

Re: [PATCH v2 1/2] t1092: add tests for `git diff-files`

[Junio C Hamano]

Shuqi Liang <cheskaqiqi@gmail.com> writes:

>> We create a new directory that is outside the cone, with or without
>> using the sparse-index feature.  We know we are violating the cone,
>> and have to override the safety with the "--sparse" option.  OK.
>>
>> What output do we expect out of "git add" to match in the two cases?
>>
>> > +     #file present on-disk without modifications
>> > +     test_sparse_match git diff-files &&
>> > +     test_sparse_match git diff-files newdirectory/testfile &&
>>
>> As "diff-files" is about comparing between the index and the working
>> tree, the new path should not appear in the output when the sparse
>> checkout feature with or without the sparse-index feature is NOT in
>> use.  Does the picture get different when we are sparse?  IOW, would
>> we notice that we now have newdirectory/testfile that is supposed to
>> be missing in the index and show that in the output?
>
> I'm a bit caught up here.
> Do you mean I need to add a test for "git add" also?

Not really.  The above two tests are happy with _any_ output coming
out of "git diff-files" (and "git diff-files nd/tf") as long as they
match between sparse checkouts, one of which uses and the other does
not use the sparse index feature.  I was wondering if we want to be
a bit stricter than that.  Thinks like "not only the two output must
match, they both must be empty".