Excluding paths with wildcard not working with add -p

Excluding paths with wildcard not working with add -p

[Robert Dailey]

If I run the command:

    git add -p -- ':^*.cs'

I get an error:

fatal: empty string is not a valid pathspec. please use . instead if
you meant to match all paths
Cannot close git diff-index --cached --numstat --summary HEAD --
:(exclude,prefix:0)*.cs  () at C:/Program
Files/Git/mingw64/libexec/git-core\git-add--interactive line 242.

However, it works if I remove `-p`. Also observed this works too:

    git add -p -- ':*.cs'

So it looks like we have a corner case where I can't do a patch add
while excluding files. Is this intentional?

Re: Excluding paths with wildcard not working with add -p

[Junio C Hamano]

Robert Dailey <rcdailey.lists@gmail.com> writes:

> If I run the command:
>
>     git add -p -- ':^*.cs'
>
> I get an error:
>
> fatal: empty string is not a valid pathspec. please use . instead if
> you meant to match all paths
> Cannot close git diff-index --cached --numstat --summary HEAD --
> :(exclude,prefix:0)*.cs  () at C:/Program
> Files/Git/mingw64/libexec/git-core\git-add--interactive line 242.
>
> However, it works if I remove `-p`. Also observed this works too:
>
>     git add -p -- ':*.cs'
>
> So it looks like we have a corner case where I can't do a patch add
> while excluding files. Is this intentional?

I do not think so.

Does this command

    git add -p -- . ':^*.cs'

work as you expect?

Re: Excluding paths with wildcard not working with add -p

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Does this command
>
>     git add -p -- . ':^*.cs'
>
> work as you expect?

We used to error out when you give a pathspec with only negative
elements in it, like the one you gave above.  Later, we tweaked this
logic at 859b7f1d (pathspec: don't error out on all-exclusionary
pathspec patterns, 2017-02-07) so that we add an empty string as an
extra element when your pathspec has only negative elements.

At around the same time, we were migrating from "an empty string is
a valid pathspec element that matches everything" to "either a dot
or ":/" is used for that purpose, and an empty string is rejected",
between d426430e (pathspec: warn on empty strings as pathspec,
2016-06-22) and 9e4e8a64 (pathspec: die on empty strings as
pathspec, 2017-06-06).  I think 9e4e8a64 was not careful enough to
turn the empty string 859b7f1d added to either a dot or ":/"

For the purpose of "add -p", I _think_ adding a "dot" is correct,
but depending on the command, the code needs to add ":/".

Here is a quick trial patch, which seems to compile and pass all the
tests we have, but the fact that this lingered with us for the past
5 years is a strong sign that we lack coverage in this area, so it
may be breaking something else in a big way.

 pathspec.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git c/pathspec.c w/pathspec.c
index ddeeba7911..1b0ae51aa4 100644
--- c/pathspec.c
+++ w/pathspec.c
@@ -628,8 +628,10 @@ void parse_pathspec(struct pathspec *pathspec,
 	 * that matches everything. We allocated an extra one for this.
 	 */
 	if (nr_exclude == n) {
-		int plen = (!(flags & PATHSPEC_PREFER_CWD)) ? 0 : prefixlen;
-		init_pathspec_item(item + n, 0, prefix, plen, "");
+		if (!(flags & PATHSPEC_PREFER_CWD))
+			init_pathspec_item(item + n, 0, NULL, 0, ":/");
+		else
+			init_pathspec_item(item + n, 0, prefix, prefixlen, ".");
 		pathspec->nr++;
 	}
 

[PATCH] pathspec: correct an empty string used as a pathspec element

[Junio C Hamano]

Pathspecs with only negative elements did not work with some
commands that pass the pathspec along to a subprocess.  For
instance,

    $ git add -p -- ':!*.txt'

should add everything except for paths ending in ".txt", but it gets
complaint from underlying "diff-index" and aborts.

We used to error out when a pathspec with only negative elements in
it, like the one in the above example.  Later, 859b7f1d (pathspec:
don't error out on all-exclusionary pathspec patterns, 2017-02-07)
updated the logic to add an empty string as an extra element.  The
intention was to let the extra element to match everything and let
the negative ones given by the user to subtract from it.

At around the same time, we were migrating from "an empty string is
a valid pathspec element that matches everything" to "either a dot
or ":/" is used to match all, and an empty string is rejected",
between d426430e (pathspec: warn on empty strings as pathspec,
2016-06-22) and 9e4e8a64 (pathspec: die on empty strings as
pathspec, 2017-06-06).  I think 9e4e8a64, which happened long after
859b7f1d happened, was not careful enough to turn the empty string
859b7f1d added to either a dot or ":/".

A care should be taken as the definition of "everything" depends on
subcommand.  For the purpose of "add -p", adding a "." to add
everything in the current directory is the right thing to do.  But
for some other commands, ":/" (i.e. really really everything, even
things outside the current subdirectory) is the right choice.

We would break commands in a big way if we get this wrong, so add a
handful of test pieces to make sure the resulting code still
excludes the paths that are expected and includes "everything" else.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 pathspec.c                  |   2 +-
 t/t6132-pathspec-exclude.sh | 181 ++++++++++++++++++++++++++++++++++++
 2 files changed, 182 insertions(+), 1 deletion(-)

diff --git a/pathspec.c b/pathspec.c
index ddeeba7911..84ad9c73cf 100644
--- a/pathspec.c
+++ b/pathspec.c
@@ -629,7 +629,7 @@ void parse_pathspec(struct pathspec *pathspec,
 	 */
 	if (nr_exclude == n) {
 		int plen = (!(flags & PATHSPEC_PREFER_CWD)) ? 0 : prefixlen;
-		init_pathspec_item(item + n, 0, prefix, plen, "");
+		init_pathspec_item(item + n, 0, prefix, plen, ".");
 		pathspec->nr++;
 	}
 
diff --git a/t/t6132-pathspec-exclude.sh b/t/t6132-pathspec-exclude.sh
index 8ff1d76f79..9fdafeb1e9 100755
--- a/t/t6132-pathspec-exclude.sh
+++ b/t/t6132-pathspec-exclude.sh
@@ -195,6 +195,7 @@ test_expect_success 'multiple exclusions' '
 '
 
 test_expect_success 't_e_i() exclude case #8' '
+	test_when_finished "rm -fr case8" &&
 	git init case8 &&
 	(
 		cd case8 &&
@@ -244,4 +245,184 @@ test_expect_success 'grep --untracked PATTERN :(exclude)*FILE' '
 	test_cmp expect-grep actual-grep
 '
 
+# Depending on the command, all negative pathspec needs to subtract
+# either from the full tree, or from the current directory.
+#
+# The sample tree checked out at this point has:
+# file
+# sub/file
+# sub/file2
+# sub/sub/file
+# sub/sub/sub/file
+# sub2/file
+#
+# but there may also be some cruft that interferes with "git clean"
+# and "git add" tests.
+
+test_expect_success 'archive with all negative' '
+	git reset --hard &&
+	git clean -f &&
+	git -C sub archive --format=tar HEAD -- ":!sub/" >archive &&
+	"$TAR" tf archive >actual &&
+	cat >expect <<-\EOF &&
+	file
+	file2
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'add with all negative' '
+	H=$(git rev-parse HEAD) &&
+	git reset --hard $H &&
+	git clean -f &&
+	test_when_finished "git reset --hard $H" &&
+	for path in file sub/file sub/sub/file sub2/file
+	do
+		echo smudge >>"$path" || return 1
+	done &&
+	git -C sub add -- ":!sub/" &&
+	git diff --name-only --no-renames --cached >actual &&
+	cat >expect <<-\EOF &&
+	file
+	sub/file
+	sub2/file
+	EOF
+	test_cmp expect actual &&
+	git diff --name-only --no-renames >actual &&
+	echo sub/sub/file >expect &&
+	test_cmp expect actual
+'
+
+test_expect_success 'add -p with all negative' '
+	H=$(git rev-parse HEAD) &&
+	git reset --hard $H &&
+	git clean -f &&
+	test_when_finished "git reset --hard $H" &&
+	for path in file sub/file sub/sub/file sub2/file
+	do
+		echo smudge >>"$path" || return 1
+	done &&
+	yes | git -C sub add -p -- ":!sub/" &&
+	git diff --name-only --no-renames --cached >actual &&
+	cat >expect <<-\EOF &&
+	file
+	sub/file
+	sub2/file
+	EOF
+	test_cmp expect actual &&
+	git diff --name-only --no-renames >actual &&
+	echo sub/sub/file >expect &&
+	test_cmp expect actual
+'
+
+test_expect_success 'clean with all negative' '
+	H=$(git rev-parse HEAD) &&
+	git reset --hard $H &&
+	test_when_finished "git reset --hard $H && git clean -f" &&
+	git clean -f &&
+	for path in file9 sub/file9 sub/sub/file9 sub2/file9
+	do
+		echo cruft >"$path" || return 1
+	done &&
+	git -C sub clean -f -- ":!sub" &&
+	test_path_is_file file9 &&
+	test_path_is_missing sub/file9 &&
+	test_path_is_file sub/sub/file9 &&
+	test_path_is_file sub2/file9
+'
+
+test_expect_success 'commit with all negative' '
+	H=$(git rev-parse HEAD) &&
+	git reset --hard $H &&
+	test_when_finished "git reset --hard $H" &&
+	for path in file sub/file sub/sub/file sub2/file
+	do
+		echo smudge >>"$path" || return 1
+	done &&
+	git -C sub commit -m sample -- ":!sub/" &&
+	git diff --name-only --no-renames HEAD^ HEAD >actual &&
+	cat >expect <<-\EOF &&
+	file
+	sub/file
+	sub2/file
+	EOF
+	test_cmp expect actual &&
+	git diff --name-only --no-renames HEAD >actual &&
+	echo sub/sub/file >expect &&
+	test_cmp expect actual
+'
+
+test_expect_success 'reset with all negative' '
+	H=$(git rev-parse HEAD) &&
+	git reset --hard $H &&
+	test_when_finished "git reset --hard $H" &&
+	for path in file sub/file sub/sub/file sub2/file
+	do
+		echo smudge >>"$path" &&
+		git add "$path" || return 1
+	done &&
+	git -C sub reset --quiet -- ":!sub/" &&
+	git diff --name-only --no-renames --cached >actual &&
+	echo sub/sub/file >expect &&
+	test_cmp expect actual
+'
+
+test_expect_success 'grep with all negative' '
+	H=$(git rev-parse HEAD) &&
+	git reset --hard $H &&
+	test_when_finished "git reset --hard $H" &&
+	for path in file sub/file sub/sub/file sub2/file
+	do
+		echo "needle $path" >>"$path" || return 1
+	done &&
+	git -C sub grep -h needle -- ":!sub/" >actual &&
+	cat >expect <<-\EOF &&
+	needle sub/file
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'ls-files with all negative' '
+	git reset --hard &&
+	git -C sub ls-files -- ":!sub/" >actual &&
+	cat >expect <<-\EOF &&
+	file
+	file2
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'rm with all negative' '
+	git reset --hard &&
+	test_when_finished "git reset --hard" &&
+	git -C sub rm -r --cached -- ":!sub/" >actual &&
+	git diff --name-only --no-renames --diff-filter=D --cached >actual &&
+	cat >expect <<-\EOF &&
+	sub/file
+	sub/file2
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'stash with all negative' '
+	H=$(git rev-parse HEAD) &&
+	git reset --hard $H &&
+	test_when_finished "git reset --hard $H" &&
+	for path in file sub/file sub/sub/file sub2/file
+	do
+		echo smudge >>"$path" || return 1
+	done &&
+	git -C sub stash push -m sample -- ":!sub/" &&
+	git diff --name-only --no-renames HEAD >actual &&
+	echo sub/sub/file >expect &&
+	test_cmp expect actual &&
+	git stash show --name-only >actual &&
+	cat >expect <<-\EOF &&
+	file
+	sub/file
+	sub2/file
+	EOF
+	test_cmp expect actual
+'
+
 test_done
-- 
2.36.1-385-g60203f3fdb

RE: [PATCH] pathspec: correct an empty string used as a pathspec element

[rsbecker]

On May 29, 2022 6:40 PM, Junio C Hamano wrote:
>Pathspecs with only negative elements did not work with some commands that
>pass the pathspec along to a subprocess.  For instance,
>
>    $ git add -p -- ':!*.txt'
>
>should add everything except for paths ending in ".txt", but it gets
complaint from
>underlying "diff-index" and aborts.
>
>We used to error out when a pathspec with only negative elements in it,
like the
>one in the above example.  Later, 859b7f1d (pathspec:
>don't error out on all-exclusionary pathspec patterns, 2017-02-07) updated
the
>logic to add an empty string as an extra element.  The intention was to let
the
>extra element to match everything and let the negative ones given by the
user to
>subtract from it.
>
>At around the same time, we were migrating from "an empty string is a valid
>pathspec element that matches everything" to "either a dot or ":/" is used
to
>match all, and an empty string is rejected", between d426430e (pathspec:
warn on
>empty strings as pathspec,
>2016-06-22) and 9e4e8a64 (pathspec: die on empty strings as pathspec,
2017-06-
>06).  I think 9e4e8a64, which happened long after 859b7f1d happened, was
not
>careful enough to turn the empty string 859b7f1d added to either a dot or
":/".
>
>A care should be taken as the definition of "everything" depends on
>subcommand.  For the purpose of "add -p", adding a "." to add everything in
the
>current directory is the right thing to do.  But for some other commands,
":/" (i.e.
>really really everything, even things outside the current subdirectory) is
the right
>choice.
>
>We would break commands in a big way if we get this wrong, so add a handful
of
>test pieces to make sure the resulting code still excludes the paths that
are
>expected and includes "everything" else.

Thanks for the heads up. I have to check into this for some scripting. Not
worried but glad to know.
Thanks,
--Randall

Re: Excluding paths with wildcard not working with add -p

[Robert Dailey]

On Sun, May 29, 2022 at 1:25 PM Junio C Hamano <gitster@pobox.com> wrote:
> Does this command
>
>     git add -p -- . ':^*.cs'
>
> work as you expect?

yes I can confirm this works.

Re: Excluding paths with wildcard not working with add -p

[Junio C Hamano]

Robert Dailey <rcdailey.lists@gmail.com> writes:

> On Sun, May 29, 2022 at 1:25 PM Junio C Hamano <gitster@pobox.com> wrote:
>> Does this command
>>
>>     git add -p -- . ':^*.cs'
>>
>> work as you expect?
>
> yes I can confirm this works.

Thanks for confirming; you helped a lot to come up with a fix that
is found at

https://lore.kernel.org/git/xmqqpmjwx8so.fsf_-_@gitster.g/