git-add ignores exclude markers for ignored files

git-add ignores exclude markers for ignored files

[Rahn, René]

Hey all,

I’ve run into a strange behavior of git-add that I believe is not correct. I couldn’t find something on the mailing list regarding this, so I am writing a new mail. The issue is that if you use git-add with an explicit pathspec with an exclude pattern of a file that is ignored by some .gitignore file (locally or globally) the git-add command errors with the following message:
 
git add --":(exclude)ignored.txt"
The following paths are ignored by one of your ".gitignore" files:
ignored.txt
Note: Use -f if you really want to add them.
Note: Disable this message with "git config set advice.addIgnoredFile false"

Steps to reproduce:

mkdir git-add-test 
cd git-add-test
git init
touch .gitignore
echo “ignored.txt” >>.gitignore
touch ignored.txt
git add --“:(exclude)ignored.txt”
 
Expected behavior:
 
The git add command recognizes that the file ignored.txt is already excluded despite of it being ignored by some gitignore and thus does not check if it is ignored or not. It simply will not be added. 
Note forcing git-add will do the trick, but this could also have side effects for some files that are ignored but not present in the list of excluded files.
Hence, this can’t be the right solution.

Any advice or confirmation are much appreciated.
 
Thank you
René Rahn

Re: git-add ignores exclude markers for ignored files

[Junio C Hamano]

"Rahn, René" <Rene.Rahn@pfizer.com> writes:

> Steps to reproduce:
>
> mkdir git-add-test
> cd git-add-test
> git init
> touch .gitignore
> echo “ignored.txt” >>.gitignore
> touch ignored.txt
> git add --“:(exclude)ignored.txt”
> 
> Expected behavior:
> 
> The git add command recognizes that the file ignored.txt is
> already excluded despite of it being ignored by some gitignore and
> thus does not check if it is ignored or not. It simply will not be
> added.  Note forcing git-add will do the trick, but this could
> also have side effects for some files that are ignored but not
> present in the list of excluded files.  Hence, this can’t be the
> right solution.

It is not quite clear what you want to see.  The command would not
add ignored.text even if you give ":(exclude)ignored.txt" from the
command line, would it?

This may be an ancient regression when e1b8c7bd (dir: remove struct
path_simplify, 2017-01-04) was rewritten exclude_matches_pathspec()
function, which was written in 29209cbe (dir: fix COLLECT_IGNORED on
excluded prefixes, 2010-03-11), back in the days before ":(exclude)"
and other pathspec magic was even invented.

Perhaps try this patch?

I have no idea what the ramifications of the change is, though.
There may be unintended fallouts in some distant corner, even though
it does not seem to break any existing tests.

----- >8 -----
Subject: dir.c: do not be fooled by :(exclude) pathspec elements

When exclude_matches_pathspec() tries to determine if an otherwise
excluded item matches the pathspec given, it goes through each
pathspec element and declares a hit, without checking if the element
is a negative ":(exclude)" element.  Fix it be applying the usual "a
path matches if it matches any one of positive pathspec element, and
if it matches none of negative pathspec elements" rule in the
function.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
diff --git a/dir.c b/dir.c
index 5b2181e589..5b1258a09d 100644
--- a/dir.c
+++ b/dir.c
@@ -2219,6 +2219,8 @@ static int exclude_matches_pathspec(const char *path, int pathlen,
 				    const struct pathspec *pathspec)
 {
 	int i;
+	int matches_exclude_magic = 0;
+	int matches_pathspec_elem = 0;
 
 	if (!pathspec || !pathspec->nr)
 		return 0;
@@ -2235,15 +2237,23 @@ static int exclude_matches_pathspec(const char *path, int pathlen,
 	for (i = 0; i < pathspec->nr; i++) {
 		const struct pathspec_item *item = &pathspec->items[i];
 		int len = item->nowildcard_len;
+		int *matches;
+
+		if (item->magic & PATHSPEC_EXCLUDE)
+			matches = &matches_exclude_magic;
+		else
+			matches = &matches_pathspec_elem;
 
 		if (len == pathlen &&
 		    !ps_strncmp(item, item->match, path, pathlen))
-			return 1;
+			*matches = 1;
 		if (len > pathlen &&
 		    item->match[pathlen] == '/' &&
 		    !ps_strncmp(item, item->match, path, pathlen))
-			return 1;
+			*matches = 1;
 	}
+	if (matches_pathspec_elem && !matches_exclude_magic)
+		return 1;
 	return 0;
 }
 
diff --git a/t/t2204-add-ignored.sh b/t/t2204-add-ignored.sh
index 31eb233df5..aa55b219ab 100755
--- a/t/t2204-add-ignored.sh
+++ b/t/t2204-add-ignored.sh
@@ -89,4 +89,21 @@ do
 	'
 done
 
+test_expect_success "exclude magic would not interfere with .gitignore" '
+	test_write_lines dir file sub ign err out "*.o" >.gitignore &&
+	>foo.o &&
+	>foo.c &&
+	test_must_fail git add foo.o 2>err &&
+	test_grep "are ignored by one" err &&
+	test_grep "hint: Use -f" err &&
+
+	git add ":(exclude)foo.o" &&
+	git ls-files >actual &&
+	cat >expect <<-\EOF &&
+	.gitignore
+	foo.c
+	EOF
+	test_cmp expect actual
+'
+
 test_done

Re: git-add ignores exclude markers for ignored files

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> This may be an ancient regression when e1b8c7bd (dir: remove struct
> path_simplify, 2017-01-04) was rewritten exclude_matches_pathspec()
> function, which was written in 29209cbe (dir: fix COLLECT_IGNORED on
> excluded prefixes, 2010-03-11), back in the days before ":(exclude)"
> and other pathspec magic was even invented.
>
> Perhaps try this patch?
>
> I have no idea what the ramifications of the change is, though.
> There may be unintended fallouts in some distant corner, even though
> it does not seem to break any existing tests.

One thing that I am extremely uncomfortable about this change is
that I do not quite see why we use this function to begin with.
Given a <path, pathlen>, it seems if the pathspec matches, but why
aren't we using dir.c:match_pathspec() for and have our own separate
and much simple-minded variant instead?

Re: git-add ignores exclude markers for ignored files

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> "Rahn, René" <Rene.Rahn@pfizer.com> writes:
>
>> Steps to reproduce:
>>
>> mkdir git-add-test
>> cd git-add-test
>> git init
>> touch .gitignore
>> echo “ignored.txt” >>.gitignore
>> touch ignored.txt
>> git add --“:(exclude)ignored.txt”
>> 
>> Expected behavior:
>> 
>> The git add command recognizes that the file ignored.txt is
>> already excluded despite of it being ignored by some gitignore and
>> thus does not check if it is ignored or not. It simply will not be
>> added.  Note forcing git-add will do the trick, but this could
>> also have side effects for some files that are ignored but not
>> present in the list of excluded files.  Hence, this can’t be the
>> right solution.
>
> It is not quite clear what you want to see.  The command would not
> add ignored.text even if you give ":(exclude)ignored.txt" from the
> command line, would it?
>
> This may be an ancient regression when e1b8c7bd (dir: remove struct
> path_simplify, 2017-01-04) was rewritten exclude_matches_pathspec()
> function, which was written in 29209cbe (dir: fix COLLECT_IGNORED on
> excluded prefixes, 2010-03-11), back in the days before ":(exclude)"
> and other pathspec magic was even invented.
>
> Perhaps try this patch?
>
> I have no idea what the ramifications of the change is, though.
> There may be unintended fallouts in some distant corner, even though
> it does not seem to break any existing tests.

As I haven't heard from anybody if this patch improves the reported
situation or there are unintended behaviour changes that degrades
the system, I'll drop this topic from 'seen' soonish, as this topic
is not exactly my itch.  That does not mean that it is unwelcome if
motivated others pick up the patch and polish it to bring it to the
finish line.  Anybody doing so should consult the original
discussion thread [*].

Thanks.


[Reference]

* https://lore.kernel.org/git/xmqqtsz9o3cn.fsf@gitster.g/