[PATCH] diff --no-index: fix logic for paths ending in '/'

[PATCH] diff --no-index: fix logic for paths ending in '/'

[Jacob Keller]

If one of the two provided paths for git diff --no-index ends in a '/',
a failure similar to the following occurs:

  $ git diff --no-index -- /tmp/ /tmp/ ':!'
  fatal: `pos + len' is too far after the end of the buffer

This occurs because of an incorrect calculation of the skip lengths in
diff_no_index(). The code wants to calculate the length of the string,
but add one in case the string doesn't end with a slash.

The method it uses is incorrect, as it always checks the trailing NUL
character of the string. This will never be a '/', so we always add one.
In the event that we *do* have a trailing slash, this will create an
off-by-one length error later when using the skip value.

The most straightforward fix would be to correct the skip1 and skip2
lengths by using ends_with().

However, Johannes made a good point that the existing logic is wasting a
lot of computation. We generate the match string by copying the path in
and then skipping almost all of it immediately with a potentially
expensive memmove() from the strbuf_remove() call. We also re-initialize
the match stringbuf each time we call read_directory_contents.

The read_directory_contents really wants a path that is rooted at the
start of the directory scan. We're currently building this by taking the
full path and stripping out the start portion. Instead, replace this
logic by building up the portion of the match as we go.

Start by initializing two strbuf in diff_no_index containing the empty
string. Pass these into queue_diff, which in turn passes the appropriate
left or right side into read_directory_contents.

As before, we build up the matches by appending elements to the match
path and then clearing them using strbuf_setlen.

In the recursive portion of the queue_diff algorithm, we build up new
match paths the same way that we build up new buffer paths, by appending
the elements and then clearing them with strbuf_setlen after each
iteration. This is cheaper as it avoids repeated allocations, and is a
bit simpler to track what is going on.

Add a couple of test cases that pass in paths already ending in '/', to
ensure the tests cover this regression.

Reported-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Closes: https://lore.kernel.org/git/c75ec5f9-407a-6555-d4fb-bb629d54ec61@gmx.de/
Signed-off-by: Jacob Keller <jacob.e.keller@intel.com>
---
This fix might feel overly complex. We can drop this and just go with a
simple ends_with() fix, but that leaves the needless strbuf_remove() in the
read_directory_contents.

I tried some alternatives in that function, including trying to use
&path[skip], but this doesn't quite work since path might not end with a
slash, while the skip length does need to account for the possible slash,
since we don't want to end up accidentally using an absolute path.
---
 diff-no-index.c          | 61 ++++++++++++++++++++++++++----------------------
 t/t4053-diff-no-index.sh | 16 +++++++++++++
 2 files changed, 49 insertions(+), 28 deletions(-)

diff --git a/diff-no-index.c b/diff-no-index.c
index 88ae4cee56ba..c70f82b80559 100644
--- a/diff-no-index.c
+++ b/diff-no-index.c
@@ -21,30 +21,21 @@
 
 static int read_directory_contents(const char *path, struct string_list *list,
 				   const struct pathspec *pathspec,
-				   int skip)
+				   struct strbuf *match)
 {
-	struct strbuf match = STRBUF_INIT;
-	int len;
+	int len = match->len;
 	DIR *dir;
 	struct dirent *e;
 
 	if (!(dir = opendir(path)))
 		return error("Could not open directory %s", path);
 
-	if (pathspec) {
-		strbuf_addstr(&match, path);
-		strbuf_complete(&match, '/');
-		strbuf_remove(&match, 0, skip);
-
-		len = match.len;
-	}
-
 	while ((e = readdir_skip_dot_and_dotdot(dir))) {
 		if (pathspec) {
 			int is_dir = 0;
 
-			strbuf_setlen(&match, len);
-			strbuf_addstr(&match, e->d_name);
+			strbuf_setlen(match, len);
+			strbuf_addstr(match, e->d_name);
 			if (NOT_CONSTANT(DTYPE(e)) != DT_UNKNOWN) {
 				is_dir = (DTYPE(e) == DT_DIR);
 			} else {
@@ -57,7 +48,7 @@ static int read_directory_contents(const char *path, struct string_list *list,
 			}
 
 			if (!match_leading_pathspec(NULL, pathspec,
-						    match.buf, match.len,
+						    match->buf, match->len,
 						    0, NULL, is_dir))
 				continue;
 		}
@@ -65,7 +56,7 @@ static int read_directory_contents(const char *path, struct string_list *list,
 		string_list_insert(list, e->d_name);
 	}
 
-	strbuf_release(&match);
+	strbuf_setlen(match, len);
 	closedir(dir);
 	return 0;
 }
@@ -169,7 +160,8 @@ static struct diff_filespec *noindex_filespec(const struct git_hash_algo *algop,
 
 static int queue_diff(struct diff_options *o, const struct git_hash_algo *algop,
 		      const char *name1, const char *name2, int recursing,
-		      const struct pathspec *ps, int skip1, int skip2)
+		      const struct pathspec *ps,
+		      struct strbuf *ps_match1, struct strbuf *ps_match2)
 {
 	int mode1 = 0, mode2 = 0;
 	enum special special1 = SPECIAL_NONE, special2 = SPECIAL_NONE;
@@ -208,10 +200,12 @@ static int queue_diff(struct diff_options *o, const struct git_hash_algo *algop,
 		struct string_list p2 = STRING_LIST_INIT_DUP;
 		int i1, i2, ret = 0;
 		size_t len1 = 0, len2 = 0;
+		size_t match1_len = ps_match1->len;
+		size_t match2_len = ps_match2->len;
 
-		if (name1 && read_directory_contents(name1, &p1, ps, skip1))
+		if (name1 && read_directory_contents(name1, &p1, ps, ps_match1))
 			return -1;
-		if (name2 && read_directory_contents(name2, &p2, ps, skip2)) {
+		if (name2 && read_directory_contents(name2, &p2, ps, ps_match2)) {
 			string_list_clear(&p1, 0);
 			return -1;
 		}
@@ -235,6 +229,11 @@ static int queue_diff(struct diff_options *o, const struct git_hash_algo *algop,
 			strbuf_setlen(&buffer1, len1);
 			strbuf_setlen(&buffer2, len2);
 
+			if (ps) {
+				strbuf_setlen(ps_match1, match1_len);
+				strbuf_setlen(ps_match2, match2_len);
+			}
+
 			if (i1 == p1.nr)
 				comp = 1;
 			else if (i2 == p2.nr)
@@ -245,18 +244,28 @@ static int queue_diff(struct diff_options *o, const struct git_hash_algo *algop,
 			if (comp > 0)
 				n1 = NULL;
 			else {
-				strbuf_addstr(&buffer1, p1.items[i1++].string);
+				strbuf_addstr(&buffer1, p1.items[i1].string);
+				if (ps) {
+					strbuf_addstr(ps_match1, p1.items[i1].string);
+					strbuf_complete(ps_match1, '/');
+				}
 				n1 = buffer1.buf;
+				i1++;
 			}
 
 			if (comp < 0)
 				n2 = NULL;
 			else {
-				strbuf_addstr(&buffer2, p2.items[i2++].string);
+				strbuf_addstr(&buffer2, p2.items[i2].string);
+				if (ps) {
+					strbuf_addstr(ps_match2, p2.items[i2].string);
+					strbuf_complete(ps_match2, '/');
+				}
 				n2 = buffer2.buf;
+				i2++;
 			}
 
-			ret = queue_diff(o, algop, n1, n2, 1, ps, skip1, skip2);
+			ret = queue_diff(o, algop, n1, n2, 1, ps, ps_match1, ps_match2);
 		}
 		string_list_clear(&p1, 0);
 		string_list_clear(&p2, 0);
@@ -346,7 +355,8 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
 		  int implicit_no_index, int argc, const char **argv)
 {
 	struct pathspec pathspec, *ps = NULL;
-	int i, no_index, skip1 = 0, skip2 = 0;
+	struct strbuf ps_match1 = STRBUF_INIT, ps_match2 = STRBUF_INIT;
+	int i, no_index;
 	int ret = 1;
 	const char *paths[2];
 	char *to_free[ARRAY_SIZE(paths)] = { 0 };
@@ -387,11 +397,6 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
 			       NULL, &argv[2]);
 		if (pathspec.nr)
 			ps = &pathspec;
-
-		skip1 = strlen(paths[0]);
-		skip1 += paths[0][skip1] == '/' ? 0 : 1;
-		skip2 = strlen(paths[1]);
-		skip2 += paths[1][skip2] == '/' ? 0 : 1;
 	} else if (argc > 2) {
 		warning(_("Limiting comparison with pathspecs is only "
 			  "supported if both paths are directories."));
@@ -415,7 +420,7 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
 	revs->diffopt.flags.exit_with_status = 1;
 
 	if (queue_diff(&revs->diffopt, algop, paths[0], paths[1], 0, ps,
-		       skip1, skip2))
+		       &ps_match1, &ps_match2))
 		goto out;
 	diff_set_mnemonic_prefix(&revs->diffopt, "1/", "2/");
 	diffcore_std(&revs->diffopt);
diff --git a/t/t4053-diff-no-index.sh b/t/t4053-diff-no-index.sh
index 01db9243abfe..e0ea437685b0 100755
--- a/t/t4053-diff-no-index.sh
+++ b/t/t4053-diff-no-index.sh
@@ -322,6 +322,22 @@ test_expect_success 'diff --no-index with pathspec' '
 	test_cmp expect actual
 '
 
+test_expect_success 'diff --no-index first path ending in slash with pathspec' '
+	test_expect_code 1 git diff --name-status --no-index a/ b 1 >actual &&
+	cat >expect <<-EOF &&
+	D	a/1
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'diff --no-index second path ending in slash with pathspec' '
+	test_expect_code 1 git diff --name-status --no-index a b/ 1 >actual &&
+	cat >expect <<-EOF &&
+	D	a/1
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success 'diff --no-index with pathspec no matches' '
 	test_expect_code 0 git diff --name-status --no-index a b missing
 '

---
base-commit: 8d2709d075d65ba386a4dac157129ef868c283e5
change-id: 20250924-jk-fix-no-index-path-with-slash-788785be16c1

Best regards,
--  
Jacob Keller <jacob.e.keller@intel.com>

Re: [PATCH] diff --no-index: fix logic for paths ending in '/'

[Junio C Hamano]

Jacob Keller <jacob.e.keller@intel.com> writes:

> If one of the two provided paths for git diff --no-index ends in a '/',
> a failure similar to the following occurs:
>
>   $ git diff --no-index -- /tmp/ /tmp/ ':!'
>   fatal: `pos + len' is too far after the end of the buffer
>
> This occurs because of an incorrect calculation of the skip lengths in
> diff_no_index(). The code wants to calculate the length of the string,
> but add one in case the string doesn't end with a slash.
>
> The method it uses is incorrect, as it always checks the trailing NUL
> character of the string. This will never be a '/', so we always add one.
> In the event that we *do* have a trailing slash, this will create an
> off-by-one length error later when using the skip value.
>
> The most straightforward fix would be to correct the skip1 and skip2
> lengths by using ends_with().

Meaning, "ah, this one ends with '/' so let's trim it and do
everything else the same as before"?

It certainly is how we usually handle regression fixes.

There were two topics that touched "git diff --no-index" in Git 2.51
timeframe, and the pathspec support was a new feature added by them,
so this is not exactly a regression.

> This fix might feel overly complex. We can drop this and just go with a
> simple ends_with() fix, but that leaves the needless strbuf_remove() in the
> read_directory_contents.

We can do it as a two-step patch series, which would allow us to
revert the more complex part relatively cleanly if it turns out to
break the cases that the current code handles fine.  But as this is
a new feature in 2.51, it is OK to declare that the code never
worked correctly and we are making it right this time with a single
more ambitious patch.

Will queue.  Thanks, both of you.

Re: [PATCH] diff --no-index: fix logic for paths ending in '/'

[Junio C Hamano]

Jacob Keller <jacob.e.keller@intel.com> writes:

> diff --git a/diff-no-index.c b/diff-no-index.c
> index 88ae4cee56ba..c70f82b80559 100644
> --- a/diff-no-index.c
> +++ b/diff-no-index.c
> ...
> @@ -346,7 +355,8 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
>  		  int implicit_no_index, int argc, const char **argv)
>  {
>  	struct pathspec pathspec, *ps = NULL;
> -	int i, no_index, skip1 = 0, skip2 = 0;
> +	struct strbuf ps_match1 = STRBUF_INIT, ps_match2 = STRBUF_INIT;
> +	int i, no_index;
>  	int ret = 1;
>  	const char *paths[2];
>  	char *to_free[ARRAY_SIZE(paths)] = { 0 };
> @@ -387,11 +397,6 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
>  			       NULL, &argv[2]);
>  		if (pathspec.nr)
>  			ps = &pathspec;
> -
> -		skip1 = strlen(paths[0]);
> -		skip1 += paths[0][skip1] == '/' ? 0 : 1;
> -		skip2 = strlen(paths[1]);
> -		skip2 += paths[1][skip2] == '/' ? 0 : 1;
>  	} else if (argc > 2) {
>  		warning(_("Limiting comparison with pathspecs is only "
>  			  "supported if both paths are directories."));
> @@ -415,7 +420,7 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
>  	revs->diffopt.flags.exit_with_status = 1;
>  
>  	if (queue_diff(&revs->diffopt, algop, paths[0], paths[1], 0, ps,
> -		       skip1, skip2))
> +		       &ps_match1, &ps_match2))

Inside queue_diff() that makes recursive calls to itself, lenthens
these strbuf to hold longer paths while using setlen when it wants
to trim the tail end of the paths.

So it is likely that ps_match.buf would never become NUL even when
ps_match.len goes down to 0 after the recursion and queue_diff()
uses setlen to trim the string back to what was originally in there.

Hence, I think the clean-up code of this function this goto ...

>  		goto out;

... jumps to would need

	strbuf_release(&ps_match1);
	strbuf_release(&ps_match2);

added after that "out:" label?

If we run this test with leak sanitizer, wouldn't it find leak in
these (I haven't tried it myself---I just am speculating)?

> diff --git a/t/t4053-diff-no-index.sh b/t/t4053-diff-no-index.sh
> index 01db9243abfe..e0ea437685b0 100755
> --- a/t/t4053-diff-no-index.sh
> +++ b/t/t4053-diff-no-index.sh
> @@ -322,6 +322,22 @@ test_expect_success 'diff --no-index with pathspec' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 'diff --no-index first path ending in slash with pathspec' '
> +	test_expect_code 1 git diff --name-status --no-index a/ b 1 >actual &&
> +	cat >expect <<-EOF &&
> +	D	a/1
> +	EOF
> +	test_cmp expect actual
> +'
> +
> +test_expect_success 'diff --no-index second path ending in slash with pathspec' '
> +	test_expect_code 1 git diff --name-status --no-index a b/ 1 >actual &&
> +	cat >expect <<-EOF &&
> +	D	a/1
> +	EOF
> +	test_cmp expect actual
> +'
> +
>  test_expect_success 'diff --no-index with pathspec no matches' '
>  	test_expect_code 0 git diff --name-status --no-index a b missing
>  '

Re: [PATCH] diff --no-index: fix logic for paths ending in '/'

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Hence, I think the clean-up code of this function this goto ...
>
>>  		goto out;
>
> ... jumps to would need
>
> 	strbuf_release(&ps_match1);
> 	strbuf_release(&ps_match2);
>
> added after that "out:" label?
>
> If we run this test with leak sanitizer, wouldn't it find leak in
> these (I haven't tried it myself---I just am speculating)?

Now I did, and my speculations were both correct.  The SANITIZE=leak
build fails, and with these two releases the test passes.

You can squash this in, or I can do so myself if you like, if this
is the only change that is required.

 diff-no-index.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git c/diff-no-index.c w/diff-no-index.c
index c70f82b805..f320424f05 100644
--- c/diff-no-index.c
+++ w/diff-no-index.c
@@ -436,6 +436,8 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
 	for (i = 0; i < ARRAY_SIZE(to_free); i++)
 		free(to_free[i]);
 	strbuf_release(&replacement);
+	strbuf_release(&ps_match1);
+	strbuf_release(&ps_match2);
 	if (ps)
 		clear_pathspec(ps);
 	return ret;

Re: [PATCH] diff --no-index: fix logic for paths ending in '/'

[Jacob Keller]

[-- Attachment #1.1: Type: text/plain, Size: 1320 bytes --]



On 9/24/2025 3:24 PM, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
> 
>> Hence, I think the clean-up code of this function this goto ...
>>
>>>  		goto out;
>>
>> ... jumps to would need
>>
>> 	strbuf_release(&ps_match1);
>> 	strbuf_release(&ps_match2);
>>
>> added after that "out:" label?
>>
>> If we run this test with leak sanitizer, wouldn't it find leak in
>> these (I haven't tried it myself---I just am speculating)?
> 
> Now I did, and my speculations were both correct.  The SANITIZE=leak
> build fails, and with these two releases the test passes.
> 
> You can squash this in, or I can do so myself if you like, if this
> is the only change that is required.
> 
>  diff-no-index.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git c/diff-no-index.c w/diff-no-index.c
> index c70f82b805..f320424f05 100644
> --- c/diff-no-index.c
> +++ w/diff-no-index.c
> @@ -436,6 +436,8 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
>  	for (i = 0; i < ARRAY_SIZE(to_free); i++)
>  		free(to_free[i]);
>  	strbuf_release(&replacement);
> +	strbuf_release(&ps_match1);
> +	strbuf_release(&ps_match2);
>  	if (ps)
>  		clear_pathspec(ps);
>  	return ret;

Please squash this in. I'll fix it if we need a v2 otherwise.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [PATCH] diff --no-index: fix logic for paths ending in '/'

[Junio C Hamano]

Jacob Keller <jacob.e.keller@intel.com> writes:

>> Now I did, and my speculations were both correct.  The SANITIZE=leak
>> build fails, and with these two releases the test passes.
>> 
>> You can squash this in, or I can do so myself if you like, if this
>> is the only change that is required.

Will do.  Thanks.

Re: [PATCH] diff --no-index: fix logic for paths ending in '/'

[Junio C Hamano]

Jacob Keller <jacob.e.keller@intel.com> writes:

>> You can squash this in, or I can do so myself if you like, if this
>> is the only change that is required.
>> 
>>  diff-no-index.c | 2 ++
>>  1 file changed, 2 insertions(+)
>> 
>> diff --git c/diff-no-index.c w/diff-no-index.c
>> index c70f82b805..f320424f05 100644
>> --- c/diff-no-index.c
>> +++ w/diff-no-index.c
>> @@ -436,6 +436,8 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
>>  	for (i = 0; i < ARRAY_SIZE(to_free); i++)
>>  		free(to_free[i]);
>>  	strbuf_release(&replacement);
>> +	strbuf_release(&ps_match1);
>> +	strbuf_release(&ps_match2);
>>  	if (ps)
>>  		clear_pathspec(ps);
>>  	return ret;
>
> Please squash this in. I'll fix it if we need a v2 otherwise.

Since this exchange things have gone quiet, so shall we declare
victory and merge it down to 'next' and then eventually to 'master'?

Thanks.

Re: [PATCH] diff --no-index: fix logic for paths ending in '/'

[Jacob Keller]

[-- Attachment #1.1: Type: text/plain, Size: 1171 bytes --]



On 10/10/2025 9:13 AM, Junio C Hamano wrote:
> Jacob Keller <jacob.e.keller@intel.com> writes:
> 
>>> You can squash this in, or I can do so myself if you like, if this
>>> is the only change that is required.
>>>
>>>  diff-no-index.c | 2 ++
>>>  1 file changed, 2 insertions(+)
>>>
>>> diff --git c/diff-no-index.c w/diff-no-index.c
>>> index c70f82b805..f320424f05 100644
>>> --- c/diff-no-index.c
>>> +++ w/diff-no-index.c
>>> @@ -436,6 +436,8 @@ int diff_no_index(struct rev_info *revs, const struct git_hash_algo *algop,
>>>  	for (i = 0; i < ARRAY_SIZE(to_free); i++)
>>>  		free(to_free[i]);
>>>  	strbuf_release(&replacement);
>>> +	strbuf_release(&ps_match1);
>>> +	strbuf_release(&ps_match2);
>>>  	if (ps)
>>>  		clear_pathspec(ps);
>>>  	return ret;
>>
>> Please squash this in. I'll fix it if we need a v2 otherwise.
> 
> Since this exchange things have gone quiet, so shall we declare
> victory and merge it down to 'next' and then eventually to 'master'?
> 
> Thanks.

That would be my preference. I haven't seen any other reports or
comments about this since the v1 besides the missing strbuf release.

Thanks,
Jake

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]