[PATCH] diff: run arguments through precompose_argv

[PATCH] diff: run arguments through precompose_argv

[Alexander Rinass]

I found an issue where file paths containing decomposed unicode chars
are not converted to precomposed unicode form when passed to the diff
command family.

As a result, no diff is displayed.

With the help of Torsten Bögershausen and Junio C Hamano I came up 
with the following patch to fix the issue.

Alexander Rinass (1):
  diff: run arguments through precompose_argv

 builtin/diff-files.c         |  1 +
 builtin/diff-index.c         |  1 +
 builtin/diff-tree.c          |  2 ++
 builtin/diff.c               |  1 +
 t/t3910-mac-os-precompose.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 47 insertions(+)

-- 
2.7.2

[PATCH] diff: run arguments through precompose_argv

[Alexander Rinass]

File paths containing decomposed unicode chars passed to diff
command are not converted to precomposed unicode form.

As a result, no diff is displayed when feeding such a file path to the
diff command.

Opposite to most builtin commands, the diff builtin is missing the
parse_options call, which internally runs arguments through the
precompose_argv call, which ensures all arguments are in precomposed
unicode form.

Fix the problem by adding a precompose_argv call directly, as a call to
parse_options would require additional work to call it.

Also applies to diff-index, diff-files and diff-tree.

Signed-off-by: Alexander Rinass <alex@fournova.com>
Thanks-to: Torsten Bögershausen <tboegi@web.de>
Thanks-to: Junio C Hamano <gitster@pobox.com>
---
 builtin/diff-files.c         |  1 +
 builtin/diff-index.c         |  1 +
 builtin/diff-tree.c          |  2 ++
 builtin/diff.c               |  1 +
 t/t3910-mac-os-precompose.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 47 insertions(+)

diff --git a/builtin/diff-files.c b/builtin/diff-files.c
index 8ed2eb8..15c61fd 100644
--- a/builtin/diff-files.c
+++ b/builtin/diff-files.c
@@ -24,6 +24,7 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix)
 	gitmodules_config();
 	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
 	rev.abbrev = 0;
+	precompose_argv(argc, argv);
 
 	argc = setup_revisions(argc, argv, &rev, NULL);
 	while (1 < argc && argv[1][0] == '-') {
diff --git a/builtin/diff-index.c b/builtin/diff-index.c
index d979824..1af373d 100644
--- a/builtin/diff-index.c
+++ b/builtin/diff-index.c
@@ -21,6 +21,7 @@ int cmd_diff_index(int argc, const char **argv, const char *prefix)
 	gitmodules_config();
 	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
 	rev.abbrev = 0;
+	precompose_argv(argc, argv);
 
 	argc = setup_revisions(argc, argv, &rev, NULL);
 	for (i = 1; i < argc; i++) {
diff --git a/builtin/diff-tree.c b/builtin/diff-tree.c
index 2a12b81..806dd7a 100644
--- a/builtin/diff-tree.c
+++ b/builtin/diff-tree.c
@@ -114,6 +114,8 @@ int cmd_diff_tree(int argc, const char **argv, const char *prefix)
 	opt->disable_stdin = 1;
 	memset(&s_r_opt, 0, sizeof(s_r_opt));
 	s_r_opt.tweak = diff_tree_tweak_rev;
+
+	precompose_argv(argc, argv);
 	argc = setup_revisions(argc, argv, opt, &s_r_opt);
 
 	while (--argc > 0) {
diff --git a/builtin/diff.c b/builtin/diff.c
index 52c98a9..d6b8f98 100644
--- a/builtin/diff.c
+++ b/builtin/diff.c
@@ -319,6 +319,7 @@ int cmd_diff(int argc, const char **argv, const char *prefix)
 	if (!no_index)
 		gitmodules_config();
 	git_config(git_diff_ui_config, NULL);
+	precompose_argv(argc, argv);
 
 	init_revisions(&rev, prefix);
 
diff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh
index 8319356..26dd5b7 100755
--- a/t/t3910-mac-os-precompose.sh
+++ b/t/t3910-mac-os-precompose.sh
@@ -49,12 +49,54 @@ test_expect_success "setup" '
 test_expect_success "setup case mac" '
 	git checkout -b mac_os
 '
+# This will test nfd2nfc in git diff
+test_expect_success "git diff f.Adiar" '
+	touch f.$Adiarnfc &&
+	git add f.$Adiarnfc &&
+	echo f.Adiarnfc >f.$Adiarnfc &&
+	git diff f.$Adiarnfd >expect &&
+	git diff f.$Adiarnfc >actual &&
+	test_cmp expect actual &&
+	git reset HEAD f.Adiarnfc &&
+	rm f.$Adiarnfc expect actual
+'
+# This will test nfd2nfc in git diff-files
+test_expect_success "git diff-files f.Adiar" '
+	touch f.$Adiarnfc &&
+	git add f.$Adiarnfc &&
+	echo f.Adiarnfc >f.$Adiarnfc &&
+	git diff-files f.$Adiarnfd >expect &&
+	git diff-files f.$Adiarnfc >actual &&
+	test_cmp expect actual &&
+	git reset HEAD f.Adiarnfc &&
+	rm f.$Adiarnfc expect actual
+'
+# This will test nfd2nfc in git diff-index
+test_expect_success "git diff-index f.Adiar" '
+	touch f.$Adiarnfc &&
+	git add f.$Adiarnfc &&
+	echo f.Adiarnfc >f.$Adiarnfc &&
+	git diff-index HEAD f.$Adiarnfd >expect &&
+	git diff-index HEAD f.$Adiarnfc >actual &&
+	test_cmp expect actual &&
+	git reset HEAD f.Adiarnfc &&
+	rm f.$Adiarnfc expect actual
+'
 # This will test nfd2nfc in readdir()
 test_expect_success "add file Adiarnfc" '
 	echo f.Adiarnfc >f.$Adiarnfc &&
 	git add f.$Adiarnfc &&
 	git commit -m "add f.$Adiarnfc"
 '
+# This will test nfd2nfc in git diff-tree
+test_expect_success "git diff-tree f.Adiar" '
+	echo f.Adiarnfc >>f.$Adiarnfc &&
+	git diff-tree HEAD f.$Adiarnfd >expect &&
+	git diff-tree HEAD f.$Adiarnfc >actual &&
+	test_cmp expect actual &&
+	git checkout f.$Adiarnfc &&
+	rm expect actual
+'
 # This will test nfd2nfc in git stage()
 test_expect_success "stage file d.Adiarnfd/f.Adiarnfd" '
 	mkdir d.$Adiarnfd &&
-- 
2.7.2

Re: [PATCH] diff: run arguments through precompose_argv

[Junio C Hamano]

Alexander Rinass <alex@fournova.com> writes:

> File paths containing decomposed unicode chars passed to diff
> command are not converted to precomposed unicode form.
>
> As a result, no diff is displayed when feeding such a file path to the
> diff command.
>
> Opposite to most builtin commands, the diff builtin is missing the
> parse_options call, which internally runs arguments through the
> precompose_argv call, which ensures all arguments are in precomposed
> unicode form.
>
> Fix the problem by adding a precompose_argv call directly, as a call to
> parse_options would require additional work to call it.
>
> Also applies to diff-index, diff-files and diff-tree.

Thanks.  The log message talks about "such a file path", but
precompose_argv() applies indiscriminately on any command line
arguments, so things like -G<string> would also get the same
treatment, which I think is what most users would want).

Will queue.

> Signed-off-by: Alexander Rinass <alex@fournova.com>
> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> Thanks-to: Junio C Hamano <gitster@pobox.com>
> ---
>  builtin/diff-files.c         |  1 +
>  builtin/diff-index.c         |  1 +
>  builtin/diff-tree.c          |  2 ++
>  builtin/diff.c               |  1 +
>  t/t3910-mac-os-precompose.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 47 insertions(+)
>
> diff --git a/builtin/diff-files.c b/builtin/diff-files.c
> index 8ed2eb8..15c61fd 100644
> --- a/builtin/diff-files.c
> +++ b/builtin/diff-files.c
> @@ -24,6 +24,7 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix)
>  	gitmodules_config();
>  	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
>  	rev.abbrev = 0;
> +	precompose_argv(argc, argv);
>  
>  	argc = setup_revisions(argc, argv, &rev, NULL);
>  	while (1 < argc && argv[1][0] == '-') {
> diff --git a/builtin/diff-index.c b/builtin/diff-index.c
> index d979824..1af373d 100644
> --- a/builtin/diff-index.c
> +++ b/builtin/diff-index.c
> @@ -21,6 +21,7 @@ int cmd_diff_index(int argc, const char **argv, const char *prefix)
>  	gitmodules_config();
>  	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
>  	rev.abbrev = 0;
> +	precompose_argv(argc, argv);
>  
>  	argc = setup_revisions(argc, argv, &rev, NULL);
>  	for (i = 1; i < argc; i++) {
> diff --git a/builtin/diff-tree.c b/builtin/diff-tree.c
> index 2a12b81..806dd7a 100644
> --- a/builtin/diff-tree.c
> +++ b/builtin/diff-tree.c
> @@ -114,6 +114,8 @@ int cmd_diff_tree(int argc, const char **argv, const char *prefix)
>  	opt->disable_stdin = 1;
>  	memset(&s_r_opt, 0, sizeof(s_r_opt));
>  	s_r_opt.tweak = diff_tree_tweak_rev;
> +
> +	precompose_argv(argc, argv);
>  	argc = setup_revisions(argc, argv, opt, &s_r_opt);
>  
>  	while (--argc > 0) {
> diff --git a/builtin/diff.c b/builtin/diff.c
> index 52c98a9..d6b8f98 100644
> --- a/builtin/diff.c
> +++ b/builtin/diff.c
> @@ -319,6 +319,7 @@ int cmd_diff(int argc, const char **argv, const char *prefix)
>  	if (!no_index)
>  		gitmodules_config();
>  	git_config(git_diff_ui_config, NULL);
> +	precompose_argv(argc, argv);
>  
>  	init_revisions(&rev, prefix);
>  
> diff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh
> index 8319356..26dd5b7 100755
> --- a/t/t3910-mac-os-precompose.sh
> +++ b/t/t3910-mac-os-precompose.sh
> @@ -49,12 +49,54 @@ test_expect_success "setup" '
>  test_expect_success "setup case mac" '
>  	git checkout -b mac_os
>  '
> +# This will test nfd2nfc in git diff
> +test_expect_success "git diff f.Adiar" '
> +	touch f.$Adiarnfc &&
> +	git add f.$Adiarnfc &&
> +	echo f.Adiarnfc >f.$Adiarnfc &&
> +	git diff f.$Adiarnfd >expect &&
> +	git diff f.$Adiarnfc >actual &&
> +	test_cmp expect actual &&
> +	git reset HEAD f.Adiarnfc &&
> +	rm f.$Adiarnfc expect actual
> +'
> +# This will test nfd2nfc in git diff-files
> +test_expect_success "git diff-files f.Adiar" '
> +	touch f.$Adiarnfc &&
> +	git add f.$Adiarnfc &&
> +	echo f.Adiarnfc >f.$Adiarnfc &&
> +	git diff-files f.$Adiarnfd >expect &&
> +	git diff-files f.$Adiarnfc >actual &&
> +	test_cmp expect actual &&
> +	git reset HEAD f.Adiarnfc &&
> +	rm f.$Adiarnfc expect actual
> +'
> +# This will test nfd2nfc in git diff-index
> +test_expect_success "git diff-index f.Adiar" '
> +	touch f.$Adiarnfc &&
> +	git add f.$Adiarnfc &&
> +	echo f.Adiarnfc >f.$Adiarnfc &&
> +	git diff-index HEAD f.$Adiarnfd >expect &&
> +	git diff-index HEAD f.$Adiarnfc >actual &&
> +	test_cmp expect actual &&
> +	git reset HEAD f.Adiarnfc &&
> +	rm f.$Adiarnfc expect actual
> +'
>  # This will test nfd2nfc in readdir()
>  test_expect_success "add file Adiarnfc" '
>  	echo f.Adiarnfc >f.$Adiarnfc &&
>  	git add f.$Adiarnfc &&
>  	git commit -m "add f.$Adiarnfc"
>  '
> +# This will test nfd2nfc in git diff-tree
> +test_expect_success "git diff-tree f.Adiar" '
> +	echo f.Adiarnfc >>f.$Adiarnfc &&
> +	git diff-tree HEAD f.$Adiarnfd >expect &&
> +	git diff-tree HEAD f.$Adiarnfc >actual &&
> +	test_cmp expect actual &&
> +	git checkout f.$Adiarnfc &&
> +	rm expect actual
> +'
>  # This will test nfd2nfc in git stage()
>  test_expect_success "stage file d.Adiarnfd/f.Adiarnfd" '
>  	mkdir d.$Adiarnfd &&

Re: [PATCH] diff: run arguments through precompose_argv

[Johannes Sixt]

Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>> Thanks-to: Torsten Bögershausen <tboegi@web.de>

I sense NFD disease: The combining diaresis should combine with the o, 
not the g. Here is a correct line to copy-and-paste if you like:

Thanks-to: Torsten Bögershausen <tboegi@web.de>

-- Hannes

Re: [PATCH] diff: run arguments through precompose_argv

[Torsten Bögershausen]

On 05.04.16 21:15, Johannes Sixt wrote:
> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> 
> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
> 
> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> 
> -- Hannes

Good eyes.

And thanks to Alexander for doing this patch

Re: [PATCH] diff: run arguments through precompose_argv

[Junio C Hamano]

Torsten Bögershausen <tboegi@web.de> writes:

> On 05.04.16 21:15, Johannes Sixt wrote:
>> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>> 
>> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
>> 
>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>> 
>> -- Hannes
>
> Good eyes.
>
> And thanks to Alexander for doing this patch

Yup.  It is funny that I wrote:

    Thanks.  The log message talks about "such a file path", but
    precompose_argv() applies indiscriminately on any command line
    arguments, so things like -G<string> would also get the same
    treatment, which I think is what most users would want.

but "git log --grep=" with your name would not have found the one
that would have resulted from the original e-mail message applied.

Re: [PATCH] diff: run arguments through precompose_argv

[Alexander Rinass]

> On 05 Apr 2016, at 21:15, Johannes Sixt <j6t@kdbg.org> wrote:
> 
> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> 
> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
> 
> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> 
> -- Hannes

Thanks for reviewing and catching the NFD encoding error.

I will send in a patch v2 with the correct NFC encoding.

Would you also like me to alter the commit message as mentioned by Junio?

I could rewrite the sentence:

“As a result, no diff is displayed when feeding such a file path to the
diff command.”

into simply saying:

“As a result, no diff is displayed.”

However, I don't read the original message as it would imply that only
file paths are affected by the precompose_argv call. 

Are there other suggestions on improving the commit message?

Re: [PATCH] diff: run arguments through precompose_argv

[Torsten Bögershausen]

On 06.04.16 08:51, Alexander Rinass wrote:
> 
>> On 05 Apr 2016, at 21:15, Johannes Sixt <j6t@kdbg.org> wrote:
>>
>> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>>
>> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
>>
>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>>
>> -- Hannes
> 
> Thanks for reviewing and catching the NFD encoding error.
> 
> I will send in a patch v2 with the correct NFC encoding.
> 
> Would you also like me to alter the commit message as mentioned by Junio?
> 
> I could rewrite the sentence:
> 
> “As a result, no diff is displayed when feeding such a file path to the
> diff command.”
> 
> into simply saying:
> 
> “As a result, no diff is displayed.”
> 
> However, I don't read the original message as it would imply that only
> file paths are affected by the precompose_argv call. 
> 
> Are there other suggestions on improving the commit message?
May be something like this, (but this is highly a personal taste question)

When running diff commands, file paths containing decomposed unicode code points
are not converted to precomposed unicode form under Mac OS X.

As a result, no diff is displayed.

Opposite to most builtin commands, the diff builtin is missing the
parse_options call, which internally runs arguments through the
precompose_argv call, which ensures all arguments are in precomposed
unicode form.

Fix the problem by adding a precompose_argv call to diff, diff-index, diff-files and diff-tree.

Signed-off-by: Alexander Rinass <alex@fournova.com>
Helped-By: Torsten Bögershausen <tboegi@web.de>
Helped-By: Junio C Hamano <gitster@pobox.com>

Re: [PATCH] diff: run arguments through precompose_argv

[Junio C Hamano]

Torsten Bögershausen <tboegi@web.de> writes:

> May be something like this, (but this is highly a personal taste question)

My observation was not a suggestion to rewrite the log message, but
primarily about describing the change in behaviour.

I didn't mean to nitpick the phrasing but let me think aloud about
it, using this as a template.  I rewrapped overlong lines from the
original.

> When running diff commands, file paths containing decomposed
> unicode code points are not converted to precomposed unicode form
> under Mac OS X.
>
> As a result, no diff is displayed.

Add ", but we normalize the paths in the index and the history to
precomposed form on that platform." at the end of the first sentence, to
make sure you mention both causes of "As a result" that follows.
Also, these are better to be a single paragraph.  I.e.

    When running diff commands, a pathspec containing decomposed
    unicode code points is not converted to precomposed unicode form
    under Mac OS X, but we normalize the paths in the index and the
    history to precomposed form on that platform.  As a result, the
    pathspec would not match and no diff is shown.

> Opposite to most builtin commands, the diff builtin is missing the
> parse_options call, which internally runs arguments through the
> precompose_argv call, which ensures all arguments are in precomposed
> unicode form.
>
> Fix the problem by adding a precompose_argv call to diff,
> diff-index, diff-files and diff-tree.

Perhaps it is just me but I found "Opposite to most" harder to grok
for some reason.

And the "backward compatibility note" is missing.  I'd write these
two paragraphs like this if I were doing this patch.

    Unlike many builtin commands, the "diff" family of commands do
    not use parse_options(), which is how other builtin commands
    indirectly call precompose_argv() to normalize argv[] into
    precomposed form on Mac OSX.  Teach these commands to call
    precompose_argv() themselves.

    Note that precomopose_argv() normalizes not just paths but all
    command line arguments, so things like "git diff -G $string"
    when $string has the decomposed form would first normalized into
    the precomposed form and would stop hitting the same string in
    the decomposed form in the diff output with this change.  It is
    not a problem per-se, as "log" family of commands already use
    parse_options() and call precompose_argv()--we can think of it
    as making the "diff" family of commands behave in a similar way
    as the commands in the "log" family.

Re: [PATCH] diff: run arguments through precompose_argv

[Junio C Hamano]

Alexander Rinass <alex@fournova.com> writes:

>> On 05 Apr 2016, at 21:15, Johannes Sixt <j6t@kdbg.org> wrote:
>> 
>> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>> 
>> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
>> 
>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>> 
>> -- Hannes
>
> Thanks for reviewing and catching the NFD encoding error.
>
> I will send in a patch v2 with the correct NFC encoding.
>
> Would you also like me to alter the commit message as mentioned by Junio?
>
> I could rewrite the sentence:
>
> “As a result, no diff is displayed when feeding such a file path to the
> diff command.”
>
> into simply saying:
>
> “As a result, no diff is displayed.”
>
> However, I don't read the original message as it would imply that only
> file paths are affected by the precompose_argv call. 
>
> Are there other suggestions on improving the commit message?

I think after this message there were a few suggestions, and then we
heard nothing.  Should we still be waiting for a response from you?

Thanks.

Re: [PATCH] diff: run arguments through precompose_argv

[Alexander Rinass]

> On 12 May 2016, at 00:08, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Alexander Rinass <alex@fournova.com> writes:
> 
>>> On 05 Apr 2016, at 21:15, Johannes Sixt <j6t@kdbg.org> wrote:
>>> 
>>> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>>> 
>>> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
>>> 
>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>>> 
>>> -- Hannes
>> 
>> Thanks for reviewing and catching the NFD encoding error.
>> 
>> I will send in a patch v2 with the correct NFC encoding.
>> 
>> Would you also like me to alter the commit message as mentioned by Junio?
>> 
>> I could rewrite the sentence:
>> 
>> “As a result, no diff is displayed when feeding such a file path to the
>> diff command.”
>> 
>> into simply saying:
>> 
>> “As a result, no diff is displayed.”
>> 
>> However, I don't read the original message as it would imply that only
>> file paths are affected by the precompose_argv call. 
>> 
>> Are there other suggestions on improving the commit message?
> 
> I think after this message there were a few suggestions, and then we
> heard nothing.  Should we still be waiting for a response from you?
> 
> Thanks.

Sorry for not replying earlier. 

I will create a v2 patch until the weekend and send it to the mailing list.

Re: [PATCH] diff: run arguments through precompose_argv

[Junio C Hamano]

Alexander Rinass <alex@fournova.com> writes:

> I will create a v2 patch until the weekend and send it to the mailing list.

Thanks.