[PATCH] pathspec: warn on empty strings as pathspec

[PATCH] pathspec: warn on empty strings as pathspec

[Emily Xie]

For any command that takes a pathspec, passing an empty string will
execute the command on all files in the current directory. This
results in unexpected behavior. For example, git add "" adds all
files to staging, while git rm -rf "" recursively removes all files
from the working tree and index. A two-step implemetation will
prevent such cases.

This patch, as step one, invokes a warning whenever an empty
string is detected as a pathspec, introducing users to the upcoming
change. For step two, a follow-up patch several release cycles later
will remove the warnings and actually implement the change by
throwing an error instead.

Signed-off-by: Emily Xie <emilyxxie@gmail.com>
Reported-by: David Turner <novalis@novalis.org>
Mentored-by: Michail Denchev <mdenchev@gmail.com>
Thanks-to: Sarah Sharp <sarah@thesharps.us> and James Sharp <jamey@minilop.net>
---
 pathspec.c     | 6 +++++-
 t/t3600-rm.sh  | 6 +++++-
 t/t3700-add.sh | 4 ++++
 3 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/pathspec.c b/pathspec.c
index c9e9b6c..79e370e 100644
--- a/pathspec.c
+++ b/pathspec.c
@@ -402,8 +402,12 @@ void parse_pathspec(struct pathspec *pathspec,
 	}
 
 	n = 0;
-	while (argv[n])
+	while (argv[n]) {
+		if (*argv[n] == '\0')
+			warning(_("empty strings are not valid pathspecs and will no longer "
+			          "be supported in upcoming releases"));
 		n++;
+	}
 
 	pathspec->nr = n;
 	ALLOC_ARRAY(pathspec->items, n);
diff --git a/t/t3600-rm.sh b/t/t3600-rm.sh
index d046d98..4503a14 100755
--- a/t/t3600-rm.sh
+++ b/t/t3600-rm.sh
@@ -881,4 +881,8 @@ test_expect_success 'rm files with two different errors' '
 	test_i18ncmp expect actual
 '
 
-test_done
+test_expect_success 'rm empty string should invoke warning' '
+	git rm -rf "" 2>&1 | grep "warning: empty string"
+'
+
+test_done
\ No newline at end of file
diff --git a/t/t3700-add.sh b/t/t3700-add.sh
index f14a665..5dbe8c2 100755
--- a/t/t3700-add.sh
+++ b/t/t3700-add.sh
@@ -207,6 +207,10 @@ test_expect_success POSIXPERM,SANITY 'git add should fail atomically upon an unr
 	! ( git ls-files foo1 | grep foo1 )
 '
 
+test_expect_success 'git add empty string should invoke warning' '
+	git add "" 2>&1 | grep "warning: empty string"
+'
+
 rm -f foo2
 
 test_expect_success POSIXPERM,SANITY 'git add --ignore-errors' '
-- 
2.8.4

Re: [PATCH] pathspec: warn on empty strings as pathspec

[Junio C Hamano]

Emily Xie <emilyxxie@gmail.com> writes:

> For any command that takes a pathspec, passing an empty string will
> execute the command on all files in the current directory.

OK.

> This
> results in unexpected behavior.

Technically speaking, your expectation is what is wrong here.  An
empty string as a pathspec matches all paths.

> For example, git add "" adds all
> files to staging, while git rm -rf "" recursively removes all files
> from the working tree and index.

That is a logical consequence that an empty string is a pathspec
that matches everything.  If somebody wants to add everything in
their current directory, they can say 'git add ""'.  If you do not
want to do so, don't say 'git add ""'.

You need to argue why these are bad things to convince those who are
used to the current behaviour that it is OK to break them.

Here is my attempt.

	An pathspec that is an empty string has been interpreted to
	match any path, as pathspec match is a leading substring
	match that honours directory boundary.  Just like pathspec
	"dir/" (or "dir") matches "dir/file", "" matches "file".

	However, a carelessly-written script may result in an empty
	string assigned to a variable (perhaps caused by a bug in
	it), and may end up passing an empty string to a Git command
	invocation, i.e.

        	git rm "$path"
        	git add "$path"
		
	which would affect all paths in the current directory.

	We cannot simply reject an empty string given as a pathspec
	to prevent this kind of mistake.  Because there surely are
	existing scripts that take advantage of the fact that an
	empty string matches all paths, such a change will break
	scripts that legitimately have been using an empty string
	for that purpose.

	Instead, we need two step approach.  The first step is to
	notice that the caller used an empty string as a pathspec,
	give a warning to (1) declare that the use of an empty
	string as "match all" will become invalid and (2) ask them
	to use "." instead if they mean "match all".

	After some release cycles, we can remove the warning and
	turn an empty string used as a pathspec element as an error.

	This patch is the first step.


> A two-step implemetation will
> prevent such cases.

There is some leap/gap in logic here.  

> This patch, as step one, invokes a warning whenever an empty
> string is detected as a pathspec, introducing users to the upcoming
> change. For step two, a follow-up patch several release cycles later
> will remove the warnings and actually implement the change by
> throwing an error instead.

This paragraph is OK, but I think I ended up covering the whole
thing in my attempt ;-).


> Signed-off-by: Emily Xie <emilyxxie@gmail.com>
> Reported-by: David Turner <novalis@novalis.org>
> Mentored-by: Michail Denchev <mdenchev@gmail.com>
> Thanks-to: Sarah Sharp <sarah@thesharps.us> and James Sharp <jamey@minilop.net>
> ---
>  pathspec.c     | 6 +++++-
>  t/t3600-rm.sh  | 6 +++++-
>  t/t3700-add.sh | 4 ++++
>  3 files changed, 14 insertions(+), 2 deletions(-)
>
> diff --git a/pathspec.c b/pathspec.c
> index c9e9b6c..79e370e 100644
> --- a/pathspec.c
> +++ b/pathspec.c
> @@ -402,8 +402,12 @@ void parse_pathspec(struct pathspec *pathspec,
>  	}
>  
>  	n = 0;
> -	while (argv[n])
> +	while (argv[n]) {
> +		if (*argv[n] == '\0')
> +			warning(_("empty strings are not valid pathspecs and will no longer "
> +			          "be supported in upcoming releases"));
>  		n++;

Three issues:

 * if argv[1] and argv[2] are both emtpy, the user will see the same
   message twice.  Is it intended?  Is it acceptable?

 * "Empty strings are not valid pathspecs" is just plain wrong.  It
   has been valid, but the warning message notifies that we are
   going to make it invalid what has been valid.

 * "Will no longer be supported" is just plain useless.  We are
   notifying that we will turn what they have been using as a valid
   feature invalid.  What needs to accompany that notification is
   how to update their script that have been happily working, which
   we are going to break with the future change, in a way that will
   keep working, i.e. "please use . instead if you meant to match
   all".


> +test_expect_success 'rm empty string should invoke warning' '
> +	git rm -rf "" 2>&1 | grep "warning: empty string"
> +'

As your warning is in _("..."), you would need test_i18grep here, I think.

> +test_done
> \ No newline at end of file

Oops.

[PATCH] pathspec: warn on empty strings as pathspec

[Emily Xie]

Currently, passing an empty string as a pathspec results in
a match on all paths. This is because a pathspec match is a
leading substring match that honors directory boundaries.
So, just as pathspec "dir/" (or "dir") matches "dir/file",
"" matches "file".

However, this causes problems. Namely, a user's
carelessly-written script could accidentally assign an
empty string to a variable that then gets passed to a Git
command invocation, e.g.:

        git rm -r "$path"
        git add "$path"

which would unintentionally affect all paths in the current
directory.

The fix for this issue requires a two-step approach. As
there may be existing scripts that knowingly use empty
strings in this manner, the first step simply invokes a
warning that (1) declares using empty strings to
match all paths will become invalid and (2) asks the user
to use "." if their intent was to match all.

For step two, a follow-up patch several release cycles
later will remove the warnings and throw an error instead.

This patch is the first step.

Signed-off-by: Emily Xie <emilyxxie@gmail.com>
Reported-by: David Turner <novalis@novalis.org>
Mentored-by: Michail Denchev <mdenchev@gmail.com>
Thanks-to: Sarah Sharp <sarah@thesharps.us> and James Sharp <jamey@minilop.net>
---
 pathspec.c     | 11 +++++++++--
 t/t3600-rm.sh  |  5 +++++
 t/t3700-add.sh |  5 +++++
 3 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/pathspec.c b/pathspec.c
index c9e9b6c..02aa691 100644
--- a/pathspec.c
+++ b/pathspec.c
@@ -364,7 +364,7 @@ void parse_pathspec(struct pathspec *pathspec,
 {
 	struct pathspec_item *item;
 	const char *entry = argv ? *argv : NULL;
-	int i, n, prefixlen, nr_exclude = 0;
+	int i, n, prefixlen, warn_empty_string, nr_exclude = 0;
 
 	memset(pathspec, 0, sizeof(*pathspec));
 
@@ -402,8 +402,15 @@ void parse_pathspec(struct pathspec *pathspec,
 	}
 
 	n = 0;
-	while (argv[n])
+	warn_empty_string = 1;
+	while (argv[n]) {
+		if (*argv[n] == '\0' && warn_empty_string) {
+			warning(_("empty strings as pathspecs will be made invalid in upcoming releases. "
+			          "please use . instead if you meant to match all paths"));
+			warn_empty_string = 0;
+		}
 		n++;
+	}
 
 	pathspec->nr = n;
 	ALLOC_ARRAY(pathspec->items, n);
diff --git a/t/t3600-rm.sh b/t/t3600-rm.sh
index d046d98..14f0edc 100755
--- a/t/t3600-rm.sh
+++ b/t/t3600-rm.sh
@@ -881,4 +881,9 @@ test_expect_success 'rm files with two different errors' '
 	test_i18ncmp expect actual
 '
 
+test_expect_success 'rm empty string should invoke warning' '
+	git rm -rf "" 2>output &&
+	test_i18ngrep "warning: empty strings" output
+'
+
 test_done
diff --git a/t/t3700-add.sh b/t/t3700-add.sh
index f14a665..05379d0 100755
--- a/t/t3700-add.sh
+++ b/t/t3700-add.sh
@@ -332,4 +332,9 @@ test_expect_success 'git add --dry-run --ignore-missing of non-existing file out
 	test_i18ncmp expect.err actual.err
 '
 
+test_expect_success 'git add empty string should invoke warning' '
+	git add "" 2>output &&
+	test_i18ngrep "warning: empty strings" output
+'
+
 test_done
-- 
2.8.4

Re: [PATCH] pathspec: warn on empty strings as pathspec

[Junio C Hamano]

Emily Xie <emilyxxie@gmail.com> writes:

> The fix for this issue requires a two-step approach. As
> there may be existing scripts that knowingly use empty
> strings in this manner, the first step simply invokes a
> warning that (1) declares using empty strings to
> match all paths will become invalid and (2) asks the user
> to use "." if their intent was to match all.
>
> For step two, a follow-up patch several release cycles
> later will remove the warnings and throw an error instead.
>
> This patch is the first step.
>
> Signed-off-by: Emily Xie <emilyxxie@gmail.com>
> Reported-by: David Turner <novalis@novalis.org>
> Mentored-by: Michail Denchev <mdenchev@gmail.com>
> Thanks-to: Sarah Sharp <sarah@thesharps.us> and James Sharp <jamey@minilop.net>
> ---

Looks sensible.  I personally do not think warn-only-once matters in
practice, but now you have implemented it, it is an additional nice
touch ;-)

Will queue.

Thanks.

Re: [PATCH] pathspec: warn on empty strings as pathspec

[Emily Xie]

Awesome. Thanks, Junio---this is exciting!

As for step 2: do you have a good sense on the timing? Around how long
should I wait to submit the patch for it?

Otherwise, let me know if there is anything else that I need to do!

- Emily

Re: [PATCH] pathspec: warn on empty strings as pathspec

[Junio C Hamano]

Emily Xie <emilyxxie@gmail.com> writes:

> Awesome. Thanks, Junio---this is exciting!

No, thank _you_.

> As for step 2: do you have a good sense on the timing? Around how long
> should I wait to submit the patch for it?

Not so fast.  We'll wait for others to comment first.

I am queuing this change but that does not mean anything more than
that I am not outright rejecting the idea.

It is possible that others may assess the cost of having to do the
two-step migration differently, and the list concensus may end up
being "if it hurts, don't pass an empty string", i.e. we'd better
without this patch or subsequent second step.  If that happens, the
first step dies without hitting 'next'.  I'd say we'd wait at least
for a week.

Otherwise, if the change is received favourably, we'll merge it to
'next', do the same waiting for comments.  Repeat the same and then
merge to 'master'.  After it hits the next feature release (probably
at around the end of August), the change will be seen by wider
general public, and at that point we may see strong opposition from
them with a good argument neither of us thought of why we shouldn't
do this change, in which case we might have to revise the plan and
scrap the whole thing.

So, we'll wait and see.

Re: [PATCH] pathspec: warn on empty strings as pathspec

[Emily Xie]

I suppose I got ahead of myself there. :-) Thanks for the
clarification on the process.

On Thu, Jun 23, 2016 at 2:17 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Emily Xie <emilyxxie@gmail.com> writes:
>
>> Awesome. Thanks, Junio---this is exciting!
>
> No, thank _you_.
>
>> As for step 2: do you have a good sense on the timing? Around how long
>> should I wait to submit the patch for it?
>
> Not so fast.  We'll wait for others to comment first.
>
> I am queuing this change but that does not mean anything more than
> that I am not outright rejecting the idea.
>
> It is possible that others may assess the cost of having to do the
> two-step migration differently, and the list concensus may end up
> being "if it hurts, don't pass an empty string", i.e. we'd better
> without this patch or subsequent second step.  If that happens, the
> first step dies without hitting 'next'.  I'd say we'd wait at least
> for a week.
>
> Otherwise, if the change is received favourably, we'll merge it to
> 'next', do the same waiting for comments.  Repeat the same and then
> merge to 'master'.  After it hits the next feature release (probably
> at around the end of August), the change will be seen by wider
> general public, and at that point we may see strong opposition from
> them with a good argument neither of us thought of why we shouldn't
> do this change, in which case we might have to revise the plan and
> scrap the whole thing.
>
> So, we'll wait and see.
>



-- 
Emily Xie
xie-emily.com
207.399.6370