[PATCH] cvsimport: strip question marks from tags

[PATCH] cvsimport: strip question marks from tags

[Ken Dreyer]

The "?" character can be present in a CVS tag name, but git's
bad_ref_char does not allow question marks in git tags. If
git-cvsimport encounters a CVS tag with a question mark, it will error
and refuse to continue the import beyond that point.

When importing CVS tags, strip "?" characters from the tag names as we
translate them to git tag names.

Signed-off-by: Ken Dreyer <ktdreyer@ktdreyer.com>
---
 git-cvsimport.perl | 1 +
 1 file changed, 1 insertion(+)

diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index 8d41610..36f59fe 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -890,6 +890,7 @@ sub commit {
 		$xtag =~ tr/_/\./ if ( $opt_u );
 		$xtag =~ s/[\/]/$opt_s/g;
 		$xtag =~ s/\[//g;
+		$xtag =~ s/\?//g;
 
 		system('git' , 'tag', '-f', $xtag, $cid) == 0
 			or die "Cannot create tag $xtag: $!\n";
-- 
1.7.11.4

Re: [PATCH] cvsimport: strip question marks from tags

[Junio C Hamano]

Ken Dreyer <ktdreyer@ktdreyer.com> writes:

> The "?" character can be present in a CVS tag name, but git's
> bad_ref_char does not allow question marks in git tags. If
> git-cvsimport encounters a CVS tag with a question mark, it will error
> and refuse to continue the import beyond that point.
>
> When importing CVS tags, strip "?" characters from the tag names as we
> translate them to git tag names.
>
> Signed-off-by: Ken Dreyer <ktdreyer@ktdreyer.com>
> ---
>  git-cvsimport.perl | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/git-cvsimport.perl b/git-cvsimport.perl
> index 8d41610..36f59fe 100755
> --- a/git-cvsimport.perl
> +++ b/git-cvsimport.perl
> @@ -890,6 +890,7 @@ sub commit {
>  		$xtag =~ tr/_/\./ if ( $opt_u );
>  		$xtag =~ s/[\/]/$opt_s/g;
>  		$xtag =~ s/\[//g;
> +		$xtag =~ s/\?//g;

I do not think this is a right and sustainable approach.  The next
patch would probably be to strip "~" and then another patch that
strips "^", and yet another that squashes ".." into one would surely
follow.

How about extending the s/\[//g we can see in the context to cover
everything that are unacceptable (see refs.c:bad_ref_char()) once
and for all?  The result needs to be further massaged to avoid
component that has two or more dots in a row, a dot at the beginning
or at the end (see the comment at the beginning of refs.c, and also
refs.c:check_refname_component()).

>  		system('git' , 'tag', '-f', $xtag, $cid) == 0
>  			or die "Cannot create tag $xtag: $!\n";

[PATCH] cvsimport: strip all inappropriate tag strings

[Ken Dreyer]

Certain characters such as "?" can be present in a CVS tag name, but
git does not allow these characters in tags. If git-cvsimport
encounters a CVS tag that git cannot handle, cvsimport will error and
refuse to continue the import beyond that point.

When importing CVS tags, strip all the inappropriate strings from the
tag names as we translate them to git tag names.

Signed-off-by: Ken Dreyer <ktdreyer@ktdreyer.com>
---

Thank you Junio for the review. I've taken your suggestion and amended
my patch to eliminate all the bad strings in ref.c. 

 git-cvsimport.perl | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index 8d41610..0dc598d 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -889,7 +889,25 @@ sub commit {
 		$xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY **
 		$xtag =~ tr/_/\./ if ( $opt_u );
 		$xtag =~ s/[\/]/$opt_s/g;
-		$xtag =~ s/\[//g;
+
+		# See ref.c for these rules.
+		# Tag cannot end with a '/' - this is already handled above.
+		# Tag cannot contain bad chars. See bad_ref_char in ref.c.
+		$xtag =~ s/[ ~\^:\\\*\?\[]//g;
+		# Tag cannot contain '..'.
+		$xtag =~ s/\.\.//g;
+		# Tag cannot contain '@{'.
+		$xtag =~ s/\@{//g;
+		# Tag cannot end with '.lock'.
+		$xtag =~ s/(?:\.lock)+$//;
+		# Tag cannot begin or end with '.'.
+		$xtag =~ s/^\.+//;
+		$xtag =~ s/\.+$//;
+		# Tag cannot consist of a single '.' - already handled above.
+		# Tag cannot be empty.
+		if ($xtag eq '') {
+			return;
+		}
 
 		system('git' , 'tag', '-f', $xtag, $cid) == 0
 			or die "Cannot create tag $xtag: $!\n";
-- 
1.7.11.4

Re: [PATCH] cvsimport: strip all inappropriate tag strings

[Junio C Hamano]

Ken Dreyer <ktdreyer@ktdreyer.com> writes:

> Certain characters such as "?" can be present in a CVS tag name, but
> git does not allow these characters in tags. If git-cvsimport
> encounters a CVS tag that git cannot handle, cvsimport will error and
> refuse to continue the import beyond that point.
>
> When importing CVS tags, strip all the inappropriate strings from the
> tag names as we translate them to git tag names.
>
> Signed-off-by: Ken Dreyer <ktdreyer@ktdreyer.com>
> ---

Thanks, will queue.

I think we also forbid tagnames (or branchnames for that matter)
that begin with a dash on the creation side, even though the reading
side tries to be lenient (i.e. if for some bad tool already created
a file .git/refs/tags/-foobar, we allow "git show tags/-foobar" to
show it).  The routines in refs.c enforces primarily on the reading
codepath.  So this part:

> +		# Tag cannot begin or end with '.'.
> +		$xtag =~ s/^\.+//;
> +		$xtag =~ s/\.+$//;

may need to become

	# Tag cannot begin with '.' or '-', or end with '.'.
	$xtag =~ s/^[-.]+//;
	$xtag =~ s/\.+$//;

or something.


>  git-cvsimport.perl | 20 +++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/git-cvsimport.perl b/git-cvsimport.perl
> index 8d41610..0dc598d 100755
> --- a/git-cvsimport.perl
> +++ b/git-cvsimport.perl
> @@ -889,7 +889,25 @@ sub commit {
>  		$xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY **
>  		$xtag =~ tr/_/\./ if ( $opt_u );
>  		$xtag =~ s/[\/]/$opt_s/g;
> -		$xtag =~ s/\[//g;
> +
> +		# See ref.c for these rules.
> +		# Tag cannot end with a '/' - this is already handled above.
> +		# Tag cannot contain bad chars. See bad_ref_char in ref.c.
> +		$xtag =~ s/[ ~\^:\\\*\?\[]//g;
> +		# Tag cannot contain '..'.
> +		$xtag =~ s/\.\.//g;
> +		# Tag cannot contain '@{'.
> +		$xtag =~ s/\@{//g;
> +		# Tag cannot end with '.lock'.
> +		$xtag =~ s/(?:\.lock)+$//;
> +		# Tag cannot begin or end with '.'.
> +		$xtag =~ s/^\.+//;
> +		$xtag =~ s/\.+$//;
> +		# Tag cannot consist of a single '.' - already handled above.
> +		# Tag cannot be empty.
> +		if ($xtag eq '') {
> +			return;
> +		}
>  
>  		system('git' , 'tag', '-f', $xtag, $cid) == 0
>  			or die "Cannot create tag $xtag: $!\n";

Re: [PATCH] cvsimport: strip all inappropriate tag strings

[Alex Vandiver]

On Tue, 2012-09-04 at 22:26 -0600, Ken Dreyer wrote:
> When importing CVS tags, strip all the inappropriate strings from the
> tag names as we translate them to git tag names.
>
> [snip]
> diff --git a/git-cvsimport.perl b/git-cvsimport.perl
> index 8d41610..0dc598d 100755
> --- a/git-cvsimport.perl
> +++ b/git-cvsimport.perl
> @@ -889,7 +889,25 @@ sub commit {
>  		$xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY **
>  		$xtag =~ tr/_/\./ if ( $opt_u );
>  		$xtag =~ s/[\/]/$opt_s/g;
> -		$xtag =~ s/\[//g;
> +
> +		# See ref.c for these rules.
> +		# Tag cannot end with a '/' - this is already handled above.
> +		# Tag cannot contain bad chars. See bad_ref_char in ref.c.
> +		$xtag =~ s/[ ~\^:\\\*\?\[]//g;
> +		# Tag cannot contain '..'.
> +		$xtag =~ s/\.\.//g;
> +		# Tag cannot contain '@{'.
> +		$xtag =~ s/\@{//g;
> +		# Tag cannot end with '.lock'.
> +		$xtag =~ s/(?:\.lock)+$//;
> +		# Tag cannot begin or end with '.'.
> +		$xtag =~ s/^\.+//;
> +		$xtag =~ s/\.+$//;
> +		# Tag cannot consist of a single '.' - already handled above.
> +		# Tag cannot be empty.
> +		if ($xtag eq '') {
> +			return;
> +		}

Unfortunately, this isn't quite sufficient.  Consider the case of a tag
named "foo.lock."  The .lock rule doesn't match, because it's not at the
end of the string -- but after s/\.+$// runs, it _is_ at the end, and
hence invalid.  A similar problem exists with a tag named "a.@{.b",
given the ordering of @{ and .. removal.

Something like the following would suffice:

    1 while $xtag =~ s/
               (?: \.\.        # Tag cannot contain '..'.
               |   \@{         # Tag cannot contain '@{'.
               |   \.lock $    # Tag cannot end with '.lock'.
               | ^ \.          # Tag cannot begin...
               |   \. $        # ...or end with '.'
               )//xg;

 - Alex

[PATCH] cvsimport: strip all inappropriate tag strings

[Ken Dreyer]

Certain characters such as "?" can be present in a CVS tag name, but
git does not allow these characters in tags. If git-cvsimport
encounters a CVS tag that git cannot handle, cvsimport will error and
refuse to continue the import beyond that point.

When importing CVS tags, strip all the inappropriate strings from the
tag names as we translate them to git tag names.

Signed-off-by: Ken Dreyer <ktdreyer@ktdreyer.com>
---

Thanks Junio and Alex for your review and comments. I've implemented
both of your suggestions in this patch.

 git-cvsimport.perl | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index 8d41610..dda8a6d 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -889,7 +889,23 @@ sub commit {
 		$xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY **
 		$xtag =~ tr/_/\./ if ( $opt_u );
 		$xtag =~ s/[\/]/$opt_s/g;
-		$xtag =~ s/\[//g;
+
+		# See ref.c for these rules.
+		# Tag cannot contain bad chars. See bad_ref_char in ref.c.
+		$xtag =~ s/[ ~\^:\\\*\?\[]//g;
+		# Other bad strings for tags:
+		1 while $xtag =~ s/
+			(?: \.\.        # Tag cannot contain '..'.
+			|   \@{         # Tag cannot contain '@{'.
+			| ^ -           # Tag cannot begin with '-'.
+			|   \.lock $    # Tag cannot end with '.lock'.
+			| ^ \.          # Tag cannot begin...
+			|   \. $        # ...or end with '.'
+			)//xg;
+		# Tag cannot be empty.
+		if ($xtag eq '') {
+			return;
+		}
 
 		system('git' , 'tag', '-f', $xtag, $cid) == 0
 			or die "Cannot create tag $xtag: $!\n";
-- 
1.7.11.4

Re: [PATCH] cvsimport: strip all inappropriate tag strings

[Junio C Hamano]

Ken Dreyer <ktdreyer@ktdreyer.com> writes:

> Certain characters such as "?" can be present in a CVS tag name, but
> git does not allow these characters in tags. If git-cvsimport
> encounters a CVS tag that git cannot handle, cvsimport will error and
> refuse to continue the import beyond that point.
>
> When importing CVS tags, strip all the inappropriate strings from the
> tag names as we translate them to git tag names.
>
> Signed-off-by: Ken Dreyer <ktdreyer@ktdreyer.com>
> ---
>
> Thanks Junio and Alex for your review and comments. I've implemented
> both of your suggestions in this patch.

Thanks.

Do we want to give a warning instead of silently dropping a tag on
the floor, or is the output verbose enough that such a warning will
be drowned in the noise?

>  git-cvsimport.perl | 18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/git-cvsimport.perl b/git-cvsimport.perl
> index 8d41610..dda8a6d 100755
> --- a/git-cvsimport.perl
> +++ b/git-cvsimport.perl
> @@ -889,7 +889,23 @@ sub commit {
>  		$xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY **
>  		$xtag =~ tr/_/\./ if ( $opt_u );
>  		$xtag =~ s/[\/]/$opt_s/g;
> -		$xtag =~ s/\[//g;
> +
> +		# See ref.c for these rules.
> +		# Tag cannot contain bad chars. See bad_ref_char in ref.c.
> +		$xtag =~ s/[ ~\^:\\\*\?\[]//g;
> +		# Other bad strings for tags:
> +		1 while $xtag =~ s/
> +			(?: \.\.        # Tag cannot contain '..'.
> +			|   \@{         # Tag cannot contain '@{'.
> +			| ^ -           # Tag cannot begin with '-'.
> +			|   \.lock $    # Tag cannot end with '.lock'.
> +			| ^ \.          # Tag cannot begin...
> +			|   \. $        # ...or end with '.'
> +			)//xg;
> +		# Tag cannot be empty.
> +		if ($xtag eq '') {

That is, adding something like:

	print STDERR "warning: ignoring tag '$tag' with invalid tagname";

here.

> +			return;
> +		}
>  
>  		system('git' , 'tag', '-f', $xtag, $cid) == 0
>  			or die "Cannot create tag $xtag: $!\n";

It also may be worthwhile to show the original tagname ($tag)
somewhere in this message to help diagnosis.

[PATCH] cvsimport: strip all inappropriate tag strings

[Ken Dreyer]

Certain characters such as "?" can be present in a CVS tag name, but
git does not allow these characters in tags. If git-cvsimport
encounters a CVS tag that git cannot handle, cvsimport will error and
refuse to continue the import beyond that point.

When importing CVS tags, strip all the inappropriate strings from the
tag names as we translate them to git tag names.

Provide more debugging information to the user if we've altered the
tag and the "git tag" command still fails. Also, warn the user if we
end up skipping an (unusable) tag altogether.

Signed-off-by: Ken Dreyer <ktdreyer@ktdreyer.com>
---

Thanks Junio for your suggestion about diagnosis messages. I've
implemented your suggestion by adding a warning statement if we skip a
tag altogether, and I also added some output if we've translated a tag
and the system() call still fails.

 git-cvsimport.perl | 32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)

diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index 8d41610..3a30754 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -889,10 +889,36 @@ sub commit {
 		$xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY **
 		$xtag =~ tr/_/\./ if ( $opt_u );
 		$xtag =~ s/[\/]/$opt_s/g;
-		$xtag =~ s/\[//g;
 
-		system('git' , 'tag', '-f', $xtag, $cid) == 0
-			or die "Cannot create tag $xtag: $!\n";
+		# See ref.c for these rules.
+		# Tag cannot contain bad chars. See bad_ref_char in ref.c.
+		$xtag =~ s/[ ~\^:\\\*\?\[]//g;
+		# Other bad strings for tags:
+		1 while $xtag =~ s/
+			(?: \.\.        # Tag cannot contain '..'.
+			|   \@{         # Tag cannot contain '@{'.
+			| ^ -           # Tag cannot begin with '-'.
+			|   \.lock $    # Tag cannot end with '.lock'.
+			| ^ \.          # Tag cannot begin...
+			|   \. $        # ...or end with '.'
+			)//xg;
+		# Tag cannot be empty.
+		if ($xtag eq '') {
+			warn("warning: ignoring tag '$tag'",
+			" with invalid tagname\n");
+			return;
+		}
+
+		if (system('git' , 'tag', '-f', $xtag, $cid) != 0) {
+			# We did our best to sanitize the tag, but still failed
+			# for whatever reason. Bail out, and give the user
+			# enough information to understand if/how we should
+			# improve the translation in the future.
+			if ($tag ne $xtag) {
+				print "Translated '$tag' tag to '$xtag'\n";
+			}
+			die "Cannot create tag $xtag: $!\n";
+		}
 
 		print "Created tag '$xtag' on '$branch'\n" if $opt_v;
 	}
-- 
1.7.11.4

Re: [PATCH] cvsimport: strip all inappropriate tag strings

[Andreas Schwab]

Ken Dreyer <ktdreyer@ktdreyer.com> writes:

> +		# See ref.c for these rules.
> +		# Tag cannot contain bad chars. See bad_ref_char in ref.c.

s/ref.c/refs.c/

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

[PATCH] cvsimport: strip all inappropriate tag strings

[Ken Dreyer]

Certain characters such as "?" can be present in a CVS tag name, but
git does not allow these characters in tags. If git-cvsimport
encounters a CVS tag that git cannot handle, cvsimport will error and
refuse to continue the import beyond that point.

When importing CVS tags, strip all the inappropriate strings from the
tag names as we translate them to git tag names.

Provide more debugging information to the user if we've altered the
tag and the "git tag" command still fails. Also, warn the user if we
end up skipping an (unusable) tag altogether.

Signed-off-by: Ken Dreyer <ktdreyer@ktdreyer.com>
---

Thanks Andreas for catching that "ref.c" in the comments ought to be
"refs.c". I've corrected that in this latest version of the patch.

 git-cvsimport.perl | 33 ++++++++++++++++++++++++++++++---
 1 file changed, 30 insertions(+), 3 deletions(-)

diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index 8d41610..8032f23 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -889,10 +889,37 @@ sub commit {
 		$xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY **
 		$xtag =~ tr/_/\./ if ( $opt_u );
 		$xtag =~ s/[\/]/$opt_s/g;
-		$xtag =~ s/\[//g;
 
-		system('git' , 'tag', '-f', $xtag, $cid) == 0
-			or die "Cannot create tag $xtag: $!\n";
+		# See refs.c for these rules.
+		# Tag cannot contain bad chars. (See bad_ref_char in refs.c.)
+		$xtag =~ s/[ ~\^:\\\*\?\[]//g;
+		# Other bad strings for tags:
+		# (See check_refname_component in refs.c.)
+		1 while $xtag =~ s/
+			(?: \.\.        # Tag cannot contain '..'.
+			|   \@{         # Tag cannot contain '@{'.
+			| ^ -           # Tag cannot begin with '-'.
+			|   \.lock $    # Tag cannot end with '.lock'.
+			| ^ \.          # Tag cannot begin...
+			|   \. $        # ...or end with '.'
+			)//xg;
+		# Tag cannot be empty.
+		if ($xtag eq '') {
+			warn("warning: ignoring tag '$tag'",
+			" with invalid tagname\n");
+			return;
+		}
+
+		if (system('git' , 'tag', '-f', $xtag, $cid) != 0) {
+			# We did our best to sanitize the tag, but still failed
+			# for whatever reason. Bail out, and give the user
+			# enough information to understand if/how we should
+			# improve the translation in the future.
+			if ($tag ne $xtag) {
+				print "Translated '$tag' tag to '$xtag'\n";
+			}
+			die "Cannot create tag $xtag: $!\n";
+		}
 
 		print "Created tag '$xtag' on '$branch'\n" if $opt_v;
 	}
-- 
1.7.11.4

Re: [PATCH] cvsimport: strip all inappropriate tag strings

[Junio C Hamano]

Ken Dreyer <ktdreyer@ktdreyer.com> writes:

> Thanks Andreas for catching that "ref.c" in the comments ought to be
> "refs.c". I've corrected that in this latest version of the patch.

Yeah, thanks, all.  Will queue.