[RFC PATCH] gitweb: use CGI with -utf8

[RFC PATCH] gitweb: use CGI with -utf8

[Michał Kiedrowicz]

I noticed that gitweb tries a lot to properly process UTF-8 data, for
example it prints my name correctly in log and commit information, but
it echos junk in the search field. It looks like:

	Michał Kiedrowicz

I don't know CGI well and I never touched gitewb code, but I found this
on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI '-utf8';
	my $value = params ('input');

I tried it and that fixed my problem. I'm not sure about the
consequences, maybe someone more experienced in CGI might help?
---
 gitweb/gitweb.perl |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index abb5a79..74d45b1 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -10,7 +10,7 @@
 use 5.008;
 use strict;
 use warnings;
-use CGI qw(:standard :escapeHTML -nosticky);
+use CGI qw(:standard :escapeHTML -nosticky -utf8);
 use CGI::Util qw(unescape);
 use CGI::Carp qw(fatalsToBrowser set_message);
 use Encode;
-- 
1.7.3.4

Re: [RFC PATCH] gitweb: use CGI with -utf8

[Jakub Narebski]

Michał Kiedrowicz <michal.kiedrowicz@gmail.com> writes:

> I noticed that gitweb tries a lot to properly process UTF-8 data, for
> example it prints my name correctly in log and commit information, but
> it echos junk in the search field. It looks like:
> 
> 	Michał Kiedrowicz
> 
> I don't know CGI well and I never touched gitewb code, but I found this
> on http://www.lemoda.net/cgi/perl-unicode/index.html:
> 
> 	use CGI '-utf8';
> 	my $value = params ('input');
> 
> I tried it and that fixed my problem. I'm not sure about the
> consequences, maybe someone more experienced in CGI might help?

I have reworded this to form a proper commit message (see
Documentation/SubmittingPatches) and I'll resend this as a reply to
this email.

> ---
>  gitweb/gitweb.perl |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index abb5a79..74d45b1 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -10,7 +10,7 @@
>  use 5.008;
>  use strict;
>  use warnings;
> -use CGI qw(:standard :escapeHTML -nosticky);
> +use CGI qw(:standard :escapeHTML -nosticky -utf8);
>  use CGI::Util qw(unescape);
>  use CGI::Carp qw(fatalsToBrowser set_message);
>  use Encode;
> -- 

Does this actually work for you?  Because it doesn't work for me
(perhaps I have too old CGI module: what CGI.pm and what Perl version
do you use?).

See other solution to this in other reply to this email.

-- 
Jakub Narebski

[PATCH/RFC (version A)] gitweb: use CGI with -utf8 to process Unicode query parameters correctly

[Jakub Narebski]

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

According to "Using Unicode in a Perl CGI script" article on
http://www.lemoda.net/cgi/perl-unicode/index.html the simplest
solution is to just import '-utf8' pragma for CGI module:

	use CGI '-utf8';
	my $value = params('input');

According to CGI module documentation, the '-utf8' pragma may cause
problems with POST requests containing binary files... but gitweb
currently do not use POST requests at all, so this should be not a
problem now.

Alternate solution would be to explicity decode query parameters when
storing them in %input_params (and perhaps also path_info).

[jn: reworded / rewritten commit message]

Signed-off-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
Signed-off-by: Jakub Narębski <jnareb@gmail.com>
---
 gitweb/gitweb.perl |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 9cf7e71..a7441ef 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -10,7 +10,7 @@
 use 5.008;
 use strict;
 use warnings;
-use CGI qw(:standard :escapeHTML -nosticky);
+use CGI qw(:standard :escapeHTML -nosticky -utf8);
 use CGI::Util qw(unescape);
 use CGI::Carp qw(fatalsToBrowser set_message);
 use Encode;
-- 
1.7.6

Re: [PATCH/RFC (version A)] gitweb: use CGI with -utf8 to process Unicode query parameters correctly

[Jakub Narebski]

> According to "Using Unicode in a Perl CGI script" article on
> http://www.lemoda.net/cgi/perl-unicode/index.html the simplest
> solution is to just import '-utf8' pragma for CGI module:
> 
> 	use CGI '-utf8';
> 	my $value = params('input');
[...]
> ---

Except it doesn't work for me...

>  gitweb/gitweb.perl |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 9cf7e71..a7441ef 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -10,7 +10,7 @@
>  use 5.008;
>  use strict;
>  use warnings;
> -use CGI qw(:standard :escapeHTML -nosticky);
> +use CGI qw(:standard :escapeHTML -nosticky -utf8);
>  use CGI::Util qw(unescape);
>  use CGI::Carp qw(fatalsToBrowser set_message);
>  use Encode;
> -- 
> 1.7.6
> 
> 

-- 
Jakub Narebski
Poland

Re: [PATCH/RFC (version A)] gitweb: use CGI with -utf8 to process Unicode query parameters correctly

[Michał Kiedrowicz]

Jakub Narebski <jnareb@gmail.com> wrote:

> Gitweb tries hard to properly process UTF-8 data, by marking output
> from git commands and contents of files as UTF-8 with to_utf8()
> subroutine.  This ensures that gitweb would print correctly UTF-8
> e.g. in 'log' and 'commit' views.
> 
> Unfortunately it misses another source of potentially Unicode input,
> namely query parameters.  The result is that one cannot search for a
> string containing characters outside US-ASCII.  For example searching
> for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> incorrect data in search field
> 
> 	Michał Kiedrowicz
> 
> This is caused by CGI by default treating '0xc5 0x82' bytes as two
> characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> query parameter is not processed explicitly as UTF-8 encoded string.
> 
> According to "Using Unicode in a Perl CGI script" article on
> http://www.lemoda.net/cgi/perl-unicode/index.html the simplest
> solution is to just import '-utf8' pragma for CGI module:
> 
> 	use CGI '-utf8';
> 	my $value = params('input');
> 
> According to CGI module documentation, the '-utf8' pragma may cause
> problems with POST requests containing binary files... but gitweb
> currently do not use POST requests at all, so this should be not a
> problem now.

This was exactly my feeling  when I sent this patch :).

> 
> Alternate solution would be to explicity decode query parameters when
> storing them in %input_params (and perhaps also path_info).
> 
> [jn: reworded / rewritten commit message]
> 
> Signed-off-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>

Thanks, I forgot about that.

> Signed-off-by: Jakub Narębski <jnareb@gmail.com>
> ---
>  gitweb/gitweb.perl |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 9cf7e71..a7441ef 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -10,7 +10,7 @@
>  use 5.008;
>  use strict;
>  use warnings;
> -use CGI qw(:standard :escapeHTML -nosticky);
> +use CGI qw(:standard :escapeHTML -nosticky -utf8);
>  use CGI::Util qw(unescape);
>  use CGI::Carp qw(fatalsToBrowser set_message);
>  use Encode;

[PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Jakub Narebski]

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

This is done when filling %input_params hash; this required to move
from explicit $cgi->param(<label>) to $input_params{<name>} in a few
places.

Alternate solution would be to simply use the '-utf8' pragma (via
"use CGI '-utf8';"), but according to CGI.pm documentation it may
cause problems with POST requests containing binary files... and
it doesn't work with old CGI.pm version 3.10 from Perl v5.8.6.

Noticed-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
Signed-off-by: Jakub Narębski <jnareb@gmail.com>
---
 gitweb/gitweb.perl |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 9cf7e71..55b2c24 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -52,7 +52,7 @@ sub evaluate_uri {
 	# as base URL.
 	# Therefore, if we needed to strip PATH_INFO, then we know that we have
 	# to build the base URL ourselves:
-	our $path_info = $ENV{"PATH_INFO"};
+	our $path_info = decode_utf8($ENV{"PATH_INFO"});
 	if ($path_info) {
 		if ($my_url =~ s,\Q$path_info\E$,, &&
 		    $my_uri =~ s,\Q$path_info\E$,, &&
@@ -816,9 +816,9 @@ sub evaluate_query_params {
 
 	while (my ($name, $symbol) = each %cgi_param_mapping) {
 		if ($symbol eq 'opt') {
-			$input_params{$name} = [ $cgi->param($symbol) ];
+			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
 		} else {
-			$input_params{$name} = $cgi->param($symbol);
+			$input_params{$name} = decode_utf8($cgi->param($symbol));
 		}
 	}
 }
@@ -2767,7 +2767,7 @@ sub git_populate_project_tagcloud {
 	}
 
 	my $cloud;
-	my $matched = $cgi->param('by_tag');
+	my $matched = $input_params{'ctag'};
 	if (eval { require HTML::TagCloud; 1; }) {
 		$cloud = HTML::TagCloud->new;
 		foreach my $ctag (sort keys %ctags_lc) {
@@ -5282,7 +5282,7 @@ sub git_project_list_body {
 
 	my $check_forks = gitweb_check_feature('forks');
 	my $show_ctags  = gitweb_check_feature('ctags');
-	my $tagfilter = $show_ctags ? $cgi->param('by_tag') : undef;
+	my $tagfilter = $show_ctags ? $input_params{'ctag'} : undef;
 	$check_forks = undef
 		if ($tagfilter || $searchtext);
 
@@ -6197,7 +6197,7 @@ sub git_tag {
 
 sub git_blame_common {
 	my $format = shift || 'porcelain';
-	if ($format eq 'porcelain' && $cgi->param('js')) {
+	if ($format eq 'porcelain' && $input_params{'javascript'}) {
 		$format = 'incremental';
 		$action = 'blame_incremental'; # for page title etc
 	}
-- 
1.7.6

Re: [PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Michał Kiedrowicz]

Jakub Narebski <jnareb@gmail.com> wrote:

> Gitweb tries hard to properly process UTF-8 data, by marking output
> from git commands and contents of files as UTF-8 with to_utf8()
> subroutine.  This ensures that gitweb would print correctly UTF-8
> e.g. in 'log' and 'commit' views.
> 
> Unfortunately it misses another source of potentially Unicode input,
> namely query parameters.  The result is that one cannot search for a
> string containing characters outside US-ASCII.  For example searching
> for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> incorrect data in search field
> 
> 	Michał Kiedrowicz
> 
> This is caused by CGI by default treating '0xc5 0x82' bytes as two
> characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> query parameter is not processed explicitly as UTF-8 encoded string.
> 
> The solution used here follows "Using Unicode in a Perl CGI script"
> article on http://www.lemoda.net/cgi/perl-unicode/index.html:
> 
> 	use CGI;
> 	use Encode 'decode_utf8;
> 	my $value = params('input');
> 	$value = decode_utf8($value);
> 
> This is done when filling %input_params hash; this required to move
> from explicit $cgi->param(<label>) to $input_params{<name>} in a few
> places.

I'm sorry but this doesn't work for me. I would be happy to help if you
have some questions about it.

> 
> Alternate solution would be to simply use the '-utf8' pragma (via
> "use CGI '-utf8';"), but according to CGI.pm documentation it may
> cause problems with POST requests containing binary files... and
> it doesn't work with old CGI.pm version 3.10 from Perl v5.8.6.
> 
> Noticed-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
> Signed-off-by: Jakub Narębski <jnareb@gmail.com>
> ---
>  gitweb/gitweb.perl |   12 ++++++------
>  1 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 9cf7e71..55b2c24 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -52,7 +52,7 @@ sub evaluate_uri {
>  	# as base URL.
>  	# Therefore, if we needed to strip PATH_INFO, then we know that we have
>  	# to build the base URL ourselves:
> -	our $path_info = $ENV{"PATH_INFO"};
> +	our $path_info = decode_utf8($ENV{"PATH_INFO"});
>  	if ($path_info) {
>  		if ($my_url =~ s,\Q$path_info\E$,, &&
>  		    $my_uri =~ s,\Q$path_info\E$,, &&
> @@ -816,9 +816,9 @@ sub evaluate_query_params {
>  
>  	while (my ($name, $symbol) = each %cgi_param_mapping) {
>  		if ($symbol eq 'opt') {
> -			$input_params{$name} = [ $cgi->param($symbol) ];
> +			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
>  		} else {
> -			$input_params{$name} = $cgi->param($symbol);
> +			$input_params{$name} = decode_utf8($cgi->param($symbol));
>  		}
>  	}
>  }
> @@ -2767,7 +2767,7 @@ sub git_populate_project_tagcloud {
>  	}
>  
>  	my $cloud;
> -	my $matched = $cgi->param('by_tag');
> +	my $matched = $input_params{'ctag'};
>  	if (eval { require HTML::TagCloud; 1; }) {
>  		$cloud = HTML::TagCloud->new;
>  		foreach my $ctag (sort keys %ctags_lc) {
> @@ -5282,7 +5282,7 @@ sub git_project_list_body {
>  
>  	my $check_forks = gitweb_check_feature('forks');
>  	my $show_ctags  = gitweb_check_feature('ctags');
> -	my $tagfilter = $show_ctags ? $cgi->param('by_tag') : undef;
> +	my $tagfilter = $show_ctags ? $input_params{'ctag'} : undef;
>  	$check_forks = undef
>  		if ($tagfilter || $searchtext);
>  
> @@ -6197,7 +6197,7 @@ sub git_tag {
>  
>  sub git_blame_common {
>  	my $format = shift || 'porcelain';
> -	if ($format eq 'porcelain' && $cgi->param('js')) {
> +	if ($format eq 'porcelain' && $input_params{'javascript'}) {
>  		$format = 'incremental';
>  		$action = 'blame_incremental'; # for page title etc
>  	}

Re: [PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Jakub Narebski]

On Thu, 2 Jan 2012, Michał Kiedrowicz wrote:
> Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > Gitweb tries hard to properly process UTF-8 data, by marking output
> > from git commands and contents of files as UTF-8 with to_utf8()
> > subroutine.  This ensures that gitweb would print correctly UTF-8
> > e.g. in 'log' and 'commit' views.
> > 
> > Unfortunately it misses another source of potentially Unicode input,
> > namely query parameters.  The result is that one cannot search for a
> > string containing characters outside US-ASCII.  For example searching
> > for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> > WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> > bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> > incorrect data in search field
> > 
> > 	Michał Kiedrowicz
> > 
> > This is caused by CGI by default treating '0xc5 0x82' bytes as two
> > characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> > query parameter is not processed explicitly as UTF-8 encoded string.
> > 
> > The solution used here follows "Using Unicode in a Perl CGI script"
> > article on http://www.lemoda.net/cgi/perl-unicode/index.html:
> > 
> > 	use CGI;
> > 	use Encode 'decode_utf8;
> > 	my $value = params('input');
> > 	$value = decode_utf8($value);
> > 
> > This is done when filling %input_params hash; this required to move
> > from explicit $cgi->param(<label>) to $input_params{<name>} in a few
> > places.
> 
> I'm sorry but this doesn't work for me. I would be happy to help if you
> have some questions about it.

Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says that
those two approaches should be equivalent.  The -utf8 pragma version
doesn't work for me at all, while this one works in that if finds what
it is supposed to, but shows garbage in search form.

Will investigate.
 
> > Alternate solution would be to simply use the '-utf8' pragma (via
> > "use CGI '-utf8';"), but according to CGI.pm documentation it may
> > cause problems with POST requests containing binary files... and
> > it doesn't work with old CGI.pm version 3.10 from Perl v5.8.6.

[...]
> > @@ -816,9 +816,9 @@ sub evaluate_query_params {
> >  
> >  	while (my ($name, $symbol) = each %cgi_param_mapping) {
> >  		if ($symbol eq 'opt') {
> > -			$input_params{$name} = [ $cgi->param($symbol) ];
> > +			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
> >  		} else {
> > -			$input_params{$name} = $cgi->param($symbol);
> > +			$input_params{$name} = decode_utf8($cgi->param($symbol));
> >  		}
> >  	}
> >  }
-- 
Jakub Narebski
Poland

Re: [PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Jakub Narebski]

On Thu, 2 Feb 2012, Jakub Narebski wrote:
> On Thu, 2 Feb 2012, Michał Kiedrowicz wrote:
> > Jakub Narebski <jnareb@gmail.com> wrote:
> > 
> > > Gitweb tries hard to properly process UTF-8 data, by marking output
> > > from git commands and contents of files as UTF-8 with to_utf8()
> > > subroutine.  This ensures that gitweb would print correctly UTF-8
> > > e.g. in 'log' and 'commit' views.
> > > 
> > > Unfortunately it misses another source of potentially Unicode input,
> > > namely query parameters.  The result is that one cannot search for a
> > > string containing characters outside US-ASCII.  For example searching
> > > for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> > > WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> > > bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> > > incorrect data in search field
> > > 
> > > 	Michał Kiedrowicz
> > > 
> > > This is caused by CGI by default treating '0xc5 0x82' bytes as two
> > > characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> > > query parameter is not processed explicitly as UTF-8 encoded string.
> > > 
> > > The solution used here follows "Using Unicode in a Perl CGI script"
> > > article on http://www.lemoda.net/cgi/perl-unicode/index.html:
> > > 
> > > 	use CGI;
> > > 	use Encode 'decode_utf8;
> > > 	my $value = params('input');
> > > 	$value = decode_utf8($value);
> > > 
> > > This is done when filling %input_params hash; this required to move
> > > from explicit $cgi->param(<label>) to $input_params{<name>} in a few
> > > places.
> > 
> > I'm sorry but this doesn't work for me. I would be happy to help if you
> > have some questions about it.
> 
> Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says that
> those two approaches should be equivalent.  The -utf8 pragma version
> doesn't work for me at all, while this one works in that if finds what
> it is supposed to, but shows garbage in search form.

Is it what you mean by "this doesn't work for me", i.e. working search,
garbage in search field?
 
> Will investigate.

Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext) like
in gitweb, CGI.pm would read $cgi->param("s") by itself - without decoding.
To skip this we need to pass -force=>1  or  -override=>1 (i.e. further
changes to gitweb).

-utf8 pragma works with more modern CGI.pm, but does not with 3.10.

-- 
Jakub Narebski
Poland

Re: [PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Michal Kiedrowicz]

Jakub Narebski <jnareb@gmail.com> wrote:

> On Thu, 2 Feb 2012, Jakub Narebski wrote:
> > On Thu, 2 Feb 2012, Michał Kiedrowicz wrote:
> > > Jakub Narebski <jnareb@gmail.com> wrote:
> > > 
> > > > Gitweb tries hard to properly process UTF-8 data, by marking
> > > > output from git commands and contents of files as UTF-8 with
> > > > to_utf8() subroutine.  This ensures that gitweb would print
> > > > correctly UTF-8 e.g. in 'log' and 'commit' views.
> > > > 
> > > > Unfortunately it misses another source of potentially Unicode
> > > > input, namely query parameters.  The result is that one cannot
> > > > search for a string containing characters outside US-ASCII.
> > > > For example searching for "Michał Kiedrowicz" (containing
> > > > letter 'ł' - LATIN SMALL LETTER L WITH STROKE, with Unicode
> > > > codepoint U+0142, represented with 0xc5 0x82 bytes in UTF-8 and
> > > > percent-encoded as %C5%81) result in the following incorrect
> > > > data in search field
> > > > 
> > > > 	Michał Kiedrowicz
> > > > 
> > > > This is caused by CGI by default treating '0xc5 0x82' bytes as
> > > > two characters in Perl legacy encoding latin-1 (iso-8859-1),
> > > > because 's' query parameter is not processed explicitly as
> > > > UTF-8 encoded string.
> > > > 
> > > > The solution used here follows "Using Unicode in a Perl CGI
> > > > script" article on
> > > > http://www.lemoda.net/cgi/perl-unicode/index.html:
> > > > 
> > > > 	use CGI;
> > > > 	use Encode 'decode_utf8;
> > > > 	my $value = params('input');
> > > > 	$value = decode_utf8($value);
> > > > 
> > > > This is done when filling %input_params hash; this required to
> > > > move from explicit $cgi->param(<label>) to
> > > > $input_params{<name>} in a few places.
> > > 
> > > I'm sorry but this doesn't work for me. I would be happy to help
> > > if you have some questions about it.
> > 
> > Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says
> > that those two approaches should be equivalent.  The -utf8 pragma
> > version doesn't work for me at all, while this one works in that if
> > finds what it is supposed to, but shows garbage in search form.
> 
> Is it what you mean by "this doesn't work for me", i.e. working
> search, garbage in search field?

I mean "garbage in search field". Search works even without the patch
(at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
don't have my notebook nearby at the moment to check).

>  
> > Will investigate.

Thanks for your time spending on this. I wouldn't call this problem
"production critial" but it seems wrong to support UTF-8 everywhere
properly except for one place.

> 
> Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext)
> like in gitweb, CGI.pm would read $cgi->param("s") by itself -
> without decoding. 

Makes sense. When I tried calling to_utf8() in the line that defines
textfield (this was my first approach to this problem), it haven't
changed anything.

> To skip this we need to pass -force=>1  or
> -override=>1 (i.e. further changes to gitweb).
> 
> -utf8 pragma works with more modern CGI.pm, but does not with 3.10.
> 

[PATCH/RFCv2 (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Jakub Narebski]

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

Decoding UTF-8 is done when filling %input_params hash and $path_info
variable; the former required to move from explicit $cgi->param(<label>)
to $input_params{<name>} in a few places, which is a good idea anyway.

Another required change was to add -override=>1 parameter to
$cgi->textfield() invocation (in search form).  Otherwise CGI would
use values from query string if it is present, filling value from
$cgi->param... without decode_utf8().  As we are using value of
appropriate parameter anyway, -override=>1 doesn't change the
situation but makes gitweb fill search field correctly.

Alternate solution would be to simply use the '-utf8' pragma (via
"use CGI '-utf8';"), but according to CGI.pm documentation it may
cause problems with POST requests containing binary files... and
it requires CGI 3.31 (I think), released with perl v5.8.9.

Noticed-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
Signed-off-by: Jakub Narębski <jnareb@gmail.com>
---
On Fri, 3 Feb 2012, Michal Kiedrowicz wrote:
> Jakub Narebski <jnareb@gmail.com> wrote:

> > Is it what you mean by "this doesn't work for me", i.e. working
> > search, garbage in search field?
> 
> I mean "garbage in search field". Search works even without the patch
> (at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
> don't have my notebook nearby at the moment to check).
[...]

> > Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext)
> > like in gitweb, CGI.pm would read $cgi->param("s") by itself -
> > without decoding. 
> 
> Makes sense. When I tried calling to_utf8() in the line that defines
> textfield (this was my first approach to this problem), it haven't
> changed anything.

Yes, and it doesn't makes sense in gitweb case - we use value of 
$cgi->param("s") as default value of text field anyway, but in
Unicode-aware way.
 
> > To skip this we need to pass -force=>1  or
> > -override=>1 (i.e. further changes to gitweb).

This patch does this.  

Does it make work for you?

> > -utf8 pragma works with more modern CGI.pm, but does not with 3.10.

-utf8 pragma was added in CVS revision 1.238 of CGI.pm, which I think
is present in CGI 3.31, released with perl v5.8.9.  Theoretically gitweb
maintains backward compatibility with perl v5.8.3 or something 
("use 5.008;" but IIRC 5.8.3 is needed for correct Unicde handling anyway).

 gitweb/gitweb.perl |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 9cf7e71..bd5fff9 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -52,7 +52,7 @@ sub evaluate_uri {
 	# as base URL.
 	# Therefore, if we needed to strip PATH_INFO, then we know that we have
 	# to build the base URL ourselves:
-	our $path_info = $ENV{"PATH_INFO"};
+	our $path_info = decode_utf8($ENV{"PATH_INFO"});
 	if ($path_info) {
 		if ($my_url =~ s,\Q$path_info\E$,, &&
 		    $my_uri =~ s,\Q$path_info\E$,, &&
@@ -816,9 +816,9 @@ sub evaluate_query_params {
 
 	while (my ($name, $symbol) = each %cgi_param_mapping) {
 		if ($symbol eq 'opt') {
-			$input_params{$name} = [ $cgi->param($symbol) ];
+			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
 		} else {
-			$input_params{$name} = $cgi->param($symbol);
+			$input_params{$name} = decode_utf8($cgi->param($symbol));
 		}
 	}
 }
@@ -2767,7 +2767,7 @@ sub git_populate_project_tagcloud {
 	}
 
 	my $cloud;
-	my $matched = $cgi->param('by_tag');
+	my $matched = $input_params{'ctag'};
 	if (eval { require HTML::TagCloud; 1; }) {
 		$cloud = HTML::TagCloud->new;
 		foreach my $ctag (sort keys %ctags_lc) {
@@ -3873,7 +3873,7 @@ sub print_search_form {
 	                       -values => ['commit', 'grep', 'author', 'committer', 'pickaxe']) .
 	      $cgi->sup($cgi->a({-href => href(action=>"search_help")}, "?")) .
 	      " search:\n",
-	      $cgi->textfield(-name => "s", -value => $searchtext) . "\n" .
+	      $cgi->textfield(-name => "s", -value => $searchtext, -override => 1) . "\n" .
 	      "<span title=\"Extended regular expression\">" .
 	      $cgi->checkbox(-name => 'sr', -value => 1, -label => 're',
 	                     -checked => $search_use_regexp) .
@@ -5282,7 +5282,7 @@ sub git_project_list_body {
 
 	my $check_forks = gitweb_check_feature('forks');
 	my $show_ctags  = gitweb_check_feature('ctags');
-	my $tagfilter = $show_ctags ? $cgi->param('by_tag') : undef;
+	my $tagfilter = $show_ctags ? $input_params{'ctag'} : undef;
 	$check_forks = undef
 		if ($tagfilter || $searchtext);
 
@@ -5994,7 +5994,7 @@ sub git_project_list {
 	}
 	print $cgi->startform(-method => "get") .
 	      "<p class=\"projsearch\">Search:\n" .
-	      $cgi->textfield(-name => "s", -value => $searchtext) . "\n" .
+	      $cgi->textfield(-name => "s", -value => $searchtext, -override => 1) . "\n" .
 	      "</p>" .
 	      $cgi->end_form() . "\n";
 	git_project_list_body(\@list, $order);
@@ -6197,7 +6197,7 @@ sub git_tag {
 
 sub git_blame_common {
 	my $format = shift || 'porcelain';
-	if ($format eq 'porcelain' && $cgi->param('js')) {
+	if ($format eq 'porcelain' && $input_params{'javascript'}) {
 		$format = 'incremental';
 		$action = 'blame_incremental'; # for page title etc
 	}
-- 
1.7.6

Re: [PATCH/RFCv2 (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Michał Kiedrowicz]

Jakub Narebski <jnareb@gmail.com> wrote:

> Gitweb tries hard to properly process UTF-8 data, by marking output
> from git commands and contents of files as UTF-8 with to_utf8()
> subroutine.  This ensures that gitweb would print correctly UTF-8
> e.g. in 'log' and 'commit' views.
> 
> Unfortunately it misses another source of potentially Unicode input,
> namely query parameters.  The result is that one cannot search for a
> string containing characters outside US-ASCII.  For example searching
> for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> incorrect data in search field
> 
> 	Michał Kiedrowicz
> 
> This is caused by CGI by default treating '0xc5 0x82' bytes as two
> characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> query parameter is not processed explicitly as UTF-8 encoded string.
> 
> The solution used here follows "Using Unicode in a Perl CGI script"
> article on http://www.lemoda.net/cgi/perl-unicode/index.html:
> 
> 	use CGI;
> 	use Encode 'decode_utf8;
> 	my $value = params('input');
> 	$value = decode_utf8($value);
> 
> Decoding UTF-8 is done when filling %input_params hash and $path_info
> variable; the former required to move from explicit $cgi->param(<label>)
> to $input_params{<name>} in a few places, which is a good idea anyway.
> 
> Another required change was to add -override=>1 parameter to
> $cgi->textfield() invocation (in search form).  Otherwise CGI would
> use values from query string if it is present, filling value from
> $cgi->param... without decode_utf8().  As we are using value of
> appropriate parameter anyway, -override=>1 doesn't change the
> situation but makes gitweb fill search field correctly.
> 
> Alternate solution would be to simply use the '-utf8' pragma (via
> "use CGI '-utf8';"), but according to CGI.pm documentation it may
> cause problems with POST requests containing binary files... and
> it requires CGI 3.31 (I think), released with perl v5.8.9.
> 
> Noticed-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
> Signed-off-by: Jakub Narębski <jnareb@gmail.com>
> ---
> On Fri, 3 Feb 2012, Michal Kiedrowicz wrote:
> > Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > > Is it what you mean by "this doesn't work for me", i.e. working
> > > search, garbage in search field?
> > 
> > I mean "garbage in search field". Search works even without the patch
> > (at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
> > don't have my notebook nearby at the moment to check).
> [...]
> 
> > > Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext)
> > > like in gitweb, CGI.pm would read $cgi->param("s") by itself -
> > > without decoding. 
> > 
> > Makes sense. When I tried calling to_utf8() in the line that defines
> > textfield (this was my first approach to this problem), it haven't
> > changed anything.
> 
> Yes, and it doesn't makes sense in gitweb case - we use value of 
> $cgi->param("s") as default value of text field anyway, but in
> Unicode-aware way.
>  
> > > To skip this we need to pass -force=>1  or
> > > -override=>1 (i.e. further changes to gitweb).
> 
> This patch does this.  
> 
> Does it make work for you?
> 

Yes, it works for me. Search form properly displays "ł". Thanks!

Re: [PATCH/RFCv2 (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> Gitweb tries hard to properly process UTF-8 data, by marking output
> from git commands and contents of files as UTF-8 with to_utf8()
> subroutine.  This ensures that gitweb would print correctly UTF-8
> e.g. in 'log' and 'commit' views.
>
> Unfortunately it misses another source of potentially Unicode input,
> namely query parameters.  The result is that one cannot search for a

I think two lines should suffice instead of the above two paragraphs.

        Gitweb forgot to turn query parameters into UTF-8. This results in
        a bug that one cannot search for a

> string containing characters outside US-ASCII.  For example searching
> ...

The remainder explains the problem and the solution very well modulo minor
typos.

> Noticed-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
> Signed-off-by: Jakub Narębski <jnareb@gmail.com>
> ---

We can add "Tested-by:" to this now.  Will queue.

Thanks.

Re: [RFC PATCH] gitweb: use CGI with -utf8

[Michał Kiedrowicz]

Jakub Narebski <jnareb@gmail.com> wrote:

> Michał Kiedrowicz <michal.kiedrowicz@gmail.com> writes:
> 
> > I noticed that gitweb tries a lot to properly process UTF-8 data, for
> > example it prints my name correctly in log and commit information, but
> > it echos junk in the search field. It looks like:
> > 
> > 	Michał Kiedrowicz
> > 
> > I don't know CGI well and I never touched gitewb code, but I found this
> > on http://www.lemoda.net/cgi/perl-unicode/index.html:
> > 
> > 	use CGI '-utf8';
> > 	my $value = params ('input');
> > 
> > I tried it and that fixed my problem. I'm not sure about the
> > consequences, maybe someone more experienced in CGI might help?
> 
> I have reworded this to form a proper commit message (see
> Documentation/SubmittingPatches) and I'll resend this as a reply to
> this email.

Thanks, your message is much better.

> 
> > ---
> >  gitweb/gitweb.perl |    2 +-
> >  1 files changed, 1 insertions(+), 1 deletions(-)
> > 
> > diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> > index abb5a79..74d45b1 100755
> > --- a/gitweb/gitweb.perl
> > +++ b/gitweb/gitweb.perl
> > @@ -10,7 +10,7 @@
> >  use 5.008;
> >  use strict;
> >  use warnings;
> > -use CGI qw(:standard :escapeHTML -nosticky);
> > +use CGI qw(:standard :escapeHTML -nosticky -utf8);
> >  use CGI::Util qw(unescape);
> >  use CGI::Carp qw(fatalsToBrowser set_message);
> >  use Encode;
> > -- 
> 
> Does this actually work for you? 

Yes. It correctly displays "ł" in the search form.

>  Because it doesn't work for me
> (perhaps I have too old CGI module: what CGI.pm and what Perl version
> do you use?).
> 

$ perl --version

This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux
(with 12 registered patches, see perl -V for more detail)

$ eix -e CGI -c
[I] perl-core/CGI (3.510@01.02.2012): Simple Common Gateway Interface Class

> See other solution to this in other reply to this email.
>