Re: [PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Jakub Narebski <jnareb@gmail.com>]

On Thu, 2 Jan 2012, Michał Kiedrowicz wrote:
> Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > Gitweb tries hard to properly process UTF-8 data, by marking output
> > from git commands and contents of files as UTF-8 with to_utf8()
> > subroutine.  This ensures that gitweb would print correctly UTF-8
> > e.g. in 'log' and 'commit' views.
> > 
> > Unfortunately it misses another source of potentially Unicode input,
> > namely query parameters.  The result is that one cannot search for a
> > string containing characters outside US-ASCII.  For example searching
> > for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> > WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> > bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> > incorrect data in search field
> > 
> > 	Michał Kiedrowicz
> > 
> > This is caused by CGI by default treating '0xc5 0x82' bytes as two
> > characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> > query parameter is not processed explicitly as UTF-8 encoded string.
> > 
> > The solution used here follows "Using Unicode in a Perl CGI script"
> > article on http://www.lemoda.net/cgi/perl-unicode/index.html:
> > 
> > 	use CGI;
> > 	use Encode 'decode_utf8;
> > 	my $value = params('input');
> > 	$value = decode_utf8($value);
> > 
> > This is done when filling %input_params hash; this required to move
> > from explicit $cgi->param(<label>) to $input_params{<name>} in a few
> > places.
> 
> I'm sorry but this doesn't work for me. I would be happy to help if you
> have some questions about it.

Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says that
those two approaches should be equivalent.  The -utf8 pragma version
doesn't work for me at all, while this one works in that if finds what
it is supposed to, but shows garbage in search form.

Will investigate.
 
> > Alternate solution would be to simply use the '-utf8' pragma (via
> > "use CGI '-utf8';"), but according to CGI.pm documentation it may
> > cause problems with POST requests containing binary files... and
> > it doesn't work with old CGI.pm version 3.10 from Perl v5.8.6.

[...]
> > @@ -816,9 +816,9 @@ sub evaluate_query_params {
> >  
> >  	while (my ($name, $symbol) = each %cgi_param_mapping) {
> >  		if ($symbol eq 'opt') {
> > -			$input_params{$name} = [ $cgi->param($symbol) ];
> > +			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
> >  		} else {
> > -			$input_params{$name} = $cgi->param($symbol);
> > +			$input_params{$name} = decode_utf8($cgi->param($symbol));
> >  		}
> >  	}
> >  }
-- 
Jakub Narebski
Poland