Re: [PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

[Jakub Narebski <jnareb@gmail.com>]

On Thu, 2 Feb 2012, Jakub Narebski wrote:
> On Thu, 2 Feb 2012, Michał Kiedrowicz wrote:
> > Jakub Narebski <jnareb@gmail.com> wrote:
> > 
> > > Gitweb tries hard to properly process UTF-8 data, by marking output
> > > from git commands and contents of files as UTF-8 with to_utf8()
> > > subroutine.  This ensures that gitweb would print correctly UTF-8
> > > e.g. in 'log' and 'commit' views.
> > > 
> > > Unfortunately it misses another source of potentially Unicode input,
> > > namely query parameters.  The result is that one cannot search for a
> > > string containing characters outside US-ASCII.  For example searching
> > > for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> > > WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> > > bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> > > incorrect data in search field
> > > 
> > > 	Michał Kiedrowicz
> > > 
> > > This is caused by CGI by default treating '0xc5 0x82' bytes as two
> > > characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> > > query parameter is not processed explicitly as UTF-8 encoded string.
> > > 
> > > The solution used here follows "Using Unicode in a Perl CGI script"
> > > article on http://www.lemoda.net/cgi/perl-unicode/index.html:
> > > 
> > > 	use CGI;
> > > 	use Encode 'decode_utf8;
> > > 	my $value = params('input');
> > > 	$value = decode_utf8($value);
> > > 
> > > This is done when filling %input_params hash; this required to move
> > > from explicit $cgi->param(<label>) to $input_params{<name>} in a few
> > > places.
> > 
> > I'm sorry but this doesn't work for me. I would be happy to help if you
> > have some questions about it.
> 
> Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says that
> those two approaches should be equivalent.  The -utf8 pragma version
> doesn't work for me at all, while this one works in that if finds what
> it is supposed to, but shows garbage in search form.

Is it what you mean by "this doesn't work for me", i.e. working search,
garbage in search field?
 
> Will investigate.

Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext) like
in gitweb, CGI.pm would read $cgi->param("s") by itself - without decoding.
To skip this we need to pass -force=>1  or  -override=>1 (i.e. further
changes to gitweb).

-utf8 pragma works with more modern CGI.pm, but does not with 3.10.

-- 
Jakub Narebski
Poland