Re: [PATCH/RFC] Gitweb: Convert UTF-8 encoded file names

[Michael Wagner <accounts@mwagner.org>]

On Thu, May 15, 2014 at 12:25:45AM +0200, Jakub Narębski wrote:
> On Wed, May 14, 2014 at 11:57 PM, Junio C Hamano <gitster@pobox.com> wrote:
> > Michael Wagner <accounts@mwagner.org> writes:
> >
> >> Perl has an internal encoding used to store text strings. Currently, trying to
> >> view files with UTF-8 encoded names results in an error (either "404 - Cannot
> >> find file" [blob_plain] or "XML Parsing Error" [blob]). Converting these UTF-8
> >> encoded file names into Perl's internal format resolves these errors.
> 
> Could you give us an example?  What is important is whether filename
> is passed via path_info or via query string.
> 

There is a file named "Gütekriterien.txt" in my repository. Trying to
view this file as "blob_plain" produces an 404 error (displaying the
file name with an additional print statement):

$ REQUEST_METHOD=GET QUERY_STRING='p=notes.git;a=blob_plain;f=work/G%C3%83%C2%BCtekriterien.txt;hb=HEAD' ./gitweb.cgi

work/Gütekriterien.txt
Status: 404 Not Found

Decoding the UTF-8 encoded file name (again with an additional print
statement):

$ REQUEST_METHOD=GET QUERY_STRING='p=notes.git;a=blob_plain;f=work/G%C3%83%C2%BCtekriterien.txt;hb=HEAD' ./gitweb.cgi

work/Gütekriterien.txt
Content-disposition: inline; filename="work/Gütekriterien.txt"

> Because in evaluate_uri() there is
> 
>      our $path_info = decode_utf8($ENV{"PATH_INFO"});
> 
> and in evaluate_query_params() there is
> 
>     $input_params{$name} = decode_utf8($cgi->param($symbol));
> 
> >> Signed-off-by: Michael Wagner <accounts@mwagner.org>
> >> ---
> >
> > Cc'ing Jakub, who have been the area maintainer, for comments.
> >
> > One thing I wonder is that, if there are some additional calls to
> > encode() necessary before we embed $file_name (which are now decoded
> > to the internal string form, not a byte-sequence that happens to be
> > in utf-8) in the generated pages, if we were to do this change.

The generated pages show the correct file names. 

> 
> There should be no problem with output encoding.  esc_path(), which
> should be used for filenames, includes to_utf8, which in turn uses
> decode($fallback_encoding, $str, Encode::FB_DEFAULT);
> 
> >>  gitweb/gitweb.perl | 2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> >> index a9f57d6..6046977 100755
> >> --- a/gitweb/gitweb.perl
> >> +++ b/gitweb/gitweb.perl
> >> @@ -1056,7 +1056,7 @@ sub evaluate_and_validate_params {
> >>               }
> >>       }
> >>
> >> -     our $file_name = $input_params{'file_name'};
> >> +     our $file_name = decode("utf-8", $input_params{'file_name'});
> >>       if (defined $file_name) {
> >>               if (!is_valid_pathname($file_name)) {
> >>                       die_error(400, "Invalid file parameter");
> 
> Hmm... all %input_params should have been properly decoded
> already, how it was missed?
> 
> Also, branchname (hash_base etc.), search query, filename in file_parent,
> project name can be UTF-8 too, so it is at best partial fix.
> 
> -- 
> Jakub Narębski