Re: [PATCH 1/2] t/unit-tests: add UTF-8 width tests for CJK chars

[Junio C Hamano <gitster@pobox.com>]

Jiang Xin <worldhello.net@gmail.com> writes:

[jc: the same question about the choice of Cc addresses applies]

> This commit adds a new test suite (u-utf8-width.c) to test the UTF-8
> width functions in Git, particularly focusing on multi-byte characters
> from East Asian languages like Chinese, Japanese, and Korean that
> typically require 2 display columns per character.
>
> The test suite includes:
> - Tests for utf8_strnwidth with Chinese strings
> - Tests for utf8_strwidth with Chinese strings
> - Tests for Japanese and Korean characters
> - Edge case tests with invalid UTF-8 sequences
> - Proper test function naming following the Clar framework convention
>
> Also updated the build configuration in Makefile and meson.build to
> include the new test suite in the build process.

The usual way to compose a log message of this project is to

 - Give an observation on how the current system works in the
   present tense (so no need to say "Currently X is Y", or
   "Previously X was Y" to describe the state before your change;
   just "X is Y" is enough), and discuss what you perceive as a
   problem in it.

 - Propose a solution (optional---often, problem description
   trivially leads to an obvious solution in reader's minds).

 - Give commands to somebody editing the codebase to "make it so",
   instead of saying "This commit does X".

in this order.

> +	/* Test length limiting */
> +	str = "你好世界";
> +	cl_assert_equal_i(2, utf8_strnwidth(str, 3, 0));  /* Only first char "你"(2 columns) within 3 bytes */
> +	cl_assert_equal_i(4, utf8_strnwidth(str, 6, 0));  /* First two chars "你好"(4 columns) in 6 bytes */

We also should test utf8_strwidth() on the same string here.

> +/*
> + * Test edge cases with partial UTF-8 sequences
> + */

All tests before these make sense, but I am not sure if we want to
hold utf8_strnwidth() to the requirement that it will tolerate "len"
to end in the middle of a single character, as such a requirement by
itself does not do application any good.

A caller may have "你好世界" in str, learn that the first 4 bytes
would only need two display columns to show (i.e., 3-byte "你" plus
a single garbage byte, that would make UTF-8 encoded "好" if the
remaining two bytes were included), and may want to learn how to
show only enough to fill the two display columns.  But there is not
enough information given back by utf8_strnwidth() for such a caller
to figure out that it needs to feed only the first three bytes (not
four) of str to printf() to do so.