terminal enhancement

terminal enhancement

[Yoshinori K. Okuji]

To support Unicode in the menu interface, I think we need to add one 
more function into the terminal system. For now, the menu only thinks 
about ASCII characters. So the number of bytes in a string is identical 
to the number of columns to be used for displaying the string.

This is not true, generally speaking. In UTF-8, most Latin characters 
are 2-bytes but one column is used to show each character. Simply 
speaking, width(string) != length(string).

In POSIX, the function wcwidth is defined to get the width of a string 
of wide characters. We need a similar function in GRUB.

Basically, only terminal drivers know how many columns are used for a 
given string. If a terminal driver uses the font manager completely, we 
can get the information from the font manager, but not all terminals 
use the font manager (such as the PC console).

So I'd like to propose adding a new member 'getwidth' into struct 
grub_term. This function would be like this:

int
getwidth (grub_uint32_t code)
{
  /* CODE is encoded in UCS-4.  */

  /* Do something here.  */

  width = ...;
  return width;
}

Anyway, you need this kind of function to implement a terminal.

What do you think?

Okuji

Re: terminal enhancement

[Marco Gerards]

"Yoshinori K. Okuji" <okuji@enbug.org> writes:

> To support Unicode in the menu interface, I think we need to add one 
> more function into the terminal system. For now, the menu only thinks 
> about ASCII characters. So the number of bytes in a string is identical 
> to the number of columns to be used for displaying the string.
>
> This is not true, generally speaking. In UTF-8, most Latin characters 
> are 2-bytes but one column is used to show each character. Simply 
> speaking, width(string) != length(string).
>
> In POSIX, the function wcwidth is defined to get the width of a string 
> of wide characters. We need a similar function in GRUB.

[...]

> Anyway, you need this kind of function to implement a terminal.
>
> What do you think?

If it is required, I agree.  But I know little about unicode to say
more sane things about this.  I hope you can explain some things to
me.

First of all, why does the terminal has to determine the width?  Isn't
that something generic and how the POSIX function wcwidth (which I
could not find...) works?  What I mean is that the character 'a'
always has the same size, unless you are using some ttf-fonts.  Do you
have bigger characters in mind?

So basically, my question is why POSIX seems to implement it
independent of a terminal while we need to integrate it there.  I'm
sure you are correct about this, but I want to understand this so I
can implement the terminals for the PPC ports properly.

Thanks,
Marco

Re: terminal enhancement

[Yoshinori K. Okuji]

On Sunday 19 September 2004 13:07, Marco Gerards wrote:
> First of all, why does the terminal has to determine the width? 

Because it is dependent upon the implementation of a terminal. For 
example, some Japanese characters can be shown both in one-column and 
two-column. I think this problem has been discussed in a xfree86 
mailing list.

Another problem is that not all Unicode characters may not be shown in a 
given terminal. For example, some glyphs might be missing in a font. 
Then, the font manager uses a default glyph. This glyph might use a 
different width from real ones.

One way to solve this problem independently is to define standard widths 
used in GRUB, and terminal drivers and the font manager follow them. 
I'd like to avoid this, because I want to map some Unicode characters 
to built-in glyphs in a ROM (for instance, in the PC console driver). 
Another reason why I am willing to avoid this is that we need to create 
our own font if any font does not follow our standard.

> Isn't that something generic and how the POSIX function wcwidth
> (which I could not find...) works?

I don't think it is very useful in reality, since Unicode does not 
define how each character code is displayed.

> What I mean is that the character 
> 'a' always has the same size, unless you are using some ttf-fonts. 
> Do you have bigger characters in mind?

Surely. For example, look at this one:

http://yudit.org/images/yudit-2.4.8.gif

Even if you don't understand what the characters mean, you should be 
able to distinguish characters.

> So basically, my question is why POSIX seems to implement it
> independent of a terminal while we need to integrate it there.  I'm
> sure you are correct about this, but I want to understand this so I
> can implement the terminals for the PPC ports properly.

The SUSP v2 says that this function was removed from the final ISO C 
Amendment 1. So probably they understand that this function is 
meaningless.

http://www.opengroup.org/onlinepubs/007908799/xsh/wcwidth.html

When implementing a terminal, never assume (the number of bytes) == (the 
number of columns) or (the number of characters) < (the number of 
columns). A given character can be one-column or two-column in the 
current implementation of GRUB. If we want to support Hindi, things 
will be more complicated.

Okuji

Re: terminal enhancement

[Marco Gerards]

"Yoshinori K. Okuji" <okuji@enbug.org> writes:

>> What I mean is that the character 
>> 'a' always has the same size, unless you are using some ttf-fonts. 
>> Do you have bigger characters in mind?
>
> Surely. For example, look at this one:
>
> http://yudit.org/images/yudit-2.4.8.gif

[...]

> When implementing a terminal, never assume (the number of bytes) == (the 
> number of columns) or (the number of characters) < (the number of 
> columns). A given character can be one-column or two-column in the 
> current implementation of GRUB. If we want to support Hindi, things 
> will be more complicated.

I understand.

Thanks,
Marco