Re: [2.6 patch] UTF-8 fixes in comments

Re: [2.6 patch] UTF-8 fixes in comments

[Samuel Thibault]

Willy Tarreau wrote:
> 3) if I enter Alt-196, I get a "Ä". Flushing the buffer shows that od
> got two bytes: c3 84.

Confirmed.

Try init=/bin/stty -a, that will show

-iutf8

So there is little wonder that canonical mode does not work as expected.

Try init=/bin/sh, from that shell run stty iutf8. Then things will work
fine.  The fix is thus just to make the VT's tty initial iutf8 setup
follow vt.default_utf8.

Samuel

Re: [2.6 patch] UTF-8 fixes in comments

[Chris Adams]

Once upon a time, Samuel Thibault  <samuel.thibault@ens-lyon.org> said:
>Try init=/bin/sh, from that shell run stty iutf8. Then things will work
>fine.  The fix is thus just to make the VT's tty initial iutf8 setup
>follow vt.default_utf8.

You may also need to select a UTF-8 locale (e.g. LANG="en_US.UTF-8") for
programs like bash to handle this correctly.

-- 
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: [2.6 patch] UTF-8 fixes in comments

[Samuel Thibault]

Chris Adams wrote:
> Once upon a time, Samuel Thibault  <samuel.thibault@ens-lyon.org> said:
> >Try init=/bin/sh, from that shell run stty iutf8. Then things will work
> >fine.  The fix is thus just to make the VT's tty initial iutf8 setup
> >follow vt.default_utf8.
> 
> You may also need to select a UTF-8 locale (e.g. LANG="en_US.UTF-8") for
> programs like bash to handle this correctly.

Yes of course, but here the purpose was _not_ programs like bash, but
the canonical mode (i.e. programs like cat etc.), for which the LANG
variable has no effect, only iutf8 has.

Samuel

Re: [2.6 patch] UTF-8 fixes in comments

[Willy Tarreau]

On Wed, Apr 30, 2008 at 10:38:32AM +0100, Samuel Thibault wrote:
> Chris Adams wrote:
> > Once upon a time, Samuel Thibault  <samuel.thibault@ens-lyon.org> said:
> > >Try init=/bin/sh, from that shell run stty iutf8. Then things will work
> > >fine.  The fix is thus just to make the VT's tty initial iutf8 setup
> > >follow vt.default_utf8.
> > 
> > You may also need to select a UTF-8 locale (e.g. LANG="en_US.UTF-8") for
> > programs like bash to handle this correctly.
> 
> Yes of course, but here the purpose was _not_ programs like bash, but
> the canonical mode (i.e. programs like cat etc.), for which the LANG
> variable has no effect, only iutf8 has.

exactly, thanks for understanding my problem Samuel :-)

Willy

Re: [2.6 patch] UTF-8 fixes in comments

[Willy Tarreau]

On Wed, Apr 30, 2008 at 01:08:51AM +0100, Samuel Thibault wrote:
> Willy Tarreau wrote:
> > 3) if I enter Alt-196, I get a "Ä". Flushing the buffer shows that od
> > got two bytes: c3 84.
> 
> Confirmed.
> 
> Try init=/bin/stty -a, that will show
> 
> -iutf8
> 
> So there is little wonder that canonical mode does not work as expected.
> 
> Try init=/bin/sh, from that shell run stty iutf8. Then things will work
> fine.  The fix is thus just to make the VT's tty initial iutf8 setup
> follow vt.default_utf8.

Will try that on a more recent install. Mine's stty does not support
this option. Your analysis makes quite a lot of sense, and such a fix
would wipe part of my annoyances/anger with this recent change.

Thanks,
Willy

Re: [2.6 patch] UTF-8 fixes in comments

[Samuel Thibault]

Hello,

Willy Tarreau, le Wed 30 Apr 2008 21:49:20 +0200, a écrit :
> On Wed, Apr 30, 2008 at 01:08:51AM +0100, Samuel Thibault wrote:
> > Willy Tarreau wrote:
> > > 3) if I enter Alt-196, I get a "Ä". Flushing the buffer shows that od
> > > got two bytes: c3 84.
> > 
> > Confirmed.
> > 
> > Try init=/bin/stty -a, that will show
> > 
> > -iutf8
> > 
> > So there is little wonder that canonical mode does not work as expected.
> > 
> > Try init=/bin/sh, from that shell run stty iutf8. Then things will work
> > fine.  The fix is thus just to make the VT's tty initial iutf8 setup
> > follow vt.default_utf8.
> 
> Will try that on a more recent install. Mine's stty does not support
> this option. Your analysis makes quite a lot of sense, and such a fix
> would wipe part of my annoyances/anger with this recent change.

Can you give the patch below a try?
Dynamic per-VT utf-8 switch should also work, provided that you reopen
the VT (i.e. log out).

Samuel



Set IUTF8 as appropriate on VT tty open.

Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>

--- linux/drivers/char/vt.c.orig	2008-05-04 00:37:50.000000000 +0100
+++ linux/drivers/char/vt.c	2008-05-04 00:47:39.000000000 +0100
@@ -2723,6 +2723,10 @@ static int con_open(struct tty_struct *t
 				tty->winsize.ws_row = vc_cons[currcons].d->vc_rows;
 				tty->winsize.ws_col = vc_cons[currcons].d->vc_cols;
 			}
+			if (vc->vc_utf)
+				tty->termios->c_iflag |= IUTF8;
+			else
+				tty->termios->c_iflag &= ~IUTF8;
 			release_console_sem();
 			vcs_make_sysfs(tty);
 			return ret;
@@ -2899,6 +2903,8 @@ int __init vty_init(void)
 	console_driver->minor_start = 1;
 	console_driver->type = TTY_DRIVER_TYPE_CONSOLE;
 	console_driver->init_termios = tty_std_termios;
+	if (default_utf8)
+		console_driver->init_termios.c_iflag |= IUTF8;
 	console_driver->flags = TTY_DRIVER_REAL_RAW | TTY_DRIVER_RESET_TERMIOS;
 	tty_set_operations(console_driver, &con_ops);
 	if (tty_register_driver(console_driver))

Re: [2.6 patch] UTF-8 fixes in comments

[Willy Tarreau]

Hi Samuel,

On Sun, May 04, 2008 at 12:50:28AM +0100, Samuel Thibault wrote:
> Can you give the patch below a try?
> Dynamic per-VT utf-8 switch should also work, provided that you reopen
> the VT (i.e. log out).

I confirm that your patch works perfectly for me. Now backspace correctly
removes multi-byte characters. My bash is still fooled though but as Alan
explained it, it's readline which has to be upgraded now.

Thanks!
Willy

Fix VT canonical input in UTF-8 mode [Was: UTF-8 fixes in comments]

[Samuel Thibault]

Samuel Thibault, le Sun 04 May 2008 00:50:27 +0100, a écrit :
> Willy Tarreau, le Wed 30 Apr 2008 21:49:20 +0200, a écrit :
> > On Wed, Apr 30, 2008 at 01:08:51AM +0100, Samuel Thibault wrote:
> > > Willy Tarreau wrote:
> > > > 3) if I enter Alt-196, I get a "Ä". Flushing the buffer shows that od
> > > > got two bytes: c3 84.
> > > 
> > > Confirmed.
> > > 
> > > Try init=/bin/stty -a, that will show
> > > 
> > > -iutf8
> > > 
> > > So there is little wonder that canonical mode does not work as expected.
> > > 
> > > Try init=/bin/sh, from that shell run stty iutf8. Then things will work
> > > fine.  The fix is thus just to make the VT's tty initial iutf8 setup
> > > follow vt.default_utf8.
> > 
> > Will try that on a more recent install. Mine's stty does not support
> > this option. Your analysis makes quite a lot of sense, and such a fix
> > would wipe part of my annoyances/anger with this recent change.
> 
> Can you give the patch below a try?
> Dynamic per-VT utf-8 switch should also work, provided that you reopen
> the VT (i.e. log out).

Willy Tarreau, le Sun 04 May 2008 10:55:14 +0200, a écrit :
> I confirm that your patch works perfectly for me. Now backspace correctly
> removes multi-byte characters. My bash is still fooled though but as Alan
> explained it, it's readline which has to be upgraded now.

I guess this is suitable for the stable trees of 2.6.24 and 2.6.25
(where UTF-8 is by default now).




Set IUTF8 as appropriate on VT tty open.

Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>

--- linux/drivers/char/vt.c.orig	2008-05-04 00:37:50.000000000 +0100
+++ linux/drivers/char/vt.c	2008-05-04 00:47:39.000000000 +0100
@@ -2723,6 +2723,10 @@ static int con_open(struct tty_struct *t
 				tty->winsize.ws_row = vc_cons[currcons].d->vc_rows;
 				tty->winsize.ws_col = vc_cons[currcons].d->vc_cols;
 			}
+			if (vc->vc_utf)
+				tty->termios->c_iflag |= IUTF8;
+			else
+				tty->termios->c_iflag &= ~IUTF8;
 			release_console_sem();
 			vcs_make_sysfs(tty);
 			return ret;
@@ -2899,6 +2903,8 @@ int __init vty_init(void)
 	console_driver->minor_start = 1;
 	console_driver->type = TTY_DRIVER_TYPE_CONSOLE;
 	console_driver->init_termios = tty_std_termios;
+	if (default_utf8)
+		console_driver->init_termios.c_iflag |= IUTF8;
 	console_driver->flags = TTY_DRIVER_REAL_RAW | TTY_DRIVER_RESET_TERMIOS;
 	tty_set_operations(console_driver, &con_ops);
 	if (tty_register_driver(console_driver))

Re: Fix VT canonical input in UTF-8 mode [Was: UTF-8 fixes in comments]

[Willy Tarreau]

On Sun, May 04, 2008 at 11:25:54AM +0100, Samuel Thibault wrote:
> Willy Tarreau, le Sun 04 May 2008 10:55:14 +0200, a écrit :
> > I confirm that your patch works perfectly for me. Now backspace correctly
> > removes multi-byte characters. My bash is still fooled though but as Alan
> > explained it, it's readline which has to be upgraded now.
> 
> I guess this is suitable for the stable trees of 2.6.24 and 2.6.25
> (where UTF-8 is by default now).

agreed.

> Set IUTF8 as appropriate on VT tty open.
> 
> Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>

You should have added:  CC: stable@kernel.org here so that the stable
team automatically gets notified when it's merged into mainline.

Thanks!
Willy

Re: Fix VT canonical input in UTF-8 mode [Was: UTF-8 fixes in comments]

[Andrew Morton]

On Sun, 4 May 2008 11:25:54 +0100
Samuel Thibault <samuel.thibault@ens-lyon.org> wrote:

> Samuel Thibault, le Sun 04 May 2008 00:50:27 +0100, a écrit :
> > Willy Tarreau, le Wed 30 Apr 2008 21:49:20 +0200, a écrit :
> > > On Wed, Apr 30, 2008 at 01:08:51AM +0100, Samuel Thibault wrote:
> > > > Willy Tarreau wrote:
> > > > > 3) if I enter Alt-196, I get a "Ä". Flushing the buffer shows that od
> > > > > got two bytes: c3 84.
> > > > 
> > > > Confirmed.
> > > > 
> > > > Try init=/bin/stty -a, that will show
> > > > 
> > > > -iutf8
> > > > 
> > > > So there is little wonder that canonical mode does not work as expected.
> > > > 
> > > > Try init=/bin/sh, from that shell run stty iutf8. Then things will work
> > > > fine.  The fix is thus just to make the VT's tty initial iutf8 setup
> > > > follow vt.default_utf8.
> > > 
> > > Will try that on a more recent install. Mine's stty does not support
> > > this option. Your analysis makes quite a lot of sense, and such a fix
> > > would wipe part of my annoyances/anger with this recent change.
> > 
> > Can you give the patch below a try?
> > Dynamic per-VT utf-8 switch should also work, provided that you reopen
> > the VT (i.e. log out).
> 
> Willy Tarreau, le Sun 04 May 2008 10:55:14 +0200, a écrit :
> > I confirm that your patch works perfectly for me. Now backspace correctly
> > removes multi-byte characters. My bash is still fooled though but as Alan
> > explained it, it's readline which has to be upgraded now.
> 
> I guess this is suitable for the stable trees of 2.6.24 and 2.6.25
> (where UTF-8 is by default now).
> 
> 
> 
> 
> Set IUTF8 as appropriate on VT tty open.
> 
> Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>

That changelog is pretty darn terse :(  I'll often go through
the email ladder and try to extract the missing information
but this time I don't really see it there.

Things like: what is the kernel's current behaviour, why does
it behave that way, how does the patch fix it?

Thanks.

Re: Fix VT canonical input in UTF-8 mode [Was: UTF-8 fixes in comments]

[Samuel Thibault]

Andrew Morton, le Mon 05 May 2008 16:00:44 -0700, a écrit :
> > Set IUTF8 as appropriate on VT tty open.
> > 
> > Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
> 
> That changelog is pretty darn terse :( 

Erf, sorry.

> I'll often go through
> the email ladder and try to extract the missing information
> but this time I don't really see it there.
> 
> Things like: what is the kernel's current behaviour, why does
> it behave that way, how does the patch fix it?

Well, it's more an implementation than a fix. Let's try again:



For e.g. proper TTY canonical support, IUTF8 termios flag has to be set
as appropriate.  Linux used to not care about setting that flag for VT
TTYs.

This patch fixes that by activating it according to the current mode
of the VT, and sets the default value according to the vt.default_utf8
parameter.



Samuel