Unicode in dosemu under xterm

Unicode in dosemu under xterm

[Grigory Batalov]

Hello!

I got a problem running dosemu in terminal (xterm): there is no Russian symbols.
Xdosemu works well. This is my settings:

$_external_char_set = "utf8"
$_internal_char_set = "cp866"
$_layout = "ru"
$_X_font = "vgacyr"

$ locale
LANG=ru_RU.UTF-8
LC_CTYPE="ru_RU.UTF-8"
LC_NUMERIC="ru_RU.UTF-8"
LC_TIME="ru_RU.UTF-8"
LC_COLLATE="ru_RU.UTF-8"
LC_MONETARY="ru_RU.UTF-8"
LC_MESSAGES="ru_RU.UTF-8"
LC_PAPER="ru_RU.UTF-8"
LC_NAME="ru_RU.UTF-8"
LC_ADDRESS="ru_RU.UTF-8"
LC_TELEPHONE="ru_RU.UTF-8"
LC_MEASUREMENT="ru_RU.UTF-8"
LC_IDENTIFICATION="ru_RU.UTF-8"
LC_ALL=

I've found some interesting changes in boot.log
(first - xdosemu run, second - dosemu in xterm):

...
 CONF: DPMI base addr = 0xffffffff
 CONF: PM DOS API Translator on
 CONF: No DJGPP NULL deref checks: off
-CONF: dosemu running on X
+CONF: dosemu running on _xterm
 U: charset->unicode charset:utf8 00 -> symbol:0000...1
 U: charset->unicode charset:utf8 01 -> symbol:0001...1
...
 U: charset->unicode charset:utf8 02 -> symbol:0002...1
 U: charset->unicode charset:terminal_cp866 ff -> symbol:00a0...1
 U: unicode->charset charset:terminal_cp866 symbol:00a0 -> char:ff...1
-U: charset->unicode charset:X_keysym 61,00,00,00,00,00,00,00 -> symbol:0061...8
-U: charset->unicode charset:X_keysym 61,00,00,00,00,00,00,00 -> symbol:0061...8
-U: charset->unicode charset:X_keysym 73,00,00,00,00,00,00,00 -> symbol:0073...8
-U: charset->unicode charset:X_keysym 73,00,00,00,00,00,00,00 -> symbol:0073...8
-U: charset->unicode charset:X_keysym 64,00,00,00,00,00,00,00 -> symbol:0064...8
-U: charset->unicode charset:X_keysym 64,00,00,00,00,00,00,00 -> symbol:0064...8
-U: charset->unicode charset:X_keysym e4,ff,00,00,00,00,00,00 -> symbol:e134...8
-U: charset->unicode charset:X_keysym 08,fe,00,00,00,00,00,00 -> symbol:ffff...8
-U: charset->unicode charset:X_keysym e4,ff,00,00,00,00,00,00 -> symbol:e134...8
-U: charset->unicode charset:X_keysym e2,ff,00,00,00,00,00,00 -> symbol:e136...8
<Russian starts here>
-U: charset->unicode charset:X_keysym c6,06,00,00,00,00,00,00 -> symbol:0444...8
-U: charset->unicode charset:X_keysym c6,06,00,00,00,00,00,00 -> symbol:0444...8
-U: charset->unicode charset:X_keysym d9,06,00,00,00,00,00,00 -> symbol:044b...8
-U: charset->unicode charset:X_keysym d9,06,00,00,00,00,00,00 -> symbol:044b...8
-U: charset->unicode charset:X_keysym d7,06,00,00,00,00,00,00 -> symbol:0432...8
-U: charset->unicode charset:X_keysym d7,06,00,00,00,00,00,00 -> symbol:0432...8
-U: charset->unicode charset:X_keysym c1,06,00,00,00,00,00,00 -> symbol:0430...8
-U: charset->unicode charset:X_keysym c1,06,00,00,00,00,00,00 -> symbol:0430...8
+U: charset->unicode charset:utf8 61 -> symbol:0061...1
+U: charset->unicode charset:utf8 73 -> symbol:0073...1
+U: charset->unicode charset:utf8 64 -> symbol:0064...1
<Russian starts here>
+U: charset->unicode charset:utf8 d1,84 -> symbol:3444...2
+U: charset->unicode charset:utf8  -> symbol:ffff...18446744073709551615:84(Invalid or incomplete multibyte or wide character)
+U: charset->unicode charset:utf8 d1,8b -> symbol:344b...2
+U: charset->unicode charset:utf8  -> symbol:ffff...18446744073709551615:84(Invalid or incomplete multibyte or wide character)
+U: charset->unicode charset:utf8 d0,b2 -> symbol:3432...2
+U: charset->unicode charset:utf8  -> symbol:ffff...18446744073709551615:84(Invalid or incomplete multibyte or wide character)
+U: charset->unicode charset:utf8 d0,b0 -> symbol:3430...2
+U: charset->unicode charset:utf8  -> symbol:ffff...18446744073709551615:84(Invalid or incomplete multibyte or wide character)

Above I've typed in 'asd' letters, and then 'asdf' keys in Russian locale
(that sounds like 'fyva'). Also there was

+U: charset->unicode charset:cp866 e4 -> symbol:0444...1
+U: unicode->charset charset:utf8 symbol:0444 -> char:d1,84...2

So I don't understand, why d1,84 became later symbol:3444 instead of symbol:0444.
This is a 2.6.18 kernel on x86_64 platform.

Could anybody help me?

Re: Unicode in dosemu under xterm

[Bart Oldeman]

[-- Attachment #1: Type: text/plain, Size: 565 bytes --]

On 4/19/07, Grigory Batalov <bga@altlinux.org> wrote:
> Hello!
>
> I got a problem running dosemu in terminal (xterm): there is no Russian symbols.
> Xdosemu works well. This is my settings:
>
> $_external_char_set = "utf8"

One workaround is to leave this to "default" which takes the current
locale setting.
That does not solve flickering when you input non-ASCII though.

> So I don't understand, why d1,84 became later symbol:3444 instead of symbol:0444.

A bug in the UTF-8 code...

It's fixed in SVN rev 1736; you could also apply the attached instead.

Bart

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: utf8.diff --]
[-- Type: text/x-patch; name="utf8.diff", Size: 957 bytes --]

Index: src/plugin/term/keyb_slang.c
===================================================================
--- src/plugin/term/keyb_slang.c	(revision 1735)
+++ src/plugin/term/keyb_slang.c	(revision 1736)
@@ -1274,6 +1274,8 @@
 			/* rough draft version don't stop here... */
 			result = charset_to_unicode(&keyb_state.translate_state,
 				&symbol, keyb_state.kbp, keyb_state.kbcount);
+			if (result != -1 && result > keyb_state.Keystr_Len)
+				keyb_state.Keystr_Len = result;
 			k_printf("KBD: got %08x, result=%zx\n", symbol, result);
 		}
 
Index: src/plugin/translate/charsets/utf8.c
===================================================================
--- src/plugin/translate/charsets/utf8.c	(revision 1735)
+++ src/plugin/translate/charsets/utf8.c	(revision 1736)
@@ -47,7 +47,6 @@
 		goto bad_string;
 	}
 	if (len >= bytes_desired) {
-		result = ch;
 		for(i = 1; i < bytes_desired; i++) {
 			ch = str[i];
 			if ((ch >= 0x80) && (ch <= 0xBF)) {

Re: Unicode in dosemu under xterm

[Grigory Batalov]

On Fri, 20 Apr 2007 09:23:56 -0400
Bart Oldeman wrote:

> > I got a problem running dosemu in terminal (xterm): there is no Russian symbols.
> > Xdosemu works well. This is my settings:
> >
> > $_external_char_set = "utf8"
> 
> One workaround is to leave this to "default" which takes the current
> locale setting.
> That does not solve flickering when you input non-ASCII though.
> 
> > So I don't understand, why d1,84 became later symbol:3444 instead of symbol:0444.
> 
> A bug in the UTF-8 code...
> 
> It's fixed in SVN rev 1736; you could also apply the attached instead.

  Thanks a lot, now all goes well !