[PATCH] Re: Improved console UTF-8 support for the Linux kernel?

[PATCH] Re: Improved console UTF-8 support for the Linux kernel?

[Chris Heath]

Here is the patch that makes compose keys work.  I consider it a hack...
but it's simple and effective enough that it may be worth including. 
It's a hack because:

* Compose keys are still stored as 8-bit chars internally, but get
converted to UTF-8 as they are output.  Therefore you are still
restricted to a single 8-bit character set.

* Logically, keyboard input and console display are separated in the
kernel. When you switch in and out of Unicode mode you have to switch
both the keyboard and the display separately.  However, this patch
intertwines the two because it does its 8-bit to Unicode conversion
using a table that is designed for use by the display module.  (When the
display is in Unicode mode, this conversion table is unused, so why not
use it for the keyboard module?)

As you have already figured out, Suse is using this patch in their
distribution, so I figure it has had pretty wide testing already.

I have a couple of other patches on my website, which I am happy to
submit (or you are welcome to take), but this is the simplest and the
most popular.

Chris


--- a/include/linux/consolemap.h	2003-08-03 21:10:43.000000000 -0400
+++ b/include/linux/consolemap.h	2003-08-02 10:55:33.000000000 -0400
@@ -13,3 +13,4 @@
 extern unsigned char inverse_translate(struct vc_data *conp, int
glyph);
 extern unsigned short *set_translate(int m,int currcons);
 extern int conv_uni_to_pc(struct vc_data *conp, long ucs);
+extern u32 conv_8bit_to_uni(unsigned char c);
--- a/drivers/char/consolemap.c	2003-08-03 21:10:43.000000000 -0400
+++ b/drivers/char/consolemap.c	2003-08-02 10:52:55.000000000 -0400
@@ -633,6 +633,19 @@
 	if (p) p->readonly = rdonly;
 }
 
+/* may be called during an interrupt */
+u32 conv_8bit_to_uni(unsigned char c)
+{
+	/* 
+	 * Always use USER_MAP. This function is used by the keyboard,
+	 * which shouldn't be affected by G0/G1 switching, etc.
+	 * If the user map still contains default values, i.e. the 
+	 * direct-to-font mapping, then assume user is using Latin1.
+	 */
+	unsigned short uni = translations[USER_MAP][c];
+	return uni == (0xf000 | c) ? c : uni;
+}
+
 int
 conv_uni_to_pc(struct vc_data *conp, long ucs) 
 {
--- a/drivers/char/keyboard.c	2003-08-03 21:10:43.000000000 -0400
+++ b/drivers/char/keyboard.c	2003-08-02 10:58:49.000000000 -0400
@@ -35,6 +35,7 @@
 #include <linux/init.h>
 #include <linux/slab.h>
 
+#include <linux/consolemap.h>
 #include <linux/kbd_kern.h>
 #include <linux/kbd_diacr.h>
 #include <linux/vt_kern.h>
@@ -347,6 +348,15 @@
     	}
 }
 
+static void put_8bit(struct vc_data *vc, u8 c)
+{
+	if (kbd->kbdmode != VC_UNICODE || c < 32 || c == 127) 
+		/* Don't translate control chars */
+		put_queue(vc, c);
+	else
+		to_utf8(vc, conv_8bit_to_uni(c));
+}
+
 /* 
  * Called after returning from RAW mode or when changing consoles -
recompute
  * shift_down[] and shift_state from key_down[] maybe called when
keymap is
@@ -407,7 +417,7 @@
 	if (ch == ' ' || ch == d)
 		return d;
 
-	put_queue(vc, d);
+	put_8bit(vc, d);
 	return ch;
 }
 
@@ -417,7 +427,7 @@
 static void fn_enter(struct vc_data *vc, struct pt_regs *regs)
 {
 	if (diacr) {
-		put_queue(vc, diacr);
+		put_8bit(vc, diacr);
 		diacr = 0;
 	}
 	put_queue(vc, 13);
@@ -626,7 +636,7 @@
 		diacr = value;
 		return;
 	}
-	put_queue(vc, value);
+	put_8bit(vc, value);
 }
 
 /*

Re: [PATCH] Re: Improved console UTF-8 support for the Linux kernel?

[Jan Engelhardt]

>* Logically, keyboard input and console display are separated in the
>kernel. When you switch in and out of Unicode mode you have to switch

But they get agglumerated when it comes to ttys.

>As you have already figured out, Suse is using this patch in their
>distribution, so I figure it has had pretty wide testing already.

They actually don't, as of KOTD 20041202. Patching the tree with the three 
-CDH1 patches gave no fatal rejects (except a few linenoise), so the patches 
are not in there ATM.

>I have a couple of other patches on my website, which I am happy to
>submit (or you are welcome to take), but this is the simplest and the
>most popular.
>

Jan Engelhardt
-- 
ENOSPC

Re: [PATCH] Re: Improved console UTF-8 support for the Linux kernel?

[Andries Brouwer]

On Mon, Dec 13, 2004 at 01:50:24AM -0500, Chris Heath wrote:

> Logically, keyboard input and console display are separated in the
> kernel. When you switch in and out of Unicode mode you have to switch
> both the keyboard and the display separately.  However, this patch
> intertwines the two because it does its 8-bit to Unicode conversion
> using a table that is designed for use by the display module.
> 
> I have a couple of other patches on my website, which I am happy to
> submit (or you are welcome to take), but this is the simplest and the
> most popular.

Wouldnt mind looking at your other patches.
Will not submit this one - perhaps someone else likes it.
I consider the below completely unacceptable.

You cannot use knowledge about the setup of the output side
in the keyboard handler. These are independent in principle.

Andries


> +u32 conv_8bit_to_uni(unsigned char c)
> +{
> +	/* 
> +	 * Always use USER_MAP. This function is used by the keyboard,
> +	 * which shouldn't be affected by G0/G1 switching, etc.
> +	 * If the user map still contains default values, i.e. the 
> +	 * direct-to-font mapping, then assume user is using Latin1.
> +	 */
> +	unsigned short uni = translations[USER_MAP][c];
> +	return uni == (0xf000 | c) ? c : uni;
> +}

Re: [PATCH] Re: Improved console UTF-8 support for the Linux kernel?

[Simos Xenitellis]

Hi All,
I tried the patch of Chris on Fedora Core 2 and here is my take, from
the user's point of view.

The patch works, surprisingly very well.
It can be easily intergrated to the various distributions simply by
modifying the /etc/sysconfig/i18n and /etc/sysconfig/keyboard
configuration files.

The system scripts essentially call the following two commands (assuming
we are already in Unicode mode):
% setfont <font-name> -m <console-screen-map>
% loadkeys <keymap>

For example: 
For Spanish:
% setfont latarcyrheb-sun16 -m 8859-1
% loadkeys es

For Finish:
% setfont latarcyrheb-sun16 -m 8859-1
% loadkeys fi

For Greek:
% setfont iso07u-16 -m 8859-7
% loadkeys gr

The character and key maps used are the "old" 8-bit versions. setfonts
loads a Unicode map with the "-u" options instead of "-m". Also, the key
maps for a few languages have been updates (for example, "gr-utf" for
Greek). The new files (very few) cannot be used here. No need to update
them ;-).

I tried a few languages and what follows shows characters produced from
the console with composing. I used "vim" as my editor.

gr: Greek    ά έ ί ό ύ ώ ϊ ϋ ϊ Ά Έ Ί Ή Ύ Ϋ
es: Spanish  ñ á é í ý ú ü ï ÿ ä ë
nl: Dutch    á é í ó ú ý à è ì ò ù
cz: Czech    ä ë ö
us-ascentos: á é í ó ú ý ä ë ï ö ü ÿ
cf: french-canadian  à è ì ò ù
fi: Finish   ä ë ï ö ü â ê î ô û
fr  French   â ê î ô û ä ë ï ö ü ÿ

Therefore, from the user's point of view the patch works.

Cheers,
Simos Xenitellis