[PATCH 0/3] vt: make VS16 emoji widening controllable

[PATCH 0/3] vt: make VS16 emoji widening controllable

[Nicolas Pitre]

The VT turns a single-width character followed by VS16 (U+FE0F) into a
double-width one, as many terminals do for emoji presentation sequences
such as ‼️. Applications that compute widths with wcwidth() consider
VS16 zero-width though, and those tracking the cursor position
themselves get out of sync. With mosh, a single subject line in an
alpine message index was enough to leave the whole screen scrolled by
one line.

Terminals are split roughly evenly on this, so this series keeps the
current behavior by default and provides a way out:

Patch 1 fixes VS16 widening only taking effect after a double-width
character had been displayed, because the Unicode screen memory it
relies on is allocated on demand.

Patch 2 lets applications control VS16 widening with DEC private mode
2027 (from the Contour terminal's Unicode core proposal), with
vt.default_vs16_wide setting its initial and reset state.

Patch 3 adds DECRQM so applications can query mode states, including
2027, rather than waiting for a reply that never comes.

Tested with a program checking the DECRQM replies for all supported modes,
the VS16 width with the mode set, reset and after RIS, with and without
vt.default_vs16_wide=0, and that other CSI ... $ sequences are still
ignored.

diffstat:
 .../admin-guide/kernel-parameters.txt         |  10 ++
 drivers/tty/vt/vt.c                           | 114 +++++++++++++++++-
 include/linux/console_struct.h                |   1 +
 3 files changed, 124 insertions(+), 1 deletion(-)

-- 
2.55.0

[PATCH 1/3] vt: don't make VS16 widening depend on earlier screen contents

[Nicolas Pitre]

From: Nicolas Pitre <npitre@baylibre.com>

VS16 (U+FE0F) widens the preceding character only when that character
can be found in the Unicode screen memory. But that memory is allocated
on demand, when the first double-width character is displayed. Until
then vc_uniscr_getc() returns 0 and the VS16 is ignored.

The same sequence, e.g. U+203C U+FE0F, therefore takes one column on a
console that hasn't displayed any double-width character yet, and two
columns afterwards.

Allocate the Unicode screen memory when a VS16 shows up so the
preceding character is always available. vc_uniscr_check() populates
it from the current screen content, which includes that character.

Fixes: 95b05de0a566 ("vt: properly support zero-width Unicode code points")
Assisted-by: LLM
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
---
 drivers/tty/vt/vt.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
index 49a140b49eff..335253aec549 100644
--- a/drivers/tty/vt/vt.c
+++ b/drivers/tty/vt/vt.c
@@ -3128,6 +3128,15 @@ static int vc_process_ucs(struct vc_data *vc, int *c, int *tc)
 	}
 
 	/* From here the preceding character, if any, must be single-width. */
+
+	/*
+	 * The Unicode screen memory is allocated only when required. VS16
+	 * needs it to find the character it applies to, so it can't depend
+	 * on whether a double-width character was displayed earlier.
+	 */
+	if (curr_c == UCS_VS16)
+		vc_uniscr_check(vc);
+
 	prev_c = vc_uniscr_getc(vc, -1);
 
 	if (curr_c == UCS_VS16 && prev_c != 0) {
-- 
2.55.0

Re: [PATCH 1/3] vt: don't make VS16 widening depend on earlier screen contents

[sashiko-bot]

> From: Nicolas Pitre <npitre@baylibre.com>
> 
> VS16 (U+FE0F) widens the preceding character only when that character
> can be found in the Unicode screen memory. But that memory is allocated
> on demand, when the first double-width character is displayed. Until
> then vc_uniscr_getc() returns 0 and the VS16 is ignored.
> 
> The same sequence, e.g. U+203C U+FE0F, therefore takes one column on a
> console that hasn't displayed any double-width character yet, and two
> columns afterwards.
> 
> Allocate the Unicode screen memory when a VS16 shows up so the
> preceding character is always available. vc_uniscr_check() populates
> it from the current screen content, which includes that character.
> 
> Fixes: 95b05de0a566 ("vt: properly support zero-width Unicode code points")
> Assisted-by: LLM
> Signed-off-by: Nicolas Pitre <npitre@baylibre.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1792443833.git.npitre@baylibre.com?part=1

[PATCH 2/3] vt: add DEC private mode 2027 to control VS16 widening

[Nicolas Pitre]

From: Nicolas Pitre <npitre@baylibre.com>

VS16 (U+FE0F) after a single-width character currently turns it into a
double-width one, giving emoji presentation sequences such as U+203C
U+FE0F (‼️) a square aspect.

There is no consensus on this among terminals. According to the
ucs-detect survey [1], these widen such sequences to 2 cells, as the
Linux console does:

  kitty, foot, Ghostty, Contour, Konsole, iTerm2, Windows Terminal,
  mintty, Rio, tmux (3.4)

while these keep them at 1 cell:

  xterm, VTE based terminals (GNOME Terminal, XFCE Terminal, ...),
  Alacritty, st, urxvt, Apple Terminal, GNU Screen, libvterm,
  xterm.js (WezTerm only partially widens)

On the application side, glibc's wcwidth() is per code point and
returns 0 for VS16, and so do the tools relying on it. Even vim, which
has its own emoji width table, counts U+203C U+FE0F as 1 column.

Applications that track the cursor position themselves therefore get
out of sync with a widening terminal. For example, mosh writes full
width rows and moves with relative CR/LF. A row containing ‼️ is one
column wider on the console than in mosh's model, so its last cell
wraps to the next line and the whole screen ends up scrolled by one.
Others are reconsidering widening by default too: tmux 3.6 added the
"variation-selector-always-wide" option to turn it off, and xterm's
emoji width handling is configurable.

DEC private mode 2027, from the Contour terminal's Unicode core
proposal [2], is meant to give applications this control. With it set,
VS16 promotes a grapheme cluster to emoji presentation and a width of 2
cells. With it reset, the behavior is left to the terminal. Contour,
Ghostty, foot and WezTerm implement it.

Make VS16 widening follow that mode. Applications, or the user before
starting them, can opt out with CSI ? 2027 l, in which case VS16 is
zero-width as wcwidth() says. Only the VS16 part of the proposal is
implemented here.

The initial and reset state of the mode comes from the new
vt.default_vs16_wide parameter, similar to vt.default_utf8. It
defaults to 1, preserving the existing behavior, and can be set to 0
on systems where applications relying on wcwidth() are the norm.

[1] https://ucs-detect.readthedocs.io/results.html
[2] https://github.com/contour-terminal/terminal-unicode-core

Assisted-by: LLM
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
---
 Documentation/admin-guide/kernel-parameters.txt | 10 ++++++++++
 drivers/tty/vt/vt.c                             | 15 ++++++++++++++-
 include/linux/console_struct.h                  |  1 +
 3 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index e75344f4e0cd..983afb6c531e 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -8451,6 +8451,16 @@ Kernel parameters
 			Default is 1, i.e. UTF-8 mode is enabled for all
 			newly opened terminals.
 
+	vt.default_vs16_wide=
+			[VT]
+			Format=<0|1>
+			Set system-wide default for whether VS16 (U+FE0F)
+			turns the preceding single-width character into a
+			double-width one, as in emoji presentation sequences.
+			This is the initial and reset state of DEC private
+			mode 2027 on all tty's, which applications can also
+			change with CSI ? 2027 h/l. Default is 1.
+
 	vt.global_cursor_default=
 			[VT]
 			Format=<-1|0|1>
diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
index 335253aec549..4d5c9cabd058 100644
--- a/drivers/tty/vt/vt.c
+++ b/drivers/tty/vt/vt.c
@@ -166,6 +166,9 @@ EXPORT_SYMBOL(global_cursor_default);
 static int cur_default = CUR_UNDERLINE;
 module_param(cur_default, int, S_IRUGO | S_IWUSR);
 
+static bool default_vs16_wide = true;
+module_param(default_vs16_wide, bool, 0644);
+
 /*
  * ignore_poke: don't unblank the screen when things are typed.  This is
  * mainly for the privacy of braille terminal users.
@@ -1984,6 +1987,7 @@ enum {
 	CSI_DEC_hl_MOUSE_VT200	= 1000,
 	CSI_DEC_hl_ALT_SCREEN	= 1049,
 	CSI_DEC_hl_BRACKETED_PASTE = 2004,
+	CSI_DEC_hl_GRAPHEME_WIDTH = 2027,	/* VS16 widening */
 };
 
 /* console_lock is held */
@@ -2039,6 +2043,9 @@ static void csi_DEC_hl(struct vc_data *vc, bool on_off)
 		case CSI_DEC_hl_BRACKETED_PASTE:
 			vc->vc_bracketed_paste = on_off;
 			break;
+		case CSI_DEC_hl_GRAPHEME_WIDTH:
+			vc->vc_vs16_wide = on_off;
+			break;
 		case CSI_DEC_hl_ALT_SCREEN:
 			if (on_off)
 				enter_alt_screen(vc);
@@ -2284,6 +2291,7 @@ static void reset_terminal(struct vc_data *vc, int do_clear)
 	vc->vc_decawm		= 1;
 	vc->vc_deccm		= global_cursor_default;
 	vc->vc_decim		= 0;
+	vc->vc_vs16_wide	= default_vs16_wide;
 
 	if (vc->vc_saved_screen != NULL) {
 		kfree(vc->vc_saved_screen);
@@ -3139,12 +3147,17 @@ static int vc_process_ucs(struct vc_data *vc, int *c, int *tc)
 
 	prev_c = vc_uniscr_getc(vc, -1);
 
-	if (curr_c == UCS_VS16 && prev_c != 0) {
+	if (curr_c == UCS_VS16 && prev_c != 0 && vc->vc_vs16_wide) {
 		/*
 		 * VS16 (U+FE0F) is special. It typically turns the preceding
 		 * single-width character into a double-width one. Let it
 		 * have a width of 1 effectively making the combination with
 		 * the preceding character double-width.
+		 *
+		 * This can be disabled with DEC private mode 2027 for
+		 * applications that compute widths per code point with
+		 * wcwidth(), where VS16 is always zero-width. Otherwise
+		 * they lose track of the cursor position.
 		 */
 		*tc = ' ';
 		return 1;
diff --git a/include/linux/console_struct.h b/include/linux/console_struct.h
index fe915afdece5..8a1031cd097e 100644
--- a/include/linux/console_struct.h
+++ b/include/linux/console_struct.h
@@ -167,6 +167,7 @@ struct vc_data {
 	unsigned int	vc_decawm	: 1;	/* Autowrap Mode */
 	unsigned int	vc_deccm	: 1;	/* Cursor Visible */
 	unsigned int	vc_decim	: 1;	/* Insert Mode */
+	unsigned int	vc_vs16_wide	: 1;	/* VS16 makes preceding char wide */
 	/* misc */
 	unsigned int	vc_priv		: 3;
 	unsigned int	vc_need_wrap	: 1;
-- 
2.55.0

Re: [PATCH 2/3] vt: add DEC private mode 2027 to control VS16 widening

[sashiko-bot]

> From: Nicolas Pitre <npitre@baylibre.com>
> 
> VS16 (U+FE0F) after a single-width character currently turns it into a
> double-width one, giving emoji presentation sequences such as U+203C
> U+FE0F (‼️) a square aspect.
> 
> There is no consensus on this among terminals. According to the
> ucs-detect survey [1], these widen such sequences to 2 cells, as the
> Linux console does:
> 
>   kitty, foot, Ghostty, Contour, Konsole, iTerm2, Windows Terminal,
>   mintty, Rio, tmux (3.4)
> 
> while these keep them at 1 cell:
> 
> [ ... ]
> [1] https://ucs-detect.readthedocs.io/results.html
> [2] https://github.com/contour-terminal/terminal-unicode-core
> 
> Assisted-by: LLM
> Signed-off-by: Nicolas Pitre <npitre@baylibre.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1792443833.git.npitre@baylibre.com?part=2

[PATCH 3/3] vt: add DECRQM support to report mode states

[Nicolas Pitre]

From: Nicolas Pitre <npitre@baylibre.com>

Applications currently have no way to find out whether the VT supports
a given mode, nor what its current state is. DECRQM, "CSI Ps $ p" for
ANSI modes and "CSI ? Ps $ p" for DEC private modes, is the standard
query for this. But the '$' intermediate makes the parser ignore the
whole sequence, leaving applications waiting for a reply that never
comes.

This matters especially for mode 2027, whose specification relies on
DECRQM for applications to detect support before enabling it.

Add an ESdollar parser state for "CSI parameters $", and answer a final
'p' with DECRPM, "CSI [?] Ps ; Pm $ y", where Pm is 1 (set), 2 (reset),
0 (not recognized) or 4 (permanently reset). All modes handled by
csi_DEC_hl() and csi_hl() are reported, with 132-column mode reported
as permanently reset since it isn't implemented. Other sequences using
the '$' intermediate are still ignored.

Assisted-by: LLM
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
---
 drivers/tty/vt/vt.c | 90 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)

diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
index 4d5c9cabd058..22dfab8de7bc 100644
--- a/drivers/tty/vt/vt.c
+++ b/drivers/tty/vt/vt.c
@@ -2083,6 +2083,82 @@ static void csi_hl(struct vc_data *vc, bool on_off)
 		}
 }
 
+/* DECRPM mode states */
+enum {
+	DECRPM_NOT_RECOGNIZED	= 0,
+	DECRPM_SET		= 1,
+	DECRPM_RESET		= 2,
+	DECRPM_PERMANENTLY_RESET = 4,
+};
+
+static int decrpm_state(bool set)
+{
+	return set ? DECRPM_SET : DECRPM_RESET;
+}
+
+/* State of a DEC private mode as handled by csi_DEC_hl() */
+static int csi_DEC_mode(struct vc_data *vc, unsigned int mode)
+{
+	switch (mode) {
+	case CSI_DEC_hl_CURSOR_KEYS:
+		return decrpm_state(is_kbd(vc, decckm));
+	case CSI_DEC_hl_132_COLUMNS:
+		return DECRPM_PERMANENTLY_RESET;
+	case CSI_DEC_hl_REVERSE_VIDEO:
+		return decrpm_state(vc->vc_decscnm);
+	case CSI_DEC_hl_ORIGIN_MODE:
+		return decrpm_state(vc->vc_decom);
+	case CSI_DEC_hl_AUTOWRAP:
+		return decrpm_state(vc->vc_decawm);
+	case CSI_DEC_hl_AUTOREPEAT:
+		return decrpm_state(is_kbd(vc, decarm));
+	case CSI_DEC_hl_MOUSE_X10:
+		return decrpm_state(vc->vc_report_mouse == 1);
+	case CSI_DEC_hl_SHOW_CURSOR:
+		return decrpm_state(vc->vc_deccm);
+	case CSI_DEC_hl_MOUSE_VT200:
+		return decrpm_state(vc->vc_report_mouse == 2);
+	case CSI_DEC_hl_ALT_SCREEN:
+		return decrpm_state(vc->vc_saved_screen != NULL);
+	case CSI_DEC_hl_BRACKETED_PASTE:
+		return decrpm_state(vc->vc_bracketed_paste);
+	case CSI_DEC_hl_GRAPHEME_WIDTH:
+		return decrpm_state(vc->vc_vs16_wide);
+	}
+	return DECRPM_NOT_RECOGNIZED;
+}
+
+/* State of an ANSI mode as handled by csi_hl() */
+static int csi_ANSI_mode(struct vc_data *vc, unsigned int mode)
+{
+	switch (mode) {
+	case CSI_hl_DISPLAY_CTRL:
+		return decrpm_state(vc->vc_disp_ctrl);
+	case CSI_hl_INSERT:
+		return decrpm_state(vc->vc_decim);
+	case CSI_hl_AUTO_NL:
+		return decrpm_state(is_kbd(vc, lnm));
+	}
+	return DECRPM_NOT_RECOGNIZED;
+}
+
+/*
+ * DECRQM: request mode, "CSI Ps $ p" for ANSI modes and "CSI ? Ps $ p" for
+ * DEC private modes. Reply with DECRPM: "CSI [?] Ps ; Pm $ y".
+ */
+/* console_lock is held */
+static void csi_RQM(struct tty_struct *tty, struct vc_data *vc)
+{
+	unsigned int mode = vc->vc_par[0];
+	bool dec = vc->vc_priv == EPdec;
+	char buf[32];
+	int len;
+
+	len = snprintf(buf, sizeof(buf), "\033[%s%u;%d$y", dec ? "?" : "",
+		       mode, dec ? csi_DEC_mode(vc, mode) : csi_ANSI_mode(vc, mode));
+	respond_string(buf, len, tty->port);
+}
+
 enum CSI_right_square_bracket {
 	CSI_RSB_COLOR_FOR_UNDERLINE		= 1,
 	CSI_RSB_COLOR_FOR_HALF_BRIGHT		= 2,
@@ -2233,6 +2309,7 @@ static void restore_cur(struct vc_data *vc)
  * @ESsetG1:		ESC ) parsed
  * @ESpercent:		ESC % parsed
  * @EScsiignore:	CSI [0x20-0x3f] parsed
+ * @ESdollar:		CSI parameters $ parsed -- final char expected
  * @ESnonstd:		OSC parsed
  * @ESpalette:		OSC P parsed
  * @ESosc:		OSC [0-9] parsed
@@ -2254,6 +2331,7 @@ enum vc_ctl_state {
 	ESsetG1,
 	ESpercent,
 	EScsiignore,
+	ESdollar,
 	ESnonstd,
 	ESpalette,
 	ESosc,
@@ -2788,6 +2866,9 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, u8 c)
 			vc->vc_par[vc->vc_npar] *= 10;
 			vc->vc_par[vc->vc_npar] += c - '0';
 			return;
+		case '$':
+			vc->vc_state = ESdollar;
+			return;
 		}
 		if (c >= ASCII_CSI_IGNORE_FIRST && c <= ASCII_CSI_IGNORE_LAST) {
 			vc->vc_state = EScsiignore;
@@ -2846,6 +2927,15 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, u8 c)
 			return;
 		vc->vc_state = ESnormal;
 		return;
+	case ESdollar:	/* CSI parameters $, final char expected */
+		if (c >= ASCII_CSI_IGNORE_FIRST && c <= ASCII_CSI_IGNORE_LAST) {
+			vc->vc_state = EScsiignore;
+			return;
+		}
+		vc->vc_state = ESnormal;
+		if (c == 'p' && (vc->vc_priv == EPdec || vc->vc_priv == EPecma))
+			csi_RQM(tty, vc);
+		return;
 	case ESpercent:	/* ESC % */
 		vc->vc_state = ESnormal;
 		switch (c) {
-- 
2.55.0

Re: [PATCH 3/3] vt: add DECRQM support to report mode states

[sashiko-bot]

> From: Nicolas Pitre <npitre@baylibre.com>
> 
> Applications currently have no way to find out whether the VT supports
> a given mode, nor what its current state is. DECRQM, "CSI Ps $ p" for
> ANSI modes and "CSI ? Ps $ p" for DEC private modes, is the standard
> query for this. But the '$' intermediate makes the parser ignore the
> whole sequence, leaving applications waiting for a reply that never
> comes.
> 
> This matters especially for mode 2027, whose specification relies on
> DECRQM for applications to detect support before enabling it.
> 
> Add an ESdollar parser state for "CSI parameters $", and answer a final
> 'p' with DECRPM, "CSI [?] Ps ; Pm $ y", where Pm is 1 (set), 2 (reset),
> 0 (not recognized) or 4 (permanently reset). All modes handled by
> csi_DEC_hl() and csi_hl() are reported, with 132-column mode reported
> as permanently reset since it isn't implemented. Other sequences using
> the '$' intermediate are still ignored.
> 
> Assisted-by: LLM
> Signed-off-by: Nicolas Pitre <npitre@baylibre.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1792443833.git.npitre@baylibre.com?part=3