[PATCH v3 1/3] fbcon: Include font geometry in the rotated font cache

[PATCH v3 1/3] fbcon: Include font geometry in the rotated font cache

[Yao Yiqi]

From: Yao Yiqi <yaoyiqi3@huawei.com>

fbcon_rotate_font() caches a rotated copy of the console font in
par->rotated.buf and skips the rebuild when the cache appears to be
up to date. The cache is keyed on the source font's data pointer and
the rotation alone, on the assumption that an unchanged fontdata
pointer implies an unchanged font.

That assumption does not hold. Raw font data (font_data_t) stores no
glyph geometry; width, height and charcount are tracked separately in
vc->vc_font. When a font is uploaded, fbcon_set_font() deduplicates
it against the fonts of all consoles with font_data_is_equal(), which
compares only the byte count, the checksum and the raw bytes. Two
uploads of the same data that declare different dimensions therefore
end up sharing one font_data_t, and a console can install a new
geometry while its fontdata pointer stays the same.

The early-out in fbcon_rotate_font() then skips the rebuild, and the
rotated blitting routines index the stale buffer with cell sizes
computed from the new geometry:

	src = par->rotated.buf + (scr_readw(s) & charmask) * cellsize;

Rotation swaps a glyph's width and height, so the size of the rotated
cell depends on pitch rounding and differs between geometries. A 4x16
font whose 4096 bytes are re-uploaded as 16x8 on a CW-rotated console,
for example, grows the rotated cell size from 8 to 16 bytes, and the
largest character then reads past the end of the 2048-byte buffer that
was built for the 4x16 geometry.

The key has never included the geometry. Its exposure was limited,
but not eliminated, while the deduplication in fbcon_set_font() also
compared the consoles' font widths and total byte counts: with the
charcount restricted to 256 or 512, differing uploads could then only
trade height against charcount (such as 4x18x256 against 4x9x512),
and even those can change the rotated cell size. Commit 1e3c49aa03fb
("lib/fonts: Compare font data for equality with font_data_is_equal()")
dropped the width comparison, so identical bytes now share font data
across arbitrary dimensions.

Record the geometry that the buffer was built from and compare it in
the early-out, so a geometry change always triggers a rebuild.

Fixes: 6cc50e1c5b57 ("[PATCH] fbcon: Console Rotation - Add support to rotate font bitmap")
Cc: stable@vger.kernel.org
Signed-off-by: Yao Yiqi <yaoyiqi3@huawei.com>
---
 drivers/video/fbdev/core/fbcon.h        | 9 ++++++---
 drivers/video/fbdev/core/fbcon_rotate.c | 9 ++++++++-
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/drivers/video/fbdev/core/fbcon.h b/drivers/video/fbdev/core/fbcon.h
index 407d207b14f1..bfbd70f8b1cd 100644
--- a/drivers/video/fbdev/core/fbcon.h
+++ b/drivers/video/fbdev/core/fbcon.h
@@ -89,10 +89,13 @@ struct fbcon_par {
 	char  *cursor_data;
 #ifdef CONFIG_FRAMEBUFFER_CONSOLE_ROTATION
 	struct {
-		font_data_t *fontdata;  /* source font */
-		u8 *buf;                /* rotated glyphs */
+		font_data_t *fontdata;       /* source font */
+		u8 *buf;                     /* rotated glyphs */
 		size_t bufsize;
-		int buf_rotate;         /* rotation of buf */
+		int buf_rotate;              /* rotation of buf */
+		unsigned int buf_width;      /* geometry of source font */
+		unsigned int buf_height;
+		unsigned int buf_charcount;
 	} rotated;
 #endif
 	u8    *cursor_src;
diff --git a/drivers/video/fbdev/core/fbcon_rotate.c b/drivers/video/fbdev/core/fbcon_rotate.c
index 6cdbc96eeca6..660a3fa81bae 100644
--- a/drivers/video/fbdev/core/fbcon_rotate.c
+++ b/drivers/video/fbdev/core/fbcon_rotate.c
@@ -21,11 +21,18 @@ int fbcon_rotate_font(struct fb_info *info, struct vc_data *vc)
 	unsigned char *buf;
 	int ret;
 
-	if (par->p->fontdata == par->rotated.fontdata && par->rotate == par->rotated.buf_rotate)
+	if (par->p->fontdata == par->rotated.fontdata &&
+	    par->rotate == par->rotated.buf_rotate &&
+	    vc->vc_font.width == par->rotated.buf_width &&
+	    vc->vc_font.height == par->rotated.buf_height &&
+	    vc->vc_font.charcount == par->rotated.buf_charcount)
 		return 0;
 
 	par->rotated.fontdata = par->p->fontdata;
 	par->rotated.buf_rotate = par->rotate;
+	par->rotated.buf_width = vc->vc_font.width;
+	par->rotated.buf_height = vc->vc_font.height;
+	par->rotated.buf_charcount = vc->vc_font.charcount;
 
 	if (info->fbops->fb_sync)
 		info->fbops->fb_sync(info);
-- 
2.56.0

Re: [PATCH v3 1/3] fbcon: Include font geometry in the rotated font cache

[Helge Deller]

On 10/16/26 09:30, Yao Yiqi wrote:
> From: Yao Yiqi <yaoyiqi3@huawei.com>
> 
> fbcon_rotate_font() caches a rotated copy of the console font in
> par->rotated.buf and skips the rebuild when the cache appears to be
> up to date. The cache is keyed on the source font's data pointer and
> the rotation alone, on the assumption that an unchanged fontdata
> pointer implies an unchanged font.
> 
> That assumption does not hold. Raw font data (font_data_t) stores no
> glyph geometry; width, height and charcount are tracked separately in
> vc->vc_font. When a font is uploaded, fbcon_set_font() deduplicates
> it against the fonts of all consoles with font_data_is_equal(), which
> compares only the byte count, the checksum and the raw bytes. Two
> uploads of the same data that declare different dimensions therefore
> end up sharing one font_data_t, and a console can install a new
> geometry while its fontdata pointer stays the same.
> 
> The early-out in fbcon_rotate_font() then skips the rebuild, and the
> rotated blitting routines index the stale buffer with cell sizes
> computed from the new geometry:
> 
> 	src = par->rotated.buf + (scr_readw(s) & charmask) * cellsize;
> 
> Rotation swaps a glyph's width and height, so the size of the rotated
> cell depends on pitch rounding and differs between geometries. A 4x16
> font whose 4096 bytes are re-uploaded as 16x8 on a CW-rotated console,
> for example, grows the rotated cell size from 8 to 16 bytes, and the
> largest character then reads past the end of the 2048-byte buffer that
> was built for the 4x16 geometry.
> 
> The key has never included the geometry. Its exposure was limited,
> but not eliminated, while the deduplication in fbcon_set_font() also
> compared the consoles' font widths and total byte counts: with the
> charcount restricted to 256 or 512, differing uploads could then only
> trade height against charcount (such as 4x18x256 against 4x9x512),
> and even those can change the rotated cell size. Commit 1e3c49aa03fb
> ("lib/fonts: Compare font data for equality with font_data_is_equal()")
> dropped the width comparison, so identical bytes now share font data
> across arbitrary dimensions.
> 
> Record the geometry that the buffer was built from and compare it in
> the early-out, so a geometry change always triggers a rebuild.
> 
> Fixes: 6cc50e1c5b57 ("[PATCH] fbcon: Console Rotation - Add support to rotate font bitmap")
> Cc: stable@vger.kernel.org
> Signed-off-by: Yao Yiqi <yaoyiqi3@huawei.com>
> ---
>   drivers/video/fbdev/core/fbcon.h        | 9 ++++++---
>   drivers/video/fbdev/core/fbcon_rotate.c | 9 ++++++++-
>   2 files changed, 14 insertions(+), 4 deletions(-)
I've replaced the v1 version of this series with this v3 version
in fbdev/for-next git tree for further testing.

Thanks!
Helge

Re: [PATCH v3 1/3] fbcon: Include font geometry in the rotated font cache

[yyaqia]

On 2026-10-18 16:05, Helge Deller wrote:
> I've replaced the v1 version of this series with this v3 version
> in fbdev/for-next git tree for further testing.

Sorry, the patch was not good enough, Zimmermann reviewed the code and I
did some modification after that. So we might need to switch to v4
version, sadly. Anyway, appreciate your continuous work for testing.

Thanks,
Yao Yiqi

[PATCH v3 2/3] fbcon: Rebuild rotated font buffer on font change

[Yao Yiqi]

From: Yao Yiqi <yaoyiqi3@huawei.com>

fbcon_rotate_font() caches a rotated copy of the console font in
par->rotated.buf. The buffer is sized from the font's width, height
and charcount, and is rebuilt in fbcon_init(), fbcon_switch() and
fbcon_modechanged(), but not in fbcon_do_set_font().

When the console font is changed through KDFONTOP/KD_FONT_OP_SET, the
new font data and geometry are installed and vc_resize() is called.
The redraw that follows reaches the rotated bitops while the buffer
still holds the old font: the rotated putcs/cursor ops index
par->rotated.buf with the new cell size and character codes, so a
stale, too-small buffer is read out of bounds.

KASAN reports, for example:

  BUG: KASAN: slab-out-of-bounds in soft_cursor
  Read of size 64 at addr ...
   memcpy
   soft_cursor
   cw_cursor
   fbcon_cursor
   hide_cursor
   redraw_screen
   vc_do_resize
   fbcon_do_set_font
   fbcon_set_font
   con_font_op
   vt_ioctl

Rebuild the buffer for the visible console right after the new font
geometry has been installed and before any redraw. Select the
console's blitting state with set_blitting_type() first, as
fbcon_init(), fbcon_switch() and fbcon_modechanged() do before their
rotate_font() calls: fbcon_rotate_font() reads the source font
through par->p, which may otherwise still refer to another console of
this framebuffer. On tile-blitting hardware, the same call uploads
the new font's map to the text engine, which previously happened only
when the console was switched. Background consoles keep their rebuild
at fbcon_switch() time, as the buffer is shared per framebuffer and
must not be rebuilt for a console that is not being displayed.

If vc_resize() fails, err_out restores the previous font. The rotated
buffer that was rebuilt for the new font earlier in the function
cannot be restored with it: font_data_rotate() has already reused or
freed the buffer that belonged to the old font. Select the blitting
type and rebuild the buffer again after the restore: the upload to
tile-blitting hardware and the rotated cache would otherwise keep
describing the rejected font, and par->rotated.fontdata would keep
referencing the font data whose last reference the error path drops.
A failing rebuild leaves par->rotated.buf NULL, so the rotated ops
skip drawing instead of reading a mismatched buffer; the next patch
retries the allocation on the following call.

Unlike the other callers, this one does not fall back to unrotated
rendering on failure. The "par->rotate = FB_ROTATE_UR;
set_blitting_type(vc, info);" fallback they use does not actually
switch to unrotated rendering: set_blitting_type() calls
fbcon_set_rotation(), which immediately re-derives par->rotate from
par->p->con_rotate. Replicating that pattern here would not change
the outcome; repairing the fallback is left to separate work.

Fixes: 6cc50e1c5b57 ("[PATCH] fbcon: Console Rotation - Add support to rotate font bitmap")
Cc: stable@vger.kernel.org
Signed-off-by: Yao Yiqi <yaoyiqi3@huawei.com>
---
Changes in v3:

- Also select the blitting type again in the error path of
  fbcon_do_set_font(). On tile-blitting hardware, set_blitting_type()
  uploads the font map to the text engine; without the second call, a
  failed vc_resize() left the engine with the rejected font (found by
  the Sashiko review).

Changes in v2:

- Rebuild the buffer again in the error path of fbcon_do_set_font().
  Without this, a failed vc_resize() left the rotated cache describing
  the rejected font while the console rendered with the restored one,
  and par->rotated.fontdata referenced freed font data (found by the
  Sashiko review).
- Select the console's blitting state with set_blitting_type() before
  rebuilding, like the other rotate_font() callers, instead of writing
  par->p directly.
- Reworded the note about the unrotated-rendering fallback, which is
  now left to separate work.
---
 drivers/video/fbdev/core/fbcon.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/drivers/video/fbdev/core/fbcon.c b/drivers/video/fbdev/core/fbcon.c
index 01715873ea49..d90a2ad45053 100644
--- a/drivers/video/fbdev/core/fbcon.c
+++ b/drivers/video/fbdev/core/fbcon.c
@@ -2432,6 +2432,17 @@ static int fbcon_do_set_font(struct vc_data *vc, int w, int h, int charcount,
 	else if (!vc->vc_hi_font_mask && charcount == 512)
 		set_vc_hi_font(vc, true);
 
+	/*
+	 * Select the console's blitting state and update par->rotated to
+	 * match the new font, so that the rendering below does not use a
+	 * cache that was built for the old font.
+	 */
+	if (con_is_visible(vc)) {
+		set_blitting_type(vc, info);
+		if (par->bitops->rotate_font)
+			par->bitops->rotate_font(info, vc);
+	}
+
 	if (resize) {
 		int cols, rows;
 
@@ -2466,6 +2477,18 @@ static int fbcon_do_set_font(struct vc_data *vc, int w, int h, int charcount,
 	else if (!old_hi_font_mask && vc->vc_hi_font_mask)
 		set_vc_hi_font(vc, false);
 
+	/*
+	 * Restore the rendering state for the old font: re-select the
+	 * blitting type, which re-uploads the font map on tile-blitting
+	 * hardware, and overwrite par->rotated with the restored font's
+	 * state.
+	 */
+	if (con_is_visible(vc)) {
+		set_blitting_type(vc, info);
+		if (par->bitops->rotate_font)
+			par->bitops->rotate_font(info, vc);
+	}
+
 	font_data_put(data);
 
 	return ret;
-- 
2.56.0

Re: [PATCH v3 2/3] fbcon: Rebuild rotated font buffer on font change

[Thomas Zimmermann]

Hi

Am 16.10.26 um 09:30 schrieb Yao Yiqi:
> From: Yao Yiqi <yaoyiqi3@huawei.com>
>
> fbcon_rotate_font() caches a rotated copy of the console font in
> par->rotated.buf. The buffer is sized from the font's width, height
> and charcount, and is rebuilt in fbcon_init(), fbcon_switch() and
> fbcon_modechanged(), but not in fbcon_do_set_font().
>
> When the console font is changed through KDFONTOP/KD_FONT_OP_SET, the
> new font data and geometry are installed and vc_resize() is called.
> The redraw that follows reaches the rotated bitops while the buffer
> still holds the old font: the rotated putcs/cursor ops index
> par->rotated.buf with the new cell size and character codes, so a
> stale, too-small buffer is read out of bounds.
>
> KASAN reports, for example:
>
>    BUG: KASAN: slab-out-of-bounds in soft_cursor
>    Read of size 64 at addr ...
>     memcpy
>     soft_cursor
>     cw_cursor
>     fbcon_cursor
>     hide_cursor
>     redraw_screen
>     vc_do_resize
>     fbcon_do_set_font
>     fbcon_set_font
>     con_font_op
>     vt_ioctl
>
> Rebuild the buffer for the visible console right after the new font
> geometry has been installed and before any redraw. Select the
> console's blitting state with set_blitting_type() first, as
> fbcon_init(), fbcon_switch() and fbcon_modechanged() do before their
> rotate_font() calls: fbcon_rotate_font() reads the source font
> through par->p, which may otherwise still refer to another console of
> this framebuffer. On tile-blitting hardware, the same call uploads
> the new font's map to the text engine, which previously happened only
> when the console was switched. Background consoles keep their rebuild
> at fbcon_switch() time, as the buffer is shared per framebuffer and
> must not be rebuilt for a console that is not being displayed.
>
> If vc_resize() fails, err_out restores the previous font. The rotated
> buffer that was rebuilt for the new font earlier in the function
> cannot be restored with it: font_data_rotate() has already reused or
> freed the buffer that belonged to the old font. Select the blitting
> type and rebuild the buffer again after the restore: the upload to
> tile-blitting hardware and the rotated cache would otherwise keep
> describing the rejected font, and par->rotated.fontdata would keep
> referencing the font data whose last reference the error path drops.
> A failing rebuild leaves par->rotated.buf NULL, so the rotated ops
> skip drawing instead of reading a mismatched buffer; the next patch
> retries the allocation on the following call.
>
> Unlike the other callers, this one does not fall back to unrotated
> rendering on failure. The "par->rotate = FB_ROTATE_UR;
> set_blitting_type(vc, info);" fallback they use does not actually
> switch to unrotated rendering: set_blitting_type() calls
> fbcon_set_rotation(), which immediately re-derives par->rotate from
> par->p->con_rotate. Replicating that pattern here would not change
> the outcome; repairing the fallback is left to separate work.

It's way too long and boring to read. Can you please try to shorten the 
AI-generated commit description?


>
> Fixes: 6cc50e1c5b57 ("[PATCH] fbcon: Console Rotation - Add support to rotate font bitmap")
> Cc: stable@vger.kernel.org
> Signed-off-by: Yao Yiqi <yaoyiqi3@huawei.com>
> ---
> Changes in v3:
>
> - Also select the blitting type again in the error path of
>    fbcon_do_set_font(). On tile-blitting hardware, set_blitting_type()
>    uploads the font map to the text engine; without the second call, a
>    failed vc_resize() left the engine with the rejected font (found by
>    the Sashiko review).

The problem is that rotate_font can fail as well. Handling that in the 
error path is not possible; hence results might be worse than keeping 
the rejected font.  Can you try to keep the original rotated font and 
reinstall that directly?

Best regards
Thomas


>
> Changes in v2:
>
> - Rebuild the buffer again in the error path of fbcon_do_set_font().
>    Without this, a failed vc_resize() left the rotated cache describing
>    the rejected font while the console rendered with the restored one,
>    and par->rotated.fontdata referenced freed font data (found by the
>    Sashiko review).
> - Select the console's blitting state with set_blitting_type() before
>    rebuilding, like the other rotate_font() callers, instead of writing
>    par->p directly.
> - Reworded the note about the unrotated-rendering fallback, which is
>    now left to separate work.
> ---
>   drivers/video/fbdev/core/fbcon.c | 23 +++++++++++++++++++++++
>   1 file changed, 23 insertions(+)
>
> diff --git a/drivers/video/fbdev/core/fbcon.c b/drivers/video/fbdev/core/fbcon.c
> index 01715873ea49..d90a2ad45053 100644
> --- a/drivers/video/fbdev/core/fbcon.c
> +++ b/drivers/video/fbdev/core/fbcon.c
> @@ -2432,6 +2432,17 @@ static int fbcon_do_set_font(struct vc_data *vc, int w, int h, int charcount,
>   	else if (!vc->vc_hi_font_mask && charcount == 512)
>   		set_vc_hi_font(vc, true);
>   
> +	/*
> +	 * Select the console's blitting state and update par->rotated to
> +	 * match the new font, so that the rendering below does not use a
> +	 * cache that was built for the old font.
> +	 */
> +	if (con_is_visible(vc)) {
> +		set_blitting_type(vc, info);
> +		if (par->bitops->rotate_font)
> +			par->bitops->rotate_font(info, vc);
> +	}
> +
>   	if (resize) {
>   		int cols, rows;
>   
> @@ -2466,6 +2477,18 @@ static int fbcon_do_set_font(struct vc_data *vc, int w, int h, int charcount,
>   	else if (!old_hi_font_mask && vc->vc_hi_font_mask)
>   		set_vc_hi_font(vc, false);
>   
> +	/*
> +	 * Restore the rendering state for the old font: re-select the
> +	 * blitting type, which re-uploads the font map on tile-blitting
> +	 * hardware, and overwrite par->rotated with the restored font's
> +	 * state.
> +	 */
> +	if (con_is_visible(vc)) {
> +		set_blitting_type(vc, info);
> +		if (par->bitops->rotate_font)
> +			par->bitops->rotate_font(info, vc);
> +	}
> +
>   	font_data_put(data);
>   
>   	return ret;

-- 
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstr. 146, 90461 Nürnberg, Germany, www.suse.com
GF: Stefan Gaiser, Jochen Jaser, Abhinav Puri, (HRB 36809, AG Nürnberg)

Re: [PATCH v3 2/3] fbcon: Rebuild rotated font buffer on font change

[yyaqia]

On 2026-10-16 09:59, Thomas Zimmermann wrote:
> 
> It's way too long and boring to read. Can you please try to shorten the
> AI-generated commit description?
> 
> 

Sorry for that, I was a bit lazy to think commit desc by myself. I am
new to kernel and a little afraid that my message does not cover all
things.

> > - Also select the blitting type again in the error path of
> >    fbcon_do_set_font(). On tile-blitting hardware, set_blitting_type()
> >    uploads the font map to the text engine; without the second call, a
> >    failed vc_resize() left the engine with the rejected font (found by
> >    the Sashiko review).
> 
> The problem is that rotate_font can fail as well. Handling that in the error
> path is not possible; hence results might be worse than keeping the rejected
> font.  Can you try to keep the original rotated font and reinstall that
> directly?
> 

Good advice, I will fix this in V4.

Thanks for your review,
Yao Yiqi

[PATCH v3 3/3] fbcon: Retry rotated font allocation after failure

[Yao Yiqi]

From: Yao Yiqi <yaoyiqi3@huawei.com>

fbcon_rotate_font() records the font, its geometry and the rotation it
is about to handle in par->rotated before calling font_data_rotate().
If the allocation fails, the error path clears par->rotated.buf, but
the recorded font, geometry and rotation already look up to date, so
the next call takes the early-out and never retries. The rotated
putcs/cursor ops keep returning early on the NULL buffer and the
console stays blank until a font or rotation with a different key is
installed, even once memory is available again.

Also test par->rotated.buf in the early-out, so a failed allocation is
retried the next time fbcon_rotate_font() runs: at the next
fbcon_switch(), fbcon_modechanged() or font change.

Fixes: e4ef723d8975 ("fbcon: Avoid OOB font access if console rotation fails")
Cc: stable@vger.kernel.org
Signed-off-by: Yao Yiqi <yaoyiqi3@huawei.com>
---
Changes in v2:

- The early-out now also compares the font geometry (previous patch in
  this series); the retry covers the combined key.
---
 drivers/video/fbdev/core/fbcon_rotate.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/video/fbdev/core/fbcon_rotate.c b/drivers/video/fbdev/core/fbcon_rotate.c
index 660a3fa81bae..7afd63dcee7b 100644
--- a/drivers/video/fbdev/core/fbcon_rotate.c
+++ b/drivers/video/fbdev/core/fbcon_rotate.c
@@ -25,7 +25,8 @@ int fbcon_rotate_font(struct fb_info *info, struct vc_data *vc)
 	    par->rotate == par->rotated.buf_rotate &&
 	    vc->vc_font.width == par->rotated.buf_width &&
 	    vc->vc_font.height == par->rotated.buf_height &&
-	    vc->vc_font.charcount == par->rotated.buf_charcount)
+	    vc->vc_font.charcount == par->rotated.buf_charcount &&
+	    par->rotated.buf)
 		return 0;
 
 	par->rotated.fontdata = par->p->fontdata;
-- 
2.56.0