[PATCH] vga: relax restriction on display width

[PATCH] vga: relax restriction on display width

[Simon Rowe]

When validating the parameters of VBE ioport writes the X co-ordinate
is silently rounded down to a multiple of 8. For valid resolutions
(such as 1366x768) which are not divisible by 8 this causes
miscalculations because the display surface has shorter lines than
expected. For example, a VNC client receives updates with a
"staircase" effect.

Reduce the rounding to a multiple of two.

Signed-off-by: Simon Rowe <simon.rowe@nutanix.com>
---
 hw/display/vga.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/hw/display/vga.c b/hw/display/vga.c
index 892fedc8dc..ea659e2812 100644
--- a/hw/display/vga.c
+++ b/hw/display/vga.c
@@ -581,14 +581,14 @@ static void vbe_fixup_regs(VGACommonState *s)
     }
 
     /* check width */
-    r[VBE_DISPI_INDEX_XRES] &= ~7u;
+    r[VBE_DISPI_INDEX_XRES] &= ~1u;
     if (r[VBE_DISPI_INDEX_XRES] == 0) {
         r[VBE_DISPI_INDEX_XRES] = 8;
     }
     if (r[VBE_DISPI_INDEX_XRES] > VBE_DISPI_MAX_XRES) {
         r[VBE_DISPI_INDEX_XRES] = VBE_DISPI_MAX_XRES;
     }
-    r[VBE_DISPI_INDEX_VIRT_WIDTH] &= ~7u;
+    r[VBE_DISPI_INDEX_VIRT_WIDTH] &= ~1u;
     if (r[VBE_DISPI_INDEX_VIRT_WIDTH] > VBE_DISPI_MAX_XRES) {
         r[VBE_DISPI_INDEX_VIRT_WIDTH] = VBE_DISPI_MAX_XRES;
     }
-- 
2.22.3

Re: [PATCH] vga: relax restriction on display width

[Simon Rowe]

[-- Attachment #1: Type: text/plain, Size: 1372 bytes --]

Ping?

On 04/10/2024, 09:39, "Simon Rowe" <simon.rowe@nutanix.com> wrote:

When validating the parameters of VBE ioport writes the X co-ordinate
is silently rounded down to a multiple of 8. For valid resolutions
(such as 1366x768) which are not divisible by 8 this causes
miscalculations because the display surface has shorter lines than
expected. For example, a VNC client receives updates with a
"staircase" effect.

Reduce the rounding to a multiple of two.

Signed-off-by: Simon Rowe <simon.rowe@nutanix.com>
---
 hw/display/vga.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/hw/display/vga.c b/hw/display/vga.c
index 892fedc8dc..ea659e2812 100644
--- a/hw/display/vga.c
+++ b/hw/display/vga.c
@@ -581,14 +581,14 @@ static void vbe_fixup_regs(VGACommonState *s)
     }

     /* check width */
-    r[VBE_DISPI_INDEX_XRES] &= ~7u;
+    r[VBE_DISPI_INDEX_XRES] &= ~1u;
     if (r[VBE_DISPI_INDEX_XRES] == 0) {
         r[VBE_DISPI_INDEX_XRES] = 8;
     }
     if (r[VBE_DISPI_INDEX_XRES] > VBE_DISPI_MAX_XRES) {
         r[VBE_DISPI_INDEX_XRES] = VBE_DISPI_MAX_XRES;
     }
-    r[VBE_DISPI_INDEX_VIRT_WIDTH] &= ~7u;
+    r[VBE_DISPI_INDEX_VIRT_WIDTH] &= ~1u;
     if (r[VBE_DISPI_INDEX_VIRT_WIDTH] > VBE_DISPI_MAX_XRES) {
         r[VBE_DISPI_INDEX_VIRT_WIDTH] = VBE_DISPI_MAX_XRES;
     }
--
2.22.3

[-- Attachment #2: Type: text/html, Size: 3613 bytes --]