[PATCH] vesafb memory size mismatch

[Aurelien Jacobs <aurel@gnuage.org>]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

Hi,

I found a bug in the vesafb driver. The video memory size which is
reserved by vesafb do not correspond to the really needed memory.
Currently the needed memory is calculated using
screen_info.lfb_width * vesafb_defined.bits_per_pixel >> 3
This works with most cards because it's usually equal to
vesafb_fix.line_length.
But with some cards (voodoo3 in 24 bits mode) line_length is
greater than width * bytes_per_pixel (probably for alignment
purpose, line_length = 4096 on voodoo3 800x600-24).
Su currently, with such cards, writting in the second buffer
(as memory is allocated for double buffering) can go past the
end of reserved memory, overwritting some other memory area
and leading to a kernel crash.

So I wrote this tiny patch wich correct this bug.
I tested it successfully with different video cards (Matrox,
Nvidia, Sis, 3dfx) in different modes.

I don't know if it's the right place to send a patch for vesafb ?
Could you please direct me to the right place if it's not ?

Aurel

[-- Attachment #2: vesafb-memory.diff --]
[-- Type: text/plain, Size: 696 bytes --]

diff -Naur linux-2.6.8.1.orig/drivers/video/vesafb.c linux-2.6.8.1/drivers/video/vesafb.c
--- linux-2.6.8.1.orig/drivers/video/vesafb.c	2004-09-29 17:47:41.997524992 +0200
+++ linux-2.6.8.1/drivers/video/vesafb.c	2004-09-29 17:48:03.946188288 +0200
@@ -234,7 +234,7 @@
 	vesafb_fix.line_length = screen_info.lfb_linelength;
 
 	/* Allocate enough memory for double buffering */
-	vesafb_fix.smem_len = screen_info.lfb_width * screen_info.lfb_height * vesafb_defined.bits_per_pixel >> 2;
+	vesafb_fix.smem_len = 2 * screen_info.lfb_linelength * screen_info.lfb_height;
 
 	/* check that we don't remap more memory than old cards have */
 	if (vesafb_fix.smem_len > (screen_info.lfb_size * 65536))