Re: Re: [PATCH] vesafb memory size mismatch

["Antonino A. Daplas" <adaplas@hotpop.com>]

On Sunday 03 October 2004 09:59, Andrew Morton wrote:
> (Gerd re-added to Cc)
>
> Aurelien Jacobs <aurel@gnuage.org> wrote:
> > On 01 Oct 2004 17:48:21 +0200
> >
> > Gerd Knorr <kraxel@bytesex.org> wrote:
> > > Aurelien Jacobs <aurel@gnuage.org> writes:
> > > > Hi,
> > > >
> > > > I found a bug in the vesafb driver. The video memory size which is
> > > > reserved by vesafb do not correspond to the really needed memory.
> > >
> > > Oh, while we are at it, I've hacked a patch as well some time ago.
> > > All the size calculation in vesafb is a bit tricky.  I've tried to
> > > cleanup that a bit and fix some bugs along the way, but never managed
> > > to submit it.  I think the boundary check cleanups should also fix the
> > > bug you've seen.
> >
> > Unfortunately not :-(
> > Your patch still keep the same way to calculate minimum memory
> > required for the video mode (based on xres*bytes_per_pixel). It's
> > wrong ! With video cards such as 3dfx in 24 bits mode, line_length
> > is bigger than xres*bytes_per_pixel (probably for alignment pupose).
> > So not enough memory is allocated, leading to a crash.
> >
> > Here is an updated version of your patch with the right way to
> > calculate size_vmode.
>
> OK, I'll drop the patch which I added to -mm.  Could someone please resend
> the final version when it's all sorted out?
>

Yes, Aurelien is correct, calculating the memory needed for a mode is more
accurate if line_length * yres is used.

I'll submit a patch combining Gerd's, Aurelien's and a few of my changes
later today.

Tony




-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl