From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Hellstrom Subject: Re: [PATCH -fixes] vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create() Date: Wed, 21 Dec 2011 13:22:57 +0100 Message-ID: <4EF1CFA1.9040005@vmware.com> References: <4EF1B279.2090907@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: Received: from smtp-outbound-1.vmware.com (smtp-outbound-1.vmware.com [65.115.85.69]) by gabe.freedesktop.org (Postfix) with ESMTP id 041159E747 for ; Wed, 21 Dec 2011 04:26:09 -0800 (PST) In-Reply-To: <4EF1B279.2090907@gmail.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dri-devel-bounces+sf-dri-devel=m.gmane.org@lists.freedesktop.org Errors-To: dri-devel-bounces+sf-dri-devel=m.gmane.org@lists.freedesktop.org To: Xi Wang Cc: David Airlie , dri-devel@lists.freedesktop.org List-Id: dri-devel@lists.freedesktop.org This looks good, although I want to do a quick test to verify that it doesn't break anything. I'll get back as soon as possible. /Thomas On 12/21/2011 11:18 AM, Xi Wang wrote: > Commit e133e737 didn't correctly fix the integer overflow issue. > > - unsigned int required_size; > + u64 required_size; > ... > required_size = mode_cmd->pitch * mode_cmd->height; > - if (unlikely(required_size> dev_priv->vram_size)) { > + if (unlikely(required_size> (u64) dev_priv->vram_size)) { > > Note that both pitch and height are u32. Their product is still u32 and > would overflow before being assigned to required_size. A correct way is > to convert pitch and height to u64 before the multiplication. > > required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height; > > This patch calls the existing vmw_kms_validate_mode_vram() for > validation. > > Signed-off-by: Xi Wang > --- > drivers/gpu/drm/vmwgfx/vmwgfx_kms.c | 6 +++--- > 1 files changed, 3 insertions(+), 3 deletions(-) > > diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c > index 8aa1dbb..f94b33a 100644 > --- a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c > +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c > @@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev, > struct vmw_surface *surface = NULL; > struct vmw_dma_buffer *bo = NULL; > struct ttm_base_object *user_obj; > - u64 required_size; > int ret; > > /** > @@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev, > * requested framebuffer. > */ > > - required_size = mode_cmd->pitch * mode_cmd->height; > - if (unlikely(required_size> (u64) dev_priv->vram_size)) { > + if (!vmw_kms_validate_mode_vram(dev_priv, > + mode_cmd->pitch, > + mode_cmd->height)) { > DRM_ERROR("VRAM size is too small for requested mode.\n"); > return ERR_PTR(-ENOMEM); > } >