From mboxrd@z Thu Jan 1 00:00:00 1970 From: Xi Wang Subject: [PATCH RESEND] vmwgfx: fix incorrect vram size check in vmw_kms_fb_create() Date: Tue, 20 Dec 2011 16:08:32 -0500 Message-ID: <4EF0F950.7000707@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail-gx0-f177.google.com (mail-gx0-f177.google.com [209.85.161.177]) by gabe.freedesktop.org (Postfix) with ESMTP id AD6739E7AD for ; Tue, 20 Dec 2011 13:08:38 -0800 (PST) Received: by ggnp4 with SMTP id p4so6440677ggn.36 for ; Tue, 20 Dec 2011 13:08:38 -0800 (PST) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dri-devel-bounces+sf-dri-devel=m.gmane.org@lists.freedesktop.org Errors-To: dri-devel-bounces+sf-dri-devel=m.gmane.org@lists.freedesktop.org To: Jakob Bornecrantz , Thomas Hellstrom Cc: Dave Airlie , dri-devel@lists.freedesktop.org List-Id: dri-devel@lists.freedesktop.org The previous commit didn't correctly fix the integer overflow issue. http://git.kernel.org/linus/e133e737 - unsigned int required_size; + u64 required_size; ... required_size = mode_cmd->pitch * mode_cmd->height; - if (unlikely(required_size > dev_priv->vram_size)) { + if (unlikely(required_size > (u64) dev_priv->vram_size)) { Note that both pitch and height are u32, their product is still u32 and would overflow before being assigned to required_size. A correct way is to convert pitch and height to u64 before the multiplication. required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height; This patch calls an existing function vmw_kms_validate_mode_vram() for validation. Signed-off-by: Xi Wang --- vmwgfx_kms.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/vmwgfx_kms.c b/vmwgfx_kms.c index b87afdf..6b8857e 100644 --- a/vmwgfx_kms.c +++ b/vmwgfx_kms.c @@ -1101,7 +1101,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev, struct vmw_surface *surface = NULL; struct vmw_dma_buffer *bo = NULL; struct ttm_base_object *user_obj; - u64 required_size; int ret; /** @@ -1110,8 +1109,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev, * requested framebuffer. */ - required_size = mode_cmd->pitch * mode_cmd->height; - if (unlikely(required_size > (u64) dev_priv->vram_size)) { + if (!vmw_kms_validate_mode_vram(dev_priv, + mode_cmd->pitch, + mode_cmd->height)) { DRM_ERROR("VRAM size is too small for requested mode.\n"); return ERR_PTR(-ENOMEM); } -- 1.7.5.4