* [PATCH RESEND] vmwgfx: fix incorrect vram size check in vmw_kms_fb_create()
@ 2011-12-20 21:08 Xi Wang
2011-12-21 9:30 ` David Airlie
0 siblings, 1 reply; 6+ messages in thread
From: Xi Wang @ 2011-12-20 21:08 UTC (permalink / raw)
To: Jakob Bornecrantz, Thomas Hellstrom; +Cc: Dave Airlie, dri-devel
The previous commit didn't correctly fix the integer overflow issue.
http://git.kernel.org/linus/e133e737
- unsigned int required_size;
+ u64 required_size;
...
required_size = mode_cmd->pitch * mode_cmd->height;
- if (unlikely(required_size > dev_priv->vram_size)) {
+ if (unlikely(required_size > (u64) dev_priv->vram_size)) {
Note that both pitch and height are u32, their product is still u32 and
would overflow before being assigned to required_size. A correct way is
to convert pitch and height to u64 before the multiplication.
required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;
This patch calls an existing function vmw_kms_validate_mode_vram()
for validation.
Signed-off-by: Xi Wang <xi.wang@gmail.com>
---
vmwgfx_kms.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/vmwgfx_kms.c b/vmwgfx_kms.c
index b87afdf..6b8857e 100644
--- a/vmwgfx_kms.c
+++ b/vmwgfx_kms.c
@@ -1101,7 +1101,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
struct vmw_surface *surface = NULL;
struct vmw_dma_buffer *bo = NULL;
struct ttm_base_object *user_obj;
- u64 required_size;
int ret;
/**
@@ -1110,8 +1109,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
* requested framebuffer.
*/
- required_size = mode_cmd->pitch * mode_cmd->height;
- if (unlikely(required_size > (u64) dev_priv->vram_size)) {
+ if (!vmw_kms_validate_mode_vram(dev_priv,
+ mode_cmd->pitch,
+ mode_cmd->height)) {
DRM_ERROR("VRAM size is too small for requested mode.\n");
return ERR_PTR(-ENOMEM);
}
--
1.7.5.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH RESEND] vmwgfx: fix incorrect vram size check in vmw_kms_fb_create()
2011-12-20 21:08 [PATCH RESEND] vmwgfx: fix incorrect vram size check in vmw_kms_fb_create() Xi Wang
@ 2011-12-21 9:30 ` David Airlie
2011-12-21 9:33 ` Xi Wang
2011-12-21 10:18 ` [PATCH -fixes] vmwgfx: fix incorrect VRAM " Xi Wang
0 siblings, 2 replies; 6+ messages in thread
From: David Airlie @ 2011-12-21 9:30 UTC (permalink / raw)
To: Xi Wang; +Cc: Thomas Hellstrom, dri-devel
----- Original Message -----
> From: "Xi Wang" <xi.wang@gmail.com>
> To: "Jakob Bornecrantz" <jakob@vmware.com>, "Thomas Hellstrom" <thellstrom@vmware.com>
> Cc: dri-devel@lists.freedesktop.org, "Dave Airlie" <airlied@redhat.com>
> Sent: Tuesday, 20 December, 2011 9:08:32 PM
> Subject: [PATCH RESEND] vmwgfx: fix incorrect vram size check in vmw_kms_fb_create()
This patch doesn't apply with git am, please regenerate it,
Thomas, can you ack as well please?
Is this for -fixes or -next?
Dave.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH RESEND] vmwgfx: fix incorrect vram size check in vmw_kms_fb_create()
2011-12-21 9:30 ` David Airlie
@ 2011-12-21 9:33 ` Xi Wang
2011-12-21 10:18 ` [PATCH -fixes] vmwgfx: fix incorrect VRAM " Xi Wang
1 sibling, 0 replies; 6+ messages in thread
From: Xi Wang @ 2011-12-21 9:33 UTC (permalink / raw)
To: David Airlie; +Cc: Thomas Hellstrom, dri-devel
On Dec 21, 2011, at 4:30 AM, David Airlie wrote:
>
> This patch doesn't apply with git am, please regenerate it,
>
> Thomas, can you ack as well please?
>
> Is this for -fixes or -next?
Sorry this patch is for vmwgfx.git. I will redo a version that can be applied to the mainline kernel.
- xi
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH -fixes] vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()
2011-12-21 9:30 ` David Airlie
2011-12-21 9:33 ` Xi Wang
@ 2011-12-21 10:18 ` Xi Wang
2011-12-21 12:22 ` Thomas Hellstrom
2011-12-21 21:33 ` Thomas Hellstrom
1 sibling, 2 replies; 6+ messages in thread
From: Xi Wang @ 2011-12-21 10:18 UTC (permalink / raw)
To: David Airlie; +Cc: Thomas Hellstrom, dri-devel
Commit e133e737 didn't correctly fix the integer overflow issue.
- unsigned int required_size;
+ u64 required_size;
...
required_size = mode_cmd->pitch * mode_cmd->height;
- if (unlikely(required_size > dev_priv->vram_size)) {
+ if (unlikely(required_size > (u64) dev_priv->vram_size)) {
Note that both pitch and height are u32. Their product is still u32 and
would overflow before being assigned to required_size. A correct way is
to convert pitch and height to u64 before the multiplication.
required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;
This patch calls the existing vmw_kms_validate_mode_vram() for
validation.
Signed-off-by: Xi Wang <xi.wang@gmail.com>
---
drivers/gpu/drm/vmwgfx/vmwgfx_kms.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
index 8aa1dbb..f94b33a 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
@@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
struct vmw_surface *surface = NULL;
struct vmw_dma_buffer *bo = NULL;
struct ttm_base_object *user_obj;
- u64 required_size;
int ret;
/**
@@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
* requested framebuffer.
*/
- required_size = mode_cmd->pitch * mode_cmd->height;
- if (unlikely(required_size > (u64) dev_priv->vram_size)) {
+ if (!vmw_kms_validate_mode_vram(dev_priv,
+ mode_cmd->pitch,
+ mode_cmd->height)) {
DRM_ERROR("VRAM size is too small for requested mode.\n");
return ERR_PTR(-ENOMEM);
}
--
1.7.5.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH -fixes] vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()
2011-12-21 10:18 ` [PATCH -fixes] vmwgfx: fix incorrect VRAM " Xi Wang
@ 2011-12-21 12:22 ` Thomas Hellstrom
2011-12-21 21:33 ` Thomas Hellstrom
1 sibling, 0 replies; 6+ messages in thread
From: Thomas Hellstrom @ 2011-12-21 12:22 UTC (permalink / raw)
To: Xi Wang; +Cc: David Airlie, dri-devel
This looks good, although I want to do a quick test to verify that it
doesn't break anything.
I'll get back as soon as possible.
/Thomas
On 12/21/2011 11:18 AM, Xi Wang wrote:
> Commit e133e737 didn't correctly fix the integer overflow issue.
>
> - unsigned int required_size;
> + u64 required_size;
> ...
> required_size = mode_cmd->pitch * mode_cmd->height;
> - if (unlikely(required_size> dev_priv->vram_size)) {
> + if (unlikely(required_size> (u64) dev_priv->vram_size)) {
>
> Note that both pitch and height are u32. Their product is still u32 and
> would overflow before being assigned to required_size. A correct way is
> to convert pitch and height to u64 before the multiplication.
>
> required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;
>
> This patch calls the existing vmw_kms_validate_mode_vram() for
> validation.
>
> Signed-off-by: Xi Wang<xi.wang@gmail.com>
> ---
> drivers/gpu/drm/vmwgfx/vmwgfx_kms.c | 6 +++---
> 1 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
> index 8aa1dbb..f94b33a 100644
> --- a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
> +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
> @@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
> struct vmw_surface *surface = NULL;
> struct vmw_dma_buffer *bo = NULL;
> struct ttm_base_object *user_obj;
> - u64 required_size;
> int ret;
>
> /**
> @@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
> * requested framebuffer.
> */
>
> - required_size = mode_cmd->pitch * mode_cmd->height;
> - if (unlikely(required_size> (u64) dev_priv->vram_size)) {
> + if (!vmw_kms_validate_mode_vram(dev_priv,
> + mode_cmd->pitch,
> + mode_cmd->height)) {
> DRM_ERROR("VRAM size is too small for requested mode.\n");
> return ERR_PTR(-ENOMEM);
> }
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH -fixes] vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()
2011-12-21 10:18 ` [PATCH -fixes] vmwgfx: fix incorrect VRAM " Xi Wang
2011-12-21 12:22 ` Thomas Hellstrom
@ 2011-12-21 21:33 ` Thomas Hellstrom
1 sibling, 0 replies; 6+ messages in thread
From: Thomas Hellstrom @ 2011-12-21 21:33 UTC (permalink / raw)
To: David Airlie; +Cc: dri-devel, Xi Wang
Reviewed- and tested by
Thomas Hellstrom <thellstrom@vmware.com>
Thanks,
Thomas
On 12/21/2011 11:18 AM, Xi Wang wrote:
> Commit e133e737 didn't correctly fix the integer overflow issue.
>
> - unsigned int required_size;
> + u64 required_size;
> ...
> required_size = mode_cmd->pitch * mode_cmd->height;
> - if (unlikely(required_size> dev_priv->vram_size)) {
> + if (unlikely(required_size> (u64) dev_priv->vram_size)) {
>
> Note that both pitch and height are u32. Their product is still u32 and
> would overflow before being assigned to required_size. A correct way is
> to convert pitch and height to u64 before the multiplication.
>
> required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;
>
> This patch calls the existing vmw_kms_validate_mode_vram() for
> validation.
>
> Signed-off-by: Xi Wang<xi.wang@gmail.com>
> ---
> drivers/gpu/drm/vmwgfx/vmwgfx_kms.c | 6 +++---
> 1 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
> index 8aa1dbb..f94b33a 100644
> --- a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
> +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
> @@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
> struct vmw_surface *surface = NULL;
> struct vmw_dma_buffer *bo = NULL;
> struct ttm_base_object *user_obj;
> - u64 required_size;
> int ret;
>
> /**
> @@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
> * requested framebuffer.
> */
>
> - required_size = mode_cmd->pitch * mode_cmd->height;
> - if (unlikely(required_size> (u64) dev_priv->vram_size)) {
> + if (!vmw_kms_validate_mode_vram(dev_priv,
> + mode_cmd->pitch,
> + mode_cmd->height)) {
> DRM_ERROR("VRAM size is too small for requested mode.\n");
> return ERR_PTR(-ENOMEM);
> }
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-12-21 21:36 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-20 21:08 [PATCH RESEND] vmwgfx: fix incorrect vram size check in vmw_kms_fb_create() Xi Wang
2011-12-21 9:30 ` David Airlie
2011-12-21 9:33 ` Xi Wang
2011-12-21 10:18 ` [PATCH -fixes] vmwgfx: fix incorrect VRAM " Xi Wang
2011-12-21 12:22 ` Thomas Hellstrom
2011-12-21 21:33 ` Thomas Hellstrom
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.