Re: [PATCH] drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs()

[Thomas Zimmermann <tzimmermann@suse.de>]

Am 09.04.26 um 18:41 schrieb Ashutosh Desai:
> drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions
> using plain integer division:
>
>    unsigned int width  = mode_cmd->width  / (i ? info->hsub : 1);
>    unsigned int height = mode_cmd->height / (i ? info->vsub : 1);
>
> However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses
> drm_format_info_plane_width/height() which round up dimensions via
> DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object
> size check for certain pixel format and dimension combinations.
>
> For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the
> GEM size validation path sees height=0 instead of height=1. The
> expression (height - 1) then wraps to UINT_MAX as an unsigned int,
> causing min_size to overflow and wrap back to a small value. A tiny
> GEM object therefore passes the size guard, yet when the GPU accesses
> the chroma plane it will read or write memory beyond the object's
> bounds.
>
> Fix by replacing the open-coded divisions with drm_format_info_plane_width()
> and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match
> the calculation already used in framebuffer_check().
>
> Signed-off-by: Ashutosh Desai <ashutoshdesai993@gmail.com>

Thanks for the fix.

Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de>

Fixes: 4c3dbb2c312c ("drm: Add GEM backed framebuffer library")
Cc: <stable@vger.kernel.org> # v4.14+



> ---
>   drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/drm_gem_framebuffer_helper.c b/drivers/gpu/drm/drm_gem_framebuffer_helper.c
> index 9166c353f..88808e972 100644
> --- a/drivers/gpu/drm/drm_gem_framebuffer_helper.c
> +++ b/drivers/gpu/drm/drm_gem_framebuffer_helper.c
> @@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev,
>   	}
>   
>   	for (i = 0; i < info->num_planes; i++) {
> -		unsigned int width = mode_cmd->width / (i ? info->hsub : 1);
> -		unsigned int height = mode_cmd->height / (i ? info->vsub : 1);
> +		unsigned int width = drm_format_info_plane_width(info, mode_cmd->width, i);
> +		unsigned int height = drm_format_info_plane_height(info, mode_cmd->height, i);
>   		unsigned int min_size;
>   
>   		objs[i] = drm_gem_object_lookup(file, mode_cmd->handles[i]);

-- 
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstr. 146, 90461 Nürnberg, Germany, www.suse.com
GF: Jochen Jaser, Andrew McDonald, Werner Knoblich, (HRB 36809, AG Nürnberg)