From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64A801925BC for ; Mon, 20 Apr 2026 01:37:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776649039; cv=none; b=oIIpHgH4NJmmw8gkWqZwY7grodCB5+h7QD8CBL9jWSwhWeaUoKkaWqccZnxijZTVNnK//BKqxjmVP9ifi9eaAGK1cgntoFymqxujHVXrDLQSy0dYgK+tGeHqTy9+EiJYNcfn0jJJRe/PyFZCG68fxyXEtfqurX2m4ciWEoJ0PYo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776649039; c=relaxed/simple; bh=bEu+schdl3H87Rb4LAH9J3MCpCQdId0KAn+oSl3OdAQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UaQd41QYYD3l7gddCmfGBjpGId2leAxo8VBQ94yqmmwk1tF4tWO9aOfDZg2Rv4duPCiTVtLHnFBI7Alv2fM4le3TKmY334rdLfBNcD4UFAm6dptSG68LS17LVsnol+O6/F1rLDgBKldtDMUQ/ARpkWa9YDkgY54EOGq6io9qOWg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=m3+LFlKC; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m3+LFlKC" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-79a46ebe2beso23094137b3.2 for ; Sun, 19 Apr 2026 18:37:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1776649037; x=1777253837; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PKU11ok0UdmISqg+TyGzbe+U8sWE0AZJ6IK5DjOXMLY=; b=m3+LFlKC3LSVSnRvUeBWEj8CjTaHyPrTgQNubUOs014Wq/mhMl4D4y7SVbOeZkFVNx IcweaJN4hMt4Son1OsZfsrmF6V+s54wwPfyuOTzelAK9AID9h6KWzW2Vhj5xPBZDiQ/w iKj8nuJM1SGdGDA+dqyJqrEtI83FTMbQ2rg36j6amYZxnncESx3NlbttdZZ8YVqpl/4Z 70nfQC85RVGG9HlZ1tAOuBnHMT38htIu7+KgXn0/bNxeFEbVleoefLWlVZ++I+HVUeu+ Qy6WdW1mC1p52I4ZAREIbkUOnE7OoQrbA0/lkpt+1DaTZIJgiIGl5ejlaRbGviAM1cE/ xX/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776649037; x=1777253837; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=PKU11ok0UdmISqg+TyGzbe+U8sWE0AZJ6IK5DjOXMLY=; b=NU/Wzgc8T92XKYWt8I9yN+h6OTwLJKoMnhLe6qGE56nHJIG6QMRTLizCiUXcew3LoJ btGFrd13v3FZXezO6ubD/ewe4k9HnlazUbAc8brJTdB2V6nl/Mxq/WvxGOGzQUGqWGPj ePtcuM45mp5PynHnCXXMfAEIaDfKlxy3Z892Sfc5ubNnpB2wsVK1Ey+oaTZD4Hv13LMJ hBjzzhsloMZB2LKhISGWdlXWfZhP5qB2svWZ8bIPOBK2G5ZB8WFinB/7eHuBiFIi7Jv3 M1JpeaHPbcXlyHp5tg3icRwj5JTgP0JDr/rdeKvkUINnSoa7yPe/S3xEzgVpcC465FzC cLvQ== X-Forwarded-Encrypted: i=1; AFNElJ+6ifL23nbr26FFViSea3iZVfKdlC0N+mVI78M1D+Pe6mZx68dbQ9ywXq1ksFkxKvVuRgdudOWuxYViwok=@vger.kernel.org X-Gm-Message-State: AOJu0YwNfEbCZeQM0odVRpDimzqNKruw/8KBJ/WaHM3KV1ceL3KLaFjJ EFRQy1l0absZ60s9YM3Ww9W6053z6W8dOQX2ltcKpF1pG64kCjvurtWX X-Gm-Gg: AeBDieuC301ZuJV6Lq9oQVl8nt1GpOlHMpbF0td+zV3R17BDf4fAHSFLHGsMKPPmssq 0DiQLL6BNSgyqTBS60VSrBscMmt4RSpFPEgdX+sw8uWPVkYGxvbTJQqcVuTmLgl3rU5AGHDIzwn QxFH9L9jQFL5a1ekBW1OMkEdD03gUFTh2KymN0kr+tqEBdN3pll/KwYG0S6qs0eY4vrKJ9vE3tp 28/jxkTkro8n+yRpQBI5rCwSw1ooi6gmcThkhfLl2ybh5ahx1DGvnTqN6sWR5hmCjgZGPv0jiE1 ImQ7y8Zl3fOKDkAIYBZS5GQ9RkDsrgiBiAp5POy7m+2w347vCRt68UenC3x2OWYxb0QjI5+e9CP nOK6A72fcVEgLJlJ9PSH7fFVscmJzY6RthkkQP8Q3lKTEhWZf59ZuUiLFp41hl+dpA/BsZaQYwj feuTMjFb4+qdpEnmxamwnIfuhwYHArdh1/OHgGcETGKaLTHG5qbj+UaWYFaC4qsXQr+AZhWsrzm bfZSnB7CpnQP/Hd2U2kZQLHKe4EuTjVIZgbdiM= X-Received: by 2002:a05:690c:9:b0:7a2:7b00:67e6 with SMTP id 00721157ae682-7b9ecee8228mr133713637b3.21.1776649037367; Sun, 19 Apr 2026 18:37:17 -0700 (PDT) Received: from TDC4045031631.e0cglfehwr0e5gttmepj3hi3hf.ux.internal.cloudapp.net ([20.63.37.123]) by smtp.gmail.com with ESMTPSA id 00721157ae682-7b9ee9b1c4dsm37350767b3.33.2026.04.19.18.37.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Apr 2026 18:37:16 -0700 (PDT) From: Ashutosh Desai To: maarten.lankhorst@linux.intel.com, mripard@kernel.org, tzimmermann@suse.de, airlied@gmail.com, simona@ffwll.ch Cc: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Ashutosh Desai Subject: [PATCH v2] drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs() Date: Mon, 20 Apr 2026 01:36:37 +0000 Message-Id: <20260420013637.457751-1-ashutoshdesai993@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <2f9cd84b-0642-418b-a4ed-7863716a8531@suse.de> References: <2f9cd84b-0642-418b-a4ed-7863716a8531@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division: unsigned int width = mode_cmd->width / (i ? info->hsub : 1); unsigned int height = mode_cmd->height / (i ? info->vsub : 1); However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations. For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds. Fix by replacing the open-coded divisions with drm_format_info_plane_width() and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check(). Fixes: 4c3dbb2c312c ("drm: Add GEM backed framebuffer library") Cc: stable@vger.kernel.org # v4.14+ Reviewed-by: Thomas Zimmermann Signed-off-by: Ashutosh Desai --- V1 -> V2: add Fixes: tag, Cc: stable@vger.kernel.org, incorporate Reviewed-by from Thomas Zimmermann. Link: https://lore.kernel.org/dri-devel/20260409164156.2235189-1-ashutoshdesai993@gmail.com/ drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/drm_gem_framebuffer_helper.c b/drivers/gpu/drm/drm_gem_framebuffer_helper.c index 9166c353f131..88808e972cc1 100644 --- a/drivers/gpu/drm/drm_gem_framebuffer_helper.c +++ b/drivers/gpu/drm/drm_gem_framebuffer_helper.c @@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev, } for (i = 0; i < info->num_planes; i++) { - unsigned int width = mode_cmd->width / (i ? info->hsub : 1); - unsigned int height = mode_cmd->height / (i ? info->vsub : 1); + unsigned int width = drm_format_info_plane_width(info, mode_cmd->width, i); + unsigned int height = drm_format_info_plane_height(info, mode_cmd->height, i); unsigned int min_size; objs[i] = drm_gem_object_lookup(file, mode_cmd->handles[i]); -- 2.34.1