From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92CACC44501 for ; Thu, 9 Jul 2026 04:02:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF7F66B0088; Thu, 9 Jul 2026 00:02:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA8CD6B008A; Thu, 9 Jul 2026 00:02:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B70E66B008C; Thu, 9 Jul 2026 00:02:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7B7A36B0088 for ; Thu, 9 Jul 2026 00:02:10 -0400 (EDT) Received: from smtpin22.hostedemail.com (lb01a-stub [10.200.18.249]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EF2631C8364 for ; Thu, 9 Jul 2026 04:02:09 +0000 (UTC) X-FDA: 84967890378.22.B8F029B Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf10.hostedemail.com (Postfix) with ESMTP id 22989C000A for ; Thu, 9 Jul 2026 04:02:07 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20251104 header.b=BPotYqLT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of realwujing@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=realwujing@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1783569728; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=8zdSKyJHGaSQVXICXuU1S4Ua7zP2wyhTHAVBqc1s6N4=; b=FtFv4TuFvIiU41LHMDHdRvu9NLuJC2YtDFYCvs9hmWJQr4X2FD0oK9g28ZL5gmwcI28g1/ equFmBrsLSal/QTBLYqJpN+Pc081VVmF2c2agEe9wimDUkZHUe196vye1JtqLC7Go2dOti Kt52EGSEF2ZNRqjEYjNBH4I+6mSZFr8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20251104 header.b=BPotYqLT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of realwujing@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=realwujing@gmail.com ARC-Seal: i=1; a=rsa-sha256; d=hostedemail.com; s=arc-20220608; cv=none; t=1783569728; b=4qjWbgmIEQNyndzz1LYL7W9+L7/EQhjg2zBCiwTVPN4+s5Ac9VweLy5Wpe1r5HuoI5HAcq y6jZa+N09xJzHlmzqIzWB6Lu4MyebV8JpUKHBisdd8Sy7it66g1/6VhUhGAUiSBXVPKTYm 4giC7oDdZyMQQUczKWHcoBiHDjuz0PM= Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-84536ecfc5bso1541205b3a.2 for ; Wed, 08 Jul 2026 21:02:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783569727; x=1784174527; darn=kvack.org; h=cc:to:message-id:content-transfer-encoding:content-type :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=8zdSKyJHGaSQVXICXuU1S4Ua7zP2wyhTHAVBqc1s6N4=; b=BPotYqLTYxX4xNQqMzhtWU6J1BVVX5IArHlmAGXYWEKDv5PQFI6x+jjvjHUYY7rhvx CV3sI1BB/YJL730U7mF4nIHVb7mUviSbVUxDQ0EyMQMaDJn0fi1x0mSbKenwaGyDvazb H01jxJyFQL+EceMIapTCCwqSrhQsK4LSOBHwZTVax0qXo70mVWindDenAmDCMwSPx6K9 FpTaQ2G1iyr3o3yMB33HjpQC79URDniKAAuyeHKWX6OzEUed9SR8TCCQQo9eSmeFANuK QbK4gX8n4jpJiO5gWK41sKXAZadsLRoN1Z2Bth8swKkLQ25yYVPYo9oIfEna0Qo97F/b Cjew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783569727; x=1784174527; h=cc:to:message-id:content-transfer-encoding:content-type :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=8zdSKyJHGaSQVXICXuU1S4Ua7zP2wyhTHAVBqc1s6N4=; b=D6R81mbJx1jQ2wzRvnaNsfSv9REVFTsbPkhbU1UymcbLjQHlWYdFPU4U7F7Hmj1hRK 5w+GTP+xw3lo7FfOXlSBE9e78X/TzNbUVz827a0wkaLewoHz39VWAT5J1y4BiTXpdaBV XgMMEe0hJQfCmD57Yi+764bME87MraQ7CLP4eZ42IuBLiLXrIgsHiu9FCIjIdAKs2A1d W3QqIxPt9V83IDPMyjJI9wmjNOEscEfWbaOwDM7Z7M+YF4kxbm3/9unF6eQtSFRznC68 vmUyaEYVDMao9009A1L48Vvd63vbN2P/feyUGxCzX+yLm5gWF7jpOhQof9Ww61aZicLs IBoQ== X-Forwarded-Encrypted: i=1; AHgh+RqvjX2T5MbzqH2WjaTXrj0mxowkXtnLLcwjW6cy4bR6dp3tLunP//5Q3hInoTmQfCJwsQqhDzY7kw==@kvack.org X-Gm-Message-State: AOJu0YzwhZZ9OHkO7QY7nwo5DotyVW3V8/XO2RO9LrL5/4emaPcL34+u WdyzQsR49r6Aq0FnsBzhMdpRt/w1+NrV55rgjQ3RrhmubsszUoEvqlO7 X-Gm-Gg: AfdE7clZ3qlz4UqQHpV5tDjicxcZSdrnxqvLUuq5/3AoBMzp7eOjR6B5elm7NIrFWzS kTLAxC1M/N1S8RBrIB2lVA/9YG1cy/7s45k/wwd7LPKtGuZuS9XPkr7ly2sxTgoIIPQgFWI2hr5 aELaYtL7O8zRlX4CFQ0g/kx3In3S+QTKyeGUdEjvg9hSy9Py94p1DB3tmkTlxPg9htfhSCtOid+ t3m4+9ubUhym7Eulq0vQOLDHhBDOdQ8McnG9O3CRtDiLk7v6UOVVeTw2fMYtVQ8YmHFm3EzXvv8 HDpHWptlbml8WIJGe35HKTiO6fWVnwoQL/7CqZkbdIoRI/2gX5VkCyJqdBf1+/ml4lEmZCwd8TJ Oexza4p1x/QJeYeXN08XTXxR63ILo/IgjaaRao2lcqAh7/JaN8o6yjTJCFNqlUmYLqjK/wdshuU VnQuK7wbgCKK8= X-Received: by 2002:a05:6a00:4405:b0:845:ebe2:c656 with SMTP id d2e1a72fcca58-84842fec34cmr5532174b3a.47.1783569726713; Wed, 08 Jul 2026 21:02:06 -0700 (PDT) Received: from [127.0.1.1] ([138.199.21.246]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-847f6b97bbesm8188838b3a.17.2026.07.08.21.02.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jul 2026 21:02:06 -0700 (PDT) From: Qiliang Yuan Date: Thu, 09 Jul 2026 12:02:00 +0800 Subject: [PATCH v7] cgroup/dmem: implement dmem.high soft limit with proactive reclaim MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260709-feature-dmem-high-v7-1-de559e7e7768@gmail.com> X-B4-Tracking: v=1; b=H4sIADcdT2oC/22NQQ6CMBBFr0Jm7RhapIAr72FYkHZoJ7FAWmw0h LtbiUuX7yX//Q0iBaYI12KDQIkjz1OG5lSAdsNkCdlkBllKVdaiw5GG9RkIjSePjq1DobquEZf W6KqFvFsCjfw6mvc+s+O4zuF9XCT1tb9aJf7UkkKBsqxVRdooo5ub9QM/znr20O/7/gEIbGJqs gAAAA== To: Christian Koenig , Huang Rui , Matthew Auld , Matthew Brost , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Tejun Heo , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Natalie Vock Cc: Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Qiliang Yuan , Jing Wu X-Mailer: b4 0.13.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 22989C000A X-Stat-Signature: rrps9fca1bbrxhdgfk9rd3ds7up3r7jj X-HE-Tag: 1783569727-631097 X-HE-Meta: U2FsdGVkX1+S4glorSDXP/E7YyucHtvRpRa2J3gxzyyrkCE5x/nsl4uHDoNzNcrEpvHg2bOCDbuLlhvQiAPX4zusaXrW165VY4Yx8YyUBaYuzf9LXGyV8Lp3IivjNgo0/VCWbW+24L5O/cUJHDW+pw6fb5KB8nixFIXfhAHj+NXsmHrhm87pEdTYxOnn2Uc9W9z2TpGEW3HR92rwEMV5ujBfGUXhZ3E/Y+yVkcbhR3DR+QjatfefMhZR9XHfOadLNlZu4ilDyIirj4PFA8htV2NT+pK4Hk7YXd1X3VtkUMngkfWJNgRPluU7C0K7DC0LWysVukan+hMmmlpk1WnxVbl3vwB8e1Totcx7bQdS7+Gs3dSD9oKqNRNLu5tjKIbT2f0swfteAawQnJn/59MM72i81eYKGYQRdOl+E8Xym9ZukxU05yaMhIm4erHn+mmVBYEWIanUGVzJyGzhCvYXr8MsuuuE/fLpXbP1azB9D17OgHsfRWIOUq2ESWik0jqshxb8gnbCXvJw8HpAShils3Gbe8Qd5wiSByLhj+mg8r0/LNzeDednqgdaNTzJW+/HPJOIlLKCC1P2qTAihk7gjvNxIMUPyEn7EQLJdITUn1N2ye4uR+azvgmBBHvMdNFqetZOiptMc5DV6F4+JIIINcHnzXrkRGssutf7Sp4RSyy83mmsF0F4w7cOrDxEGfds/GA/ZE/7h+0IAAJqSomawNBr3NrNUig9Hbhr5z3PQcoHyI7ruUhp12hWb5wco52kYMbqOFcKx8XMVtrjQMLf99GhKbNgc4VA1Wd/ZfcNaYBGxlAOeb81cu7KjaX+fcNmfJOlQSDrxDzjPRycSwqim6cXySssYy1tLvFOT5xrHDDX3hZWtR9EIy80kegXmrF+M0GKivNCTRwRUKYIHDWE3BXWhpCAMPA231XKKIvWQ2Do+8LuVXejANVEqUKmZ9mc+bmmQ4vSh/ipQhYOHcR CaMmwPYJ PmGD7OSmBae6lT+8QsLNrmAJeANXZYemdSYYDAnm/kgfaNtZee/+WlVYeBqfPErFOjQgDratnx+d2mxTAN9FG6YpoEStThfthkUL5QzZbnUX3cSi35t8ZM+nvA4SEde7qaBH9qwlbz3tgma3KyBeFAkIWb0D96O+hGZSWtOsnvAmJyVoofOYzwzWdTrdM81IyJne5Yk1PLuYW0U4/QQ3yM6EPzbPjb7t6SP0Vx4OEAhTpgDUznqF4zmi0TmLvBSO3YbdSJwrxuLLZOUQ27KPD5k2F8cPuFegeNf41HoH+LyYr75g49uETLsqFhyNPr62M40AfIivcyFZCE7MeQYphLSuKz1FU6UWLmT0oqAuauGNj/HpSBplZeG7BjyJxUYi3yQjMYuP0QybW5jr58hiB3ky+tI9Bnxx+tffc8bQ0ZoEu+egOxbkRKGcVoERe8lveCfrdgH3WlXaCVMSadc7kpgPfP0m+opGRjitetRDBtV9otTcRpyiGspqis/G8jx2bNTeWGQb6KM+4PU529ik24Jq19D7o4simr3eh3IzJ8gcbbA7LEej0TrJxssxpQvsq9EfG5Iyj0r621L3SA9w2gmc3dHjlF/3YcbvJ Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The dmem cgroup v2 controller currently only provides a hard "max" limit, which causes immediate allocation failures when a cgroup's device memory usage reaches its quota. GPU-bound AI workloads need smoother over-subscription support: a soft limit that provides backpressure through reclaim before the hard limit is reached. Implement dmem.high as a proper soft limit following the memory.high semantics: the limit is checked in the charge path on every successful allocation, and proactive reclaim is triggered when usage crosses the threshold, rather than waiting for a dmem.max hit to drive eviction. Expose "high" as a new cgroupfs control file per region via set_resource_high() and get_resource_high(), initialized to PAGE_COUNTER_MAX in reset_all_resource_limits(). Like get_resource_max(), get_resource_high() returns PAGE_COUNTER_MAX when the pool is NULL. Extend dmem_cgroup_try_charge() with a ret_over_high_pool output parameter. On a successful charge, if the pool's usage now exceeds its dmem.high threshold, ret_over_high_pool is set so the caller can trigger proactive reclaim. Propagate this signal through ttm_resource_alloc() up to ttm_bo_alloc_resource(). Add ttm_bo_proactive_evict_high() in TTM, which walks the LRU and evicts one BO from the over-limit cgroup (using the existing try_high logic in dmem_cgroup_state_evict_valuable()). This is best-effort: the allocation already succeeded since dmem.high is a soft limit. A blocking lock is used when a ww_acquire_ctx ticket is available, trylock otherwise. Remove the try_high first pass from ttm_bo_evict_alloc(): that pass tied the interface semantics to a specific eviction ordering detail rather than a proper soft limit. Proactive reclaim in the charge path is the correct place to enforce the soft limit. Co-developed-by: Qiliang Yuan Signed-off-by: Qiliang Yuan Signed-off-by: Jing Wu --- Implement dmem.high as a proper soft limit for the dmem cgroup v2 controller, following memory.high semantics: the limit is checked in the charge path on every successful allocation, and proactive reclaim is triggered when usage crosses the threshold. The dmem cgroup currently only supports a hard "max" limit, which causes allocation failures for GPU-bound workloads. A soft limit enables smoother over-subscription by providing backpressure through reclaim before the hard limit is reached. The implementation extends dmem_cgroup_try_charge() with a ret_over_high_pool output parameter that signals callers when a successful charge pushes usage above the dmem.high threshold. ttm_resource_alloc() propagates this signal up to ttm_bo_alloc_resource(), where ttm_bo_proactive_evict_high() evicts one BO from the over-limit cgroup on a best-effort basis. --- V6 -> V7: - Replace prioritized eviction (eviction-time ordering) with proactive reclaim (charge-time enforcement): dmem.high is now checked in dmem_cgroup_try_charge() on every successful allocation, matching memory.high semantics as requested by Tejun Heo. - Add ret_over_high_pool output parameter to dmem_cgroup_try_charge() to signal callers when a successful charge crosses the high threshold. - Add ttm_bo_proactive_evict_high() to evict one BO from the over-limit cgroup on each allocation that crosses dmem.high (best-effort; the allocation already succeeded since dmem.high is a soft limit). - Remove the try_high first pass from ttm_bo_evict_alloc(): the high limit is no longer enforced via eviction ordering in the max path. - Propagate ret_over_high_pool through ttm_resource_alloc() and update all callers (including TTM test files) to pass the new parameter. - Add Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song to Cc per Tejun's request for memcg input on soft limit semantics. V5 -> V6: - Guard the try_high dereference of test_pool->cnt with a NULL check to prevent a kernel panic during global memory pressure eviction when a BO has no associated cgroup. - Make the disabled-cgroup stub for dmem_cgroup_state_evict_valuable() return false in try_high mode so the stub does not incorrectly enable Pass 1 when CONFIG_CGROUP_DMEM=n. V4 -> V5: - Restore the original control flow in dmem_cgroup_state_evict_valuable(): test_pool is no longer dereferenced before the ancestry checks, fixing a NULL pointer dereference on BOs without a cgroup. The limit_pool NULL-to-root-cgroup resolution is now performed before the try_high block, fixing a panic during global memory pressure eviction. - Keep the try_high check for limit_pool == test_pool inside the existing early-return branch to avoid bypassing the hierarchy constraint check that prevents cross-cgroup eviction. - Use a blocking lock in Pass 1 only when a ticket is available (trylock otherwise), addressing the deadlock risk of blocking without a valid ww_acquire_ctx. - Explicitly reset trylock_only to true before Pass 2 so it does not inherit Pass 1's blocking behavior. V3 -> V4: - Use a blocking lock in Pass 1 instead of trylock to ensure over-limit cgroups are penalized even when their BOs are actively in use, as requested by Maarten Lankhorst. - Evaluate the try_high condition before the limit_pool == test_pool early-return so that the limit-hitting cgroup's own BOs are also filtered by dmem.high. - Remove the high-priority compensation retry at the start of Pass 3, which is no longer needed now that Pass 1 uses a blocking lock. V2 -> V3: - Walk the page_counter parent chain in the try_high pass to prevent child cgroups from evading the penalty when a parent cgroup exceeds its dmem.high limit. - Check dmem.min protection in the try_high pass to avoid evicting BOs below the effective minimum. - Add a properly-locked high-priority retry at the beginning of Pass 3 so that actively-used over-limit BOs (which failed trylock in Pass 1) are not skipped while innocent cgroups are evicted. - Fix get_resource_high(NULL) returning 0 instead of PAGE_COUNTER_MAX to match the behavior of get_resource_max(). V1 -> V2: - Replace sleep-on-allocation throttling with prioritized eviction. - Remove task throttling entirely. - Add dmem.high cgroupfs control file per region. - Extend dmem_cgroup_state_evict_valuable() with try_high parameter. - Refactor ttm_bo_evict_alloc() into a 3-pass eviction strategy. - Initialize high to PAGE_COUNTER_MAX in reset_all_resource_limits(). v6: https://lore.kernel.org/r/20260531-feature-dmem-high-v6-1-20563ecd6dc7@gmail.com v5: https://lore.kernel.org/r/20260531-feature-dmem-high-v5-1-1c6c532b26a9@gmail.com v4: https://lore.kernel.org/r/20260530-feature-dmem-high-v4-1-ee7c6ec1c8da@gmail.com v3: https://lore.kernel.org/r/20260528-feature-dmem-high-v3-1-c642b34bcb2f@gmail.com v2: https://lore.kernel.org/r/20260522-feature-dmem-high-v2-1-1d7d4a0fa5da@gmail.com v1: https://lore.kernel.org/all/20260520-feature-dmem-high-v1-1-97ca0cb7f95a@gmail.com --- drivers/gpu/drm/ttm/tests/ttm_bo_test.c | 18 ++-- drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c | 4 +- drivers/gpu/drm/ttm/tests/ttm_resource_test.c | 2 +- drivers/gpu/drm/ttm/ttm_bo.c | 60 ++++++++++--- drivers/gpu/drm/ttm/ttm_resource.c | 8 +- include/drm/ttm/ttm_resource.h | 3 +- include/linux/cgroup_dmem.h | 15 +++- kernel/cgroup/dmem.c | 104 +++++++++++++++++++++-- 8 files changed, 179 insertions(+), 35 deletions(-) diff --git a/drivers/gpu/drm/ttm/tests/ttm_bo_test.c b/drivers/gpu/drm/ttm/tests/ttm_bo_test.c index f3103307b5df9..7a03f6a04f4e8 100644 --- a/drivers/gpu/drm/ttm/tests/ttm_bo_test.c +++ b/drivers/gpu/drm/ttm/tests/ttm_bo_test.c @@ -258,13 +258,13 @@ static void ttm_bo_unreserve_basic(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, BO_SIZE, NULL); bo->priority = bo_prio; - err = ttm_resource_alloc(bo, place, &res1, NULL); + err = ttm_resource_alloc(bo, place, &res1, NULL, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res1; /* Add a dummy resource to populate LRU */ - ttm_resource_alloc(bo, place, &res2, NULL); + ttm_resource_alloc(bo, place, &res2, NULL, NULL); dma_resv_lock(bo->base.resv, NULL); ttm_bo_unreserve(bo); @@ -300,12 +300,12 @@ static void ttm_bo_unreserve_pinned(struct kunit *test) dma_resv_lock(bo->base.resv, NULL); ttm_bo_pin(bo); - err = ttm_resource_alloc(bo, place, &res1, NULL); + err = ttm_resource_alloc(bo, place, &res1, NULL, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res1; /* Add a dummy resource to the pinned list */ - err = ttm_resource_alloc(bo, place, &res2, NULL); + err = ttm_resource_alloc(bo, place, &res2, NULL, NULL); KUNIT_ASSERT_EQ(test, err, 0); KUNIT_ASSERT_EQ(test, list_is_last(&res2->lru.link, &priv->ttm_dev->unevictable), 1); @@ -355,7 +355,7 @@ static void ttm_bo_unreserve_bulk(struct kunit *test) ttm_bo_set_bulk_move(bo1, &lru_bulk_move); dma_resv_unlock(bo1->base.resv); - err = ttm_resource_alloc(bo1, place, &res1, NULL); + err = ttm_resource_alloc(bo1, place, &res1, NULL, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo1->resource = res1; @@ -363,7 +363,7 @@ static void ttm_bo_unreserve_bulk(struct kunit *test) ttm_bo_set_bulk_move(bo2, &lru_bulk_move); dma_resv_unlock(bo2->base.resv); - err = ttm_resource_alloc(bo2, place, &res2, NULL); + err = ttm_resource_alloc(bo2, place, &res2, NULL, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo2->resource = res2; @@ -401,7 +401,7 @@ static void ttm_bo_fini_basic(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, BO_SIZE, NULL); bo->type = ttm_bo_type_device; - err = ttm_resource_alloc(bo, place, &res, NULL); + err = ttm_resource_alloc(bo, place, &res, NULL, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res; @@ -518,7 +518,7 @@ static void ttm_bo_pin_unpin_resource(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, BO_SIZE, NULL); - err = ttm_resource_alloc(bo, place, &res, NULL); + err = ttm_resource_alloc(bo, place, &res, NULL, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res; @@ -569,7 +569,7 @@ static void ttm_bo_multiple_pin_one_unpin(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, BO_SIZE, NULL); - err = ttm_resource_alloc(bo, place, &res, NULL); + err = ttm_resource_alloc(bo, place, &res, NULL, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res; diff --git a/drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c b/drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c index 2db221f6fc3a1..cd40f5b2ab4f1 100644 --- a/drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c +++ b/drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c @@ -547,7 +547,7 @@ static void ttm_bo_validate_no_placement_signaled(struct kunit *test) ttm_bo_reserve(bo, false, false, NULL); - err = ttm_resource_alloc(bo, place, &bo->resource, NULL); + err = ttm_resource_alloc(bo, place, &bo->resource, NULL, NULL); KUNIT_EXPECT_EQ(test, err, 0); KUNIT_ASSERT_EQ(test, man->usage, size); @@ -604,7 +604,7 @@ static void ttm_bo_validate_no_placement_not_signaled(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, size, NULL); bo->type = params->bo_type; - err = ttm_resource_alloc(bo, place, &bo->resource, NULL); + err = ttm_resource_alloc(bo, place, &bo->resource, NULL, NULL); KUNIT_EXPECT_EQ(test, err, 0); placement = kunit_kzalloc(test, sizeof(*placement), GFP_KERNEL); diff --git a/drivers/gpu/drm/ttm/tests/ttm_resource_test.c b/drivers/gpu/drm/ttm/tests/ttm_resource_test.c index c0e4e35e04426..41fe4d89cd714 100644 --- a/drivers/gpu/drm/ttm/tests/ttm_resource_test.c +++ b/drivers/gpu/drm/ttm/tests/ttm_resource_test.c @@ -303,7 +303,7 @@ static void ttm_sys_man_free_basic(struct kunit *test) res = kunit_kzalloc(test, sizeof(*res), GFP_KERNEL); KUNIT_ASSERT_NOT_NULL(test, res); - ttm_resource_alloc(bo, place, &res, NULL); + ttm_resource_alloc(bo, place, &res, NULL, NULL); man = ttm_manager_type(priv->devs->ttm_dev, mem_type); man->func->free(man, res); diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index bcd76f6bb7f02..bdfcfe5c7d1e2 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -505,6 +505,8 @@ struct ttm_bo_evict_walk { /** @limit_pool: Which pool limit we should test against */ struct dmem_cgroup_pool_state *limit_pool; + /** @try_high: Whether to only evict BO's above the high watermark (first pass) */ + bool try_high; /** @try_low: Whether we should attempt to evict BO's with low watermark threshold */ bool try_low; /** @hit_low: If we cannot evict a bo when @try_low is false (first pass) */ @@ -518,7 +520,8 @@ static s64 ttm_bo_evict_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object * s64 lret; if (!dmem_cgroup_state_evict_valuable(evict_walk->limit_pool, bo->resource->css, - evict_walk->try_low, &evict_walk->hit_low)) + evict_walk->try_high, evict_walk->try_low, + &evict_walk->hit_low)) return 0; if (bo->pin_count || !bo->bdev->funcs->eviction_valuable(bo, evict_walk->place)) @@ -538,7 +541,7 @@ static s64 ttm_bo_evict_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object * evict_walk->evicted++; if (evict_walk->res) lret = ttm_resource_alloc(evict_walk->evictor, evict_walk->place, - evict_walk->res, NULL); + evict_walk->res, NULL, NULL); if (lret == 0) return 1; out: @@ -553,6 +556,38 @@ static const struct ttm_lru_walk_ops ttm_evict_walk_ops = { .process_bo = ttm_bo_evict_cb, }; +/* + * Proactive reclaim: evict one BO from a cgroup that exceeds its dmem.high + * soft limit. Called after a successful charge that pushed usage over the + * high threshold. Best-effort; allocation already succeeded (soft limit). + */ +static void ttm_bo_proactive_evict_high(struct ttm_device *bdev, + struct ttm_resource_manager *man, + const struct ttm_place *place, + struct ttm_buffer_object *evictor, + struct ttm_operation_ctx *ctx, + struct ww_acquire_ctx *ticket, + struct dmem_cgroup_pool_state *over_high_pool) +{ + struct ttm_bo_evict_walk evict_walk = { + .walk = { + .ops = &ttm_evict_walk_ops, + .arg = { + .ctx = ctx, + .ticket = ticket, + .trylock_only = !ticket, + } + }, + .place = place, + .evictor = evictor, + .res = NULL, + .limit_pool = over_high_pool, + .try_high = true, + }; + + ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1); +} + static int ttm_bo_evict_alloc(struct ttm_device *bdev, struct ttm_resource_manager *man, const struct ttm_place *place, @@ -579,29 +614,24 @@ static int ttm_bo_evict_alloc(struct ttm_device *bdev, evict_walk.walk.arg.trylock_only = true; lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1); - - /* One more attempt if we hit low limit? */ if (!lret && evict_walk.hit_low) { evict_walk.try_low = true; lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1); } + if (lret || !ticket) goto out; - /* Reset low limit */ evict_walk.try_low = evict_walk.hit_low = false; - /* If ticket-locking, repeat while making progress. */ evict_walk.walk.arg.trylock_only = false; retry: do { - /* The walk may clear the evict_walk.walk.ticket field */ evict_walk.walk.arg.ticket = ticket; evict_walk.evicted = 0; lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1); } while (!lret && evict_walk.evicted); - /* We hit the low limit? Try once more */ if (!lret && evict_walk.hit_low && !evict_walk.try_low) { evict_walk.try_low = true; goto retry; @@ -737,7 +767,17 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo, continue; may_evict = (force_space && place->mem_type != TTM_PL_SYSTEM); - ret = ttm_resource_alloc(bo, place, res, force_space ? &limit_pool : NULL); + { + struct dmem_cgroup_pool_state *over_high_pool = NULL; + + ret = ttm_resource_alloc(bo, place, res, + force_space ? &limit_pool : NULL, + &over_high_pool); + if (!ret && over_high_pool) + ttm_bo_proactive_evict_high(bdev, man, place, bo, + ctx, ticket, over_high_pool); + dmem_cgroup_pool_state_put(over_high_pool); + } if (ret) { if (ret != -ENOSPC) { dmem_cgroup_pool_state_put(limit_pool); @@ -1152,7 +1192,7 @@ ttm_bo_swapout_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo) memset(&hop, 0, sizeof(hop)); place.mem_type = TTM_PL_SYSTEM; - ret = ttm_resource_alloc(bo, &place, &evict_mem, NULL); + ret = ttm_resource_alloc(bo, &place, &evict_mem, NULL, NULL); if (ret) goto out; diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c index 154d6739256f8..e6ad46d9ff181 100644 --- a/drivers/gpu/drm/ttm/ttm_resource.c +++ b/drivers/gpu/drm/ttm/ttm_resource.c @@ -389,7 +389,8 @@ EXPORT_SYMBOL(ttm_resource_fini); int ttm_resource_alloc(struct ttm_buffer_object *bo, const struct ttm_place *place, struct ttm_resource **res_ptr, - struct dmem_cgroup_pool_state **ret_limit_pool) + struct dmem_cgroup_pool_state **ret_limit_pool, + struct dmem_cgroup_pool_state **ret_over_high_pool) { struct ttm_resource_manager *man = ttm_manager_type(bo->bdev, place->mem_type); @@ -397,7 +398,8 @@ int ttm_resource_alloc(struct ttm_buffer_object *bo, int ret; if (man->cg) { - ret = dmem_cgroup_try_charge(man->cg, bo->base.size, &pool, ret_limit_pool); + ret = dmem_cgroup_try_charge(man->cg, bo->base.size, &pool, + ret_limit_pool, ret_over_high_pool); if (ret) { if (ret == -EAGAIN) ret = -ENOSPC; @@ -409,6 +411,8 @@ int ttm_resource_alloc(struct ttm_buffer_object *bo, if (ret) { if (pool) dmem_cgroup_uncharge(pool, bo->base.size); + if (ret_over_high_pool) + dmem_cgroup_pool_state_put(*ret_over_high_pool); return ret; } diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h index a5d386583fb6e..88fb402acfdc6 100644 --- a/include/drm/ttm/ttm_resource.h +++ b/include/drm/ttm/ttm_resource.h @@ -461,7 +461,8 @@ void ttm_resource_fini(struct ttm_resource_manager *man, int ttm_resource_alloc(struct ttm_buffer_object *bo, const struct ttm_place *place, struct ttm_resource **res, - struct dmem_cgroup_pool_state **ret_limit_pool); + struct dmem_cgroup_pool_state **ret_limit_pool, + struct dmem_cgroup_pool_state **ret_over_high_pool); void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res); bool ttm_resource_intersects(struct ttm_device *bdev, struct ttm_resource *res, diff --git a/include/linux/cgroup_dmem.h b/include/linux/cgroup_dmem.h index dd4869f1d736e..1808bfbbc9a31 100644 --- a/include/linux/cgroup_dmem.h +++ b/include/linux/cgroup_dmem.h @@ -19,11 +19,12 @@ struct dmem_cgroup_region *dmem_cgroup_register_region(u64 size, const char *nam void dmem_cgroup_unregister_region(struct dmem_cgroup_region *region); int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size, struct dmem_cgroup_pool_state **ret_pool, - struct dmem_cgroup_pool_state **ret_limit_pool); + struct dmem_cgroup_pool_state **ret_limit_pool, + struct dmem_cgroup_pool_state **ret_over_high_pool); void dmem_cgroup_uncharge(struct dmem_cgroup_pool_state *pool, u64 size); bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool, struct dmem_cgroup_pool_state *test_pool, - bool ignore_low, bool *ret_hit_low); + bool try_high, bool ignore_low, bool *ret_hit_low); void dmem_cgroup_pool_state_put(struct dmem_cgroup_pool_state *pool); #else @@ -38,13 +39,17 @@ static inline void dmem_cgroup_unregister_region(struct dmem_cgroup_region *regi static inline int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size, struct dmem_cgroup_pool_state **ret_pool, - struct dmem_cgroup_pool_state **ret_limit_pool) + struct dmem_cgroup_pool_state **ret_limit_pool, + struct dmem_cgroup_pool_state **ret_over_high_pool) { *ret_pool = NULL; if (ret_limit_pool) *ret_limit_pool = NULL; + if (ret_over_high_pool) + *ret_over_high_pool = NULL; + return 0; } @@ -54,8 +59,10 @@ static inline void dmem_cgroup_uncharge(struct dmem_cgroup_pool_state *pool, u64 static inline bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool, struct dmem_cgroup_pool_state *test_pool, - bool ignore_low, bool *ret_hit_low) + bool try_high, bool ignore_low, bool *ret_hit_low) { + if (try_high) + return false; return true; } diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c index 4753a67d0f0f2..b322c8a7e2a67 100644 --- a/kernel/cgroup/dmem.c +++ b/kernel/cgroup/dmem.c @@ -156,6 +156,12 @@ set_resource_low(struct dmem_cgroup_pool_state *pool, u64 val) page_counter_set_low(&pool->cnt, val); } +static void +set_resource_high(struct dmem_cgroup_pool_state *pool, u64 val) +{ + page_counter_set_high(&pool->cnt, val); +} + static void set_resource_max(struct dmem_cgroup_pool_state *pool, u64 val) { @@ -167,6 +173,11 @@ static u64 get_resource_low(struct dmem_cgroup_pool_state *pool) return pool ? READ_ONCE(pool->cnt.low) : 0; } +static u64 get_resource_high(struct dmem_cgroup_pool_state *pool) +{ + return pool ? READ_ONCE(pool->cnt.high) : PAGE_COUNTER_MAX; +} + static u64 get_resource_min(struct dmem_cgroup_pool_state *pool) { return pool ? READ_ONCE(pool->cnt.min) : 0; @@ -186,6 +197,7 @@ static void reset_all_resource_limits(struct dmem_cgroup_pool_state *rpool) { set_resource_min(rpool, 0); set_resource_low(rpool, 0); + set_resource_high(rpool, PAGE_COUNTER_MAX); set_resource_max(rpool, PAGE_COUNTER_MAX); } @@ -289,10 +301,13 @@ dmem_cgroup_calculate_protection(struct dmem_cgroup_pool_state *limit_pool, * dmem_cgroup_state_evict_valuable() - Check if we should evict from test_pool * @limit_pool: The pool for which we hit limits * @test_pool: The pool for which to test + * @try_high: Only evict BOs whose usage exceeds the high limit (first pass) * @ignore_low: Whether we have to respect low watermarks. * @ret_hit_low: Pointer to whether it makes sense to consider low watermark. * * This function returns true if we can evict from @test_pool, false if not. + * When @try_high is set, only pools with usage above their high limit are + * evictable, enabling prioritized eviction of over-limit cgroups. * When returning false and @ignore_low is false, @ret_hit_low may * be set to true to indicate this function can be retried with @ignore_low * set to true. @@ -301,15 +316,26 @@ dmem_cgroup_calculate_protection(struct dmem_cgroup_pool_state *limit_pool, */ bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool, struct dmem_cgroup_pool_state *test_pool, - bool ignore_low, bool *ret_hit_low) + bool try_high, bool ignore_low, bool *ret_hit_low) { struct dmem_cgroup_pool_state *pool = test_pool; struct page_counter *ctest; u64 used, min, low; - /* Can always evict from current pool, despite limits */ - if (limit_pool == test_pool) + /* + * When the limit-hitting cgroup's own BOs are being considered + * in try_high mode, only evict them if their pool exceeds its + * own dmem.high limit. For non-try_high mode, maintain the + * existing behavior: always evict from the limit-hitting pool. + */ + if (limit_pool == test_pool) { + if (try_high && test_pool) { + ctest = &test_pool->cnt; + used = page_counter_read(ctest); + return used > READ_ONCE(ctest->high); + } return true; + } if (limit_pool) { if (!parent_dmemcs(limit_pool->cs)) @@ -330,10 +356,38 @@ bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool, } ctest = &test_pool->cnt; + used = page_counter_read(ctest); + + if (try_high) { + struct page_counter *c; + + /* + * Walk the page_counter parent chain to check whether any + * ancestor cgroup exceeds its dmem.high limit. This prevents + * child cgroups from evading the penalty when a parent cgroup + * is over its high limit. + */ + if (used <= READ_ONCE(ctest->high)) { + for (c = ctest->parent; c; c = c->parent) { + if (page_counter_read(c) > READ_ONCE(c->high)) + break; + } + if (!c) + return false; + } + + /* + * Respect dmem.min protection: do not evict BOs below the + * effective minimum even during the high-priority pass. + */ + dmem_cgroup_calculate_protection(limit_pool, test_pool); + min = READ_ONCE(ctest->emin); + + return used > min; + } dmem_cgroup_calculate_protection(limit_pool, test_pool); - used = page_counter_read(ctest); min = READ_ONCE(ctest->emin); if (used <= min) @@ -634,8 +688,9 @@ EXPORT_SYMBOL_GPL(dmem_cgroup_uncharge); * dmem_cgroup_try_charge() - Try charging a new allocation to a region. * @region: dmem region to charge * @size: Size (in bytes) to charge. - * @ret_pool: On succesfull allocation, the pool that is charged. + * @ret_pool: On successful allocation, the pool that is charged. * @ret_limit_pool: On a failed allocation, the limiting pool. + * @ret_over_high_pool: On successful allocation, set if usage exceeds dmem.high. * * This function charges the @region region for a size of @size bytes. * @@ -647,11 +702,17 @@ EXPORT_SYMBOL_GPL(dmem_cgroup_uncharge); * eviction as argument to dmem_cgroup_evict_valuable(). This reference must be freed * with @dmem_cgroup_pool_state_put(). * + * When the function succeeds and @ret_over_high_pool is non-null, it will be + * set if the charged pool's usage now exceeds its dmem.high soft limit. The + * caller should trigger proactive eviction to bring usage back under the limit. + * This reference must be freed with @dmem_cgroup_pool_state_put(). + * * Return: 0 on success, -EAGAIN on hitting a limit, or a negative errno on failure. */ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size, struct dmem_cgroup_pool_state **ret_pool, - struct dmem_cgroup_pool_state **ret_limit_pool) + struct dmem_cgroup_pool_state **ret_limit_pool, + struct dmem_cgroup_pool_state **ret_over_high_pool) { struct dmemcg_state *cg; struct dmem_cgroup_pool_state *pool; @@ -661,6 +722,8 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size, *ret_pool = NULL; if (ret_limit_pool) *ret_limit_pool = NULL; + if (ret_over_high_pool) + *ret_over_high_pool = NULL; /* * hold on to css, as cgroup can be removed but resource @@ -685,6 +748,18 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size, goto err; } + /* + * Charge succeeded. Check if usage now exceeds the soft high limit so + * the caller can trigger proactive reclaim to bring the cgroup back + * under its dmem.high threshold. + */ + if (ret_over_high_pool && + page_counter_read(&pool->cnt) > READ_ONCE(pool->cnt.high)) { + *ret_over_high_pool = pool; + css_get(&pool->cs->css); + dmemcg_pool_get(*ret_over_high_pool); + } + /* On success, reference from get_current_dmemcs is transferred to *ret_pool */ *ret_pool = pool; return 0; @@ -835,6 +910,17 @@ static ssize_t dmem_cgroup_region_low_write(struct kernfs_open_file *of, return dmemcg_limit_write(of, buf, nbytes, off, set_resource_low); } +static int dmem_cgroup_region_high_show(struct seq_file *sf, void *v) +{ + return dmemcg_limit_show(sf, v, get_resource_high); +} + +static ssize_t dmem_cgroup_region_high_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) +{ + return dmemcg_limit_write(of, buf, nbytes, off, set_resource_high); +} + static int dmem_cgroup_region_max_show(struct seq_file *sf, void *v) { return dmemcg_limit_show(sf, v, get_resource_max); @@ -868,6 +954,12 @@ static struct cftype files[] = { .seq_show = dmem_cgroup_region_low_show, .flags = CFTYPE_NOT_ON_ROOT, }, + { + .name = "high", + .write = dmem_cgroup_region_high_write, + .seq_show = dmem_cgroup_region_high_show, + .flags = CFTYPE_NOT_ON_ROOT, + }, { .name = "max", .write = dmem_cgroup_region_max_write, --- base-commit: ab5fce87a778cb780a05984a2ca448f2b41aafbf change-id: 20260519-feature-dmem-high-16997148dc38 Best regards, -- Jing Wu