From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA924606A31F for ; Sat, 31 Oct 2026 09:42:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=OOG8QuQD; dkim=temperror header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=Z5e53RJF DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Unsubscribe-Post: List-Owner; bh=PCXji8t2YghBn8zNO97WnfB06mhFIwWKCiaxVt0zvRk=; b=OOG8QuQD3i95q2 d8pPKhd1+7fTBS9kYTgBMFYNStJEb8mpgujRH9adPBNszld+kp+b6hq9qB1CyLkaY5kmDCAWDNiXV /o2fQKtfRvLa62pCrG88zoMm4avPMa/Oo4p2LC1c+EDBrERI/OYLai7pDhGBxF9Kraaw25/tP98Rh 5knOecQCH/uzxpXXCbCPGPJAn0yFbC2vpUY+Z/wykoPhJQgkRRiUfqHg8OkEAKVLNIKVfV+6UJv4K ztGR0LaD5cn1k56zJluEy5biaIpZXYLxxbWEG0XuTG2ubIBsv2/EQBZPLdNS7YNGOYfN7dFqBqhFA VtpBXWZeL729CMWkIISg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.100.1 #2 (Red Hat Linux)) id 1xN5bh-0000000D9Np-2nBK; Sat, 31 Oct 2026 09:42:37 +0000 Received: from tor.source.kernel.org ([172.105.4.254]) by bombadil.infradead.org with esmtps (Exim 4.100.1 #2 (Red Hat Linux)) id 1xN5bg-0000000D9NV-0Z8e for kexec@lists.infradead.org; Sat, 31 Oct 2026 09:42:36 +0000 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by tor.source.kernel.org (Postfix) with ESMTP id 904B8600C8; Sat, 31 Oct 2026 09:42:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C6831F0093A; Sat, 31 Oct 2026 09:42:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1793439755; bh=PCXji8t2YghBn8zNO97WnfB06mhFIwWKCiaxVt0zvRk=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=Z5e53RJFYXZm94nrWWOWfz98i5n0k7Vq1mlTIUq5K8+xJc8k6dpaYYzSZsF4CgBGh v4KYqINsYG9kt2cDdfKy0eTyGyYpv6J5W5NiuEPWdzqHf88nGhb0V60FfXxzWiLdX3 QmdhnZrltq4DP3/sy1XSUvGTC8FipG5eXCbz9AOB+hjr5+MIg4VzYpuGk4ssaJpufB TQO0s/RKG+JX4ALqjZDzvmreezhmYOe5sHjpVmP0tZoPQLouAFH7bKEQyTb3phuSOY jUGQIHVgorDoKoHm4liePBrx/LWneHktnw6p09RxrbfVYcoo6un0Z7fj/dn2+TFapn MVKDKVJ5kogzA== From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:38 +0100 Subject: [PATCH v3 09/47] mm/rmap: handle device-exclusive entries in folio_referenced_one() X-Archived-By: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Message-Id: <20261031-fixes_core-v3-9-d6437171ecc3@kernel.org> References: <20261031-fixes_core-v3-0-d6437171ecc3@kernel.org> In-Reply-To: <20261031-fixes_core-v3-0-d6437171ecc3@kernel.org> To: Andrew Morton , Johannes Weiner , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Baolin Wang , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Peter Zijlstra , Jann Horn , Pedro Falcato , Hugh Dickins , Pasha Tatashin , Pratyush Yadav , "Matthew Wilcox (Oracle)" , Jan Kara , Zi Yan , Lance Yang , Rik van Riel , Harry Yoo , Alistair Popple , Nico Pache , Ryan Roberts , Dev Jain , Usama Arif , Kiryl Shutsemau , Matthew Brost , John Hubbard , Peter Xu , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Oscar Salvador , Minchan Kim , Muchun Song , Xu Xin , Chengming Zhou , Chris Li , Kemeng Shi , Nhat Pham , Youngjun Park , Gregory Price , Joshua Hahn , Rakie Kim , Ying Huang , Jason Gunthorpe Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kexec@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-cxl@vger.kernel.org, stable@vger.kernel.org, Josef Bacik , Muhammad Usama Anjum , "David Hildenbrand (Arm)" X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3393; i=david@kernel.org; h=from:subject:message-id; bh=glmREPecf3Soxf23A8cAtJsaeHFT2ex5UzHoQI9g+28=; b=owGbwMvMwCF2/OVpCdfQ3ZWMp9WSGLKebk8TXHrRW3j5ImPXqRM2+EWeLvBgOzKltzWl/Yf1J /Wu2evmdpSyMIhxMMiKKbIEBd83ePlyXW21Gx8zzBxWJpAhDFycAjARpheMDFNaAhtDhL1kNO1c dGdue3/ixn6rTU5ud9Ss+l5+3C7szsbwk7Fgs0P+Td/7KsFuhndbzDKmPefPbc5Li/3Itfb6dNY p3AA= X-Developer-Key: i=david@kernel.org; a=openpgp; fpr=1BD9CAAD735C4C3A460DFCCA4DDE10F700FF835A X-BeenThere: kexec@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kexec" X-Spam-Status: No, score=-18.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=4.0.1 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on aws-us-west-2-korg-listserv-alma10-1 page_vma_mapped_walk() can return non-present device-exclusive entries, but folio_referenced_one() passes them to PTE young-bit accessors that require present PTEs. We must neither call clear_flush_young_ptes_notify() nor folio_pte_batch() with non-present entries. For example, clearing the young bit on non-present PTEs can corrupt the entry: on arm64 PTE_AF overlaps the swap type, so clearing it turns a device-exclusive entry into an ordinary swap entry. folio_pte_batch() will VM_WARN_ON_FOLIO. Apply the same approach as the previous fixes to e.g., damon_ptep_mkold(): treat these entries as old from the CPU's perspective and obtain their young state through the MMU notifier. Skip PTE batching and MGLRU look-around for non-present entries, and don't modify non-present PTEs. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Reviewed-by: Lorenzo Stoakes (ARM) Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Harry Yoo (Meta) Reviewed-by: Barry Song Reviewed-by: Dev Jain Tested-by: Muhammad Usama Anjum Signed-off-by: David Hildenbrand (Arm) --- mm/rmap.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 1131b76bbbc28..0faef46806e5c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -924,6 +924,7 @@ static bool folio_referenced_one(struct folio *folio, DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); int ptes = 0, referenced = 0; unsigned int nr; + pte_t pteval; while (page_vma_mapped_walk(&pvmw)) { address = pvmw.address; @@ -968,10 +969,12 @@ static bool folio_referenced_one(struct folio *folio, return false; } - if (pvmw.pte && folio_test_large(folio)) { + if (pvmw.pte) + pteval = ptep_get(pvmw.pte); + + if (pvmw.pte && pte_present(pteval) && folio_test_large(folio)) { const unsigned long end_addr = pmd_addr_end(address, vma->vm_end); const unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT; - pte_t pteval = ptep_get(pvmw.pte); nr = folio_pte_batch(folio, pvmw.pte, pteval, max_nr); } @@ -981,11 +984,23 @@ static bool folio_referenced_one(struct folio *folio, * are. —they could be on active/inactive lists or on MGLRU. So the * simplest approach is to disable this look-around optimization. */ - if (lru_gen_enabled() && !lru_gen_switching() && pvmw.pte) { + if (lru_gen_enabled() && !lru_gen_switching() && pvmw.pte && + pte_present(pteval)) { if (lru_gen_look_around(&pvmw, nr)) referenced++; } else if (pvmw.pte) { - if (clear_flush_young_ptes_notify(vma, address, pvmw.pte, nr)) + bool young = false; + + /* + * PFN swap PTEs, such as device-exclusive ones, that + * actually map pages are "old" from a CPU perspective. + * The MMU notifier takes care of any device aspects. + */ + if (pte_present(pteval)) + young = clear_flush_young_ptes(vma, address, pvmw.pte, nr); + young |= mmu_notifier_clear_flush_young(vma->vm_mm, address, + address + nr * PAGE_SIZE); + if (young) referenced++; } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { if (pmdp_clear_flush_young_notify(vma, address, -- 2.43.0