All of lore.kernel.org
 help / color / mirror / Atom feed
From: David Hildenbrand <david@redhat.com>
To: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org, David Hildenbrand <david@redhat.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Matthew Wilcox <willy@infradead.org>,
	Ryan Roberts <ryan.roberts@arm.com>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will@kernel.org>,
	"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
	Nick Piggin <npiggin@gmail.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Christophe Leroy <christophe.leroy@csgroup.eu>,
	"Naveen N. Rao" <naveen.n.rao@linux.ibm.com>,
	Heiko Carstens <hca@linux.ibm.com>,
	Vasily Gorbik <gor@linux.ibm.com>,
	Alexander Gordeev <agordeev@linux.ibm.com>,
	Christian Borntraeger <borntraeger@linux.ibm.com>,
	Sven Schnelle <svens@linux.ibm.com>,
	Arnd Bergmann <arnd@arndb.de>,
	linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
	linux-s390@vger.kernel.org
Subject: [PATCH v1 3/9] mm/memory: further separate anon and pagecache folio handling in zap_present_pte()
Date: Mon, 29 Jan 2024 15:32:15 +0100	[thread overview]
Message-ID: <20240129143221.263763-4-david@redhat.com> (raw)
In-Reply-To: <20240129143221.263763-1-david@redhat.com>

We don't need up-to-date accessed-dirty information for anon folios and can
simply work with the ptent we already have. Also, we know the RSS counter
we want to update.

We can safely move arch_check_zapped_pte() + tlb_remove_tlb_entry() +
zap_install_uffd_wp_if_needed() after updating the folio and RSS.

While at it, only call zap_install_uffd_wp_if_needed() if there is even
any chance that pte_install_uffd_wp_if_needed() would do *something*.
That is, just don't bother if uffd-wp does not apply.

Signed-off-by: David Hildenbrand <david@redhat.com>
---
 mm/memory.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 69502cdc0a7d..20bc13ab8db2 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1552,12 +1552,9 @@ static inline void zap_present_pte(struct mmu_gather *tlb,
 	folio = page_folio(page);
 	if (unlikely(!should_zap_folio(details, folio)))
 		return;
-	ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm);
-	arch_check_zapped_pte(vma, ptent);
-	tlb_remove_tlb_entry(tlb, pte, addr);
-	zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent);
 
 	if (!folio_test_anon(folio)) {
+		ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm);
 		if (pte_dirty(ptent)) {
 			folio_mark_dirty(folio);
 			if (tlb_delay_rmap(tlb)) {
@@ -1567,8 +1564,17 @@ static inline void zap_present_pte(struct mmu_gather *tlb,
 		}
 		if (pte_young(ptent) && likely(vma_has_recency(vma)))
 			folio_mark_accessed(folio);
+		rss[mm_counter(folio)]--;
+	} else {
+		/* We don't need up-to-date accessed/dirty bits. */
+		ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm);
+		rss[MM_ANONPAGES]--;
 	}
-	rss[mm_counter(folio)]--;
+	arch_check_zapped_pte(vma, ptent);
+	tlb_remove_tlb_entry(tlb, pte, addr);
+	if (unlikely(userfaultfd_pte_wp(vma, ptent)))
+		zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent);
+
 	if (!delay_rmap) {
 		folio_remove_rmap_pte(folio, page, vma);
 		if (unlikely(page_mapcount(page) < 0))
-- 
2.43.0


WARNING: multiple messages have this Message-ID (diff)
From: David Hildenbrand <david@redhat.com>
To: linux-kernel@vger.kernel.org
Cc: David Hildenbrand <david@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	linux-mm@kvack.org, Alexander Gordeev <agordeev@linux.ibm.com>,
	Will Deacon <will@kernel.org>,
	linux-arch@vger.kernel.org, linux-s390@vger.kernel.org,
	Vasily Gorbik <gor@linux.ibm.com>,
	Matthew Wilcox <willy@infradead.org>,
	"Naveen N. Rao" <naveen.n.rao@linux.ibm.com>,
	Christian Borntraeger <borntraeger@linux.ibm.com>,
	Ryan Roberts <ryan.roberts@arm.com>,
	Arnd Bergmann <arnd@arndb.de>, Heiko Carstens <hca@linux.ibm.com>,
	Nick Piggin <npiggin@gmail.com>,
	Sven Schnelle <svens@linux.ibm.com>,
	"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	linuxppc-dev@lists.ozlabs.org
Subject: [PATCH v1 3/9] mm/memory: further separate anon and pagecache folio handling in zap_present_pte()
Date: Mon, 29 Jan 2024 15:32:15 +0100	[thread overview]
Message-ID: <20240129143221.263763-4-david@redhat.com> (raw)
In-Reply-To: <20240129143221.263763-1-david@redhat.com>

We don't need up-to-date accessed-dirty information for anon folios and can
simply work with the ptent we already have. Also, we know the RSS counter
we want to update.

We can safely move arch_check_zapped_pte() + tlb_remove_tlb_entry() +
zap_install_uffd_wp_if_needed() after updating the folio and RSS.

While at it, only call zap_install_uffd_wp_if_needed() if there is even
any chance that pte_install_uffd_wp_if_needed() would do *something*.
That is, just don't bother if uffd-wp does not apply.

Signed-off-by: David Hildenbrand <david@redhat.com>
---
 mm/memory.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 69502cdc0a7d..20bc13ab8db2 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1552,12 +1552,9 @@ static inline void zap_present_pte(struct mmu_gather *tlb,
 	folio = page_folio(page);
 	if (unlikely(!should_zap_folio(details, folio)))
 		return;
-	ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm);
-	arch_check_zapped_pte(vma, ptent);
-	tlb_remove_tlb_entry(tlb, pte, addr);
-	zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent);
 
 	if (!folio_test_anon(folio)) {
+		ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm);
 		if (pte_dirty(ptent)) {
 			folio_mark_dirty(folio);
 			if (tlb_delay_rmap(tlb)) {
@@ -1567,8 +1564,17 @@ static inline void zap_present_pte(struct mmu_gather *tlb,
 		}
 		if (pte_young(ptent) && likely(vma_has_recency(vma)))
 			folio_mark_accessed(folio);
+		rss[mm_counter(folio)]--;
+	} else {
+		/* We don't need up-to-date accessed/dirty bits. */
+		ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm);
+		rss[MM_ANONPAGES]--;
 	}
-	rss[mm_counter(folio)]--;
+	arch_check_zapped_pte(vma, ptent);
+	tlb_remove_tlb_entry(tlb, pte, addr);
+	if (unlikely(userfaultfd_pte_wp(vma, ptent)))
+		zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent);
+
 	if (!delay_rmap) {
 		folio_remove_rmap_pte(folio, page, vma);
 		if (unlikely(page_mapcount(page) < 0))
-- 
2.43.0


  parent reply	other threads:[~2024-01-29 14:32 UTC|newest]

Thread overview: 80+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-01-29 14:32 [PATCH v1 0/9] mm/memory: optimize unmap/zap with PTE-mapped THP David Hildenbrand
2024-01-29 14:32 ` David Hildenbrand
2024-01-29 14:32 ` [PATCH v1 1/9] mm/memory: factor out zapping of present pte into zap_present_pte() David Hildenbrand
2024-01-29 14:32   ` David Hildenbrand
2024-01-30  8:13   ` Ryan Roberts
2024-01-30  8:13     ` Ryan Roberts
2024-01-30  8:41     ` David Hildenbrand
2024-01-30  8:41       ` David Hildenbrand
2024-01-30  8:46       ` Ryan Roberts
2024-01-30  8:46         ` Ryan Roberts
2024-01-30  8:49         ` David Hildenbrand
2024-01-30  8:49           ` David Hildenbrand
2024-01-29 14:32 ` [PATCH v1 2/9] mm/memory: handle !page case in zap_present_pte() separately David Hildenbrand
2024-01-29 14:32   ` David Hildenbrand
2024-01-30  8:20   ` Ryan Roberts
2024-01-30  8:20     ` Ryan Roberts
2024-01-29 14:32 ` David Hildenbrand [this message]
2024-01-29 14:32   ` [PATCH v1 3/9] mm/memory: further separate anon and pagecache folio handling in zap_present_pte() David Hildenbrand
2024-01-30  8:31   ` Ryan Roberts
2024-01-30  8:31     ` Ryan Roberts
2024-01-30  8:37     ` David Hildenbrand
2024-01-30  8:37       ` David Hildenbrand
2024-01-30  8:45       ` Ryan Roberts
2024-01-30  8:45         ` Ryan Roberts
2024-01-30  8:47         ` David Hildenbrand
2024-01-30  8:47           ` David Hildenbrand
2024-01-29 14:32 ` [PATCH v1 4/9] mm/memory: factor out zapping folio pte into zap_present_folio_pte() David Hildenbrand
2024-01-29 14:32   ` David Hildenbrand
2024-01-30  8:47   ` Ryan Roberts
2024-01-30  8:47     ` Ryan Roberts
2024-01-29 14:32 ` [PATCH v1 5/9] mm/mmu_gather: pass "delay_rmap" instead of encoded page to __tlb_remove_page_size() David Hildenbrand
2024-01-29 14:32   ` David Hildenbrand
2024-01-30  8:41   ` Ryan Roberts
2024-01-30  8:41     ` Ryan Roberts
2024-01-29 14:32 ` [PATCH v1 6/9] mm/mmu_gather: define ENCODED_PAGE_FLAG_DELAY_RMAP David Hildenbrand
2024-01-29 14:32   ` David Hildenbrand
2024-01-30  9:03   ` Ryan Roberts
2024-01-30  9:03     ` Ryan Roberts
2024-01-29 14:32 ` [PATCH v1 7/9] mm/mmu_gather: add __tlb_remove_folio_pages() David Hildenbrand
2024-01-29 14:32   ` David Hildenbrand
2024-01-30  9:21   ` Ryan Roberts
2024-01-30  9:21     ` Ryan Roberts
2024-01-30  9:33     ` David Hildenbrand
2024-01-30  9:33       ` David Hildenbrand
2024-01-29 14:32 ` [PATCH v1 8/9] mm/mmu_gather: add tlb_remove_tlb_entries() David Hildenbrand
2024-01-29 14:32   ` David Hildenbrand
2024-01-30  9:33   ` Ryan Roberts
2024-01-30  9:33     ` Ryan Roberts
2024-01-29 14:32 ` [PATCH v1 9/9] mm/memory: optimize unmap/zap with PTE-mapped THP David Hildenbrand
2024-01-29 14:32   ` David Hildenbrand
2024-01-30  9:08   ` David Hildenbrand
2024-01-30  9:08     ` David Hildenbrand
2024-01-30  9:48   ` Ryan Roberts
2024-01-30  9:48     ` Ryan Roberts
2024-01-31 10:21     ` David Hildenbrand
2024-01-31 10:21       ` David Hildenbrand
2024-01-31 10:31       ` Ryan Roberts
2024-01-31 10:31         ` Ryan Roberts
2024-01-31 11:13         ` David Hildenbrand
2024-01-31 11:13           ` David Hildenbrand
2024-01-31  2:30   ` Yin Fengwei
2024-01-31  2:30     ` Yin Fengwei
2024-01-31 10:30     ` David Hildenbrand
2024-01-31 10:30       ` David Hildenbrand
2024-01-31 10:43       ` Yin, Fengwei
2024-01-31 10:43         ` Yin, Fengwei
2024-01-31  2:20 ` [PATCH v1 0/9] " Yin Fengwei
2024-01-31  2:20   ` Yin Fengwei
2024-01-31 10:16   ` David Hildenbrand
2024-01-31 10:16     ` David Hildenbrand
2024-01-31 10:26     ` Ryan Roberts
2024-01-31 10:26       ` Ryan Roberts
2024-01-31 14:08       ` Michal Hocko
2024-01-31 14:08         ` Michal Hocko
2024-01-31 14:20         ` David Hildenbrand
2024-01-31 14:20           ` David Hildenbrand
2024-01-31 14:03     ` Michal Hocko
2024-01-31 14:03       ` Michal Hocko
2024-01-31 10:43   ` David Hildenbrand
2024-01-31 10:43     ` David Hildenbrand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240129143221.263763-4-david@redhat.com \
    --to=david@redhat.com \
    --cc=agordeev@linux.ibm.com \
    --cc=akpm@linux-foundation.org \
    --cc=aneesh.kumar@linux.ibm.com \
    --cc=arnd@arndb.de \
    --cc=borntraeger@linux.ibm.com \
    --cc=catalin.marinas@arm.com \
    --cc=christophe.leroy@csgroup.eu \
    --cc=gor@linux.ibm.com \
    --cc=hca@linux.ibm.com \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-s390@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=mpe@ellerman.id.au \
    --cc=naveen.n.rao@linux.ibm.com \
    --cc=npiggin@gmail.com \
    --cc=peterz@infradead.org \
    --cc=ryan.roberts@arm.com \
    --cc=svens@linux.ibm.com \
    --cc=will@kernel.org \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.