[merged mm-stable] mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch removed from -mm tree

[merged mm-stable] mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch removed from -mm tree

[Andrew Morton]

The quilt patch titled
     Subject: mm/madvise: optimize lazyfreeing with mTHP in madvise_free
has been removed from the -mm tree.  Its filename was
     mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch

This patch was dropped because it was merged into the mm-stable branch
of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

------------------------------------------------------
From: Lance Yang <ioworker0@gmail.com>
Subject: mm/madvise: optimize lazyfreeing with mTHP in madvise_free
Date: Thu, 18 Apr 2024 21:44:35 +0800

This patch optimizes lazyfreeing with PTE-mapped mTHP[1] (Inspired by
David Hildenbrand[2]).  We aim to avoid unnecessary folio splitting if the
large folio is fully mapped within the target range.

If a large folio is locked or shared, or if we fail to split it, we just
leave it in place and advance to the next PTE in the range.  But note that
the behavior is changed; previously, any failure of this sort would cause
the entire operation to give up.  As large folios become more common,
sticking to the old way could result in wasted opportunities.

On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of
the same size results in the following runtimes for madvise(MADV_FREE) in
seconds (shorter is better):

Folio Size |   Old    |   New    | Change
------------------------------------------
      4KiB | 0.590251 | 0.590259 |    0%
     16KiB | 2.990447 | 0.185655 |  -94%
     32KiB | 2.547831 | 0.104870 |  -95%
     64KiB | 2.457796 | 0.052812 |  -97%
    128KiB | 2.281034 | 0.032777 |  -99%
    256KiB | 2.230387 | 0.017496 |  -99%
    512KiB | 2.189106 | 0.010781 |  -99%
   1024KiB | 2.183949 | 0.007753 |  -99%
   2048KiB | 0.002799 | 0.002804 |    0%

[1] https://lkml.kernel.org/r/20231207161211.2374093-5-ryan.roberts@arm.com
[2] https://lore.kernel.org/linux-mm/20240214204435.167852-1-david@redhat.com

Link: https://lkml.kernel.org/r/20240418134435.6092-5-ioworker0@gmail.com
Signed-off-by: Lance Yang <ioworker0@gmail.com>
Reviewed-by: Ryan Roberts <ryan.roberts@arm.com>
Acked-by: David Hildenbrand <david@redhat.com>
Cc: Barry Song <21cnbao@gmail.com>
Cc: Jeff Xie <xiehuan09@gmail.com>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Muchun Song <songmuchun@bytedance.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Yang Shi <shy828301@gmail.com>
Cc: Yin Fengwei <fengwei.yin@intel.com>
Cc: Zach O'Keefe <zokeefe@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/madvise.c |   85 +++++++++++++++++++++++++------------------------
 1 file changed, 44 insertions(+), 41 deletions(-)

--- a/mm/madvise.c~mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free
+++ a/mm/madvise.c
@@ -643,6 +643,7 @@ static int madvise_free_pte_range(pmd_t
 				unsigned long end, struct mm_walk *walk)
 
 {
+	const cydp_t cydp_flags = CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY;
 	struct mmu_gather *tlb = walk->private;
 	struct mm_struct *mm = tlb->mm;
 	struct vm_area_struct *vma = walk->vma;
@@ -697,44 +698,57 @@ static int madvise_free_pte_range(pmd_t
 			continue;
 
 		/*
-		 * If pmd isn't transhuge but the folio is large and
-		 * is owned by only this process, split it and
-		 * deactivate all pages.
+		 * If we encounter a large folio, only split it if it is not
+		 * fully mapped within the range we are operating on. Otherwise
+		 * leave it as is so that it can be marked as lazyfree. If we
+		 * fail to split a folio, leave it in place and advance to the
+		 * next pte in the range.
 		 */
 		if (folio_test_large(folio)) {
-			int err;
+			bool any_young, any_dirty;
 
-			if (folio_likely_mapped_shared(folio))
-				break;
-			if (!folio_trylock(folio))
-				break;
-			folio_get(folio);
-			arch_leave_lazy_mmu_mode();
-			pte_unmap_unlock(start_pte, ptl);
-			start_pte = NULL;
-			err = split_folio(folio);
-			folio_unlock(folio);
-			folio_put(folio);
-			if (err)
-				break;
-			start_pte = pte =
-				pte_offset_map_lock(mm, pmd, addr, &ptl);
-			if (!start_pte)
-				break;
-			arch_enter_lazy_mmu_mode();
-			pte--;
-			addr -= PAGE_SIZE;
-			continue;
+			nr = madvise_folio_pte_batch(addr, end, folio, pte,
+						     ptent, &any_young, &any_dirty);
+
+			if (nr < folio_nr_pages(folio)) {
+				int err;
+
+				if (folio_likely_mapped_shared(folio))
+					continue;
+				if (!folio_trylock(folio))
+					continue;
+				folio_get(folio);
+				arch_leave_lazy_mmu_mode();
+				pte_unmap_unlock(start_pte, ptl);
+				start_pte = NULL;
+				err = split_folio(folio);
+				folio_unlock(folio);
+				folio_put(folio);
+				pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
+				start_pte = pte;
+				if (!start_pte)
+					break;
+				arch_enter_lazy_mmu_mode();
+				if (!err)
+					nr = 0;
+				continue;
+			}
+
+			if (any_young)
+				ptent = pte_mkyoung(ptent);
+			if (any_dirty)
+				ptent = pte_mkdirty(ptent);
 		}
 
 		if (folio_test_swapcache(folio) || folio_test_dirty(folio)) {
 			if (!folio_trylock(folio))
 				continue;
 			/*
-			 * If folio is shared with others, we mustn't clear
-			 * the folio's dirty flag.
+			 * If we have a large folio at this point, we know it is
+			 * fully mapped so if its mapcount is the same as its
+			 * number of pages, it must be exclusive.
 			 */
-			if (folio_mapcount(folio) != 1) {
+			if (folio_mapcount(folio) != folio_nr_pages(folio)) {
 				folio_unlock(folio);
 				continue;
 			}
@@ -750,19 +764,8 @@ static int madvise_free_pte_range(pmd_t
 		}
 
 		if (pte_young(ptent) || pte_dirty(ptent)) {
-			/*
-			 * Some of architecture(ex, PPC) don't update TLB
-			 * with set_pte_at and tlb_remove_tlb_entry so for
-			 * the portability, remap the pte with old|clean
-			 * after pte clearing.
-			 */
-			ptent = ptep_get_and_clear_full(mm, addr, pte,
-							tlb->fullmm);
-
-			ptent = pte_mkold(ptent);
-			ptent = pte_mkclean(ptent);
-			set_pte_at(mm, addr, pte, ptent);
-			tlb_remove_tlb_entry(tlb, pte, addr);
+			clear_young_dirty_ptes(vma, addr, pte, nr, cydp_flags);
+			tlb_remove_tlb_entries(tlb, pte, nr, addr);
 		}
 		folio_mark_lazyfree(folio);
 	}
_

Patches currently in -mm which might be from ioworker0@gmail.com are

Re: [merged mm-stable] mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch removed from -mm tree

[David Hildenbrand]

On 06.05.24 02:57, Andrew Morton wrote:
> 
> The quilt patch titled
>       Subject: mm/madvise: optimize lazyfreeing with mTHP in madvise_free
> has been removed from the -mm tree.  Its filename was
>       mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch
> 
> This patch was dropped because it was merged into the mm-stable branch
> of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
> 

Andrew,

this series was moved to mm-stable a bit too fast for my taste.

Review is still going on, can we still remove that from mm-stable?

-- 
Cheers,

David / dhildenb

Re: [merged mm-stable] mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch removed from -mm tree

[Andrew Morton]

On Tue, 7 May 2024 10:15:24 +0200 David Hildenbrand <david@redhat.com> wrote:

> On 06.05.24 02:57, Andrew Morton wrote:
> > 
> > The quilt patch titled
> >       Subject: mm/madvise: optimize lazyfreeing with mTHP in madvise_free
> > has been removed from the -mm tree.  Its filename was
> >       mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch
> > 
> > This patch was dropped because it was merged into the mm-stable branch
> > of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
> > 
> 
> Andrew,
> 
> this series was moved to mm-stable a bit too fast for my taste.

It was nearly three weeks.

> Review is still going on, can we still remove that from mm-stable?

I can rebase if needed.  Is there a reason to do so?  Nothing that
can't be fixed up with followup patches?

Re: [merged mm-stable] mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch removed from -mm tree

[David Hildenbrand]

On 07.05.24 17:55, Andrew Morton wrote:
> On Tue, 7 May 2024 10:15:24 +0200 David Hildenbrand <david@redhat.com> wrote:
> 
>> On 06.05.24 02:57, Andrew Morton wrote:
>>>
>>> The quilt patch titled
>>>        Subject: mm/madvise: optimize lazyfreeing with mTHP in madvise_free
>>> has been removed from the -mm tree.  Its filename was
>>>        mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch
>>>
>>> This patch was dropped because it was merged into the mm-stable branch
>>> of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
>>>
>>
>> Andrew,
>>
>> this series was moved to mm-stable a bit too fast for my taste.
> 
> It was nearly three weeks.

I'm sorry, I replied to the wrong mail. I thought this was:

[1] https://lkml.kernel.org/r/20240501042700.83974-1-ioworker0@gmail.com

Because *that* is in mm-stable now as well. The major rewrite in v3 
(sent April 29) that had no reviews but broke compilation and was resend 
in v4 (send May 1), to be moved to mm-stable on May 6.

That's certainly too rushed for my taste for something that is not high 
provide just before the merge window is about to open.

I have it on my review list, but was expecting this to not get moved to 
mm-stable at this point. I won't be able to review it this/next week. I 
see other people are reviewing it now.

> 
>> Review is still going on, can we still remove that from mm-stable?
> 
> I can rebase if needed.  Is there a reason to do so?  Nothing that
> can't be fixed up with followup patches?

This here is good to go and saw sufficient review. [1] was too fast for 
my taste, but it's up to you to decide if we want to fix that up piece 
by piece as more review happens.

-- 
Cheers,

David / dhildenb

Re: [merged mm-stable] mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch removed from -mm tree

[Andrew Morton]

On Tue, 7 May 2024 18:54:09 +0200 David Hildenbrand <david@redhat.com> wrote:

> > 
> >> Review is still going on, can we still remove that from mm-stable?
> > 
> > I can rebase if needed.  Is there a reason to do so?  Nothing that
> > can't be fixed up with followup patches?
> 
> This here is good to go and saw sufficient review. [1] was too fast for 
> my taste, but it's up to you to decide if we want to fix that up piece 
> by piece as more review happens.
> 

OK.

Both series are cleanly droppable.  Maybe drop the later series
"Reclaim lazyfree THP without splitting" - let's see what the required
fixups look like?