[PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

Unmapped pagecache pages can be demoted to low-tier memory, but 
they can only be promoted if a process maps the pages into the
memory space (so that NUMA hint faults can be caught).  This can
cause significant performance degradation as the pagecache ages
and unmapped, cached files are accessed.

This patch series enables the pagecache to request a promotion of
a folio when it is accessed via the pagecache.

We add a new `numa_hint_page_cache` counter in vmstat to capture
information on when these migrations occur.

Gregory Price (3):
  migrate: Allow migrate_misplaced_folio APIs without a VMA
  memory: allow non-fault migration in numa_migrate_prep path
  swap: enable promotion for unmapped pagecache on access

 include/linux/migrate.h       |  6 ++----
 include/linux/vm_event_item.h |  1 +
 mm/huge_memory.c              |  2 +-
 mm/memory.c                   | 21 +++++++++++----------
 mm/mempolicy.c                | 25 +++++++++++++++++--------
 mm/migrate.c                  |  7 +++----
 mm/swap.c                     | 20 ++++++++++++++++++++
 mm/vmstat.c                   |  1 +
 8 files changed, 56 insertions(+), 27 deletions(-)

-- 
2.43.0

[PATCH 1/3] migrate: Allow migrate_misplaced_folio APIs without a VMA

[Gregory Price]

To migrate unmapped pagecache folios, migrate_misplaced_folio and
migrate_misplaced_folio_prepare must handle folios without VMAs.

migrate_misplaced_folio_prepare checks VMA for exec bits, so allow
a NULL VMA when it does not have a mapping. migrate_misplaced_folio
does not require a VMA, remove it from the argument list.

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Gregory Price <gourry@gourry.net>
---
 include/linux/migrate.h | 6 ++----
 mm/huge_memory.c        | 2 +-
 mm/memory.c             | 2 +-
 mm/migrate.c            | 7 +++----
 4 files changed, 7 insertions(+), 10 deletions(-)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 644be30b69c8..553bdb4a65db 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -141,16 +141,14 @@ const struct movable_operations *page_movable_ops(struct page *page)
 #ifdef CONFIG_NUMA_BALANCING
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
-int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
-			   int node);
+int migrate_misplaced_folio(struct folio *folio, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
-static inline int migrate_misplaced_folio(struct folio *folio,
-					 struct vm_area_struct *vma, int node)
+static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index f4be468e06a4..f79c42e36d37 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1725,7 +1725,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	spin_unlock(vmf->ptl);
 	writable = false;
 
-	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
+	if (!migrate_misplaced_folio(folio, target_nid)) {
 		flags |= TNF_MIGRATED;
 		nid = target_nid;
 	} else {
diff --git a/mm/memory.c b/mm/memory.c
index 34f8402d2046..1d97bdfd0ed6 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5355,7 +5355,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	ignore_writable = true;
 
 	/* Migrate to the requested node */
-	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
+	if (!migrate_misplaced_folio(folio, target_nid)) {
 		nid = target_nid;
 		flags |= TNF_MIGRATED;
 	} else {
diff --git a/mm/migrate.c b/mm/migrate.c
index e7296c0fb5d5..c648dc08758b 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2546,7 +2546,7 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
 
 /*
  * Prepare for calling migrate_misplaced_folio() by isolating the folio if
- * permitted. Must be called with the PTL still held.
+ * permitted. If VMA is not NULL, Must be called with the PTL still held.
  */
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -2563,7 +2563,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		 * See folio_likely_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) &&
+		if (vma && (vma->vm_flags & VM_EXEC) &&
 		    folio_likely_mapped_shared(folio))
 			return -EACCES;
 
@@ -2614,8 +2614,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
  * elevated reference count on the folio. This function will un-isolate the
  * folio, dereferencing the folio before returning.
  */
-int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
-			    int node)
+int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	pg_data_t *pgdat = NODE_DATA(node);
 	int nr_remaining;
-- 
2.43.0

[PATCH 2/3] memory: allow non-fault migration in numa_migrate_prep path

[Gregory Price]

numa_migrate_prep and mpol_misplaced presume callers are in the
fault path.  To enable migrations from the pagecache, re-using
the same logic to handle migration prep would be preferable.

Mildly refactor numa_migrate_prep and mpol_misplaced so that it
may be called with (vmf=NULL) from non-faulting paths.

Count non-fault events as page-cache numa hints instead of
fault hints.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 include/linux/vm_event_item.h |  1 +
 mm/memory.c                   | 19 ++++++++++---------
 mm/mempolicy.c                | 25 +++++++++++++++++--------
 mm/vmstat.c                   |  1 +
 4 files changed, 29 insertions(+), 17 deletions(-)

diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 747943bc8cc2..b917bcfffe6d 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -62,6 +62,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		NUMA_HUGE_PTE_UPDATES,
 		NUMA_HINT_FAULTS,
 		NUMA_HINT_FAULTS_LOCAL,
+		NUMA_HINT_PAGE_CACHE,
 		NUMA_PAGE_MIGRATE,
 #endif
 #ifdef CONFIG_MIGRATION
diff --git a/mm/memory.c b/mm/memory.c
index 1d97bdfd0ed6..8b664b08915c 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5205,15 +5205,16 @@ static vm_fault_t do_fault(struct vm_fault *vmf)
 int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf,
 		      unsigned long addr, int page_nid, int *flags)
 {
-	struct vm_area_struct *vma = vmf->vma;
-
-	/* Record the current PID acceesing VMA */
-	vma_set_access_pid_bit(vma);
-
-	count_vm_numa_event(NUMA_HINT_FAULTS);
-	if (page_nid == numa_node_id()) {
-		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
-		*flags |= TNF_FAULT_LOCAL;
+	/* If accessed in fault path, record the current PID acceesing VMA */
+	if (vmf) {
+		vma_set_access_pid_bit(vmf->vma);
+		count_vm_numa_event(NUMA_HINT_FAULTS);
+		if (page_nid == numa_node_id()) {
+			count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
+			*flags |= TNF_FAULT_LOCAL;
+		}
+	} else {
+		count_vm_numa_event(NUMA_HINT_PAGE_CACHE);
 	}
 
 	return mpol_misplaced(folio, vmf, addr);
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index b858e22b259d..0f654aff477a 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2726,12 +2726,16 @@ static void sp_free(struct sp_node *n)
  * mpol_misplaced - check whether current folio node is valid in policy
  *
  * @folio: folio to be checked
- * @vmf: structure describing the fault
+ * @vmf: structure describing the fault (NULL if called outside fault path)
  * @addr: virtual address in @vma for shared policy lookup and interleave policy
+ *        Ignored if vmf is NULL.
  *
  * Lookup current policy node id for vma,addr and "compare to" folio's
- * node id.  Policy determination "mimics" alloc_page_vma().
- * Called from fault path where we know the vma and faulting address.
+ * node id - or task's policy node id if vmf is NULL.  Policy determination
+ * "mimics" alloc_page_vma()
+ *
+ * vmf must be non-NULL if called from fault path where we know the vma and
+ * faulting address. The PTL must be held by caller if vmf is not NULL.
  *
  * Return: NUMA_NO_NODE if the page is in a node that is valid for this
  * policy, or a suitable node ID to allocate a replacement folio from.
@@ -2743,7 +2747,6 @@ int mpol_misplaced(struct folio *folio, struct vm_fault *vmf,
 	pgoff_t ilx;
 	struct zoneref *z;
 	int curnid = folio_nid(folio);
-	struct vm_area_struct *vma = vmf->vma;
 	int thiscpu = raw_smp_processor_id();
 	int thisnid = numa_node_id();
 	int polnid = NUMA_NO_NODE;
@@ -2753,18 +2756,24 @@ int mpol_misplaced(struct folio *folio, struct vm_fault *vmf,
 	 * Make sure ptl is held so that we don't preempt and we
 	 * have a stable smp processor id
 	 */
-	lockdep_assert_held(vmf->ptl);
-	pol = get_vma_policy(vma, addr, folio_order(folio), &ilx);
+	if (vmf) {
+		lockdep_assert_held(vmf->ptl);
+		pol = get_vma_policy(vmf->vma, addr, folio_order(folio), &ilx);
+	} else
+		pol = get_task_policy(current);
+
 	if (!(pol->flags & MPOL_F_MOF))
 		goto out;
 
 	switch (pol->mode) {
 	case MPOL_INTERLEAVE:
-		polnid = interleave_nid(pol, ilx);
+		polnid = vmf ? interleave_nid(pol, ilx) :
+			       interleave_nodes(pol);
 		break;
 
 	case MPOL_WEIGHTED_INTERLEAVE:
-		polnid = weighted_interleave_nid(pol, ilx);
+		polnid = vmf ? weighted_interleave_nid(pol, ilx) :
+			       weighted_interleave_nodes(pol);
 		break;
 
 	case MPOL_PREFERRED:
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 04a1cb6cc636..5a02e6ff043d 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1316,6 +1316,7 @@ const char * const vmstat_text[] = {
 	"numa_huge_pte_updates",
 	"numa_hint_faults",
 	"numa_hint_faults_local",
+	"numa_hint_page_cache",
 	"numa_pages_migrated",
 #endif
 #ifdef CONFIG_MIGRATION
-- 
2.43.0

[PATCH 3/3] swap: enable promotion for unmapped pagecache on access

[Gregory Price]

Unmapped pagecache presently cannot be promoted off of lower tiers,
even if they are being accessed by read/write calls. Enable migration
in mark_folio_accessed if a given folio presently resides on a lower
tier of memory.

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Gregory Price <gourry@gourry.net>
---
 mm/swap.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/mm/swap.c b/mm/swap.c
index 9caf6b017cf0..08f76d748e29 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -37,6 +37,9 @@
 #include <linux/page_idle.h>
 #include <linux/local_lock.h>
 #include <linux/buffer_head.h>
+#include <linux/sched/sysctl.h>
+#include <linux/memory-tiers.h>
+#include <linux/migrate.h>
 
 #include "internal.h"
 
@@ -463,9 +466,26 @@ static void folio_inc_refs(struct folio *folio)
  *
  * When a newly allocated folio is not yet visible, so safe for non-atomic ops,
  * __folio_set_referenced() may be substituted for folio_mark_accessed().
+ *
+ * This call may also attempt to migrate the folio memory to the local node
+ * if it presently resides on a lower memory tier.
  */
 void folio_mark_accessed(struct folio *folio)
 {
+	int nid = folio_nid(folio);
+
+	/* If folio is on lower tier, try to promote to local node */
+	if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
+	    (nid == NUMA_NO_NODE || !node_is_toptier(nid))) {
+		int flags;
+
+		nid = numa_migrate_prep(folio, NULL, 0, nid, &flags);
+		if ((nid != NUMA_NO_NODE) &&
+		    !migrate_misplaced_folio_prepare(folio, NULL, nid)) {
+			migrate_misplaced_folio(folio, nid);
+		}
+	}
+
 	if (lru_gen_enabled()) {
 		folio_inc_refs(folio);
 		return;
-- 
2.43.0

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Andrew Morton]

On Sat,  3 Aug 2024 05:47:12 -0400 Gregory Price <gourry@gourry.net> wrote:

> Unmapped pagecache pages can be demoted to low-tier memory, but 
> they can only be promoted if a process maps the pages into the
> memory space (so that NUMA hint faults can be caught).  This can
> cause significant performance degradation as the pagecache ages
> and unmapped, cached files are accessed.

It would be helpful to share some testing results so the magnitude of
this degradation can be understood.

What is the potential downside to this change?  The local node now gets
stuffed full of pagecache and other things get evicted?

> This patch series enables the pagecache to request a promotion of
> a folio when it is accessed via the pagecache.
> 
> We add a new `numa_hint_page_cache` counter in vmstat to capture
> information on when these migrations occur.

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

On Thu, Aug 08, 2024 at 04:20:11PM -0700, Andrew Morton wrote:
> On Sat,  3 Aug 2024 05:47:12 -0400 Gregory Price <gourry@gourry.net> wrote:
> 
> > Unmapped pagecache pages can be demoted to low-tier memory, but 
> > they can only be promoted if a process maps the pages into the
> > memory space (so that NUMA hint faults can be caught).  This can
> > cause significant performance degradation as the pagecache ages
> > and unmapped, cached files are accessed.
> 
> It would be helpful to share some testing results so the magnitude of
> this degradation can be understood.

Apologies, this should have been an RFC - testing results forthcoming.

> 
> What is the potential downside to this change?  The local node now gets
> stuffed full of pagecache and other things get evicted?
> 

That is one possible degenerate case if there exists a large amount of
free memory in the local node.  We're testing it now against TPP demotion
logic, but the expectation should be that if the local node is already
pressured the pagecache would be trapped on CXL until TPP frees up local
node pages.

> > This patch series enables the pagecache to request a promotion of
> > a folio when it is accessed via the pagecache.
> > 
> > We add a new `numa_hint_page_cache` counter in vmstat to capture
> > information on when these migrations occur.

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

On Tue, Aug 13, 2024 at 11:04:59AM -0400, Gregory Price wrote:
> On Thu, Aug 08, 2024 at 04:20:11PM -0700, Andrew Morton wrote:
> > On Sat,  3 Aug 2024 05:47:12 -0400 Gregory Price <gourry@gourry.net> wrote:
> > 
> > > Unmapped pagecache pages can be demoted to low-tier memory, but 
> > > they can only be promoted if a process maps the pages into the
> > > memory space (so that NUMA hint faults can be caught).  This can
> > > cause significant performance degradation as the pagecache ages
> > > and unmapped, cached files are accessed.
> > 
> > It would be helpful to share some testing results so the magnitude of
> > this degradation can be understood.
> 
> Apologies, this should have been an RFC - testing results forthcoming.
> 
> > 
> > What is the potential downside to this change?  The local node now gets
> > stuffed full of pagecache and other things get evicted?
> > 
> 
> That is one possible degenerate case if there exists a large amount of
> free memory in the local node.  We're testing it now against TPP demotion
> logic, but the expectation should be that if the local node is already
> pressured the pagecache would be trapped on CXL until TPP frees up local
> node pages.
> 
> > > This patch series enables the pagecache to request a promotion of
> > > a folio when it is accessed via the pagecache.
> > > 
> > > We add a new `numa_hint_page_cache` counter in vmstat to capture
> > > information on when these migrations occur.

Worth noting for interested parties: This patch is not stable.  After some
extended testing, we find some soft locks.  So please disregard until v2+.

~Gregory

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Huang, Ying]

Gregory Price <gourry@gourry.net> writes:

> Unmapped pagecache pages can be demoted to low-tier memory, but 
> they can only be promoted if a process maps the pages into the
> memory space (so that NUMA hint faults can be caught).  This can
> cause significant performance degradation as the pagecache ages
> and unmapped, cached files are accessed.
>
> This patch series enables the pagecache to request a promotion of
> a folio when it is accessed via the pagecache.
>
> We add a new `numa_hint_page_cache` counter in vmstat to capture
> information on when these migrations occur.

It appears that you will promote page cache page on the second access.
Do you have some better way to identify hot pages from the not-so-hot
pages?  How to balance between unmapped and mapped pages?  We have hot
page selection for hot pages.

[snip]

--
Best Regards,
Huang, Ying

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

On Mon, Aug 19, 2024 at 03:46:00PM +0800, Huang, Ying wrote:
> Gregory Price <gourry@gourry.net> writes:
> 
> > Unmapped pagecache pages can be demoted to low-tier memory, but 
> > they can only be promoted if a process maps the pages into the
> > memory space (so that NUMA hint faults can be caught).  This can
> > cause significant performance degradation as the pagecache ages
> > and unmapped, cached files are accessed.
> >
> > This patch series enables the pagecache to request a promotion of
> > a folio when it is accessed via the pagecache.
> >
> > We add a new `numa_hint_page_cache` counter in vmstat to capture
> > information on when these migrations occur.
> 
> It appears that you will promote page cache page on the second access.
> Do you have some better way to identify hot pages from the not-so-hot
> pages?  How to balance between unmapped and mapped pages?  We have hot
> page selection for hot pages.
> 
> [snip]
> 

I've since explored moving this down under a (referenced && active) check.

This would be more like promotion on third access within an LRU shrink
round (the LRU should, in theory, hack off the active bits on some decent
time interval when the system is pressured).

Barring adding new counters to folios to track hits, I don't see a clear
and obvious way way to track hotness.  The primary observation here is 
that pagecache is un-mapped, and so cannot use numa-fault hints.

This is more complicated with MGLRU, but I'm saving that for after I
figure out the plan for plain old LRU.

~Gregory

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Huang, Ying]

Gregory Price <gourry@gourry.net> writes:

> On Mon, Aug 19, 2024 at 03:46:00PM +0800, Huang, Ying wrote:
>> Gregory Price <gourry@gourry.net> writes:
>> 
>> > Unmapped pagecache pages can be demoted to low-tier memory, but 
>> > they can only be promoted if a process maps the pages into the
>> > memory space (so that NUMA hint faults can be caught).  This can
>> > cause significant performance degradation as the pagecache ages
>> > and unmapped, cached files are accessed.
>> >
>> > This patch series enables the pagecache to request a promotion of
>> > a folio when it is accessed via the pagecache.
>> >
>> > We add a new `numa_hint_page_cache` counter in vmstat to capture
>> > information on when these migrations occur.
>> 
>> It appears that you will promote page cache page on the second access.
>> Do you have some better way to identify hot pages from the not-so-hot
>> pages?  How to balance between unmapped and mapped pages?  We have hot
>> page selection for hot pages.
>> 
>> [snip]
>> 
>
> I've since explored moving this down under a (referenced && active) check.
>
> This would be more like promotion on third access within an LRU shrink
> round (the LRU should, in theory, hack off the active bits on some decent
> time interval when the system is pressured).
>
> Barring adding new counters to folios to track hits, I don't see a clear
> and obvious way way to track hotness.  The primary observation here is 
> that pagecache is un-mapped, and so cannot use numa-fault hints.
>
> This is more complicated with MGLRU, but I'm saving that for after I
> figure out the plan for plain old LRU.

Several years ago, we have tried to use the access time tracking
mechanism of NUMA balancing to track the access time latency of unmapped
file cache folios.  The original implementation is as follows,

https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/commit/?h=tiering-0.8&id=5f2e64ce75c0322602c2ec8c70b64bb69b1f1329

What do you think about this?

--
Best Regards,
Huang, Ying

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

On Mon, Sep 02, 2024 at 02:53:26PM +0800, Huang, Ying wrote:
> Gregory Price <gourry@gourry.net> writes:
> 
> > On Mon, Aug 19, 2024 at 03:46:00PM +0800, Huang, Ying wrote:
> >> Gregory Price <gourry@gourry.net> writes:
> >> 
> >> > Unmapped pagecache pages can be demoted to low-tier memory, but 
> >> > they can only be promoted if a process maps the pages into the
> >> > memory space (so that NUMA hint faults can be caught).  This can
> >> > cause significant performance degradation as the pagecache ages
> >> > and unmapped, cached files are accessed.
> >> >
> >> > This patch series enables the pagecache to request a promotion of
> >> > a folio when it is accessed via the pagecache.
> >> >
> >> > We add a new `numa_hint_page_cache` counter in vmstat to capture
> >> > information on when these migrations occur.
> >> 
> >> It appears that you will promote page cache page on the second access.
> >> Do you have some better way to identify hot pages from the not-so-hot
> >> pages?  How to balance between unmapped and mapped pages?  We have hot
> >> page selection for hot pages.
> >> 
> >> [snip]
> >> 
> >
> > I've since explored moving this down under a (referenced && active) check.
> >
> > This would be more like promotion on third access within an LRU shrink
> > round (the LRU should, in theory, hack off the active bits on some decent
> > time interval when the system is pressured).
> >
> > Barring adding new counters to folios to track hits, I don't see a clear
> > and obvious way way to track hotness.  The primary observation here is 
> > that pagecache is un-mapped, and so cannot use numa-fault hints.
> >
> > This is more complicated with MGLRU, but I'm saving that for after I
> > figure out the plan for plain old LRU.
> 
> Several years ago, we have tried to use the access time tracking
> mechanism of NUMA balancing to track the access time latency of unmapped
> file cache folios.  The original implementation is as follows,
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/commit/?h=tiering-0.8&id=5f2e64ce75c0322602c2ec8c70b64bb69b1f1329
> 
> What do you think about this?
>

Also seems like an interesting option. I've been looking at another old
proposal to simply add a new LRU that was implemented by kbusch a few
years back.

https://git.kernel.org/pub/scm/linux/kernel/git/kbusch/linux.git/commit/?h=lru-promote&id=6616afe9a722f6ebedbb27ade3848cf07b9a3af7

I may spend a little time to add a few different methods in with a switch
I can flip to test them side by side / with each other and see what results
we can get.
 
> --
> Best Regards,
> Huang, Ying

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

On Mon, Sep 02, 2024 at 02:53:26PM +0800, Huang, Ying wrote:
> Gregory Price <gourry@gourry.net> writes:
> 
> > On Mon, Aug 19, 2024 at 03:46:00PM +0800, Huang, Ying wrote:
> >> Gregory Price <gourry@gourry.net> writes:
> >> 
> >> > Unmapped pagecache pages can be demoted to low-tier memory, but 
> >> > they can only be promoted if a process maps the pages into the
> >> > memory space (so that NUMA hint faults can be caught).  This can
> >> > cause significant performance degradation as the pagecache ages
> >> > and unmapped, cached files are accessed.
> >> >
> >> > This patch series enables the pagecache to request a promotion of
> >> > a folio when it is accessed via the pagecache.
> >> >
> >> > We add a new `numa_hint_page_cache` counter in vmstat to capture
> >> > information on when these migrations occur.
> >> 
> >> It appears that you will promote page cache page on the second access.
> >> Do you have some better way to identify hot pages from the not-so-hot
> >> pages?  How to balance between unmapped and mapped pages?  We have hot
> >> page selection for hot pages.
> >> 
> >> [snip]
> >> 
> >
> > I've since explored moving this down under a (referenced && active) check.
> >
> > This would be more like promotion on third access within an LRU shrink
> > round (the LRU should, in theory, hack off the active bits on some decent
> > time interval when the system is pressured).
> >
> > Barring adding new counters to folios to track hits, I don't see a clear
> > and obvious way way to track hotness.  The primary observation here is 
> > that pagecache is un-mapped, and so cannot use numa-fault hints.
> >
> > This is more complicated with MGLRU, but I'm saving that for after I
> > figure out the plan for plain old LRU.
> 
> Several years ago, we have tried to use the access time tracking
> mechanism of NUMA balancing to track the access time latency of unmapped
> file cache folios.  The original implementation is as follows,
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/commit/?h=tiering-0.8&id=5f2e64ce75c0322602c2ec8c70b64bb69b1f1329
> 
> What do you think about this?
> 

Coming back around to explore this topic a bit more, dug into this old
patch and the LRU patch by Keith - I'm struggling find a good option
that doesn't over-complicate or propose something contentious.


I did a browse through lore and did not see any discussion on this patch
or on Keith's LRU patch, so i presume discussion on this happened largely
off-list.  So if you have any context as to why this wasn't RFC'd officially
I would like more information.


My observations between these 3 proposals:

- The page-lock state is complex while trying interpose in mark_folio_accessed,
  meaning inline promotion inside that interface is a non-starter.

  We found one deadlock during task exit due to the PTL being held. 

  This worries me more generally, but we did find some success changing certain
  calls to mark_folio_accessed to mark_folio_accessed_and_promote - rather than
  modifying mark_folio_accessed. This ends up changing code in similar places
  to your hook - but catches a more conditions that mark a page accessed.

- For Keith's proposal, promotions via LRU requires memory pressure on the lower
  tier to cause a shrink and therefore promotions. I'm not well versed in LRU
  LRU sematics, but it seems we could try proactive reclaim here.
  
  Doing promote-reclaim and demote/swap/evict reclaim on the same triggers
  seems counter-intuitive.

- Doing promotions inline with access creates overhead.  I've seen some research
  suggesting 60us+ per migration - so aggressiveness could harm performance.

  Doing it async would alleviate inline access overheads - but it could also make
  promotion pointless if time-to-promote is to far from liveliness of the pages.

- Doing async-promotion may also require something like PG_PROMOTABLE (as proposed
  by Keith's patch), which will obviously be a very contentious topic.


tl;dr: I'm learning towards a solution like you have here, but we may need to
make a sysfs switch similar to demotion_enabled in case of poor performance due
to heuristically degenerate access patterns, and we may need to expose some
form of adjustable aggressiveness value to make it tunable.

Reading more into the code surrounding this and other migration logic, I also
think we should explore an optimization to mempolicy that tries to aggressively
keep certain classes of memory on the local node (RX memory and stack for example).

Other areas of reclaim try to actively prevent demoting this type of memory, so we
should try not to allocate it there in the first place.

~Gregory

> --
> Best Regards,
> Huang, Ying

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Huang, Ying]

Hi, Gregory,

Gregory Price <gourry@gourry.net> writes:

> On Mon, Sep 02, 2024 at 02:53:26PM +0800, Huang, Ying wrote:
>> Gregory Price <gourry@gourry.net> writes:
>> 
>> > On Mon, Aug 19, 2024 at 03:46:00PM +0800, Huang, Ying wrote:
>> >> Gregory Price <gourry@gourry.net> writes:
>> >> 
>> >> > Unmapped pagecache pages can be demoted to low-tier memory, but 
>> >> > they can only be promoted if a process maps the pages into the
>> >> > memory space (so that NUMA hint faults can be caught).  This can
>> >> > cause significant performance degradation as the pagecache ages
>> >> > and unmapped, cached files are accessed.
>> >> >
>> >> > This patch series enables the pagecache to request a promotion of
>> >> > a folio when it is accessed via the pagecache.
>> >> >
>> >> > We add a new `numa_hint_page_cache` counter in vmstat to capture
>> >> > information on when these migrations occur.
>> >> 
>> >> It appears that you will promote page cache page on the second access.
>> >> Do you have some better way to identify hot pages from the not-so-hot
>> >> pages?  How to balance between unmapped and mapped pages?  We have hot
>> >> page selection for hot pages.
>> >> 
>> >> [snip]
>> >> 
>> >
>> > I've since explored moving this down under a (referenced && active) check.
>> >
>> > This would be more like promotion on third access within an LRU shrink
>> > round (the LRU should, in theory, hack off the active bits on some decent
>> > time interval when the system is pressured).
>> >
>> > Barring adding new counters to folios to track hits, I don't see a clear
>> > and obvious way way to track hotness.  The primary observation here is 
>> > that pagecache is un-mapped, and so cannot use numa-fault hints.
>> >
>> > This is more complicated with MGLRU, but I'm saving that for after I
>> > figure out the plan for plain old LRU.
>> 
>> Several years ago, we have tried to use the access time tracking
>> mechanism of NUMA balancing to track the access time latency of unmapped
>> file cache folios.  The original implementation is as follows,
>> 
>> https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/commit/?h=tiering-0.8&id=5f2e64ce75c0322602c2ec8c70b64bb69b1f1329
>> 
>> What do you think about this?
>> 
>
> Coming back around to explore this topic a bit more, dug into this old
> patch and the LRU patch by Keith - I'm struggling find a good option
> that doesn't over-complicate or propose something contentious.
>
>
> I did a browse through lore and did not see any discussion on this patch
> or on Keith's LRU patch, so i presume discussion on this happened largely
> off-list.  So if you have any context as to why this wasn't RFC'd officially
> I would like more information.

Thanks for doing this.  There's no much discussion offline.  We just
don't have enough time to work on the solution.

> My observations between these 3 proposals:
>
> - The page-lock state is complex while trying interpose in mark_folio_accessed,
>   meaning inline promotion inside that interface is a non-starter.
>
>   We found one deadlock during task exit due to the PTL being held. 
>
>   This worries me more generally, but we did find some success changing certain
>   calls to mark_folio_accessed to mark_folio_accessed_and_promote - rather than
>   modifying mark_folio_accessed. This ends up changing code in similar places
>   to your hook - but catches a more conditions that mark a page accessed.
>
> - For Keith's proposal, promotions via LRU requires memory pressure on the lower
>   tier to cause a shrink and therefore promotions. I'm not well versed in LRU
>   LRU sematics, but it seems we could try proactive reclaim here.
>   
>   Doing promote-reclaim and demote/swap/evict reclaim on the same triggers
>   seems counter-intuitive.

IIUC, in TPP paper (https://arxiv.org/abs/2206.02878), a similar method
is proposed for page promoting.  I guess that it works together with
proactive reclaiming.

> - Doing promotions inline with access creates overhead.  I've seen some research
>   suggesting 60us+ per migration - so aggressiveness could harm performance.
>
>   Doing it async would alleviate inline access overheads - but it could also make
>   promotion pointless if time-to-promote is to far from liveliness of the pages.

Async promotion needs to deal with the resource (CPU/memory) charging
too.  You do some work for a task, so you need to charge the consumed
resource for the task.

> - Doing async-promotion may also require something like PG_PROMOTABLE (as proposed
>   by Keith's patch), which will obviously be a very contentious topic.

Some additional data structure can be used to record pages.

> tl;dr: I'm learning towards a solution like you have here, but we may need to
> make a sysfs switch similar to demotion_enabled in case of poor performance due
> to heuristically degenerate access patterns, and we may need to expose some
> form of adjustable aggressiveness value to make it tunable.

Yes.  We may need that, because the performance benefit may be lower
than the overhead introduced.

> Reading more into the code surrounding this and other migration logic, I also
> think we should explore an optimization to mempolicy that tries to aggressively
> keep certain classes of memory on the local node (RX memory and stack
> for example).
>
> Other areas of reclaim try to actively prevent demoting this type of memory, so we
> should try not to allocate it there in the first place.

We have already used DRAM first allocation policy.  So, we need to
measure its effect firstly.

--
Best Regards,
Huang, Ying

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

On Tue, Nov 05, 2024 at 10:00:59AM +0800, Huang, Ying wrote:
> Hi, Gregory,
> 
> Gregory Price <gourry@gourry.net> writes:
> 
> > My observations between these 3 proposals:
> >
> > - The page-lock state is complex while trying interpose in mark_folio_accessed,
> >   meaning inline promotion inside that interface is a non-starter.
> >
> >   We found one deadlock during task exit due to the PTL being held. 
> >
> >   This worries me more generally, but we did find some success changing certain
> >   calls to mark_folio_accessed to mark_folio_accessed_and_promote - rather than
> >   modifying mark_folio_accessed. This ends up changing code in similar places
> >   to your hook - but catches a more conditions that mark a page accessed.
> >
> > - For Keith's proposal, promotions via LRU requires memory pressure on the lower
> >   tier to cause a shrink and therefore promotions. I'm not well versed in LRU
> >   LRU sematics, but it seems we could try proactive reclaim here.
> >   
> >   Doing promote-reclaim and demote/swap/evict reclaim on the same triggers
> >   seems counter-intuitive.
> 
> IIUC, in TPP paper (https://arxiv.org/abs/2206.02878), a similar method
> is proposed for page promoting.  I guess that it works together with
> proactive reclaiming.
> 

Each process is responsible for doing page table scanning for numa hint faults
and producing a promotion.  Since the structure used there is the page tables
themselves, there isn't an existing recording mechanism for us to piggy-back on
to defer migrations to later.

> > - Doing promotions inline with access creates overhead.  I've seen some research
> >   suggesting 60us+ per migration - so aggressiveness could harm performance.
> >
> >   Doing it async would alleviate inline access overheads - but it could also make
> >   promotion pointless if time-to-promote is to far from liveliness of the pages.
> 
> Async promotion needs to deal with the resource (CPU/memory) charging
> too.  You do some work for a task, so you need to charge the consumed
> resource for the task.
> 

This is a good point, and would heavily complicate things. Simple is better,
let's avoid that.

> > - Doing async-promotion may also require something like PG_PROMOTABLE (as proposed
> >   by Keith's patch), which will obviously be a very contentious topic.
> 
> Some additional data structure can be used to record pages.
> 

I have an idea inspired by these three sets, i'll bumble my way through a prototype.

> > Reading more into the code surrounding this and other migration logic, I also
> > think we should explore an optimization to mempolicy that tries to aggressively
> > keep certain classes of memory on the local node (RX memory and stack
> > for example).
> >
> > Other areas of reclaim try to actively prevent demoting this type of memory, so we
> > should try not to allocate it there in the first place.
> 
> We have already used DRAM first allocation policy.  So, we need to
> measure its effect firstly.
> 

Yes, but also as the weighted interleave patch set demonstrated, it can be beneficial
to change this to distribute allocations from the outset - however, distributing all
allocations lead to less reliable performance than just distributing the heap.

Another topic for another thread.
~Gregory

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

On Tue, Nov 05, 2024 at 10:00:59AM +0800, Huang, Ying wrote:
> Hi, Gregory,
> >> 
> >> Several years ago, we have tried to use the access time tracking
> >> mechanism of NUMA balancing to track the access time latency of unmapped
> >> file cache folios.  The original implementation is as follows,
> >> 
> >> https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/commit/?h=tiering-0.8&id=5f2e64ce75c0322602c2ec8c70b64bb69b1f1329
> >> 
> >> What do you think about this?
> >> 
> >
> > Coming back around to explore this topic a bit more, dug into this old
> > patch and the LRU patch by Keith - I'm struggling find a good option
> > that doesn't over-complicate or propose something contentious.
> >
> >
> > I did a browse through lore and did not see any discussion on this patch
> > or on Keith's LRU patch, so i presume discussion on this happened largely
> > off-list.  So if you have any context as to why this wasn't RFC'd officially
> > I would like more information.
> 
> Thanks for doing this.  There's no much discussion offline.  We just
> don't have enough time to work on the solution.
> 

Exploring and testing this a little further, I brought this up to current
folio work in 6.9 and found this solution to be unstable as-is.

After some work to fix lock/reference issues, Johannes pointed out that
__filemap_get_folio can be called from an atomic context - which means it
may not be safe to do migrations in this context.

We're back to looking at something like an LRU-esque system, but now we're
thinking about isolating the folios in folio_mark_accessed into a task-local
list, and then process the list on resume.

Basically we're thinking

1) hook folio_mark_accessed and use PG_ACTIVE/PG_ACCESSED to determine whether
   the page is a promotion candidate.
2) if it is, isolate it from the LRU - which is safe because folio_mark_accessed
   already does this elsewhere, and place it onto current->promo_queue
3) set_notify_resume
4) add logic to resume_user_mode_work() to run through current->promo_queue and
   either promote the pages accordingly, or do folio_putback_lru on failure.

Going to RFC this up

~Gregory

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Huang, Ying]

Gregory Price <gourry@gourry.net> writes:

> On Tue, Nov 05, 2024 at 10:00:59AM +0800, Huang, Ying wrote:
>> Hi, Gregory,
>> >> 
>> >> Several years ago, we have tried to use the access time tracking
>> >> mechanism of NUMA balancing to track the access time latency of unmapped
>> >> file cache folios.  The original implementation is as follows,
>> >> 
>> >> https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/commit/?h=tiering-0.8&id=5f2e64ce75c0322602c2ec8c70b64bb69b1f1329
>> >> 
>> >> What do you think about this?
>> >> 
>> >
>> > Coming back around to explore this topic a bit more, dug into this old
>> > patch and the LRU patch by Keith - I'm struggling find a good option
>> > that doesn't over-complicate or propose something contentious.
>> >
>> >
>> > I did a browse through lore and did not see any discussion on this patch
>> > or on Keith's LRU patch, so i presume discussion on this happened largely
>> > off-list.  So if you have any context as to why this wasn't RFC'd officially
>> > I would like more information.
>> 
>> Thanks for doing this.  There's no much discussion offline.  We just
>> don't have enough time to work on the solution.
>> 
>
> Exploring and testing this a little further, I brought this up to current
> folio work in 6.9 and found this solution to be unstable as-is.
>
> After some work to fix lock/reference issues, Johannes pointed out that
> __filemap_get_folio can be called from an atomic context - which means it
> may not be safe to do migrations in this context.

Sorry, I don't understand this, the above patch changes
filemap_get_pages() and grab_cache_page_write_begin() instead of
__filemap_get_folio().

> We're back to looking at something like an LRU-esque system, but now we're
> thinking about isolating the folios in folio_mark_accessed into a task-local
> list, and then process the list on resume.

If necessary, we can use a similar method for above solution too.  And
we can filter accessed once folios with folio_mark_accessed() firstly.
That is, only promote a page if,

- record the folio access time in folio_mark_accessed() only
- when the folio are accessed again, and "access_time - record_time <
  threshold", promote the folio.

> Basically we're thinking
>
> 1) hook folio_mark_accessed and use PG_ACTIVE/PG_ACCESSED to determine whether
>    the page is a promotion candidate.
> 2) if it is, isolate it from the LRU - which is safe because folio_mark_accessed
>    already does this elsewhere, and place it onto current->promo_queue
> 3) set_notify_resume
> 4) add logic to resume_user_mode_work() to run through current->promo_queue and
>    either promote the pages accordingly, or do folio_putback_lru on failure.

Use a task_work?

> Going to RFC this up

--
Best Regards,
Huang, Ying

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Gregory Price]

On Mon, Nov 11, 2024 at 09:35:09AM +0800, Huang, Ying wrote:
> Gregory Price <gourry@gourry.net> writes:
> 
> >
> > Exploring and testing this a little further, I brought this up to current
> > folio work in 6.9 and found this solution to be unstable as-is.
> >
> > After some work to fix lock/reference issues, Johannes pointed out that
> > __filemap_get_folio can be called from an atomic context - which means it
> > may not be safe to do migrations in this context.
> 
> Sorry, I don't understand this, the above patch changes
> filemap_get_pages() and grab_cache_page_write_begin() instead of
> __filemap_get_folio().
>

on newer kernels, grab_cache_page_write_begin is a compat wrapper for
__filemap_get_folio and folio_file_page.  This chunk of code has changed
somewhat significantly, actually.
 
> > We're back to looking at something like an LRU-esque system, but now we're
> > thinking about isolating the folios in folio_mark_accessed into a task-local
> > list, and then process the list on resume.
> 
> If necessary, we can use a similar method for above solution too.  And
> we can filter accessed once folios with folio_mark_accessed() firstly.
> That is, only promote a page if,
> 
> - record the folio access time in folio_mark_accessed() only
> - when the folio are accessed again, and "access_time - record_time <
>   threshold", promote the folio.
> 

yes this was the thought.

> > Basically we're thinking
> >
> > 1) hook folio_mark_accessed and use PG_ACTIVE/PG_ACCESSED to determine whether
> >    the page is a promotion candidate.
> > 2) if it is, isolate it from the LRU - which is safe because folio_mark_accessed
> >    already does this elsewhere, and place it onto current->promo_queue
> > 3) set_notify_resume
> > 4) add logic to resume_user_mode_work() to run through current->promo_queue and
> >    either promote the pages accordingly, or do folio_putback_lru on failure.
> 
> Use a task_work?
> 

probably more correct, had a discussion about kernel threads accessing
file cache and we weren't sure if that situation even existed - so probably
going to try task_work first.

~Gregory

Re: [PATCH 0/3] mm,TPP: Enable promotion of unmapped pagecache

[Huang, Ying]

Gregory Price <gourry@gourry.net> writes:

> On Mon, Nov 11, 2024 at 09:35:09AM +0800, Huang, Ying wrote:
>> Gregory Price <gourry@gourry.net> writes:
>> 
>> >
>> > Exploring and testing this a little further, I brought this up to current
>> > folio work in 6.9 and found this solution to be unstable as-is.
>> >
>> > After some work to fix lock/reference issues, Johannes pointed out that
>> > __filemap_get_folio can be called from an atomic context - which means it
>> > may not be safe to do migrations in this context.
>> 
>> Sorry, I don't understand this, the above patch changes
>> filemap_get_pages() and grab_cache_page_write_begin() instead of
>> __filemap_get_folio().
>>
>
> on newer kernels, grab_cache_page_write_begin is a compat wrapper for
> __filemap_get_folio and folio_file_page.  This chunk of code has changed
> somewhat significantly, actually.
>  
>> > We're back to looking at something like an LRU-esque system, but now we're
>> > thinking about isolating the folios in folio_mark_accessed into a task-local
>> > list, and then process the list on resume.
>> 
>> If necessary, we can use a similar method for above solution too.  And
>> we can filter accessed once folios with folio_mark_accessed() firstly.
>> That is, only promote a page if,
>> 
>> - record the folio access time in folio_mark_accessed() only
>> - when the folio are accessed again, and "access_time - record_time <
>>   threshold", promote the folio.
>> 
>
> yes this was the thought.
>
>> > Basically we're thinking
>> >
>> > 1) hook folio_mark_accessed and use PG_ACTIVE/PG_ACCESSED to determine whether
>> >    the page is a promotion candidate.
>> > 2) if it is, isolate it from the LRU - which is safe because folio_mark_accessed
>> >    already does this elsewhere, and place it onto current->promo_queue
>> > 3) set_notify_resume
>> > 4) add logic to resume_user_mode_work() to run through current->promo_queue and
>> >    either promote the pages accordingly, or do folio_putback_lru on failure.
>> 
>> Use a task_work?
>> 
>
> probably more correct, had a discussion about kernel threads accessing
> file cache and we weren't sure if that situation even existed - so probably

We can ignore kthread when collecting promoting candidates folios.

> going to try task_work first.

--
Best Regards,
Huang, Ying