Re: [PATCH v2 1/2] mm: use aligned address in clear_gigantic_page()

Huang, Ying
2024-10-31T08:43:42Z

Kefeng Wang <wangkefeng.wang@huawei.com> writes:

[snip]
>
>>> 1) Will test some rand test to check the different of performance as
>>> David suggested.
>>>
>>> 2) Hope the LKP to run more tests since it is very useful(more test
>>> set and different machines)
>> I'm starting to use LKP to test.
>
> Greet.

I have run some tests with LKP to test.

Firstly, there's almost no measurable difference between clearing pages
from start to end or from end to start on Intel server CPU. I guess
that there's some similar optimization for both direction.

For multiple processes (same as logical CPU number)
vm-scalability/anon-w-seq test case, the benchmark score increases
about 22.4%.

For multiple processes vm-scalability/anon-w-rand test case, no
measurable difference for benchmark score.

So, the optimization helps sequential workload mainly.

In summary, on x86, process_huge_page() will not introduce any
regression. And it helps some workload.

However, on ARM64, it does introduce some regression for clearing pages
from end to start. That needs to be addressed. I guess that the
regression can be resolved via using more clearing from start to end
(but not all). For example, can you take a look at the patch below?
Which uses the similar framework as before, but clear each small trunk
(mpage) from start to end. You can adjust MPAGE_NRPAGES to check when
the regression can be restored.

WARNING: the patch is only build tested.

Best Regards,
Huang, Ying

-----------------------------------8<----------------------------------------
From 406bcd1603987fdd7130d2df6f7d4aee4cc6b978 Mon Sep 17 00:00:00 2001
From: Huang Ying <ying.huang@intel.com>
Date: Thu, 31 Oct 2024 11:13:57 +0800
Subject: [PATCH] mpage clear

---
mm/memory.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 67 insertions(+), 3 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 3ccee51adfbb..1fdc548c4275 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -6769,6 +6769,68 @@ static inline int process_huge_page(
return 0;
}

+#define MPAGE_NRPAGES (1<<4)
+#define MPAGE_SIZE (PAGE_SIZE * MPAGE_NRPAGES)
+static inline int clear_huge_page(
+ unsigned long addr_hint, unsigned int nr_pages,
+ int (*process_subpage)(unsigned long addr, int idx, void *arg),
+ void *arg)
+{
+ int i, n, base, l, ret;
+ unsigned long addr = addr_hint &
+ ~(((unsigned long)nr_pages << PAGE_SHIFT) - 1);
+ unsigned long nr_mpages = ((unsigned long)nr_pages << PAGE_SHIFT) / MPAGE_SIZE;
+
+ /* Process target subpage last to keep its cache lines hot */
+ might_sleep();
+ n = (addr_hint - addr) / MPAGE_SIZE;
+ if (2 * n <= nr_mpages) {
+ /* If target subpage in first half of huge page */
+ base = 0;
+ l = n;
+ /* Process subpages at the end of huge page */
+ for (i = nr_mpages - 1; i >= 2 * n; i--) {
+ cond_resched();
+ ret = process_subpage(addr + i * MPAGE_SIZE,
+ i * MPAGE_NRPAGES, arg);
+ if (ret)
+ return ret;
+ }
+ } else {
+ /* If target subpage in second half of huge page */
+ base = nr_mpages - 2 * (nr_mpages - n);
+ l = nr_mpages - n;
+ /* Process subpages at the begin of huge page */
+ for (i = 0; i < base; i++) {
+ cond_resched();
+ ret = process_subpage(addr + i * MPAGE_SIZE,
+ i * MPAGE_NRPAGES, arg);
+ if (ret)
+ return ret;
+ }
+ }
+ /*
+ * Process remaining subpages in left-right-left-right pattern
+ * towards the target subpage
+ */
+ for (i = 0; i < l; i++) {
+ int left_idx = base + i;
+ int right_idx = base + 2 * l - 1 - i;
+
+ cond_resched();
+ ret = process_subpage(addr + left_idx * MPAGE_SIZE,
+ left_idx * MPAGE_NRPAGES, arg);
+ if (ret)
+ return ret;
+ cond_resched();
+ ret = process_subpage(addr + right_idx * MPAGE_SIZE,
+ right_idx * MPAGE_NRPAGES, arg);
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
static void clear_gigantic_page(struct folio *folio, unsigned long addr,
unsigned int nr_pages)
{
@@ -6784,8 +6846,10 @@ static void clear_gigantic_page(struct folio *folio, unsigned long addr,
static int clear_subpage(unsigned long addr, int idx, void *arg)
{
struct folio *folio = arg;
+ int i;

- clear_user_highpage(folio_page(folio, idx), addr);
+ for (i = 0; i < MPAGE_NRPAGES; i++)
+ clear_user_highpage(folio_page(folio, idx + i), addr + i * PAGE_SIZE);
return 0;
}

@@ -6798,10 +6862,10 @@ void folio_zero_user(struct folio *folio, unsigned long addr_hint)
{
unsigned int nr_pages = folio_nr_pages(folio);

- if (unlikely(nr_pages > MAX_ORDER_NR_PAGES))
+ if (unlikely(nr_pages != HPAGE_PMD_NR))
clear_gigantic_page(folio, addr_hint, nr_pages);
else
- process_huge_page(addr_hint, nr_pages, clear_subpage, folio);
+ clear_huge_page(addr_hint, nr_pages, clear_subpage, folio);
}

static int copy_user_gigantic_page(struct folio *dst, struct folio *src,
--
2.39.2