* [PATCH v2 0/1] mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()
@ 2023-09-21 12:02 Jiexun Wang
[not found] ` <85363861af65fac66c7a98c251906afc0d9c8098.1695291046.git.wangjiexun@tinylab.org>
0 siblings, 1 reply; 6+ messages in thread
From: Jiexun Wang @ 2023-09-21 12:02 UTC (permalink / raw)
To: akpm, brauner; +Cc: linux-mm, linux-kernel, falcon, tangjinyu, Jiexun Wang
I conducted real-time testing and observed that
madvise_cold_or_pageout_pte_range() causes significant
latency under memory pressure, which can be effectively
reduced by adding cond_resched() within the loop.
I tested on the LicheePi 4A board using Cylictest for latency
testing and Ftrace for latency tracing. The board uses TH1520
processor and has a memory size of 8GB. The kernel version is 6.5.0
with the PREEMPT_RT patch applied.
The script I tested is as follows:
echo wakeup_rt > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/tracing_on
echo 0 > /sys/kernel/tracing/tracing_max_latency
stress-ng --vm 8 --vm-bytes 2G &
cyclictest --mlockall --smp --priority=99 --distance=0 --duration=30m
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace
The tracing results before modification are as follows:
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 6.5.0-rt6-r1208-00003-g999d221864bf
# --------------------------------------------------------------------
# latency: 2552 us, #6/6, CPU#3 | (M:preempt_rt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: cyclictest-196 (uid:0 nice:0 policy:1 rt_prio:99)
# -----------------
#
# _--------=> CPU#
# / _-------=> irqs-off/BH-disabled
# | / _------=> need-resched
# || / _-----=> need-resched-lazy
# ||| / _----=> hardirq/softirq
# |||| / _---=> preempt-depth
# ||||| / _--=> preempt-lazy-depth
# |||||| / _-=> migrate-disable
# ||||||| / delay
# cmd pid |||||||| time | caller
# \ / |||||||| \ | /
stress-n-206 3dn.h512 2us : 206:120:R + [003] 196: 0:R cyclictest
stress-n-206 3dn.h512 7us : <stack trace>
=> __ftrace_trace_stack
=> __trace_stack
=> probe_wakeup
=> ttwu_do_activate
=> try_to_wake_up
=> wake_up_process
=> hrtimer_wakeup
=> __hrtimer_run_queues
=> hrtimer_interrupt
=> riscv_timer_interrupt
=> handle_percpu_devid_irq
=> generic_handle_domain_irq
=> riscv_intc_irq
=> handle_riscv_irq
=> do_irq
stress-n-206 3dn.h512 9us#: 0
stress-n-206 3d...3.. 2544us : __schedule
stress-n-206 3d...3.. 2545us : 206:120:R ==> [003] 196: 0:R cyclictest
stress-n-206 3d...3.. 2551us : <stack trace>
=> __ftrace_trace_stack
=> __trace_stack
=> probe_wakeup_sched_switch
=> __schedule
=> preempt_schedule
=> migrate_enable
=> rt_spin_unlock
=> madvise_cold_or_pageout_pte_range
=> walk_pgd_range
=> __walk_page_range
=> walk_page_range
=> madvise_pageout
=> madvise_vma_behavior
=> do_madvise
=> sys_madvise
=> do_trap_ecall_u
=> ret_from_exception
The tracing results after modification are as follows:
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 6.5.0-rt6-r1208-00004-gca3876fc69a6-dirty
# --------------------------------------------------------------------
# latency: 1689 us, #6/6, CPU#0 | (M:preempt_rt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: cyclictest-217 (uid:0 nice:0 policy:1 rt_prio:99)
# -----------------
#
# _--------=> CPU#
# / _-------=> irqs-off/BH-disabled
# | / _------=> need-resched
# || / _-----=> need-resched-lazy
# ||| / _----=> hardirq/softirq
# |||| / _---=> preempt-depth
# ||||| / _--=> preempt-lazy-depth
# |||||| / _-=> migrate-disable
# ||||||| / delay
# cmd pid |||||||| time | caller
# \ / |||||||| \ | /
stress-n-232 0dn.h413 1us+: 232:120:R + [000] 217: 0:R cyclictest
stress-n-232 0dn.h413 12us : <stack trace>
=> __ftrace_trace_stack
=> __trace_stack
=> probe_wakeup
=> ttwu_do_activate
=> try_to_wake_up
=> wake_up_process
=> hrtimer_wakeup
=> __hrtimer_run_queues
=> hrtimer_interrupt
=> riscv_timer_interrupt
=> handle_percpu_devid_irq
=> generic_handle_domain_irq
=> riscv_intc_irq
=> handle_riscv_irq
=> do_irq
stress-n-232 0dn.h413 19us#: 0
stress-n-232 0d...3.. 1671us : __schedule
stress-n-232 0d...3.. 1676us+: 232:120:R ==> [000] 217: 0:R cyclictest
stress-n-232 0d...3.. 1687us : <stack trace>
=> __ftrace_trace_stack
=> __trace_stack
=> probe_wakeup_sched_switch
=> __schedule
=> preempt_schedule
=> migrate_enable
=> free_unref_page_list
=> release_pages
=> free_pages_and_swap_cache
=> tlb_batch_pages_flush
=> tlb_flush_mmu
=> unmap_page_range
=> unmap_vmas
=> unmap_region
=> do_vmi_align_munmap.constprop.0
=> do_vmi_munmap
=> __vm_munmap
=> sys_munmap
=> do_trap_ecall_u
=> ret_from_exception
After the modification, the cause of maximum latency is no longer
madvise_cold_or_pageout_pte_range(), so this modification can
reduce the latency caused by madvise_cold_or_pageout_pte_range().
Changes in v2:
- re-enter the loop after the cond_resched() method
Jiexun Wang (1):
mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()
mm/madvise.c | 11 +++++++++++
1 file changed, 11 insertions(+)
--
2.34.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/1] mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()
[not found] ` <85363861af65fac66c7a98c251906afc0d9c8098.1695291046.git.wangjiexun@tinylab.org>
@ 2023-09-22 0:41 ` Andrew Morton
2024-01-26 3:12 ` Sergey Senozhatsky
2024-01-26 3:25 ` [PATCH] mm/madvise: don't forget to leave lazy MMU mode " Sergey Senozhatsky
0 siblings, 2 replies; 6+ messages in thread
From: Andrew Morton @ 2023-09-22 0:41 UTC (permalink / raw)
To: Jiexun Wang; +Cc: brauner, falcon, linux-kernel, linux-mm, tangjinyu
On Thu, 21 Sep 2023 20:27:51 +0800 Jiexun Wang <wangjiexun@tinylab.org> wrote:
> Currently the madvise_cold_or_pageout_pte_range() function exhibits
> significant latency under memory pressure, which can be effectively
> reduced by adding cond_resched() within the loop.
>
> When the batch_count reaches SWAP_CLUSTER_MAX, we reschedule
> the task to ensure fairness and avoid long lock holding times.
>
> ...
>
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -354,6 +354,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> struct folio *folio = NULL;
> LIST_HEAD(folio_list);
> bool pageout_anon_only_filter;
> + unsigned int batch_count = 0;
>
> if (fatal_signal_pending(current))
> return -EINTR;
> @@ -433,6 +434,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> regular_folio:
> #endif
> tlb_change_page_size(tlb, PAGE_SIZE);
> +restart:
> start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
The handling of start_pte looks OK.
> if (!start_pte)
> return 0;
> @@ -441,6 +443,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> for (; addr < end; pte++, addr += PAGE_SIZE) {
> ptent = ptep_get(pte);
>
> + if (++batch_count == SWAP_CLUSTER_MAX) {
> + batch_count = 0;
> + if (need_resched()) {
> + pte_unmap_unlock(start_pte, ptl);
> + cond_resched();
> + goto restart;
> + }
> + }
> +
> if (pte_none(ptent))
> continue;
>
I think this patch looks OK, but would appreciate careful review from
others, please.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/1] mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()
2023-09-22 0:41 ` [PATCH v2 1/1] " Andrew Morton
@ 2024-01-26 3:12 ` Sergey Senozhatsky
2024-01-26 3:25 ` [PATCH] mm/madvise: don't forget to leave lazy MMU mode " Sergey Senozhatsky
1 sibling, 0 replies; 6+ messages in thread
From: Sergey Senozhatsky @ 2024-01-26 3:12 UTC (permalink / raw)
To: Andrew Morton, Jiexun Wang
Cc: brauner, falcon, linux-kernel, linux-mm, tangjinyu
On (23/09/21 17:41), Andrew Morton wrote:
> > Currently the madvise_cold_or_pageout_pte_range() function exhibits
> > significant latency under memory pressure, which can be effectively
> > reduced by adding cond_resched() within the loop.
> >
> > When the batch_count reaches SWAP_CLUSTER_MAX, we reschedule
> > the task to ensure fairness and avoid long lock holding times.
> >
> > ...
> >
> > --- a/mm/madvise.c
> > +++ b/mm/madvise.c
> > @@ -354,6 +354,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> > struct folio *folio = NULL;
> > LIST_HEAD(folio_list);
> > bool pageout_anon_only_filter;
> > + unsigned int batch_count = 0;
> >
> > if (fatal_signal_pending(current))
> > return -EINTR;
> > @@ -433,6 +434,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> > regular_folio:
> > #endif
> > tlb_change_page_size(tlb, PAGE_SIZE);
> > +restart:
> > start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
>
> The handling of start_pte looks OK.
>
> > if (!start_pte)
> > return 0;
> > @@ -441,6 +443,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> > for (; addr < end; pte++, addr += PAGE_SIZE) {
> > ptent = ptep_get(pte);
> >
> > + if (++batch_count == SWAP_CLUSTER_MAX) {
> > + batch_count = 0;
> > + if (need_resched()) {
> > + pte_unmap_unlock(start_pte, ptl);
Shouldn't it leave lazy MMU mode here?
---
diff --git a/mm/madvise.c b/mm/madvise.c
index 0f222d464254..127f0c7b69ac 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -451,6 +451,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
if (++batch_count == SWAP_CLUSTER_MAX) {
batch_count = 0;
if (need_resched()) {
+ arch_leave_lazy_mmu_mode();
pte_unmap_unlock(start_pte, ptl);
cond_resched();
goto restart;
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH] mm/madvise: don't forget to leave lazy MMU mode in madvise_cold_or_pageout_pte_range()
2023-09-22 0:41 ` [PATCH v2 1/1] " Andrew Morton
2024-01-26 3:12 ` Sergey Senozhatsky
@ 2024-01-26 3:25 ` Sergey Senozhatsky
2024-01-26 6:53 ` Andrew Morton
1 sibling, 1 reply; 6+ messages in thread
From: Sergey Senozhatsky @ 2024-01-26 3:25 UTC (permalink / raw)
To: Andrew Morton; +Cc: Jiexun Wang, linux-kernel, linux-mm, Sergey Senozhatsky
We need to leave lazy MMU mode before unlocking.
Fixes: b2f557a21bc8 ("mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()")
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
mm/madvise.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/mm/madvise.c b/mm/madvise.c
index 0f222d464254..127f0c7b69ac 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -451,6 +451,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
if (++batch_count == SWAP_CLUSTER_MAX) {
batch_count = 0;
if (need_resched()) {
+ arch_leave_lazy_mmu_mode();
pte_unmap_unlock(start_pte, ptl);
cond_resched();
goto restart;
--
2.43.0.429.g432eaa2c6b-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] mm/madvise: don't forget to leave lazy MMU mode in madvise_cold_or_pageout_pte_range()
2024-01-26 3:25 ` [PATCH] mm/madvise: don't forget to leave lazy MMU mode " Sergey Senozhatsky
@ 2024-01-26 6:53 ` Andrew Morton
2024-01-26 7:00 ` Sergey Senozhatsky
0 siblings, 1 reply; 6+ messages in thread
From: Andrew Morton @ 2024-01-26 6:53 UTC (permalink / raw)
To: Sergey Senozhatsky; +Cc: Jiexun Wang, linux-kernel, linux-mm
On Fri, 26 Jan 2024 12:25:48 +0900 Sergey Senozhatsky <senozhatsky@chromium.org> wrote:
> We need to leave lazy MMU mode before unlocking.
What might be the userspace-visible effects of this?
> Fixes: b2f557a21bc8 ("mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()"
> Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
I'll add a cc:stable.
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -451,6 +451,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> if (++batch_count == SWAP_CLUSTER_MAX) {
> batch_count = 0;
> if (need_resched()) {
> + arch_leave_lazy_mmu_mode();
> pte_unmap_unlock(start_pte, ptl);
> cond_resched();
> goto restart;
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] mm/madvise: don't forget to leave lazy MMU mode in madvise_cold_or_pageout_pte_range()
2024-01-26 6:53 ` Andrew Morton
@ 2024-01-26 7:00 ` Sergey Senozhatsky
0 siblings, 0 replies; 6+ messages in thread
From: Sergey Senozhatsky @ 2024-01-26 7:00 UTC (permalink / raw)
To: Andrew Morton; +Cc: Sergey Senozhatsky, Jiexun Wang, linux-kernel, linux-mm
On (24/01/25 22:53), Andrew Morton wrote:
> Date: Thu, 25 Jan 2024 22:53:36 -0800
> From: Andrew Morton <akpm@linux-foundation.org>
> To: Sergey Senozhatsky <senozhatsky@chromium.org>
> Cc: Jiexun Wang <wangjiexun@tinylab.org>, linux-kernel@vger.kernel.org,
> linux-mm@kvack.org
> Subject: Re: [PATCH] mm/madvise: don't forget to leave lazy MMU mode in
> madvise_cold_or_pageout_pte_range()
> Message-Id: <20240125225336.6a444c01d9d9812a23a6890b@linux-foundation.org>
> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu)
>
> On Fri, 26 Jan 2024 12:25:48 +0900 Sergey Senozhatsky <senozhatsky@chromium.org> wrote:
>
> > We need to leave lazy MMU mode before unlocking.
It depends on the arch, as far as I understand it.
We can enter lazy MMU mode (on each goto restart) more times than leave
it, and, for isntance, on powerpc that means that we can preempt_disable()
more times than preempt_enable(). That's how enter/leave lazy MMU mode is
implemented there:
static inline void arch_enter_lazy_mmu_mode(void)
{
struct ppc64_tlb_batch *batch;
if (radix_enabled())
return;
/*
* apply_to_page_range can call us this preempt enabled when
* operating on kernel page tables.
*/
preempt_disable();
batch = this_cpu_ptr(&ppc64_tlb_batch);
batch->active = 1;
}
static inline void arch_leave_lazy_mmu_mode(void)
{
struct ppc64_tlb_batch *batch;
if (radix_enabled())
return;
batch = this_cpu_ptr(&ppc64_tlb_batch);
if (batch->index)
__flush_tlb_pending(batch);
batch->active = 0;
preempt_enable();
}
> What might be the userspace-visible effects of this?
>
> > Fixes: b2f557a21bc8 ("mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()"
> > Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
>
> I'll add a cc:stable.
Thanks.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-01-26 7:00 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-09-21 12:02 [PATCH v2 0/1] mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range() Jiexun Wang
[not found] ` <85363861af65fac66c7a98c251906afc0d9c8098.1695291046.git.wangjiexun@tinylab.org>
2023-09-22 0:41 ` [PATCH v2 1/1] " Andrew Morton
2024-01-26 3:12 ` Sergey Senozhatsky
2024-01-26 3:25 ` [PATCH] mm/madvise: don't forget to leave lazy MMU mode " Sergey Senozhatsky
2024-01-26 6:53 ` Andrew Morton
2024-01-26 7:00 ` Sergey Senozhatsky
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).