From: Mel Gorman <mgorman@suse.de>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Alex Thorlton <athorlton@sgi.com>, Rik van Riel <riel@redhat.com>,
Linux-MM <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>, Mel Gorman <mgorman@suse.de>
Subject: [PATCH 02/18] mm: numa: Call MMU notifiers on THP migration
Date: Mon, 9 Dec 2013 07:08:56 +0000 [thread overview]
Message-ID: <1386572952-1191-3-git-send-email-mgorman@suse.de> (raw)
In-Reply-To: <1386572952-1191-1-git-send-email-mgorman@suse.de>
MMU notifiers must be called on THP page migration or secondary MMUs will
get very confused.
Cc: stable@vger.kernel.org
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/migrate.c | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/mm/migrate.c b/mm/migrate.c
index 2cabbd5..be787d5 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -36,6 +36,7 @@
#include <linux/hugetlb_cgroup.h>
#include <linux/gfp.h>
#include <linux/balloon_compaction.h>
+#include <linux/mmu_notifier.h>
#include <asm/tlbflush.h>
@@ -1716,12 +1717,13 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
struct page *page, int node)
{
spinlock_t *ptl;
- unsigned long haddr = address & HPAGE_PMD_MASK;
pg_data_t *pgdat = NODE_DATA(node);
int isolated = 0;
struct page *new_page = NULL;
struct mem_cgroup *memcg = NULL;
int page_lru = page_is_file_cache(page);
+ unsigned long mmun_start = address & HPAGE_PMD_MASK;
+ unsigned long mmun_end = mmun_start + HPAGE_PMD_SIZE;
pmd_t orig_entry;
/*
@@ -1756,10 +1758,12 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
WARN_ON(PageLRU(new_page));
/* Recheck the target PMD */
+ mmu_notifier_invalidate_range_start(mm, mmun_start, mmun_end);
ptl = pmd_lock(mm, pmd);
if (unlikely(!pmd_same(*pmd, entry) || page_count(page) != 2)) {
fail_putback:
spin_unlock(ptl);
+ mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
/* Reverse changes made by migrate_page_copy() */
if (TestClearPageActive(new_page))
@@ -1800,15 +1804,16 @@ fail_putback:
* The SetPageUptodate on the new page and page_add_new_anon_rmap
* guarantee the copy is visible before the pagetable update.
*/
- flush_cache_range(vma, haddr, haddr + HPAGE_PMD_SIZE);
- page_add_new_anon_rmap(new_page, vma, haddr);
- pmdp_clear_flush(vma, haddr, pmd);
- set_pmd_at(mm, haddr, pmd, entry);
+ flush_cache_range(vma, mmun_start, mmun_end);
+ page_add_new_anon_rmap(new_page, vma, mmun_start);
+ pmdp_clear_flush(vma, mmun_start, pmd);
+ set_pmd_at(mm, mmun_start, pmd, entry);
+ flush_tlb_range(vma, mmun_start, mmun_end);
update_mmu_cache_pmd(vma, address, &entry);
if (page_count(page) != 2) {
- set_pmd_at(mm, haddr, pmd, orig_entry);
- flush_tlb_range(vma, haddr, haddr + HPAGE_PMD_SIZE);
+ set_pmd_at(mm, mmun_start, pmd, orig_entry);
+ flush_tlb_range(vma, mmun_start, mmun_end);
update_mmu_cache_pmd(vma, address, &entry);
page_remove_rmap(new_page);
goto fail_putback;
@@ -1823,6 +1828,7 @@ fail_putback:
*/
mem_cgroup_end_migration(memcg, page, new_page, true);
spin_unlock(ptl);
+ mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
unlock_page(new_page);
unlock_page(page);
@@ -1843,7 +1849,7 @@ out_dropref:
ptl = pmd_lock(mm, pmd);
if (pmd_same(*pmd, entry)) {
entry = pmd_mknonnuma(entry);
- set_pmd_at(mm, haddr, pmd, entry);
+ set_pmd_at(mm, mmun_start, pmd, entry);
update_mmu_cache_pmd(vma, address, &entry);
}
spin_unlock(ptl);
--
1.8.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2013-12-09 7:09 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-12-09 7:08 [PATCH 00/18] NUMA balancing segmentation fault fixes and misc followups v3 Mel Gorman
2013-12-09 7:08 ` [PATCH 01/18] mm: numa: Serialise parallel get_user_page against THP migration Mel Gorman
2013-12-09 14:08 ` Rik van Riel
2013-12-09 7:08 ` Mel Gorman [this message]
2013-12-09 14:09 ` [PATCH 02/18] mm: numa: Call MMU notifiers on " Rik van Riel
2013-12-09 7:08 ` [PATCH 03/18] mm: Clear pmd_numa before invalidating Mel Gorman
2013-12-09 14:14 ` Rik van Riel
2013-12-09 7:08 ` [PATCH 04/18] mm: numa: Do not clear PMD during PTE update scan Mel Gorman
2013-12-09 14:22 ` Rik van Riel
2013-12-09 7:08 ` [PATCH 05/18] mm: numa: Do not clear PTE for pte_numa update Mel Gorman
2013-12-09 14:31 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 06/18] mm: numa: Ensure anon_vma is locked to prevent parallel THP splits Mel Gorman
2013-12-09 14:34 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 07/18] mm: numa: Avoid unnecessary work on the failure path Mel Gorman
2013-12-09 14:42 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 08/18] sched: numa: Skip inaccessible VMAs Mel Gorman
2013-12-09 14:50 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 09/18] mm: numa: Clear numa hinting information on mprotect Mel Gorman
2013-12-09 15:57 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 10/18] mm: numa: Avoid unnecessary disruption of NUMA hinting during migration Mel Gorman
2013-12-09 16:10 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 11/18] mm: fix TLB flush race between migration, and change_protection_range Mel Gorman
2013-12-10 14:25 ` Rik van Riel
2013-12-10 17:19 ` Mel Gorman
2013-12-10 18:02 ` Paul E. McKenney
2013-12-11 11:21 ` Mel Gorman
2013-12-09 7:09 ` [PATCH 12/18] mm: numa: Defer TLB flush for THP migration as long as possible Mel Gorman
2013-12-09 16:13 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 13/18] mm: numa: Make NUMA-migrate related functions static Mel Gorman
2013-12-09 7:20 ` Wanpeng Li
[not found] ` <20131209072010.GA3716@hacker.(null)>
2013-12-09 8:46 ` Mel Gorman
2013-12-09 8:57 ` Wanpeng Li
[not found] ` <20131209085720.GA16251@hacker.(null)>
2013-12-09 9:08 ` Mel Gorman
2013-12-09 9:13 ` Wanpeng Li
2013-12-09 16:14 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 14/18] mm: numa: Limit scope of lock for NUMA migrate rate limiting Mel Gorman
2013-12-09 16:47 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 15/18] mm: numa: Trace tasks that fail migration due to " Mel Gorman
2013-12-09 16:57 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 16/18] mm: numa: Do not automatically migrate KSM pages Mel Gorman
2013-12-09 16:57 ` Rik van Riel
2013-12-09 7:09 ` [PATCH 17/18] sched: Tracepoint task movement Mel Gorman
2013-12-09 18:54 ` Rik van Riel
2013-12-10 8:42 ` Mel Gorman
2013-12-10 9:06 ` Andrew Jones
2013-12-09 7:09 ` [PATCH 18/18] sched: Add tracepoints related to NUMA task migration Mel Gorman
2013-12-09 19:06 ` Rik van Riel
-- strict thread matches above, loose matches on Subject: below --
2013-12-10 15:51 [PATCH 00/17] NUMA balancing segmentation fault fixes and misc followups v4 Mel Gorman
2013-12-10 15:51 ` [PATCH 02/18] mm: numa: Call MMU notifiers on THP migration Mel Gorman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1386572952-1191-3-git-send-email-mgorman@suse.de \
--to=mgorman@suse.de \
--cc=akpm@linux-foundation.org \
--cc=athorlton@sgi.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=riel@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).