diff for duplicates of <20160531000117.GB18314@bbox> diff --git a/a/1.txt b/N1/1.txt index 3ed70b1..cb90e48 100644 --- a/a/1.txt +++ b/N1/1.txt @@ -3,939 +3,3 @@ Per Vlastimi's review comment. Thanks for the detail review, Vlastimi! If you have another concern, feel free to say. After I resolve all thing, I will send v7 rebased on recent mmotm. - ->From b14aaeeeeb2d3ac0702c7b2eec36409d74406d43 Mon Sep 17 00:00:00 2001 -From: Minchan Kim <minchan@kernel.org> -Date: Fri, 8 Apr 2016 10:34:49 +0900 -Subject: [PATCH] mm: migrate: support non-lru movable page migration - -We have allowed migration for only LRU pages until now and it was -enough to make high-order pages. But recently, embedded system(e.g., -webOS, android) uses lots of non-movable pages(e.g., zram, GPU memory) -so we have seen several reports about troubles of small high-order -allocation. For fixing the problem, there were several efforts -(e,g,. enhance compaction algorithm, SLUB fallback to 0-order page, -reserved memory, vmalloc and so on) but if there are lots of -non-movable pages in system, their solutions are void in the long run. - -So, this patch is to support facility to change non-movable pages -with movable. For the feature, this patch introduces functions related -to migration to address_space_operations as well as some page flags. - -If a driver want to make own pages movable, it should define three functions -which are function pointers of struct address_space_operations. - -1. bool (*isolate_page) (struct page *page, isolate_mode_t mode); - -What VM expects on isolate_page function of driver is to return *true* -if driver isolates page successfully. On returing true, VM marks the page -as PG_isolated so concurrent isolation in several CPUs skip the page -for isolation. If a driver cannot isolate the page, it should return *false*. - -Once page is successfully isolated, VM uses page.lru fields so driver -shouldn't expect to preserve values in that fields. - -2. int (*migratepage) (struct address_space *mapping, - struct page *newpage, struct page *oldpage, enum migrate_mode); - -After isolation, VM calls migratepage of driver with isolated page. -The function of migratepage is to move content of the old page to new page -and set up fields of struct page newpage. Keep in mind that you should -indicate to the VM the oldpage is no longer movable via __ClearPageMovable() -under page_lock if you migrated the oldpage successfully and returns 0. -If driver cannot migrate the page at the moment, driver can return -EAGAIN. -On -EAGAIN, VM will retry page migration in a short time because VM interprets --EAGAIN as "temporal migration failure". On returning any error except -EAGAIN, -VM will give up the page migration without retrying in this time. - -Driver shouldn't touch page.lru field VM using in the functions. - -3. void (*putback_page)(struct page *); - -If migration fails on isolated page, VM should return the isolated page -to the driver so VM calls driver's putback_page with migration failed page. -In this function, driver should put the isolated page back to the own data -structure. - -4. non-lru movable page flags - -There are two page flags for supporting non-lru movable page. - -* PG_movable - -Driver should use the below function to make page movable under page_lock. - - void __SetPageMovable(struct page *page, struct address_space *mapping) - -It needs argument of address_space for registering migration family functions -which will be called by VM. Exactly speaking, PG_movable is not a real flag of -struct page. Rather than, VM reuses page->mapping's lower bits to represent it. - - #define PAGE_MAPPING_MOVABLE 0x2 - page->mapping = page->mapping | PAGE_MAPPING_MOVABLE; - -so driver shouldn't access page->mapping directly. Instead, driver should -use page_mapping which mask off the low two bits of page->mapping so it can get -right struct address_space. - -For testing of non-lru movable page, VM supports __PageMovable function. -However, it doesn't guarantee to identify non-lru movable page because -page->mapping field is unified with other variables in struct page. -As well, if driver releases the page after isolation by VM, page->mapping -doesn't have stable value although it has PAGE_MAPPING_MOVABLE -(Look at __ClearPageMovable). But __PageMovable is cheap to catch whether -page is LRU or non-lru movable once the page has been isolated. Because -LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also -good for just peeking to test non-lru movable pages before more expensive -checking with lock_page in pfn scanning to select victim. - -For guaranteeing non-lru movable page, VM provides PageMovable function. -Unlike __PageMovable, PageMovable functions validates page->mapping and -mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden -destroying of page->mapping. - -Driver using __SetPageMovable should clear the flag via __ClearMovablePage -under page_lock before the releasing the page. - -* PG_isolated - -To prevent concurrent isolation among several CPUs, VM marks isolated page -as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru -movable page, it can skip it. Driver doesn't need to manipulate the flag -because VM will set/clear it automatically. Keep in mind that if driver -sees PG_isolated page, it means the page have been isolated by VM so it -shouldn't touch page.lru field. -PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag -for own purpose. - -Cc: Rik van Riel <riel@redhat.com> -Cc: Vlastimil Babka <vbabka@suse.cz> -Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> -Cc: Mel Gorman <mgorman@suse.de> -Cc: Hugh Dickins <hughd@google.com> -Cc: Rafael Aquini <aquini@redhat.com> -Cc: virtualization@lists.linux-foundation.org -Cc: Jonathan Corbet <corbet@lwn.net> -Cc: John Einar Reitan <john.reitan@foss.arm.com> -Cc: dri-devel@lists.freedesktop.org -Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> -Signed-off-by: Gioh Kim <gi-oh.kim@profitbricks.com> -Signed-off-by: Minchan Kim <minchan@kernel.org> ---- - Documentation/filesystems/Locking | 4 + - Documentation/filesystems/vfs.txt | 11 +++ - Documentation/vm/page_migration | 107 ++++++++++++++++++++- - include/linux/compaction.h | 17 ++++ - include/linux/fs.h | 2 + - include/linux/ksm.h | 3 +- - include/linux/migrate.h | 2 + - include/linux/mm.h | 1 + - include/linux/page-flags.h | 33 +++++-- - mm/compaction.c | 85 +++++++++++++---- - mm/ksm.c | 4 +- - mm/migrate.c | 192 ++++++++++++++++++++++++++++++++++---- - mm/page_alloc.c | 2 +- - mm/util.c | 6 +- - 14 files changed, 417 insertions(+), 52 deletions(-) - -diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking -index af7c030a0368..3991a976cf43 100644 ---- a/Documentation/filesystems/Locking -+++ b/Documentation/filesystems/Locking -@@ -195,7 +195,9 @@ unlocks and drops the reference. - int (*releasepage) (struct page *, int); - void (*freepage)(struct page *); - int (*direct_IO)(struct kiocb *, struct iov_iter *iter); -+ bool (*isolate_page) (struct page *, isolate_mode_t); - int (*migratepage)(struct address_space *, struct page *, struct page *); -+ void (*putback_page) (struct page *); - int (*launder_page)(struct page *); - int (*is_partially_uptodate)(struct page *, unsigned long, unsigned long); - int (*error_remove_page)(struct address_space *, struct page *); -@@ -219,7 +221,9 @@ invalidatepage: yes - releasepage: yes - freepage: yes - direct_IO: -+isolate_page: yes - migratepage: yes (both) -+putback_page: yes - launder_page: yes - is_partially_uptodate: yes - error_remove_page: yes -diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt -index 19366fef2652..9d4ae317fdcb 100644 ---- a/Documentation/filesystems/vfs.txt -+++ b/Documentation/filesystems/vfs.txt -@@ -591,9 +591,14 @@ struct address_space_operations { - int (*releasepage) (struct page *, int); - void (*freepage)(struct page *); - ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); -+ /* isolate a page for migration */ -+ bool (*isolate_page) (struct page *, isolate_mode_t); - /* migrate the contents of a page to the specified target */ - int (*migratepage) (struct page *, struct page *); -+ /* put migration-failed page back to right list */ -+ void (*putback_page) (struct page *); - int (*launder_page) (struct page *); -+ - int (*is_partially_uptodate) (struct page *, unsigned long, - unsigned long); - void (*is_dirty_writeback) (struct page *, bool *, bool *); -@@ -739,6 +744,10 @@ struct address_space_operations { - and transfer data directly between the storage and the - application's address space. - -+ isolate_page: Called by the VM when isolating a movable non-lru page. -+ If page is successfully isolated, VM marks the page as PG_isolated -+ via __SetPageIsolated. -+ - migrate_page: This is used to compact the physical memory usage. - If the VM wants to relocate a page (maybe off a memory card - that is signalling imminent failure) it will pass a new page -@@ -746,6 +755,8 @@ struct address_space_operations { - transfer any private data across and update any references - that it has to the page. - -+ putback_page: Called by the VM when isolated page's migration fails. -+ - launder_page: Called before freeing a page - it writes back the dirty page. To - prevent redirtying the page, it is kept locked during the whole - operation. -diff --git a/Documentation/vm/page_migration b/Documentation/vm/page_migration -index fea5c0864170..18d37c7ac50b 100644 ---- a/Documentation/vm/page_migration -+++ b/Documentation/vm/page_migration -@@ -142,5 +142,110 @@ is increased so that the page cannot be freed while page migration occurs. - 20. The new page is moved to the LRU and can be scanned by the swapper - etc again. - --Christoph Lameter, May 8, 2006. -+C. Non-LRU page migration -+------------------------- -+ -+Although original migration aimed for reducing the latency of memory access -+for NUMA, compaction who want to create high-order page is also main customer. -+ -+Current problem of the implementation is that it is designed to migrate only -+*LRU* pages. However, there are potential non-lru pages which can be migrated -+in drivers, for example, zsmalloc, virtio-balloon pages. -+ -+For virtio-balloon pages, some parts of migration code path have been hooked -+up and added virtio-balloon specific functions to intercept migration logics. -+It's too specific to a driver so other drivers who want to make their pages -+movable would have to add own specific hooks in migration path. -+ -+To overclome the problem, VM supports non-LRU page migration which provides -+generic functions for non-LRU movable pages without driver specific hooks -+migration path. -+ -+If a driver want to make own pages movable, it should define three functions -+which are function pointers of struct address_space_operations. -+ -+1. bool (*isolate_page) (struct page *page, isolate_mode_t mode); -+ -+What VM expects on isolate_page function of driver is to return *true* -+if driver isolates page successfully. On returing true, VM marks the page -+as PG_isolated so concurrent isolation in several CPUs skip the page -+for isolation. If a driver cannot isolate the page, it should return *false*. -+ -+Once page is successfully isolated, VM uses page.lru fields so driver -+shouldn't expect to preserve values in that fields. -+ -+2. int (*migratepage) (struct address_space *mapping, -+ struct page *newpage, struct page *oldpage, enum migrate_mode); -+ -+After isolation, VM calls migratepage of driver with isolated page. -+The function of migratepage is to move content of the old page to new page -+and set up fields of struct page newpage. Keep in mind that you should -+indicate to the VM the oldpage is no longer movable via __ClearPageMovable() -+under page_lock if you migrated the oldpage successfully and returns 0. -+If driver cannot migrate the page at the moment, driver can return -EAGAIN. -+On -EAGAIN, VM will retry page migration in a short time because VM interprets -+-EAGAIN as "temporal migration failure". On returning any error except -EAGAIN, -+VM will give up the page migration without retrying in this time. -+ -+Driver shouldn't touch page.lru field VM using in the functions. -+ -+3. void (*putback_page)(struct page *); -+ -+If migration fails on isolated page, VM should return the isolated page -+to the driver so VM calls driver's putback_page with migration failed page. -+In this function, driver should put the isolated page back to the own data -+structure. - -+4. non-lru movable page flags -+ -+There are two page flags for supporting non-lru movable page. -+ -+* PG_movable -+ -+Driver should use the below function to make page movable under page_lock. -+ -+ void __SetPageMovable(struct page *page, struct address_space *mapping) -+ -+It needs argument of address_space for registering migration family functions -+which will be called by VM. Exactly speaking, PG_movable is not a real flag of -+struct page. Rather than, VM reuses page->mapping's lower bits to represent it. -+ -+ #define PAGE_MAPPING_MOVABLE 0x2 -+ page->mapping = page->mapping | PAGE_MAPPING_MOVABLE; -+ -+so driver shouldn't access page->mapping directly. Instead, driver should -+use page_mapping which mask off the low two bits of page->mapping under -+page lock so it can get right struct address_space. -+ -+For testing of non-lru movable page, VM supports __PageMovable function. -+However, it doesn't guarantee to identify non-lru movable page because -+page->mapping field is unified with other variables in struct page. -+As well, if driver releases the page after isolation by VM, page->mapping -+doesn't have stable value although it has PAGE_MAPPING_MOVABLE -+(Look at __ClearPageMovable). But __PageMovable is cheap to catch whether -+page is LRU or non-lru movable once the page has been isolated. Because -+LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also -+good for just peeking to test non-lru movable pages before more expensive -+checking with lock_page in pfn scanning to select victim. -+ -+For guaranteeing non-lru movable page, VM provides PageMovable function. -+Unlike __PageMovable, PageMovable functions validates page->mapping and -+mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden -+destroying of page->mapping. -+ -+Driver using __SetPageMovable should clear the flag via __ClearMovablePage -+under page_lock before the releasing the page. -+ -+* PG_isolated -+ -+To prevent concurrent isolation among several CPUs, VM marks isolated page -+as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru -+movable page, it can skip it. Driver doesn't need to manipulate the flag -+because VM will set/clear it automatically. Keep in mind that if driver -+sees PG_isolated page, it means the page have been isolated by VM so it -+shouldn't touch page.lru field. -+PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag -+for own purpose. -+ -+Christoph Lameter, May 8, 2006. -+Minchan Kim, Mar 28, 2016. -diff --git a/include/linux/compaction.h b/include/linux/compaction.h -index a58c852a268f..c6b47c861cea 100644 ---- a/include/linux/compaction.h -+++ b/include/linux/compaction.h -@@ -54,6 +54,9 @@ enum compact_result { - struct alloc_context; /* in mm/internal.h */ - - #ifdef CONFIG_COMPACTION -+extern int PageMovable(struct page *page); -+extern void __SetPageMovable(struct page *page, struct address_space *mapping); -+extern void __ClearPageMovable(struct page *page); - extern int sysctl_compact_memory; - extern int sysctl_compaction_handler(struct ctl_table *table, int write, - void __user *buffer, size_t *length, loff_t *ppos); -@@ -151,6 +154,19 @@ extern void kcompactd_stop(int nid); - extern void wakeup_kcompactd(pg_data_t *pgdat, int order, int classzone_idx); - - #else -+static inline int PageMovable(struct page *page) -+{ -+ return 0; -+} -+static inline void __SetPageMovable(struct page *page, -+ struct address_space *mapping) -+{ -+} -+ -+static inline void __ClearPageMovable(struct page *page) -+{ -+} -+ - static inline enum compact_result try_to_compact_pages(gfp_t gfp_mask, - unsigned int order, int alloc_flags, - const struct alloc_context *ac, -@@ -212,6 +228,7 @@ static inline void wakeup_kcompactd(pg_data_t *pgdat, int order, int classzone_i - #endif /* CONFIG_COMPACTION */ - - #if defined(CONFIG_COMPACTION) && defined(CONFIG_SYSFS) && defined(CONFIG_NUMA) -+struct node; - extern int compaction_register_node(struct node *node); - extern void compaction_unregister_node(struct node *node); - -diff --git a/include/linux/fs.h b/include/linux/fs.h -index 0cfdf2aec8f7..39ef97414033 100644 ---- a/include/linux/fs.h -+++ b/include/linux/fs.h -@@ -402,6 +402,8 @@ struct address_space_operations { - */ - int (*migratepage) (struct address_space *, - struct page *, struct page *, enum migrate_mode); -+ bool (*isolate_page)(struct page *, isolate_mode_t); -+ void (*putback_page)(struct page *); - int (*launder_page) (struct page *); - int (*is_partially_uptodate) (struct page *, unsigned long, - unsigned long); -diff --git a/include/linux/ksm.h b/include/linux/ksm.h -index 7ae216a39c9e..481c8c4627ca 100644 ---- a/include/linux/ksm.h -+++ b/include/linux/ksm.h -@@ -43,8 +43,7 @@ static inline struct stable_node *page_stable_node(struct page *page) - static inline void set_page_stable_node(struct page *page, - struct stable_node *stable_node) - { -- page->mapping = (void *)stable_node + -- (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); -+ page->mapping = (void *)((unsigned long)stable_node | PAGE_MAPPING_KSM); - } - - /* -diff --git a/include/linux/migrate.h b/include/linux/migrate.h -index 9b50325e4ddf..404fbfefeb33 100644 ---- a/include/linux/migrate.h -+++ b/include/linux/migrate.h -@@ -37,6 +37,8 @@ extern int migrate_page(struct address_space *, - struct page *, struct page *, enum migrate_mode); - extern int migrate_pages(struct list_head *l, new_page_t new, free_page_t free, - unsigned long private, enum migrate_mode mode, int reason); -+extern bool isolate_movable_page(struct page *page, isolate_mode_t mode); -+extern void putback_movable_page(struct page *page); - - extern int migrate_prep(void); - extern int migrate_prep_local(void); -diff --git a/include/linux/mm.h b/include/linux/mm.h -index a00ec816233a..33eaec57e997 100644 ---- a/include/linux/mm.h -+++ b/include/linux/mm.h -@@ -1035,6 +1035,7 @@ static inline pgoff_t page_file_index(struct page *page) - } - - bool page_mapped(struct page *page); -+struct address_space *page_mapping(struct page *page); - - /* - * Return true only if the page has been allocated with -diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h -index e5a32445f930..f36dbb3a3060 100644 ---- a/include/linux/page-flags.h -+++ b/include/linux/page-flags.h -@@ -129,6 +129,9 @@ enum pageflags { - - /* Compound pages. Stored in first tail page's flags */ - PG_double_map = PG_private_2, -+ -+ /* non-lru isolated movable page */ -+ PG_isolated = PG_reclaim, - }; - - #ifndef __GENERATING_BOUNDS_H -@@ -357,29 +360,37 @@ PAGEFLAG(Idle, idle, PF_ANY) - * with the PAGE_MAPPING_ANON bit set to distinguish it. See rmap.h. - * - * On an anonymous page in a VM_MERGEABLE area, if CONFIG_KSM is enabled, -- * the PAGE_MAPPING_KSM bit may be set along with the PAGE_MAPPING_ANON bit; -- * and then page->mapping points, not to an anon_vma, but to a private -+ * the PAGE_MAPPING_MOVABLE bit may be set along with the PAGE_MAPPING_ANON -+ * bit; and then page->mapping points, not to an anon_vma, but to a private - * structure which KSM associates with that merged page. See ksm.h. - * -- * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is currently never used. -+ * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is used for non-lru movable -+ * page and then page->mapping points a struct address_space. - * - * Please note that, confusingly, "page_mapping" refers to the inode - * address_space which maps the page from disk; whereas "page_mapped" - * refers to user virtual address space into which the page is mapped. - */ --#define PAGE_MAPPING_ANON 1 --#define PAGE_MAPPING_KSM 2 --#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM) -+#define PAGE_MAPPING_ANON 0x1 -+#define PAGE_MAPPING_MOVABLE 0x2 -+#define PAGE_MAPPING_KSM (PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE) -+#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE) - --static __always_inline int PageAnonHead(struct page *page) -+static __always_inline int PageMappingFlags(struct page *page) - { -- return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0; -+ return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) != 0; - } - - static __always_inline int PageAnon(struct page *page) - { - page = compound_head(page); -- return PageAnonHead(page); -+ return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0; -+} -+ -+static __always_inline int __PageMovable(struct page *page) -+{ -+ return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) == -+ PAGE_MAPPING_MOVABLE; - } - - #ifdef CONFIG_KSM -@@ -393,7 +404,7 @@ static __always_inline int PageKsm(struct page *page) - { - page = compound_head(page); - return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) == -- (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); -+ PAGE_MAPPING_KSM; - } - #else - TESTPAGEFLAG_FALSE(Ksm) -@@ -641,6 +652,8 @@ static inline void __ClearPageBalloon(struct page *page) - atomic_set(&page->_mapcount, -1); - } - -+__PAGEFLAG(Isolated, isolated, PF_ANY); -+ - /* - * If network-based swap is enabled, sl*b must keep track of whether pages - * were allocated from pfmemalloc reserves. -diff --git a/mm/compaction.c b/mm/compaction.c -index 1427366ad673..a680b52e190b 100644 ---- a/mm/compaction.c -+++ b/mm/compaction.c -@@ -81,6 +81,44 @@ static inline bool migrate_async_suitable(int migratetype) - - #ifdef CONFIG_COMPACTION - -+int PageMovable(struct page *page) -+{ -+ struct address_space *mapping; -+ -+ VM_BUG_ON_PAGE(!PageLocked(page), page); -+ if (!__PageMovable(page)) -+ return 0; -+ -+ mapping = page_mapping(page); -+ if (mapping && mapping->a_ops && mapping->a_ops->isolate_page) -+ return 1; -+ -+ return 0; -+} -+EXPORT_SYMBOL(PageMovable); -+ -+void __SetPageMovable(struct page *page, struct address_space *mapping) -+{ -+ VM_BUG_ON_PAGE(!PageLocked(page), page); -+ VM_BUG_ON_PAGE((unsigned long)mapping & PAGE_MAPPING_MOVABLE, page); -+ page->mapping = (void *)((unsigned long)mapping | PAGE_MAPPING_MOVABLE); -+} -+EXPORT_SYMBOL(__SetPageMovable); -+ -+void __ClearPageMovable(struct page *page) -+{ -+ VM_BUG_ON_PAGE(!PageLocked(page), page); -+ VM_BUG_ON_PAGE(!PageMovable(page), page); -+ /* -+ * Clear registered address_space val with keeping PAGE_MAPPING_MOVABLE -+ * flag so that VM can catch up released page by driver after isolation. -+ * With it, VM migration doesn't try to put it back. -+ */ -+ page->mapping = (void *)((unsigned long)page->mapping & -+ PAGE_MAPPING_MOVABLE); -+} -+EXPORT_SYMBOL(__ClearPageMovable); -+ - /* Do not skip compaction more than 64 times */ - #define COMPACT_MAX_DEFER_SHIFT 6 - -@@ -735,21 +773,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, - } - - /* -- * Check may be lockless but that's ok as we recheck later. -- * It's possible to migrate LRU pages and balloon pages -- * Skip any other type of page -- */ -- is_lru = PageLRU(page); -- if (!is_lru) { -- if (unlikely(balloon_page_movable(page))) { -- if (balloon_page_isolate(page)) { -- /* Successfully isolated */ -- goto isolate_success; -- } -- } -- } -- -- /* - * Regardless of being on LRU, compound pages such as THP and - * hugetlbfs are not to be compacted. We can potentially save - * a lot of iterations if we skip them at once. The check is -@@ -765,8 +788,38 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, - goto isolate_fail; - } - -- if (!is_lru) -+ /* -+ * Check may be lockless but that's ok as we recheck later. -+ * It's possible to migrate LRU and non-lru movable pages. -+ * Skip any other type of page -+ */ -+ is_lru = PageLRU(page); -+ if (!is_lru) { -+ if (unlikely(balloon_page_movable(page))) { -+ if (balloon_page_isolate(page)) { -+ /* Successfully isolated */ -+ goto isolate_success; -+ } -+ } -+ -+ /* -+ * __PageMovable can return false positive so we need -+ * to verify it under page_lock. -+ */ -+ if (unlikely(__PageMovable(page)) && -+ !PageIsolated(page)) { -+ if (locked) { -+ spin_unlock_irqrestore(&zone->lru_lock, -+ flags); -+ locked = false; -+ } -+ -+ if (isolate_movable_page(page, isolate_mode)) -+ goto isolate_success; -+ } -+ - goto isolate_fail; -+ } - - /* - * Migration will fail if an anonymous page is pinned in memory, -diff --git a/mm/ksm.c b/mm/ksm.c -index 4786b4150f62..35b8aef867a9 100644 ---- a/mm/ksm.c -+++ b/mm/ksm.c -@@ -532,8 +532,8 @@ static struct page *get_ksm_page(struct stable_node *stable_node, bool lock_it) - void *expected_mapping; - unsigned long kpfn; - -- expected_mapping = (void *)stable_node + -- (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); -+ expected_mapping = (void *)((unsigned long)stable_node | -+ PAGE_MAPPING_KSM); - again: - kpfn = READ_ONCE(stable_node->kpfn); - page = pfn_to_page(kpfn); -diff --git a/mm/migrate.c b/mm/migrate.c -index 2666f28b5236..60abcf379b51 100644 ---- a/mm/migrate.c -+++ b/mm/migrate.c -@@ -31,6 +31,7 @@ - #include <linux/vmalloc.h> - #include <linux/security.h> - #include <linux/backing-dev.h> -+#include <linux/compaction.h> - #include <linux/syscalls.h> - #include <linux/hugetlb.h> - #include <linux/hugetlb_cgroup.h> -@@ -73,6 +74,81 @@ int migrate_prep_local(void) - return 0; - } - -+bool isolate_movable_page(struct page *page, isolate_mode_t mode) -+{ -+ struct address_space *mapping; -+ -+ /* -+ * Avoid burning cycles with pages that are yet under __free_pages(), -+ * or just got freed under us. -+ * -+ * In case we 'win' a race for a movable page being freed under us and -+ * raise its refcount preventing __free_pages() from doing its job -+ * the put_page() at the end of this block will take care of -+ * release this page, thus avoiding a nasty leakage. -+ */ -+ if (unlikely(!get_page_unless_zero(page))) -+ goto out; -+ -+ /* -+ * Check PageMovable before holding a PG_lock because page's owner -+ * assumes anybody doesn't touch PG_lock of newly allocated page -+ * so unconditionally grapping the lock ruins page's owner side. -+ */ -+ if (unlikely(!__PageMovable(page))) -+ goto out_putpage; -+ /* -+ * As movable pages are not isolated from LRU lists, concurrent -+ * compaction threads can race against page migration functions -+ * as well as race against the releasing a page. -+ * -+ * In order to avoid having an already isolated movable page -+ * being (wrongly) re-isolated while it is under migration, -+ * or to avoid attempting to isolate pages being released, -+ * lets be sure we have the page lock -+ * before proceeding with the movable page isolation steps. -+ */ -+ if (unlikely(!trylock_page(page))) -+ goto out_putpage; -+ -+ if (!PageMovable(page) || PageIsolated(page)) -+ goto out_no_isolated; -+ -+ mapping = page_mapping(page); -+ VM_BUG_ON_PAGE(!mapping, page); -+ -+ if (!mapping->a_ops->isolate_page(page, mode)) -+ goto out_no_isolated; -+ -+ /* Driver shouldn't use PG_isolated bit of page->flags */ -+ WARN_ON_ONCE(PageIsolated(page)); -+ __SetPageIsolated(page); -+ unlock_page(page); -+ -+ return true; -+ -+out_no_isolated: -+ unlock_page(page); -+out_putpage: -+ put_page(page); -+out: -+ return false; -+} -+ -+/* It should be called on page which is PG_movable */ -+void putback_movable_page(struct page *page) -+{ -+ struct address_space *mapping; -+ -+ VM_BUG_ON_PAGE(!PageLocked(page), page); -+ VM_BUG_ON_PAGE(!PageMovable(page), page); -+ VM_BUG_ON_PAGE(!PageIsolated(page), page); -+ -+ mapping = page_mapping(page); -+ mapping->a_ops->putback_page(page); -+ __ClearPageIsolated(page); -+} -+ - /* - * Put previously isolated pages back onto the appropriate lists - * from where they were once taken off for compaction/migration. -@@ -94,10 +170,25 @@ void putback_movable_pages(struct list_head *l) - list_del(&page->lru); - dec_zone_page_state(page, NR_ISOLATED_ANON + - page_is_file_cache(page)); -- if (unlikely(isolated_balloon_page(page))) -+ if (unlikely(isolated_balloon_page(page))) { - balloon_page_putback(page); -- else -+ /* -+ * We isolated non-lru movable page so here we can use -+ * __PageMovable because LRU page's mapping cannot have -+ * PAGE_MAPPING_MOVABLE. -+ */ -+ } else if (unlikely(__PageMovable(page))) { -+ VM_BUG_ON_PAGE(!PageIsolated(page), page); -+ lock_page(page); -+ if (PageMovable(page)) -+ putback_movable_page(page); -+ else -+ __ClearPageIsolated(page); -+ unlock_page(page); -+ put_page(page); -+ } else { - putback_lru_page(page); -+ } - } - } - -@@ -592,7 +683,7 @@ void migrate_page_copy(struct page *newpage, struct page *page) - ***********************************************************/ - - /* -- * Common logic to directly migrate a single page suitable for -+ * Common logic to directly migrate a single LRU page suitable for - * pages that do not use PagePrivate/PagePrivate2. - * - * Pages are locked upon entry and exit. -@@ -755,33 +846,72 @@ static int move_to_new_page(struct page *newpage, struct page *page, - enum migrate_mode mode) - { - struct address_space *mapping; -- int rc; -+ int rc = -EAGAIN; -+ bool is_lru = !__PageMovable(page); - - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(!PageLocked(newpage), newpage); - - mapping = page_mapping(page); -- if (!mapping) -- rc = migrate_page(mapping, newpage, page, mode); -- else if (mapping->a_ops->migratepage) -+ -+ if (likely(is_lru)) { -+ if (!mapping) -+ rc = migrate_page(mapping, newpage, page, mode); -+ else if (mapping->a_ops->migratepage) -+ /* -+ * Most pages have a mapping and most filesystems -+ * provide a migratepage callback. Anonymous pages -+ * are part of swap space which also has its own -+ * migratepage callback. This is the most common path -+ * for page migration. -+ */ -+ rc = mapping->a_ops->migratepage(mapping, newpage, -+ page, mode); -+ else -+ rc = fallback_migrate_page(mapping, newpage, -+ page, mode); -+ } else { - /* -- * Most pages have a mapping and most filesystems provide a -- * migratepage callback. Anonymous pages are part of swap -- * space which also has its own migratepage callback. This -- * is the most common path for page migration. -+ * In case of non-lru page, it could be released after -+ * isolation step. In that case, we shouldn't try migration. - */ -- rc = mapping->a_ops->migratepage(mapping, newpage, page, mode); -- else -- rc = fallback_migrate_page(mapping, newpage, page, mode); -+ VM_BUG_ON_PAGE(!PageIsolated(page), page); -+ if (!PageMovable(page)) { -+ rc = MIGRATEPAGE_SUCCESS; -+ __ClearPageIsolated(page); -+ goto out; -+ } -+ -+ rc = mapping->a_ops->migratepage(mapping, newpage, -+ page, mode); -+ WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && -+ !PageIsolated(page)); -+ } - - /* - * When successful, old pagecache page->mapping must be cleared before - * page is freed; but stats require that PageAnon be left as PageAnon. - */ - if (rc == MIGRATEPAGE_SUCCESS) { -- if (!PageAnon(page)) -+ if (__PageMovable(page)) { -+ VM_BUG_ON_PAGE(!PageIsolated(page), page); -+ -+ /* -+ * We clear PG_movable under page_lock so any compactor -+ * cannot try to migrate this page. -+ */ -+ __ClearPageIsolated(page); -+ } -+ -+ /* -+ * Anonymous and movable page->mapping will be cleard by -+ * free_pages_prepare so don't reset it here for keeping -+ * the type to work PageAnon, for example. -+ */ -+ if (!PageMappingFlags(page)) - page->mapping = NULL; - } -+out: - return rc; - } - -@@ -791,6 +921,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage, - int rc = -EAGAIN; - int page_was_mapped = 0; - struct anon_vma *anon_vma = NULL; -+ bool is_lru = !__PageMovable(page); - - if (!trylock_page(page)) { - if (!force || mode == MIGRATE_ASYNC) -@@ -871,6 +1002,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage, - goto out_unlock_both; - } - -+ if (unlikely(!is_lru)) { -+ rc = move_to_new_page(newpage, page, mode); -+ goto out_unlock_both; -+ } -+ - /* - * Corner case handling: - * 1. When a new swap-cache page is read into, it is added to the LRU -@@ -920,7 +1056,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage, - * list in here. - */ - if (rc == MIGRATEPAGE_SUCCESS) { -- if (unlikely(__is_movable_balloon_page(newpage))) -+ if (unlikely(__is_movable_balloon_page(newpage) || -+ __PageMovable(newpage))) - put_page(newpage); - else - putback_lru_page(newpage); -@@ -961,6 +1098,12 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, - /* page was freed from under us. So we are done. */ - ClearPageActive(page); - ClearPageUnevictable(page); -+ if (unlikely(__PageMovable(page))) { -+ lock_page(page); -+ if (!PageMovable(page)) -+ __ClearPageIsolated(page); -+ unlock_page(page); -+ } - if (put_new_page) - put_new_page(newpage, private); - else -@@ -1010,8 +1153,21 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, - num_poisoned_pages_inc(); - } - } else { -- if (rc != -EAGAIN) -- putback_lru_page(page); -+ if (rc != -EAGAIN) { -+ if (likely(!__PageMovable(page))) { -+ putback_lru_page(page); -+ goto put_new; -+ } -+ -+ lock_page(page); -+ if (PageMovable(page)) -+ putback_movable_page(page); -+ else -+ __ClearPageIsolated(page); -+ unlock_page(page); -+ put_page(page); -+ } -+put_new: - if (put_new_page) - put_new_page(newpage, private); - else -diff --git a/mm/page_alloc.c b/mm/page_alloc.c -index d27e8b968ac3..4d0ba8f6cfee 100644 ---- a/mm/page_alloc.c -+++ b/mm/page_alloc.c -@@ -1014,7 +1014,7 @@ static __always_inline bool free_pages_prepare(struct page *page, - (page + i)->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; - } - } -- if (PageAnonHead(page)) -+ if (PageMappingFlags(page)) - page->mapping = NULL; - if (check_free) - bad += free_pages_check(page); -diff --git a/mm/util.c b/mm/util.c -index 917e0e3d0f8e..b756ee36f7f0 100644 ---- a/mm/util.c -+++ b/mm/util.c -@@ -399,10 +399,12 @@ struct address_space *page_mapping(struct page *page) - } - - mapping = page->mapping; -- if ((unsigned long)mapping & PAGE_MAPPING_FLAGS) -+ if ((unsigned long)mapping & PAGE_MAPPING_ANON) - return NULL; -- return mapping; -+ -+ return (void *)((unsigned long)mapping & ~PAGE_MAPPING_FLAGS); - } -+EXPORT_SYMBOL(page_mapping); - - /* Slow path of page_mapcount() for compound pages */ - int __page_mapcount(struct page *page) --- -1.9.1 - --- -To unsubscribe, send a message with 'unsubscribe linux-mm' in -the body to majordomo@kvack.org. For more info on Linux MM, -see: http://www.linux-mm.org/ . -Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> diff --git a/a/content_digest b/N1/content_digest index 25ea410..00c22c3 100644 --- a/a/content_digest +++ b/N1/content_digest @@ -25,942 +25,6 @@ "\n" "Thanks for the detail review, Vlastimi!\n" "If you have another concern, feel free to say.\n" - "After I resolve all thing, I will send v7 rebased on recent mmotm.\n" - "\n" - ">From b14aaeeeeb2d3ac0702c7b2eec36409d74406d43 Mon Sep 17 00:00:00 2001\n" - "From: Minchan Kim <minchan@kernel.org>\n" - "Date: Fri, 8 Apr 2016 10:34:49 +0900\n" - "Subject: [PATCH] mm: migrate: support non-lru movable page migration\n" - "\n" - "We have allowed migration for only LRU pages until now and it was\n" - "enough to make high-order pages. But recently, embedded system(e.g.,\n" - "webOS, android) uses lots of non-movable pages(e.g., zram, GPU memory)\n" - "so we have seen several reports about troubles of small high-order\n" - "allocation. For fixing the problem, there were several efforts\n" - "(e,g,. enhance compaction algorithm, SLUB fallback to 0-order page,\n" - "reserved memory, vmalloc and so on) but if there are lots of\n" - "non-movable pages in system, their solutions are void in the long run.\n" - "\n" - "So, this patch is to support facility to change non-movable pages\n" - "with movable. For the feature, this patch introduces functions related\n" - "to migration to address_space_operations as well as some page flags.\n" - "\n" - "If a driver want to make own pages movable, it should define three functions\n" - "which are function pointers of struct address_space_operations.\n" - "\n" - "1. bool (*isolate_page) (struct page *page, isolate_mode_t mode);\n" - "\n" - "What VM expects on isolate_page function of driver is to return *true*\n" - "if driver isolates page successfully. On returing true, VM marks the page\n" - "as PG_isolated so concurrent isolation in several CPUs skip the page\n" - "for isolation. If a driver cannot isolate the page, it should return *false*.\n" - "\n" - "Once page is successfully isolated, VM uses page.lru fields so driver\n" - "shouldn't expect to preserve values in that fields.\n" - "\n" - "2. int (*migratepage) (struct address_space *mapping,\n" - "\t\tstruct page *newpage, struct page *oldpage, enum migrate_mode);\n" - "\n" - "After isolation, VM calls migratepage of driver with isolated page.\n" - "The function of migratepage is to move content of the old page to new page\n" - "and set up fields of struct page newpage. Keep in mind that you should\n" - "indicate to the VM the oldpage is no longer movable via __ClearPageMovable()\n" - "under page_lock if you migrated the oldpage successfully and returns 0.\n" - "If driver cannot migrate the page at the moment, driver can return -EAGAIN.\n" - "On -EAGAIN, VM will retry page migration in a short time because VM interprets\n" - "-EAGAIN as \"temporal migration failure\". On returning any error except -EAGAIN,\n" - "VM will give up the page migration without retrying in this time.\n" - "\n" - "Driver shouldn't touch page.lru field VM using in the functions.\n" - "\n" - "3. void (*putback_page)(struct page *);\n" - "\n" - "If migration fails on isolated page, VM should return the isolated page\n" - "to the driver so VM calls driver's putback_page with migration failed page.\n" - "In this function, driver should put the isolated page back to the own data\n" - "structure.\n" - "\n" - "4. non-lru movable page flags\n" - "\n" - "There are two page flags for supporting non-lru movable page.\n" - "\n" - "* PG_movable\n" - "\n" - "Driver should use the below function to make page movable under page_lock.\n" - "\n" - "\tvoid __SetPageMovable(struct page *page, struct address_space *mapping)\n" - "\n" - "It needs argument of address_space for registering migration family functions\n" - "which will be called by VM. Exactly speaking, PG_movable is not a real flag of\n" - "struct page. Rather than, VM reuses page->mapping's lower bits to represent it.\n" - "\n" - "\t#define PAGE_MAPPING_MOVABLE 0x2\n" - "\tpage->mapping = page->mapping | PAGE_MAPPING_MOVABLE;\n" - "\n" - "so driver shouldn't access page->mapping directly. Instead, driver should\n" - "use page_mapping which mask off the low two bits of page->mapping so it can get\n" - "right struct address_space.\n" - "\n" - "For testing of non-lru movable page, VM supports __PageMovable function.\n" - "However, it doesn't guarantee to identify non-lru movable page because\n" - "page->mapping field is unified with other variables in struct page.\n" - "As well, if driver releases the page after isolation by VM, page->mapping\n" - "doesn't have stable value although it has PAGE_MAPPING_MOVABLE\n" - "(Look at __ClearPageMovable). But __PageMovable is cheap to catch whether\n" - "page is LRU or non-lru movable once the page has been isolated. Because\n" - "LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also\n" - "good for just peeking to test non-lru movable pages before more expensive\n" - "checking with lock_page in pfn scanning to select victim.\n" - "\n" - "For guaranteeing non-lru movable page, VM provides PageMovable function.\n" - "Unlike __PageMovable, PageMovable functions validates page->mapping and\n" - "mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden\n" - "destroying of page->mapping.\n" - "\n" - "Driver using __SetPageMovable should clear the flag via __ClearMovablePage\n" - "under page_lock before the releasing the page.\n" - "\n" - "* PG_isolated\n" - "\n" - "To prevent concurrent isolation among several CPUs, VM marks isolated page\n" - "as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru\n" - "movable page, it can skip it. Driver doesn't need to manipulate the flag\n" - "because VM will set/clear it automatically. Keep in mind that if driver\n" - "sees PG_isolated page, it means the page have been isolated by VM so it\n" - "shouldn't touch page.lru field.\n" - "PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag\n" - "for own purpose.\n" - "\n" - "Cc: Rik van Riel <riel@redhat.com>\n" - "Cc: Vlastimil Babka <vbabka@suse.cz>\n" - "Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>\n" - "Cc: Mel Gorman <mgorman@suse.de>\n" - "Cc: Hugh Dickins <hughd@google.com>\n" - "Cc: Rafael Aquini <aquini@redhat.com>\n" - "Cc: virtualization@lists.linux-foundation.org\n" - "Cc: Jonathan Corbet <corbet@lwn.net>\n" - "Cc: John Einar Reitan <john.reitan@foss.arm.com>\n" - "Cc: dri-devel@lists.freedesktop.org\n" - "Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>\n" - "Signed-off-by: Gioh Kim <gi-oh.kim@profitbricks.com>\n" - "Signed-off-by: Minchan Kim <minchan@kernel.org>\n" - "---\n" - " Documentation/filesystems/Locking | 4 +\n" - " Documentation/filesystems/vfs.txt | 11 +++\n" - " Documentation/vm/page_migration | 107 ++++++++++++++++++++-\n" - " include/linux/compaction.h | 17 ++++\n" - " include/linux/fs.h | 2 +\n" - " include/linux/ksm.h | 3 +-\n" - " include/linux/migrate.h | 2 +\n" - " include/linux/mm.h | 1 +\n" - " include/linux/page-flags.h | 33 +++++--\n" - " mm/compaction.c | 85 +++++++++++++----\n" - " mm/ksm.c | 4 +-\n" - " mm/migrate.c | 192 ++++++++++++++++++++++++++++++++++----\n" - " mm/page_alloc.c | 2 +-\n" - " mm/util.c | 6 +-\n" - " 14 files changed, 417 insertions(+), 52 deletions(-)\n" - "\n" - "diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking\n" - "index af7c030a0368..3991a976cf43 100644\n" - "--- a/Documentation/filesystems/Locking\n" - "+++ b/Documentation/filesystems/Locking\n" - "@@ -195,7 +195,9 @@ unlocks and drops the reference.\n" - " \tint (*releasepage) (struct page *, int);\n" - " \tvoid (*freepage)(struct page *);\n" - " \tint (*direct_IO)(struct kiocb *, struct iov_iter *iter);\n" - "+\tbool (*isolate_page) (struct page *, isolate_mode_t);\n" - " \tint (*migratepage)(struct address_space *, struct page *, struct page *);\n" - "+\tvoid (*putback_page) (struct page *);\n" - " \tint (*launder_page)(struct page *);\n" - " \tint (*is_partially_uptodate)(struct page *, unsigned long, unsigned long);\n" - " \tint (*error_remove_page)(struct address_space *, struct page *);\n" - "@@ -219,7 +221,9 @@ invalidatepage:\t\tyes\n" - " releasepage:\t\tyes\n" - " freepage:\t\tyes\n" - " direct_IO:\n" - "+isolate_page:\t\tyes\n" - " migratepage:\t\tyes (both)\n" - "+putback_page:\t\tyes\n" - " launder_page:\t\tyes\n" - " is_partially_uptodate:\tyes\n" - " error_remove_page:\tyes\n" - "diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt\n" - "index 19366fef2652..9d4ae317fdcb 100644\n" - "--- a/Documentation/filesystems/vfs.txt\n" - "+++ b/Documentation/filesystems/vfs.txt\n" - "@@ -591,9 +591,14 @@ struct address_space_operations {\n" - " \tint (*releasepage) (struct page *, int);\n" - " \tvoid (*freepage)(struct page *);\n" - " \tssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter);\n" - "+\t/* isolate a page for migration */\n" - "+\tbool (*isolate_page) (struct page *, isolate_mode_t);\n" - " \t/* migrate the contents of a page to the specified target */\n" - " \tint (*migratepage) (struct page *, struct page *);\n" - "+\t/* put migration-failed page back to right list */\n" - "+\tvoid (*putback_page) (struct page *);\n" - " \tint (*launder_page) (struct page *);\n" - "+\n" - " \tint (*is_partially_uptodate) (struct page *, unsigned long,\n" - " \t\t\t\t\tunsigned long);\n" - " \tvoid (*is_dirty_writeback) (struct page *, bool *, bool *);\n" - "@@ -739,6 +744,10 @@ struct address_space_operations {\n" - " and transfer data directly between the storage and the\n" - " application's address space.\n" - " \n" - "+ isolate_page: Called by the VM when isolating a movable non-lru page.\n" - "+\tIf page is successfully isolated, VM marks the page as PG_isolated\n" - "+\tvia __SetPageIsolated.\n" - "+\n" - " migrate_page: This is used to compact the physical memory usage.\n" - " If the VM wants to relocate a page (maybe off a memory card\n" - " that is signalling imminent failure) it will pass a new page\n" - "@@ -746,6 +755,8 @@ struct address_space_operations {\n" - " \ttransfer any private data across and update any references\n" - " that it has to the page.\n" - " \n" - "+ putback_page: Called by the VM when isolated page's migration fails.\n" - "+\n" - " launder_page: Called before freeing a page - it writes back the dirty page. To\n" - " \tprevent redirtying the page, it is kept locked during the whole\n" - " \toperation.\n" - "diff --git a/Documentation/vm/page_migration b/Documentation/vm/page_migration\n" - "index fea5c0864170..18d37c7ac50b 100644\n" - "--- a/Documentation/vm/page_migration\n" - "+++ b/Documentation/vm/page_migration\n" - "@@ -142,5 +142,110 @@ is increased so that the page cannot be freed while page migration occurs.\n" - " 20. The new page is moved to the LRU and can be scanned by the swapper\n" - " etc again.\n" - " \n" - "-Christoph Lameter, May 8, 2006.\n" - "+C. Non-LRU page migration\n" - "+-------------------------\n" - "+\n" - "+Although original migration aimed for reducing the latency of memory access\n" - "+for NUMA, compaction who want to create high-order page is also main customer.\n" - "+\n" - "+Current problem of the implementation is that it is designed to migrate only\n" - "+*LRU* pages. However, there are potential non-lru pages which can be migrated\n" - "+in drivers, for example, zsmalloc, virtio-balloon pages.\n" - "+\n" - "+For virtio-balloon pages, some parts of migration code path have been hooked\n" - "+up and added virtio-balloon specific functions to intercept migration logics.\n" - "+It's too specific to a driver so other drivers who want to make their pages\n" - "+movable would have to add own specific hooks in migration path.\n" - "+\n" - "+To overclome the problem, VM supports non-LRU page migration which provides\n" - "+generic functions for non-LRU movable pages without driver specific hooks\n" - "+migration path.\n" - "+\n" - "+If a driver want to make own pages movable, it should define three functions\n" - "+which are function pointers of struct address_space_operations.\n" - "+\n" - "+1. bool (*isolate_page) (struct page *page, isolate_mode_t mode);\n" - "+\n" - "+What VM expects on isolate_page function of driver is to return *true*\n" - "+if driver isolates page successfully. On returing true, VM marks the page\n" - "+as PG_isolated so concurrent isolation in several CPUs skip the page\n" - "+for isolation. If a driver cannot isolate the page, it should return *false*.\n" - "+\n" - "+Once page is successfully isolated, VM uses page.lru fields so driver\n" - "+shouldn't expect to preserve values in that fields.\n" - "+\n" - "+2. int (*migratepage) (struct address_space *mapping,\n" - "+\t\tstruct page *newpage, struct page *oldpage, enum migrate_mode);\n" - "+\n" - "+After isolation, VM calls migratepage of driver with isolated page.\n" - "+The function of migratepage is to move content of the old page to new page\n" - "+and set up fields of struct page newpage. Keep in mind that you should\n" - "+indicate to the VM the oldpage is no longer movable via __ClearPageMovable()\n" - "+under page_lock if you migrated the oldpage successfully and returns 0.\n" - "+If driver cannot migrate the page at the moment, driver can return -EAGAIN.\n" - "+On -EAGAIN, VM will retry page migration in a short time because VM interprets\n" - "+-EAGAIN as \"temporal migration failure\". On returning any error except -EAGAIN,\n" - "+VM will give up the page migration without retrying in this time.\n" - "+\n" - "+Driver shouldn't touch page.lru field VM using in the functions.\n" - "+\n" - "+3. void (*putback_page)(struct page *);\n" - "+\n" - "+If migration fails on isolated page, VM should return the isolated page\n" - "+to the driver so VM calls driver's putback_page with migration failed page.\n" - "+In this function, driver should put the isolated page back to the own data\n" - "+structure.\n" - " \n" - "+4. non-lru movable page flags\n" - "+\n" - "+There are two page flags for supporting non-lru movable page.\n" - "+\n" - "+* PG_movable\n" - "+\n" - "+Driver should use the below function to make page movable under page_lock.\n" - "+\n" - "+\tvoid __SetPageMovable(struct page *page, struct address_space *mapping)\n" - "+\n" - "+It needs argument of address_space for registering migration family functions\n" - "+which will be called by VM. Exactly speaking, PG_movable is not a real flag of\n" - "+struct page. Rather than, VM reuses page->mapping's lower bits to represent it.\n" - "+\n" - "+\t#define PAGE_MAPPING_MOVABLE 0x2\n" - "+\tpage->mapping = page->mapping | PAGE_MAPPING_MOVABLE;\n" - "+\n" - "+so driver shouldn't access page->mapping directly. Instead, driver should\n" - "+use page_mapping which mask off the low two bits of page->mapping under\n" - "+page lock so it can get right struct address_space.\n" - "+\n" - "+For testing of non-lru movable page, VM supports __PageMovable function.\n" - "+However, it doesn't guarantee to identify non-lru movable page because\n" - "+page->mapping field is unified with other variables in struct page.\n" - "+As well, if driver releases the page after isolation by VM, page->mapping\n" - "+doesn't have stable value although it has PAGE_MAPPING_MOVABLE\n" - "+(Look at __ClearPageMovable). But __PageMovable is cheap to catch whether\n" - "+page is LRU or non-lru movable once the page has been isolated. Because\n" - "+LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also\n" - "+good for just peeking to test non-lru movable pages before more expensive\n" - "+checking with lock_page in pfn scanning to select victim.\n" - "+\n" - "+For guaranteeing non-lru movable page, VM provides PageMovable function.\n" - "+Unlike __PageMovable, PageMovable functions validates page->mapping and\n" - "+mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden\n" - "+destroying of page->mapping.\n" - "+\n" - "+Driver using __SetPageMovable should clear the flag via __ClearMovablePage\n" - "+under page_lock before the releasing the page.\n" - "+\n" - "+* PG_isolated\n" - "+\n" - "+To prevent concurrent isolation among several CPUs, VM marks isolated page\n" - "+as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru\n" - "+movable page, it can skip it. Driver doesn't need to manipulate the flag\n" - "+because VM will set/clear it automatically. Keep in mind that if driver\n" - "+sees PG_isolated page, it means the page have been isolated by VM so it\n" - "+shouldn't touch page.lru field.\n" - "+PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag\n" - "+for own purpose.\n" - "+\n" - "+Christoph Lameter, May 8, 2006.\n" - "+Minchan Kim, Mar 28, 2016.\n" - "diff --git a/include/linux/compaction.h b/include/linux/compaction.h\n" - "index a58c852a268f..c6b47c861cea 100644\n" - "--- a/include/linux/compaction.h\n" - "+++ b/include/linux/compaction.h\n" - "@@ -54,6 +54,9 @@ enum compact_result {\n" - " struct alloc_context; /* in mm/internal.h */\n" - " \n" - " #ifdef CONFIG_COMPACTION\n" - "+extern int PageMovable(struct page *page);\n" - "+extern void __SetPageMovable(struct page *page, struct address_space *mapping);\n" - "+extern void __ClearPageMovable(struct page *page);\n" - " extern int sysctl_compact_memory;\n" - " extern int sysctl_compaction_handler(struct ctl_table *table, int write,\n" - " \t\t\tvoid __user *buffer, size_t *length, loff_t *ppos);\n" - "@@ -151,6 +154,19 @@ extern void kcompactd_stop(int nid);\n" - " extern void wakeup_kcompactd(pg_data_t *pgdat, int order, int classzone_idx);\n" - " \n" - " #else\n" - "+static inline int PageMovable(struct page *page)\n" - "+{\n" - "+\treturn 0;\n" - "+}\n" - "+static inline void __SetPageMovable(struct page *page,\n" - "+\t\t\tstruct address_space *mapping)\n" - "+{\n" - "+}\n" - "+\n" - "+static inline void __ClearPageMovable(struct page *page)\n" - "+{\n" - "+}\n" - "+\n" - " static inline enum compact_result try_to_compact_pages(gfp_t gfp_mask,\n" - " \t\t\tunsigned int order, int alloc_flags,\n" - " \t\t\tconst struct alloc_context *ac,\n" - "@@ -212,6 +228,7 @@ static inline void wakeup_kcompactd(pg_data_t *pgdat, int order, int classzone_i\n" - " #endif /* CONFIG_COMPACTION */\n" - " \n" - " #if defined(CONFIG_COMPACTION) && defined(CONFIG_SYSFS) && defined(CONFIG_NUMA)\n" - "+struct node;\n" - " extern int compaction_register_node(struct node *node);\n" - " extern void compaction_unregister_node(struct node *node);\n" - " \n" - "diff --git a/include/linux/fs.h b/include/linux/fs.h\n" - "index 0cfdf2aec8f7..39ef97414033 100644\n" - "--- a/include/linux/fs.h\n" - "+++ b/include/linux/fs.h\n" - "@@ -402,6 +402,8 @@ struct address_space_operations {\n" - " \t */\n" - " \tint (*migratepage) (struct address_space *,\n" - " \t\t\tstruct page *, struct page *, enum migrate_mode);\n" - "+\tbool (*isolate_page)(struct page *, isolate_mode_t);\n" - "+\tvoid (*putback_page)(struct page *);\n" - " \tint (*launder_page) (struct page *);\n" - " \tint (*is_partially_uptodate) (struct page *, unsigned long,\n" - " \t\t\t\t\tunsigned long);\n" - "diff --git a/include/linux/ksm.h b/include/linux/ksm.h\n" - "index 7ae216a39c9e..481c8c4627ca 100644\n" - "--- a/include/linux/ksm.h\n" - "+++ b/include/linux/ksm.h\n" - "@@ -43,8 +43,7 @@ static inline struct stable_node *page_stable_node(struct page *page)\n" - " static inline void set_page_stable_node(struct page *page,\n" - " \t\t\t\t\tstruct stable_node *stable_node)\n" - " {\n" - "-\tpage->mapping = (void *)stable_node +\n" - "-\t\t\t\t(PAGE_MAPPING_ANON | PAGE_MAPPING_KSM);\n" - "+\tpage->mapping = (void *)((unsigned long)stable_node | PAGE_MAPPING_KSM);\n" - " }\n" - " \n" - " /*\n" - "diff --git a/include/linux/migrate.h b/include/linux/migrate.h\n" - "index 9b50325e4ddf..404fbfefeb33 100644\n" - "--- a/include/linux/migrate.h\n" - "+++ b/include/linux/migrate.h\n" - "@@ -37,6 +37,8 @@ extern int migrate_page(struct address_space *,\n" - " \t\t\tstruct page *, struct page *, enum migrate_mode);\n" - " extern int migrate_pages(struct list_head *l, new_page_t new, free_page_t free,\n" - " \t\tunsigned long private, enum migrate_mode mode, int reason);\n" - "+extern bool isolate_movable_page(struct page *page, isolate_mode_t mode);\n" - "+extern void putback_movable_page(struct page *page);\n" - " \n" - " extern int migrate_prep(void);\n" - " extern int migrate_prep_local(void);\n" - "diff --git a/include/linux/mm.h b/include/linux/mm.h\n" - "index a00ec816233a..33eaec57e997 100644\n" - "--- a/include/linux/mm.h\n" - "+++ b/include/linux/mm.h\n" - "@@ -1035,6 +1035,7 @@ static inline pgoff_t page_file_index(struct page *page)\n" - " }\n" - " \n" - " bool page_mapped(struct page *page);\n" - "+struct address_space *page_mapping(struct page *page);\n" - " \n" - " /*\n" - " * Return true only if the page has been allocated with\n" - "diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h\n" - "index e5a32445f930..f36dbb3a3060 100644\n" - "--- a/include/linux/page-flags.h\n" - "+++ b/include/linux/page-flags.h\n" - "@@ -129,6 +129,9 @@ enum pageflags {\n" - " \n" - " \t/* Compound pages. Stored in first tail page's flags */\n" - " \tPG_double_map = PG_private_2,\n" - "+\n" - "+\t/* non-lru isolated movable page */\n" - "+\tPG_isolated = PG_reclaim,\n" - " };\n" - " \n" - " #ifndef __GENERATING_BOUNDS_H\n" - "@@ -357,29 +360,37 @@ PAGEFLAG(Idle, idle, PF_ANY)\n" - " * with the PAGE_MAPPING_ANON bit set to distinguish it. See rmap.h.\n" - " *\n" - " * On an anonymous page in a VM_MERGEABLE area, if CONFIG_KSM is enabled,\n" - "- * the PAGE_MAPPING_KSM bit may be set along with the PAGE_MAPPING_ANON bit;\n" - "- * and then page->mapping points, not to an anon_vma, but to a private\n" - "+ * the PAGE_MAPPING_MOVABLE bit may be set along with the PAGE_MAPPING_ANON\n" - "+ * bit; and then page->mapping points, not to an anon_vma, but to a private\n" - " * structure which KSM associates with that merged page. See ksm.h.\n" - " *\n" - "- * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is currently never used.\n" - "+ * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is used for non-lru movable\n" - "+ * page and then page->mapping points a struct address_space.\n" - " *\n" - " * Please note that, confusingly, \"page_mapping\" refers to the inode\n" - " * address_space which maps the page from disk; whereas \"page_mapped\"\n" - " * refers to user virtual address space into which the page is mapped.\n" - " */\n" - "-#define PAGE_MAPPING_ANON\t1\n" - "-#define PAGE_MAPPING_KSM\t2\n" - "-#define PAGE_MAPPING_FLAGS\t(PAGE_MAPPING_ANON | PAGE_MAPPING_KSM)\n" - "+#define PAGE_MAPPING_ANON\t0x1\n" - "+#define PAGE_MAPPING_MOVABLE\t0x2\n" - "+#define PAGE_MAPPING_KSM\t(PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE)\n" - "+#define PAGE_MAPPING_FLAGS\t(PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE)\n" - " \n" - "-static __always_inline int PageAnonHead(struct page *page)\n" - "+static __always_inline int PageMappingFlags(struct page *page)\n" - " {\n" - "-\treturn ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0;\n" - "+\treturn ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) != 0;\n" - " }\n" - " \n" - " static __always_inline int PageAnon(struct page *page)\n" - " {\n" - " \tpage = compound_head(page);\n" - "-\treturn PageAnonHead(page);\n" - "+\treturn ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0;\n" - "+}\n" - "+\n" - "+static __always_inline int __PageMovable(struct page *page)\n" - "+{\n" - "+\treturn ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) ==\n" - "+\t\t\t\tPAGE_MAPPING_MOVABLE;\n" - " }\n" - " \n" - " #ifdef CONFIG_KSM\n" - "@@ -393,7 +404,7 @@ static __always_inline int PageKsm(struct page *page)\n" - " {\n" - " \tpage = compound_head(page);\n" - " \treturn ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) ==\n" - "-\t\t\t\t(PAGE_MAPPING_ANON | PAGE_MAPPING_KSM);\n" - "+\t\t\t\tPAGE_MAPPING_KSM;\n" - " }\n" - " #else\n" - " TESTPAGEFLAG_FALSE(Ksm)\n" - "@@ -641,6 +652,8 @@ static inline void __ClearPageBalloon(struct page *page)\n" - " \tatomic_set(&page->_mapcount, -1);\n" - " }\n" - " \n" - "+__PAGEFLAG(Isolated, isolated, PF_ANY);\n" - "+\n" - " /*\n" - " * If network-based swap is enabled, sl*b must keep track of whether pages\n" - " * were allocated from pfmemalloc reserves.\n" - "diff --git a/mm/compaction.c b/mm/compaction.c\n" - "index 1427366ad673..a680b52e190b 100644\n" - "--- a/mm/compaction.c\n" - "+++ b/mm/compaction.c\n" - "@@ -81,6 +81,44 @@ static inline bool migrate_async_suitable(int migratetype)\n" - " \n" - " #ifdef CONFIG_COMPACTION\n" - " \n" - "+int PageMovable(struct page *page)\n" - "+{\n" - "+\tstruct address_space *mapping;\n" - "+\n" - "+\tVM_BUG_ON_PAGE(!PageLocked(page), page);\n" - "+\tif (!__PageMovable(page))\n" - "+\t\treturn 0;\n" - "+\n" - "+\tmapping = page_mapping(page);\n" - "+\tif (mapping && mapping->a_ops && mapping->a_ops->isolate_page)\n" - "+\t\treturn 1;\n" - "+\n" - "+\treturn 0;\n" - "+}\n" - "+EXPORT_SYMBOL(PageMovable);\n" - "+\n" - "+void __SetPageMovable(struct page *page, struct address_space *mapping)\n" - "+{\n" - "+\tVM_BUG_ON_PAGE(!PageLocked(page), page);\n" - "+\tVM_BUG_ON_PAGE((unsigned long)mapping & PAGE_MAPPING_MOVABLE, page);\n" - "+\tpage->mapping = (void *)((unsigned long)mapping | PAGE_MAPPING_MOVABLE);\n" - "+}\n" - "+EXPORT_SYMBOL(__SetPageMovable);\n" - "+\n" - "+void __ClearPageMovable(struct page *page)\n" - "+{\n" - "+\tVM_BUG_ON_PAGE(!PageLocked(page), page);\n" - "+\tVM_BUG_ON_PAGE(!PageMovable(page), page);\n" - "+\t/*\n" - "+\t * Clear registered address_space val with keeping PAGE_MAPPING_MOVABLE\n" - "+\t * flag so that VM can catch up released page by driver after isolation.\n" - "+\t * With it, VM migration doesn't try to put it back.\n" - "+\t */\n" - "+\tpage->mapping = (void *)((unsigned long)page->mapping &\n" - "+\t\t\t\tPAGE_MAPPING_MOVABLE);\n" - "+}\n" - "+EXPORT_SYMBOL(__ClearPageMovable);\n" - "+\n" - " /* Do not skip compaction more than 64 times */\n" - " #define COMPACT_MAX_DEFER_SHIFT 6\n" - " \n" - "@@ -735,21 +773,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,\n" - " \t\t}\n" - " \n" - " \t\t/*\n" - "-\t\t * Check may be lockless but that's ok as we recheck later.\n" - "-\t\t * It's possible to migrate LRU pages and balloon pages\n" - "-\t\t * Skip any other type of page\n" - "-\t\t */\n" - "-\t\tis_lru = PageLRU(page);\n" - "-\t\tif (!is_lru) {\n" - "-\t\t\tif (unlikely(balloon_page_movable(page))) {\n" - "-\t\t\t\tif (balloon_page_isolate(page)) {\n" - "-\t\t\t\t\t/* Successfully isolated */\n" - "-\t\t\t\t\tgoto isolate_success;\n" - "-\t\t\t\t}\n" - "-\t\t\t}\n" - "-\t\t}\n" - "-\n" - "-\t\t/*\n" - " \t\t * Regardless of being on LRU, compound pages such as THP and\n" - " \t\t * hugetlbfs are not to be compacted. We can potentially save\n" - " \t\t * a lot of iterations if we skip them at once. The check is\n" - "@@ -765,8 +788,38 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,\n" - " \t\t\tgoto isolate_fail;\n" - " \t\t}\n" - " \n" - "-\t\tif (!is_lru)\n" - "+\t\t/*\n" - "+\t\t * Check may be lockless but that's ok as we recheck later.\n" - "+\t\t * It's possible to migrate LRU and non-lru movable pages.\n" - "+\t\t * Skip any other type of page\n" - "+\t\t */\n" - "+\t\tis_lru = PageLRU(page);\n" - "+\t\tif (!is_lru) {\n" - "+\t\t\tif (unlikely(balloon_page_movable(page))) {\n" - "+\t\t\t\tif (balloon_page_isolate(page)) {\n" - "+\t\t\t\t\t/* Successfully isolated */\n" - "+\t\t\t\t\tgoto isolate_success;\n" - "+\t\t\t\t}\n" - "+\t\t\t}\n" - "+\n" - "+\t\t\t/*\n" - "+\t\t\t * __PageMovable can return false positive so we need\n" - "+\t\t\t * to verify it under page_lock.\n" - "+\t\t\t */\n" - "+\t\t\tif (unlikely(__PageMovable(page)) &&\n" - "+\t\t\t\t\t!PageIsolated(page)) {\n" - "+\t\t\t\tif (locked) {\n" - "+\t\t\t\t\tspin_unlock_irqrestore(&zone->lru_lock,\n" - "+\t\t\t\t\t\t\t\t\tflags);\n" - "+\t\t\t\t\tlocked = false;\n" - "+\t\t\t\t}\n" - "+\n" - "+\t\t\t\tif (isolate_movable_page(page, isolate_mode))\n" - "+\t\t\t\t\tgoto isolate_success;\n" - "+\t\t\t}\n" - "+\n" - " \t\t\tgoto isolate_fail;\n" - "+\t\t}\n" - " \n" - " \t\t/*\n" - " \t\t * Migration will fail if an anonymous page is pinned in memory,\n" - "diff --git a/mm/ksm.c b/mm/ksm.c\n" - "index 4786b4150f62..35b8aef867a9 100644\n" - "--- a/mm/ksm.c\n" - "+++ b/mm/ksm.c\n" - "@@ -532,8 +532,8 @@ static struct page *get_ksm_page(struct stable_node *stable_node, bool lock_it)\n" - " \tvoid *expected_mapping;\n" - " \tunsigned long kpfn;\n" - " \n" - "-\texpected_mapping = (void *)stable_node +\n" - "-\t\t\t\t(PAGE_MAPPING_ANON | PAGE_MAPPING_KSM);\n" - "+\texpected_mapping = (void *)((unsigned long)stable_node |\n" - "+\t\t\t\t\tPAGE_MAPPING_KSM);\n" - " again:\n" - " \tkpfn = READ_ONCE(stable_node->kpfn);\n" - " \tpage = pfn_to_page(kpfn);\n" - "diff --git a/mm/migrate.c b/mm/migrate.c\n" - "index 2666f28b5236..60abcf379b51 100644\n" - "--- a/mm/migrate.c\n" - "+++ b/mm/migrate.c\n" - "@@ -31,6 +31,7 @@\n" - " #include <linux/vmalloc.h>\n" - " #include <linux/security.h>\n" - " #include <linux/backing-dev.h>\n" - "+#include <linux/compaction.h>\n" - " #include <linux/syscalls.h>\n" - " #include <linux/hugetlb.h>\n" - " #include <linux/hugetlb_cgroup.h>\n" - "@@ -73,6 +74,81 @@ int migrate_prep_local(void)\n" - " \treturn 0;\n" - " }\n" - " \n" - "+bool isolate_movable_page(struct page *page, isolate_mode_t mode)\n" - "+{\n" - "+\tstruct address_space *mapping;\n" - "+\n" - "+\t/*\n" - "+\t * Avoid burning cycles with pages that are yet under __free_pages(),\n" - "+\t * or just got freed under us.\n" - "+\t *\n" - "+\t * In case we 'win' a race for a movable page being freed under us and\n" - "+\t * raise its refcount preventing __free_pages() from doing its job\n" - "+\t * the put_page() at the end of this block will take care of\n" - "+\t * release this page, thus avoiding a nasty leakage.\n" - "+\t */\n" - "+\tif (unlikely(!get_page_unless_zero(page)))\n" - "+\t\tgoto out;\n" - "+\n" - "+\t/*\n" - "+\t * Check PageMovable before holding a PG_lock because page's owner\n" - "+\t * assumes anybody doesn't touch PG_lock of newly allocated page\n" - "+\t * so unconditionally grapping the lock ruins page's owner side.\n" - "+\t */\n" - "+\tif (unlikely(!__PageMovable(page)))\n" - "+\t\tgoto out_putpage;\n" - "+\t/*\n" - "+\t * As movable pages are not isolated from LRU lists, concurrent\n" - "+\t * compaction threads can race against page migration functions\n" - "+\t * as well as race against the releasing a page.\n" - "+\t *\n" - "+\t * In order to avoid having an already isolated movable page\n" - "+\t * being (wrongly) re-isolated while it is under migration,\n" - "+\t * or to avoid attempting to isolate pages being released,\n" - "+\t * lets be sure we have the page lock\n" - "+\t * before proceeding with the movable page isolation steps.\n" - "+\t */\n" - "+\tif (unlikely(!trylock_page(page)))\n" - "+\t\tgoto out_putpage;\n" - "+\n" - "+\tif (!PageMovable(page) || PageIsolated(page))\n" - "+\t\tgoto out_no_isolated;\n" - "+\n" - "+\tmapping = page_mapping(page);\n" - "+\tVM_BUG_ON_PAGE(!mapping, page);\n" - "+\n" - "+\tif (!mapping->a_ops->isolate_page(page, mode))\n" - "+\t\tgoto out_no_isolated;\n" - "+\n" - "+\t/* Driver shouldn't use PG_isolated bit of page->flags */\n" - "+\tWARN_ON_ONCE(PageIsolated(page));\n" - "+\t__SetPageIsolated(page);\n" - "+\tunlock_page(page);\n" - "+\n" - "+\treturn true;\n" - "+\n" - "+out_no_isolated:\n" - "+\tunlock_page(page);\n" - "+out_putpage:\n" - "+\tput_page(page);\n" - "+out:\n" - "+\treturn false;\n" - "+}\n" - "+\n" - "+/* It should be called on page which is PG_movable */\n" - "+void putback_movable_page(struct page *page)\n" - "+{\n" - "+\tstruct address_space *mapping;\n" - "+\n" - "+\tVM_BUG_ON_PAGE(!PageLocked(page), page);\n" - "+\tVM_BUG_ON_PAGE(!PageMovable(page), page);\n" - "+\tVM_BUG_ON_PAGE(!PageIsolated(page), page);\n" - "+\n" - "+\tmapping = page_mapping(page);\n" - "+\tmapping->a_ops->putback_page(page);\n" - "+\t__ClearPageIsolated(page);\n" - "+}\n" - "+\n" - " /*\n" - " * Put previously isolated pages back onto the appropriate lists\n" - " * from where they were once taken off for compaction/migration.\n" - "@@ -94,10 +170,25 @@ void putback_movable_pages(struct list_head *l)\n" - " \t\tlist_del(&page->lru);\n" - " \t\tdec_zone_page_state(page, NR_ISOLATED_ANON +\n" - " \t\t\t\tpage_is_file_cache(page));\n" - "-\t\tif (unlikely(isolated_balloon_page(page)))\n" - "+\t\tif (unlikely(isolated_balloon_page(page))) {\n" - " \t\t\tballoon_page_putback(page);\n" - "-\t\telse\n" - "+\t\t/*\n" - "+\t\t * We isolated non-lru movable page so here we can use\n" - "+\t\t * __PageMovable because LRU page's mapping cannot have\n" - "+\t\t * PAGE_MAPPING_MOVABLE.\n" - "+\t\t */\n" - "+\t\t} else if (unlikely(__PageMovable(page))) {\n" - "+\t\t\tVM_BUG_ON_PAGE(!PageIsolated(page), page);\n" - "+\t\t\tlock_page(page);\n" - "+\t\t\tif (PageMovable(page))\n" - "+\t\t\t\tputback_movable_page(page);\n" - "+\t\t\telse\n" - "+\t\t\t\t__ClearPageIsolated(page);\n" - "+\t\t\tunlock_page(page);\n" - "+\t\t\tput_page(page);\n" - "+\t\t} else {\n" - " \t\t\tputback_lru_page(page);\n" - "+\t\t}\n" - " \t}\n" - " }\n" - " \n" - "@@ -592,7 +683,7 @@ void migrate_page_copy(struct page *newpage, struct page *page)\n" - " ***********************************************************/\n" - " \n" - " /*\n" - "- * Common logic to directly migrate a single page suitable for\n" - "+ * Common logic to directly migrate a single LRU page suitable for\n" - " * pages that do not use PagePrivate/PagePrivate2.\n" - " *\n" - " * Pages are locked upon entry and exit.\n" - "@@ -755,33 +846,72 @@ static int move_to_new_page(struct page *newpage, struct page *page,\n" - " \t\t\t\tenum migrate_mode mode)\n" - " {\n" - " \tstruct address_space *mapping;\n" - "-\tint rc;\n" - "+\tint rc = -EAGAIN;\n" - "+\tbool is_lru = !__PageMovable(page);\n" - " \n" - " \tVM_BUG_ON_PAGE(!PageLocked(page), page);\n" - " \tVM_BUG_ON_PAGE(!PageLocked(newpage), newpage);\n" - " \n" - " \tmapping = page_mapping(page);\n" - "-\tif (!mapping)\n" - "-\t\trc = migrate_page(mapping, newpage, page, mode);\n" - "-\telse if (mapping->a_ops->migratepage)\n" - "+\n" - "+\tif (likely(is_lru)) {\n" - "+\t\tif (!mapping)\n" - "+\t\t\trc = migrate_page(mapping, newpage, page, mode);\n" - "+\t\telse if (mapping->a_ops->migratepage)\n" - "+\t\t\t/*\n" - "+\t\t\t * Most pages have a mapping and most filesystems\n" - "+\t\t\t * provide a migratepage callback. Anonymous pages\n" - "+\t\t\t * are part of swap space which also has its own\n" - "+\t\t\t * migratepage callback. This is the most common path\n" - "+\t\t\t * for page migration.\n" - "+\t\t\t */\n" - "+\t\t\trc = mapping->a_ops->migratepage(mapping, newpage,\n" - "+\t\t\t\t\t\t\tpage, mode);\n" - "+\t\telse\n" - "+\t\t\trc = fallback_migrate_page(mapping, newpage,\n" - "+\t\t\t\t\t\t\tpage, mode);\n" - "+\t} else {\n" - " \t\t/*\n" - "-\t\t * Most pages have a mapping and most filesystems provide a\n" - "-\t\t * migratepage callback. Anonymous pages are part of swap\n" - "-\t\t * space which also has its own migratepage callback. This\n" - "-\t\t * is the most common path for page migration.\n" - "+\t\t * In case of non-lru page, it could be released after\n" - "+\t\t * isolation step. In that case, we shouldn't try migration.\n" - " \t\t */\n" - "-\t\trc = mapping->a_ops->migratepage(mapping, newpage, page, mode);\n" - "-\telse\n" - "-\t\trc = fallback_migrate_page(mapping, newpage, page, mode);\n" - "+\t\tVM_BUG_ON_PAGE(!PageIsolated(page), page);\n" - "+\t\tif (!PageMovable(page)) {\n" - "+\t\t\trc = MIGRATEPAGE_SUCCESS;\n" - "+\t\t\t__ClearPageIsolated(page);\n" - "+\t\t\tgoto out;\n" - "+\t\t}\n" - "+\n" - "+\t\trc = mapping->a_ops->migratepage(mapping, newpage,\n" - "+\t\t\t\t\t\tpage, mode);\n" - "+\t\tWARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS &&\n" - "+\t\t\t!PageIsolated(page));\n" - "+\t}\n" - " \n" - " \t/*\n" - " \t * When successful, old pagecache page->mapping must be cleared before\n" - " \t * page is freed; but stats require that PageAnon be left as PageAnon.\n" - " \t */\n" - " \tif (rc == MIGRATEPAGE_SUCCESS) {\n" - "-\t\tif (!PageAnon(page))\n" - "+\t\tif (__PageMovable(page)) {\n" - "+\t\t\tVM_BUG_ON_PAGE(!PageIsolated(page), page);\n" - "+\n" - "+\t\t\t/*\n" - "+\t\t\t * We clear PG_movable under page_lock so any compactor\n" - "+\t\t\t * cannot try to migrate this page.\n" - "+\t\t\t */\n" - "+\t\t\t__ClearPageIsolated(page);\n" - "+\t\t}\n" - "+\n" - "+\t\t/*\n" - "+\t\t * Anonymous and movable page->mapping will be cleard by\n" - "+\t\t * free_pages_prepare so don't reset it here for keeping\n" - "+\t\t * the type to work PageAnon, for example.\n" - "+\t\t */\n" - "+\t\tif (!PageMappingFlags(page))\n" - " \t\t\tpage->mapping = NULL;\n" - " \t}\n" - "+out:\n" - " \treturn rc;\n" - " }\n" - " \n" - "@@ -791,6 +921,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,\n" - " \tint rc = -EAGAIN;\n" - " \tint page_was_mapped = 0;\n" - " \tstruct anon_vma *anon_vma = NULL;\n" - "+\tbool is_lru = !__PageMovable(page);\n" - " \n" - " \tif (!trylock_page(page)) {\n" - " \t\tif (!force || mode == MIGRATE_ASYNC)\n" - "@@ -871,6 +1002,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage,\n" - " \t\tgoto out_unlock_both;\n" - " \t}\n" - " \n" - "+\tif (unlikely(!is_lru)) {\n" - "+\t\trc = move_to_new_page(newpage, page, mode);\n" - "+\t\tgoto out_unlock_both;\n" - "+\t}\n" - "+\n" - " \t/*\n" - " \t * Corner case handling:\n" - " \t * 1. When a new swap-cache page is read into, it is added to the LRU\n" - "@@ -920,7 +1056,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage,\n" - " \t * list in here.\n" - " \t */\n" - " \tif (rc == MIGRATEPAGE_SUCCESS) {\n" - "-\t\tif (unlikely(__is_movable_balloon_page(newpage)))\n" - "+\t\tif (unlikely(__is_movable_balloon_page(newpage) ||\n" - "+\t\t\t\t__PageMovable(newpage)))\n" - " \t\t\tput_page(newpage);\n" - " \t\telse\n" - " \t\t\tputback_lru_page(newpage);\n" - "@@ -961,6 +1098,12 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page,\n" - " \t\t/* page was freed from under us. So we are done. */\n" - " \t\tClearPageActive(page);\n" - " \t\tClearPageUnevictable(page);\n" - "+\t\tif (unlikely(__PageMovable(page))) {\n" - "+\t\t\tlock_page(page);\n" - "+\t\t\tif (!PageMovable(page))\n" - "+\t\t\t\t__ClearPageIsolated(page);\n" - "+\t\t\tunlock_page(page);\n" - "+\t\t}\n" - " \t\tif (put_new_page)\n" - " \t\t\tput_new_page(newpage, private);\n" - " \t\telse\n" - "@@ -1010,8 +1153,21 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page,\n" - " \t\t\t\tnum_poisoned_pages_inc();\n" - " \t\t}\n" - " \t} else {\n" - "-\t\tif (rc != -EAGAIN)\n" - "-\t\t\tputback_lru_page(page);\n" - "+\t\tif (rc != -EAGAIN) {\n" - "+\t\t\tif (likely(!__PageMovable(page))) {\n" - "+\t\t\t\tputback_lru_page(page);\n" - "+\t\t\t\tgoto put_new;\n" - "+\t\t\t}\n" - "+\n" - "+\t\t\tlock_page(page);\n" - "+\t\t\tif (PageMovable(page))\n" - "+\t\t\t\tputback_movable_page(page);\n" - "+\t\t\telse\n" - "+\t\t\t\t__ClearPageIsolated(page);\n" - "+\t\t\tunlock_page(page);\n" - "+\t\t\tput_page(page);\n" - "+\t\t}\n" - "+put_new:\n" - " \t\tif (put_new_page)\n" - " \t\t\tput_new_page(newpage, private);\n" - " \t\telse\n" - "diff --git a/mm/page_alloc.c b/mm/page_alloc.c\n" - "index d27e8b968ac3..4d0ba8f6cfee 100644\n" - "--- a/mm/page_alloc.c\n" - "+++ b/mm/page_alloc.c\n" - "@@ -1014,7 +1014,7 @@ static __always_inline bool free_pages_prepare(struct page *page,\n" - " \t\t\t(page + i)->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;\n" - " \t\t}\n" - " \t}\n" - "-\tif (PageAnonHead(page))\n" - "+\tif (PageMappingFlags(page))\n" - " \t\tpage->mapping = NULL;\n" - " \tif (check_free)\n" - " \t\tbad += free_pages_check(page);\n" - "diff --git a/mm/util.c b/mm/util.c\n" - "index 917e0e3d0f8e..b756ee36f7f0 100644\n" - "--- a/mm/util.c\n" - "+++ b/mm/util.c\n" - "@@ -399,10 +399,12 @@ struct address_space *page_mapping(struct page *page)\n" - " \t}\n" - " \n" - " \tmapping = page->mapping;\n" - "-\tif ((unsigned long)mapping & PAGE_MAPPING_FLAGS)\n" - "+\tif ((unsigned long)mapping & PAGE_MAPPING_ANON)\n" - " \t\treturn NULL;\n" - "-\treturn mapping;\n" - "+\n" - "+\treturn (void *)((unsigned long)mapping & ~PAGE_MAPPING_FLAGS);\n" - " }\n" - "+EXPORT_SYMBOL(page_mapping);\n" - " \n" - " /* Slow path of page_mapcount() for compound pages */\n" - " int __page_mapcount(struct page *page)\n" - "-- \n" - "1.9.1\n" - "\n" - "--\n" - "To unsubscribe, send a message with 'unsubscribe linux-mm' in\n" - "the body to majordomo@kvack.org. For more info on Linux MM,\n" - "see: http://www.linux-mm.org/ .\n" - "Don't email: <a href=mailto:\"dont@kvack.org\"> email@kvack.org </a>" + After I resolve all thing, I will send v7 rebased on recent mmotm. -2fa3355d740734ac1a27f0a6983e3b2c06c93ea900abfc7387590d05b257c857 +8867f12be355db7e048f6059a50bf36c38f45eebaf76915947fc5870fefc60df
diff --git a/a/1.txt b/N2/1.txt index 3ed70b1..8082ddd 100644 --- a/a/1.txt +++ b/N2/1.txt @@ -933,9 +933,3 @@ index 917e0e3d0f8e..b756ee36f7f0 100644 int __page_mapcount(struct page *page) -- 1.9.1 - --- -To unsubscribe, send a message with 'unsubscribe linux-mm' in -the body to majordomo@kvack.org. For more info on Linux MM, -see: http://www.linux-mm.org/ . -Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> diff --git a/a/content_digest b/N2/content_digest index 25ea410..9164cc1 100644 --- a/a/content_digest +++ b/N2/content_digest @@ -5,18 +5,18 @@ "Subject\0[PATCH v6v3 02/12] mm: migrate: support non-lru movable page migration\0" "Date\0Tue, 31 May 2016 09:01:17 +0900\0" "To\0Andrew Morton <akpm@linux-foundation.org>\0" - "Cc\0linux-mm@kvack.org" - linux-kernel@vger.kernel.org + "Cc\0<linux-mm@kvack.org>" + <linux-kernel@vger.kernel.org> Rik van Riel <riel@redhat.com> Vlastimil Babka <vbabka@suse.cz> Joonsoo Kim <iamjoonsoo.kim@lge.com> Mel Gorman <mgorman@suse.de> Hugh Dickins <hughd@google.com> Rafael Aquini <aquini@redhat.com> - virtualization@lists.linux-foundation.org + <virtualization@lists.linux-foundation.org> Jonathan Corbet <corbet@lwn.net> John Einar Reitan <john.reitan@foss.arm.com> - dri-devel@lists.freedesktop.org + <dri-devel@lists.freedesktop.org> Sergey Senozhatsky <sergey.senozhatsky@gmail.com> " Gioh Kim <gi-oh.kim@profitbricks.com>\0" "\00:1\0" @@ -955,12 +955,6 @@ " /* Slow path of page_mapcount() for compound pages */\n" " int __page_mapcount(struct page *page)\n" "-- \n" - "1.9.1\n" - "\n" - "--\n" - "To unsubscribe, send a message with 'unsubscribe linux-mm' in\n" - "the body to majordomo@kvack.org. For more info on Linux MM,\n" - "see: http://www.linux-mm.org/ .\n" - "Don't email: <a href=mailto:\"dont@kvack.org\"> email@kvack.org </a>" + 1.9.1 -2fa3355d740734ac1a27f0a6983e3b2c06c93ea900abfc7387590d05b257c857 +a8d3b6adc0ccb143551854cca08e374bb39358bb25db36c6e095574125417810
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.