* [PATCH RFC v0 1/6] pagevec: segmented page vectors
2015-06-15 7:50 [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Konstantin Khlebnikov
@ 2015-06-15 7:51 ` Konstantin Khlebnikov
2015-06-15 7:51 ` [PATCH RFC v0 2/6] mm/migrate: move putback of old page out of unmap_and_move Konstantin Khlebnikov
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Konstantin Khlebnikov @ 2015-06-15 7:51 UTC (permalink / raw)
To: linux-mm
This patch adds helpers for linking several page vectors into
segmented chain and macro for iterating over this chain.
For linking it uses space formerly used for field 'cold' which
is now stored as a lower bit in pointer to the next segment.
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
---
include/linux/pagevec.h | 48 ++++++++++++++++++++++++++++++++++++++++++++---
mm/swap.c | 44 +++++++++++++++++++++++++++++++++++++++++--
2 files changed, 87 insertions(+), 5 deletions(-)
diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index b45d391..de3ea58 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -16,7 +16,7 @@ struct address_space;
struct pagevec {
unsigned long nr;
- unsigned long cold;
+ unsigned long _next;
struct page *pages[PAGEVEC_SIZE];
};
@@ -32,11 +32,23 @@ unsigned pagevec_lookup(struct pagevec *pvec, struct address_space *mapping,
unsigned pagevec_lookup_tag(struct pagevec *pvec,
struct address_space *mapping, pgoff_t *index, int tag,
unsigned nr_pages);
+struct pagevec *pagevec_extend(struct pagevec *pvec, gfp_t gfpmask);
+void pagevec_shrink(struct pagevec *pvec);
-static inline void pagevec_init(struct pagevec *pvec, int cold)
+static inline bool pagevec_cold(struct pagevec *pvec)
+{
+ return pvec->_next & 1;
+}
+
+static inline struct pagevec *pagevec_next(struct pagevec *pvec)
+{
+ return (struct pagevec *)(pvec->_next & ~1ul);
+}
+
+static inline void pagevec_init(struct pagevec *pvec, bool cold)
{
pvec->nr = 0;
- pvec->cold = cold;
+ pvec->_next = cold;
}
static inline void pagevec_reinit(struct pagevec *pvec)
@@ -69,4 +81,34 @@ static inline void pagevec_release(struct pagevec *pvec)
__pagevec_release(pvec);
}
+/**
+ * pagevec_for_each_page - iterate over all pages in single page vector *
+ * @pv pointer to page vector
+ * @i int variable used as index
+ * @page pointer to struct page
+ */
+#define pagevec_for_each_page(pv, i, page) \
+ for (i = 0; (i < (pv)->nr) && (page = (pv)->pages[i], true); i++)
+
+/**
+ * pagevec_for_each_vec - iterate over all segments in page vector
+ * @pv pointer to head page vector
+ * @v pointer for current page vector segment
+ */
+#define pagevec_for_each_vec(pv, v) \
+ for (v = (pv); v; v = pagevec_next(v))
+
+/**
+ * pagevec_for_each_vec_and_page - iterate over all pages in segmented vector
+ * @pv pointer to head page vector
+ * @v pointer for current page vector segment
+ * @i int variable used index in current segment
+ * @page pointer to struct page
+ *
+ * Warning: this is double loop, "break" does not work.
+ */
+#define pagevec_for_each_vec_and_page(pv, v, i, page) \
+ pagevec_for_each_vec(pv, v) \
+ pagevec_for_each_page(v, i, page)
+
#endif /* _LINUX_PAGEVEC_H */
diff --git a/mm/swap.c b/mm/swap.c
index a7251a8..3ec0eb5 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -32,6 +32,7 @@
#include <linux/gfp.h>
#include <linux/uio.h>
#include <linux/hugetlb.h>
+#include <linux/slab.h>
#include "internal.h"
@@ -440,7 +441,7 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,
}
if (zone)
spin_unlock_irqrestore(&zone->lru_lock, flags);
- release_pages(pvec->pages, pvec->nr, pvec->cold);
+ release_pages(pvec->pages, pagevec_count(pvec), pagevec_cold(pvec));
pagevec_reinit(pvec);
}
@@ -982,7 +983,7 @@ EXPORT_SYMBOL(release_pages);
void __pagevec_release(struct pagevec *pvec)
{
lru_add_drain();
- release_pages(pvec->pages, pagevec_count(pvec), pvec->cold);
+ release_pages(pvec->pages, pagevec_count(pvec), pagevec_cold(pvec));
pagevec_reinit(pvec);
}
EXPORT_SYMBOL(__pagevec_release);
@@ -1137,6 +1138,45 @@ unsigned pagevec_lookup_tag(struct pagevec *pvec, struct address_space *mapping,
}
EXPORT_SYMBOL(pagevec_lookup_tag);
+/**
+ * pagevec_extend - allocate, initialize and link next page vector
+ * @pvec: page vector for extending
+ * @gfpmask:
+ *
+ * Returns pointer to new page vector or NULL if allocation failed.
+ */
+struct pagevec *pagevec_extend(struct pagevec *pvec, gfp_t gfpmask)
+{
+ struct pagevec *next;
+
+ next = kmalloc(sizeof(struct pagevec), gfpmask);
+ if (next) {
+ bool cold = pagevec_cold(pvec);
+ pagevec_init(next, cold);
+ pvec->_next = (unsigned long)next + cold;
+ }
+ return next;
+}
+
+/**
+ * pagevec_shrink - free all following page vectors segments
+ * @pvec: head page vector
+ *
+ * Head vector and pages are not freed.
+ */
+void pagevec_shrink(struct pagevec *head)
+{
+ struct pagevec *pvec, *next;
+
+ pvec = pagevec_next(head);
+ head->_next &= 1ul;
+ while (pvec) {
+ next = pagevec_next(pvec);
+ kfree(pvec);
+ pvec = next;
+ }
+}
+
/*
* Perform any setup for the swap system
*/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH RFC v0 2/6] mm/migrate: move putback of old page out of unmap_and_move
2015-06-15 7:50 [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Konstantin Khlebnikov
2015-06-15 7:51 ` [PATCH RFC v0 1/6] pagevec: segmented page vectors Konstantin Khlebnikov
@ 2015-06-15 7:51 ` Konstantin Khlebnikov
2015-06-15 7:51 ` [PATCH RFC v0 3/6] mm/cma: repalce reclaim_clean_pages_from_list with try_to_reclaim_page Konstantin Khlebnikov
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Konstantin Khlebnikov @ 2015-06-15 7:51 UTC (permalink / raw)
To: linux-mm
This is preparation for migrating non-isolated pages.
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
---
mm/migrate.c | 34 ++++++++++++++--------------------
1 file changed, 14 insertions(+), 20 deletions(-)
diff --git a/mm/migrate.c b/mm/migrate.c
index f53838f..eca80b3 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -939,19 +939,6 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page,
rc = __unmap_and_move(page, newpage, force, mode);
out:
- if (rc != -EAGAIN) {
- /*
- * A page that has been migrated has all references
- * removed and will be freed. A page that has not been
- * migrated will have kepts its references and be
- * restored.
- */
- list_del(&page->lru);
- dec_zone_page_state(page, NR_ISOLATED_ANON +
- page_is_file_cache(page));
- putback_lru_page(page);
- }
-
/*
* If migration was not successful and there's a freeing callback, use
* it. Otherwise, putback_lru_page() will drop the reference grabbed
@@ -1011,10 +998,8 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,
* tables or check whether the hugepage is pmd-based or not before
* kicking migration.
*/
- if (!hugepage_migration_supported(page_hstate(hpage))) {
- putback_active_hugepage(hpage);
+ if (!hugepage_migration_supported(page_hstate(hpage)))
return -ENOSYS;
- }
new_hpage = get_new_page(hpage, private, &result);
if (!new_hpage)
@@ -1051,9 +1036,6 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,
unlock_page(hpage);
out:
- if (rc != -EAGAIN)
- putback_active_hugepage(hpage);
-
/*
* If migration was not successful and there's a freeing callback, use
* it. Otherwise, put_page() will drop the reference grabbed during
@@ -1129,7 +1111,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
goto out;
case -EAGAIN:
retry++;
- break;
+ /* Keep that page for next pass */
+ continue;
case MIGRATEPAGE_SUCCESS:
nr_succeeded++;
break;
@@ -1143,6 +1126,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
nr_failed++;
break;
}
+
+ /* Putback migrated or permanently failed page. */
+ if (PageHuge(page)) {
+ putback_active_hugepage(page);
+ } else {
+ list_del(&page->lru);
+ if (unlikely(isolated_balloon_page(page)))
+ balloon_page_putback(page);
+ else
+ putback_lru_page(page);
+ }
}
}
rc = nr_failed + retry;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH RFC v0 3/6] mm/cma: repalce reclaim_clean_pages_from_list with try_to_reclaim_page
2015-06-15 7:50 [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Konstantin Khlebnikov
2015-06-15 7:51 ` [PATCH RFC v0 1/6] pagevec: segmented page vectors Konstantin Khlebnikov
2015-06-15 7:51 ` [PATCH RFC v0 2/6] mm/migrate: move putback of old page out of unmap_and_move Konstantin Khlebnikov
@ 2015-06-15 7:51 ` Konstantin Khlebnikov
2015-06-15 7:51 ` [PATCH RFC v0 4/6] mm/migrate: page migration without page isolation Konstantin Khlebnikov
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Konstantin Khlebnikov @ 2015-06-15 7:51 UTC (permalink / raw)
To: linux-mm
This gives almost the same behavior but makes code much less ugly.
Reclaimer works only with isolated pages, try_to_reclaim_page doesn't
require that. Of course it fails if page is currently isolated by
somebody else because in this case page has elevated refcount.
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
---
include/linux/mm.h | 1 +
mm/filemap.c | 20 ++++++++++++++++++++
mm/internal.h | 2 --
mm/page_alloc.c | 13 +++++++++----
mm/vmscan.c | 42 +++++-------------------------------------
5 files changed, 35 insertions(+), 43 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0755b9f..ed1e76bb 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1204,6 +1204,7 @@ int get_kernel_page(unsigned long start, int write, struct page **pages);
struct page *get_dump_page(unsigned long addr);
extern int try_to_release_page(struct page * page, gfp_t gfp_mask);
+extern int try_to_reclaim_page(struct page *page);
extern void do_invalidatepage(struct page *page, unsigned int offset,
unsigned int length);
diff --git a/mm/filemap.c b/mm/filemap.c
index 6bf5e42..a06324d 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2663,3 +2663,23 @@ int try_to_release_page(struct page *page, gfp_t gfp_mask)
}
EXPORT_SYMBOL(try_to_release_page);
+
+/**
+ * try_to_reclaim_page() - unmap and invalidate clean page cache page
+ *
+ * @page: the page which the kernel is trying to free
+ */
+int try_to_reclaim_page(struct page *page)
+{
+ int ret;
+
+ if (PageDirty(page) || PageWriteback(page))
+ return 0;
+ if (!trylock_page(page))
+ return 0;
+ if (page_mapped(page))
+ try_to_unmap(page, TTU_UNMAP | TTU_IGNORE_ACCESS);
+ ret = invalidate_inode_page(page);
+ unlock_page(page);
+ return ret;
+}
diff --git a/mm/internal.h b/mm/internal.h
index a25e359..1cf2eb9 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -416,8 +416,6 @@ extern unsigned long vm_mmap_pgoff(struct file *, unsigned long,
unsigned long, unsigned long);
extern void set_pageblock_order(void);
-unsigned long reclaim_clean_pages_from_list(struct zone *zone,
- struct list_head *page_list);
/* The ALLOC_WMARK bits are used as an index to zone->watermark */
#define ALLOC_WMARK_MIN WMARK_MIN
#define ALLOC_WMARK_LOW WMARK_LOW
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index ebffa0e..9adf4d07 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6341,9 +6341,9 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
unsigned long start, unsigned long end)
{
/* This function is based on compact_zone() from compaction.c. */
- unsigned long nr_reclaimed;
unsigned long pfn = start;
unsigned int tries = 0;
+ struct page *page;
int ret = 0;
migrate_prep();
@@ -6367,9 +6367,14 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
break;
}
- nr_reclaimed = reclaim_clean_pages_from_list(cc->zone,
- &cc->migratepages);
- cc->nr_migratepages -= nr_reclaimed;
+ /*
+ * Try to reclaim clean page cache pages.
+ * Migration simply skips pages where page_count == 1.
+ */
+ list_for_each_entry(page, &cc->migratepages, lru) {
+ if (!PageAnon(page))
+ try_to_reclaim_page(page);
+ }
ret = migrate_pages(&cc->migratepages, alloc_migrate_target,
NULL, 0, cc->mode, MR_CMA);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 5e8eadd..ae2d50d 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -843,13 +843,11 @@ static void page_check_dirty_writeback(struct page *page,
static unsigned long shrink_page_list(struct list_head *page_list,
struct zone *zone,
struct scan_control *sc,
- enum ttu_flags ttu_flags,
unsigned long *ret_nr_dirty,
unsigned long *ret_nr_unqueued_dirty,
unsigned long *ret_nr_congested,
unsigned long *ret_nr_writeback,
- unsigned long *ret_nr_immediate,
- bool force_reclaim)
+ unsigned long *ret_nr_immediate)
{
LIST_HEAD(ret_pages);
LIST_HEAD(free_pages);
@@ -991,8 +989,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
}
}
- if (!force_reclaim)
- references = page_check_references(page, sc);
+ references = page_check_references(page, sc);
switch (references) {
case PAGEREF_ACTIVATE:
@@ -1024,7 +1021,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
* processes. Try to unmap it here.
*/
if (page_mapped(page) && mapping) {
- switch (try_to_unmap(page, ttu_flags)) {
+ switch (try_to_unmap(page, TTU_UNMAP)) {
case SWAP_FAIL:
goto activate_locked;
case SWAP_AGAIN:
@@ -1188,34 +1185,6 @@ keep:
return nr_reclaimed;
}
-unsigned long reclaim_clean_pages_from_list(struct zone *zone,
- struct list_head *page_list)
-{
- struct scan_control sc = {
- .gfp_mask = GFP_KERNEL,
- .priority = DEF_PRIORITY,
- .may_unmap = 1,
- };
- unsigned long ret, dummy1, dummy2, dummy3, dummy4, dummy5;
- struct page *page, *next;
- LIST_HEAD(clean_pages);
-
- list_for_each_entry_safe(page, next, page_list, lru) {
- if (page_is_file_cache(page) && !PageDirty(page) &&
- !isolated_balloon_page(page)) {
- ClearPageActive(page);
- list_move(&page->lru, &clean_pages);
- }
- }
-
- ret = shrink_page_list(&clean_pages, zone, &sc,
- TTU_UNMAP|TTU_IGNORE_ACCESS,
- &dummy1, &dummy2, &dummy3, &dummy4, &dummy5, true);
- list_splice(&clean_pages, page_list);
- mod_zone_page_state(zone, NR_ISOLATED_FILE, -ret);
- return ret;
-}
-
/*
* Attempt to remove the specified page from its LRU. Only take this page
* if it is of the appropriate PageActive status. Pages which are being
@@ -1563,10 +1532,9 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
if (nr_taken == 0)
return 0;
- nr_reclaimed = shrink_page_list(&page_list, zone, sc, TTU_UNMAP,
+ nr_reclaimed = shrink_page_list(&page_list, zone, sc,
&nr_dirty, &nr_unqueued_dirty, &nr_congested,
- &nr_writeback, &nr_immediate,
- false);
+ &nr_writeback, &nr_immediate);
spin_lock_irq(&zone->lru_lock);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH RFC v0 4/6] mm/migrate: page migration without page isolation
2015-06-15 7:50 [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Konstantin Khlebnikov
` (2 preceding siblings ...)
2015-06-15 7:51 ` [PATCH RFC v0 3/6] mm/cma: repalce reclaim_clean_pages_from_list with try_to_reclaim_page Konstantin Khlebnikov
@ 2015-06-15 7:51 ` Konstantin Khlebnikov
2015-06-15 7:51 ` [PATCH RFC v0 5/6] mm/compaction: use migration without isolation Konstantin Khlebnikov
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Konstantin Khlebnikov @ 2015-06-15 7:51 UTC (permalink / raw)
To: linux-mm
migrate_pagevec() does the same job as migrate_pages() but it works with
chained page vector instead of list of isolated pages.
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
---
include/linux/migrate.h | 4 ++
mm/migrate.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 102 insertions(+)
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index cac1c09..04553f5 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -33,6 +33,10 @@ extern int migrate_page(struct address_space *,
struct page *, struct page *, enum migrate_mode);
extern int migrate_pages(struct list_head *l, new_page_t new, free_page_t free,
unsigned long private, enum migrate_mode mode, int reason);
+struct pagevec;
+extern int migrate_pagevec(struct pagevec *pages, new_page_t get_new_page,
+ free_page_t put_new_page, unsigned long private,
+ enum migrate_mode mode, int reason);
extern int migrate_prep(void);
extern int migrate_prep_local(void);
diff --git a/mm/migrate.c b/mm/migrate.c
index eca80b3..775cc9d 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1153,6 +1153,104 @@ out:
return rc;
}
+/*
+ * migrate_pagevec - migrate the pages specified in a page vector, to the free
+ * pages supplied as the target for the page migration
+ *
+ * @pages: The vector of pages to be migrated.
+ * @get_new_page: The function used to allocate free pages to be used
+ * as the target of the page migration.
+ * @put_new_page: The function used to free target pages if migration
+ * fails, or NULL if no special handling is necessary.
+ * @private: Private data to be passed on to get_new_page()
+ * @mode: The migration mode that specifies the constraints for
+ * page migration, if any.
+ * @reason: The reason for page migration.
+ *
+ * The function returns after 10 attempts or if no pages are movable any more
+ * because the vector has become empty or no retryable pages exist any more.
+ * This function keeps all pages in the page vector but reorders them.
+ *
+ * Returns the number of pages that were not migrated, or an error code.
+ */
+int migrate_pagevec(struct pagevec *pages, new_page_t get_new_page,
+ free_page_t put_new_page, unsigned long private,
+ enum migrate_mode mode, int reason)
+{
+ int nr_to_scan = INT_MAX;
+ int nr_failed = 0;
+ int nr_succeeded = 0;
+ int nr_retry, pass;
+ struct page *page;
+ int swapwrite = current->flags & PF_SWAPWRITE;
+ int rc;
+
+ if (!swapwrite)
+ current->flags |= PF_SWAPWRITE;
+
+ for (pass = 0; pass < 10 && nr_to_scan; pass++) {
+ struct pagevec *pvec, *retry_pvec = pages;
+ int index, retry_index = 0;
+
+ nr_retry = 0;
+ pagevec_for_each_vec_and_page(pages, pvec, index, page) {
+ cond_resched();
+
+ if (!nr_to_scan--)
+ goto next_pass;
+
+ if (PageHuge(page))
+ rc = unmap_and_move_huge_page(get_new_page,
+ put_new_page, private, page,
+ pass > 2, mode);
+ else
+ rc = unmap_and_move(get_new_page, put_new_page,
+ private, page, pass > 2, mode);
+
+ switch(rc) {
+ case -ENOMEM:
+ goto out;
+ case -EAGAIN:
+ nr_retry++;
+ /* move page to the head for next pass */
+ swap(pvec->pages[index],
+ retry_pvec->pages[retry_index]);
+ if (++retry_index == retry_pvec->nr) {
+ retry_pvec = pagevec_next(retry_pvec);
+ retry_index = 0;
+ }
+ break;
+ case MIGRATEPAGE_SUCCESS:
+ nr_succeeded++;
+ break;
+ default:
+ /*
+ * Permanent failure (-EBUSY, -ENOSYS, etc.):
+ * unlike -EAGAIN case, the failed page is
+ * removed from migration page list and not
+ * retried in the next outer loop.
+ */
+ nr_failed++;
+ break;
+ }
+ }
+next_pass:
+ nr_to_scan = nr_retry;
+ }
+ rc = nr_failed + nr_retry;
+out:
+ if (nr_succeeded)
+ count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded);
+ if (nr_failed)
+ count_vm_events(PGMIGRATE_FAIL, nr_failed);
+ trace_mm_migrate_pages(nr_succeeded, nr_failed, mode, reason);
+
+ if (!swapwrite)
+ current->flags &= ~PF_SWAPWRITE;
+
+ return rc;
+}
+
#ifdef CONFIG_NUMA
/*
* Move a list of individual pages
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH RFC v0 5/6] mm/compaction: use migration without isolation
2015-06-15 7:50 [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Konstantin Khlebnikov
` (3 preceding siblings ...)
2015-06-15 7:51 ` [PATCH RFC v0 4/6] mm/migrate: page migration without page isolation Konstantin Khlebnikov
@ 2015-06-15 7:51 ` Konstantin Khlebnikov
2015-06-15 7:51 ` [PATCH RFC v0 6/6] mm/migrate: preserve lru order if possible Konstantin Khlebnikov
2015-06-15 9:52 ` [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Vlastimil Babka
6 siblings, 0 replies; 8+ messages in thread
From: Konstantin Khlebnikov @ 2015-06-15 7:51 UTC (permalink / raw)
To: linux-mm
TODO
* fix interaction with too_many_isolated (account pages as pinned?)
* rename the rest of isolated_* stuff, pages are not isolated
* fix racy check page->mapping->a_ops->migratepage
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
---
include/trace/events/compaction.h | 12 +-
mm/compaction.c | 205 +++++++++++++++++++++----------------
mm/internal.h | 9 +-
mm/migrate.c | 2
mm/page_alloc.c | 24 ++--
5 files changed, 142 insertions(+), 110 deletions(-)
diff --git a/include/trace/events/compaction.h b/include/trace/events/compaction.h
index 9a6a3fe..c5b3260 100644
--- a/include/trace/events/compaction.h
+++ b/include/trace/events/compaction.h
@@ -66,7 +66,7 @@ TRACE_EVENT(mm_compaction_migratepages,
TP_PROTO(unsigned long nr_all,
int migrate_rc,
- struct list_head *migratepages),
+ struct pagevec *migratepages),
TP_ARGS(nr_all, migrate_rc, migratepages),
@@ -77,7 +77,9 @@ TRACE_EVENT(mm_compaction_migratepages,
TP_fast_assign(
unsigned long nr_failed = 0;
- struct list_head *page_lru;
+ struct pagevec *pvec;
+ struct page *page;
+ int i;
/*
* migrate_pages() returns either a non-negative number
@@ -88,8 +90,10 @@ TRACE_EVENT(mm_compaction_migratepages,
if (migrate_rc >= 0)
nr_failed = migrate_rc;
else
- list_for_each(page_lru, migratepages)
- nr_failed++;
+ pagevec_for_each_vec_and_page(migratepages,
+ pvec, i, page)
+ if (page_count(page) != 1)
+ nr_failed++;
__entry->nr_migrated = nr_all - nr_failed;
__entry->nr_failed = nr_failed;
diff --git a/mm/compaction.c b/mm/compaction.c
index 018f08d..9f3fe19 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -378,6 +378,11 @@ static bool compact_unlock_should_abort(spinlock_t *lock,
*/
static inline bool compact_should_abort(struct compact_control *cc)
{
+ if (fatal_signal_pending(current)) {
+ cc->contended = COMPACT_CONTENDED_SCHED;
+ return true;
+ }
+
/* async compaction aborts if contended */
if (need_resched()) {
if (cc->mode == MIGRATE_ASYNC) {
@@ -591,22 +596,6 @@ isolate_freepages_range(struct compact_control *cc,
return pfn;
}
-/* Update the number of anon and file isolated pages in the zone */
-static void acct_isolated(struct zone *zone, struct compact_control *cc)
-{
- struct page *page;
- unsigned int count[2] = { 0, };
-
- if (list_empty(&cc->migratepages))
- return;
-
- list_for_each_entry(page, &cc->migratepages, lru)
- count[!!page_is_file_cache(page)]++;
-
- mod_zone_page_state(zone, NR_ISOLATED_ANON, count[0]);
- mod_zone_page_state(zone, NR_ISOLATED_FILE, count[1]);
-}
-
/* Similar to reclaim, but different enough that they don't share logic */
static bool too_many_isolated(struct zone *zone)
{
@@ -623,7 +612,7 @@ static bool too_many_isolated(struct zone *zone)
}
/**
- * isolate_migratepages_block() - isolate all migrate-able pages within
+ * collect_migratepages_block() - collect all migrate-able pages within
* a single pageblock
* @cc: Compaction control structure.
* @low_pfn: The first PFN to isolate
@@ -641,15 +630,11 @@ static bool too_many_isolated(struct zone *zone)
* is neither read nor updated.
*/
static unsigned long
-isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
+collect_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
unsigned long end_pfn, isolate_mode_t isolate_mode)
{
struct zone *zone = cc->zone;
unsigned long nr_scanned = 0, nr_isolated = 0;
- struct list_head *migratelist = &cc->migratepages;
- struct lruvec *lruvec;
- unsigned long flags = 0;
- bool locked = false;
struct page *page = NULL, *valid_page = NULL;
unsigned long start_pfn = low_pfn;
@@ -672,6 +657,10 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
if (compact_should_abort(cc))
return 0;
+ /* pagevec_extend() has failed */
+ if (!cc->migratepages_tail)
+ return 0;
+
/* Time to isolate some pages for migration */
for (; low_pfn < end_pfn; low_pfn++) {
/*
@@ -679,9 +668,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
* contention, to give chance to IRQs. Abort async compaction
* if contended.
*/
- if (!(low_pfn % SWAP_CLUSTER_MAX)
- && compact_unlock_should_abort(&zone->lru_lock, flags,
- &locked, cc))
+ if (!(low_pfn % SWAP_CLUSTER_MAX) &&
+ compact_should_abort(cc))
break;
if (!pfn_valid_within(low_pfn))
@@ -728,22 +716,13 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
}
/*
- * PageLRU is set. lru_lock normally excludes isolation
- * splitting and collapsing (collapsing has already happened
- * if PageLRU is set) but the lock is not necessarily taken
- * here and it is wasteful to take it just to check transhuge.
- * Check TransHuge without lock and skip the whole pageblock if
- * it's either a transhuge or hugetlbfs page, as calling
- * compound_order() without preventing THP from splitting the
- * page underneath us may return surprising results.
+ * Check PageTransCompound without lock and skip the whole
+ * pageblock if it's either a transhuge or hugetlbfs page,
+ * as calling compound_order() without preventing THP from
+ * splitting the page may return surprising results.
*/
- if (PageTransHuge(page)) {
- if (!locked)
- low_pfn = ALIGN(low_pfn + 1,
- pageblock_nr_pages) - 1;
- else
- low_pfn += (1 << compound_order(page)) - 1;
-
+ if (PageTransCompound(page)) {
+ low_pfn = ALIGN(low_pfn + 1, pageblock_nr_pages) - 1;
continue;
}
@@ -756,37 +735,60 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
page_count(page) > page_mapcount(page))
continue;
- /* If we already hold the lock, we can skip some rechecking */
- if (!locked) {
- locked = compact_trylock_irqsave(&zone->lru_lock,
- &flags, cc);
- if (!locked)
- break;
+ /* Compaction should not handle unevictable pages but CMA can do so */
+ if (PageUnevictable(page) &&
+ !(isolate_mode & ISOLATE_UNEVICTABLE))
+ continue;
- /* Recheck PageLRU and PageTransHuge under lock */
- if (!PageLRU(page))
- continue;
- if (PageTransHuge(page)) {
- low_pfn += (1 << compound_order(page)) - 1;
+ /*
+ * ISOLATE_ASYNC_MIGRATE is used to indicate that it only wants to pages
+ * that it is possible to migrate without blocking
+ */
+ if (isolate_mode & ISOLATE_ASYNC_MIGRATE) {
+ /* All the caller can do on PageWriteback is block */
+ if (PageWriteback(page))
continue;
+
+ if (PageDirty(page)) {
+ struct address_space *mapping;
+
+ /*
+ * Only pages without mappings or that have
+ * a ->migratepage callback are possible to
+ * migrate without blocking.
+ *
+ * FIXME this is unsafe without page_lock
+ * present __isolate_lru_page does this too
+ */
+ mapping = page_mapping(page);
+ if (mapping && !mapping->a_ops->migratepage)
+ continue;
}
}
- lruvec = mem_cgroup_page_lruvec(page, zone);
-
- /* Try isolate the page */
- if (__isolate_lru_page(page, isolate_mode) != 0)
+ if (!get_page_unless_zero(page))
continue;
- VM_BUG_ON_PAGE(PageTransCompound(page), page);
-
- /* Successfully isolated */
- del_page_from_lru_list(page, lruvec, page_lru(page));
+ /*
+ * Without PageLRU that might any type of kernel page
+ * we could also check page->mapping but without PageLRU
+ * migraion likely fails because elevated page refcount.
+ */
+ if (!PageLRU(page) || PageTransCompound(page)) {
+ VM_BUG_ON_PAGE(PageTail(page), page);
+ put_page(page);
+ continue;
+ }
isolate_success:
- list_add(&page->lru, migratelist);
- cc->nr_migratepages++;
nr_isolated++;
+ cc->nr_migratepages++;
+ if (!pagevec_add(cc->migratepages_tail, page)) {
+ cc->migratepages_tail = pagevec_extend(
+ cc->migratepages_tail, GFP_ATOMIC);
+ if (!cc->migratepages_tail)
+ return 0;
+ }
/* Avoid isolating too much */
if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) {
@@ -802,9 +804,6 @@ isolate_success:
if (unlikely(low_pfn > end_pfn))
low_pfn = end_pfn;
- if (locked)
- spin_unlock_irqrestore(&zone->lru_lock, flags);
-
/*
* Update the pageblock-skip information and cached scanner pfn,
* if the whole pageblock was scanned without isolating any page.
@@ -823,7 +822,7 @@ isolate_success:
}
/**
- * isolate_migratepages_range() - isolate migrate-able pages in a PFN range
+ * collect_migratepages_range() - collect migrate-able pages in a PFN range
* @cc: Compaction control structure.
* @start_pfn: The first PFN to start isolating.
* @end_pfn: The one-past-last PFN.
@@ -833,7 +832,7 @@ isolate_success:
* (which may be greater than end_pfn if end fell in a middle of a THP page).
*/
unsigned long
-isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn,
+collect_migratepages_range(struct compact_control *cc, unsigned long start_pfn,
unsigned long end_pfn)
{
unsigned long pfn, block_end_pfn;
@@ -850,8 +849,8 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn,
if (!pageblock_pfn_to_page(pfn, block_end_pfn, cc->zone))
continue;
- pfn = isolate_migratepages_block(cc, pfn, block_end_pfn,
- ISOLATE_UNEVICTABLE);
+ pfn = collect_migratepages_block(cc, pfn, block_end_pfn,
+ ISOLATE_UNEVICTABLE);
/*
* In case of fatal failure, release everything that might
@@ -859,19 +858,28 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn,
* the failure back to caller.
*/
if (!pfn) {
- putback_movable_pages(&cc->migratepages);
- cc->nr_migratepages = 0;
+ release_migratepages(cc);
break;
}
if (cc->nr_migratepages == COMPACT_CLUSTER_MAX)
break;
}
- acct_isolated(cc->zone, cc);
return pfn;
}
+void release_migratepages(struct compact_control *cc)
+{
+ struct pagevec *pvec;
+
+ pagevec_for_each_vec(&cc->migratepages, pvec)
+ pagevec_release(pvec);
+ pagevec_shrink(&cc->migratepages);
+ cc->migratepages_tail = &cc->migratepages;
+ cc->nr_migratepages = 0;
+}
+
#endif /* CONFIG_COMPACTION || CONFIG_CMA */
#ifdef CONFIG_COMPACTION
@@ -1040,7 +1048,7 @@ static void compaction_free(struct page *page, unsigned long data)
cc->nr_freepages++;
}
-/* possible outcome of isolate_migratepages */
+/* possible outcome of collect_migratepages */
typedef enum {
ISOLATE_ABORT, /* Abort compaction now */
ISOLATE_NONE, /* No pages isolated, continue scanning */
@@ -1058,7 +1066,7 @@ int sysctl_compact_unevictable_allowed __read_mostly = 1;
* starting at the block pointed to by the migrate scanner pfn within
* compact_control.
*/
-static isolate_migrate_t isolate_migratepages(struct zone *zone,
+static isolate_migrate_t collect_migratepages(struct zone *zone,
struct compact_control *cc)
{
unsigned long low_pfn, end_pfn;
@@ -1109,14 +1117,11 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,
!migrate_async_suitable(get_pageblock_migratetype(page)))
continue;
- /* Perform the isolation */
- low_pfn = isolate_migratepages_block(cc, low_pfn, end_pfn,
+ low_pfn = collect_migratepages_block(cc, low_pfn, end_pfn,
isolate_mode);
- if (!low_pfn || cc->contended) {
- acct_isolated(zone, cc);
+ if (!low_pfn || cc->contended)
return ISOLATE_ABORT;
- }
/*
* Either we isolated something and proceed with migration. Or
@@ -1126,7 +1131,6 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,
break;
}
- acct_isolated(zone, cc);
/*
* Record where migration scanner will be restarted. If we end up in
* the same pageblock as the free scanner, make the scanners fully
@@ -1344,11 +1348,14 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
int err;
unsigned long isolate_start_pfn = cc->migrate_pfn;
- switch (isolate_migratepages(zone, cc)) {
+ VM_BUG_ON(cc->nr_migratepages);
+ VM_BUG_ON(pagevec_count(&cc->migratepages));
+ VM_BUG_ON(pagevec_next(&cc->migratepages));
+
+ switch (collect_migratepages(zone, cc)) {
case ISOLATE_ABORT:
ret = COMPACT_PARTIAL;
- putback_movable_pages(&cc->migratepages);
- cc->nr_migratepages = 0;
+ release_migratepages(cc);
goto out;
case ISOLATE_NONE:
/*
@@ -1361,7 +1368,22 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
;
}
- err = migrate_pages(&cc->migratepages, compaction_alloc,
+ {
+ struct pagevec *pvec;
+ int index;
+ struct page *page;
+ int count = 0;
+
+ pagevec_for_each_vec_and_page(&cc->migratepages, pvec, index, page) {
+ VM_BUG_ON_PAGE(page_count(page) < 1, page);
+ VM_BUG_ON_PAGE(PageTransCompound(page), page);
+ count++;
+ }
+
+ VM_BUG_ON(count != cc->nr_migratepages);
+ }
+
+ err = migrate_pagevec(&cc->migratepages, compaction_alloc,
compaction_free, (unsigned long)cc, cc->mode,
MR_COMPACTION);
@@ -1369,11 +1391,10 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
&cc->migratepages);
/* All pages were either migrated or will be released */
- cc->nr_migratepages = 0;
+ release_migratepages(cc);
if (err) {
- putback_movable_pages(&cc->migratepages);
/*
- * migrate_pages() may return -ENOMEM when scanners meet
+ * migrate_pagevec() may return -ENOMEM when scanners meet
* and we want compact_finished() to detect it
*/
if (err == -ENOMEM && cc->free_pfn > cc->migrate_pfn) {
@@ -1385,7 +1406,7 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
/*
* Record where we could have freed pages by migration and not
* yet flushed them to buddy allocator. We use the pfn that
- * isolate_migratepages() started from in this loop iteration
+ * collect_migratepages() started from in this loop iteration
* - this is the lowest page that could have been isolated and
* then freed by migration.
*/
@@ -1459,12 +1480,13 @@ static unsigned long compact_zone_order(struct zone *zone, int order,
.classzone_idx = classzone_idx,
};
INIT_LIST_HEAD(&cc.freepages);
- INIT_LIST_HEAD(&cc.migratepages);
+ pagevec_init(&cc.migratepages, 0);
+ cc.migratepages_tail = &cc.migratepages;
ret = compact_zone(zone, &cc);
VM_BUG_ON(!list_empty(&cc.freepages));
- VM_BUG_ON(!list_empty(&cc.migratepages));
+ VM_BUG_ON(pagevec_count(&cc.migratepages));
*contended = cc.contended;
return ret;
@@ -1604,7 +1626,8 @@ static void __compact_pgdat(pg_data_t *pgdat, struct compact_control *cc)
cc->nr_migratepages = 0;
cc->zone = zone;
INIT_LIST_HEAD(&cc->freepages);
- INIT_LIST_HEAD(&cc->migratepages);
+ pagevec_init(&cc->migratepages, 0);
+ cc->migratepages_tail = &cc->migratepages;
/*
* When called via /proc/sys/vm/compact_memory
@@ -1624,7 +1647,7 @@ static void __compact_pgdat(pg_data_t *pgdat, struct compact_control *cc)
}
VM_BUG_ON(!list_empty(&cc->freepages));
- VM_BUG_ON(!list_empty(&cc->migratepages));
+ VM_BUG_ON(pagevec_count(&cc->migratepages));
}
}
diff --git a/mm/internal.h b/mm/internal.h
index 1cf2eb9..19081ba 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -13,6 +13,7 @@
#include <linux/fs.h>
#include <linux/mm.h>
+#include <linux/pagevec.h>
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
@@ -176,11 +177,12 @@ extern int user_min_free_kbytes;
*/
struct compact_control {
struct list_head freepages; /* List of free pages to migrate to */
- struct list_head migratepages; /* List of pages being migrated */
+ struct pagevec migratepages; /* Vector of pages being migrated */
+ struct pagevec *migratepages_tail;
unsigned long nr_freepages; /* Number of isolated free pages */
unsigned long nr_migratepages; /* Number of pages to migrate */
unsigned long free_pfn; /* isolate_freepages search base */
- unsigned long migrate_pfn; /* isolate_migratepages search base */
+ unsigned long migrate_pfn; /* collect_migratepages search base */
enum migrate_mode mode; /* Async or sync migration mode */
bool ignore_skip_hint; /* Scan blocks even if marked skip */
int order; /* order a direct compactor needs */
@@ -198,8 +200,9 @@ unsigned long
isolate_freepages_range(struct compact_control *cc,
unsigned long start_pfn, unsigned long end_pfn);
unsigned long
-isolate_migratepages_range(struct compact_control *cc,
+collect_migratepages_range(struct compact_control *cc,
unsigned long low_pfn, unsigned long end_pfn);
+void release_migratepages(struct compact_control *cc);
int find_suitable_fallback(struct free_area *area, unsigned int order,
int migratetype, bool only_stealable, bool *can_steal);
diff --git a/mm/migrate.c b/mm/migrate.c
index 775cc9d..c060991 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -76,7 +76,7 @@ int migrate_prep_local(void)
* from where they were once taken off for compaction/migration.
*
* This function shall be used whenever the isolated pageset has been
- * built from lru, balloon, hugetlbfs page. See isolate_migratepages_range()
+ * built from lru, balloon, hugetlbfs page. See collect_migratepages_range()
* and isolate_huge_page().
*/
void putback_movable_pages(struct list_head *l)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9adf4d07..ca37e71 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6343,20 +6343,21 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
/* This function is based on compact_zone() from compaction.c. */
unsigned long pfn = start;
unsigned int tries = 0;
+ struct pagevec *pvec;
struct page *page;
- int ret = 0;
+ int index, ret = 0;
migrate_prep();
- while (pfn < end || !list_empty(&cc->migratepages)) {
+ while (pfn < end || pagevec_count(&cc->migratepages)) {
if (fatal_signal_pending(current)) {
ret = -EINTR;
break;
}
- if (list_empty(&cc->migratepages)) {
+ if (!pagevec_count(&cc->migratepages)) {
cc->nr_migratepages = 0;
- pfn = isolate_migratepages_range(cc, pfn, end);
+ pfn = collect_migratepages_range(cc, pfn, end);
if (!pfn) {
ret = -EINTR;
break;
@@ -6371,18 +6372,18 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
* Try to reclaim clean page cache pages.
* Migration simply skips pages where page_count == 1.
*/
- list_for_each_entry(page, &cc->migratepages, lru) {
+ pagevec_for_each_vec_and_page(&cc->migratepages,
+ pvec, index, page) {
if (!PageAnon(page))
try_to_reclaim_page(page);
}
- ret = migrate_pages(&cc->migratepages, alloc_migrate_target,
- NULL, 0, cc->mode, MR_CMA);
+ ret = migrate_pagevec(&cc->migratepages, alloc_migrate_target,
+ NULL, 0, cc->mode, MR_CMA);
}
- if (ret < 0) {
- putback_movable_pages(&cc->migratepages);
+ release_migratepages(cc);
+ if (ret < 0)
return ret;
- }
return 0;
}
@@ -6419,7 +6420,8 @@ int alloc_contig_range(unsigned long start, unsigned long end,
.mode = MIGRATE_SYNC,
.ignore_skip_hint = true,
};
- INIT_LIST_HEAD(&cc.migratepages);
+ pagevec_init(&cc.migratepages, 0);
+ cc.migratepages_tail = &cc.migratepages;
/*
* What we do here is we mark all pageblocks in range as
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH RFC v0 6/6] mm/migrate: preserve lru order if possible
2015-06-15 7:50 [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Konstantin Khlebnikov
` (4 preceding siblings ...)
2015-06-15 7:51 ` [PATCH RFC v0 5/6] mm/compaction: use migration without isolation Konstantin Khlebnikov
@ 2015-06-15 7:51 ` Konstantin Khlebnikov
2015-06-15 9:52 ` [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Vlastimil Babka
6 siblings, 0 replies; 8+ messages in thread
From: Konstantin Khlebnikov @ 2015-06-15 7:51 UTC (permalink / raw)
To: linux-mm
TODO
* link old and new pages and insert them as a batch later
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
---
mm/internal.h | 1 +
mm/migrate.c | 7 +++++--
mm/swap.c | 25 +++++++++++++++++++++++++
3 files changed, 31 insertions(+), 2 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index 19081ba..6184fc2 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -99,6 +99,7 @@ extern unsigned long highest_memmap_pfn;
*/
extern int isolate_lru_page(struct page *page);
extern void putback_lru_page(struct page *page);
+extern void insert_lru_page(struct page *page, struct page *pos);
extern bool zone_reclaimable(struct zone *zone);
/*
diff --git a/mm/migrate.c b/mm/migrate.c
index c060991..e171981 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -947,10 +947,13 @@ out:
if (rc != MIGRATEPAGE_SUCCESS && put_new_page) {
ClearPageSwapBacked(newpage);
put_new_page(newpage, private);
- } else if (unlikely(__is_movable_balloon_page(newpage))) {
+ } else if (rc != MIGRATEPAGE_SUCCESS ||
+ unlikely(__is_movable_balloon_page(newpage))) {
/* drop our reference, page already in the balloon */
put_page(newpage);
- } else
+ } else if (PageLRU(page))
+ insert_lru_page(newpage, page);
+ else
putback_lru_page(newpage);
if (result) {
diff --git a/mm/swap.c b/mm/swap.c
index 3ec0eb5..40559d6 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -518,6 +518,31 @@ static void __activate_page(struct page *page, struct lruvec *lruvec,
}
}
+void insert_lru_page(struct page *page, struct page *pos)
+{
+ struct zone *zone = page_zone(page);
+ int lru = page_lru_base_type(page);
+ struct lruvec *lruvec;
+ unsigned long flags;
+
+ if (page_evictable(page) && lru == page_lru(pos) &&
+#ifdef CONFIG_MEMCG
+ page->mem_cgroup == pos->mem_cgroup &&
+#endif
+ zone == page_zone(pos)) {
+ spin_lock_irqsave(&zone->lru_lock, flags);
+ lruvec = mem_cgroup_page_lruvec(page, zone);
+ SetPageLRU(page);
+ add_page_to_lru_list(page, lruvec, lru);
+ trace_mm_lru_insertion(page, lru);
+ if (PageLRU(pos) &&
+ lruvec == mem_cgroup_page_lruvec(pos, zone))
+ list_move(&page->lru, &pos->lru);
+ spin_unlock_irqrestore(&zone->lru_lock, flags);
+ } else
+ putback_lru_page(page);
+}
+
#ifdef CONFIG_SMP
static DEFINE_PER_CPU(struct pagevec, activate_page_pvecs);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation
2015-06-15 7:50 [PATCH RFC v0 0/6] mm: proof-of-concept memory compaction without isolation Konstantin Khlebnikov
` (5 preceding siblings ...)
2015-06-15 7:51 ` [PATCH RFC v0 6/6] mm/migrate: preserve lru order if possible Konstantin Khlebnikov
@ 2015-06-15 9:52 ` Vlastimil Babka
6 siblings, 0 replies; 8+ messages in thread
From: Vlastimil Babka @ 2015-06-15 9:52 UTC (permalink / raw)
To: Konstantin Khlebnikov, linux-mm
On 06/15/2015 09:50 AM, Konstantin Khlebnikov wrote:
> This is incomplete implementation of non-isolating memory migration and
> compaction. It's alive!
>
> The main reason -- it can preserve lru order during compaction.
That's nice, and there's also another benefit - no lru_lock taken during
migration scanner.
So I think it's worth pursuing. But after brief checking, I'm not sure
it can work as it is (but maybe I just overlooked something). What
prevents somebody else to isolate the old page from the lru while you
are migrating it? Especially in Patch 6, it appears that a PageLRU() is
tested without any lock and only then insert_lru_page() takes the
lru_lock to do something. This seems racy to me.
>
> Also it makes implementation of migration for various types of pages: zram,
> balloon, ptes, kernel stacks [ Why not? I've already migrated them accidentally
> and kernel have crashed in very funny places ] much easier: owner just have to
> set page->mappingw with valid method a_ops->migratepage.
>
> ---
>
> Konstantin Khlebnikov (6):
> pagevec: segmented page vectors
> mm/migrate: move putback of old page out of unmap_and_move
> mm/cma: repalce reclaim_clean_pages_from_list with try_to_reclaim_page
> mm/migrate: page migration without page isolation
> mm/compaction: use migration without isolation
> mm/migrate: preserve lru order if possible
>
>
> include/linux/migrate.h | 4 +
> include/linux/mm.h | 1
> include/linux/pagevec.h | 48 ++++++++-
> include/trace/events/compaction.h | 12 +-
> mm/compaction.c | 205 +++++++++++++++++++++----------------
> mm/filemap.c | 20 ++++
> mm/internal.h | 12 +-
> mm/migrate.c | 141 +++++++++++++++++++++----
> mm/page_alloc.c | 35 ++++--
> mm/swap.c | 69 ++++++++++++
> mm/vmscan.c | 42 +-------
> 11 files changed, 410 insertions(+), 179 deletions(-)
>
> --
> Konstantin
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 8+ messages in thread