[PATCH v5] mm: support madvise(MADV_FREE)

[PATCH v5] mm: support madvise(MADV_FREE)

[Minchan Kim]

Linux doesn't have an ability to free pages lazy while other OS
already have been supported that named by madvise(MADV_FREE).

The gain is clear that kernel can discard freed pages rather than
swapping out or OOM if memory pressure happens.

Without memory pressure, freed pages would be reused by userspace
without another additional overhead(ex, page fault + allocation
+ zeroing).

How to work is following as.

When madvise syscall is called, VM clears dirty bit of ptes of
the range. If memory pressure happens, VM checks dirty bit of
page table and if it found still "clean", it means it's a
"lazyfree pages" so VM could discard the page instead of swapping out.
Once there was store operation for the page before VM peek a page
to reclaim, dirty bit is set so VM can swap out the page instead of
discarding.

Firstly, heavy users would be general allocators(ex, jemalloc,
tcmalloc and hope glibc supports it) and jemalloc/tcmalloc already
have supported the feature for other OS(ex, FreeBSD)

barrios@blaptop:~/benchmark/ebizzy$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Stepping:              7
CPU MHz:               2801.000
BogoMIPS:              5581.64
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              4096K
NUMA node0 CPU(s):     0-3

ebizzy benchmark(./ebizzy -S 10 -n 512)

 vanilla-jemalloc		MADV_free-jemalloc

1 thread
records:  10              records:  10
avg:      7436.70         avg:      15292.70
std:      48.01(0.65%)    std:      496.40(3.25%)
max:      7542.00         max:      15944.00
min:      7366.00         min:      14478.00

2 thread
records:  10              records:  10
avg:      12190.50        avg:      24975.50
std:      1011.51(8.30%)  std:      1127.22(4.51%)
max:      13012.00        max:      26382.00
min:      10192.00        min:      23265.00

4 thread
records:  10              records:  10
avg:      16875.30        avg:      36320.90
std:      562.59(3.33%)   std:      1503.75(4.14%)
max:      17465.00        max:      38314.00
min:      15552.00        min:      33863.00

8 thread
records:  10              records:  10
avg:      16966.80        avg:      35915.20
std:      229.35(1.35%)   std:      2153.89(6.00%)
max:      17456.00        max:      37943.00
min:      16742.00        min:      29891.00

16 thread
records:  10              records:  10
avg:      20590.90        avg:      37388.40
std:      362.33(1.76%)   std:      1282.59(3.43%)
max:      20954.00        max:      38911.00
min:      19985.00        min:      34928.00

32 thread
records:  10              records:  10
avg:      22633.40        avg:      37118.00
std:      413.73(1.83%)   std:      766.36(2.06%)
max:      23120.00        max:      38328.00
min:      22071.00        min:      35557.00

In summary, MADV_FREE is about 2 time faster than MADV_DONTNEED.

Patchset is based on v3.15-rc1-mmotm-2014-04-15-16-14

* From v4
 * Add Reviewed-by: Zhang Yanfei
 * Rebase on v3.15-rc1-mmotm-2014-04-15-16-14

* From v3
 * Add "how to work part" in description - Zhang
 * Add page_discardable utility function - Zhang
 * Clean up

* From v2
 * Remove forceful dirty marking of swap-readed page - Johannes
 * Remove deactivation logic of lazyfreed page
 * Rebased on 3.14
 * Remove RFC tag

* From v1
 * Use custom page table walker for madvise_free - Johannes
 * Remove PG_lazypage flag - Johannes
 * Do madvise_dontneed instead of madvise_freein swapless system

Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Jason Evans <je@fb.com>
Reviewed-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Signed-off-by: Minchan Kim <minchan@kernel.org>
---
 include/linux/mm.h                     |   2 +
 include/linux/rmap.h                   |  21 ++++-
 include/linux/vm_event_item.h          |   1 +
 include/uapi/asm-generic/mman-common.h |   1 +
 mm/madvise.c                           |  25 ++++++
 mm/memory.c                            | 140 +++++++++++++++++++++++++++++++++
 mm/rmap.c                              |  83 +++++++++++++++++--
 mm/vmscan.c                            |  29 ++++++-
 mm/vmstat.c                            |   1 +
 9 files changed, 291 insertions(+), 12 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9a3744d98b00..98f55ccef1a9 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1082,6 +1082,8 @@ int zap_vma_ptes(struct vm_area_struct *vma, unsigned long address,
 		unsigned long size);
 void zap_page_range(struct vm_area_struct *vma, unsigned long address,
 		unsigned long size, struct zap_details *);
+void lazyfree_range(struct vm_area_struct *vma, unsigned long address,
+		unsigned long size);
 void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
 		unsigned long start, unsigned long end);
 
diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index b66c2110cb1f..7c842675bb1b 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -181,8 +181,11 @@ static inline void page_dup_rmap(struct page *page)
 /*
  * Called from mm/vmscan.c to handle paging out
  */
+
+bool page_discardable(struct page *page, bool pte_dirty);
 int page_referenced(struct page *, int is_locked,
-			struct mem_cgroup *memcg, unsigned long *vm_flags);
+			struct mem_cgroup *memcg, unsigned long *vm_flags,
+			int *is_dirty);
 int page_referenced_one(struct page *, struct vm_area_struct *,
 	unsigned long address, void *arg);
 
@@ -235,6 +238,11 @@ struct anon_vma *page_lock_anon_vma_read(struct page *page);
 void page_unlock_anon_vma_read(struct anon_vma *anon_vma);
 int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma);
 
+struct rmap_private {
+	enum ttu_flags flags;
+	int pte_dirty;	/* used for lazyfree */
+};
+
 /*
  * rmap_walk_control: To control rmap traversing for specific needs
  *
@@ -263,11 +271,19 @@ int rmap_walk(struct page *page, struct rmap_walk_control *rwc);
 #define anon_vma_prepare(vma)	(0)
 #define anon_vma_link(vma)	do {} while (0)
 
+bool page_discardable(struct page *page, bool pte_dirty)
+{
+	return false;
+}
+
 static inline int page_referenced(struct page *page, int is_locked,
 				  struct mem_cgroup *memcg,
-				  unsigned long *vm_flags)
+				  unsigned long *vm_flags,
+				  int *is_pte_dirty)
 {
 	*vm_flags = 0;
+	if (is_pte_dirty)
+		*is_pte_dirty = 0;
 	return 0;
 }
 
@@ -288,5 +304,6 @@ static inline int page_mkclean(struct page *page)
 #define SWAP_AGAIN	1
 #define SWAP_FAIL	2
 #define SWAP_MLOCK	3
+#define SWAP_DISCARD	4
 
 #endif	/* _LINUX_RMAP_H */
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index ced92345c963..e2d3fb1e9814 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -25,6 +25,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		FOR_ALL_ZONES(PGALLOC),
 		PGFREE, PGACTIVATE, PGDEACTIVATE,
 		PGFAULT, PGMAJFAULT,
+		PGLAZYFREED,
 		FOR_ALL_ZONES(PGREFILL),
 		FOR_ALL_ZONES(PGSTEAL_KSWAPD),
 		FOR_ALL_ZONES(PGSTEAL_DIRECT),
diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
index ddc3b36f1046..7a94102b7a02 100644
--- a/include/uapi/asm-generic/mman-common.h
+++ b/include/uapi/asm-generic/mman-common.h
@@ -34,6 +34,7 @@
 #define MADV_SEQUENTIAL	2		/* expect sequential page references */
 #define MADV_WILLNEED	3		/* will need these pages */
 #define MADV_DONTNEED	4		/* don't need these pages */
+#define MADV_FREE	5		/* free pages only if memory pressure */
 
 /* common parameters: try to keep these consistent across architectures */
 #define MADV_REMOVE	9		/* remove these pages & resources */
diff --git a/mm/madvise.c b/mm/madvise.c
index 539eeb96b323..4a2aa96b2db3 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -31,6 +31,7 @@ static int madvise_need_mmap_write(int behavior)
 	case MADV_REMOVE:
 	case MADV_WILLNEED:
 	case MADV_DONTNEED:
+	case MADV_FREE:
 		return 0;
 	default:
 		/* be safe, default to 1. list exceptions explicitly */
@@ -251,6 +252,22 @@ static long madvise_willneed(struct vm_area_struct *vma,
 	return 0;
 }
 
+static long madvise_lazyfree(struct vm_area_struct *vma,
+			     struct vm_area_struct **prev,
+			     unsigned long start, unsigned long end)
+{
+	*prev = vma;
+	if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
+		return -EINVAL;
+
+	/* MADV_FREE works for only anon vma at the moment */
+	if (vma->vm_file)
+		return -EINVAL;
+
+	lazyfree_range(vma, start, end - start);
+	return 0;
+}
+
 /*
  * Application no longer needs these pages.  If the pages are dirty,
  * it's OK to just throw them away.  The app will be more careful about
@@ -384,6 +401,13 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev,
 		return madvise_remove(vma, prev, start, end);
 	case MADV_WILLNEED:
 		return madvise_willneed(vma, prev, start, end);
+	case MADV_FREE:
+		/*
+		 * In this implementation, MADV_FREE works like MADV_DONTNEED
+		 * on swapless system or full swap.
+		 */
+		if (get_nr_swap_pages() > 0)
+			return madvise_lazyfree(vma, prev, start, end);
 	case MADV_DONTNEED:
 		return madvise_dontneed(vma, prev, start, end);
 	default:
@@ -403,6 +427,7 @@ madvise_behavior_valid(int behavior)
 	case MADV_REMOVE:
 	case MADV_WILLNEED:
 	case MADV_DONTNEED:
+	case MADV_FREE:
 #ifdef CONFIG_KSM
 	case MADV_MERGEABLE:
 	case MADV_UNMERGEABLE:
diff --git a/mm/memory.c b/mm/memory.c
index c4b5bc250820..ca427f258204 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1270,6 +1270,104 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
 	return addr;
 }
 
+static unsigned long lazyfree_pte_range(struct mmu_gather *tlb,
+				struct vm_area_struct *vma, pmd_t *pmd,
+				unsigned long addr, unsigned long end)
+{
+	struct mm_struct *mm = tlb->mm;
+	spinlock_t *ptl;
+	pte_t *start_pte;
+	pte_t *pte;
+
+	start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
+	pte = start_pte;
+	arch_enter_lazy_mmu_mode();
+	do {
+		pte_t ptent = *pte;
+
+		if (pte_none(ptent))
+			continue;
+
+		if (!pte_present(ptent))
+			continue;
+
+		ptent = pte_mkold(ptent);
+		ptent = pte_mkclean(ptent);
+		set_pte_at(mm, addr, pte, ptent);
+		tlb_remove_tlb_entry(tlb, pte, addr);
+	} while (pte++, addr += PAGE_SIZE, addr != end);
+	arch_leave_lazy_mmu_mode();
+	pte_unmap_unlock(start_pte, ptl);
+
+	return addr;
+}
+
+static inline unsigned long lazyfree_pmd_range(struct mmu_gather *tlb,
+				struct vm_area_struct *vma, pud_t *pud,
+				unsigned long addr, unsigned long end)
+{
+	pmd_t *pmd;
+	unsigned long next;
+
+	pmd = pmd_offset(pud, addr);
+	do {
+		next = pmd_addr_end(addr, end);
+		if (pmd_trans_huge(*pmd))
+			split_huge_page_pmd(vma, addr, pmd);
+		/*
+		 * Here there can be other concurrent MADV_DONTNEED or
+		 * trans huge page faults running, and if the pmd is
+		 * none or trans huge it can change under us. This is
+		 * because MADV_LAZYFREE holds the mmap_sem in read
+		 * mode.
+		 */
+		if (pmd_none_or_trans_huge_or_clear_bad(pmd))
+			goto next;
+		next = lazyfree_pte_range(tlb, vma, pmd, addr, next);
+next:
+		cond_resched();
+	} while (pmd++, addr = next, addr != end);
+
+	return addr;
+}
+
+static inline unsigned long lazyfree_pud_range(struct mmu_gather *tlb,
+				struct vm_area_struct *vma, pgd_t *pgd,
+				unsigned long addr, unsigned long end)
+{
+	pud_t *pud;
+	unsigned long next;
+
+	pud = pud_offset(pgd, addr);
+	do {
+		next = pud_addr_end(addr, end);
+		if (pud_none_or_clear_bad(pud))
+			continue;
+		next = lazyfree_pmd_range(tlb, vma, pud, addr, next);
+	} while (pud++, addr = next, addr != end);
+
+	return addr;
+}
+
+static void lazyfree_page_range(struct mmu_gather *tlb,
+			     struct vm_area_struct *vma,
+			     unsigned long addr, unsigned long end)
+{
+	pgd_t *pgd;
+	unsigned long next;
+
+	BUG_ON(addr >= end);
+	tlb_start_vma(tlb, vma);
+	pgd = pgd_offset(vma->vm_mm, addr);
+	do {
+		next = pgd_addr_end(addr, end);
+		if (pgd_none_or_clear_bad(pgd))
+			continue;
+		next = lazyfree_pud_range(tlb, vma, pgd, addr, next);
+	} while (pgd++, addr = next, addr != end);
+	tlb_end_vma(tlb, vma);
+}
+
 static void unmap_page_range(struct mmu_gather *tlb,
 			     struct vm_area_struct *vma,
 			     unsigned long addr, unsigned long end,
@@ -1296,6 +1394,23 @@ static void unmap_page_range(struct mmu_gather *tlb,
 }
 
 
+static void lazyfree_single_vma(struct mmu_gather *tlb,
+		struct vm_area_struct *vma, unsigned long start_addr,
+		unsigned long end_addr)
+{
+	unsigned long start = max(vma->vm_start, start_addr);
+	unsigned long end;
+
+	if (start >= vma->vm_end)
+		return;
+	end = min(vma->vm_end, end_addr);
+	if (end <= vma->vm_start)
+		return;
+
+	if (start != end)
+		lazyfree_page_range(tlb, vma, start, end);
+}
+
 static void unmap_single_vma(struct mmu_gather *tlb,
 		struct vm_area_struct *vma, unsigned long start_addr,
 		unsigned long end_addr,
@@ -1370,6 +1485,31 @@ void unmap_vmas(struct mmu_gather *tlb,
 }
 
 /**
+ * lazyfree_range - clear dirty bit of pte in a given range
+ * @vma: vm_area_struct holding the applicable pages
+ * @start: starting address of pages
+ * @size: number of bytes to do lazyfree
+ *
+ * Caller must protect the VMA list
+ */
+void lazyfree_range(struct vm_area_struct *vma, unsigned long start,
+		unsigned long size)
+{
+	struct mm_struct *mm = vma->vm_mm;
+	struct mmu_gather tlb;
+	unsigned long end = start + size;
+
+	lru_add_drain();
+	tlb_gather_mmu(&tlb, mm, start, end);
+	update_hiwater_rss(mm);
+	mmu_notifier_invalidate_range_start(mm, start, end);
+	for ( ; vma && vma->vm_start < end; vma = vma->vm_next)
+		lazyfree_single_vma(&tlb, vma, start, end);
+	mmu_notifier_invalidate_range_end(mm, start, end);
+	tlb_finish_mmu(&tlb, start, end);
+}
+
+/**
  * zap_page_range - remove user pages in a given range
  * @vma: vm_area_struct holding the applicable pages
  * @start: starting address of pages to zap
diff --git a/mm/rmap.c b/mm/rmap.c
index e065ba798fde..a87661a7ef2c 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -657,6 +657,7 @@ int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma)
 }
 
 struct page_referenced_arg {
+	int dirtied;
 	int mapcount;
 	int referenced;
 	unsigned long vm_flags;
@@ -671,6 +672,7 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma,
 	struct mm_struct *mm = vma->vm_mm;
 	spinlock_t *ptl;
 	int referenced = 0;
+	int dirty = 0;
 	struct page_referenced_arg *pra = arg;
 
 	if (unlikely(PageTransHuge(page))) {
@@ -723,6 +725,10 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma,
 			if (likely(!(vma->vm_flags & VM_SEQ_READ)))
 				referenced++;
 		}
+
+		if (pte_dirty(*pte))
+			dirty++;
+
 		pte_unmap_unlock(pte, ptl);
 	}
 
@@ -731,6 +737,9 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma,
 		pra->vm_flags |= vma->vm_flags;
 	}
 
+	if (dirty)
+		pra->dirtied++;
+
 	pra->mapcount--;
 	if (!pra->mapcount)
 		return SWAP_SUCCESS; /* To break the loop */
@@ -750,11 +759,34 @@ static bool invalid_page_referenced_vma(struct vm_area_struct *vma, void *arg)
 }
 
 /**
+ * page_discardable - test if the page could be discardable instead of swap
+ * @page: the page to test
+ * @pte_dirty: ptes which have marked dirty bit
+ *
+ * Should check PageDirty because swap-in page by read fault
+ * would put on swapcache and pte point out the page doesn't have
+ * dirty bit so only pte dirtiness check isn't enough to detect
+ * lazyfree page so we need to check PG_swapcache to filter it out.
+ * But, if the page is removed from swapcache, it must have PG_dirty
+ * so we should check it to prevent purging non-lazyfree page.
+ */
+bool page_discardable(struct page *page, bool pte_dirty)
+{
+	bool ret = false;
+
+	if (PageAnon(page) && !pte_dirty && !PageSwapCache(page) &&
+			!PageDirty(page))
+		ret = true;
+	return ret;
+}
+
+/**
  * page_referenced - test if the page was referenced
  * @page: the page to test
  * @is_locked: caller holds lock on the page
  * @memcg: target memory cgroup
  * @vm_flags: collect encountered vma->vm_flags who actually referenced the page
+ * @is_pte_dirty: ptes which have marked dirty bit - used for lazyfree page
  *
  * Quick test_and_clear_referenced for all mappings to a page,
  * returns the number of ptes which referenced the page.
@@ -762,7 +794,8 @@ static bool invalid_page_referenced_vma(struct vm_area_struct *vma, void *arg)
 int page_referenced(struct page *page,
 		    int is_locked,
 		    struct mem_cgroup *memcg,
-		    unsigned long *vm_flags)
+		    unsigned long *vm_flags,
+		    int *is_pte_dirty)
 {
 	int ret;
 	int we_locked = 0;
@@ -777,6 +810,9 @@ int page_referenced(struct page *page,
 	};
 
 	*vm_flags = 0;
+	if (is_pte_dirty)
+		*is_pte_dirty = 0;
+
 	if (!page_mapped(page))
 		return 0;
 
@@ -804,6 +840,9 @@ int page_referenced(struct page *page,
 	if (we_locked)
 		unlock_page(page);
 
+	if (is_pte_dirty)
+		*is_pte_dirty = pra.dirtied;
+
 	return pra.referenced;
 }
 
@@ -1116,7 +1155,9 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 	pte_t pteval;
 	spinlock_t *ptl;
 	int ret = SWAP_AGAIN;
-	enum ttu_flags flags = (enum ttu_flags)arg;
+	struct rmap_private *rp = (struct rmap_private *)arg;
+	enum ttu_flags flags = rp->flags;
+	int dirty = 0;
 
 	pte = page_check_address(page, mm, address, &ptl, 0);
 	if (!pte)
@@ -1146,7 +1187,8 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 	pteval = ptep_clear_flush(vma, address, pte);
 
 	/* Move the dirty bit to the physical page now the pte is gone. */
-	if (pte_dirty(pteval))
+	dirty = pte_dirty(pteval);
+	if (dirty)
 		set_page_dirty(page);
 
 	/* Update high watermark before we lower rss */
@@ -1175,6 +1217,15 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 		swp_entry_t entry = { .val = page_private(page) };
 		pte_t swp_pte;
 
+		if ((TTU_ACTION(flags) == TTU_UNMAP) &&
+				page_discardable(page, dirty)) {
+			dec_mm_counter(mm, MM_ANONPAGES);
+			goto discard;
+		}
+
+		if (dirty)
+			rp->pte_dirty++;
+
 		if (PageSwapCache(page)) {
 			/*
 			 * Store the swap location in the pte.
@@ -1193,6 +1244,10 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 			}
 			dec_mm_counter(mm, MM_ANONPAGES);
 			inc_mm_counter(mm, MM_SWAPENTS);
+		} else if (TTU_ACTION(flags) == TTU_UNMAP) {
+			set_pte_at(mm, address, pte, pteval);
+			ret = SWAP_FAIL;
+			goto out_unmap;
 		} else if (IS_ENABLED(CONFIG_MIGRATION)) {
 			/*
 			 * Store the pfn of the page in a special migration
@@ -1216,6 +1271,7 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 	} else
 		dec_mm_counter(mm, MM_FILEPAGES);
 
+discard:
 	page_remove_rmap(page);
 	page_cache_release(page);
 
@@ -1486,13 +1542,19 @@ static int page_not_mapped(struct page *page)
  * SWAP_AGAIN	- we missed a mapping, try again later
  * SWAP_FAIL	- the page is unswappable
  * SWAP_MLOCK	- page is mlocked.
+ * SWAP_DISCARD - same with SWAP_SUCCESS but no need to swap out
  */
 int try_to_unmap(struct page *page, enum ttu_flags flags)
 {
 	int ret;
+
+	struct rmap_private rp = {
+		.flags = flags,
+	};
+
 	struct rmap_walk_control rwc = {
 		.rmap_one = try_to_unmap_one,
-		.arg = (void *)flags,
+		.arg = &rp,
 		.done = page_not_mapped,
 		.file_nonlinear = try_to_unmap_nonlinear,
 		.anon_lock = page_lock_anon_vma_read,
@@ -1513,8 +1575,13 @@ int try_to_unmap(struct page *page, enum ttu_flags flags)
 
 	ret = rmap_walk(page, &rwc);
 
-	if (ret != SWAP_MLOCK && !page_mapped(page))
+	if (ret != SWAP_MLOCK && !page_mapped(page)) {
 		ret = SWAP_SUCCESS;
+		if (TTU_ACTION(flags) == TTU_UNMAP &&
+			page_discardable(page, rp.pte_dirty))
+			ret = SWAP_DISCARD;
+	}
+
 	return ret;
 }
 
@@ -1536,9 +1603,13 @@ int try_to_unmap(struct page *page, enum ttu_flags flags)
 int try_to_munlock(struct page *page)
 {
 	int ret;
+	struct rmap_private rp = {
+		.flags = TTU_MUNLOCK,
+	};
+
 	struct rmap_walk_control rwc = {
 		.rmap_one = try_to_unmap_one,
-		.arg = (void *)TTU_MUNLOCK,
+		.arg = &rp,
 		.done = page_not_mapped,
 		/*
 		 * We don't bother to try to find the munlocked page in
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 6cdda104f629..243da05cc24d 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -698,6 +698,7 @@ enum page_references {
 	PAGEREF_RECLAIM_CLEAN,
 	PAGEREF_KEEP,
 	PAGEREF_ACTIVATE,
+	PAGEREF_DISCARD,
 };
 
 static enum page_references page_check_references(struct page *page,
@@ -705,9 +706,12 @@ static enum page_references page_check_references(struct page *page,
 {
 	int referenced_ptes, referenced_page;
 	unsigned long vm_flags;
+	int is_pte_dirty;
+
+	VM_BUG_ON_PAGE(!PageLocked(page), page);
 
 	referenced_ptes = page_referenced(page, 1, sc->target_mem_cgroup,
-					  &vm_flags);
+					  &vm_flags, &is_pte_dirty);
 	referenced_page = TestClearPageReferenced(page);
 
 	/*
@@ -748,6 +752,9 @@ static enum page_references page_check_references(struct page *page,
 		return PAGEREF_KEEP;
 	}
 
+	if (page_discardable(page, is_pte_dirty))
+		return PAGEREF_DISCARD;
+
 	/* Reclaim if clean, defer dirty pages to writeback */
 	if (referenced_page && !PageSwapBacked(page))
 		return PAGEREF_RECLAIM_CLEAN;
@@ -817,6 +824,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		int may_enter_fs;
 		enum page_references references = PAGEREF_RECLAIM_CLEAN;
 		bool dirty, writeback;
+		bool discard = false;
 
 		cond_resched();
 
@@ -946,6 +954,12 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 			goto activate_locked;
 		case PAGEREF_KEEP:
 			goto keep_locked;
+		case PAGEREF_DISCARD:
+			/*
+			 * skip to add the page into swapcache  then
+			 * unmap without mapping
+			 */
+			discard = true;
 		case PAGEREF_RECLAIM:
 		case PAGEREF_RECLAIM_CLEAN:
 			; /* try to reclaim the page below */
@@ -955,7 +969,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		 * Anonymous process memory has backing store?
 		 * Try to allocate it some swap space here.
 		 */
-		if (PageAnon(page) && !PageSwapCache(page)) {
+		if (PageAnon(page) && !PageSwapCache(page) && !discard) {
 			if (!(sc->gfp_mask & __GFP_IO))
 				goto keep_locked;
 			if (!add_to_swap(page, page_list))
@@ -970,7 +984,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		 * The page is mapped into the page tables of one or more
 		 * processes. Try to unmap it here.
 		 */
-		if (page_mapped(page) && mapping) {
+		if (page_mapped(page) && (mapping || discard)) {
 			switch (try_to_unmap(page, ttu_flags)) {
 			case SWAP_FAIL:
 				goto activate_locked;
@@ -978,6 +992,13 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 				goto keep_locked;
 			case SWAP_MLOCK:
 				goto cull_mlocked;
+			case SWAP_DISCARD:
+				VM_BUG_ON_PAGE(PageSwapCache(page), page);
+				if (!page_freeze_refs(page, 1))
+					goto keep_locked;
+				__clear_page_locked(page);
+				count_vm_event(PGLAZYFREED);
+				goto free_it;
 			case SWAP_SUCCESS:
 				; /* try to free the page below */
 			}
@@ -1702,7 +1723,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 		}
 
 		if (page_referenced(page, 0, sc->target_mem_cgroup,
-				    &vm_flags)) {
+				    &vm_flags, NULL)) {
 			nr_rotated += hpage_nr_pages(page);
 			/*
 			 * Identify referenced, file-backed active pages and
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 0b295fd8a60a..dcf6c6ab723e 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -792,6 +792,7 @@ const char * const vmstat_text[] = {
 
 	"pgfault",
 	"pgmajfault",
+	"pglazyfreed",
 
 	TEXTS_FOR_ZONES("pgrefill")
 	TEXTS_FOR_ZONES("pgsteal_kswapd")
-- 
1.9.2

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v5] mm: support madvise(MADV_FREE)

[Minchan Kim]

bump

On Mon, Apr 21, 2014 at 10:56:08AM +0900, Minchan Kim wrote:
> Linux doesn't have an ability to free pages lazy while other OS
> already have been supported that named by madvise(MADV_FREE).
> 
> The gain is clear that kernel can discard freed pages rather than
> swapping out or OOM if memory pressure happens.
> 
> Without memory pressure, freed pages would be reused by userspace
> without another additional overhead(ex, page fault + allocation
> + zeroing).
> 
> How to work is following as.
> 
> When madvise syscall is called, VM clears dirty bit of ptes of
> the range. If memory pressure happens, VM checks dirty bit of
> page table and if it found still "clean", it means it's a
> "lazyfree pages" so VM could discard the page instead of swapping out.
> Once there was store operation for the page before VM peek a page
> to reclaim, dirty bit is set so VM can swap out the page instead of
> discarding.
> 
> Firstly, heavy users would be general allocators(ex, jemalloc,
> tcmalloc and hope glibc supports it) and jemalloc/tcmalloc already
> have supported the feature for other OS(ex, FreeBSD)
> 
> barrios@blaptop:~/benchmark/ebizzy$ lscpu
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                4
> On-line CPU(s) list:   0-3
> Thread(s) per core:    2
> Core(s) per socket:    2
> Socket(s):             1
> NUMA node(s):          1
> Vendor ID:             GenuineIntel
> CPU family:            6
> Model:                 42
> Stepping:              7
> CPU MHz:               2801.000
> BogoMIPS:              5581.64
> Virtualization:        VT-x
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              256K
> L3 cache:              4096K
> NUMA node0 CPU(s):     0-3
> 
> ebizzy benchmark(./ebizzy -S 10 -n 512)
> 
>  vanilla-jemalloc		MADV_free-jemalloc
> 
> 1 thread
> records:  10              records:  10
> avg:      7436.70         avg:      15292.70
> std:      48.01(0.65%)    std:      496.40(3.25%)
> max:      7542.00         max:      15944.00
> min:      7366.00         min:      14478.00
> 
> 2 thread
> records:  10              records:  10
> avg:      12190.50        avg:      24975.50
> std:      1011.51(8.30%)  std:      1127.22(4.51%)
> max:      13012.00        max:      26382.00
> min:      10192.00        min:      23265.00
> 
> 4 thread
> records:  10              records:  10
> avg:      16875.30        avg:      36320.90
> std:      562.59(3.33%)   std:      1503.75(4.14%)
> max:      17465.00        max:      38314.00
> min:      15552.00        min:      33863.00
> 
> 8 thread
> records:  10              records:  10
> avg:      16966.80        avg:      35915.20
> std:      229.35(1.35%)   std:      2153.89(6.00%)
> max:      17456.00        max:      37943.00
> min:      16742.00        min:      29891.00
> 
> 16 thread
> records:  10              records:  10
> avg:      20590.90        avg:      37388.40
> std:      362.33(1.76%)   std:      1282.59(3.43%)
> max:      20954.00        max:      38911.00
> min:      19985.00        min:      34928.00
> 
> 32 thread
> records:  10              records:  10
> avg:      22633.40        avg:      37118.00
> std:      413.73(1.83%)   std:      766.36(2.06%)
> max:      23120.00        max:      38328.00
> min:      22071.00        min:      35557.00
> 
> In summary, MADV_FREE is about 2 time faster than MADV_DONTNEED.
> 
> Patchset is based on v3.15-rc1-mmotm-2014-04-15-16-14
> 
> * From v4
>  * Add Reviewed-by: Zhang Yanfei
>  * Rebase on v3.15-rc1-mmotm-2014-04-15-16-14
> 
> * From v3
>  * Add "how to work part" in description - Zhang
>  * Add page_discardable utility function - Zhang
>  * Clean up
> 
> * From v2
>  * Remove forceful dirty marking of swap-readed page - Johannes
>  * Remove deactivation logic of lazyfreed page
>  * Rebased on 3.14
>  * Remove RFC tag
> 
> * From v1
>  * Use custom page table walker for madvise_free - Johannes
>  * Remove PG_lazypage flag - Johannes
>  * Do madvise_dontneed instead of madvise_freein swapless system
> 
> Cc: Hugh Dickins <hughd@google.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Rik van Riel <riel@redhat.com>
> Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Jason Evans <je@fb.com>
> Reviewed-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
> Signed-off-by: Minchan Kim <minchan@kernel.org>
> ---
>  include/linux/mm.h                     |   2 +
>  include/linux/rmap.h                   |  21 ++++-
>  include/linux/vm_event_item.h          |   1 +
>  include/uapi/asm-generic/mman-common.h |   1 +
>  mm/madvise.c                           |  25 ++++++
>  mm/memory.c                            | 140 +++++++++++++++++++++++++++++++++
>  mm/rmap.c                              |  83 +++++++++++++++++--
>  mm/vmscan.c                            |  29 ++++++-
>  mm/vmstat.c                            |   1 +
>  9 files changed, 291 insertions(+), 12 deletions(-)
> 
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 9a3744d98b00..98f55ccef1a9 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1082,6 +1082,8 @@ int zap_vma_ptes(struct vm_area_struct *vma, unsigned long address,
>  		unsigned long size);
>  void zap_page_range(struct vm_area_struct *vma, unsigned long address,
>  		unsigned long size, struct zap_details *);
> +void lazyfree_range(struct vm_area_struct *vma, unsigned long address,
> +		unsigned long size);
>  void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
>  		unsigned long start, unsigned long end);
>  
> diff --git a/include/linux/rmap.h b/include/linux/rmap.h
> index b66c2110cb1f..7c842675bb1b 100644
> --- a/include/linux/rmap.h
> +++ b/include/linux/rmap.h
> @@ -181,8 +181,11 @@ static inline void page_dup_rmap(struct page *page)
>  /*
>   * Called from mm/vmscan.c to handle paging out
>   */
> +
> +bool page_discardable(struct page *page, bool pte_dirty);
>  int page_referenced(struct page *, int is_locked,
> -			struct mem_cgroup *memcg, unsigned long *vm_flags);
> +			struct mem_cgroup *memcg, unsigned long *vm_flags,
> +			int *is_dirty);
>  int page_referenced_one(struct page *, struct vm_area_struct *,
>  	unsigned long address, void *arg);
>  
> @@ -235,6 +238,11 @@ struct anon_vma *page_lock_anon_vma_read(struct page *page);
>  void page_unlock_anon_vma_read(struct anon_vma *anon_vma);
>  int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma);
>  
> +struct rmap_private {
> +	enum ttu_flags flags;
> +	int pte_dirty;	/* used for lazyfree */
> +};
> +
>  /*
>   * rmap_walk_control: To control rmap traversing for specific needs
>   *
> @@ -263,11 +271,19 @@ int rmap_walk(struct page *page, struct rmap_walk_control *rwc);
>  #define anon_vma_prepare(vma)	(0)
>  #define anon_vma_link(vma)	do {} while (0)
>  
> +bool page_discardable(struct page *page, bool pte_dirty)
> +{
> +	return false;
> +}
> +
>  static inline int page_referenced(struct page *page, int is_locked,
>  				  struct mem_cgroup *memcg,
> -				  unsigned long *vm_flags)
> +				  unsigned long *vm_flags,
> +				  int *is_pte_dirty)
>  {
>  	*vm_flags = 0;
> +	if (is_pte_dirty)
> +		*is_pte_dirty = 0;
>  	return 0;
>  }
>  
> @@ -288,5 +304,6 @@ static inline int page_mkclean(struct page *page)
>  #define SWAP_AGAIN	1
>  #define SWAP_FAIL	2
>  #define SWAP_MLOCK	3
> +#define SWAP_DISCARD	4
>  
>  #endif	/* _LINUX_RMAP_H */
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
> index ced92345c963..e2d3fb1e9814 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -25,6 +25,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
>  		FOR_ALL_ZONES(PGALLOC),
>  		PGFREE, PGACTIVATE, PGDEACTIVATE,
>  		PGFAULT, PGMAJFAULT,
> +		PGLAZYFREED,
>  		FOR_ALL_ZONES(PGREFILL),
>  		FOR_ALL_ZONES(PGSTEAL_KSWAPD),
>  		FOR_ALL_ZONES(PGSTEAL_DIRECT),
> diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
> index ddc3b36f1046..7a94102b7a02 100644
> --- a/include/uapi/asm-generic/mman-common.h
> +++ b/include/uapi/asm-generic/mman-common.h
> @@ -34,6 +34,7 @@
>  #define MADV_SEQUENTIAL	2		/* expect sequential page references */
>  #define MADV_WILLNEED	3		/* will need these pages */
>  #define MADV_DONTNEED	4		/* don't need these pages */
> +#define MADV_FREE	5		/* free pages only if memory pressure */
>  
>  /* common parameters: try to keep these consistent across architectures */
>  #define MADV_REMOVE	9		/* remove these pages & resources */
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 539eeb96b323..4a2aa96b2db3 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -31,6 +31,7 @@ static int madvise_need_mmap_write(int behavior)
>  	case MADV_REMOVE:
>  	case MADV_WILLNEED:
>  	case MADV_DONTNEED:
> +	case MADV_FREE:
>  		return 0;
>  	default:
>  		/* be safe, default to 1. list exceptions explicitly */
> @@ -251,6 +252,22 @@ static long madvise_willneed(struct vm_area_struct *vma,
>  	return 0;
>  }
>  
> +static long madvise_lazyfree(struct vm_area_struct *vma,
> +			     struct vm_area_struct **prev,
> +			     unsigned long start, unsigned long end)
> +{
> +	*prev = vma;
> +	if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
> +		return -EINVAL;
> +
> +	/* MADV_FREE works for only anon vma at the moment */
> +	if (vma->vm_file)
> +		return -EINVAL;
> +
> +	lazyfree_range(vma, start, end - start);
> +	return 0;
> +}
> +
>  /*
>   * Application no longer needs these pages.  If the pages are dirty,
>   * it's OK to just throw them away.  The app will be more careful about
> @@ -384,6 +401,13 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev,
>  		return madvise_remove(vma, prev, start, end);
>  	case MADV_WILLNEED:
>  		return madvise_willneed(vma, prev, start, end);
> +	case MADV_FREE:
> +		/*
> +		 * In this implementation, MADV_FREE works like MADV_DONTNEED
> +		 * on swapless system or full swap.
> +		 */
> +		if (get_nr_swap_pages() > 0)
> +			return madvise_lazyfree(vma, prev, start, end);
>  	case MADV_DONTNEED:
>  		return madvise_dontneed(vma, prev, start, end);
>  	default:
> @@ -403,6 +427,7 @@ madvise_behavior_valid(int behavior)
>  	case MADV_REMOVE:
>  	case MADV_WILLNEED:
>  	case MADV_DONTNEED:
> +	case MADV_FREE:
>  #ifdef CONFIG_KSM
>  	case MADV_MERGEABLE:
>  	case MADV_UNMERGEABLE:
> diff --git a/mm/memory.c b/mm/memory.c
> index c4b5bc250820..ca427f258204 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -1270,6 +1270,104 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
>  	return addr;
>  }
>  
> +static unsigned long lazyfree_pte_range(struct mmu_gather *tlb,
> +				struct vm_area_struct *vma, pmd_t *pmd,
> +				unsigned long addr, unsigned long end)
> +{
> +	struct mm_struct *mm = tlb->mm;
> +	spinlock_t *ptl;
> +	pte_t *start_pte;
> +	pte_t *pte;
> +
> +	start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
> +	pte = start_pte;
> +	arch_enter_lazy_mmu_mode();
> +	do {
> +		pte_t ptent = *pte;
> +
> +		if (pte_none(ptent))
> +			continue;
> +
> +		if (!pte_present(ptent))
> +			continue;
> +
> +		ptent = pte_mkold(ptent);
> +		ptent = pte_mkclean(ptent);
> +		set_pte_at(mm, addr, pte, ptent);
> +		tlb_remove_tlb_entry(tlb, pte, addr);
> +	} while (pte++, addr += PAGE_SIZE, addr != end);
> +	arch_leave_lazy_mmu_mode();
> +	pte_unmap_unlock(start_pte, ptl);
> +
> +	return addr;
> +}
> +
> +static inline unsigned long lazyfree_pmd_range(struct mmu_gather *tlb,
> +				struct vm_area_struct *vma, pud_t *pud,
> +				unsigned long addr, unsigned long end)
> +{
> +	pmd_t *pmd;
> +	unsigned long next;
> +
> +	pmd = pmd_offset(pud, addr);
> +	do {
> +		next = pmd_addr_end(addr, end);
> +		if (pmd_trans_huge(*pmd))
> +			split_huge_page_pmd(vma, addr, pmd);
> +		/*
> +		 * Here there can be other concurrent MADV_DONTNEED or
> +		 * trans huge page faults running, and if the pmd is
> +		 * none or trans huge it can change under us. This is
> +		 * because MADV_LAZYFREE holds the mmap_sem in read
> +		 * mode.
> +		 */
> +		if (pmd_none_or_trans_huge_or_clear_bad(pmd))
> +			goto next;
> +		next = lazyfree_pte_range(tlb, vma, pmd, addr, next);
> +next:
> +		cond_resched();
> +	} while (pmd++, addr = next, addr != end);
> +
> +	return addr;
> +}
> +
> +static inline unsigned long lazyfree_pud_range(struct mmu_gather *tlb,
> +				struct vm_area_struct *vma, pgd_t *pgd,
> +				unsigned long addr, unsigned long end)
> +{
> +	pud_t *pud;
> +	unsigned long next;
> +
> +	pud = pud_offset(pgd, addr);
> +	do {
> +		next = pud_addr_end(addr, end);
> +		if (pud_none_or_clear_bad(pud))
> +			continue;
> +		next = lazyfree_pmd_range(tlb, vma, pud, addr, next);
> +	} while (pud++, addr = next, addr != end);
> +
> +	return addr;
> +}
> +
> +static void lazyfree_page_range(struct mmu_gather *tlb,
> +			     struct vm_area_struct *vma,
> +			     unsigned long addr, unsigned long end)
> +{
> +	pgd_t *pgd;
> +	unsigned long next;
> +
> +	BUG_ON(addr >= end);
> +	tlb_start_vma(tlb, vma);
> +	pgd = pgd_offset(vma->vm_mm, addr);
> +	do {
> +		next = pgd_addr_end(addr, end);
> +		if (pgd_none_or_clear_bad(pgd))
> +			continue;
> +		next = lazyfree_pud_range(tlb, vma, pgd, addr, next);
> +	} while (pgd++, addr = next, addr != end);
> +	tlb_end_vma(tlb, vma);
> +}
> +
>  static void unmap_page_range(struct mmu_gather *tlb,
>  			     struct vm_area_struct *vma,
>  			     unsigned long addr, unsigned long end,
> @@ -1296,6 +1394,23 @@ static void unmap_page_range(struct mmu_gather *tlb,
>  }
>  
>  
> +static void lazyfree_single_vma(struct mmu_gather *tlb,
> +		struct vm_area_struct *vma, unsigned long start_addr,
> +		unsigned long end_addr)
> +{
> +	unsigned long start = max(vma->vm_start, start_addr);
> +	unsigned long end;
> +
> +	if (start >= vma->vm_end)
> +		return;
> +	end = min(vma->vm_end, end_addr);
> +	if (end <= vma->vm_start)
> +		return;
> +
> +	if (start != end)
> +		lazyfree_page_range(tlb, vma, start, end);
> +}
> +
>  static void unmap_single_vma(struct mmu_gather *tlb,
>  		struct vm_area_struct *vma, unsigned long start_addr,
>  		unsigned long end_addr,
> @@ -1370,6 +1485,31 @@ void unmap_vmas(struct mmu_gather *tlb,
>  }
>  
>  /**
> + * lazyfree_range - clear dirty bit of pte in a given range
> + * @vma: vm_area_struct holding the applicable pages
> + * @start: starting address of pages
> + * @size: number of bytes to do lazyfree
> + *
> + * Caller must protect the VMA list
> + */
> +void lazyfree_range(struct vm_area_struct *vma, unsigned long start,
> +		unsigned long size)
> +{
> +	struct mm_struct *mm = vma->vm_mm;
> +	struct mmu_gather tlb;
> +	unsigned long end = start + size;
> +
> +	lru_add_drain();
> +	tlb_gather_mmu(&tlb, mm, start, end);
> +	update_hiwater_rss(mm);
> +	mmu_notifier_invalidate_range_start(mm, start, end);
> +	for ( ; vma && vma->vm_start < end; vma = vma->vm_next)
> +		lazyfree_single_vma(&tlb, vma, start, end);
> +	mmu_notifier_invalidate_range_end(mm, start, end);
> +	tlb_finish_mmu(&tlb, start, end);
> +}
> +
> +/**
>   * zap_page_range - remove user pages in a given range
>   * @vma: vm_area_struct holding the applicable pages
>   * @start: starting address of pages to zap
> diff --git a/mm/rmap.c b/mm/rmap.c
> index e065ba798fde..a87661a7ef2c 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -657,6 +657,7 @@ int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma)
>  }
>  
>  struct page_referenced_arg {
> +	int dirtied;
>  	int mapcount;
>  	int referenced;
>  	unsigned long vm_flags;
> @@ -671,6 +672,7 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma,
>  	struct mm_struct *mm = vma->vm_mm;
>  	spinlock_t *ptl;
>  	int referenced = 0;
> +	int dirty = 0;
>  	struct page_referenced_arg *pra = arg;
>  
>  	if (unlikely(PageTransHuge(page))) {
> @@ -723,6 +725,10 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma,
>  			if (likely(!(vma->vm_flags & VM_SEQ_READ)))
>  				referenced++;
>  		}
> +
> +		if (pte_dirty(*pte))
> +			dirty++;
> +
>  		pte_unmap_unlock(pte, ptl);
>  	}
>  
> @@ -731,6 +737,9 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma,
>  		pra->vm_flags |= vma->vm_flags;
>  	}
>  
> +	if (dirty)
> +		pra->dirtied++;
> +
>  	pra->mapcount--;
>  	if (!pra->mapcount)
>  		return SWAP_SUCCESS; /* To break the loop */
> @@ -750,11 +759,34 @@ static bool invalid_page_referenced_vma(struct vm_area_struct *vma, void *arg)
>  }
>  
>  /**
> + * page_discardable - test if the page could be discardable instead of swap
> + * @page: the page to test
> + * @pte_dirty: ptes which have marked dirty bit
> + *
> + * Should check PageDirty because swap-in page by read fault
> + * would put on swapcache and pte point out the page doesn't have
> + * dirty bit so only pte dirtiness check isn't enough to detect
> + * lazyfree page so we need to check PG_swapcache to filter it out.
> + * But, if the page is removed from swapcache, it must have PG_dirty
> + * so we should check it to prevent purging non-lazyfree page.
> + */
> +bool page_discardable(struct page *page, bool pte_dirty)
> +{
> +	bool ret = false;
> +
> +	if (PageAnon(page) && !pte_dirty && !PageSwapCache(page) &&
> +			!PageDirty(page))
> +		ret = true;
> +	return ret;
> +}
> +
> +/**
>   * page_referenced - test if the page was referenced
>   * @page: the page to test
>   * @is_locked: caller holds lock on the page
>   * @memcg: target memory cgroup
>   * @vm_flags: collect encountered vma->vm_flags who actually referenced the page
> + * @is_pte_dirty: ptes which have marked dirty bit - used for lazyfree page
>   *
>   * Quick test_and_clear_referenced for all mappings to a page,
>   * returns the number of ptes which referenced the page.
> @@ -762,7 +794,8 @@ static bool invalid_page_referenced_vma(struct vm_area_struct *vma, void *arg)
>  int page_referenced(struct page *page,
>  		    int is_locked,
>  		    struct mem_cgroup *memcg,
> -		    unsigned long *vm_flags)
> +		    unsigned long *vm_flags,
> +		    int *is_pte_dirty)
>  {
>  	int ret;
>  	int we_locked = 0;
> @@ -777,6 +810,9 @@ int page_referenced(struct page *page,
>  	};
>  
>  	*vm_flags = 0;
> +	if (is_pte_dirty)
> +		*is_pte_dirty = 0;
> +
>  	if (!page_mapped(page))
>  		return 0;
>  
> @@ -804,6 +840,9 @@ int page_referenced(struct page *page,
>  	if (we_locked)
>  		unlock_page(page);
>  
> +	if (is_pte_dirty)
> +		*is_pte_dirty = pra.dirtied;
> +
>  	return pra.referenced;
>  }
>  
> @@ -1116,7 +1155,9 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  	pte_t pteval;
>  	spinlock_t *ptl;
>  	int ret = SWAP_AGAIN;
> -	enum ttu_flags flags = (enum ttu_flags)arg;
> +	struct rmap_private *rp = (struct rmap_private *)arg;
> +	enum ttu_flags flags = rp->flags;
> +	int dirty = 0;
>  
>  	pte = page_check_address(page, mm, address, &ptl, 0);
>  	if (!pte)
> @@ -1146,7 +1187,8 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  	pteval = ptep_clear_flush(vma, address, pte);
>  
>  	/* Move the dirty bit to the physical page now the pte is gone. */
> -	if (pte_dirty(pteval))
> +	dirty = pte_dirty(pteval);
> +	if (dirty)
>  		set_page_dirty(page);
>  
>  	/* Update high watermark before we lower rss */
> @@ -1175,6 +1217,15 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  		swp_entry_t entry = { .val = page_private(page) };
>  		pte_t swp_pte;
>  
> +		if ((TTU_ACTION(flags) == TTU_UNMAP) &&
> +				page_discardable(page, dirty)) {
> +			dec_mm_counter(mm, MM_ANONPAGES);
> +			goto discard;
> +		}
> +
> +		if (dirty)
> +			rp->pte_dirty++;
> +
>  		if (PageSwapCache(page)) {
>  			/*
>  			 * Store the swap location in the pte.
> @@ -1193,6 +1244,10 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  			}
>  			dec_mm_counter(mm, MM_ANONPAGES);
>  			inc_mm_counter(mm, MM_SWAPENTS);
> +		} else if (TTU_ACTION(flags) == TTU_UNMAP) {
> +			set_pte_at(mm, address, pte, pteval);
> +			ret = SWAP_FAIL;
> +			goto out_unmap;
>  		} else if (IS_ENABLED(CONFIG_MIGRATION)) {
>  			/*
>  			 * Store the pfn of the page in a special migration
> @@ -1216,6 +1271,7 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  	} else
>  		dec_mm_counter(mm, MM_FILEPAGES);
>  
> +discard:
>  	page_remove_rmap(page);
>  	page_cache_release(page);
>  
> @@ -1486,13 +1542,19 @@ static int page_not_mapped(struct page *page)
>   * SWAP_AGAIN	- we missed a mapping, try again later
>   * SWAP_FAIL	- the page is unswappable
>   * SWAP_MLOCK	- page is mlocked.
> + * SWAP_DISCARD - same with SWAP_SUCCESS but no need to swap out
>   */
>  int try_to_unmap(struct page *page, enum ttu_flags flags)
>  {
>  	int ret;
> +
> +	struct rmap_private rp = {
> +		.flags = flags,
> +	};
> +
>  	struct rmap_walk_control rwc = {
>  		.rmap_one = try_to_unmap_one,
> -		.arg = (void *)flags,
> +		.arg = &rp,
>  		.done = page_not_mapped,
>  		.file_nonlinear = try_to_unmap_nonlinear,
>  		.anon_lock = page_lock_anon_vma_read,
> @@ -1513,8 +1575,13 @@ int try_to_unmap(struct page *page, enum ttu_flags flags)
>  
>  	ret = rmap_walk(page, &rwc);
>  
> -	if (ret != SWAP_MLOCK && !page_mapped(page))
> +	if (ret != SWAP_MLOCK && !page_mapped(page)) {
>  		ret = SWAP_SUCCESS;
> +		if (TTU_ACTION(flags) == TTU_UNMAP &&
> +			page_discardable(page, rp.pte_dirty))
> +			ret = SWAP_DISCARD;
> +	}
> +
>  	return ret;
>  }
>  
> @@ -1536,9 +1603,13 @@ int try_to_unmap(struct page *page, enum ttu_flags flags)
>  int try_to_munlock(struct page *page)
>  {
>  	int ret;
> +	struct rmap_private rp = {
> +		.flags = TTU_MUNLOCK,
> +	};
> +
>  	struct rmap_walk_control rwc = {
>  		.rmap_one = try_to_unmap_one,
> -		.arg = (void *)TTU_MUNLOCK,
> +		.arg = &rp,
>  		.done = page_not_mapped,
>  		/*
>  		 * We don't bother to try to find the munlocked page in
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 6cdda104f629..243da05cc24d 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -698,6 +698,7 @@ enum page_references {
>  	PAGEREF_RECLAIM_CLEAN,
>  	PAGEREF_KEEP,
>  	PAGEREF_ACTIVATE,
> +	PAGEREF_DISCARD,
>  };
>  
>  static enum page_references page_check_references(struct page *page,
> @@ -705,9 +706,12 @@ static enum page_references page_check_references(struct page *page,
>  {
>  	int referenced_ptes, referenced_page;
>  	unsigned long vm_flags;
> +	int is_pte_dirty;
> +
> +	VM_BUG_ON_PAGE(!PageLocked(page), page);
>  
>  	referenced_ptes = page_referenced(page, 1, sc->target_mem_cgroup,
> -					  &vm_flags);
> +					  &vm_flags, &is_pte_dirty);
>  	referenced_page = TestClearPageReferenced(page);
>  
>  	/*
> @@ -748,6 +752,9 @@ static enum page_references page_check_references(struct page *page,
>  		return PAGEREF_KEEP;
>  	}
>  
> +	if (page_discardable(page, is_pte_dirty))
> +		return PAGEREF_DISCARD;
> +
>  	/* Reclaim if clean, defer dirty pages to writeback */
>  	if (referenced_page && !PageSwapBacked(page))
>  		return PAGEREF_RECLAIM_CLEAN;
> @@ -817,6 +824,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
>  		int may_enter_fs;
>  		enum page_references references = PAGEREF_RECLAIM_CLEAN;
>  		bool dirty, writeback;
> +		bool discard = false;
>  
>  		cond_resched();
>  
> @@ -946,6 +954,12 @@ static unsigned long shrink_page_list(struct list_head *page_list,
>  			goto activate_locked;
>  		case PAGEREF_KEEP:
>  			goto keep_locked;
> +		case PAGEREF_DISCARD:
> +			/*
> +			 * skip to add the page into swapcache  then
> +			 * unmap without mapping
> +			 */
> +			discard = true;
>  		case PAGEREF_RECLAIM:
>  		case PAGEREF_RECLAIM_CLEAN:
>  			; /* try to reclaim the page below */
> @@ -955,7 +969,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
>  		 * Anonymous process memory has backing store?
>  		 * Try to allocate it some swap space here.
>  		 */
> -		if (PageAnon(page) && !PageSwapCache(page)) {
> +		if (PageAnon(page) && !PageSwapCache(page) && !discard) {
>  			if (!(sc->gfp_mask & __GFP_IO))
>  				goto keep_locked;
>  			if (!add_to_swap(page, page_list))
> @@ -970,7 +984,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
>  		 * The page is mapped into the page tables of one or more
>  		 * processes. Try to unmap it here.
>  		 */
> -		if (page_mapped(page) && mapping) {
> +		if (page_mapped(page) && (mapping || discard)) {
>  			switch (try_to_unmap(page, ttu_flags)) {
>  			case SWAP_FAIL:
>  				goto activate_locked;
> @@ -978,6 +992,13 @@ static unsigned long shrink_page_list(struct list_head *page_list,
>  				goto keep_locked;
>  			case SWAP_MLOCK:
>  				goto cull_mlocked;
> +			case SWAP_DISCARD:
> +				VM_BUG_ON_PAGE(PageSwapCache(page), page);
> +				if (!page_freeze_refs(page, 1))
> +					goto keep_locked;
> +				__clear_page_locked(page);
> +				count_vm_event(PGLAZYFREED);
> +				goto free_it;
>  			case SWAP_SUCCESS:
>  				; /* try to free the page below */
>  			}
> @@ -1702,7 +1723,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
>  		}
>  
>  		if (page_referenced(page, 0, sc->target_mem_cgroup,
> -				    &vm_flags)) {
> +				    &vm_flags, NULL)) {
>  			nr_rotated += hpage_nr_pages(page);
>  			/*
>  			 * Identify referenced, file-backed active pages and
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 0b295fd8a60a..dcf6c6ab723e 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -792,6 +792,7 @@ const char * const vmstat_text[] = {
>  
>  	"pgfault",
>  	"pgmajfault",
> +	"pglazyfreed",
>  
>  	TEXTS_FOR_ZONES("pgrefill")
>  	TEXTS_FOR_ZONES("pgsteal_kswapd")
> -- 
> 1.9.2
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v5] mm: support madvise(MADV_FREE)

[Rik van Riel]

On 04/20/2014 09:56 PM, Minchan Kim wrote:

> In summary, MADV_FREE is about 2 time faster than MADV_DONTNEED.

This is awesome.

I have a few nitpicks with the patch, though :)

> +static long madvise_lazyfree(struct vm_area_struct *vma,
> +			     struct vm_area_struct **prev,
> +			     unsigned long start, unsigned long end)
> +{
> +	*prev = vma;
> +	if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
> +		return -EINVAL;
> +
> +	/* MADV_FREE works for only anon vma at the moment */
> +	if (vma->vm_file)
> +		return -EINVAL;
> +
> +	lazyfree_range(vma, start, end - start);
> +	return 0;
> +}

This code checks whether lazyfree_range would work on
the VMA...

> diff --git a/mm/memory.c b/mm/memory.c
> index c4b5bc250820..ca427f258204 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -1270,6 +1270,104 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
>   	return addr;
>   }
>
> +static unsigned long lazyfree_pte_range(struct mmu_gather *tlb,
> +				struct vm_area_struct *vma, pmd_t *pmd,
> +				unsigned long addr, unsigned long end)
> +{
> +	struct mm_struct *mm = tlb->mm;
> +	spinlock_t *ptl;
> +	pte_t *start_pte;
> +	pte_t *pte;
> +
> +	start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
> +	pte = start_pte;
> +	arch_enter_lazy_mmu_mode();
> +	do {
> +		pte_t ptent = *pte;
> +
> +		if (pte_none(ptent))
> +			continue;
> +
> +		if (!pte_present(ptent))
> +			continue;
> +
> +		ptent = pte_mkold(ptent);
> +		ptent = pte_mkclean(ptent);
> +		set_pte_at(mm, addr, pte, ptent);
> +		tlb_remove_tlb_entry(tlb, pte, addr);

This may not work on PPC, which has a weird hash table for
its TLB. You will find that tlb_remove_tlb_entry does
nothing for PPC64, and set_pte_at does not remove the hash
table entry either.

> @@ -1370,6 +1485,31 @@ void unmap_vmas(struct mmu_gather *tlb,
>   }
>
>   /**
> + * lazyfree_range - clear dirty bit of pte in a given range
> + * @vma: vm_area_struct holding the applicable pages
> + * @start: starting address of pages
> + * @size: number of bytes to do lazyfree
> + *
> + * Caller must protect the VMA list
> + */
> +void lazyfree_range(struct vm_area_struct *vma, unsigned long start,
> +		unsigned long size)
> +{
> +	struct mm_struct *mm = vma->vm_mm;
> +	struct mmu_gather tlb;
> +	unsigned long end = start + size;
> +
> +	lru_add_drain();
> +	tlb_gather_mmu(&tlb, mm, start, end);
> +	update_hiwater_rss(mm);
> +	mmu_notifier_invalidate_range_start(mm, start, end);
> +	for ( ; vma && vma->vm_start < end; vma = vma->vm_next)
> +		lazyfree_single_vma(&tlb, vma, start, end);
> +	mmu_notifier_invalidate_range_end(mm, start, end);
> +	tlb_finish_mmu(&tlb, start, end);
> +}

This function, called by madvise_lazyfree, can iterate
over multiple VMAs.

However, madvise_lazyfree only checked one of them.

What should happen when the code encounters a VMA where
MADV_FREE does not work?  Should it return an error?
Should it skip over it?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v5] mm: support madvise(MADV_FREE)

[Minchan Kim]

Hello Rik,

On Thu, May 08, 2014 at 04:04:33PM -0400, Rik van Riel wrote:
> On 04/20/2014 09:56 PM, Minchan Kim wrote:
> 
> >In summary, MADV_FREE is about 2 time faster than MADV_DONTNEED.
> 
> This is awesome.

Thanks!

> 
> I have a few nitpicks with the patch, though :)
> 
> >+static long madvise_lazyfree(struct vm_area_struct *vma,
> >+			     struct vm_area_struct **prev,
> >+			     unsigned long start, unsigned long end)
> >+{
> >+	*prev = vma;
> >+	if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
> >+		return -EINVAL;
> >+
> >+	/* MADV_FREE works for only anon vma at the moment */
> >+	if (vma->vm_file)
> >+		return -EINVAL;
> >+
> >+	lazyfree_range(vma, start, end - start);
> >+	return 0;
> >+}
> 
> This code checks whether lazyfree_range would work on
> the VMA...
> 
> >diff --git a/mm/memory.c b/mm/memory.c
> >index c4b5bc250820..ca427f258204 100644
> >--- a/mm/memory.c
> >+++ b/mm/memory.c
> >@@ -1270,6 +1270,104 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
> >  	return addr;
> >  }
> >
> >+static unsigned long lazyfree_pte_range(struct mmu_gather *tlb,
> >+				struct vm_area_struct *vma, pmd_t *pmd,
> >+				unsigned long addr, unsigned long end)
> >+{
> >+	struct mm_struct *mm = tlb->mm;
> >+	spinlock_t *ptl;
> >+	pte_t *start_pte;
> >+	pte_t *pte;
> >+
> >+	start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
> >+	pte = start_pte;
> >+	arch_enter_lazy_mmu_mode();
> >+	do {
> >+		pte_t ptent = *pte;
> >+
> >+		if (pte_none(ptent))
> >+			continue;
> >+
> >+		if (!pte_present(ptent))
> >+			continue;
> >+
> >+		ptent = pte_mkold(ptent);
> >+		ptent = pte_mkclean(ptent);
> >+		set_pte_at(mm, addr, pte, ptent);
> >+		tlb_remove_tlb_entry(tlb, pte, addr);
> 
> This may not work on PPC, which has a weird hash table for
> its TLB. You will find that tlb_remove_tlb_entry does
> nothing for PPC64, and set_pte_at does not remove the hash
> table entry either.

Hmm, I didn't notice that. Thanks Rik.

Maybe I need this in asm-generic.

static inline void ptep_set_lazyfree(struct mm_struct *mm, unsigned addr, pte_t *ptep)
{
        pte_t ptent = *ptep;
        ptent = pte_mkold(ptent);
        ptent = pte_mkclean(ptent);
        set_pte_at(mm, addr, ptep, ptent);
}

For arch/powerpc/include/asm/pgtable.h

static inline void ptep_set_lazyfree(struct mm_struct *mm, unsigned long addr,
                        pte_t *ptep)
{
        pte_update(mm, addr, ptep, _PAGE_DIRTY|_PAGE_ACCESSED, 0, 0);
}

> 
> >@@ -1370,6 +1485,31 @@ void unmap_vmas(struct mmu_gather *tlb,
> >  }
> >
> >  /**
> >+ * lazyfree_range - clear dirty bit of pte in a given range
> >+ * @vma: vm_area_struct holding the applicable pages
> >+ * @start: starting address of pages
> >+ * @size: number of bytes to do lazyfree
> >+ *
> >+ * Caller must protect the VMA list
> >+ */
> >+void lazyfree_range(struct vm_area_struct *vma, unsigned long start,
> >+		unsigned long size)
> >+{
> >+	struct mm_struct *mm = vma->vm_mm;
> >+	struct mmu_gather tlb;
> >+	unsigned long end = start + size;
> >+
> >+	lru_add_drain();
> >+	tlb_gather_mmu(&tlb, mm, start, end);
> >+	update_hiwater_rss(mm);
> >+	mmu_notifier_invalidate_range_start(mm, start, end);
> >+	for ( ; vma && vma->vm_start < end; vma = vma->vm_next)
> >+		lazyfree_single_vma(&tlb, vma, start, end);
> >+	mmu_notifier_invalidate_range_end(mm, start, end);
> >+	tlb_finish_mmu(&tlb, start, end);
> >+}
> 
> This function, called by madvise_lazyfree, can iterate
> over multiple VMAs.
> 
> However, madvise_lazyfree only checked one of them.

Oops, the check should have been lazyfree_range.
Will fix.

> 
> What should happen when the code encounters a VMA where
> MADV_FREE does not work?  Should it return an error?
> Should it skip over it?

I'd like to maintain it same as DONTNEED sematic so it could return
just EINVAL.

Thanks for the review!

-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v5] mm: support madvise(MADV_FREE)

[Minchan Kim]

On Fri, May 09, 2014 at 03:17:14PM +0900, Minchan Kim wrote:
> Hello Rik,
> 
> On Thu, May 08, 2014 at 04:04:33PM -0400, Rik van Riel wrote:
> > On 04/20/2014 09:56 PM, Minchan Kim wrote:
> > 
> > >In summary, MADV_FREE is about 2 time faster than MADV_DONTNEED.
> > 
> > This is awesome.
> 
> Thanks!
> 
> > 
> > I have a few nitpicks with the patch, though :)
> > 
> > >+static long madvise_lazyfree(struct vm_area_struct *vma,
> > >+			     struct vm_area_struct **prev,
> > >+			     unsigned long start, unsigned long end)
> > >+{
> > >+	*prev = vma;
> > >+	if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
> > >+		return -EINVAL;
> > >+
> > >+	/* MADV_FREE works for only anon vma at the moment */
> > >+	if (vma->vm_file)
> > >+		return -EINVAL;
> > >+
> > >+	lazyfree_range(vma, start, end - start);
> > >+	return 0;
> > >+}
> > 
> > This code checks whether lazyfree_range would work on
> > the VMA...
> > 
> > >diff --git a/mm/memory.c b/mm/memory.c
> > >index c4b5bc250820..ca427f258204 100644
> > >--- a/mm/memory.c
> > >+++ b/mm/memory.c
> > >@@ -1270,6 +1270,104 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
> > >  	return addr;
> > >  }
> > >
> > >+static unsigned long lazyfree_pte_range(struct mmu_gather *tlb,
> > >+				struct vm_area_struct *vma, pmd_t *pmd,
> > >+				unsigned long addr, unsigned long end)
> > >+{
> > >+	struct mm_struct *mm = tlb->mm;
> > >+	spinlock_t *ptl;
> > >+	pte_t *start_pte;
> > >+	pte_t *pte;
> > >+
> > >+	start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
> > >+	pte = start_pte;
> > >+	arch_enter_lazy_mmu_mode();
> > >+	do {
> > >+		pte_t ptent = *pte;
> > >+
> > >+		if (pte_none(ptent))
> > >+			continue;
> > >+
> > >+		if (!pte_present(ptent))
> > >+			continue;
> > >+
> > >+		ptent = pte_mkold(ptent);
> > >+		ptent = pte_mkclean(ptent);
> > >+		set_pte_at(mm, addr, pte, ptent);
> > >+		tlb_remove_tlb_entry(tlb, pte, addr);
> > 
> > This may not work on PPC, which has a weird hash table for
> > its TLB. You will find that tlb_remove_tlb_entry does
> > nothing for PPC64, and set_pte_at does not remove the hash
> > table entry either.
> 
> Hmm, I didn't notice that. Thanks Rik.
> 
> Maybe I need this in asm-generic.
> 
> static inline void ptep_set_lazyfree(struct mm_struct *mm, unsigned addr, pte_t *ptep)
> {
>         pte_t ptent = *ptep;
>         ptent = pte_mkold(ptent);
>         ptent = pte_mkclean(ptent);
>         set_pte_at(mm, addr, ptep, ptent);
> }
> 
> For arch/powerpc/include/asm/pgtable.h
> 
> static inline void ptep_set_lazyfree(struct mm_struct *mm, unsigned long addr,
>                         pte_t *ptep)
> {
>         pte_update(mm, addr, ptep, _PAGE_DIRTY|_PAGE_ACCESSED, 0, 0);
> }
> 
> > 
> > >@@ -1370,6 +1485,31 @@ void unmap_vmas(struct mmu_gather *tlb,
> > >  }
> > >
> > >  /**
> > >+ * lazyfree_range - clear dirty bit of pte in a given range
> > >+ * @vma: vm_area_struct holding the applicable pages
> > >+ * @start: starting address of pages
> > >+ * @size: number of bytes to do lazyfree
> > >+ *
> > >+ * Caller must protect the VMA list
> > >+ */
> > >+void lazyfree_range(struct vm_area_struct *vma, unsigned long start,
> > >+		unsigned long size)
> > >+{
> > >+	struct mm_struct *mm = vma->vm_mm;
> > >+	struct mmu_gather tlb;
> > >+	unsigned long end = start + size;
> > >+
> > >+	lru_add_drain();
> > >+	tlb_gather_mmu(&tlb, mm, start, end);
> > >+	update_hiwater_rss(mm);
> > >+	mmu_notifier_invalidate_range_start(mm, start, end);
> > >+	for ( ; vma && vma->vm_start < end; vma = vma->vm_next)
> > >+		lazyfree_single_vma(&tlb, vma, start, end);
> > >+	mmu_notifier_invalidate_range_end(mm, start, end);
> > >+	tlb_finish_mmu(&tlb, start, end);
> > >+}
> > 
> > This function, called by madvise_lazyfree, can iterate
> > over multiple VMAs.
> > 
> > However, madvise_lazyfree only checked one of them.
> 
> Oops, the check should have been lazyfree_range.
> Will fix.

Now that I see the code, madvise_vma always pass *a* vma so madvise_lazyfree
doesn't cover multiple vma all at once so the current sematic is same with
dontneed. So, I don't see any problem. If I miss something, let me know it.

-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v5] mm: support madvise(MADV_FREE)

[Rik van Riel]

On 05/09/2014 02:28 AM, Minchan Kim wrote:
> On Fri, May 09, 2014 at 03:17:14PM +0900, Minchan Kim wrote:
>> Hello Rik,
>>
>> On Thu, May 08, 2014 at 04:04:33PM -0400, Rik van Riel wrote:
>>> On 04/20/2014 09:56 PM, Minchan Kim wrote:
>>>
>>>> In summary, MADV_FREE is about 2 time faster than MADV_DONTNEED.
>>>
>>> This is awesome.
>>
>> Thanks!
>>
>>>
>>> I have a few nitpicks with the patch, though :)
>>>
>>>> +static long madvise_lazyfree(struct vm_area_struct *vma,
>>>> +			     struct vm_area_struct **prev,
>>>> +			     unsigned long start, unsigned long end)
>>>> +{
>>>> +	*prev = vma;
>>>> +	if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
>>>> +		return -EINVAL;
>>>> +
>>>> +	/* MADV_FREE works for only anon vma at the moment */
>>>> +	if (vma->vm_file)
>>>> +		return -EINVAL;
>>>> +
>>>> +	lazyfree_range(vma, start, end - start);
>>>> +	return 0;
>>>> +}
>>>
>>> This code checks whether lazyfree_range would work on
>>> the VMA...
>>>
>>>> diff --git a/mm/memory.c b/mm/memory.c
>>>> index c4b5bc250820..ca427f258204 100644
>>>> --- a/mm/memory.c
>>>> +++ b/mm/memory.c
>>>> @@ -1270,6 +1270,104 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
>>>>   	return addr;
>>>>   }
>>>>
>>>> +static unsigned long lazyfree_pte_range(struct mmu_gather *tlb,
>>>> +				struct vm_area_struct *vma, pmd_t *pmd,
>>>> +				unsigned long addr, unsigned long end)
>>>> +{
>>>> +	struct mm_struct *mm = tlb->mm;
>>>> +	spinlock_t *ptl;
>>>> +	pte_t *start_pte;
>>>> +	pte_t *pte;
>>>> +
>>>> +	start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
>>>> +	pte = start_pte;
>>>> +	arch_enter_lazy_mmu_mode();
>>>> +	do {
>>>> +		pte_t ptent = *pte;
>>>> +
>>>> +		if (pte_none(ptent))
>>>> +			continue;
>>>> +
>>>> +		if (!pte_present(ptent))
>>>> +			continue;
>>>> +
>>>> +		ptent = pte_mkold(ptent);
>>>> +		ptent = pte_mkclean(ptent);
>>>> +		set_pte_at(mm, addr, pte, ptent);
>>>> +		tlb_remove_tlb_entry(tlb, pte, addr);
>>>
>>> This may not work on PPC, which has a weird hash table for
>>> its TLB. You will find that tlb_remove_tlb_entry does
>>> nothing for PPC64, and set_pte_at does not remove the hash
>>> table entry either.
>>
>> Hmm, I didn't notice that. Thanks Rik.
>>
>> Maybe I need this in asm-generic.
>>
>> static inline void ptep_set_lazyfree(struct mm_struct *mm, unsigned addr, pte_t *ptep)
>> {
>>          pte_t ptent = *ptep;
>>          ptent = pte_mkold(ptent);
>>          ptent = pte_mkclean(ptent);
>>          set_pte_at(mm, addr, ptep, ptent);
>> }
>>
>> For arch/powerpc/include/asm/pgtable.h
>>
>> static inline void ptep_set_lazyfree(struct mm_struct *mm, unsigned long addr,
>>                          pte_t *ptep)
>> {
>>          pte_update(mm, addr, ptep, _PAGE_DIRTY|_PAGE_ACCESSED, 0, 0);
>> }
>>
>>>
>>>> @@ -1370,6 +1485,31 @@ void unmap_vmas(struct mmu_gather *tlb,
>>>>   }
>>>>
>>>>   /**
>>>> + * lazyfree_range - clear dirty bit of pte in a given range
>>>> + * @vma: vm_area_struct holding the applicable pages
>>>> + * @start: starting address of pages
>>>> + * @size: number of bytes to do lazyfree
>>>> + *
>>>> + * Caller must protect the VMA list
>>>> + */
>>>> +void lazyfree_range(struct vm_area_struct *vma, unsigned long start,
>>>> +		unsigned long size)
>>>> +{
>>>> +	struct mm_struct *mm = vma->vm_mm;
>>>> +	struct mmu_gather tlb;
>>>> +	unsigned long end = start + size;
>>>> +
>>>> +	lru_add_drain();
>>>> +	tlb_gather_mmu(&tlb, mm, start, end);
>>>> +	update_hiwater_rss(mm);
>>>> +	mmu_notifier_invalidate_range_start(mm, start, end);
>>>> +	for ( ; vma && vma->vm_start < end; vma = vma->vm_next)
>>>> +		lazyfree_single_vma(&tlb, vma, start, end);
>>>> +	mmu_notifier_invalidate_range_end(mm, start, end);
>>>> +	tlb_finish_mmu(&tlb, start, end);
>>>> +}
>>>
>>> This function, called by madvise_lazyfree, can iterate
>>> over multiple VMAs.
>>>
>>> However, madvise_lazyfree only checked one of them.
>>
>> Oops, the check should have been lazyfree_range.
>> Will fix.
>
> Now that I see the code, madvise_vma always pass *a* vma so madvise_lazyfree
> doesn't cover multiple vma all at once so the current sematic is same with
> dontneed. So, I don't see any problem. If I miss something, let me know it.
>

Does that mean lazyfree_range is unnecessary, and everything
can be done inside lazyfree_single_vma ?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH v5] mm: support madvise(MADV_FREE)

[Minchan Kim]

On Fri, May 09, 2014 at 11:12:01AM -0400, Rik van Riel wrote:
> On 05/09/2014 02:28 AM, Minchan Kim wrote:
> >On Fri, May 09, 2014 at 03:17:14PM +0900, Minchan Kim wrote:
> >>Hello Rik,
> >>
> >>On Thu, May 08, 2014 at 04:04:33PM -0400, Rik van Riel wrote:
> >>>On 04/20/2014 09:56 PM, Minchan Kim wrote:
> >>>
> >>>>In summary, MADV_FREE is about 2 time faster than MADV_DONTNEED.
> >>>
> >>>This is awesome.
> >>
> >>Thanks!
> >>
> >>>
> >>>I have a few nitpicks with the patch, though :)
> >>>
> >>>>+static long madvise_lazyfree(struct vm_area_struct *vma,
> >>>>+			     struct vm_area_struct **prev,
> >>>>+			     unsigned long start, unsigned long end)
> >>>>+{
> >>>>+	*prev = vma;
> >>>>+	if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
> >>>>+		return -EINVAL;
> >>>>+
> >>>>+	/* MADV_FREE works for only anon vma at the moment */
> >>>>+	if (vma->vm_file)
> >>>>+		return -EINVAL;
> >>>>+
> >>>>+	lazyfree_range(vma, start, end - start);
> >>>>+	return 0;
> >>>>+}
> >>>
> >>>This code checks whether lazyfree_range would work on
> >>>the VMA...
> >>>
> >>>>diff --git a/mm/memory.c b/mm/memory.c
> >>>>index c4b5bc250820..ca427f258204 100644
> >>>>--- a/mm/memory.c
> >>>>+++ b/mm/memory.c
> >>>>@@ -1270,6 +1270,104 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
> >>>>  	return addr;
> >>>>  }
> >>>>
> >>>>+static unsigned long lazyfree_pte_range(struct mmu_gather *tlb,
> >>>>+				struct vm_area_struct *vma, pmd_t *pmd,
> >>>>+				unsigned long addr, unsigned long end)
> >>>>+{
> >>>>+	struct mm_struct *mm = tlb->mm;
> >>>>+	spinlock_t *ptl;
> >>>>+	pte_t *start_pte;
> >>>>+	pte_t *pte;
> >>>>+
> >>>>+	start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
> >>>>+	pte = start_pte;
> >>>>+	arch_enter_lazy_mmu_mode();
> >>>>+	do {
> >>>>+		pte_t ptent = *pte;
> >>>>+
> >>>>+		if (pte_none(ptent))
> >>>>+			continue;
> >>>>+
> >>>>+		if (!pte_present(ptent))
> >>>>+			continue;
> >>>>+
> >>>>+		ptent = pte_mkold(ptent);
> >>>>+		ptent = pte_mkclean(ptent);
> >>>>+		set_pte_at(mm, addr, pte, ptent);
> >>>>+		tlb_remove_tlb_entry(tlb, pte, addr);
> >>>
> >>>This may not work on PPC, which has a weird hash table for
> >>>its TLB. You will find that tlb_remove_tlb_entry does
> >>>nothing for PPC64, and set_pte_at does not remove the hash
> >>>table entry either.
> >>
> >>Hmm, I didn't notice that. Thanks Rik.
> >>
> >>Maybe I need this in asm-generic.
> >>
> >>static inline void ptep_set_lazyfree(struct mm_struct *mm, unsigned addr, pte_t *ptep)
> >>{
> >>         pte_t ptent = *ptep;
> >>         ptent = pte_mkold(ptent);
> >>         ptent = pte_mkclean(ptent);
> >>         set_pte_at(mm, addr, ptep, ptent);
> >>}
> >>
> >>For arch/powerpc/include/asm/pgtable.h
> >>
> >>static inline void ptep_set_lazyfree(struct mm_struct *mm, unsigned long addr,
> >>                         pte_t *ptep)
> >>{
> >>         pte_update(mm, addr, ptep, _PAGE_DIRTY|_PAGE_ACCESSED, 0, 0);
> >>}
> >>
> >>>
> >>>>@@ -1370,6 +1485,31 @@ void unmap_vmas(struct mmu_gather *tlb,
> >>>>  }
> >>>>
> >>>>  /**
> >>>>+ * lazyfree_range - clear dirty bit of pte in a given range
> >>>>+ * @vma: vm_area_struct holding the applicable pages
> >>>>+ * @start: starting address of pages
> >>>>+ * @size: number of bytes to do lazyfree
> >>>>+ *
> >>>>+ * Caller must protect the VMA list
> >>>>+ */
> >>>>+void lazyfree_range(struct vm_area_struct *vma, unsigned long start,
> >>>>+		unsigned long size)
> >>>>+{
> >>>>+	struct mm_struct *mm = vma->vm_mm;
> >>>>+	struct mmu_gather tlb;
> >>>>+	unsigned long end = start + size;
> >>>>+
> >>>>+	lru_add_drain();
> >>>>+	tlb_gather_mmu(&tlb, mm, start, end);
> >>>>+	update_hiwater_rss(mm);
> >>>>+	mmu_notifier_invalidate_range_start(mm, start, end);
> >>>>+	for ( ; vma && vma->vm_start < end; vma = vma->vm_next)
> >>>>+		lazyfree_single_vma(&tlb, vma, start, end);
> >>>>+	mmu_notifier_invalidate_range_end(mm, start, end);
> >>>>+	tlb_finish_mmu(&tlb, start, end);
> >>>>+}
> >>>
> >>>This function, called by madvise_lazyfree, can iterate
> >>>over multiple VMAs.
> >>>
> >>>However, madvise_lazyfree only checked one of them.
> >>
> >>Oops, the check should have been lazyfree_range.
> >>Will fix.
> >
> >Now that I see the code, madvise_vma always pass *a* vma so madvise_lazyfree
> >doesn't cover multiple vma all at once so the current sematic is same with
> >dontneed. So, I don't see any problem. If I miss something, let me know it.
> >
> 
> Does that mean lazyfree_range is unnecessary, and everything
> can be done inside lazyfree_single_vma ?

Yeb, Will resend.
Thanks!

-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>