[patch 0/4] mm: A few memory tracker fixes

[patch 0/4] mm: A few memory tracker fixes

[Cyrill Gorcunov]

Hi! Here is a few fixes for memory softdirty tracker inspired by
LKML thread https://lkml.org/lkml/2014/3/18/709 . It turned out
that indeed I've missed to setup softdirty bit on file mappings
in a few places. But it seems the only intensive user of this
feature is the criu tool where we dump/restore shared memory
areas completely regardless the softdirty bit status and
shared file mapped areas are not dumped at all so I'm not
sure if the patches are stable material. Please take a look,
thanks.

	Cyrill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 1/4] mm: Make freshly remapped file pages being softdirty unconditionally

[Cyrill Gorcunov]

[-- Attachment #1: mm-file-pte-softdirty-update-2 --]
[-- Type: text/plain, Size: 1769 bytes --]

Hugh reported:

 | I noticed your soft_dirty work in install_file_pte(): which looked
 | good at first, until I realized that it's propagating the soft_dirty
 | of a pte it's about to zap completely, to the unrelated entry it's
 | about to insert in its place.  Which seems very odd to me.

Indeed this code ends up being nop in result -- pte_file_mksoft_dirty()
operates with pte_t argument and returns new pte_t which were never
used after. After looking more I think what we need is to soft-dirtify
all newely remapped file pages because it should look like a new mapping
for memory tracker.

Reported-by: Hugh Dickins <hughd@google.com>
CC: Pavel Emelyanov <xemul@parallels.com>
CC: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
---
 mm/fremap.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

Index: linux-2.6.git/mm/fremap.c
===================================================================
--- linux-2.6.git.orig/mm/fremap.c
+++ linux-2.6.git/mm/fremap.c
@@ -66,13 +66,10 @@ static int install_file_pte(struct mm_st
 
 	ptfile = pgoff_to_pte(pgoff);
 
-	if (!pte_none(*pte)) {
-		if (pte_present(*pte) && pte_soft_dirty(*pte))
-			pte_file_mksoft_dirty(ptfile);
+	if (!pte_none(*pte))
 		zap_pte(mm, vma, addr, pte);
-	}
 
-	set_pte_at(mm, addr, pte, ptfile);
+	set_pte_at(mm, addr, pte, pte_file_mksoft_dirty(ptfile));
 	/*
 	 * We don't need to run update_mmu_cache() here because the "file pte"
 	 * being installed by install_file_pte() is not a real pte - it's a

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 2/4] mm: Dont forget to set softdirty on file mapped fault

[Cyrill Gorcunov]

[-- Attachment #1: mm-file-pte-softdirty-on-fault --]
[-- Type: text/plain, Size: 1046 bytes --]

Otherwise we may not notice that pte was softdirty.

CC: Pavel Emelyanov <xemul@parallels.com>
CC: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
---
 mm/memory.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux-2.6.git/mm/memory.c
===================================================================
--- linux-2.6.git.orig/mm/memory.c
+++ linux-2.6.git/mm/memory.c
@@ -3422,7 +3422,7 @@ static int __do_fault(struct mm_struct *
 		if (flags & FAULT_FLAG_WRITE)
 			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 		else if (pte_file(orig_pte) && pte_file_soft_dirty(orig_pte))
-			pte_mksoft_dirty(entry);
+			entry = pte_mksoft_dirty(entry);
 		if (anon) {
 			inc_mm_counter_fast(mm, MM_ANONPAGES);
 			page_add_new_anon_rmap(page, vma, address);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 2/4] mm: Dont forget to set softdirty on file mapped fault

[Andrew Morton]

On Mon, 24 Mar 2014 16:28:40 +0400 Cyrill Gorcunov <gorcunov@openvz.org> wrote:

> Otherwise we may not notice that pte was softdirty.
> 
> --- linux-2.6.git.orig/mm/memory.c
> +++ linux-2.6.git/mm/memory.c
> @@ -3422,7 +3422,7 @@ static int __do_fault(struct mm_struct *
>  		if (flags & FAULT_FLAG_WRITE)
>  			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
>  		else if (pte_file(orig_pte) && pte_file_soft_dirty(orig_pte))
> -			pte_mksoft_dirty(entry);
> +			entry = pte_mksoft_dirty(entry);
>  		if (anon) {
>  			inc_mm_counter_fast(mm, MM_ANONPAGES);
>  			page_add_new_anon_rmap(page, vma, address);

This will need to be redone for current kernels, please.  New patch, new
title, new changelog, retest.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 2/4] mm: Dont forget to set softdirty on file mapped fault

[Cyrill Gorcunov]

On Mon, Apr 14, 2014 at 03:27:58PM -0700, Andrew Morton wrote:
> 
> This will need to be redone for current kernels, please.  New patch, new
> title, new changelog, retest.

Sure, will resend once done.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 2/4] mm: Dont forget to set softdirty on file mapped fault

[Cyrill Gorcunov]

On Tue, Apr 15, 2014 at 02:33:09AM +0400, Cyrill Gorcunov wrote:
> On Mon, Apr 14, 2014 at 03:27:58PM -0700, Andrew Morton wrote:
> > 
> > This will need to be redone for current kernels, please.  New patch, new
> > title, new changelog, retest.
> 
> Sure, will resend once done.

Andrew, sorry, I'm a bit confused, jost got notifications you've picked them
up into -mm tree, right? So what I should do now -- by "current kernels" you
mean latest Linus's git repo?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 2/4] mm: Dont forget to set softdirty on file mapped fault

[Andrew Morton]

On Tue, 15 Apr 2014 02:38:21 +0400 Cyrill Gorcunov <gorcunov@gmail.com> wrote:

> On Tue, Apr 15, 2014 at 02:33:09AM +0400, Cyrill Gorcunov wrote:
> > On Mon, Apr 14, 2014 at 03:27:58PM -0700, Andrew Morton wrote:
> > > 
> > > This will need to be redone for current kernels, please.  New patch, new
> > > title, new changelog, retest.
> > 
> > Sure, will resend once done.
> 
> Andrew, sorry, I'm a bit confused, jost got notifications you've picked them
> up into -mm tree, right?

1, 3 and 4.

> So what I should do now -- by "current kernels" you
> mean latest Linus's git repo?

yup.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 3/4] mm: Dont forget to save file map softdiry bit on unmap

[Cyrill Gorcunov]

[-- Attachment #1: mm-file-pte-softdirty-try_to_unmap_cluster --]
[-- Type: text/plain, Size: 1124 bytes --]

pte_file_mksoft_dirty operates with argument passed by
a value and returns modified result thus need to assign
@ptfile here, otherwise it's nop operation which may lead
to lose of softdirty bit.

CC: Pavel Emelyanov <xemul@parallels.com>
CC: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
---
 mm/rmap.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux-2.6.git/mm/rmap.c
===================================================================
--- linux-2.6.git.orig/mm/rmap.c
+++ linux-2.6.git/mm/rmap.c
@@ -1339,7 +1339,7 @@ static int try_to_unmap_cluster(unsigned
 		if (page->index != linear_page_index(vma, address)) {
 			pte_t ptfile = pgoff_to_pte(page->index);
 			if (pte_soft_dirty(pteval))
-				pte_file_mksoft_dirty(ptfile);
+				ptfile = pte_file_mksoft_dirty(ptfile);
 			set_pte_at(mm, address, pte, ptfile);
 		}
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 4/4] mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty

[Cyrill Gorcunov]

[-- Attachment #1: mm-vma-softdirty-clean-vma-softdirty-bit --]
[-- Type: text/plain, Size: 1444 bytes --]

The clear_refs_write is called earlier than clear_soft_dirty and it is
more natural to clear VM_SOFTDIRTY (which belongs to VMA entry but not
PTEs) that early instead of clearing it a way deeper inside call chain.

CC: Pavel Emelyanov <xemul@parallels.com>
CC: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
---
 fs/proc/task_mmu.c |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

Index: linux-2.6.git/fs/proc/task_mmu.c
===================================================================
--- linux-2.6.git.orig/fs/proc/task_mmu.c
+++ linux-2.6.git/fs/proc/task_mmu.c
@@ -736,9 +736,6 @@ static inline void clear_soft_dirty(stru
 		ptent = pte_file_clear_soft_dirty(ptent);
 	}
 
-	if (vma->vm_flags & VM_SOFTDIRTY)
-		vma->vm_flags &= ~VM_SOFTDIRTY;
-
 	set_pte_at(vma->vm_mm, addr, pte, ptent);
 #endif
 }
@@ -843,6 +840,10 @@ static ssize_t clear_refs_write(struct f
 				continue;
 			if (type == CLEAR_REFS_MAPPED && !vma->vm_file)
 				continue;
+			if (type == CLEAR_REFS_SOFT_DIRTY) {
+				if (vma->vm_flags & VM_SOFTDIRTY)
+					vma->vm_flags &= ~VM_SOFTDIRTY;
+			}
 			walk_page_range(vma->vm_start, vma->vm_end,
 					&clear_refs_walk);
 		}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 4/4] mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty

[Andrew Morton]

On Mon, 24 Mar 2014 16:28:42 +0400 Cyrill Gorcunov <gorcunov@openvz.org> wrote:

> The clear_refs_write is called earlier than clear_soft_dirty and it is
> more natural to clear VM_SOFTDIRTY (which belongs to VMA entry but not
> PTEs) that early instead of clearing it a way deeper inside call chain.

This patch had some significant conflicts with the pagewalk patches:

mm-hugetlbfs-fix-rmapping-for-anonymous-hugepages-with-page_pgoff.patch
mm-hugetlbfs-fix-rmapping-for-anonymous-hugepages-with-page_pgoff-v2.patch
mm-hugetlbfs-fix-rmapping-for-anonymous-hugepages-with-page_pgoff-v3.patch
mm-hugetlbfs-fix-rmapping-for-anonymous-hugepages-with-page_pgoff-v3-fix.patch
pagewalk-update-page-table-walker-core.patch
pagewalk-update-page-table-walker-core-fix-end-address-calculation-in-walk_page_range.patch
pagewalk-update-page-table-walker-core-fix-end-address-calculation-in-walk_page_range-fix.patch
pagewalk-add-walk_page_vma.patch
smaps-redefine-callback-functions-for-page-table-walker.patch
clear_refs-redefine-callback-functions-for-page-table-walker.patch
pagemap-redefine-callback-functions-for-page-table-walker.patch
pagemap-redefine-callback-functions-for-page-table-walker-fix.patch
numa_maps-redefine-callback-functions-for-page-table-walker.patch
memcg-redefine-callback-functions-for-page-table-walker.patch
arch-powerpc-mm-subpage-protc-use-walk_page_vma-instead-of-walk_page_range.patch
pagewalk-remove-argument-hmask-from-hugetlb_entry.patch
pagewalk-remove-argument-hmask-from-hugetlb_entry-fix.patch
pagewalk-remove-argument-hmask-from-hugetlb_entry-fix-fix.patch
mempolicy-apply-page-table-walker-on-queue_pages_range.patch
mm-add-pte_present-check-on-existing-hugetlb_entry-callbacks.patch
mm-pagewalkc-move-pte-null-check.patch

I resolved this by merging
mm-softdirty-clear-vm_softdirty-flag-inside-clear_refs_write-instead-of-clear_soft_dirty.patch
on top of the pagewalk patches as below - please carefully review.

I'm hoping we'll be able to get the pagewalk patches merged in 3.16-rc1
- we'll see what happens when the testing gets underway again.


From: Cyrill Gorcunov <gorcunov@openvz.org>
Subject: mm: softdirty: clear VM_SOFTDIRTY flag inside clear_refs_write() instead of clear_soft_dirty()

clear_refs_write() is called earlier than clear_soft_dirty() and it is
more natural to clear VM_SOFTDIRTY (which belongs to VMA entry but not
PTEs) that early instead of clearing it a way deeper inside call chain.

Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/proc/task_mmu.c |   13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff -puN fs/proc/task_mmu.c~mm-softdirty-clear-vm_softdirty-flag-inside-clear_refs_write-instead-of-clear_soft_dirty fs/proc/task_mmu.c
--- a/fs/proc/task_mmu.c~mm-softdirty-clear-vm_softdirty-flag-inside-clear_refs_write-instead-of-clear_soft_dirty
+++ a/fs/proc/task_mmu.c
@@ -723,9 +723,6 @@ static inline void clear_soft_dirty(stru
 		ptent = pte_file_clear_soft_dirty(ptent);
 	}
 
-	if (vma->vm_flags & VM_SOFTDIRTY)
-		vma->vm_flags &= ~VM_SOFTDIRTY;
-
 	set_pte_at(vma->vm_mm, addr, pte, ptent);
 #endif
 }
@@ -762,11 +759,16 @@ static int clear_refs_test_walk(unsigned
 	 * Writing 1 to /proc/pid/clear_refs affects all pages.
 	 * Writing 2 to /proc/pid/clear_refs only affects anonymous pages.
 	 * Writing 3 to /proc/pid/clear_refs only affects file mapped pages.
+	 * Writing 4 to /proc/pid/clear_refs affects all pages.
 	 */
 	if (cp->type == CLEAR_REFS_ANON && vma->vm_file)
 		walk->skip = 1;
 	if (cp->type == CLEAR_REFS_MAPPED && !vma->vm_file)
 		walk->skip = 1;
+	if (cp->type == CLEAR_REFS_SOFT_DIRTY) {
+		if (vma->vm_flags & VM_SOFTDIRTY)
+			vma->vm_flags &= ~VM_SOFTDIRTY;
+	}
 	return 0;
 }
 
@@ -795,8 +797,9 @@ static ssize_t clear_refs_write(struct f
 
 	if (type == CLEAR_REFS_SOFT_DIRTY) {
 		soft_dirty_cleared = true;
-		pr_warn_once("The pagemap bits 55-60 has changed their meaning! "
-				"See the linux/Documentation/vm/pagemap.txt for details.\n");
+		pr_warn_once("The pagemap bits 55-60 has changed their meaning!"
+			     " See the linux/Documentation/vm/pagemap.txt for "
+			     "details.\n");
 	}
 
 	task = get_proc_task(file_inode(file));
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 4/4] mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty

[Cyrill Gorcunov]

On Tue, Apr 15, 2014 at 11:06:54AM -0700, Andrew Morton wrote:
> 
> I resolved this by merging
> mm-softdirty-clear-vm_softdirty-flag-inside-clear_refs_write-instead-of-clear_soft_dirty.patch
> on top of the pagewalk patches as below - please carefully review.

Thanks a lot, Andrew! I've updated the patches and were planning to send them to you
tonightm but because you applied it on top of pagewal patches, I think i rather need to
fetch -next repo and review this patch and update the rest of the series on top (hope
i'll do that in 3-4 hours).

> 
> I'm hoping we'll be able to get the pagewalk patches merged in 3.16-rc1
> - we'll see what happens when the testing gets underway again.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 4/4] mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty

[Andrew Morton]

On Tue, 15 Apr 2014 22:29:35 +0400 Cyrill Gorcunov <gorcunov@gmail.com> wrote:

> On Tue, Apr 15, 2014 at 11:06:54AM -0700, Andrew Morton wrote:
> > 
> > I resolved this by merging
> > mm-softdirty-clear-vm_softdirty-flag-inside-clear_refs_write-instead-of-clear_soft_dirty.patch
> > on top of the pagewalk patches as below - please carefully review.
> 
> Thanks a lot, Andrew! I've updated the patches and were planning to send them to you
> tonightm but because you applied it on top of pagewal patches, I think i rather need to
> fetch -next repo and review this patch and update the rest of the series on top (hope
> i'll do that in 3-4 hours).

-mm isn't in -next at present.  I'll get a release done later today for
tomorrow's -next.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 4/4] mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty

[Cyrill Gorcunov]

On Tue, Apr 15, 2014 at 11:44:49AM -0700, Andrew Morton wrote:
> On Tue, 15 Apr 2014 22:29:35 +0400 Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> 
> > On Tue, Apr 15, 2014 at 11:06:54AM -0700, Andrew Morton wrote:
> > > 
> > > I resolved this by merging
> > > mm-softdirty-clear-vm_softdirty-flag-inside-clear_refs_write-instead-of-clear_soft_dirty.patch
> > > on top of the pagewalk patches as below - please carefully review.
> > 
> > Thanks a lot, Andrew! I've updated the patches and were planning to send them to you
> > tonightm but because you applied it on top of pagewal patches, I think i rather need to
> > fetch -next repo and review this patch and update the rest of the series on top (hope
> > i'll do that in 3-4 hours).
> 
> -mm isn't in -next at present.  I'll get a release done later today for
> tomorrow's -next.

OK. Could you please remind me the place I could fetch the patchwalk series from?
(or better to wait until -mm get merged into -next?)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 4/4] mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty

[Andrew Morton]

On Tue, 15 Apr 2014 22:48:51 +0400 Cyrill Gorcunov <gorcunov@gmail.com> wrote:

> On Tue, Apr 15, 2014 at 11:44:49AM -0700, Andrew Morton wrote:
> > On Tue, 15 Apr 2014 22:29:35 +0400 Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> > 
> > > On Tue, Apr 15, 2014 at 11:06:54AM -0700, Andrew Morton wrote:
> > > > 
> > > > I resolved this by merging
> > > > mm-softdirty-clear-vm_softdirty-flag-inside-clear_refs_write-instead-of-clear_soft_dirty.patch
> > > > on top of the pagewalk patches as below - please carefully review.
> > > 
> > > Thanks a lot, Andrew! I've updated the patches and were planning to send them to you
> > > tonightm but because you applied it on top of pagewal patches, I think i rather need to
> > > fetch -next repo and review this patch and update the rest of the series on top (hope
> > > i'll do that in 3-4 hours).
> > 
> > -mm isn't in -next at present.  I'll get a release done later today for
> > tomorrow's -next.
> 
> OK. Could you please remind me the place I could fetch the patchwalk series from?

http://ozlabs.org/~akpm/mmots/

> (or better to wait until -mm get merged into -next?)

That's probably simpler.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 4/4] mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty

[Cyrill Gorcunov]

On Tue, Apr 15, 2014 at 11:52:19AM -0700, Andrew Morton wrote:
> > > 
> > > -mm isn't in -next at present.  I'll get a release done later today for
> > > tomorrow's -next.
> > 
> > OK. Could you please remind me the place I could fetch the patchwalk series from?
> 
> http://ozlabs.org/~akpm/mmots/
> 
> > (or better to wait until -mm get merged into -next?)
> 
> That's probably simpler.

I've fetched -next tree and updated patch looks good, thanks Andrew!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>