[RFC][PATCH 0/8] memcg: recharge at task move (Oct13)

[RFC][PATCH 0/8] memcg: recharge at task move (Oct13)

[Daisuke Nishimura]

Hi.

These are my current patches for recharge at task move.

In current memcg, charges associated with a task aren't moved to the new cgroup
at task move. These patches are for this feature, that is, for recharging to
the new cgroup and, of course, uncharging from old cgroup at task move.

I've tested these patches on 2.6.32-rc3(+ some patches) with memory pressure
and rmdir, they didn't cause any BUGs during last weekend.

Major Changes from Sep24:
- rebased on mmotm-2009-10-09-01-07 + KAMEZAWA-san's batched charge/uncharge(Oct09)
  + part of KAMEZAWA-san's cleanup/fix patches(4,5,7 of Sep25 with some fixes).
- changed the term "migrate" to "recharge".

TODO:
- update Documentation/cgroup/memory.txt
- implement madvise(2) (MADV_MEMCG_RECHARGE/NORECHARGE)

Any comments or suggestions would be welcome.


Thanks,
Dasiuke Nishimura.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 1/8] cgroup: introduce cancel_attach()

[Daisuke Nishimura]

This patch adds cancel_attach() operation to struct cgroup_subsys.
cancel_attach() can be used when can_attach() operation prepares something
for the subsys, but we should discard what can_attach() operation has prepared
if attach task fails afterwards.

Signed-off-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
---
 Documentation/cgroups/cgroups.txt |   12 ++++++++++++
 include/linux/cgroup.h            |    2 ++
 kernel/cgroup.c                   |   27 ++++++++++++++++++++-------
 3 files changed, 34 insertions(+), 7 deletions(-)

diff --git a/Documentation/cgroups/cgroups.txt b/Documentation/cgroups/cgroups.txt
index 0b33bfe..fd8e1c1 100644
--- a/Documentation/cgroups/cgroups.txt
+++ b/Documentation/cgroups/cgroups.txt
@@ -540,6 +540,18 @@ remain valid while the caller holds cgroup_mutex. If threadgroup is
 true, then a successful result indicates that all threads in the given
 thread's threadgroup can be moved together.
 
+void cancel_attach(struct cgroup_subsys *ss, struct cgroup *cgrp,
+	       struct task_struct *task, bool threadgroup)
+(cgroup_mutex held by caller)
+
+Called when a task attach operation has failed after can_attach() has succeeded.
+For example, this will be called if some subsystems are mounted on the same
+hierarchy, can_attach() operations have succeeded about part of the subsystems,
+but has failed about next subsystem. This will be called only about subsystems
+whose can_attach() operation has succeeded. A subsystem whose can_attach() has
+some side-effects should provide this function, so that the subsytem can
+implement a rollback. If not, not necessary.
+
 void attach(struct cgroup_subsys *ss, struct cgroup *cgrp,
 	    struct cgroup *old_cgrp, struct task_struct *task,
 	    bool threadgroup)
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 0008dee..d4cc200 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -427,6 +427,8 @@ struct cgroup_subsys {
 	void (*destroy)(struct cgroup_subsys *ss, struct cgroup *cgrp);
 	int (*can_attach)(struct cgroup_subsys *ss, struct cgroup *cgrp,
 			  struct task_struct *tsk, bool threadgroup);
+	void (*cancel_attach)(struct cgroup_subsys *ss, struct cgroup *cgrp,
+			  struct task_struct *tsk, bool threadgroup);
 	void (*attach)(struct cgroup_subsys *ss, struct cgroup *cgrp,
 			struct cgroup *old_cgrp, struct task_struct *tsk,
 			bool threadgroup);
diff --git a/kernel/cgroup.c b/kernel/cgroup.c
index 0249f4b..bc145f8 100644
--- a/kernel/cgroup.c
+++ b/kernel/cgroup.c
@@ -1539,7 +1539,7 @@ int cgroup_path(const struct cgroup *cgrp, char *buf, int buflen)
 int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
 {
 	int retval = 0;
-	struct cgroup_subsys *ss;
+	struct cgroup_subsys *ss, *fail = NULL;
 	struct cgroup *oldcgrp;
 	struct css_set *cg;
 	struct css_set *newcg;
@@ -1553,8 +1553,10 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
 	for_each_subsys(root, ss) {
 		if (ss->can_attach) {
 			retval = ss->can_attach(ss, cgrp, tsk, false);
-			if (retval)
-				return retval;
+			if (retval) {
+				fail = ss;
+				goto out;
+			}
 		}
 	}
 
@@ -1568,14 +1570,17 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
 	 */
 	newcg = find_css_set(cg, cgrp);
 	put_css_set(cg);
-	if (!newcg)
-		return -ENOMEM;
+	if (!newcg) {
+		retval = -ENOMEM;
+		goto out;
+	}
 
 	task_lock(tsk);
 	if (tsk->flags & PF_EXITING) {
 		task_unlock(tsk);
 		put_css_set(newcg);
-		return -ESRCH;
+		retval = -ESRCH;
+		goto out;
 	}
 	rcu_assign_pointer(tsk->cgroups, newcg);
 	task_unlock(tsk);
@@ -1601,7 +1606,15 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
 	 * is no longer empty.
 	 */
 	cgroup_wakeup_rmdir_waiter(cgrp);
-	return 0;
+out:
+	if (retval)
+		for_each_subsys(root, ss) {
+			if (ss == fail)
+				break;
+			if (ss->cancel_attach)
+				ss->cancel_attach(ss, cgrp, tsk, false);
+		}
+	return retval;
 }
 
 /*
-- 
1.5.6.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/8] cgroup: introduce cancel_attach()

[Paul Menage]

On Mon, Oct 12, 2009 at 9:50 PM, Daisuke Nishimura
<nishimura@mxp.nes.nec.co.jp> wrote:
>  int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
>  {
>        int retval = 0;
> -       struct cgroup_subsys *ss;
> +       struct cgroup_subsys *ss, *fail = NULL;

Maybe give this a more descriptive name, such as "failed_subsys" ?

> @@ -1553,8 +1553,10 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
>        for_each_subsys(root, ss) {
>                if (ss->can_attach) {
>                        retval = ss->can_attach(ss, cgrp, tsk, false);
> -                       if (retval)
> -                               return retval;
> +                       if (retval) {
> +                               fail = ss;

Comment here on why you set "fail" ?
> +out:
> +       if (retval)
> +               for_each_subsys(root, ss) {
> +                       if (ss == fail)

Comment here?

The problem with this API is that the subsystem doesn't know how long
it needs to hold on to the potential rollback state for. The design
for transactional cgroup attachment that I sent out over a year ago
presented a more robust API, but I've not had time to work on it. So
maybe this patch could be a stop-gap.

Paul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/8] cgroup: introduce cancel_attach()

[Paul Menage]

On Thu, Oct 15, 2009 at 12:37 AM, Paul Menage <menage@google.com> wrote:
>
> The problem with this API is that the subsystem doesn't know how long
> it needs to hold on to the potential rollback state for.

Sorry, I guess it does - either until the attach() or the
cancel_attach(). (Assuming that the rollback state doesn't involve
holding any spinlocks). It would still be nice to have more comments
in the code changes though.

Paul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/8] cgroup: introduce cancel_attach()

[Daisuke Nishimura]

Thank you for your comments.

On Thu, 15 Oct 2009 00:37:23 -0700
Paul Menage <menage@google.com> wrote:
> On Mon, Oct 12, 2009 at 9:50 PM, Daisuke Nishimura
> <nishimura@mxp.nes.nec.co.jp> wrote:
> > A int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
> > A {
> > A  A  A  A int retval = 0;
> > - A  A  A  struct cgroup_subsys *ss;
> > + A  A  A  struct cgroup_subsys *ss, *fail = NULL;
> 
> Maybe give this a more descriptive name, such as "failed_subsys" ?
> 
will fix.

> > @@ -1553,8 +1553,10 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
> > A  A  A  A for_each_subsys(root, ss) {
> > A  A  A  A  A  A  A  A if (ss->can_attach) {
> > A  A  A  A  A  A  A  A  A  A  A  A retval = ss->can_attach(ss, cgrp, tsk, false);
> > - A  A  A  A  A  A  A  A  A  A  A  if (retval)
> > - A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  return retval;
> > + A  A  A  A  A  A  A  A  A  A  A  if (retval) {
> > + A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  fail = ss;
> 
> Comment here on why you set "fail" ?
> > +out:
> > + A  A  A  if (retval)
> > + A  A  A  A  A  A  A  for_each_subsys(root, ss) {
> > + A  A  A  A  A  A  A  A  A  A  A  if (ss == fail)
> 
> Comment here?
> 
"fail" is used to call ->cancel_attach() only for subsystems
->can_attach() of which have succeeded.

> The problem with this API is that the subsystem doesn't know how long
> it needs to hold on to the potential rollback state for. The design
> for transactional cgroup attachment that I sent out over a year ago
> presented a more robust API, but I've not had time to work on it. So
> maybe this patch could be a stop-gap.
> 
I digged my mail box and read your RFC, and I agree those APIs would be
more desirable.
In 2-8 of this patch-set, I do similar things as you proposed in the RFC.
I use ->can_attach() as prepare_attach_sleep(), ->attach() as commit_attach(),
and ->cancel_attach() as abort_attach_sleep(), so I think it wouldn't be so
difficult to adapt my interfaces to new API when it comes.
The reason why I introduced ->cancel_attach() is I need to cancel(do rollback)
the charge reservation against @to cgroup, which I've reserved in ->can_attach(),
and this patch is enough for my purpose for now.

Anyway, I'll add more comments.


Thanks,
Daisuke Nishimura.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 2/8] memcg: cleanup mem_cgroup_move_parent()

[Daisuke Nishimura]

mem_cgroup_move_parent() calls try_charge first and cancel_charge on failure.
IMHO, charge/uncharge(especially charge) is high cost operation, so we should
avoid it as far as possible.

This patch tries to delay try_charge in mem_cgroup_move_parent() by re-ordering
checks it does.

And this patch renames mem_cgroup_move_account() to __mem_cgroup_move_account(),
changes the return value of __mem_cgroup_move_account() from int to void,
and adds a new wrapper(mem_cgroup_move_account()), which checks whether a @pc
is valid for moving account and calls __mem_cgroup_move_account().

This patch removes the last caller of trylock_page_cgroup(), so removes its
definition too.

Signed-off-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
---
 include/linux/page_cgroup.h |    7 +---
 mm/memcontrol.c             |   84 ++++++++++++++++++-------------------------
 2 files changed, 37 insertions(+), 54 deletions(-)

diff --git a/include/linux/page_cgroup.h b/include/linux/page_cgroup.h
index 4b938d4..b0e4eb1 100644
--- a/include/linux/page_cgroup.h
+++ b/include/linux/page_cgroup.h
@@ -57,6 +57,8 @@ static inline void ClearPageCgroup##uname(struct page_cgroup *pc)	\
 static inline int TestClearPageCgroup##uname(struct page_cgroup *pc)	\
 	{ return test_and_clear_bit(PCG_##lname, &pc->flags);  }
 
+TESTPCGFLAG(Locked, LOCK)
+
 /* Cache flag is set only once (at allocation) */
 TESTPCGFLAG(Cache, CACHE)
 CLEARPCGFLAG(Cache, CACHE)
@@ -86,11 +88,6 @@ static inline void lock_page_cgroup(struct page_cgroup *pc)
 	bit_spin_lock(PCG_LOCK, &pc->flags);
 }
 
-static inline int trylock_page_cgroup(struct page_cgroup *pc)
-{
-	return bit_spin_trylock(PCG_LOCK, &pc->flags);
-}
-
 static inline void unlock_page_cgroup(struct page_cgroup *pc)
 {
 	bit_spin_unlock(PCG_LOCK, &pc->flags);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index f604469..fb20564 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1646,45 +1646,29 @@ static void __mem_cgroup_commit_charge(struct mem_cgroup *mem,
 }
 
 /**
- * mem_cgroup_move_account - move account of the page
+ * __mem_cgroup_move_account - move account of the page
  * @pc:	page_cgroup of the page.
  * @from: mem_cgroup which the page is moved from.
  * @to:	mem_cgroup which the page is moved to. @from != @to.
  *
  * The caller must confirm following.
  * - page is not on LRU (isolate_page() is useful.)
- *
- * returns 0 at success,
- * returns -EBUSY when lock is busy or "pc" is unstable.
+ * - the pc is locked, used, and ->mem_cgroup points to @from.
  *
  * This function does "uncharge" from old cgroup but doesn't do "charge" to
  * new cgroup. It should be done by a caller.
  */
 
-static int mem_cgroup_move_account(struct page_cgroup *pc,
+static void __mem_cgroup_move_account(struct page_cgroup *pc,
 	struct mem_cgroup *from, struct mem_cgroup *to)
 {
-	struct mem_cgroup_per_zone *from_mz, *to_mz;
-	int nid, zid;
-	int ret = -EBUSY;
 	struct page *page;
 
 	VM_BUG_ON(from == to);
 	VM_BUG_ON(PageLRU(pc->page));
-
-	nid = page_cgroup_nid(pc);
-	zid = page_cgroup_zid(pc);
-	from_mz =  mem_cgroup_zoneinfo(from, nid, zid);
-	to_mz =  mem_cgroup_zoneinfo(to, nid, zid);
-
-	if (!trylock_page_cgroup(pc))
-		return ret;
-
-	if (!PageCgroupUsed(pc))
-		goto out;
-
-	if (pc->mem_cgroup != from)
-		goto out;
+	VM_BUG_ON(!PageCgroupLocked(pc));
+	VM_BUG_ON(!PageCgroupUsed(pc));
+	VM_BUG_ON(pc->mem_cgroup != from);
 
 	if (!mem_cgroup_is_root(from))
 		res_counter_uncharge(&from->res, PAGE_SIZE);
@@ -1705,15 +1689,28 @@ static int mem_cgroup_move_account(struct page_cgroup *pc,
 	css_get(&to->css);
 	pc->mem_cgroup = to;
 	mem_cgroup_charge_statistics(to, pc, true);
-	ret = 0;
-out:
-	unlock_page_cgroup(pc);
 	/*
 	 * We charges against "to" which may not have any tasks. Then, "to"
 	 * can be under rmdir(). But in current implementation, caller of
 	 * this function is just force_empty() and it's garanteed that
 	 * "to" is never removed. So, we don't check rmdir status here.
 	 */
+}
+
+/*
+ * check whether the @pc is valid for moving account and call
+ * __mem_cgroup_move_account()
+ */
+static int mem_cgroup_move_account(struct page_cgroup *pc,
+				struct mem_cgroup *from, struct mem_cgroup *to)
+{
+	int ret = -EINVAL;
+	lock_page_cgroup(pc);
+	if (PageCgroupUsed(pc) && pc->mem_cgroup == from) {
+		__mem_cgroup_move_account(pc, from, to);
+		ret = 0;
+	}
+	unlock_page_cgroup(pc);
 	return ret;
 }
 
@@ -1735,38 +1732,27 @@ static int mem_cgroup_move_parent(struct page_cgroup *pc,
 	if (!pcg)
 		return -EINVAL;
 
+	ret = -EBUSY;
+	if (!get_page_unless_zero(page))
+		goto out;
+	if (isolate_lru_page(page))
+		goto put;
 
 	parent = mem_cgroup_from_cgroup(pcg);
-
-
 	ret = __mem_cgroup_try_charge(NULL, gfp_mask, &parent, false, page);
 	if (ret || !parent)
-		return ret;
-
-	if (!get_page_unless_zero(page)) {
-		ret = -EBUSY;
-		goto uncharge;
-	}
-
-	ret = isolate_lru_page(page);
-
-	if (ret)
-		goto cancel;
+		goto put_back;
 
 	ret = mem_cgroup_move_account(pc, child, parent);
-
+	if (!ret)
+		css_put(&parent->css);	/* drop extra refcnt by try_charge() */
+	else
+		mem_cgroup_cancel_charge(parent);	/* does css_put */
+put_back:
 	putback_lru_page(page);
-	if (!ret) {
-		put_page(page);
-		/* drop extra refcnt by try_charge() */
-		css_put(&parent->css);
-		return 0;
-	}
-
-cancel:
+put:
 	put_page(page);
-uncharge:
-	mem_cgroup_cancel_charge(parent);
+out:
 	return ret;
 }
 
-- 
1.5.6.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 3/8] memcg: move memcg_tasklist mutex

[Daisuke Nishimura]

memcg_tasklist was introduced to serialize mem_cgroup_out_of_memory() and
mem_cgroup_move_task() to ensure tasks cannot be moved to another cgroup
during select_bad_process().

task_in_mem_cgroup(), which can be called by select_bad_process(), will check
whether a task is in the mem_cgroup or not by dereferencing task->cgroups
->subsys[]. So, it would be desirable to change task->cgroups
(rcu_assign_pointer() in cgroup_attach_task() does it) with memcg_tasklist held.

Now that we can define cancel_attach(), we can safely release memcg_tasklist
on fail path even if we hold memcg_tasklist in can_attach(). So let's move
mutex_lock/unlock() of memcg_tasklist.

Signed-off-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
---
 mm/memcontrol.c |   22 ++++++++++++++++++++--
 1 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index fb20564..7084cb1 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3395,18 +3395,34 @@ static int mem_cgroup_populate(struct cgroup_subsys *ss,
 	return ret;
 }
 
+static int mem_cgroup_can_attach(struct cgroup_subsys *ss,
+				struct cgroup *cgroup,
+				struct task_struct *p,
+				bool threadgroup)
+{
+	mutex_lock(&memcg_tasklist);
+	return 0;
+}
+
+static void mem_cgroup_cancel_attach(struct cgroup_subsys *ss,
+				struct cgroup *cgroup,
+				struct task_struct *p,
+				bool threadgroup)
+{
+	mutex_unlock(&memcg_tasklist);
+}
+
 static void mem_cgroup_move_task(struct cgroup_subsys *ss,
 				struct cgroup *cgroup,
 				struct cgroup *old_cgroup,
 				struct task_struct *p,
 				bool threadgroup)
 {
-	mutex_lock(&memcg_tasklist);
+	mutex_unlock(&memcg_tasklist);
 	/*
 	 * FIXME: It's better to move charges of this process from old
 	 * memcg to new memcg. But it's just on TODO-List now.
 	 */
-	mutex_unlock(&memcg_tasklist);
 }
 
 struct cgroup_subsys mem_cgroup_subsys = {
@@ -3416,6 +3432,8 @@ struct cgroup_subsys mem_cgroup_subsys = {
 	.pre_destroy = mem_cgroup_pre_destroy,
 	.destroy = mem_cgroup_destroy,
 	.populate = mem_cgroup_populate,
+	.can_attach = mem_cgroup_can_attach,
+	.cancel_attach = mem_cgroup_cancel_attach,
 	.attach = mem_cgroup_move_task,
 	.early_init = 0,
 	.use_id = 1,
-- 
1.5.6.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 4/8] memcg: add interface to recharge at task move

[Daisuke Nishimura]

In current memcg, charges associated with a task aren't moved to the new cgroup
at task move. These patches are for this feature, that is, for recharging to
the new cgroup and, of course, uncharging from old cgroup at task move.

This patch adds "memory.recharge_at_immigrate" file, which is a flag file to
determine whether charges should be moved to the new cgroup at task move or
not, and read/write handlers of the file.
This patch also adds no-op handlers for this feature. These handlers will be
implemented in later patche.

Signed-off-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
---
 mm/memcontrol.c |   65 +++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 files changed, 59 insertions(+), 6 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 7084cb1..66206cc 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -185,6 +185,12 @@ struct mem_cgroup {
 	bool		memsw_is_minimum;
 
 	/*
+	 * Should we recharge charges of a task when a task is moved into this
+	 * mem_cgroup ?
+	 */
+	bool	 	recharge_at_immigrate;
+
+	/*
 	 * statistics. This must be placed at the end of memcg.
 	 */
 	struct mem_cgroup_stat stat;
@@ -2885,6 +2891,30 @@ static int mem_cgroup_reset(struct cgroup *cgroup, unsigned int event)
 	return 0;
 }
 
+static u64 mem_cgroup_recharge_read(struct cgroup *cgrp,
+					struct cftype *cft)
+{
+	return mem_cgroup_from_cgroup(cgrp)->recharge_at_immigrate;
+}
+
+static int mem_cgroup_recharge_write(struct cgroup *cgrp,
+					struct cftype *cft, u64 val)
+{
+	struct mem_cgroup *mem = mem_cgroup_from_cgroup(cgrp);
+
+	if (val != 0 && val != 1)
+		return -EINVAL;
+	/*
+	 * We check this value both in can_attach() and attach(), so we need
+	 * cgroup lock to prevent this value from being inconsistent.
+	 */
+	cgroup_lock();
+	mem->recharge_at_immigrate = val;
+	cgroup_unlock();
+
+	return 0;
+}
+
 
 /* For read statistics */
 enum {
@@ -3118,6 +3148,11 @@ static struct cftype mem_cgroup_files[] = {
 		.read_u64 = mem_cgroup_swappiness_read,
 		.write_u64 = mem_cgroup_swappiness_write,
 	},
+	{
+		.name = "recharge_at_immigrate",
+		.read_u64 = mem_cgroup_recharge_read,
+		.write_u64 = mem_cgroup_recharge_write,
+	},
 };
 
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR_SWAP
@@ -3359,6 +3394,7 @@ mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cgroup)
 	if (parent)
 		mem->swappiness = get_swappiness(parent);
 	atomic_set(&mem->refcnt, 1);
+	mem->recharge_at_immigrate = 0;
 	return &mem->css;
 free_out:
 	__mem_cgroup_free(mem);
@@ -3395,13 +3431,26 @@ static int mem_cgroup_populate(struct cgroup_subsys *ss,
 	return ret;
 }
 
+/* Handlers for recharge at task move. */
+static int mem_cgroup_can_recharge(struct mem_cgroup *mem,
+					struct task_struct *p)
+{
+	return 0;
+}
+
 static int mem_cgroup_can_attach(struct cgroup_subsys *ss,
 				struct cgroup *cgroup,
 				struct task_struct *p,
 				bool threadgroup)
 {
-	mutex_lock(&memcg_tasklist);
-	return 0;
+	int ret = 0;
+	struct mem_cgroup *mem = mem_cgroup_from_cgroup(cgroup);
+
+	if (mem->recharge_at_immigrate && thread_group_leader(p))
+		ret = mem_cgroup_can_recharge(mem, p);
+	if (!ret)
+		mutex_lock(&memcg_tasklist);
+	return ret;
 }
 
 static void mem_cgroup_cancel_attach(struct cgroup_subsys *ss,
@@ -3412,17 +3461,21 @@ static void mem_cgroup_cancel_attach(struct cgroup_subsys *ss,
 	mutex_unlock(&memcg_tasklist);
 }
 
+static void mem_cgroup_recharge(void)
+{
+}
+
 static void mem_cgroup_move_task(struct cgroup_subsys *ss,
 				struct cgroup *cgroup,
 				struct cgroup *old_cgroup,
 				struct task_struct *p,
 				bool threadgroup)
 {
+	struct mem_cgroup *mem = mem_cgroup_from_cgroup(cgroup);
+
 	mutex_unlock(&memcg_tasklist);
-	/*
-	 * FIXME: It's better to move charges of this process from old
-	 * memcg to new memcg. But it's just on TODO-List now.
-	 */
+	if (mem->recharge_at_immigrate && thread_group_leader(p))
+		mem_cgroup_recharge();
 }
 
 struct cgroup_subsys mem_cgroup_subsys = {
-- 
1.5.6.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 5/8] memcg: recharge charges of mapped page

[Daisuke Nishimura]

This patch is the core part of this recharge-at-task-move feature.
It implements functions to recharge charges of pages(including anonymous, shmem,
and file caches) mapped by the task.

Implementation:
- define struct recharge_struct and a valuable of it(recharge) to remember
  the count of pre-charges and other information.
- At can_attach(), parse the page table of the task and count the number of
  mapped pages which are charged to the source mem_cgroup, and call
  __mem_cgroup_try_charge() repeatedly and count up recharge.precharge.
- At attach(), parse the page table again, find a target page as we did in
  can_attach(), and call mem_cgroup_move_account() about the page.
- Cancel all charges if recharge.precharge > 0 on failure or at the end of
  task move.

Signed-off-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
---
 mm/memcontrol.c |  266 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 263 insertions(+), 3 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 66206cc..85fee0c 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -21,6 +21,8 @@
 #include <linux/memcontrol.h>
 #include <linux/cgroup.h>
 #include <linux/mm.h>
+#include <linux/migrate.h>
+#include <linux/hugetlb.h>
 #include <linux/pagemap.h>
 #include <linux/smp.h>
 #include <linux/page-flags.h>
@@ -197,6 +199,18 @@ struct mem_cgroup {
 };
 
 /*
+ * A data structure and a valiable for recharging charges at task move.
+ * "recharge" and its members are protected by cgroup_lock
+ */
+struct recharge_struct {
+	struct mem_cgroup *from;
+	struct mem_cgroup *to;
+	struct task_struct *target;	/* the target task being moved */
+	unsigned long precharge;
+};
+static struct recharge_struct recharge;
+
+/*
  * Maximum loops in mem_cgroup_hierarchical_reclaim(), used for soft
  * limit reclaim to prevent infinite loops, if they ever occur.
  */
@@ -1529,7 +1543,7 @@ charged:
 	 * Insert ancestor (and ancestor's ancestors), to softlimit RB-tree.
 	 * if they exceeds softlimit.
 	 */
-	if (mem_cgroup_soft_limit_check(mem))
+	if (page && mem_cgroup_soft_limit_check(mem))
 		mem_cgroup_update_tree(mem, page);
 done:
 	return 0;
@@ -3432,10 +3446,161 @@ static int mem_cgroup_populate(struct cgroup_subsys *ss,
 }
 
 /* Handlers for recharge at task move. */
+/**
+ * is_target_pte_for_recharge - check a pte whether it is target for recharge
+ * @vma: the vma the pte to be checked belongs
+ * @addr: the address corresponding to the pte to be checked
+ * @ptent: the pte to be checked
+ * @target: the pointer the target page will be stored(can be NULL)
+ *
+ * Returns
+ *   0(RECHARGE_TARGET_NONE): if the pte is not a target for charge recharge.
+ *   1(RECHARGE_TARGET_PAGE): if the page corresponding to this pte is a target
+ *     for recharge. if @target is not NULL, the page is stored in target->page
+ *     with extra refcnt got(Callers should handle it).
+ *
+ * Called with pte lock held.
+ */
+/* We add a new member later. */
+union recharge_target {
+	struct page	*page;
+};
+
+/* We add a new type later. */
+enum recharge_target_type {
+	RECHARGE_TARGET_NONE,	/* not used */
+	RECHARGE_TARGET_PAGE,
+};
+
+static int is_target_pte_for_recharge(struct vm_area_struct *vma,
+		unsigned long addr, pte_t ptent, union recharge_target *target)
+{
+	struct page *page;
+	struct page_cgroup *pc;
+	int ret = 0;
+
+	if (!pte_present(ptent))
+		return 0;
+
+	page = vm_normal_page(vma, addr, ptent);
+	if (!page || !page_mapped(page))
+		return 0;
+	if (!get_page_unless_zero(page))
+		return 0;
+
+	pc = lookup_page_cgroup(page);
+	lock_page_cgroup(pc);
+	if (PageCgroupUsed(pc) && pc->mem_cgroup == recharge.from) {
+		ret = RECHARGE_TARGET_PAGE;
+		if (target)
+			target->page = page;
+	}
+	unlock_page_cgroup(pc);
+
+	if (!ret || !target)
+		put_page(page);
+
+	return ret;
+}
+
+static int mem_cgroup_recharge_do_precharge(void)
+{
+	int ret = -ENOMEM;
+	struct mem_cgroup *mem = recharge.to;
+
+	ret = __mem_cgroup_try_charge(NULL, GFP_KERNEL, &mem, false, NULL);
+	if (ret || !mem)
+		return -ENOMEM;
+
+	recharge.precharge++;
+	return ret;
+}
+
+static int mem_cgroup_recharge_prepare_pte_range(pmd_t *pmd,
+					unsigned long addr, unsigned long end,
+					struct mm_walk *walk)
+{
+	int ret = 0;
+	unsigned long count = 0;
+	struct vm_area_struct *vma = walk->private;
+	pte_t *pte;
+	spinlock_t *ptl;
+
+	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
+	for (; addr != end; pte++, addr += PAGE_SIZE)
+		if (is_target_pte_for_recharge(vma, addr, *pte, NULL))
+			count++;
+	pte_unmap_unlock(pte - 1, ptl);
+
+	while (count-- && !ret)
+		ret = mem_cgroup_recharge_do_precharge();
+
+	return ret;
+}
+
+static int mem_cgroup_recharge_prepare(void)
+{
+	int ret = 0;
+	struct mm_struct *mm;
+	struct vm_area_struct *vma;
+
+	mm = get_task_mm(recharge.target);
+	if (!mm)
+		return 0;
+
+	down_read(&mm->mmap_sem);
+	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+		struct mm_walk mem_cgroup_recharge_prepare_walk = {
+			.pmd_entry = mem_cgroup_recharge_prepare_pte_range,
+			.mm = mm,
+			.private = vma,
+		};
+		if (signal_pending(current)) {
+			ret = -EINTR;
+			break;
+		}
+		if (is_vm_hugetlb_page(vma))
+			continue;
+		ret = walk_page_range(vma->vm_start, vma->vm_end,
+					&mem_cgroup_recharge_prepare_walk);
+		if (ret)
+			break;
+		cond_resched();
+	}
+	up_read(&mm->mmap_sem);
+
+	mmput(mm);
+	return ret;
+}
+
+static void mem_cgroup_clear_recharge(void)
+{
+	while (recharge.precharge--)
+		mem_cgroup_cancel_charge(recharge.to);
+	recharge.from = NULL;
+	recharge.to = NULL;
+	recharge.target = NULL;
+}
+
 static int mem_cgroup_can_recharge(struct mem_cgroup *mem,
 					struct task_struct *p)
 {
-	return 0;
+	int ret;
+	struct mem_cgroup *from = mem_cgroup_from_task(p);
+
+	if (from == mem)
+		return 0;
+
+	recharge.from = from;
+	recharge.to = mem;
+	recharge.target = p;
+	recharge.precharge = 0;
+
+	ret = mem_cgroup_recharge_prepare();
+
+	if (ret)
+		mem_cgroup_clear_recharge();
+	return ret;
 }
 
 static int mem_cgroup_can_attach(struct cgroup_subsys *ss,
@@ -3458,11 +3623,104 @@ static void mem_cgroup_cancel_attach(struct cgroup_subsys *ss,
 				struct task_struct *p,
 				bool threadgroup)
 {
+	struct mem_cgroup *mem = mem_cgroup_from_cgroup(cgroup);
+
 	mutex_unlock(&memcg_tasklist);
+	if (mem->recharge_at_immigrate && thread_group_leader(p))
+		mem_cgroup_clear_recharge();
+}
+
+static int mem_cgroup_recharge_pte_range(pmd_t *pmd,
+				unsigned long addr, unsigned long end,
+				struct mm_walk *walk)
+{
+	int ret = 0;
+	struct vm_area_struct *vma = walk->private;
+	pte_t *pte;
+	spinlock_t *ptl;
+
+retry:
+	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
+	for (; addr != end; addr += PAGE_SIZE) {
+		pte_t ptent = *(pte++);
+		union recharge_target target;
+		int type;
+		struct page *page;
+		struct page_cgroup *pc;
+
+		if (!recharge.precharge)
+			break;
+
+		type = is_target_pte_for_recharge(vma, addr, ptent, &target);
+		switch (type) {
+		case RECHARGE_TARGET_PAGE:
+			page = target.page;
+			if (isolate_lru_page(page))
+				goto put;
+			pc = lookup_page_cgroup(page);
+			if (!mem_cgroup_move_account(pc,
+						recharge.from, recharge.to)) {
+				css_put(&recharge.to->css);
+				recharge.precharge--;
+			}
+			putback_lru_page(page);
+put:			/* is_target_pte_for_recharge() gets the page */
+			put_page(page);
+			break;
+		default:
+			continue;
+		}
+	}
+	pte_unmap_unlock(pte - 1, ptl);
+
+	if (addr != end) {
+		/*
+		 * We have consumed all precharges we got in can_attach().
+		 * We try precharge one by one, but don't do any additional
+		 * precharges nor recharges to recharge.to if we have failed in
+		 * precharge once in attach() phase.
+		 */
+		ret = mem_cgroup_recharge_do_precharge();
+		if (!ret)
+			goto retry;
+	}
+
+	return ret;
 }
 
 static void mem_cgroup_recharge(void)
 {
+	struct mm_struct *mm;
+	struct vm_area_struct *vma;
+
+	mm = get_task_mm(recharge.target);
+	if (!mm)
+		return;
+
+	lru_add_drain_all();
+	down_read(&mm->mmap_sem);
+	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+		int ret;
+		struct mm_walk mem_cgroup_recharge_walk = {
+			.pmd_entry = mem_cgroup_recharge_pte_range,
+			.mm = mm,
+			.private = vma,
+		};
+		if (is_vm_hugetlb_page(vma))
+			continue;
+		ret = walk_page_range(vma->vm_start, vma->vm_end,
+						&mem_cgroup_recharge_walk);
+		if (ret)
+			/*
+			 * means we have consumed all precharges and failed in
+			 * doing additional precharge. Just abandon here.
+			 */
+			break;
+		cond_resched();
+	}
+	up_read(&mm->mmap_sem);
+
+	mmput(mm);
 }
 
 static void mem_cgroup_move_task(struct cgroup_subsys *ss,
@@ -3474,8 +3732,10 @@ static void mem_cgroup_move_task(struct cgroup_subsys *ss,
 	struct mem_cgroup *mem = mem_cgroup_from_cgroup(cgroup);
 
 	mutex_unlock(&memcg_tasklist);
-	if (mem->recharge_at_immigrate && thread_group_leader(p))
+	if (mem->recharge_at_immigrate && thread_group_leader(p)) {
 		mem_cgroup_recharge();
+		mem_cgroup_clear_recharge();
+	}
 }
 
 struct cgroup_subsys mem_cgroup_subsys = {
-- 
1.5.6.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 6/8] memcg: avoid oom during recharge at task move

[Daisuke Nishimura]

This recharge-at-task-move feature has extra charges(pre-charges) on "to"
mem_cgroup during recharging. This means unnecessary oom can happen.

This patch tries to avoid such oom.

Signed-off-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
---
 mm/memcontrol.c |   27 +++++++++++++++++++++++++++
 1 files changed, 27 insertions(+), 0 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 85fee0c..88b3fc2 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -206,6 +206,7 @@ struct recharge_struct {
 	struct mem_cgroup *from;
 	struct mem_cgroup *to;
 	struct task_struct *target;	/* the target task being moved */
+	struct task_struct *working;	/* a task moving the target task */
 	unsigned long precharge;
 };
 static struct recharge_struct recharge;
@@ -1526,6 +1527,30 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 		if (mem_cgroup_check_under_limit(mem_over_limit))
 			continue;
 
+		/* try to avoid oom while someone is recharging */
+		if (recharge.working && current != recharge.working) {
+			struct mem_cgroup *dest;
+			bool do_continue = false;
+			/*
+			 * There is a small race that "dest" can be freed by
+			 * rmdir, so we use css_tryget().
+			 */
+			rcu_read_lock();
+			dest = recharge.to;
+			if (dest && css_tryget(&dest->css)) {
+				if (dest->use_hierarchy)
+					do_continue = css_is_ancestor(
+							&dest->css,
+							&mem_over_limit->css);
+				else
+					do_continue = (dest == mem_over_limit);
+				css_put(&dest->css);
+			}
+			rcu_read_unlock();
+			if (do_continue)
+				continue;
+		}
+
 		if (!nr_retries--) {
 			if (oom) {
 				mutex_lock(&memcg_tasklist);
@@ -3580,6 +3605,7 @@ static void mem_cgroup_clear_recharge(void)
 	recharge.from = NULL;
 	recharge.to = NULL;
 	recharge.target = NULL;
+	recharge.working = NULL;
 }
 
 static int mem_cgroup_can_recharge(struct mem_cgroup *mem,
@@ -3594,6 +3620,7 @@ static int mem_cgroup_can_recharge(struct mem_cgroup *mem,
 	recharge.from = from;
 	recharge.to = mem;
 	recharge.target = p;
+	recharge.working = current;
 	recharge.precharge = 0;
 
 	ret = mem_cgroup_recharge_prepare();
-- 
1.5.6.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 7/8] memcg: recharge charges of anonymous swap

[Daisuke Nishimura]

This patch is another core part of this recharge-at-task-move feature.
It enables recharge of anonymous swaps.

To move the charge of swap, we need to exchange swap_cgroup's record.

In current implementation, swap_cgroup's record is protected by:

  - page lock: if the entry is on swap cache.
  - swap_lock: if the entry is not on swap cache.

This works well in usual swap-in/out activity.

But this behavior make charge migration of swap check many conditions to
exchange swap_cgroup's record safely.

So I changed modification of swap_cgroup's recored(swap_cgroup_record())
to use xchg, and define a new function to cmpxchg swap_cgroup's record.

This patch also enables recharge of non pte_present but not uncharged swap
caches by getting the target pages via find_get_page() as do_mincore() does.

Signed-off-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
---
 include/linux/page_cgroup.h |    2 +
 mm/memcontrol.c             |  104 +++++++++++++++++++++++++++++++++++++-----
 mm/page_cgroup.c            |   35 ++++++++++++++-
 3 files changed, 126 insertions(+), 15 deletions(-)

diff --git a/include/linux/page_cgroup.h b/include/linux/page_cgroup.h
index b0e4eb1..30b0813 100644
--- a/include/linux/page_cgroup.h
+++ b/include/linux/page_cgroup.h
@@ -118,6 +118,8 @@ static inline void __init page_cgroup_init_flatmem(void)
 #include <linux/swap.h>
 
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR_SWAP
+extern unsigned short swap_cgroup_cmpxchg(swp_entry_t ent,
+					unsigned short old, unsigned short new);
 extern unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id);
 extern unsigned short lookup_swap_cgroup(swp_entry_t ent);
 extern int swap_cgroup_swapon(int type, unsigned long max_pages);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 88b3fc2..7e82448 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -34,6 +34,7 @@
 #include <linux/rbtree.h>
 #include <linux/slab.h>
 #include <linux/swap.h>
+#include <linux/swapops.h>
 #include <linux/spinlock.h>
 #include <linux/fs.h>
 #include <linux/seq_file.h>
@@ -2253,6 +2254,49 @@ void mem_cgroup_uncharge_swap(swp_entry_t ent)
 	}
 	rcu_read_unlock();
 }
+
+/**
+ * mem_cgroup_move_swap_account - move swap charge and swap_cgroup's record.
+ * @entry: swap entry to be moved
+ * @from:  mem_cgroup which the entry is moved from
+ * @to:  mem_cgroup which the entry is moved to
+ *
+ * It successes only when the swap_cgroup's record for this entry is the same
+ * as the mem_cgroup's id of @from.
+ *
+ * Returns 0 on success, 1 on failure.
+ *
+ * The caller must have called __mem_cgroup_try_charge on @to.
+ */
+static int mem_cgroup_move_swap_account(swp_entry_t entry,
+				struct mem_cgroup *from, struct mem_cgroup *to)
+{
+	unsigned short old_id, new_id;
+
+	old_id = css_id(&from->css);
+	new_id = css_id(&to->css);
+
+	if (swap_cgroup_cmpxchg(entry, old_id, new_id) == old_id) {
+		if (!mem_cgroup_is_root(from))
+			res_counter_uncharge(&from->memsw, PAGE_SIZE);
+		mem_cgroup_swap_statistics(from, false);
+		mem_cgroup_put(from);
+
+		if (!mem_cgroup_is_root(to))
+			res_counter_uncharge(&to->res, PAGE_SIZE);
+		mem_cgroup_swap_statistics(to, true);
+		mem_cgroup_get(to);
+
+		return 0;
+	}
+	return 1;
+}
+#else
+static inline int mem_cgroup_move_swap_account(swp_entry_t entry,
+				struct mem_cgroup *from, struct mem_cgroup *to)
+{
+	return 1;
+}
 #endif
 
 /*
@@ -3476,43 +3520,60 @@ static int mem_cgroup_populate(struct cgroup_subsys *ss,
  * @vma: the vma the pte to be checked belongs
  * @addr: the address corresponding to the pte to be checked
  * @ptent: the pte to be checked
- * @target: the pointer the target page will be stored(can be NULL)
+ * @target: the pointer the target page or entry will be stored(can be NULL)
  *
  * Returns
  *   0(RECHARGE_TARGET_NONE): if the pte is not a target for charge recharge.
  *   1(RECHARGE_TARGET_PAGE): if the page corresponding to this pte is a target
  *     for recharge. if @target is not NULL, the page is stored in target->page
  *     with extra refcnt got(Callers should handle it).
+ *   2(MIGRATION_TARGET_SWAP): if the swap entry corresponding to this pte is a
+ *     target for charge migration. if @target is not NULL, the entry is stored
+ *     in target->ent.
  *
  * Called with pte lock held.
  */
-/* We add a new member later. */
 union recharge_target {
 	struct page	*page;
+	swp_entry_t	ent;
 };
 
-/* We add a new type later. */
 enum recharge_target_type {
 	RECHARGE_TARGET_NONE,	/* not used */
 	RECHARGE_TARGET_PAGE,
+	RECHARGE_TARGET_SWAP,
 };
 
 static int is_target_pte_for_recharge(struct vm_area_struct *vma,
 		unsigned long addr, pte_t ptent, union recharge_target *target)
 {
-	struct page *page;
+	struct page *page = NULL;
 	struct page_cgroup *pc;
+	swp_entry_t ent = { .val = 0 };
 	int ret = 0;
 
-	if (!pte_present(ptent))
-		return 0;
-
-	page = vm_normal_page(vma, addr, ptent);
-	if (!page || !page_mapped(page))
-		return 0;
-	if (!get_page_unless_zero(page))
-		return 0;
-
+	if (!pte_present(ptent)) {
+		/* TODO: handle swap of shmes/tmpfs */
+		if (pte_none(ptent) || pte_file(ptent))
+			return 0;
+		else if (is_swap_pte(ptent)) {
+			ent = pte_to_swp_entry(ptent);
+			if (is_migration_entry(ent))
+				return 0;
+			page = find_get_page(&swapper_space, ent.val);
+		}
+		if (page)
+			goto check_page;
+		else
+			goto check_swap;
+	} else {
+		page = vm_normal_page(vma, addr, ptent);
+		if (!page || !page_mapped(page))
+			return 0;
+		if (!get_page_unless_zero(page))
+			return 0;
+	}
+check_page:
 	pc = lookup_page_cgroup(page);
 	lock_page_cgroup(pc);
 	if (PageCgroupUsed(pc) && pc->mem_cgroup == recharge.from) {
@@ -3524,6 +3585,14 @@ static int is_target_pte_for_recharge(struct vm_area_struct *vma,
 
 	if (!ret || !target)
 		put_page(page);
+	/* fall throught */
+check_swap:
+	if (ent.val && do_swap_account && !ret &&
+		css_id(&recharge.from->css) == lookup_swap_cgroup(ent)) {
+		ret = RECHARGE_TARGET_SWAP;
+		if (target)
+			target->ent = ent;
+	}
 
 	return ret;
 }
@@ -3674,6 +3743,7 @@ retry:
 		int type;
 		struct page *page;
 		struct page_cgroup *pc;
+		swp_entry_t ent;
 
 		if (!recharge.precharge)
 			break;
@@ -3694,6 +3764,14 @@ retry:
 put:			/* is_target_pte_for_recharge() gets the page */
 			put_page(page);
 			break;
+		case RECHARGE_TARGET_SWAP:
+			ent = target.ent;
+			if (!mem_cgroup_move_swap_account(ent,
+						recharge.from, recharge.to)) {
+				css_put(&recharge.to->css);
+				recharge.precharge--;
+			}
+			break;
 		default:
 			continue;
 		}
diff --git a/mm/page_cgroup.c b/mm/page_cgroup.c
index 3d535d5..213b0ee 100644
--- a/mm/page_cgroup.c
+++ b/mm/page_cgroup.c
@@ -9,6 +9,7 @@
 #include <linux/vmalloc.h>
 #include <linux/cgroup.h>
 #include <linux/swapops.h>
+#include <asm/cmpxchg.h>
 
 static void __meminit
 __init_page_cgroup(struct page_cgroup *pc, unsigned long pfn)
@@ -335,6 +336,37 @@ not_enough_page:
 }
 
 /**
+ * swap_cgroup_cmpxchg - cmpxchg mem_cgroup's id for this swp_entry.
+ * @end: swap entry to be cmpxchged
+ * @old: old id
+ * @new: new id
+ *
+ * Returns old id at success, 0 at failure.
+ * (There is no mem_cgroup useing 0 as its id)
+ */
+unsigned short swap_cgroup_cmpxchg(swp_entry_t ent,
+					unsigned short old, unsigned short new)
+{
+	int type = swp_type(ent);
+	unsigned long offset = swp_offset(ent);
+	unsigned long idx = offset / SC_PER_PAGE;
+	unsigned long pos = offset & SC_POS_MASK;
+	struct swap_cgroup_ctrl *ctrl;
+	struct page *mappage;
+	struct swap_cgroup *sc;
+
+	ctrl = &swap_cgroup_ctrl[type];
+
+	mappage = ctrl->map[idx];
+	sc = page_address(mappage);
+	sc += pos;
+	if (cmpxchg(&sc->id, old, new) == old)
+		return old;
+	else
+		return 0;
+}
+
+/**
  * swap_cgroup_record - record mem_cgroup for this swp_entry.
  * @ent: swap entry to be recorded into
  * @mem: mem_cgroup to be recorded
@@ -358,8 +390,7 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id)
 	mappage = ctrl->map[idx];
 	sc = page_address(mappage);
 	sc += pos;
-	old = sc->id;
-	sc->id = id;
+	old = xchg(&sc->id, id);
 
 	return old;
 }
-- 
1.5.6.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 8/8] memcg: recharge charges of shmem swap

[Daisuke Nishimura]

This patch enables recharge of shmem swaps.

To find the shmem's page or swap entry corresponding to a non pte_present pte,
this patch add a function(mem_cgroup_get_shmem_target()) to search them from the
inode and the offset.

This patch also enables recharge of non pte_present but not uncharged file
caches by getting the target pages via find_get_page() as do_mincore() does.

Signed-off-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
---
 include/linux/swap.h |    4 ++++
 mm/memcontrol.c      |   25 +++++++++++++++++++++----
 mm/shmem.c           |   37 +++++++++++++++++++++++++++++++++++++
 3 files changed, 62 insertions(+), 4 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4ec9001..e232653 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -278,6 +278,10 @@ extern int kswapd_run(int nid);
 /* linux/mm/shmem.c */
 extern int shmem_unuse(swp_entry_t entry, struct page *page);
 #endif /* CONFIG_MMU */
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR
+extern void mem_cgroup_get_shmem_target(struct inode *inode, pgoff_t pgoff,
+					struct page **pagep, swp_entry_t *ent);
+#endif
 
 extern void swap_unplug_io_fn(struct backing_dev_info *, struct page *);
 
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 7e82448..cd63403 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3553,10 +3553,27 @@ static int is_target_pte_for_recharge(struct vm_area_struct *vma,
 	int ret = 0;
 
 	if (!pte_present(ptent)) {
-		/* TODO: handle swap of shmes/tmpfs */
-		if (pte_none(ptent) || pte_file(ptent))
-			return 0;
-		else if (is_swap_pte(ptent)) {
+		if (pte_none(ptent) || pte_file(ptent)) {
+			struct inode *inode;
+			struct address_space *mapping;
+			pgoff_t pgoff = 0;
+
+			if (!vma->vm_file)
+				return 0;
+
+			inode = vma->vm_file->f_path.dentry->d_inode;
+			mapping = vma->vm_file->f_mapping;
+			if (pte_none(ptent))
+				pgoff = linear_page_index(vma, addr);
+			if (pte_file(ptent))
+				pgoff = pte_to_pgoff(ptent);
+
+			if (mapping_cap_swap_backed(mapping))
+				mem_cgroup_get_shmem_target(inode, pgoff,
+								&page, &ent);
+			else
+				page = find_get_page(mapping, pgoff);
+		} else if (is_swap_pte(ptent)) {
 			ent = pte_to_swp_entry(ptent);
 			if (is_migration_entry(ent))
 				return 0;
diff --git a/mm/shmem.c b/mm/shmem.c
index 356dd99..170ec44 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2693,3 +2693,40 @@ int shmem_zero_setup(struct vm_area_struct *vma)
 	vma->vm_ops = &shmem_vm_ops;
 	return 0;
 }
+
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR
+/**
+ * mem_cgroup_get_shmem_target - find a page or entry assigned to the shmem file
+ * @inode: the inode to be searched
+ * @pgoff: the offset to be searched
+ * @pagep: the pointer for the found page to be stored
+ * @ent: the pointer for the found swap entry to be stored
+ *
+ * If a page is found, refcount of it is incremented. Callers should handle
+ * these refcount.
+ */
+void mem_cgroup_get_shmem_target(struct inode *inode, pgoff_t pgoff,
+					struct page **pagep, swp_entry_t *ent)
+{
+	swp_entry_t entry = { .val = 0 }, *ptr;
+	struct page *page = NULL;
+	struct shmem_inode_info *info = SHMEM_I(inode);
+
+	if ((pgoff << PAGE_CACHE_SHIFT) >= i_size_read(inode))
+		goto out;
+
+	spin_lock(&info->lock);
+	ptr = shmem_swp_entry(info, pgoff, NULL);
+	if (ptr && ptr->val) {
+		entry.val = ptr->val;
+		page = find_get_page(&swapper_space, entry.val);
+	} else
+		page = find_get_page(inode->i_mapping, pgoff);
+	if (ptr)
+		shmem_swp_unmap(ptr);
+	spin_unlock(&info->lock);
+out:
+	*pagep = page;
+	*ent = entry;
+}
+#endif
-- 
1.5.6.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 0/8] memcg: recharge at task move (Oct13)

[KAMEZAWA Hiroyuki]

On Tue, 13 Oct 2009 13:49:03 +0900
Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp> wrote:

> Hi.
> 
> These are my current patches for recharge at task move.
> 
> In current memcg, charges associated with a task aren't moved to the new cgroup
> at task move. These patches are for this feature, that is, for recharging to
> the new cgroup and, of course, uncharging from old cgroup at task move.
> 
> I've tested these patches on 2.6.32-rc3(+ some patches) with memory pressure
> and rmdir, they didn't cause any BUGs during last weekend.
> 
> Major Changes from Sep24:
> - rebased on mmotm-2009-10-09-01-07 + KAMEZAWA-san's batched charge/uncharge(Oct09)
>   + part of KAMEZAWA-san's cleanup/fix patches(4,5,7 of Sep25 with some fixes).
> - changed the term "migrate" to "recharge".
> 
> TODO:
> - update Documentation/cgroup/memory.txt
> - implement madvise(2) (MADV_MEMCG_RECHARGE/NORECHARGE)
> 

Seems nice in general.

But, as 1st version, could you postpone recharging "shared pages" ?
I think automatic recharge of them is not very good. I don't like
Moving anonymous "shared" pages and file pages.

When a user use this method.
==
   if (fork()) {
	add this to cgroup
	execve()
   }
==
All parent's memory will be recharged.

I wonder, use madivse(MADV_MEMCG_AUTO_RECHARGE) to set flag to vmas
and use the flag as hint to auto-recharge will be good idea, finally.

(*) To be honest, I believe users will not want to modify their program
    only for this. So, recharging secified vma/file/shmem by external
    program will be necessary.


For another example, an admin tries to charge libXYZ.so into /group_A.
He can do
   # echo 0 > ..../group_A/tasks.
   # cat libXYZ.so > /dev/null

After that, if a user moves a program in group_A which uses libXYZ.so,
libXYZ.so will be recharged automatically.

There will be several policy around this. But start-from-minimum is not very
bad for this functionality because we have no feature now.

Could you start from recharge "not shared pages" ?
We'll be able to update feature, for example, add flag to memcg as
your 1st version does.


Thanks,
-Kame


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 0/8] memcg: recharge at task move (Oct13)

[Daisuke Nishimura]

Thank you for your comments.

On Wed, 14 Oct 2009 16:03:50 +0900, KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> On Tue, 13 Oct 2009 13:49:03 +0900
> Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp> wrote:
> 
> > Hi.
> > 
> > These are my current patches for recharge at task move.
> > 
> > In current memcg, charges associated with a task aren't moved to the new cgroup
> > at task move. These patches are for this feature, that is, for recharging to
> > the new cgroup and, of course, uncharging from old cgroup at task move.
> > 
> > I've tested these patches on 2.6.32-rc3(+ some patches) with memory pressure
> > and rmdir, they didn't cause any BUGs during last weekend.
> > 
> > Major Changes from Sep24:
> > - rebased on mmotm-2009-10-09-01-07 + KAMEZAWA-san's batched charge/uncharge(Oct09)
> >   + part of KAMEZAWA-san's cleanup/fix patches(4,5,7 of Sep25 with some fixes).
> > - changed the term "migrate" to "recharge".
> > 
> > TODO:
> > - update Documentation/cgroup/memory.txt
> > - implement madvise(2) (MADV_MEMCG_RECHARGE/NORECHARGE)
> > 
> 
> Seems nice in general.
> 
Thanks.

> But, as 1st version, could you postpone recharging "shared pages" ?
> I think automatic recharge of them is not very good. I don't like
> Moving anonymous "shared" pages and file pages.
> 
Just for clarification, you mean "page_mapcount > 1" by "shared", right?

> When a user use this method.
> ==
>    if (fork()) {
> 	add this to cgroup
> 	execve()
>    }
> ==
> All parent's memory will be recharged.
> 
yes.

> I wonder, use madivse(MADV_MEMCG_AUTO_RECHARGE) to set flag to vmas
> and use the flag as hint to auto-recharge will be good idea, finally.
> 
> (*) To be honest, I believe users will not want to modify their program
>     only for this. So, recharging secified vma/file/shmem by external
>     program will be necessary.
> 
I think adding new MADV_MEMCG_** would make sense, I agree that users
will not want to modify their program though and it would be a desirable feature
to specify a vma to be recharged by external program(I don't have any idea now to do it).

I think extending the meaning of memory.recharge_at_immigrate(or adding
new flag file) can be used to some extent for an administrator(or a middle ware)
to decide the type of pages to be recharged.

> 
> For another example, an admin tries to charge libXYZ.so into /group_A.
> He can do
>    # echo 0 > ..../group_A/tasks.
>    # cat libXYZ.so > /dev/null
> 
> After that, if a user moves a program in group_A which uses libXYZ.so,
> libXYZ.so will be recharged automatically.
> 
hmm, yes.

> There will be several policy around this. But start-from-minimum is not very
> bad for this functionality because we have no feature now.
> 
> Could you start from recharge "not shared pages" ?
> We'll be able to update feature, for example, add flag to memcg as
> your 1st version does.
> 
I see your concern and I agree that we can start from minimum and enhance later.

I adopted mandatory policy just because, without support for shared pages for example,
we can never recharge those pages even after all of the processes that uses those
shared pages are moved to new group.

I've not considered thoroughly yet, my current plan is
  - add support for new page type
    - file pages
    - shmem/tmpfs page
  - add support for shared(page_mapcount > 1) pages

Anyway, I'll repost my patches with only support for non-shared-anon pages
(and swaps of those pages if possible) as a 1st version.


Thanks,
Daisuke Nishimura.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>