[RFC][PATCH 0/2] memcg: simple hierarchy (v2)

[RFC][PATCH 0/2] memcg: simple hierarchy (v2)

[KAMEZAWA Hiroyuki]

This is rewritten version of memcg hierarchy handling.
...and I'm sorry tons of typos in v1.

Changelog:
  - fixed typo.
  - removed meaningless params (borrow)
  - renamed structure members.

not-for-test. just for discussion.  (I'll rewrite when our direction is fixed.)

Implemented Policy:
  - parent overcommits all children
     parent->usage = resource used by itself + resource moved to children.
     Of course, parent->limit > parent->usage. 
  - when child's limit is set, the resouce moves.
  - no automatic resource moving between parent <-> child

Example)
  1) Assume a cgroup with 1GB limits. (and no tasks belongs to this, now)
     - group_A limit=1G,usage=0M.

  2) create group B, C under A.
     - group A limit=1G, usage=0M
          - group B limit=0M, usage=0M.
          - group C limit=0M, usage=0M.

  3) increase group B's limit to 300M.
     - group A limit=1G, usage=300M.
          - group B limit=300M, usage=0M.
          - group C limit=0M, usage=0M.

  4) increase group C's limit to 500M
     - group A limit=1G, usage=800M.
          - group B limit=300M, usage=0M.
          - group C limit=500M, usage=0M.

  5) reduce group B's limit to 100M
     - group A limit=1G, usage=600M.
          - group B limit=100M, usage=0M.
          - group C limit=500M, usage=0M.

Why this is enough ?
  - A middleware can do various kind of resource balancing only by reseting "limit"
    in userland.


TODO(maybe)
  - rewrite force_empty to move the resource to the parent.

Thanks,
-Kame

[RFC][PATCH 0/2] memcg: simple hierarchy (v2)

[KAMEZAWA Hiroyuki]

This is rewritten version of memcg hierarchy handling.
...and I'm sorry tons of typos in v1.

Changelog:
  - fixed typo.
  - removed meaningless params (borrow)
  - renamed structure members.

not-for-test. just for discussion.  (I'll rewrite when our direction is fixed.)

Implemented Policy:
  - parent overcommits all children
     parent->usage = resource used by itself + resource moved to children.
     Of course, parent->limit > parent->usage. 
  - when child's limit is set, the resouce moves.
  - no automatic resource moving between parent <-> child

Example)
  1) Assume a cgroup with 1GB limits. (and no tasks belongs to this, now)
     - group_A limit=1G,usage=0M.

  2) create group B, C under A.
     - group A limit=1G, usage=0M
          - group B limit=0M, usage=0M.
          - group C limit=0M, usage=0M.

  3) increase group B's limit to 300M.
     - group A limit=1G, usage=300M.
          - group B limit=300M, usage=0M.
          - group C limit=0M, usage=0M.

  4) increase group C's limit to 500M
     - group A limit=1G, usage=800M.
          - group B limit=300M, usage=0M.
          - group C limit=500M, usage=0M.

  5) reduce group B's limit to 100M
     - group A limit=1G, usage=600M.
          - group B limit=100M, usage=0M.
          - group C limit=500M, usage=0M.

Why this is enough ?
  - A middleware can do various kind of resource balancing only by reseting "limit"
    in userland.


TODO(maybe)
  - rewrite force_empty to move the resource to the parent.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

This patch tries to implements _simple_ 'hierarchy policy' in res_counter.

While several policy of hierarchy can be considered, this patch implements
simple one 
   - the parent includes, over-commits the child
   - there are no shared resource
   - dynamic hierarchy resource usage management in the kernel is not necessary

works as following.

 1. create a child. set default child limits to be 0.
 2. set limit to child.
    2-a. before setting limit to child, prepare enough room in parent.
    2-b. increase 'usage' of parent by child's limit.
 3. the child sets its limit to the val moved from the parent.
    the parent remembers what amount of resource is to the children.

 Above means that
	- a directory's usage implies the sum of all sub directories +
          own usage.
	- there are no shared resource between parent <-> child.

 Pros.
  - simple and easy policy.
  - no hierarchy overhead.
  - no resource share among child <-> parent. very suitable for multilevel
    resource isolation.
 Cons.
  - not good to implement some kind of _intelligent_ hierarchy balancing
    in the _kernel_ 

Changelog:
  -removed borrow.
  -fixed tons of typos.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

---
 Documentation/controllers/resource_counter.txt |   28 +++++
 include/linux/res_counter.h                    |   72 +++++++++++++
 kernel/res_counter.c                           |  130 +++++++++++++++++++++++--
 3 files changed, 222 insertions(+), 8 deletions(-)

Index: hie-2.6.26-rc2-mm1/include/linux/res_counter.h
===================================================================
--- hie-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
+++ hie-2.6.26-rc2-mm1/include/linux/res_counter.h
@@ -39,6 +39,11 @@ struct res_counter {
 	 */
 	unsigned long long failcnt;
 	/*
+	 * the sum of all resource which is assigned to children.
+	 */
+	unsigned long long for_children;
+
+	/*
 	 * the lock to protect all of the above.
 	 * the routines below consider this to be IRQ-safe
 	 */
@@ -57,6 +62,7 @@ struct res_counter {
  * @nbytes:  its size...
  * @pos:     and the offset.
  */
+typedef int (*res_resize_callback_t)(struct res_counter *, unsigned long long);
 
 u64 res_counter_read_u64(struct res_counter *counter, int member);
 
@@ -65,8 +71,55 @@ ssize_t res_counter_read(struct res_coun
 		int (*read_strategy)(unsigned long long val, char *s));
 ssize_t res_counter_write(struct res_counter *counter, int member,
 		const char __user *buf, size_t nbytes, loff_t *pos,
-		int (*write_strategy)(char *buf, unsigned long long *val));
+		int (*write_strategy)(char *buf, unsigned long long *val),
+		res_resize_callback_t callback);
+
+/**
+ * Move resource from parent to child and set child's limit.
+ * By this,
+ *          res->usage of the parent increased by 'val'
+ *          res->for_children of the parent increased by 'val'
+ *          res->limit of the child increased by 'val'
+ *
+ * @child:    an entity to set res->limit.
+ * @parent:   parent of child and source of resource.
+ * @val:      How much does child want to move from parent ?
+ * @callback: A callback for making resource to allow this moving, called
+ *            against parent. callback should returns 0 at success,
+ *	      returns !0 at failure. _No_ lock is held while the callback is
+ *	      called. If no callback(NULL), no retry.
+ * @retry:    # of retries at calling callback for making resource.
+ *            -1 means infinite loop. At each retry, yield() is called.
+ *
+ * Returns 0 if success. !0 at failure.
+ *
+ */
+typedef int (*res_shrink_callback_t)(struct res_counter*, unsigned long long);
 
+int res_counter_borrow_resource(struct res_counter *child,
+				struct res_counter *parent,
+				unsigned long long val,
+				res_shrink_callback_t callback, int retry);
+/**
+ * Return resource to its parent.
+ * By this,
+ *           res->usage of the parent is decreased by 'val'
+ *           res->for_children of the parent is decreased by 'val'
+ *           res->limit of the child is decreased by 'val'
+ *
+ * @child:   entry to resize.
+ * @parent:  resource will be moved back to this.
+ * @val  :   How much does child repay to parent ? -1 means 'all'
+ * @callback: A callback for decreasing resource usage of child before
+ *            moving. If NULL, just deceases child's limit.
+ * @retry:   # of retries at calling callback for freeing resource.
+ *            -1 means infinite loop. At each retry, yield() is called.
+ * Returns 0 at success.
+ */
+int res_counter_repay_resource(struct res_counter *child,
+				struct res_counter *parent,
+				unsigned long long val,
+				res_shrink_callback_t callback, int retry);
 /*
  * the field descriptors. one for each member of res_counter
  */
@@ -76,6 +129,7 @@ enum {
 	RES_MAX_USAGE,
 	RES_LIMIT,
 	RES_FAILCNT,
+	RES_FOR_CHILDREN,
 };
 
 /*
@@ -104,6 +158,11 @@ enum charge_code __must_check res_counte
 
 enum charge_code __must_check res_counter_charge(struct res_counter *counter,
 		unsigned long val);
+/*
+ * Compare usage with ->borrow member instead of ->limit.
+ */
+int __must_check res_counter_charge_borrow(struct res_counter *counter,
+		unsigned long val);
 
 /*
  * uncharge - tell that some portion of the resource is released
@@ -158,4 +217,15 @@ static inline void res_counter_reset_fai
 	cnt->failcnt = 0;
 	spin_unlock_irqrestore(&cnt->lock, flags);
 }
+
+/*
+ * should be called only after cgroup creation.
+ */
+static inline void res_counter_zero_limit(struct res_counter *cnt)
+{
+	unsigned long flags;
+	spin_lock_irqsave(&cnt->lock, flags);
+	cnt->limit = 0;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+}
 #endif
Index: hie-2.6.26-rc2-mm1/kernel/res_counter.c
===================================================================
--- hie-2.6.26-rc2-mm1.orig/kernel/res_counter.c
+++ hie-2.6.26-rc2-mm1/kernel/res_counter.c
@@ -76,6 +76,8 @@ res_counter_member(struct res_counter *c
 		return &counter->limit;
 	case RES_FAILCNT:
 		return &counter->failcnt;
+	case RES_FOR_CHILDREN:
+		return &counter->for_children;
 	};
 
 	BUG();
@@ -106,7 +108,8 @@ u64 res_counter_read_u64(struct res_coun
 
 ssize_t res_counter_write(struct res_counter *counter, int member,
 		const char __user *userbuf, size_t nbytes, loff_t *pos,
-		int (*write_strategy)(char *st_buf, unsigned long long *val))
+		int (*write_strategy)(char *st_buf, unsigned long long *val),
+		res_resize_callback_t callback)
 {
 	int ret;
 	char *buf, *end;
@@ -135,13 +138,118 @@ ssize_t res_counter_write(struct res_cou
 		if (*end != '\0')
 			goto out_free;
 	}
-	spin_lock_irqsave(&counter->lock, flags);
-	val = res_counter_member(counter, member);
-	*val = tmp;
-	spin_unlock_irqrestore(&counter->lock, flags);
-	ret = nbytes;
+	if (member != RES_LIMIT || !callback) {
+		spin_lock_irqsave(&counter->lock, flags);
+		val = res_counter_member(counter, member);
+		*val = tmp;
+		spin_unlock_irqrestore(&counter->lock, flags);
+		ret = nbytes;
+	} else {
+		/* call a callback for hierarchy management */
+		ret = callback(counter, tmp);
+		if (!ret)
+			ret = nbytes;
+	}
+
 out_free:
 	kfree(buf);
 out:
 	return ret;
 }
+
+/*
+ * This tries to move 'val' resource from parent. At success,
+ * child->limit += val.
+ * parent->for_children += val.
+ * parent->usage += val.
+ */
+
+int res_counter_borrow_resource(struct res_counter *child,
+				struct res_counter *parent,
+				unsigned long long val,
+				res_shrink_callback_t callback, int retry)
+{
+	int success = 0;
+	unsigned long flags;
+
+	/* Borrow resource from parent */
+	success = 0;
+	while (1) {
+		/* res_counter_charge just handles 'long' value...*/
+		spin_lock_irqsave(&parent->lock, flags);
+		if (parent->usage + val < parent->limit) {
+			parent->usage += val;
+			parent->for_children += val;
+			success = 1;
+		}
+		spin_unlock_irqrestore(&parent->lock, flags);
+		if (success)
+			break;
+		if (!retry || !callback)
+			goto fail;
+		if (retry > 0)
+			--retry;
+		yield();
+		callback(parent, val);
+	}
+	/* ok, we successfully got enough resource. */
+	spin_lock_irqsave(&child->lock, flags);
+	child->limit += val;
+	spin_unlock_irqrestore(&child->lock, flags);
+	return 0;
+fail:
+	return 1;
+}
+
+/*
+ * Move resource to its parent.
+ *   child->limit -= val.
+ *   parent->usage -= val.
+ *   parent->limit -= val.
+ */
+
+int res_counter_repay_resource(struct res_counter *child,
+				struct res_counter *parent,
+				unsigned long long val,
+				res_shrink_callback_t callback, int retry)
+{
+	unsigned long flags;
+	int done = 0;
+	/* Enough resources ? */
+	while (1) {
+		spin_lock_irqsave(&child->lock, flags);
+
+		if (val == (unsigned long long)-1) {
+			val = child->limit;
+			child->limit = 0;
+			done = 1;
+		} else if (child->usage + val <= child->limit) {
+			child->limit -= val;
+			done = 1;
+		}
+		spin_unlock_irqrestore(&child->lock, flags);
+		if (done)
+			break;
+		if (!retry-- || !callback)
+			goto fail;
+		/*
+		 * we want to rest somewhere but right after callback is
+		 * not good place. So rest here.
+		 */
+		yield();
+		/* reduce resource usage */
+		callback(child, val);
+	}
+
+	/* ok, we successfully got enough resource. */
+	spin_lock_irqsave(&parent->lock, flags);
+	BUG_ON(parent->for_children < val);
+	BUG_ON(parent->usage < val);
+	parent->for_children -= val;
+	parent->usage -= val;
+	spin_unlock_irqrestore(&parent->lock, flags);
+
+	return 0;
+fail:
+	return 1;
+}
Index: hie-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
===================================================================
--- hie-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
+++ hie-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
@@ -39,10 +39,13 @@ to work with it.
  	The failcnt stands for "failures counter". This is the number of
 	resource allocation attempts that failed.
 
- c. spinlock_t lock
+ e. spinlock_t lock
 
  	Protects changes of the above values.
 
+ f. for_children
+        The amount of resource moved to the children.
+
 
 
 2. Basic accounting routines
@@ -179,3 +182,24 @@ counter fields. They are recommended to 
     still can help with it).
 
  c. Compile and run :)
+
+6. Hierarchy Model
+   1) simple isolation hierarchy.
+   res_counter supports a simple hierarchy model as that the child's resource
+   is moved from its parent.
+
+   When  the limit is set to a child, its parent's usage increases by the
+   amount of limit. i.e. the child borrows resource from its parent when
+   it set the limit.
+
+   hirarchical cgroup is very useful when you implements hierarchical
+   resource isolation. For example,
+   A) admin - user
+   - system admin layer ....the first level
+	- user layer    ....the second level for user A, B, C
+   B) application/service layer.
+   -  application layer ... the first level
+        - service layer ... the second level for service Gold, Silver,...
+
+  see res_counter_borrow_resource() and res_counter_repay_resource().
+

[RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

This patch tries to implements _simple_ 'hierarchy policy' in res_counter.

While several policy of hierarchy can be considered, this patch implements
simple one 
   - the parent includes, over-commits the child
   - there are no shared resource
   - dynamic hierarchy resource usage management in the kernel is not necessary

works as following.

 1. create a child. set default child limits to be 0.
 2. set limit to child.
    2-a. before setting limit to child, prepare enough room in parent.
    2-b. increase 'usage' of parent by child's limit.
 3. the child sets its limit to the val moved from the parent.
    the parent remembers what amount of resource is to the children.

 Above means that
	- a directory's usage implies the sum of all sub directories +
          own usage.
	- there are no shared resource between parent <-> child.

 Pros.
  - simple and easy policy.
  - no hierarchy overhead.
  - no resource share among child <-> parent. very suitable for multilevel
    resource isolation.
 Cons.
  - not good to implement some kind of _intelligent_ hierarchy balancing
    in the _kernel_ 

Changelog:
  -removed borrow.
  -fixed tons of typos.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

---
 Documentation/controllers/resource_counter.txt |   28 +++++
 include/linux/res_counter.h                    |   72 +++++++++++++
 kernel/res_counter.c                           |  130 +++++++++++++++++++++++--
 3 files changed, 222 insertions(+), 8 deletions(-)

Index: hie-2.6.26-rc2-mm1/include/linux/res_counter.h
===================================================================
--- hie-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
+++ hie-2.6.26-rc2-mm1/include/linux/res_counter.h
@@ -39,6 +39,11 @@ struct res_counter {
 	 */
 	unsigned long long failcnt;
 	/*
+	 * the sum of all resource which is assigned to children.
+	 */
+	unsigned long long for_children;
+
+	/*
 	 * the lock to protect all of the above.
 	 * the routines below consider this to be IRQ-safe
 	 */
@@ -57,6 +62,7 @@ struct res_counter {
  * @nbytes:  its size...
  * @pos:     and the offset.
  */
+typedef int (*res_resize_callback_t)(struct res_counter *, unsigned long long);
 
 u64 res_counter_read_u64(struct res_counter *counter, int member);
 
@@ -65,8 +71,55 @@ ssize_t res_counter_read(struct res_coun
 		int (*read_strategy)(unsigned long long val, char *s));
 ssize_t res_counter_write(struct res_counter *counter, int member,
 		const char __user *buf, size_t nbytes, loff_t *pos,
-		int (*write_strategy)(char *buf, unsigned long long *val));
+		int (*write_strategy)(char *buf, unsigned long long *val),
+		res_resize_callback_t callback);
+
+/**
+ * Move resource from parent to child and set child's limit.
+ * By this,
+ *          res->usage of the parent increased by 'val'
+ *          res->for_children of the parent increased by 'val'
+ *          res->limit of the child increased by 'val'
+ *
+ * @child:    an entity to set res->limit.
+ * @parent:   parent of child and source of resource.
+ * @val:      How much does child want to move from parent ?
+ * @callback: A callback for making resource to allow this moving, called
+ *            against parent. callback should returns 0 at success,
+ *	      returns !0 at failure. _No_ lock is held while the callback is
+ *	      called. If no callback(NULL), no retry.
+ * @retry:    # of retries at calling callback for making resource.
+ *            -1 means infinite loop. At each retry, yield() is called.
+ *
+ * Returns 0 if success. !0 at failure.
+ *
+ */
+typedef int (*res_shrink_callback_t)(struct res_counter*, unsigned long long);
 
+int res_counter_borrow_resource(struct res_counter *child,
+				struct res_counter *parent,
+				unsigned long long val,
+				res_shrink_callback_t callback, int retry);
+/**
+ * Return resource to its parent.
+ * By this,
+ *           res->usage of the parent is decreased by 'val'
+ *           res->for_children of the parent is decreased by 'val'
+ *           res->limit of the child is decreased by 'val'
+ *
+ * @child:   entry to resize.
+ * @parent:  resource will be moved back to this.
+ * @val  :   How much does child repay to parent ? -1 means 'all'
+ * @callback: A callback for decreasing resource usage of child before
+ *            moving. If NULL, just deceases child's limit.
+ * @retry:   # of retries at calling callback for freeing resource.
+ *            -1 means infinite loop. At each retry, yield() is called.
+ * Returns 0 at success.
+ */
+int res_counter_repay_resource(struct res_counter *child,
+				struct res_counter *parent,
+				unsigned long long val,
+				res_shrink_callback_t callback, int retry);
 /*
  * the field descriptors. one for each member of res_counter
  */
@@ -76,6 +129,7 @@ enum {
 	RES_MAX_USAGE,
 	RES_LIMIT,
 	RES_FAILCNT,
+	RES_FOR_CHILDREN,
 };
 
 /*
@@ -104,6 +158,11 @@ enum charge_code __must_check res_counte
 
 enum charge_code __must_check res_counter_charge(struct res_counter *counter,
 		unsigned long val);
+/*
+ * Compare usage with ->borrow member instead of ->limit.
+ */
+int __must_check res_counter_charge_borrow(struct res_counter *counter,
+		unsigned long val);
 
 /*
  * uncharge - tell that some portion of the resource is released
@@ -158,4 +217,15 @@ static inline void res_counter_reset_fai
 	cnt->failcnt = 0;
 	spin_unlock_irqrestore(&cnt->lock, flags);
 }
+
+/*
+ * should be called only after cgroup creation.
+ */
+static inline void res_counter_zero_limit(struct res_counter *cnt)
+{
+	unsigned long flags;
+	spin_lock_irqsave(&cnt->lock, flags);
+	cnt->limit = 0;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+}
 #endif
Index: hie-2.6.26-rc2-mm1/kernel/res_counter.c
===================================================================
--- hie-2.6.26-rc2-mm1.orig/kernel/res_counter.c
+++ hie-2.6.26-rc2-mm1/kernel/res_counter.c
@@ -76,6 +76,8 @@ res_counter_member(struct res_counter *c
 		return &counter->limit;
 	case RES_FAILCNT:
 		return &counter->failcnt;
+	case RES_FOR_CHILDREN:
+		return &counter->for_children;
 	};
 
 	BUG();
@@ -106,7 +108,8 @@ u64 res_counter_read_u64(struct res_coun
 
 ssize_t res_counter_write(struct res_counter *counter, int member,
 		const char __user *userbuf, size_t nbytes, loff_t *pos,
-		int (*write_strategy)(char *st_buf, unsigned long long *val))
+		int (*write_strategy)(char *st_buf, unsigned long long *val),
+		res_resize_callback_t callback)
 {
 	int ret;
 	char *buf, *end;
@@ -135,13 +138,118 @@ ssize_t res_counter_write(struct res_cou
 		if (*end != '\0')
 			goto out_free;
 	}
-	spin_lock_irqsave(&counter->lock, flags);
-	val = res_counter_member(counter, member);
-	*val = tmp;
-	spin_unlock_irqrestore(&counter->lock, flags);
-	ret = nbytes;
+	if (member != RES_LIMIT || !callback) {
+		spin_lock_irqsave(&counter->lock, flags);
+		val = res_counter_member(counter, member);
+		*val = tmp;
+		spin_unlock_irqrestore(&counter->lock, flags);
+		ret = nbytes;
+	} else {
+		/* call a callback for hierarchy management */
+		ret = callback(counter, tmp);
+		if (!ret)
+			ret = nbytes;
+	}
+
 out_free:
 	kfree(buf);
 out:
 	return ret;
 }
+
+/*
+ * This tries to move 'val' resource from parent. At success,
+ * child->limit += val.
+ * parent->for_children += val.
+ * parent->usage += val.
+ */
+
+int res_counter_borrow_resource(struct res_counter *child,
+				struct res_counter *parent,
+				unsigned long long val,
+				res_shrink_callback_t callback, int retry)
+{
+	int success = 0;
+	unsigned long flags;
+
+	/* Borrow resource from parent */
+	success = 0;
+	while (1) {
+		/* res_counter_charge just handles 'long' value...*/
+		spin_lock_irqsave(&parent->lock, flags);
+		if (parent->usage + val < parent->limit) {
+			parent->usage += val;
+			parent->for_children += val;
+			success = 1;
+		}
+		spin_unlock_irqrestore(&parent->lock, flags);
+		if (success)
+			break;
+		if (!retry || !callback)
+			goto fail;
+		if (retry > 0)
+			--retry;
+		yield();
+		callback(parent, val);
+	}
+	/* ok, we successfully got enough resource. */
+	spin_lock_irqsave(&child->lock, flags);
+	child->limit += val;
+	spin_unlock_irqrestore(&child->lock, flags);
+	return 0;
+fail:
+	return 1;
+}
+
+/*
+ * Move resource to its parent.
+ *   child->limit -= val.
+ *   parent->usage -= val.
+ *   parent->limit -= val.
+ */
+
+int res_counter_repay_resource(struct res_counter *child,
+				struct res_counter *parent,
+				unsigned long long val,
+				res_shrink_callback_t callback, int retry)
+{
+	unsigned long flags;
+	int done = 0;
+	/* Enough resources ? */
+	while (1) {
+		spin_lock_irqsave(&child->lock, flags);
+
+		if (val == (unsigned long long)-1) {
+			val = child->limit;
+			child->limit = 0;
+			done = 1;
+		} else if (child->usage + val <= child->limit) {
+			child->limit -= val;
+			done = 1;
+		}
+		spin_unlock_irqrestore(&child->lock, flags);
+		if (done)
+			break;
+		if (!retry-- || !callback)
+			goto fail;
+		/*
+		 * we want to rest somewhere but right after callback is
+		 * not good place. So rest here.
+		 */
+		yield();
+		/* reduce resource usage */
+		callback(child, val);
+	}
+
+	/* ok, we successfully got enough resource. */
+	spin_lock_irqsave(&parent->lock, flags);
+	BUG_ON(parent->for_children < val);
+	BUG_ON(parent->usage < val);
+	parent->for_children -= val;
+	parent->usage -= val;
+	spin_unlock_irqrestore(&parent->lock, flags);
+
+	return 0;
+fail:
+	return 1;
+}
Index: hie-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
===================================================================
--- hie-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
+++ hie-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
@@ -39,10 +39,13 @@ to work with it.
  	The failcnt stands for "failures counter". This is the number of
 	resource allocation attempts that failed.
 
- c. spinlock_t lock
+ e. spinlock_t lock
 
  	Protects changes of the above values.
 
+ f. for_children
+        The amount of resource moved to the children.
+
 
 
 2. Basic accounting routines
@@ -179,3 +182,24 @@ counter fields. They are recommended to 
     still can help with it).
 
  c. Compile and run :)
+
+6. Hierarchy Model
+   1) simple isolation hierarchy.
+   res_counter supports a simple hierarchy model as that the child's resource
+   is moved from its parent.
+
+   When  the limit is set to a child, its parent's usage increases by the
+   amount of limit. i.e. the child borrows resource from its parent when
+   it set the limit.
+
+   hirarchical cgroup is very useful when you implements hierarchical
+   resource isolation. For example,
+   A) admin - user
+   - system admin layer ....the first level
+	- user layer    ....the second level for user A, B, C
+   B) application/service layer.
+   -  application layer ... the first level
+        - service layer ... the second level for service Gold, Silver,...
+
+  see res_counter_borrow_resource() and res_counter_repay_resource().
+

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

KAMEZAWA Hiroyuki wrote:
> This patch tries to implements _simple_ 'hierarchy policy' in res_counter.
> 
> While several policy of hierarchy can be considered, this patch implements
> simple one 
>    - the parent includes, over-commits the child
>    - there are no shared resource

I am not sure if this is desirable. The concept of a hierarchy applies really
well when there are shared resources.

>    - dynamic hierarchy resource usage management in the kernel is not necessary
> 

Could you please elaborate as to why? I am not sure I understand your point

> works as following.
> 
>  1. create a child. set default child limits to be 0.
>  2. set limit to child.
>     2-a. before setting limit to child, prepare enough room in parent.
>     2-b. increase 'usage' of parent by child's limit.

The problem with this is that you are forcing the parent will run into a reclaim
loop even if the child is not using the assigned limit to it.

>  3. the child sets its limit to the val moved from the parent.
>     the parent remembers what amount of resource is to the children.
> 

All of this needs to be dynamic

>  Above means that
> 	- a directory's usage implies the sum of all sub directories +
>           own usage.
> 	- there are no shared resource between parent <-> child.
> 
>  Pros.
>   - simple and easy policy.
>   - no hierarchy overhead.
>   - no resource share among child <-> parent. very suitable for multilevel
>     resource isolation.

Sharing is an important aspect of hierachies. I am not convinced of this
approach. Did you look at the patches I sent out? Was there something
fundamentally broken in them?

[snip]

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

KAMEZAWA Hiroyuki wrote:
> This patch tries to implements _simple_ 'hierarchy policy' in res_counter.
> 
> While several policy of hierarchy can be considered, this patch implements
> simple one 
>    - the parent includes, over-commits the child
>    - there are no shared resource

I am not sure if this is desirable. The concept of a hierarchy applies really
well when there are shared resources.

>    - dynamic hierarchy resource usage management in the kernel is not necessary
> 

Could you please elaborate as to why? I am not sure I understand your point

> works as following.
> 
>  1. create a child. set default child limits to be 0.
>  2. set limit to child.
>     2-a. before setting limit to child, prepare enough room in parent.
>     2-b. increase 'usage' of parent by child's limit.

The problem with this is that you are forcing the parent will run into a reclaim
loop even if the child is not using the assigned limit to it.

>  3. the child sets its limit to the val moved from the parent.
>     the parent remembers what amount of resource is to the children.
> 

All of this needs to be dynamic

>  Above means that
> 	- a directory's usage implies the sum of all sub directories +
>           own usage.
> 	- there are no shared resource between parent <-> child.
> 
>  Pros.
>   - simple and easy policy.
>   - no hierarchy overhead.
>   - no resource share among child <-> parent. very suitable for multilevel
>     resource isolation.

Sharing is an important aspect of hierachies. I am not convinced of this
approach. Did you look at the patches I sent out? Was there something
fundamentally broken in them?

[snip]

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

>KAMEZAWA Hiroyuki wrote:
>> This patch tries to implements _simple_ 'hierarchy policy' in res_counter.
>> 
>> While several policy of hierarchy can be considered, this patch implements
>> simple one 
>>    - the parent includes, over-commits the child
>>    - there are no shared resource
>
>I am not sure if this is desirable. The concept of a hierarchy applies really
>well when there are shared resources.
>
>>    - dynamic hierarchy resource usage management in the kernel is not neces
sary
>> 
>
>Could you please elaborate as to why? I am not sure I understand your point
>

ok, let's consider a _miiddleware_ wchich has following paramater.

An expoterd param to the user.
   - user_memory_limit
parameters for co-operation with the kernel
   - kernel_memory_limit

And here,
   user_memory_limit >= kernel_memory_limit == cgroup's memory.limits_in_bytes

When a user ask the miidleware to set limit to 1Gbytes
   user_memory_limit = 1G
   kernel_memory_limit = 0-1G.
It moves kernel_memory_limit dynamically 0 to 1Gbytes and reset limits_in_byte
s in dynamic way with checking memory cgroup's statistics.
Of course, we can add some kind of interdace , as following
  - failure_notifier - triggered at failcnt increment.
  - threshhold_notifier - triggered as usage > threshold.

>> works as following.
>> 
>>  1. create a child. set default child limits to be 0.
>>  2. set limit to child.
>>     2-a. before setting limit to child, prepare enough room in parent.
>>     2-b. increase 'usage' of parent by child's limit.
>
>The problem with this is that you are forcing the parent will run into a recl
aim
>loop even if the child is not using the assigned limit to it.
>
That's not problem because it's avoildable by users.
But it's ok to limit the sum of child's limit to be below XX % ot the parent.

>>  3. the child sets its limit to the val moved from the parent.
>>     the parent remembers what amount of resource is to the children.
>> 
>
>All of this needs to be dynamic
>
As explained, this can be dynamic by middleware.

>>  Above means that
>> 	- a directory's usage implies the sum of all sub directories +
>>           own usage.
>> 	- there are no shared resource between parent <-> child.
>> 
>>  Pros.
>>   - simple and easy policy.
>>   - no hierarchy overhead.
>>   - no resource share among child <-> parent. very suitable for multilevel
>>     resource isolation.
>
>Sharing is an important aspect of hierachies. I am not convinced of this
>approach. Did you look at the patches I sent out? Was there something
>fundamentally broken in them?
>

Yes, I read. And tried to make it faster and found it will be complicated.
One problem is overhead of counter itself.
Another problem is overhead of shrinking multi-level LRU with feedback.
One more problem is that it's hard to implement various kinds of hierarchy
policy. I believe there are other hierarhcy policies rather than OpenVZ
want to use. Kicking out functions to middleware AMAP is what I'm thinking
now.

Thanks,
-Kame

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

>KAMEZAWA Hiroyuki wrote:
>> This patch tries to implements _simple_ 'hierarchy policy' in res_counter.
>> 
>> While several policy of hierarchy can be considered, this patch implements
>> simple one 
>>    - the parent includes, over-commits the child
>>    - there are no shared resource
>
>I am not sure if this is desirable. The concept of a hierarchy applies really
>well when there are shared resources.
>
>>    - dynamic hierarchy resource usage management in the kernel is not neces
sary
>> 
>
>Could you please elaborate as to why? I am not sure I understand your point
>

ok, let's consider a _miiddleware_ wchich has following paramater.

An expoterd param to the user.
   - user_memory_limit
parameters for co-operation with the kernel
   - kernel_memory_limit

And here,
   user_memory_limit >= kernel_memory_limit == cgroup's memory.limits_in_bytes

When a user ask the miidleware to set limit to 1Gbytes
   user_memory_limit = 1G
   kernel_memory_limit = 0-1G.
It moves kernel_memory_limit dynamically 0 to 1Gbytes and reset limits_in_byte
s in dynamic way with checking memory cgroup's statistics.
Of course, we can add some kind of interdace , as following
  - failure_notifier - triggered at failcnt increment.
  - threshhold_notifier - triggered as usage > threshold.

>> works as following.
>> 
>>  1. create a child. set default child limits to be 0.
>>  2. set limit to child.
>>     2-a. before setting limit to child, prepare enough room in parent.
>>     2-b. increase 'usage' of parent by child's limit.
>
>The problem with this is that you are forcing the parent will run into a recl
aim
>loop even if the child is not using the assigned limit to it.
>
That's not problem because it's avoildable by users.
But it's ok to limit the sum of child's limit to be below XX % ot the parent.

>>  3. the child sets its limit to the val moved from the parent.
>>     the parent remembers what amount of resource is to the children.
>> 
>
>All of this needs to be dynamic
>
As explained, this can be dynamic by middleware.

>>  Above means that
>> 	- a directory's usage implies the sum of all sub directories +
>>           own usage.
>> 	- there are no shared resource between parent <-> child.
>> 
>>  Pros.
>>   - simple and easy policy.
>>   - no hierarchy overhead.
>>   - no resource share among child <-> parent. very suitable for multilevel
>>     resource isolation.
>
>Sharing is an important aspect of hierachies. I am not convinced of this
>approach. Did you look at the patches I sent out? Was there something
>fundamentally broken in them?
>

Yes, I read. And tried to make it faster and found it will be complicated.
One problem is overhead of counter itself.
Another problem is overhead of shrinking multi-level LRU with feedback.
One more problem is that it's hard to implement various kinds of hierarchy
policy. I believe there are other hierarhcy policies rather than OpenVZ
want to use. Kicking out functions to middleware AMAP is what I'm thinking
now.

Thanks,
-Kame



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

kamezawa.hiroyu@jp.fujitsu.com wrote:
>> KAMEZAWA Hiroyuki wrote:
>>> This patch tries to implements _simple_ 'hierarchy policy' in res_counter.
>>>
>>> While several policy of hierarchy can be considered, this patch implements
>>> simple one 
>>>    - the parent includes, over-commits the child
>>>    - there are no shared resource
>> I am not sure if this is desirable. The concept of a hierarchy applies really
>> well when there are shared resources.
>>
>>>    - dynamic hierarchy resource usage management in the kernel is not neces
> sary
>> Could you please elaborate as to why? I am not sure I understand your point
>>
> 
> ok, let's consider a _miiddleware_ wchich has following paramater.
> 
> An expoterd param to the user.
>    - user_memory_limit
> parameters for co-operation with the kernel
>    - kernel_memory_limit
> 
> And here,
>    user_memory_limit >= kernel_memory_limit == cgroup's memory.limits_in_bytes
> 
> When a user ask the miidleware to set limit to 1Gbytes
>    user_memory_limit = 1G
>    kernel_memory_limit = 0-1G.
> It moves kernel_memory_limit dynamically 0 to 1Gbytes and reset limits_in_byte
> s in dynamic way with checking memory cgroup's statistics.
> Of course, we can add some kind of interdace , as following
>   - failure_notifier - triggered at failcnt increment.
>   - threshhold_notifier - triggered as usage > threshold.
> 
>>> works as following.
>>>
>>>  1. create a child. set default child limits to be 0.
>>>  2. set limit to child.
>>>     2-a. before setting limit to child, prepare enough room in parent.
>>>     2-b. increase 'usage' of parent by child's limit.
>> The problem with this is that you are forcing the parent will run into a recl
> aim
>> loop even if the child is not using the assigned limit to it.
>>
> That's not problem because it's avoildable by users.
> But it's ok to limit the sum of child's limit to be below XX % ot the parent.
> 
>>>  3. the child sets its limit to the val moved from the parent.
>>>     the parent remembers what amount of resource is to the children.
>>>
>> All of this needs to be dynamic
>>
> As explained, this can be dynamic by middleware.
> 
>>>  Above means that
>>> 	- a directory's usage implies the sum of all sub directories +
>>>           own usage.
>>> 	- there are no shared resource between parent <-> child.
>>>
>>>  Pros.
>>>   - simple and easy policy.
>>>   - no hierarchy overhead.
>>>   - no resource share among child <-> parent. very suitable for multilevel
>>>     resource isolation.
>> Sharing is an important aspect of hierachies. I am not convinced of this
>> approach. Did you look at the patches I sent out? Was there something
>> fundamentally broken in them?
>>
> 
> Yes, I read. And tried to make it faster and found it will be complicated.
> One problem is overhead of counter itself.
> Another problem is overhead of shrinking multi-level LRU with feedback.
> One more problem is that it's hard to implement various kinds of hierarchy
> policy. I believe there are other hierarhcy policies rather than OpenVZ
> want to use. Kicking out functions to middleware AMAP is what I'm thinking
> now.

One way to manage hierarchies other than via limits is to use shares (please see
the shares used by the cpu controller). Basically, what you've done with limits
is done with shares

If a parent has 100 shares, then it can decide how many to pass on to it's  children
based on the shares of the child and your logic would work well. I propose
assigning top level (high resolution) shares to the root of the cgroup and in a
hierarchy passing them down to children and sharing it with them. Based on the
shares, deduce the limit of each node in the hierarchy.

What do you think?


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

kamezawa.hiroyu@jp.fujitsu.com wrote:
>> KAMEZAWA Hiroyuki wrote:
>>> This patch tries to implements _simple_ 'hierarchy policy' in res_counter.
>>>
>>> While several policy of hierarchy can be considered, this patch implements
>>> simple one 
>>>    - the parent includes, over-commits the child
>>>    - there are no shared resource
>> I am not sure if this is desirable. The concept of a hierarchy applies really
>> well when there are shared resources.
>>
>>>    - dynamic hierarchy resource usage management in the kernel is not neces
> sary
>> Could you please elaborate as to why? I am not sure I understand your point
>>
> 
> ok, let's consider a _miiddleware_ wchich has following paramater.
> 
> An expoterd param to the user.
>    - user_memory_limit
> parameters for co-operation with the kernel
>    - kernel_memory_limit
> 
> And here,
>    user_memory_limit >= kernel_memory_limit == cgroup's memory.limits_in_bytes
> 
> When a user ask the miidleware to set limit to 1Gbytes
>    user_memory_limit = 1G
>    kernel_memory_limit = 0-1G.
> It moves kernel_memory_limit dynamically 0 to 1Gbytes and reset limits_in_byte
> s in dynamic way with checking memory cgroup's statistics.
> Of course, we can add some kind of interdace , as following
>   - failure_notifier - triggered at failcnt increment.
>   - threshhold_notifier - triggered as usage > threshold.
> 
>>> works as following.
>>>
>>>  1. create a child. set default child limits to be 0.
>>>  2. set limit to child.
>>>     2-a. before setting limit to child, prepare enough room in parent.
>>>     2-b. increase 'usage' of parent by child's limit.
>> The problem with this is that you are forcing the parent will run into a recl
> aim
>> loop even if the child is not using the assigned limit to it.
>>
> That's not problem because it's avoildable by users.
> But it's ok to limit the sum of child's limit to be below XX % ot the parent.
> 
>>>  3. the child sets its limit to the val moved from the parent.
>>>     the parent remembers what amount of resource is to the children.
>>>
>> All of this needs to be dynamic
>>
> As explained, this can be dynamic by middleware.
> 
>>>  Above means that
>>> 	- a directory's usage implies the sum of all sub directories +
>>>           own usage.
>>> 	- there are no shared resource between parent <-> child.
>>>
>>>  Pros.
>>>   - simple and easy policy.
>>>   - no hierarchy overhead.
>>>   - no resource share among child <-> parent. very suitable for multilevel
>>>     resource isolation.
>> Sharing is an important aspect of hierachies. I am not convinced of this
>> approach. Did you look at the patches I sent out? Was there something
>> fundamentally broken in them?
>>
> 
> Yes, I read. And tried to make it faster and found it will be complicated.
> One problem is overhead of counter itself.
> Another problem is overhead of shrinking multi-level LRU with feedback.
> One more problem is that it's hard to implement various kinds of hierarchy
> policy. I believe there are other hierarhcy policies rather than OpenVZ
> want to use. Kicking out functions to middleware AMAP is what I'm thinking
> now.

One way to manage hierarchies other than via limits is to use shares (please see
the shares used by the cpu controller). Basically, what you've done with limits
is done with shares

If a parent has 100 shares, then it can decide how many to pass on to it's  children
based on the shares of the child and your logic would work well. I propose
assigning top level (high resolution) shares to the root of the cgroup and in a
hierarchy passing them down to children and sharing it with them. Based on the
shares, deduce the limit of each node in the hierarchy.

What do you think?


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

----- Original Message -----

>> One more problem is that it's hard to implement various kinds of hierarchy
>> policy. I believe there are other hierarhcy policies rather than OpenVZ
>> want to use. Kicking out functions to middleware AMAP is what I'm thinking
>> now.
>
>One way to manage hierarchies other than via limits is to use shares (please 
see
>the shares used by the cpu controller). Basically, what you've done with limi
ts
>is done with shares
>
Yes, I like _share_ rather than limits.

>If a parent has 100 shares, then it can decide how many to pass on to it's  c
hildren
>based on the shares of the child and your logic would work well. I propose
>assigning top level (high resolution) shares to the root of the cgroup and in
 a
>hierarchy passing them down to children and sharing it with them. Based on th
e
>shares, deduce the limit of each node in the hierarchy.
>
>What do you think?
>
As you wrote, a middleware can do controls based on share by limits.
And it seems much easier to implement it in userland rather than in the kernel
.

Here is an example. (just an example...)
Please point out if I'm misunderstanding "share".

root_level/                   = limit 1G.
          /child_A = share=30
          /child_B = share=15
          /child_C = share=5
(and assume there is no process under root_level for make explanation easy..)

0. At first, before starting to use memory, set all kernel_memory_limit.
root_level.limit = 1G
  child_A.limit=64M,usage=0
  child_B.limit=64M,usage=0
  child_C.limit=64M,usage=0
  free_resource=808M 

1. next, a process in child_C start to run and use memory of 600M.
root_level.limit = 1G
  child_A.limit=64M
  child_B.limit=64M
  child_C.limit=600M,usage=600M
  free_resource=272M

2. now, a process in child_A start tu run and use memory of 800M.
  child_A.limit=800M,usage=800M
  child_B.limit=64M,usage=0M
  child_C.limit=136M,usage=136M
  free_resouce=0,A:C=6:1

3.Finally, a process in child_B start. and use memory of 500M.
  child_A.limit=600M,usage=600M
  child_B.limit=300M,usage=300M
  child_C.limit=100M,usage=100M
  free_resouce=0, A:B:C=6:3:1

4. one more, a process in A exits.
  child_A.limit=64M, usage=0M
  child_B.limit=500M, usage=500M
  child_C.limit=436M, usage=436M
  free_resouce=0, B:C=3:1 (but B just want to use 500M)

This is only an example and the middleware can more pricise "limit"
contols by checking statistics of memory controller hierarchy based on
their own policy.

What I think now is what kind of statistics/notifier/controls are
necessary to implement shares in middleware. How pricise/quick work the
middleware can do is based on interfaces.
Maybe the middleware should know "how fast the application runs now" by
some kind of check or co-operative interface with the application.
But I'm not sure how the kernel can help it.

Thanks,
-Kame

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

----- Original Message -----

>> One more problem is that it's hard to implement various kinds of hierarchy
>> policy. I believe there are other hierarhcy policies rather than OpenVZ
>> want to use. Kicking out functions to middleware AMAP is what I'm thinking
>> now.
>
>One way to manage hierarchies other than via limits is to use shares (please 
see
>the shares used by the cpu controller). Basically, what you've done with limi
ts
>is done with shares
>
Yes, I like _share_ rather than limits.

>If a parent has 100 shares, then it can decide how many to pass on to it's  c
hildren
>based on the shares of the child and your logic would work well. I propose
>assigning top level (high resolution) shares to the root of the cgroup and in
 a
>hierarchy passing them down to children and sharing it with them. Based on th
e
>shares, deduce the limit of each node in the hierarchy.
>
>What do you think?
>
As you wrote, a middleware can do controls based on share by limits.
And it seems much easier to implement it in userland rather than in the kernel
.

Here is an example. (just an example...)
Please point out if I'm misunderstanding "share".

root_level/                   = limit 1G.
          /child_A = share=30
          /child_B = share=15
          /child_C = share=5
(and assume there is no process under root_level for make explanation easy..)

0. At first, before starting to use memory, set all kernel_memory_limit.
root_level.limit = 1G
  child_A.limit=64M,usage=0
  child_B.limit=64M,usage=0
  child_C.limit=64M,usage=0
  free_resource=808M 

1. next, a process in child_C start to run and use memory of 600M.
root_level.limit = 1G
  child_A.limit=64M
  child_B.limit=64M
  child_C.limit=600M,usage=600M
  free_resource=272M

2. now, a process in child_A start tu run and use memory of 800M.
  child_A.limit=800M,usage=800M
  child_B.limit=64M,usage=0M
  child_C.limit=136M,usage=136M
  free_resouce=0,A:C=6:1

3.Finally, a process in child_B start. and use memory of 500M.
  child_A.limit=600M,usage=600M
  child_B.limit=300M,usage=300M
  child_C.limit=100M,usage=100M
  free_resouce=0, A:B:C=6:3:1

4. one more, a process in A exits.
  child_A.limit=64M, usage=0M
  child_B.limit=500M, usage=500M
  child_C.limit=436M, usage=436M
  free_resouce=0, B:C=3:1 (but B just want to use 500M)

This is only an example and the middleware can more pricise "limit"
contols by checking statistics of memory controller hierarchy based on
their own policy.

What I think now is what kind of statistics/notifier/controls are
necessary to implement shares in middleware. How pricise/quick work the
middleware can do is based on interfaces.
Maybe the middleware should know "how fast the application runs now" by
some kind of check or co-operative interface with the application.
But I'm not sure how the kernel can help it.

Thanks,
-Kame


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

kamezawa.hiroyu@jp.fujitsu.com wrote:
> ----- Original Message -----
> 
>>> One more problem is that it's hard to implement various kinds of hierarchy
>>> policy. I believe there are other hierarhcy policies rather than OpenVZ
>>> want to use. Kicking out functions to middleware AMAP is what I'm thinking
>>> now.
>> One way to manage hierarchies other than via limits is to use shares (please 
> see
>> the shares used by the cpu controller). Basically, what you've done with limi
> ts
>> is done with shares
>>
> Yes, I like _share_ rather than limits.
> 
>> If a parent has 100 shares, then it can decide how many to pass on to it's  c
> hildren
>> based on the shares of the child and your logic would work well. I propose
>> assigning top level (high resolution) shares to the root of the cgroup and in
>  a
>> hierarchy passing them down to children and sharing it with them. Based on th
> e
>> shares, deduce the limit of each node in the hierarchy.
>>
>> What do you think?
>>
> As you wrote, a middleware can do controls based on share by limits.
> And it seems much easier to implement it in userland rather than in the kernel
> .

The good thing about user space is that moves unnecessary code outside the
kernel, but the hard thing is standardization. If every middleware is going to
implement what you say, imagine the code duplication, unless we standardize this
into a library component. More comments below. I am not sure about the
difference between user_memory_limit and kernel_memory_limit. Could you please
elaborate.

> 
> Here is an example. (just an example...)
> Please point out if I'm misunderstanding "share".
> 
> root_level/                   = limit 1G.
>           /child_A = share=30
>           /child_B = share=15
>           /child_C = share=5
> (and assume there is no process under root_level for make explanation easy..)
> 
> 0. At first, before starting to use memory, set all kernel_memory_limit.
> root_level.limit = 1G
>   child_A.limit=64M,usage=0
>   child_B.limit=64M,usage=0
>   child_C.limit=64M,usage=0
>   free_resource=808M 
> 

This sounds incorrect, since the limits should be proportional to shares. If the
maximum shares in the root were 100 (*ideally we want higher resolution than that)
Then

child_A.limit = .3 * 1G
child_B.limit = .15 * 1G

and so on


> 1. next, a process in child_C start to run and use memory of 600M.
> root_level.limit = 1G
>   child_A.limit=64M
>   child_B.limit=64M
>   child_C.limit=600M,usage=600M
>   free_resource=272M
> 

How is that feasible, it's limit was 64M, how did it bump up to 600M? If you
want something like that, child_C should have no limits.

> 2. now, a process in child_A start tu run and use memory of 800M.
>   child_A.limit=800M,usage=800M
>   child_B.limit=64M,usage=0M
>   child_C.limit=136M,usage=136M
>   free_resouce=0,A:C=6:1
> 

Not sure I understand this step

> 3.Finally, a process in child_B start. and use memory of 500M.
>   child_A.limit=600M,usage=600M
>   child_B.limit=300M,usage=300M
>   child_C.limit=100M,usage=100M
>   free_resouce=0, A:B:C=6:3:1
> 

Not sure I understand this step

> 4. one more, a process in A exits.
>   child_A.limit=64M, usage=0M
>   child_B.limit=500M, usage=500M
>   child_C.limit=436M, usage=436M
>   free_resouce=0, B:C=3:1 (but B just want to use 500M)
> 

Not sure I understand this step

> This is only an example and the middleware can more pricise "limit"
> contols by checking statistics of memory controller hierarchy based on
> their own policy.
> 
> What I think now is what kind of statistics/notifier/controls are
> necessary to implement shares in middleware. How pricise/quick work the
> middleware can do is based on interfaces.
> Maybe the middleware should know "how fast the application runs now" by
> some kind of check or co-operative interface with the application.
> But I'm not sure how the kernel can help it.

I am not sure if I understand your proposal at this point.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

kamezawa.hiroyu@jp.fujitsu.com wrote:
> ----- Original Message -----
> 
>>> One more problem is that it's hard to implement various kinds of hierarchy
>>> policy. I believe there are other hierarhcy policies rather than OpenVZ
>>> want to use. Kicking out functions to middleware AMAP is what I'm thinking
>>> now.
>> One way to manage hierarchies other than via limits is to use shares (please 
> see
>> the shares used by the cpu controller). Basically, what you've done with limi
> ts
>> is done with shares
>>
> Yes, I like _share_ rather than limits.
> 
>> If a parent has 100 shares, then it can decide how many to pass on to it's  c
> hildren
>> based on the shares of the child and your logic would work well. I propose
>> assigning top level (high resolution) shares to the root of the cgroup and in
>  a
>> hierarchy passing them down to children and sharing it with them. Based on th
> e
>> shares, deduce the limit of each node in the hierarchy.
>>
>> What do you think?
>>
> As you wrote, a middleware can do controls based on share by limits.
> And it seems much easier to implement it in userland rather than in the kernel
> .

The good thing about user space is that moves unnecessary code outside the
kernel, but the hard thing is standardization. If every middleware is going to
implement what you say, imagine the code duplication, unless we standardize this
into a library component. More comments below. I am not sure about the
difference between user_memory_limit and kernel_memory_limit. Could you please
elaborate.

> 
> Here is an example. (just an example...)
> Please point out if I'm misunderstanding "share".
> 
> root_level/                   = limit 1G.
>           /child_A = share=30
>           /child_B = share=15
>           /child_C = share=5
> (and assume there is no process under root_level for make explanation easy..)
> 
> 0. At first, before starting to use memory, set all kernel_memory_limit.
> root_level.limit = 1G
>   child_A.limit=64M,usage=0
>   child_B.limit=64M,usage=0
>   child_C.limit=64M,usage=0
>   free_resource=808M 
> 

This sounds incorrect, since the limits should be proportional to shares. If the
maximum shares in the root were 100 (*ideally we want higher resolution than that)
Then

child_A.limit = .3 * 1G
child_B.limit = .15 * 1G

and so on


> 1. next, a process in child_C start to run and use memory of 600M.
> root_level.limit = 1G
>   child_A.limit=64M
>   child_B.limit=64M
>   child_C.limit=600M,usage=600M
>   free_resource=272M
> 

How is that feasible, it's limit was 64M, how did it bump up to 600M? If you
want something like that, child_C should have no limits.

> 2. now, a process in child_A start tu run and use memory of 800M.
>   child_A.limit=800M,usage=800M
>   child_B.limit=64M,usage=0M
>   child_C.limit=136M,usage=136M
>   free_resouce=0,A:C=6:1
> 

Not sure I understand this step

> 3.Finally, a process in child_B start. and use memory of 500M.
>   child_A.limit=600M,usage=600M
>   child_B.limit=300M,usage=300M
>   child_C.limit=100M,usage=100M
>   free_resouce=0, A:B:C=6:3:1
> 

Not sure I understand this step

> 4. one more, a process in A exits.
>   child_A.limit=64M, usage=0M
>   child_B.limit=500M, usage=500M
>   child_C.limit=436M, usage=436M
>   free_resouce=0, B:C=3:1 (but B just want to use 500M)
> 

Not sure I understand this step

> This is only an example and the middleware can more pricise "limit"
> contols by checking statistics of memory controller hierarchy based on
> their own policy.
> 
> What I think now is what kind of statistics/notifier/controls are
> necessary to implement shares in middleware. How pricise/quick work the
> middleware can do is based on interfaces.
> Maybe the middleware should know "how fast the application runs now" by
> some kind of check or co-operative interface with the application.
> But I'm not sure how the kernel can help it.

I am not sure if I understand your proposal at this point.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

----- Original Message -----
>kamezawa.hiroyu@jp.fujitsu.com wrote:
>> ----- Original Message -----
>> 
>>>> One more problem is that it's hard to implement various kinds of hierarch
y
>>>> policy. I believe there are other hierarhcy policies rather than OpenVZ
>>>> want to use. Kicking out functions to middleware AMAP is what I'm thinkin
g
>>>> now.
>>> One way to manage hierarchies other than via limits is to use shares (plea
se 
>> see
>>> the shares used by the cpu controller). Basically, what you've done with l
imi
>> ts
>>> is done with shares
>>>
>> Yes, I like _share_ rather than limits.
>> 
>>> If a parent has 100 shares, then it can decide how many to pass on to it's
  c
>> hildren
>>> based on the shares of the child and your logic would work well. I propose
>>> assigning top level (high resolution) shares to the root of the cgroup and
 in
>>  a
>>> hierarchy passing them down to children and sharing it with them. Based on
 th
>> e
>>> shares, deduce the limit of each node in the hierarchy.
>>>
>>> What do you think?
>>>
>> As you wrote, a middleware can do controls based on share by limits.
>> And it seems much easier to implement it in userland rather than in the ker
nel
>> .
>
>The good thing about user space is that moves unnecessary code outside the
>kernel, but the hard thing is standardization. If every middleware is going t
o
>implement what you say, imagine the code duplication, unless we standardize t
his
>into a library component.

It's not problem. We're not developing world-wide eco system.
It's good that there are several development groups. It's a way to evolution.
Something popular will be defacto standard. 
What we have to do is providing proper interfaces for allowing fair race.

>> 
>> Here is an example. (just an example...)
>> Please point out if I'm misunderstanding "share".
>> 
>> root_level/                   = limit 1G.
>>           /child_A = share=30
>>           /child_B = share=15
>>           /child_C = share=5
>> (and assume there is no process under root_level for make explanation easy.
.)
>> 
>> 0. At first, before starting to use memory, set all kernel_memory_limit.
>> root_level.limit = 1G
>>   child_A.limit=64M,usage=0
>>   child_B.limit=64M,usage=0
>>   child_C.limit=64M,usage=0
>>   free_resource=808M 
>> 
>
>This sounds incorrect, since the limits should be proportional to shares. If 
the
>maximum shares in the root were 100 (*ideally we want higher resolution than 
that)
>Then
>
>child_A.limit = .3 * 1G
>child_B.limit = .15 * 1G
>
>and so on
>
Above just showing param to the kernel. 
>From user's view, memory limitation is A:B:C=6:3:1 if memory is fully used.
(In above case, usage=0)

In general, "share" works only when the total usage reaches limitation.
(See how cpu scheduler works.)
When the usage doesn't reach limit, there is no limitatiuon.

>
>> 1. next, a process in child_C start to run and use memory of 600M.
>> root_level.limit = 1G
>>   child_A.limit=64M
>>   child_B.limit=64M
>>   child_C.limit=600M,usage=600M
>>   free_resource=272M
>> 
>
>How is that feasible, it's limit was 64M, how did it bump up to 600M? If you
>want something like that, child_C should have no limits.

middleware just do when child_C.failcnt hits.
echo 64M > childC.memory.limits_in_bytes.
and periodically checks A,B,C and allow C to use what it wants becasue
A and B doesn't want memory.

>
>> 2. now, a process in child_A start tu run and use memory of 800M.
>>   child_A.limit=800M,usage=800M
>>   child_B.limit=64M,usage=0M
>>   child_C.limit=136M,usage=136M
>>   free_resouce=0,A:C=6:1
>> 
>
>Not sure I understand this step
>
Middleware notices that usage in A is growing and moves resources to A.

echo current child_C's limit - 64M > child_C
echo current child_A's limit + 64M > child_A
do above in step by step with loops for making A:C = 6:1
(64M is just an example)

>> 3.Finally, a process in child_B start. and use memory of 500M.
>>   child_A.limit=600M,usage=600M
>>   child_B.limit=300M,usage=300M
>>   child_C.limit=100M,usage=100M
>>   free_resouce=0, A:B:C=6:3:1
>> 
>
>Not sure I understand this step
>
echo current child_C's limit - 64M > child_C
echo current child_A's limit - 64M > child_A
echo current child_B's limit + 64M > child_B
do above in step by step with loops for making A:B:C = 6:3:1


>> 4. one more, a process in A exits.
>>   child_A.limit=64M, usage=0M
>>   child_B.limit=500M, usage=500M
>>   child_C.limit=436M, usage=436M
>>   free_resouce=0, B:C=3:1 (but B just want to use 500M)
>> 
>
>Not sure I understand this step
>
middleware can notice memory pressure from Child_A is reduced.

echo current child_A's limit - 64M > child_A
echo current child_C's limit + 64M > child_C
echo current child_B's limit + 64M > child_B
do above in step by step with loops for making B:C = 3:1 with avoiding
the waste of resources.



>> This is only an example and the middleware can more pricise "limit"
>> contols by checking statistics of memory controller hierarchy based on
>> their own policy.
>> 
>> What I think now is what kind of statistics/notifier/controls are
>> necessary to implement shares in middleware. How pricise/quick work the
>> middleware can do is based on interfaces.
>> Maybe the middleware should know "how fast the application runs now" by
>> some kind of check or co-operative interface with the application.
>> But I'm not sure how the kernel can help it.
>
>I am not sure if I understand your proposal at this point.
>

The most important point is cgoups.memory.memory.limit_in_bytes
is _just_ a notification to ask the kernel to limit the memory
usage of process groups temporally. It changes often.
Based on user's notification to the middleware (share or limit),
the middleware changes limit_in_bytes to be suitable value
and change it dynamically and periodically. 

Thanks,
-Kame

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

----- Original Message -----
>kamezawa.hiroyu@jp.fujitsu.com wrote:
>> ----- Original Message -----
>> 
>>>> One more problem is that it's hard to implement various kinds of hierarch
y
>>>> policy. I believe there are other hierarhcy policies rather than OpenVZ
>>>> want to use. Kicking out functions to middleware AMAP is what I'm thinkin
g
>>>> now.
>>> One way to manage hierarchies other than via limits is to use shares (plea
se 
>> see
>>> the shares used by the cpu controller). Basically, what you've done with l
imi
>> ts
>>> is done with shares
>>>
>> Yes, I like _share_ rather than limits.
>> 
>>> If a parent has 100 shares, then it can decide how many to pass on to it's
  c
>> hildren
>>> based on the shares of the child and your logic would work well. I propose
>>> assigning top level (high resolution) shares to the root of the cgroup and
 in
>>  a
>>> hierarchy passing them down to children and sharing it with them. Based on
 th
>> e
>>> shares, deduce the limit of each node in the hierarchy.
>>>
>>> What do you think?
>>>
>> As you wrote, a middleware can do controls based on share by limits.
>> And it seems much easier to implement it in userland rather than in the ker
nel
>> .
>
>The good thing about user space is that moves unnecessary code outside the
>kernel, but the hard thing is standardization. If every middleware is going t
o
>implement what you say, imagine the code duplication, unless we standardize t
his
>into a library component.

It's not problem. We're not developing world-wide eco system.
It's good that there are several development groups. It's a way to evolution.
Something popular will be defacto standard. 
What we have to do is providing proper interfaces for allowing fair race.

>> 
>> Here is an example. (just an example...)
>> Please point out if I'm misunderstanding "share".
>> 
>> root_level/                   = limit 1G.
>>           /child_A = share=30
>>           /child_B = share=15
>>           /child_C = share=5
>> (and assume there is no process under root_level for make explanation easy.
.)
>> 
>> 0. At first, before starting to use memory, set all kernel_memory_limit.
>> root_level.limit = 1G
>>   child_A.limit=64M,usage=0
>>   child_B.limit=64M,usage=0
>>   child_C.limit=64M,usage=0
>>   free_resource=808M 
>> 
>
>This sounds incorrect, since the limits should be proportional to shares. If 
the
>maximum shares in the root were 100 (*ideally we want higher resolution than 
that)
>Then
>
>child_A.limit = .3 * 1G
>child_B.limit = .15 * 1G
>
>and so on
>
Above just showing param to the kernel. 
>From user's view, memory limitation is A:B:C=6:3:1 if memory is fully used.
(In above case, usage=0)

In general, "share" works only when the total usage reaches limitation.
(See how cpu scheduler works.)
When the usage doesn't reach limit, there is no limitatiuon.

>
>> 1. next, a process in child_C start to run and use memory of 600M.
>> root_level.limit = 1G
>>   child_A.limit=64M
>>   child_B.limit=64M
>>   child_C.limit=600M,usage=600M
>>   free_resource=272M
>> 
>
>How is that feasible, it's limit was 64M, how did it bump up to 600M? If you
>want something like that, child_C should have no limits.

middleware just do when child_C.failcnt hits.
echo 64M > childC.memory.limits_in_bytes.
and periodically checks A,B,C and allow C to use what it wants becasue
A and B doesn't want memory.

>
>> 2. now, a process in child_A start tu run and use memory of 800M.
>>   child_A.limit=800M,usage=800M
>>   child_B.limit=64M,usage=0M
>>   child_C.limit=136M,usage=136M
>>   free_resouce=0,A:C=6:1
>> 
>
>Not sure I understand this step
>
Middleware notices that usage in A is growing and moves resources to A.

echo current child_C's limit - 64M > child_C
echo current child_A's limit + 64M > child_A
do above in step by step with loops for making A:C = 6:1
(64M is just an example)

>> 3.Finally, a process in child_B start. and use memory of 500M.
>>   child_A.limit=600M,usage=600M
>>   child_B.limit=300M,usage=300M
>>   child_C.limit=100M,usage=100M
>>   free_resouce=0, A:B:C=6:3:1
>> 
>
>Not sure I understand this step
>
echo current child_C's limit - 64M > child_C
echo current child_A's limit - 64M > child_A
echo current child_B's limit + 64M > child_B
do above in step by step with loops for making A:B:C = 6:3:1


>> 4. one more, a process in A exits.
>>   child_A.limit=64M, usage=0M
>>   child_B.limit=500M, usage=500M
>>   child_C.limit=436M, usage=436M
>>   free_resouce=0, B:C=3:1 (but B just want to use 500M)
>> 
>
>Not sure I understand this step
>
middleware can notice memory pressure from Child_A is reduced.

echo current child_A's limit - 64M > child_A
echo current child_C's limit + 64M > child_C
echo current child_B's limit + 64M > child_B
do above in step by step with loops for making B:C = 3:1 with avoiding
the waste of resources.



>> This is only an example and the middleware can more pricise "limit"
>> contols by checking statistics of memory controller hierarchy based on
>> their own policy.
>> 
>> What I think now is what kind of statistics/notifier/controls are
>> necessary to implement shares in middleware. How pricise/quick work the
>> middleware can do is based on interfaces.
>> Maybe the middleware should know "how fast the application runs now" by
>> some kind of check or co-operative interface with the application.
>> But I'm not sure how the kernel can help it.
>
>I am not sure if I understand your proposal at this point.
>

The most important point is cgoups.memory.memory.limit_in_bytes
is _just_ a notification to ask the kernel to limit the memory
usage of process groups temporally. It changes often.
Based on user's notification to the middleware (share or limit),
the middleware changes limit_in_bytes to be suitable value
and change it dynamically and periodically. 

Thanks,
-Kame



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

Hi, Kamezawa-san,

kamezawa.hiroyu@jp.fujitsu.com wrote:
> 
> It's not problem. We're not developing world-wide eco system.
> It's good that there are several development groups. It's a way to evolution.
> Something popular will be defacto standard. 
> What we have to do is providing proper interfaces for allowing fair race.
> 

I did not claim that we were developing an eco system either :)
My point is that we should not confuse *Linux* users. Lets do the common/useful
stuff in the kernel and make it easy for users to use the cgroup subsystem.

>>> Here is an example. (just an example...)
>>> Please point out if I'm misunderstanding "share".
>>>
>>> root_level/                   = limit 1G.
>>>           /child_A = share=30
>>>           /child_B = share=15
>>>           /child_C = share=5
>>> (and assume there is no process under root_level for make explanation easy.
> .)
>>> 0. At first, before starting to use memory, set all kernel_memory_limit.
>>> root_level.limit = 1G
>>>   child_A.limit=64M,usage=0
>>>   child_B.limit=64M,usage=0
>>>   child_C.limit=64M,usage=0
>>>   free_resource=808M 
>>>
>> This sounds incorrect, since the limits should be proportional to shares. If 
> the
>> maximum shares in the root were 100 (*ideally we want higher resolution than 
> that)
>> Then
>>
>> child_A.limit = .3 * 1G
>> child_B.limit = .15 * 1G
>>
>> and so on
>>
> Above just showing param to the kernel. 
> From user's view, memory limitation is A:B:C=6:3:1 if memory is fully used.
> (In above case, usage=0)
> 
> In general, "share" works only when the total usage reaches limitation.
> (See how cpu scheduler works.)
> When the usage doesn't reach limit, there is no limitatiuon.
> 

If you are implying that shares imply a soft limit, I agree. But the only
parameter in the kernel right now is hard limits. We need to add soft limit support.

>>> 1. next, a process in child_C start to run and use memory of 600M.
>>> root_level.limit = 1G
>>>   child_A.limit=64M
>>>   child_B.limit=64M
>>>   child_C.limit=600M,usage=600M
>>>   free_resource=272M
>>>
>> How is that feasible, it's limit was 64M, how did it bump up to 600M? If you
>> want something like that, child_C should have no limits.
> 
> middleware just do when child_C.failcnt hits.
> echo 64M > childC.memory.limits_in_bytes.
> and periodically checks A,B,C and allow C to use what it wants becasue
> A and B doesn't want memory.
> 
>>> 2. now, a process in child_A start tu run and use memory of 800M.
>>>   child_A.limit=800M,usage=800M
>>>   child_B.limit=64M,usage=0M
>>>   child_C.limit=136M,usage=136M
>>>   free_resouce=0,A:C=6:1
>>>
>> Not sure I understand this step
>>
> Middleware notices that usage in A is growing and moves resources to A.
> 
> echo current child_C's limit - 64M > child_C
> echo current child_A's limit + 64M > child_A
> do above in step by step with loops for making A:C = 6:1
> (64M is just an example)
> 
>>> 3.Finally, a process in child_B start. and use memory of 500M.
>>>   child_A.limit=600M,usage=600M
>>>   child_B.limit=300M,usage=300M
>>>   child_C.limit=100M,usage=100M
>>>   free_resouce=0, A:B:C=6:3:1
>>>
>> Not sure I understand this step
>>
> echo current child_C's limit - 64M > child_C
> echo current child_A's limit - 64M > child_A
> echo current child_B's limit + 64M > child_B
> do above in step by step with loops for making A:B:C = 6:3:1
> 
> 
>>> 4. one more, a process in A exits.
>>>   child_A.limit=64M, usage=0M
>>>   child_B.limit=500M, usage=500M
>>>   child_C.limit=436M, usage=436M
>>>   free_resouce=0, B:C=3:1 (but B just want to use 500M)
>>>
>> Not sure I understand this step
>>
> middleware can notice memory pressure from Child_A is reduced.
> 
> echo current child_A's limit - 64M > child_A
> echo current child_C's limit + 64M > child_C
> echo current child_B's limit + 64M > child_B
> do above in step by step with loops for making B:C = 3:1 with avoiding
> the waste of resources.
> 
> 
> 
>>> This is only an example and the middleware can more pricise "limit"
>>> contols by checking statistics of memory controller hierarchy based on
>>> their own policy.
>>>
>>> What I think now is what kind of statistics/notifier/controls are
>>> necessary to implement shares in middleware. How pricise/quick work the
>>> middleware can do is based on interfaces.
>>> Maybe the middleware should know "how fast the application runs now" by
>>> some kind of check or co-operative interface with the application.
>>> But I'm not sure how the kernel can help it.
>> I am not sure if I understand your proposal at this point.
>>
> 
> The most important point is cgoups.memory.memory.limit_in_bytes
> is _just_ a notification to ask the kernel to limit the memory
> usage of process groups temporally. It changes often.
> Based on user's notification to the middleware (share or limit),
> the middleware changes limit_in_bytes to be suitable value
> and change it dynamically and periodically. 
> 

Why don't we add soft limits, so that we don't have to go to the kernel and
change limits frequently. One missing piece in the memory controller is that we
don't shrink the memory controller when limits change or when tasks move. I
think soft limits is a better solution.

Thanks for patiently explaining all of this.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

Hi, Kamezawa-san,

kamezawa.hiroyu@jp.fujitsu.com wrote:
> 
> It's not problem. We're not developing world-wide eco system.
> It's good that there are several development groups. It's a way to evolution.
> Something popular will be defacto standard. 
> What we have to do is providing proper interfaces for allowing fair race.
> 

I did not claim that we were developing an eco system either :)
My point is that we should not confuse *Linux* users. Lets do the common/useful
stuff in the kernel and make it easy for users to use the cgroup subsystem.

>>> Here is an example. (just an example...)
>>> Please point out if I'm misunderstanding "share".
>>>
>>> root_level/                   = limit 1G.
>>>           /child_A = share=30
>>>           /child_B = share=15
>>>           /child_C = share=5
>>> (and assume there is no process under root_level for make explanation easy.
> .)
>>> 0. At first, before starting to use memory, set all kernel_memory_limit.
>>> root_level.limit = 1G
>>>   child_A.limit=64M,usage=0
>>>   child_B.limit=64M,usage=0
>>>   child_C.limit=64M,usage=0
>>>   free_resource=808M 
>>>
>> This sounds incorrect, since the limits should be proportional to shares. If 
> the
>> maximum shares in the root were 100 (*ideally we want higher resolution than 
> that)
>> Then
>>
>> child_A.limit = .3 * 1G
>> child_B.limit = .15 * 1G
>>
>> and so on
>>
> Above just showing param to the kernel. 
> From user's view, memory limitation is A:B:C=6:3:1 if memory is fully used.
> (In above case, usage=0)
> 
> In general, "share" works only when the total usage reaches limitation.
> (See how cpu scheduler works.)
> When the usage doesn't reach limit, there is no limitatiuon.
> 

If you are implying that shares imply a soft limit, I agree. But the only
parameter in the kernel right now is hard limits. We need to add soft limit support.

>>> 1. next, a process in child_C start to run and use memory of 600M.
>>> root_level.limit = 1G
>>>   child_A.limit=64M
>>>   child_B.limit=64M
>>>   child_C.limit=600M,usage=600M
>>>   free_resource=272M
>>>
>> How is that feasible, it's limit was 64M, how did it bump up to 600M? If you
>> want something like that, child_C should have no limits.
> 
> middleware just do when child_C.failcnt hits.
> echo 64M > childC.memory.limits_in_bytes.
> and periodically checks A,B,C and allow C to use what it wants becasue
> A and B doesn't want memory.
> 
>>> 2. now, a process in child_A start tu run and use memory of 800M.
>>>   child_A.limit=800M,usage=800M
>>>   child_B.limit=64M,usage=0M
>>>   child_C.limit=136M,usage=136M
>>>   free_resouce=0,A:C=6:1
>>>
>> Not sure I understand this step
>>
> Middleware notices that usage in A is growing and moves resources to A.
> 
> echo current child_C's limit - 64M > child_C
> echo current child_A's limit + 64M > child_A
> do above in step by step with loops for making A:C = 6:1
> (64M is just an example)
> 
>>> 3.Finally, a process in child_B start. and use memory of 500M.
>>>   child_A.limit=600M,usage=600M
>>>   child_B.limit=300M,usage=300M
>>>   child_C.limit=100M,usage=100M
>>>   free_resouce=0, A:B:C=6:3:1
>>>
>> Not sure I understand this step
>>
> echo current child_C's limit - 64M > child_C
> echo current child_A's limit - 64M > child_A
> echo current child_B's limit + 64M > child_B
> do above in step by step with loops for making A:B:C = 6:3:1
> 
> 
>>> 4. one more, a process in A exits.
>>>   child_A.limit=64M, usage=0M
>>>   child_B.limit=500M, usage=500M
>>>   child_C.limit=436M, usage=436M
>>>   free_resouce=0, B:C=3:1 (but B just want to use 500M)
>>>
>> Not sure I understand this step
>>
> middleware can notice memory pressure from Child_A is reduced.
> 
> echo current child_A's limit - 64M > child_A
> echo current child_C's limit + 64M > child_C
> echo current child_B's limit + 64M > child_B
> do above in step by step with loops for making B:C = 3:1 with avoiding
> the waste of resources.
> 
> 
> 
>>> This is only an example and the middleware can more pricise "limit"
>>> contols by checking statistics of memory controller hierarchy based on
>>> their own policy.
>>>
>>> What I think now is what kind of statistics/notifier/controls are
>>> necessary to implement shares in middleware. How pricise/quick work the
>>> middleware can do is based on interfaces.
>>> Maybe the middleware should know "how fast the application runs now" by
>>> some kind of check or co-operative interface with the application.
>>> But I'm not sure how the kernel can help it.
>> I am not sure if I understand your proposal at this point.
>>
> 
> The most important point is cgoups.memory.memory.limit_in_bytes
> is _just_ a notification to ask the kernel to limit the memory
> usage of process groups temporally. It changes often.
> Based on user's notification to the middleware (share or limit),
> the middleware changes limit_in_bytes to be suitable value
> and change it dynamically and periodically. 
> 

Why don't we add soft limits, so that we don't have to go to the kernel and
change limits frequently. One missing piece in the memory controller is that we
don't shrink the memory controller when limits change or when tasks move. I
think soft limits is a better solution.

Thanks for patiently explaining all of this.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

>Why don't we add soft limits, so that we don't have to go to the kernel and
>change limits frequently. One missing piece in the memory controller is that 
we
>don't shrink the memory controller when limits change or when tasks move. I
>think soft limits is a better solution.
>
My code adds shirinking_at_limit_change. I'm now try to write migrate_resouces
_at_task_move. (But seems not so easy to be implemented in 
clean/fast way.)

I have no objection to soft-limit if it's easy to be implemented. (I wrote
my explanation was just an example and we could add more knobs.) 
_But_ I think that something to control multiple cgroups with regard to hierar
chy under some policy never be a simple one. Adding some knobs for each cgroup
s to do soft-limit will be simple one if no hirerachy.

Memory controller's  difference from scheduler's hirerachy is that we have to 
do multilevel page reclaim with feedback under some policy (not only one..). 
Even without hierarhcy, we _did_ make the kernel's LRU logic more complicated.
But we can get a help from the middleware here, I think.

My goal is never to make cgroup slow or complicated. If it's slow, 
I'd like to say "ok, please use VMware.It's simpler and enough fast for you." 
"How fast it works rather than Hardware-Virtualization" is the most
important for me. It should be much more faster.

>Thanks for patiently explaining all of this.
>
Thanks, I'm sorry for my poor explanation skill.

Regards,
-Kame

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

>Why don't we add soft limits, so that we don't have to go to the kernel and
>change limits frequently. One missing piece in the memory controller is that 
we
>don't shrink the memory controller when limits change or when tasks move. I
>think soft limits is a better solution.
>
My code adds shirinking_at_limit_change. I'm now try to write migrate_resouces
_at_task_move. (But seems not so easy to be implemented in 
clean/fast way.)

I have no objection to soft-limit if it's easy to be implemented. (I wrote
my explanation was just an example and we could add more knobs.) 
_But_ I think that something to control multiple cgroups with regard to hierar
chy under some policy never be a simple one. Adding some knobs for each cgroup
s to do soft-limit will be simple one if no hirerachy.

Memory controller's  difference from scheduler's hirerachy is that we have to 
do multilevel page reclaim with feedback under some policy (not only one..). 
Even without hierarhcy, we _did_ make the kernel's LRU logic more complicated.
But we can get a help from the middleware here, I think.

My goal is never to make cgroup slow or complicated. If it's slow, 
I'd like to say "ok, please use VMware.It's simpler and enough fast for you." 
"How fast it works rather than Hardware-Virtualization" is the most
important for me. It should be much more faster.

>Thanks for patiently explaining all of this.
>
Thanks, I'm sorry for my poor explanation skill.

Regards,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[YAMAMOTO Takashi]

> @@ -135,13 +138,118 @@ ssize_t res_counter_write(struct res_cou
>  		if (*end != '\0')
>  			goto out_free;
>  	}
> -	spin_lock_irqsave(&counter->lock, flags);
> -	val = res_counter_member(counter, member);
> -	*val = tmp;
> -	spin_unlock_irqrestore(&counter->lock, flags);
> -	ret = nbytes;
> +	if (member != RES_LIMIT || !callback) {

is there any reason to check member != RES_LIMIT here,
rather than in callers?

> +/*
> + * Move resource to its parent.
> + *   child->limit -= val.
> + *   parent->usage -= val.
> + *   parent->limit -= val.

s/limit/for_children/

> + */
> +
> +int res_counter_repay_resource(struct res_counter *child,
> +				struct res_counter *parent,
> +				unsigned long long val,
> +				res_shrink_callback_t callback, int retry)

can you reduce gratuitous differences between
res_counter_borrow_resource and res_counter_repay_resource?
eg. 'success' vs 'done', how to decrement 'retry'.

YAMAMOTO Takashi

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[YAMAMOTO Takashi]

> @@ -135,13 +138,118 @@ ssize_t res_counter_write(struct res_cou
>  		if (*end != '\0')
>  			goto out_free;
>  	}
> -	spin_lock_irqsave(&counter->lock, flags);
> -	val = res_counter_member(counter, member);
> -	*val = tmp;
> -	spin_unlock_irqrestore(&counter->lock, flags);
> -	ret = nbytes;
> +	if (member != RES_LIMIT || !callback) {

is there any reason to check member != RES_LIMIT here,
rather than in callers?

> +/*
> + * Move resource to its parent.
> + *   child->limit -= val.
> + *   parent->usage -= val.
> + *   parent->limit -= val.

s/limit/for_children/

> + */
> +
> +int res_counter_repay_resource(struct res_counter *child,
> +				struct res_counter *parent,
> +				unsigned long long val,
> +				res_shrink_callback_t callback, int retry)

can you reduce gratuitous differences between
res_counter_borrow_resource and res_counter_repay_resource?
eg. 'success' vs 'done', how to decrement 'retry'.

YAMAMOTO Takashi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

----- Original Message -----

>> @@ -135,13 +138,118 @@ ssize_t res_counter_write(struct res_cou
>>  		if (*end != '\0')
>>  			goto out_free;
>>  	}
>> -	spin_lock_irqsave(&counter->lock, flags);
>> -	val = res_counter_member(counter, member);
>> -	*val = tmp;
>> -	spin_unlock_irqrestore(&counter->lock, flags);
>> -	ret = nbytes;
>> +	if (member != RES_LIMIT || !callback) {
>
>is there any reason to check member != RES_LIMIT here,
>rather than in callers?

Hmm...ok. This is messy. I'll rearrange this.


>
>> +/*
>> + * Move resource to its parent.
>> + *   child->limit -= val.
>> + *   parent->usage -= val.
>> + *   parent->limit -= val.
>
>s/limit/for_children/
>
>> + */
>> +
>> +int res_counter_repay_resource(struct res_counter *child,
>> +				struct res_counter *parent,
>> +				unsigned long long val,
>> +				res_shrink_callback_t callback, int retry)
>
>can you reduce gratuitous differences between
>res_counter_borrow_resource and res_counter_repay_resource?
>eg. 'success' vs 'done', how to decrement 'retry'.
>

Ah, sorry. I'll rewrite.
I'll make next version's quality better.

Thanks,
-Kame

Re: Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[kamezawa.hiroyu]

----- Original Message -----

>> @@ -135,13 +138,118 @@ ssize_t res_counter_write(struct res_cou
>>  		if (*end != '\0')
>>  			goto out_free;
>>  	}
>> -	spin_lock_irqsave(&counter->lock, flags);
>> -	val = res_counter_member(counter, member);
>> -	*val = tmp;
>> -	spin_unlock_irqrestore(&counter->lock, flags);
>> -	ret = nbytes;
>> +	if (member != RES_LIMIT || !callback) {
>
>is there any reason to check member != RES_LIMIT here,
>rather than in callers?

Hmm...ok. This is messy. I'll rearrange this.


>
>> +/*
>> + * Move resource to its parent.
>> + *   child->limit -= val.
>> + *   parent->usage -= val.
>> + *   parent->limit -= val.
>
>s/limit/for_children/
>
>> + */
>> +
>> +int res_counter_repay_resource(struct res_counter *child,
>> +				struct res_counter *parent,
>> +				unsigned long long val,
>> +				res_shrink_callback_t callback, int retry)
>
>can you reduce gratuitous differences between
>res_counter_borrow_resource and res_counter_repay_resource?
>eg. 'success' vs 'done', how to decrement 'retry'.
>

Ah, sorry. I'll rewrite.
I'll make next version's quality better.

Thanks,
-Kame



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 2/2] memcg: memcg hierarchy

[KAMEZAWA Hiroyuki]

hierarchy support for memcg.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

---
 mm/memcontrol.c |   90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 89 insertions(+), 1 deletion(-)

Index: hie-2.6.26-rc2-mm1/mm/memcontrol.c
===================================================================
--- hie-2.6.26-rc2-mm1.orig/mm/memcontrol.c
+++ hie-2.6.26-rc2-mm1/mm/memcontrol.c
@@ -792,6 +792,78 @@ int mem_cgroup_shrink_usage(struct mm_st
 }
 
 /*
+ * Memory Controller hierarchy support.
+ */
+
+int memcg_shrink_callback(struct res_counter *cnt, unsigned long long val)
+{
+	struct mem_cgroup *memcg = container_of(cnt, struct mem_cgroup, res);
+	unsigned long flags;
+	int ret = 1;
+	int progress = 1;
+
+retry:
+	spin_lock_irqsave(&cnt->lock, flags);
+	/* Need to shrink ? */
+	if (cnt->usage + val <= cnt->limit)
+		ret = 0;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+
+	if (!ret)
+		return 0;
+
+	if (!progress)
+		return 1;
+	progress = try_to_free_mem_cgroup_pages(memcg, GFP_KERNEL);
+
+	goto retry;
+}
+
+
+int mem_cgroup_resize_callback(struct res_counter *cnt, unsigned long long val)
+{
+	struct mem_cgroup *child = container_of(cnt, struct mem_cgroup, res);
+	struct mem_cgroup *parent;
+	struct cgroup *my_cg;
+	unsigned long flags, borrow;
+	unsigned long long diffs;
+	int ret = 0;
+
+	my_cg = child->css.cgroup;
+	/* Is this root group ? */
+	if (!my_cg->parent) {
+		spin_lock_irqsave(&cnt->lock, flags);
+		cnt->limit = val;
+		spin_unlock_irqrestore(&cnt->lock, flags);
+		return 0;
+	}
+	spin_lock_irqsave(&cnt->lock, flags);
+	if (val > cnt->limit) {
+		diffs = val - cnt->limit;
+		borrow = 1;
+	} else {
+		diffs = cnt->limit - val;
+		borrow = 0;
+	}
+	spin_unlock_irqrestore(&cnt->lock, flags);
+
+	parent = mem_cgroup_from_cont(my_cg->parent);
+	/* When we increase resource, call borrow. When decrease, call repay*/
+	if (borrow)
+		ret = res_counter_borrow_resource(cnt, &parent->res, diffs,
+					memcg_shrink_callback, 5);
+	else
+		ret = res_counter_repay_resource(cnt, &parent->res, diffs,
+					memcg_shrink_callback, 5);
+	return ret;
+}
+
+
+
+
+
+
+/*
  * This routine traverse page_cgroup in given list and drop them all.
  * *And* this routine doesn't reclaim page itself, just removes page_cgroup.
  */
@@ -898,7 +970,8 @@ static ssize_t mem_cgroup_write(struct c
 {
 	return res_counter_write(&mem_cgroup_from_cont(cont)->res,
 				cft->private, userbuf, nbytes, ppos,
-				mem_cgroup_write_strategy);
+				mem_cgroup_write_strategy,
+				mem_cgroup_resize_callback);
 }
 
 static int mem_cgroup_reset(struct cgroup *cont, unsigned int event)
@@ -992,6 +1065,11 @@ static struct cftype mem_cgroup_files[] 
 		.name = "stat",
 		.read_map = mem_control_stat_show,
 	},
+	{
+		.name = "assigned_to_child",
+		.private = RES_FOR_CHILDREN,
+		.read_u64 = mem_cgroup_read,
+	},
 };
 
 static int alloc_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
@@ -1069,6 +1147,8 @@ mem_cgroup_create(struct cgroup_subsys *
 	}
 
 	res_counter_init(&mem->res);
+	if (cont->parent)
+		res_counter_zero_limit(&mem->res);
 
 	for_each_node_state(node, N_POSSIBLE)
 		if (alloc_mem_cgroup_per_zone_info(mem, node))
@@ -1095,6 +1175,14 @@ static void mem_cgroup_destroy(struct cg
 {
 	int node;
 	struct mem_cgroup *mem = mem_cgroup_from_cont(cont);
+	struct mem_cgroup *parent;
+
+	if (cont->parent) {
+		parent = mem_cgroup_from_cont(cont->parent);
+		/* we did what we can...just returns what we borrow */
+		res_counter_repay_resource(&mem->res,
+				&parent->res, -1, NULL, 0);
+	}
 
 	for_each_node_state(node, N_POSSIBLE)
 		free_mem_cgroup_per_zone_info(mem, node);

[RFC][PATCH 2/2] memcg: memcg hierarchy

[KAMEZAWA Hiroyuki]

hierarchy support for memcg.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

---
 mm/memcontrol.c |   90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 89 insertions(+), 1 deletion(-)

Index: hie-2.6.26-rc2-mm1/mm/memcontrol.c
===================================================================
--- hie-2.6.26-rc2-mm1.orig/mm/memcontrol.c
+++ hie-2.6.26-rc2-mm1/mm/memcontrol.c
@@ -792,6 +792,78 @@ int mem_cgroup_shrink_usage(struct mm_st
 }
 
 /*
+ * Memory Controller hierarchy support.
+ */
+
+int memcg_shrink_callback(struct res_counter *cnt, unsigned long long val)
+{
+	struct mem_cgroup *memcg = container_of(cnt, struct mem_cgroup, res);
+	unsigned long flags;
+	int ret = 1;
+	int progress = 1;
+
+retry:
+	spin_lock_irqsave(&cnt->lock, flags);
+	/* Need to shrink ? */
+	if (cnt->usage + val <= cnt->limit)
+		ret = 0;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+
+	if (!ret)
+		return 0;
+
+	if (!progress)
+		return 1;
+	progress = try_to_free_mem_cgroup_pages(memcg, GFP_KERNEL);
+
+	goto retry;
+}
+
+
+int mem_cgroup_resize_callback(struct res_counter *cnt, unsigned long long val)
+{
+	struct mem_cgroup *child = container_of(cnt, struct mem_cgroup, res);
+	struct mem_cgroup *parent;
+	struct cgroup *my_cg;
+	unsigned long flags, borrow;
+	unsigned long long diffs;
+	int ret = 0;
+
+	my_cg = child->css.cgroup;
+	/* Is this root group ? */
+	if (!my_cg->parent) {
+		spin_lock_irqsave(&cnt->lock, flags);
+		cnt->limit = val;
+		spin_unlock_irqrestore(&cnt->lock, flags);
+		return 0;
+	}
+	spin_lock_irqsave(&cnt->lock, flags);
+	if (val > cnt->limit) {
+		diffs = val - cnt->limit;
+		borrow = 1;
+	} else {
+		diffs = cnt->limit - val;
+		borrow = 0;
+	}
+	spin_unlock_irqrestore(&cnt->lock, flags);
+
+	parent = mem_cgroup_from_cont(my_cg->parent);
+	/* When we increase resource, call borrow. When decrease, call repay*/
+	if (borrow)
+		ret = res_counter_borrow_resource(cnt, &parent->res, diffs,
+					memcg_shrink_callback, 5);
+	else
+		ret = res_counter_repay_resource(cnt, &parent->res, diffs,
+					memcg_shrink_callback, 5);
+	return ret;
+}
+
+
+
+
+
+
+/*
  * This routine traverse page_cgroup in given list and drop them all.
  * *And* this routine doesn't reclaim page itself, just removes page_cgroup.
  */
@@ -898,7 +970,8 @@ static ssize_t mem_cgroup_write(struct c
 {
 	return res_counter_write(&mem_cgroup_from_cont(cont)->res,
 				cft->private, userbuf, nbytes, ppos,
-				mem_cgroup_write_strategy);
+				mem_cgroup_write_strategy,
+				mem_cgroup_resize_callback);
 }
 
 static int mem_cgroup_reset(struct cgroup *cont, unsigned int event)
@@ -992,6 +1065,11 @@ static struct cftype mem_cgroup_files[] 
 		.name = "stat",
 		.read_map = mem_control_stat_show,
 	},
+	{
+		.name = "assigned_to_child",
+		.private = RES_FOR_CHILDREN,
+		.read_u64 = mem_cgroup_read,
+	},
 };
 
 static int alloc_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
@@ -1069,6 +1147,8 @@ mem_cgroup_create(struct cgroup_subsys *
 	}
 
 	res_counter_init(&mem->res);
+	if (cont->parent)
+		res_counter_zero_limit(&mem->res);
 
 	for_each_node_state(node, N_POSSIBLE)
 		if (alloc_mem_cgroup_per_zone_info(mem, node))
@@ -1095,6 +1175,14 @@ static void mem_cgroup_destroy(struct cg
 {
 	int node;
 	struct mem_cgroup *mem = mem_cgroup_from_cont(cont);
+	struct mem_cgroup *parent;
+
+	if (cont->parent) {
+		parent = mem_cgroup_from_cont(cont->parent);
+		/* we did what we can...just returns what we borrow */
+		res_counter_repay_resource(&mem->res,
+				&parent->res, -1, NULL, 0);
+	}
 
 	for_each_node_state(node, N_POSSIBLE)
 		free_mem_cgroup_per_zone_info(mem, node);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 0/2] memcg: simple hierarchy (v2)

[Rik van Riel]

On Fri, 30 May 2008 10:43:12 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> Implemented Policy:
>   - parent overcommits all children
>      parent->usage = resource used by itself + resource moved to children.
>      Of course, parent->limit > parent->usage. 
>   - when child's limit is set, the resouce moves.
>   - no automatic resource moving between parent <-> child

> Why this is enough ?
>   - A middleware can do various kind of resource balancing only by reseting "limit"
>     in userland.

I like this idea.  The alternative could mean having a page live
on multiple cgroup LRU lists, not just the zone LRU and the one
cgroup LRU, and drastically increasing run time overhead.

Swapping memory in and out is horrendously slow anyway, so the
idea of having a daemon adjust the limits on the fly should work
just fine.

-- 
All rights reversed.

Re: [RFC][PATCH 0/2] memcg: simple hierarchy (v2)

[Rik van Riel]

On Fri, 30 May 2008 10:43:12 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> Implemented Policy:
>   - parent overcommits all children
>      parent->usage = resource used by itself + resource moved to children.
>      Of course, parent->limit > parent->usage. 
>   - when child's limit is set, the resouce moves.
>   - no automatic resource moving between parent <-> child

> Why this is enough ?
>   - A middleware can do various kind of resource balancing only by reseting "limit"
>     in userland.

I like this idea.  The alternative could mean having a page live
on multiple cgroup LRU lists, not just the zone LRU and the one
cgroup LRU, and drastically increasing run time overhead.

Swapping memory in and out is horrendously slow anyway, so the
idea of having a daemon adjust the limits on the fly should work
just fine.

-- 
All rights reversed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFC][PATCH 0/2] memcg: hierarchy support (v3)

[KAMEZAWA Hiroyuki]

Hi, this is third version.

While small changes in codes, the whole _tone_ of code is changed.
I'm not in hurry, any comments are welcome.

based on 2.6.26-rc2-mm1 + memcg patches in -mm queue.

Changes from v2:
 - Named as HardWall policy.
 - rewrote the code to be read easily. changed the name of functions.
 - Added text.
 - supported hierarchy_model parameter.
   Now, no_hierarchy and hardwall_hierarchy is implemented.

HardWall Policy:
  - designed for strict resource isolation under hierarchy.
    Usually, automatic load balancing between cgroup can break the
    users assumption even if it's implemented very well.
  - parent overcommits all children
     parent->usage = resource used by itself + resource moved to children.
     Of course, parent->limit > parent->usage. 
  - when child's limit is set, the resouce moves.
  - no automatic resource moving between parent <-> child

Example)
  1) Assume a cgroup with 1GB limits. (and no tasks belongs to this, now)
     - group_A limit=1G,usage=0M.

  2) create group B, C under A.
     - group A limit=1G, usage=0M
          - group B limit=0M, usage=0M.
          - group C limit=0M, usage=0M.

  3) increase group B's limit to 300M.
     - group A limit=1G, usage=300M.
          - group B limit=300M, usage=0M.
          - group C limit=0M, usage=0M.

  4) increase group C's limit to 500M
     - group A limit=1G, usage=800M.
          - group B limit=300M, usage=0M.
          - group C limit=500M, usage=0M.

  5) reduce group B's limit to 100M
     - group A limit=1G, usage=600M.
          - group B limit=100M, usage=0M.
          - group C limit=500M, usage=0M.


Thanks,
-Kame

[RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

A simple hard-wall hierarhcy support for res_counter.

Changelog v2->v3
 - changed the name and arguments of functions.
 - rewrote to be read easily.
 - named as HardWall hierarchy.

This implements following model
 - A cgroup's tree means hierarchy of resource.
 - All child's resource is moved from its parents.
 - The resource moved to children is charged as parent's usage.
 - The resource moves when child->limit is changed.
 - The sum of resource for children and its own usage is limited by "limit".
 
This implies
 - No dynamic automatic hierarhcy balancing in the kernel.
 - Each resource is isolated completely.
 - The kernel just supports resource-move-at-change-in-limit.
 - The user (middle-ware) is responsible to make hierarhcy balanced well.
   Good balance can be achieved by changing limit from user land.


Background:
 Recently, there are popular resource isolation technique widely used,
 i.e. Hardware-Virtualization. We can do hierarchical resource isolation
 by using cgroup on it. But supporting hierarchy management in croups
 has some advantages of performance, unity and costs of management.

 There are good resource management in other OSs, they support some kind of
 hierarchical resource management. We wonder what kind of hierarchy policy
 is good for Linux. And there is an another point. Hierarchical system can be
 implemented by the kernel and user-land co-operation.  So, there are various
 choices to do in the kernel. Doing all in the kernel or export some proper
 interfaces to the user-land. Middle-wares are tend to be used for management.
 I hope there will be Open Source one.

 At supporting hierarchy in cgroup, several aspects of characteristics of
 policy of hierarchy can be considered. Some needs automatic balancing
 between several groups.

  - fairness    ... how fairness is kept under policy

  - performance ... should be _fast_. multi-level resource balancing tend
                 to use much amount of CPU and can cause soft lockup.

  - predictability ... resource management are usually used for resource
                 isolation. the kernel must not break the isolation and
                 predictability of users against application's progress.

  - flexibility ... some sophisticated dynamic resource balancing with
 		 soft-limit is welcomed when the user doesn't want strict
		 resource isolation or when the user cannot estimate how much
		 they want correctly.

Hard Wall Hierarchy.

 This patch implements a hard-wall model of hierarchy for resources.
 Works well for users who want strict resource isolation.

 This model allows the move of resource only between a parent and its children.
 The resource is moved to a child when it declares the amount of resources to be
 used. (by limit)
 Automatic resource balancing is not supported in this code.  
 (But users can do non-automatic by changing limit dynamically.)

 - fairness    ... good. no resource sharing. works as specified by users.
 - performance ... good. each resources are capsuled to its own level.
 - predictability ... good. resources are completely isolated. balancing only
		occurs at the event of changes in limit.
 - flexibility ... bad. no flexibility and scheduling in the kernel level.
	        need middle-ware's help.

Considerations:
 - This implementation uses "limit" == "current_available_resource".
   This should be revisited when Soft-Limit one is implemented.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

---
 Documentation/controllers/resource_counter.txt |   41 +++++++++
 include/linux/res_counter.h                    |   90 +++++++++++++++++++-
 kernel/res_counter.c                           |  112 +++++++++++++++++++++++--
 3 files changed, 235 insertions(+), 8 deletions(-)

Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
===================================================================
--- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
+++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
@@ -38,6 +38,16 @@ struct res_counter {
 	 * the number of unsuccessful attempts to consume the resource
 	 */
 	unsigned long long failcnt;
+
+	/*
+	 * hierarchy support: the parent of this resource.
+	 */
+	struct res_counter *parent;
+	/*
+	 * the amount of resources assigned to children.
+	 */
+	unsigned long long for_children;
+
 	/*
 	 * the lock to protect all of the above.
 	 * the routines below consider this to be IRQ-safe
@@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
 ssize_t res_counter_read(struct res_counter *counter, int member,
 		const char __user *buf, size_t nbytes, loff_t *pos,
 		int (*read_strategy)(unsigned long long val, char *s));
+
+/*
+ * An interface for setting res_counter's member (ex. limit)
+ * the new parameter is passed by *buf and translated by write_strategy().
+ * Then, it is applied to member under the control of set_strategy().
+ * If write_strategy() and set_strategy() can be NULL. see res_counter.c
+ */
+
 ssize_t res_counter_write(struct res_counter *counter, int member,
-		const char __user *buf, size_t nbytes, loff_t *pos,
-		int (*write_strategy)(char *buf, unsigned long long *val));
+	const char __user *buf, size_t nbytes, loff_t *pos,
+        int (*write_strategy)(char *buf, unsigned long long *val),
+	int (*set_strategy)(struct res_counter *res, unsigned long long val,
+			    int what),
+	);
 
 /*
  * the field descriptors. one for each member of res_counter
@@ -76,15 +97,33 @@ enum {
 	RES_MAX_USAGE,
 	RES_LIMIT,
 	RES_FAILCNT,
+	RES_FOR_CHILDREN,
 };
 
 /*
  * helpers for accounting
  */
 
+/*
+ * initialize res_counter.
+ * @counter : the counter
+ *
+ * initialize res_counter and set default limit to very big value(unlimited)
+ */
+
 void res_counter_init(struct res_counter *counter);
 
 /*
+ * initialize res_counter under hierarchy.
+ * @counter : the counter
+ * @parent : the parent of the counter
+ *
+ * initialize res_counter and set default limit to 0. and set "parent".
+ */
+void res_counter_init_hierarchy(struct res_counter *counter,
+				struct res_counter *parent);
+
+/*
  * charge - try to consume more resource.
  *
  * @counter: the counter
@@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
 	cnt->failcnt = 0;
 	spin_unlock_irqrestore(&cnt->lock, flags);
 }
+
+/**
+ * Move resources from a parent to a child.
+ * At success,
+ *           parent->usage += val.
+ *           parent->for_children += val.
+ *           child->limit += val.
+ *
+ * @child:    an entity to set res->limit. The parent is child->parent.
+ * @val:      the amount of resource to be moved.
+ * @callback: called when the parent's free resource is not enough to be moved.
+ *            this can be NULL if no callback is necessary.
+ * @retry:    limit for the number of trying to callback.
+ *            -1 means infinite loop. At each retry, yield() is called.
+ * Returns 0 at success, !0 at failure.
+ *
+ * The callback returns 0 at success, !0 at failure.
+ *
+ */
+
+int res_counter_move_resource(struct res_counter *child,
+	unsigned long long val,
+        int (*callback)(struct res_counter *res, unsigned long long val),
+	int retry);
+
+
+/**
+ * Return resource to its parent.
+ * At success,
+ *           parent->usage  -= val.
+ *           parent->for_children -= val.
+ *           child->limit -= val.
+ *
+ * @child:   entry to resize. The parent is child->parent.
+ * @val  :   How much does child repay to parent ? -1 means 'all'
+ * @callback: A callback for decreasing resource usage of child before
+ *            returning. If NULL, just deceases child's limit.
+ * @retry:   # of retries at calling callback for freeing resource.
+ *            -1 means infinite loop. At each retry, yield() is called.
+ * Returns 0 at success.
+ */
+
+int res_counter_return_resource(struct res_counter *child,
+	unsigned long long val,
+	int (*callback)(struct res_counter *res, unsigned long long val),
+	int retry);
+
 #endif
Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
===================================================================
--- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
+++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
@@ -44,6 +44,13 @@ to work with it.
  	Protects changes of the above values.
 
 
+ f. struct res_counter *parent
+
+	Parent res_counter under hierarchy.
+
+ g. unsigned long long for_children
+
+	Resources assigned to children. This is included in usage.
 
 2. Basic accounting routines
 
@@ -179,3 +186,37 @@ counter fields. They are recommended to 
     still can help with it).
 
  c. Compile and run :)
+
+
+6. Hierarchy
+ a. No Hierarchy
+   each cgroup can use its own private resource.
+
+ b. Hard-wall Hierarhcy
+   A simple hierarchical tree system for resource isolation.
+   Allows moving resources only between a parent and its children.
+   A parent can move its resource to children and remember the amount to
+   for_children member. A child can get new resource only from its parent.
+   Limit of a child is the amount of resource which is moved from its parent.
+
+   When add "val" to a child,
+	parent->usage += val
+	parent->for_children += val
+	child->limit += val
+   When a child returns its resource
+	parent->usage -= val
+	parent->for_children -= val
+	child->limit -= val.
+
+   This implements resource isolation among each group. This works very well
+   when you want to use strict resource isolation.
+
+   Usage Hint:
+   This seems for static resource assignment but dynamic resource re-assignment
+   can be done by resetting "limit" of groups. When you consider "limit" as
+   the amount of allowed _current_ resource, a sophisticated resource management
+   system based on strict resource isolation can be implemented.
+
+c. Soft-wall Hierarchy
+   TBD.
+
Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
===================================================================
--- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
+++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
@@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
 	counter->limit = (unsigned long long)LLONG_MAX;
 }
 
+void res_counter_init_hierarchy(struct res_counter *counter,
+		struct res_counter *parent)
+{
+	spin_lock_init(&counter->lock);
+	counter->limit = 0;
+	counter->parent = parent;
+}
+
 int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 {
 	if (counter->usage + val > counter->limit) {
@@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
 		return &counter->limit;
 	case RES_FAILCNT:
 		return &counter->failcnt;
+	case RES_FOR_CHILDREN:
+		return &counter->for_children;
 	};
 
 	BUG();
@@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
 
 ssize_t res_counter_write(struct res_counter *counter, int member,
 		const char __user *userbuf, size_t nbytes, loff_t *pos,
-		int (*write_strategy)(char *st_buf, unsigned long long *val))
+		int (*write_strategy)(char *st_buf, unsigned long long *val),
+		int (*set_strategy)(struct res_counter *res,
+			unsigned long long val, int what))
 {
 	int ret;
 	char *buf, *end;
@@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
 		if (*end != '\0')
 			goto out_free;
 	}
-	spin_lock_irqsave(&counter->lock, flags);
-	val = res_counter_member(counter, member);
-	*val = tmp;
-	spin_unlock_irqrestore(&counter->lock, flags);
-	ret = nbytes;
+	if (set_strategy) {
+		ret = set_strategy(res, tmp, member);
+		if (!ret)
+			ret = nbytes;
+	} else {
+		spin_lock_irqsave(&counter->lock, flags);
+		val = res_counter_member(counter, member);
+		*val = tmp;
+		spin_unlock_irqrestore(&counter->lock, flags);
+		ret = nbytes;
+	}
 out_free:
 	kfree(buf);
 out:
 	return ret;
 }
+
+
+int res_counter_move_resource(struct res_counter *child,
+				unsigned long long val,
+	int (*callback)(struct res_counter *res, unsigned long long val),
+	int retry)
+{
+	struct res_counter *parent = child->parent;
+	unsigned long flags;
+
+	BUG_ON(!parent);
+
+	while (1) {
+		spin_lock_irqsave(&parent->lock, flags);
+		if (parent->usage + val < parent->limit) {
+			parent->for_children += val;
+			parent->usage += val;
+			break;
+		}
+		spin_unlock_irqrestore(&parent->lock, flags);
+
+		if (!retry || !callback)
+			goto failed;
+		/* -1 means  infinite loop */
+		if (retry != -1)
+			--retry;
+		yield();
+		callback(parent, val);
+	}
+	spin_unlock_irqrestore(&parent->lock, flags);
+
+	spin_lock_irqsave(&child->lock, flags);
+	child->limit += val;
+	spin_unlock_irqrestore(&child->lock, flags);
+	return 0;
+fail:
+	return 1;
+}
+
+
+int res_counter_return_resource(struct res_counter *child,
+				unsigned long long val,
+	int (*callback)(struct res_counter *res, unsigned long long val),
+	int retry)
+{
+	unsigned long flags;
+	struct res_counter *parent = child->parent;
+
+	BUG_ON(!parent);
+
+	while (1) {
+		spin_lock_irqsave(&child->lock, flags);
+		if (val == (unsigned long long) -1) {
+			val = child->limit;
+			child->limit = 0;
+			break;
+		} else if (child->usage <= child->limit - val) {
+			child->limit -= val;
+			break;
+		}
+		spin_unlock_irqrestore(&child->lock, flags);
+
+		if (!retry)
+			goto fail;
+		/* -1 means infinite loop */
+		if (retry != -1)
+			--retry;
+		yield();
+		callback(parent, val);
+	}
+	spin_unlock_irqrestore(&child->lock, flags);
+
+	spin_lock_irqsave(&parent->lock, flags);
+	BUG_ON(parent->for_children < val);
+	BUG_ON(parent->usage < val);
+	parent->for_children -= val;
+	parent->usage -= val;
+	spin_unlock_irqrestore(&parent->lock, flags);
+	return 0;
+fail:
+	return 1;
+}

[RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

A simple hard-wall hierarhcy support for res_counter.

Changelog v2->v3
 - changed the name and arguments of functions.
 - rewrote to be read easily.
 - named as HardWall hierarchy.

This implements following model
 - A cgroup's tree means hierarchy of resource.
 - All child's resource is moved from its parents.
 - The resource moved to children is charged as parent's usage.
 - The resource moves when child->limit is changed.
 - The sum of resource for children and its own usage is limited by "limit".
 
This implies
 - No dynamic automatic hierarhcy balancing in the kernel.
 - Each resource is isolated completely.
 - The kernel just supports resource-move-at-change-in-limit.
 - The user (middle-ware) is responsible to make hierarhcy balanced well.
   Good balance can be achieved by changing limit from user land.


Background:
 Recently, there are popular resource isolation technique widely used,
 i.e. Hardware-Virtualization. We can do hierarchical resource isolation
 by using cgroup on it. But supporting hierarchy management in croups
 has some advantages of performance, unity and costs of management.

 There are good resource management in other OSs, they support some kind of
 hierarchical resource management. We wonder what kind of hierarchy policy
 is good for Linux. And there is an another point. Hierarchical system can be
 implemented by the kernel and user-land co-operation.  So, there are various
 choices to do in the kernel. Doing all in the kernel or export some proper
 interfaces to the user-land. Middle-wares are tend to be used for management.
 I hope there will be Open Source one.

 At supporting hierarchy in cgroup, several aspects of characteristics of
 policy of hierarchy can be considered. Some needs automatic balancing
 between several groups.

  - fairness    ... how fairness is kept under policy

  - performance ... should be _fast_. multi-level resource balancing tend
                 to use much amount of CPU and can cause soft lockup.

  - predictability ... resource management are usually used for resource
                 isolation. the kernel must not break the isolation and
                 predictability of users against application's progress.

  - flexibility ... some sophisticated dynamic resource balancing with
 		 soft-limit is welcomed when the user doesn't want strict
		 resource isolation or when the user cannot estimate how much
		 they want correctly.

Hard Wall Hierarchy.

 This patch implements a hard-wall model of hierarchy for resources.
 Works well for users who want strict resource isolation.

 This model allows the move of resource only between a parent and its children.
 The resource is moved to a child when it declares the amount of resources to be
 used. (by limit)
 Automatic resource balancing is not supported in this code.  
 (But users can do non-automatic by changing limit dynamically.)

 - fairness    ... good. no resource sharing. works as specified by users.
 - performance ... good. each resources are capsuled to its own level.
 - predictability ... good. resources are completely isolated. balancing only
		occurs at the event of changes in limit.
 - flexibility ... bad. no flexibility and scheduling in the kernel level.
	        need middle-ware's help.

Considerations:
 - This implementation uses "limit" == "current_available_resource".
   This should be revisited when Soft-Limit one is implemented.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

---
 Documentation/controllers/resource_counter.txt |   41 +++++++++
 include/linux/res_counter.h                    |   90 +++++++++++++++++++-
 kernel/res_counter.c                           |  112 +++++++++++++++++++++++--
 3 files changed, 235 insertions(+), 8 deletions(-)

Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
===================================================================
--- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
+++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
@@ -38,6 +38,16 @@ struct res_counter {
 	 * the number of unsuccessful attempts to consume the resource
 	 */
 	unsigned long long failcnt;
+
+	/*
+	 * hierarchy support: the parent of this resource.
+	 */
+	struct res_counter *parent;
+	/*
+	 * the amount of resources assigned to children.
+	 */
+	unsigned long long for_children;
+
 	/*
 	 * the lock to protect all of the above.
 	 * the routines below consider this to be IRQ-safe
@@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
 ssize_t res_counter_read(struct res_counter *counter, int member,
 		const char __user *buf, size_t nbytes, loff_t *pos,
 		int (*read_strategy)(unsigned long long val, char *s));
+
+/*
+ * An interface for setting res_counter's member (ex. limit)
+ * the new parameter is passed by *buf and translated by write_strategy().
+ * Then, it is applied to member under the control of set_strategy().
+ * If write_strategy() and set_strategy() can be NULL. see res_counter.c
+ */
+
 ssize_t res_counter_write(struct res_counter *counter, int member,
-		const char __user *buf, size_t nbytes, loff_t *pos,
-		int (*write_strategy)(char *buf, unsigned long long *val));
+	const char __user *buf, size_t nbytes, loff_t *pos,
+        int (*write_strategy)(char *buf, unsigned long long *val),
+	int (*set_strategy)(struct res_counter *res, unsigned long long val,
+			    int what),
+	);
 
 /*
  * the field descriptors. one for each member of res_counter
@@ -76,15 +97,33 @@ enum {
 	RES_MAX_USAGE,
 	RES_LIMIT,
 	RES_FAILCNT,
+	RES_FOR_CHILDREN,
 };
 
 /*
  * helpers for accounting
  */
 
+/*
+ * initialize res_counter.
+ * @counter : the counter
+ *
+ * initialize res_counter and set default limit to very big value(unlimited)
+ */
+
 void res_counter_init(struct res_counter *counter);
 
 /*
+ * initialize res_counter under hierarchy.
+ * @counter : the counter
+ * @parent : the parent of the counter
+ *
+ * initialize res_counter and set default limit to 0. and set "parent".
+ */
+void res_counter_init_hierarchy(struct res_counter *counter,
+				struct res_counter *parent);
+
+/*
  * charge - try to consume more resource.
  *
  * @counter: the counter
@@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
 	cnt->failcnt = 0;
 	spin_unlock_irqrestore(&cnt->lock, flags);
 }
+
+/**
+ * Move resources from a parent to a child.
+ * At success,
+ *           parent->usage += val.
+ *           parent->for_children += val.
+ *           child->limit += val.
+ *
+ * @child:    an entity to set res->limit. The parent is child->parent.
+ * @val:      the amount of resource to be moved.
+ * @callback: called when the parent's free resource is not enough to be moved.
+ *            this can be NULL if no callback is necessary.
+ * @retry:    limit for the number of trying to callback.
+ *            -1 means infinite loop. At each retry, yield() is called.
+ * Returns 0 at success, !0 at failure.
+ *
+ * The callback returns 0 at success, !0 at failure.
+ *
+ */
+
+int res_counter_move_resource(struct res_counter *child,
+	unsigned long long val,
+        int (*callback)(struct res_counter *res, unsigned long long val),
+	int retry);
+
+
+/**
+ * Return resource to its parent.
+ * At success,
+ *           parent->usage  -= val.
+ *           parent->for_children -= val.
+ *           child->limit -= val.
+ *
+ * @child:   entry to resize. The parent is child->parent.
+ * @val  :   How much does child repay to parent ? -1 means 'all'
+ * @callback: A callback for decreasing resource usage of child before
+ *            returning. If NULL, just deceases child's limit.
+ * @retry:   # of retries at calling callback for freeing resource.
+ *            -1 means infinite loop. At each retry, yield() is called.
+ * Returns 0 at success.
+ */
+
+int res_counter_return_resource(struct res_counter *child,
+	unsigned long long val,
+	int (*callback)(struct res_counter *res, unsigned long long val),
+	int retry);
+
 #endif
Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
===================================================================
--- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
+++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
@@ -44,6 +44,13 @@ to work with it.
  	Protects changes of the above values.
 
 
+ f. struct res_counter *parent
+
+	Parent res_counter under hierarchy.
+
+ g. unsigned long long for_children
+
+	Resources assigned to children. This is included in usage.
 
 2. Basic accounting routines
 
@@ -179,3 +186,37 @@ counter fields. They are recommended to 
     still can help with it).
 
  c. Compile and run :)
+
+
+6. Hierarchy
+ a. No Hierarchy
+   each cgroup can use its own private resource.
+
+ b. Hard-wall Hierarhcy
+   A simple hierarchical tree system for resource isolation.
+   Allows moving resources only between a parent and its children.
+   A parent can move its resource to children and remember the amount to
+   for_children member. A child can get new resource only from its parent.
+   Limit of a child is the amount of resource which is moved from its parent.
+
+   When add "val" to a child,
+	parent->usage += val
+	parent->for_children += val
+	child->limit += val
+   When a child returns its resource
+	parent->usage -= val
+	parent->for_children -= val
+	child->limit -= val.
+
+   This implements resource isolation among each group. This works very well
+   when you want to use strict resource isolation.
+
+   Usage Hint:
+   This seems for static resource assignment but dynamic resource re-assignment
+   can be done by resetting "limit" of groups. When you consider "limit" as
+   the amount of allowed _current_ resource, a sophisticated resource management
+   system based on strict resource isolation can be implemented.
+
+c. Soft-wall Hierarchy
+   TBD.
+
Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
===================================================================
--- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
+++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
@@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
 	counter->limit = (unsigned long long)LLONG_MAX;
 }
 
+void res_counter_init_hierarchy(struct res_counter *counter,
+		struct res_counter *parent)
+{
+	spin_lock_init(&counter->lock);
+	counter->limit = 0;
+	counter->parent = parent;
+}
+
 int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 {
 	if (counter->usage + val > counter->limit) {
@@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
 		return &counter->limit;
 	case RES_FAILCNT:
 		return &counter->failcnt;
+	case RES_FOR_CHILDREN:
+		return &counter->for_children;
 	};
 
 	BUG();
@@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
 
 ssize_t res_counter_write(struct res_counter *counter, int member,
 		const char __user *userbuf, size_t nbytes, loff_t *pos,
-		int (*write_strategy)(char *st_buf, unsigned long long *val))
+		int (*write_strategy)(char *st_buf, unsigned long long *val),
+		int (*set_strategy)(struct res_counter *res,
+			unsigned long long val, int what))
 {
 	int ret;
 	char *buf, *end;
@@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
 		if (*end != '\0')
 			goto out_free;
 	}
-	spin_lock_irqsave(&counter->lock, flags);
-	val = res_counter_member(counter, member);
-	*val = tmp;
-	spin_unlock_irqrestore(&counter->lock, flags);
-	ret = nbytes;
+	if (set_strategy) {
+		ret = set_strategy(res, tmp, member);
+		if (!ret)
+			ret = nbytes;
+	} else {
+		spin_lock_irqsave(&counter->lock, flags);
+		val = res_counter_member(counter, member);
+		*val = tmp;
+		spin_unlock_irqrestore(&counter->lock, flags);
+		ret = nbytes;
+	}
 out_free:
 	kfree(buf);
 out:
 	return ret;
 }
+
+
+int res_counter_move_resource(struct res_counter *child,
+				unsigned long long val,
+	int (*callback)(struct res_counter *res, unsigned long long val),
+	int retry)
+{
+	struct res_counter *parent = child->parent;
+	unsigned long flags;
+
+	BUG_ON(!parent);
+
+	while (1) {
+		spin_lock_irqsave(&parent->lock, flags);
+		if (parent->usage + val < parent->limit) {
+			parent->for_children += val;
+			parent->usage += val;
+			break;
+		}
+		spin_unlock_irqrestore(&parent->lock, flags);
+
+		if (!retry || !callback)
+			goto failed;
+		/* -1 means  infinite loop */
+		if (retry != -1)
+			--retry;
+		yield();
+		callback(parent, val);
+	}
+	spin_unlock_irqrestore(&parent->lock, flags);
+
+	spin_lock_irqsave(&child->lock, flags);
+	child->limit += val;
+	spin_unlock_irqrestore(&child->lock, flags);
+	return 0;
+fail:
+	return 1;
+}
+
+
+int res_counter_return_resource(struct res_counter *child,
+				unsigned long long val,
+	int (*callback)(struct res_counter *res, unsigned long long val),
+	int retry)
+{
+	unsigned long flags;
+	struct res_counter *parent = child->parent;
+
+	BUG_ON(!parent);
+
+	while (1) {
+		spin_lock_irqsave(&child->lock, flags);
+		if (val == (unsigned long long) -1) {
+			val = child->limit;
+			child->limit = 0;
+			break;
+		} else if (child->usage <= child->limit - val) {
+			child->limit -= val;
+			break;
+		}
+		spin_unlock_irqrestore(&child->lock, flags);
+
+		if (!retry)
+			goto fail;
+		/* -1 means infinite loop */
+		if (retry != -1)
+			--retry;
+		yield();
+		callback(parent, val);
+	}
+	spin_unlock_irqrestore(&child->lock, flags);
+
+	spin_lock_irqsave(&parent->lock, flags);
+	BUG_ON(parent->for_children < val);
+	BUG_ON(parent->usage < val);
+	parent->for_children -= val;
+	parent->usage -= val;
+	spin_unlock_irqrestore(&parent->lock, flags);
+	return 0;
+fail:
+	return 1;
+}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Li Zefan]

> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> @@ -44,6 +44,13 @@ to work with it.
>   	Protects changes of the above values.
>  
>  
> + f. struct res_counter *parent
> +
> +	Parent res_counter under hierarchy.
> +
> + g. unsigned long long for_children
> +
> +	Resources assigned to children. This is included in usage.

Since parent and for_children are also protected by res_count->lock,
the above text should appear before 'e. spinlock_t lock'.

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Li Zefan]

> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> @@ -44,6 +44,13 @@ to work with it.
>   	Protects changes of the above values.
>  
>  
> + f. struct res_counter *parent
> +
> +	Parent res_counter under hierarchy.
> +
> + g. unsigned long long for_children
> +
> +	Resources assigned to children. This is included in usage.

Since parent and for_children are also protected by res_count->lock,
the above text should appear before 'e. spinlock_t lock'.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Wed, 04 Jun 2008 14:54:12 +0800
Li Zefan <lizf@cn.fujitsu.com> wrote:

> > Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> > +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > @@ -44,6 +44,13 @@ to work with it.
> >   	Protects changes of the above values.
> >  
> >  
> > + f. struct res_counter *parent
> > +
> > +	Parent res_counter under hierarchy.
> > +
> > + g. unsigned long long for_children
> > +
> > +	Resources assigned to children. This is included in usage.
> 
> Since parent and for_children are also protected by res_count->lock,
> the above text should appear before 'e. spinlock_t lock'.
> 
ok.

Thanks,
-Kame

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Wed, 04 Jun 2008 14:54:12 +0800
Li Zefan <lizf@cn.fujitsu.com> wrote:

> > Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> > +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > @@ -44,6 +44,13 @@ to work with it.
> >   	Protects changes of the above values.
> >  
> >  
> > + f. struct res_counter *parent
> > +
> > +	Parent res_counter under hierarchy.
> > +
> > + g. unsigned long long for_children
> > +
> > +	Resources assigned to children. This is included in usage.
> 
> Since parent and for_children are also protected by res_count->lock,
> the above text should appear before 'e. spinlock_t lock'.
> 
ok.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[YAMAMOTO Takashi]

>  ssize_t res_counter_write(struct res_counter *counter, int member,
> -		const char __user *buf, size_t nbytes, loff_t *pos,
> -		int (*write_strategy)(char *buf, unsigned long long *val));
> +	const char __user *buf, size_t nbytes, loff_t *pos,
> +        int (*write_strategy)(char *buf, unsigned long long *val),
> +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
> +			    int what),

this comma seems surplus.

> +	);

> +int res_counter_return_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{

> +		callback(parent, val);

s/parent/child/

YAMAMOTO Takashi

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[YAMAMOTO Takashi]

>  ssize_t res_counter_write(struct res_counter *counter, int member,
> -		const char __user *buf, size_t nbytes, loff_t *pos,
> -		int (*write_strategy)(char *buf, unsigned long long *val));
> +	const char __user *buf, size_t nbytes, loff_t *pos,
> +        int (*write_strategy)(char *buf, unsigned long long *val),
> +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
> +			    int what),

this comma seems surplus.

> +	);

> +int res_counter_return_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{

> +		callback(parent, val);

s/parent/child/

YAMAMOTO Takashi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Wed,  4 Jun 2008 16:20:48 +0900 (JST)
yamamoto@valinux.co.jp (YAMAMOTO Takashi) wrote:

> >  ssize_t res_counter_write(struct res_counter *counter, int member,
> > -		const char __user *buf, size_t nbytes, loff_t *pos,
> > -		int (*write_strategy)(char *buf, unsigned long long *val));
> > +	const char __user *buf, size_t nbytes, loff_t *pos,
> > +        int (*write_strategy)(char *buf, unsigned long long *val),
> > +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
> > +			    int what),
> 
> this comma seems surplus.
> 
Ouch, I thought I fixed this...maybe patch reflesh trouble. Thanks.


> > +	);
> 
> > +int res_counter_return_resource(struct res_counter *child,
> > +				unsigned long long val,
> > +	int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry)
> > +{
> 
> > +		callback(parent, val);
> 
> s/parent/child/
> 
Hmm..yes. 

Thanks,
-Kame

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Wed,  4 Jun 2008 16:20:48 +0900 (JST)
yamamoto@valinux.co.jp (YAMAMOTO Takashi) wrote:

> >  ssize_t res_counter_write(struct res_counter *counter, int member,
> > -		const char __user *buf, size_t nbytes, loff_t *pos,
> > -		int (*write_strategy)(char *buf, unsigned long long *val));
> > +	const char __user *buf, size_t nbytes, loff_t *pos,
> > +        int (*write_strategy)(char *buf, unsigned long long *val),
> > +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
> > +			    int what),
> 
> this comma seems surplus.
> 
Ouch, I thought I fixed this...maybe patch reflesh trouble. Thanks.


> > +	);
> 
> > +int res_counter_return_resource(struct res_counter *child,
> > +				unsigned long long val,
> > +	int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry)
> > +{
> 
> > +		callback(parent, val);
> 
> s/parent/child/
> 
Hmm..yes. 

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Paul Menage]

On Tue, Jun 3, 2008 at 10:01 PM, KAMEZAWA Hiroyuki
<kamezawa.hiroyu@jp.fujitsu.com> wrote:
>        int ret;
>        char *buf, *end;
> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
>                if (*end != '\0')
>                        goto out_free;
>        }
> -       spin_lock_irqsave(&counter->lock, flags);
> -       val = res_counter_member(counter, member);
> -       *val = tmp;
> -       spin_unlock_irqrestore(&counter->lock, flags);
> -       ret = nbytes;
> +       if (set_strategy) {
> +               ret = set_strategy(res, tmp, member);
> +               if (!ret)
> +                       ret = nbytes;
> +       } else {
> +               spin_lock_irqsave(&counter->lock, flags);
> +               val = res_counter_member(counter, member);
> +               *val = tmp;
> +               spin_unlock_irqrestore(&counter->lock, flags);
> +               ret = nbytes;
> +       }

I think that the hierarchy/reclaim handling that you currently have in
the memory controller should be here; the memory controller should
just be able to pass a reference to try_to_free_mem_cgroup_pages() and
have everything else handled by res_counter.

Paul

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Paul Menage]

On Tue, Jun 3, 2008 at 10:01 PM, KAMEZAWA Hiroyuki
<kamezawa.hiroyu@jp.fujitsu.com> wrote:
>        int ret;
>        char *buf, *end;
> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
>                if (*end != '\0')
>                        goto out_free;
>        }
> -       spin_lock_irqsave(&counter->lock, flags);
> -       val = res_counter_member(counter, member);
> -       *val = tmp;
> -       spin_unlock_irqrestore(&counter->lock, flags);
> -       ret = nbytes;
> +       if (set_strategy) {
> +               ret = set_strategy(res, tmp, member);
> +               if (!ret)
> +                       ret = nbytes;
> +       } else {
> +               spin_lock_irqsave(&counter->lock, flags);
> +               val = res_counter_member(counter, member);
> +               *val = tmp;
> +               spin_unlock_irqrestore(&counter->lock, flags);
> +               ret = nbytes;
> +       }

I think that the hierarchy/reclaim handling that you currently have in
the memory controller should be here; the memory controller should
just be able to pass a reference to try_to_free_mem_cgroup_pages() and
have everything else handled by res_counter.

Paul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Wed, 4 Jun 2008 01:59:31 -0700
"Paul Menage" <menage@google.com> wrote:

> On Tue, Jun 3, 2008 at 10:01 PM, KAMEZAWA Hiroyuki
> <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> >        int ret;
> >        char *buf, *end;
> > @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
> >                if (*end != '\0')
> >                        goto out_free;
> >        }
> > -       spin_lock_irqsave(&counter->lock, flags);
> > -       val = res_counter_member(counter, member);
> > -       *val = tmp;
> > -       spin_unlock_irqrestore(&counter->lock, flags);
> > -       ret = nbytes;
> > +       if (set_strategy) {
> > +               ret = set_strategy(res, tmp, member);
> > +               if (!ret)
> > +                       ret = nbytes;
> > +       } else {
> > +               spin_lock_irqsave(&counter->lock, flags);
> > +               val = res_counter_member(counter, member);
> > +               *val = tmp;
> > +               spin_unlock_irqrestore(&counter->lock, flags);
> > +               ret = nbytes;
> > +       }
> 
> I think that the hierarchy/reclaim handling that you currently have in
> the memory controller should be here; the memory controller should
> just be able to pass a reference to try_to_free_mem_cgroup_pages() and
> have everything else handled by res_counter.
> 
Sounds reasonable. I'll re-design the whole AMAP. I think I can do more.

Thanks,
-Kame

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Wed, 4 Jun 2008 01:59:31 -0700
"Paul Menage" <menage@google.com> wrote:

> On Tue, Jun 3, 2008 at 10:01 PM, KAMEZAWA Hiroyuki
> <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> >        int ret;
> >        char *buf, *end;
> > @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
> >                if (*end != '\0')
> >                        goto out_free;
> >        }
> > -       spin_lock_irqsave(&counter->lock, flags);
> > -       val = res_counter_member(counter, member);
> > -       *val = tmp;
> > -       spin_unlock_irqrestore(&counter->lock, flags);
> > -       ret = nbytes;
> > +       if (set_strategy) {
> > +               ret = set_strategy(res, tmp, member);
> > +               if (!ret)
> > +                       ret = nbytes;
> > +       } else {
> > +               spin_lock_irqsave(&counter->lock, flags);
> > +               val = res_counter_member(counter, member);
> > +               *val = tmp;
> > +               spin_unlock_irqrestore(&counter->lock, flags);
> > +               ret = nbytes;
> > +       }
> 
> I think that the hierarchy/reclaim handling that you currently have in
> the memory controller should be here; the memory controller should
> just be able to pass a reference to try_to_free_mem_cgroup_pages() and
> have everything else handled by res_counter.
> 
Sounds reasonable. I'll re-design the whole AMAP. I think I can do more.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

KAMEZAWA Hiroyuki wrote:
> A simple hard-wall hierarhcy support for res_counter.
> 
> Changelog v2->v3
>  - changed the name and arguments of functions.
>  - rewrote to be read easily.
>  - named as HardWall hierarchy.
> 
> This implements following model
>  - A cgroup's tree means hierarchy of resource.
>  - All child's resource is moved from its parents.
>  - The resource moved to children is charged as parent's usage.
>  - The resource moves when child->limit is changed.
>  - The sum of resource for children and its own usage is limited by "limit".
> 
> This implies
>  - No dynamic automatic hierarhcy balancing in the kernel.
>  - Each resource is isolated completely.
>  - The kernel just supports resource-move-at-change-in-limit.
>  - The user (middle-ware) is responsible to make hierarhcy balanced well.

We'd definitely like to see a user level tool/application as a demo of how this
can be achieved.

>    Good balance can be achieved by changing limit from user land.
> 
> 
> Background:
>  Recently, there are popular resource isolation technique widely used,
>  i.e. Hardware-Virtualization. We can do hierarchical resource isolation
>  by using cgroup on it. But supporting hierarchy management in croups
>  has some advantages of performance, unity and costs of management.
> 
>  There are good resource management in other OSs, they support some kind of
>  hierarchical resource management. We wonder what kind of hierarchy policy
>  is good for Linux. And there is an another point. Hierarchical system can be
>  implemented by the kernel and user-land co-operation.  So, there are various
>  choices to do in the kernel. Doing all in the kernel or export some proper
>  interfaces to the user-land. Middle-wares are tend to be used for management.
>  I hope there will be Open Source one.
> 
>  At supporting hierarchy in cgroup, several aspects of characteristics of
>  policy of hierarchy can be considered. Some needs automatic balancing
>  between several groups.
> 
>   - fairness    ... how fairness is kept under policy
> 
>   - performance ... should be _fast_. multi-level resource balancing tend
>                  to use much amount of CPU and can cause soft lockup.
> 
>   - predictability ... resource management are usually used for resource
>                  isolation. the kernel must not break the isolation and
>                  predictability of users against application's progress.
> 
>   - flexibility ... some sophisticated dynamic resource balancing with
>  		 soft-limit is welcomed when the user doesn't want strict
> 		 resource isolation or when the user cannot estimate how much
> 		 they want correctly.

Soft limits has been on my plate for a while now. I'll take a crack at it. At
the moment the statistics is a bit of a worry, since users/administrators need
good statistics to take further action.

> 
> Hard Wall Hierarchy.
> 
>  This patch implements a hard-wall model of hierarchy for resources.
>  Works well for users who want strict resource isolation.
> 
>  This model allows the move of resource only between a parent and its children.
>  The resource is moved to a child when it declares the amount of resources to be
>  used. (by limit)

The other reason for preferring a shares based approach is that, the it will be
more in line with the CPU controllers interfaces.


>  Automatic resource balancing is not supported in this code.  
>  (But users can do non-automatic by changing limit dynamically.)
> 
>  - fairness    ... good. no resource sharing. works as specified by users.
>  - performance ... good. each resources are capsuled to its own level.
>  - predictability ... good. resources are completely isolated. balancing only
> 		occurs at the event of changes in limit.
>  - flexibility ... bad. no flexibility and scheduling in the kernel level.
> 	        need middle-ware's help.
> 
> Considerations:
>  - This implementation uses "limit" == "current_available_resource".
>    This should be revisited when Soft-Limit one is implemented.
> 
> Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> 
> ---
>  Documentation/controllers/resource_counter.txt |   41 +++++++++
>  include/linux/res_counter.h                    |   90 +++++++++++++++++++-
>  kernel/res_counter.c                           |  112 +++++++++++++++++++++++--
>  3 files changed, 235 insertions(+), 8 deletions(-)
> 
> Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
> +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> @@ -38,6 +38,16 @@ struct res_counter {
>  	 * the number of unsuccessful attempts to consume the resource
>  	 */
>  	unsigned long long failcnt;
> +
> +	/*
> +	 * hierarchy support: the parent of this resource.
> +	 */
> +	struct res_counter *parent;
> +	/*
> +	 * the amount of resources assigned to children.
> +	 */
> +	unsigned long long for_children;
> +

I would prefer to use a better name, lent_out? reserved_for_children?
borrowed_by_children?

>  	/*
>  	 * the lock to protect all of the above.
>  	 * the routines below consider this to be IRQ-safe
> @@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
>  ssize_t res_counter_read(struct res_counter *counter, int member,
>  		const char __user *buf, size_t nbytes, loff_t *pos,
>  		int (*read_strategy)(unsigned long long val, char *s));
> +
> +/*
> + * An interface for setting res_counter's member (ex. limit)
> + * the new parameter is passed by *buf and translated by write_strategy().
> + * Then, it is applied to member under the control of set_strategy().
> + * If write_strategy() and set_strategy() can be NULL. see res_counter.c
> + */
> +
>  ssize_t res_counter_write(struct res_counter *counter, int member,
> -		const char __user *buf, size_t nbytes, loff_t *pos,
> -		int (*write_strategy)(char *buf, unsigned long long *val));
> +	const char __user *buf, size_t nbytes, loff_t *pos,
> +        int (*write_strategy)(char *buf, unsigned long long *val),
> +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
> +			    int what),
> +	);
> 
>  /*
>   * the field descriptors. one for each member of res_counter
> @@ -76,15 +97,33 @@ enum {
>  	RES_MAX_USAGE,
>  	RES_LIMIT,
>  	RES_FAILCNT,
> +	RES_FOR_CHILDREN,

RES_BORROWED? RES_BORROWED_BY_CHILDREN?

>  };
> 
>  /*
>   * helpers for accounting
>   */
> 
> +/*
> + * initialize res_counter.
> + * @counter : the counter
> + *
> + * initialize res_counter and set default limit to very big value(unlimited)
> + */
> +
>  void res_counter_init(struct res_counter *counter);
> 
>  /*
> + * initialize res_counter under hierarchy.
> + * @counter : the counter
> + * @parent : the parent of the counter
> + *
> + * initialize res_counter and set default limit to 0. and set "parent".
> + */
> +void res_counter_init_hierarchy(struct res_counter *counter,
> +				struct res_counter *parent);
> +
> +/*
>   * charge - try to consume more resource.
>   *
>   * @counter: the counter
> @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
>  	cnt->failcnt = 0;
>  	spin_unlock_irqrestore(&cnt->lock, flags);
>  }
> +
> +/**
> + * Move resources from a parent to a child.
> + * At success,
> + *           parent->usage += val.
> + *           parent->for_children += val.
> + *           child->limit += val.
> + *
> + * @child:    an entity to set res->limit. The parent is child->parent.
> + * @val:      the amount of resource to be moved.
> + * @callback: called when the parent's free resource is not enough to be moved.
> + *            this can be NULL if no callback is necessary.
> + * @retry:    limit for the number of trying to callback.
> + *            -1 means infinite loop. At each retry, yield() is called.
> + * Returns 0 at success, !0 at failure.
> + *
> + * The callback returns 0 at success, !0 at failure.
> + *
> + */
> +
> +int res_counter_move_resource(struct res_counter *child,
> +	unsigned long long val,
> +        int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry);
> +
> +
> +/**
> + * Return resource to its parent.
> + * At success,
> + *           parent->usage  -= val.
> + *           parent->for_children -= val.
> + *           child->limit -= val.
> + *
> + * @child:   entry to resize. The parent is child->parent.
> + * @val  :   How much does child repay to parent ? -1 means 'all'
> + * @callback: A callback for decreasing resource usage of child before
> + *            returning. If NULL, just deceases child's limit.
> + * @retry:   # of retries at calling callback for freeing resource.
> + *            -1 means infinite loop. At each retry, yield() is called.
> + * Returns 0 at success.
> + */
> +
> +int res_counter_return_resource(struct res_counter *child,
> +	unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry);
> +
>  #endif
> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> @@ -44,6 +44,13 @@ to work with it.
>   	Protects changes of the above values.
> 
> 
> + f. struct res_counter *parent
> +
> +	Parent res_counter under hierarchy.
> +
> + g. unsigned long long for_children
> +
> +	Resources assigned to children. This is included in usage.
> 
>  2. Basic accounting routines
> 
> @@ -179,3 +186,37 @@ counter fields. They are recommended to 
>      still can help with it).
> 
>   c. Compile and run :)
> +
> +
> +6. Hierarchy
> + a. No Hierarchy
> +   each cgroup can use its own private resource.
> +
> + b. Hard-wall Hierarhcy
> +   A simple hierarchical tree system for resource isolation.
> +   Allows moving resources only between a parent and its children.
> +   A parent can move its resource to children and remember the amount to
> +   for_children member. A child can get new resource only from its parent.
> +   Limit of a child is the amount of resource which is moved from its parent.
> +

OK, after reading this I am totally sure I want a shares based interface. Limits
are not shared like this.

A child and a parent should both be capable of having a limit of 1G, but they
could use different shares factors to govern, how much each children will get.
Doing it this way, breaks limit semantics.


> +   When add "val" to a child,
> +	parent->usage += val
> +	parent->for_children += val
> +	child->limit += val
> +   When a child returns its resource
> +	parent->usage -= val
> +	parent->for_children -= val
> +	child->limit -= val.
> +
> +   This implements resource isolation among each group. This works very well
> +   when you want to use strict resource isolation.
> +
> +   Usage Hint:
> +   This seems for static resource assignment but dynamic resource re-assignment
> +   can be done by resetting "limit" of groups. When you consider "limit" as
> +   the amount of allowed _current_ resource, a sophisticated resource management
> +   system based on strict resource isolation can be implemented.
> +
> +c. Soft-wall Hierarchy
> +   TBD.
> +
> Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
> +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
> @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
>  	counter->limit = (unsigned long long)LLONG_MAX;
>  }
> 
> +void res_counter_init_hierarchy(struct res_counter *counter,
> +		struct res_counter *parent)
> +{
> +	spin_lock_init(&counter->lock);
> +	counter->limit = 0;
> +	counter->parent = parent;
> +}
> +
>  int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
>  {
>  	if (counter->usage + val > counter->limit) {
> @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
>  		return &counter->limit;
>  	case RES_FAILCNT:
>  		return &counter->failcnt;
> +	case RES_FOR_CHILDREN:
> +		return &counter->for_children;
>  	};
> 
>  	BUG();
> @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
> 
>  ssize_t res_counter_write(struct res_counter *counter, int member,
>  		const char __user *userbuf, size_t nbytes, loff_t *pos,
> -		int (*write_strategy)(char *st_buf, unsigned long long *val))
> +		int (*write_strategy)(char *st_buf, unsigned long long *val),
> +		int (*set_strategy)(struct res_counter *res,
> +			unsigned long long val, int what))
>  {
>  	int ret;
>  	char *buf, *end;
> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
>  		if (*end != '\0')
>  			goto out_free;
>  	}
> -	spin_lock_irqsave(&counter->lock, flags);
> -	val = res_counter_member(counter, member);
> -	*val = tmp;
> -	spin_unlock_irqrestore(&counter->lock, flags);
> -	ret = nbytes;
> +	if (set_strategy) {
> +		ret = set_strategy(res, tmp, member);


I'm afraid, I don't understand the set_strategy and it's purpose.

> +		if (!ret)
> +			ret = nbytes;
> +	} else {
> +		spin_lock_irqsave(&counter->lock, flags);
> +		val = res_counter_member(counter, member);
> +		*val = tmp;
> +		spin_unlock_irqrestore(&counter->lock, flags);
> +		ret = nbytes;
> +	}
>  out_free:
>  	kfree(buf);
>  out:
>  	return ret;
>  }
> +
> +
> +int res_counter_move_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{
> +	struct res_counter *parent = child->parent;
> +	unsigned long flags;
> +
> +	BUG_ON(!parent);
> +
> +	while (1) {
> +		spin_lock_irqsave(&parent->lock, flags);
> +		if (parent->usage + val < parent->limit) {
> +			parent->for_children += val;
> +			parent->usage += val;
> +			break;
> +		}
> +		spin_unlock_irqrestore(&parent->lock, flags);
> +
> +		if (!retry || !callback)
> +			goto failed;
> +		/* -1 means  infinite loop */
> +		if (retry != -1)
> +			--retry;

I don't like the idea of spinning in an infinite loop, I would prefer to fail
things instead of burning CPU cycles.

> +		yield();
> +		callback(parent, val);

This code is not very understandable. Why do we yield before callback?

> +	}
> +	spin_unlock_irqrestore(&parent->lock, flags);
> +
> +	spin_lock_irqsave(&child->lock, flags);
> +	child->limit += val;
> +	spin_unlock_irqrestore(&child->lock, flags);
> +	return 0;
> +fail:
> +	return 1;
> +}
> +
> +
> +int res_counter_return_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{
> +	unsigned long flags;
> +	struct res_counter *parent = child->parent;
> +
> +	BUG_ON(!parent);
> +
> +	while (1) {
> +		spin_lock_irqsave(&child->lock, flags);
> +		if (val == (unsigned long long) -1) {
> +			val = child->limit;
> +			child->limit = 0;
> +			break;
> +		} else if (child->usage <= child->limit - val) {
> +			child->limit -= val;
> +			break;
> +		}
> +		spin_unlock_irqrestore(&child->lock, flags);
> +
> +		if (!retry)
> +			goto fail;
> +		/* -1 means infinite loop */
> +		if (retry != -1)
> +			--retry;
> +		yield();
> +		callback(parent, val);

Ditto comments as above.

> +	}
> +	spin_unlock_irqrestore(&child->lock, flags);
> +
> +	spin_lock_irqsave(&parent->lock, flags);
> +	BUG_ON(parent->for_children < val);
> +	BUG_ON(parent->usage < val);
> +	parent->for_children -= val;
> +	parent->usage -= val;
> +	spin_unlock_irqrestore(&parent->lock, flags);
> +	return 0;
> +fail:
> +	return 1;
> +}
> 

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

KAMEZAWA Hiroyuki wrote:
> A simple hard-wall hierarhcy support for res_counter.
> 
> Changelog v2->v3
>  - changed the name and arguments of functions.
>  - rewrote to be read easily.
>  - named as HardWall hierarchy.
> 
> This implements following model
>  - A cgroup's tree means hierarchy of resource.
>  - All child's resource is moved from its parents.
>  - The resource moved to children is charged as parent's usage.
>  - The resource moves when child->limit is changed.
>  - The sum of resource for children and its own usage is limited by "limit".
> 
> This implies
>  - No dynamic automatic hierarhcy balancing in the kernel.
>  - Each resource is isolated completely.
>  - The kernel just supports resource-move-at-change-in-limit.
>  - The user (middle-ware) is responsible to make hierarhcy balanced well.

We'd definitely like to see a user level tool/application as a demo of how this
can be achieved.

>    Good balance can be achieved by changing limit from user land.
> 
> 
> Background:
>  Recently, there are popular resource isolation technique widely used,
>  i.e. Hardware-Virtualization. We can do hierarchical resource isolation
>  by using cgroup on it. But supporting hierarchy management in croups
>  has some advantages of performance, unity and costs of management.
> 
>  There are good resource management in other OSs, they support some kind of
>  hierarchical resource management. We wonder what kind of hierarchy policy
>  is good for Linux. And there is an another point. Hierarchical system can be
>  implemented by the kernel and user-land co-operation.  So, there are various
>  choices to do in the kernel. Doing all in the kernel or export some proper
>  interfaces to the user-land. Middle-wares are tend to be used for management.
>  I hope there will be Open Source one.
> 
>  At supporting hierarchy in cgroup, several aspects of characteristics of
>  policy of hierarchy can be considered. Some needs automatic balancing
>  between several groups.
> 
>   - fairness    ... how fairness is kept under policy
> 
>   - performance ... should be _fast_. multi-level resource balancing tend
>                  to use much amount of CPU and can cause soft lockup.
> 
>   - predictability ... resource management are usually used for resource
>                  isolation. the kernel must not break the isolation and
>                  predictability of users against application's progress.
> 
>   - flexibility ... some sophisticated dynamic resource balancing with
>  		 soft-limit is welcomed when the user doesn't want strict
> 		 resource isolation or when the user cannot estimate how much
> 		 they want correctly.

Soft limits has been on my plate for a while now. I'll take a crack at it. At
the moment the statistics is a bit of a worry, since users/administrators need
good statistics to take further action.

> 
> Hard Wall Hierarchy.
> 
>  This patch implements a hard-wall model of hierarchy for resources.
>  Works well for users who want strict resource isolation.
> 
>  This model allows the move of resource only between a parent and its children.
>  The resource is moved to a child when it declares the amount of resources to be
>  used. (by limit)

The other reason for preferring a shares based approach is that, the it will be
more in line with the CPU controllers interfaces.


>  Automatic resource balancing is not supported in this code.  
>  (But users can do non-automatic by changing limit dynamically.)
> 
>  - fairness    ... good. no resource sharing. works as specified by users.
>  - performance ... good. each resources are capsuled to its own level.
>  - predictability ... good. resources are completely isolated. balancing only
> 		occurs at the event of changes in limit.
>  - flexibility ... bad. no flexibility and scheduling in the kernel level.
> 	        need middle-ware's help.
> 
> Considerations:
>  - This implementation uses "limit" == "current_available_resource".
>    This should be revisited when Soft-Limit one is implemented.
> 
> Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> 
> ---
>  Documentation/controllers/resource_counter.txt |   41 +++++++++
>  include/linux/res_counter.h                    |   90 +++++++++++++++++++-
>  kernel/res_counter.c                           |  112 +++++++++++++++++++++++--
>  3 files changed, 235 insertions(+), 8 deletions(-)
> 
> Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
> +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> @@ -38,6 +38,16 @@ struct res_counter {
>  	 * the number of unsuccessful attempts to consume the resource
>  	 */
>  	unsigned long long failcnt;
> +
> +	/*
> +	 * hierarchy support: the parent of this resource.
> +	 */
> +	struct res_counter *parent;
> +	/*
> +	 * the amount of resources assigned to children.
> +	 */
> +	unsigned long long for_children;
> +

I would prefer to use a better name, lent_out? reserved_for_children?
borrowed_by_children?

>  	/*
>  	 * the lock to protect all of the above.
>  	 * the routines below consider this to be IRQ-safe
> @@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
>  ssize_t res_counter_read(struct res_counter *counter, int member,
>  		const char __user *buf, size_t nbytes, loff_t *pos,
>  		int (*read_strategy)(unsigned long long val, char *s));
> +
> +/*
> + * An interface for setting res_counter's member (ex. limit)
> + * the new parameter is passed by *buf and translated by write_strategy().
> + * Then, it is applied to member under the control of set_strategy().
> + * If write_strategy() and set_strategy() can be NULL. see res_counter.c
> + */
> +
>  ssize_t res_counter_write(struct res_counter *counter, int member,
> -		const char __user *buf, size_t nbytes, loff_t *pos,
> -		int (*write_strategy)(char *buf, unsigned long long *val));
> +	const char __user *buf, size_t nbytes, loff_t *pos,
> +        int (*write_strategy)(char *buf, unsigned long long *val),
> +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
> +			    int what),
> +	);
> 
>  /*
>   * the field descriptors. one for each member of res_counter
> @@ -76,15 +97,33 @@ enum {
>  	RES_MAX_USAGE,
>  	RES_LIMIT,
>  	RES_FAILCNT,
> +	RES_FOR_CHILDREN,

RES_BORROWED? RES_BORROWED_BY_CHILDREN?

>  };
> 
>  /*
>   * helpers for accounting
>   */
> 
> +/*
> + * initialize res_counter.
> + * @counter : the counter
> + *
> + * initialize res_counter and set default limit to very big value(unlimited)
> + */
> +
>  void res_counter_init(struct res_counter *counter);
> 
>  /*
> + * initialize res_counter under hierarchy.
> + * @counter : the counter
> + * @parent : the parent of the counter
> + *
> + * initialize res_counter and set default limit to 0. and set "parent".
> + */
> +void res_counter_init_hierarchy(struct res_counter *counter,
> +				struct res_counter *parent);
> +
> +/*
>   * charge - try to consume more resource.
>   *
>   * @counter: the counter
> @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
>  	cnt->failcnt = 0;
>  	spin_unlock_irqrestore(&cnt->lock, flags);
>  }
> +
> +/**
> + * Move resources from a parent to a child.
> + * At success,
> + *           parent->usage += val.
> + *           parent->for_children += val.
> + *           child->limit += val.
> + *
> + * @child:    an entity to set res->limit. The parent is child->parent.
> + * @val:      the amount of resource to be moved.
> + * @callback: called when the parent's free resource is not enough to be moved.
> + *            this can be NULL if no callback is necessary.
> + * @retry:    limit for the number of trying to callback.
> + *            -1 means infinite loop. At each retry, yield() is called.
> + * Returns 0 at success, !0 at failure.
> + *
> + * The callback returns 0 at success, !0 at failure.
> + *
> + */
> +
> +int res_counter_move_resource(struct res_counter *child,
> +	unsigned long long val,
> +        int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry);
> +
> +
> +/**
> + * Return resource to its parent.
> + * At success,
> + *           parent->usage  -= val.
> + *           parent->for_children -= val.
> + *           child->limit -= val.
> + *
> + * @child:   entry to resize. The parent is child->parent.
> + * @val  :   How much does child repay to parent ? -1 means 'all'
> + * @callback: A callback for decreasing resource usage of child before
> + *            returning. If NULL, just deceases child's limit.
> + * @retry:   # of retries at calling callback for freeing resource.
> + *            -1 means infinite loop. At each retry, yield() is called.
> + * Returns 0 at success.
> + */
> +
> +int res_counter_return_resource(struct res_counter *child,
> +	unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry);
> +
>  #endif
> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> @@ -44,6 +44,13 @@ to work with it.
>   	Protects changes of the above values.
> 
> 
> + f. struct res_counter *parent
> +
> +	Parent res_counter under hierarchy.
> +
> + g. unsigned long long for_children
> +
> +	Resources assigned to children. This is included in usage.
> 
>  2. Basic accounting routines
> 
> @@ -179,3 +186,37 @@ counter fields. They are recommended to 
>      still can help with it).
> 
>   c. Compile and run :)
> +
> +
> +6. Hierarchy
> + a. No Hierarchy
> +   each cgroup can use its own private resource.
> +
> + b. Hard-wall Hierarhcy
> +   A simple hierarchical tree system for resource isolation.
> +   Allows moving resources only between a parent and its children.
> +   A parent can move its resource to children and remember the amount to
> +   for_children member. A child can get new resource only from its parent.
> +   Limit of a child is the amount of resource which is moved from its parent.
> +

OK, after reading this I am totally sure I want a shares based interface. Limits
are not shared like this.

A child and a parent should both be capable of having a limit of 1G, but they
could use different shares factors to govern, how much each children will get.
Doing it this way, breaks limit semantics.


> +   When add "val" to a child,
> +	parent->usage += val
> +	parent->for_children += val
> +	child->limit += val
> +   When a child returns its resource
> +	parent->usage -= val
> +	parent->for_children -= val
> +	child->limit -= val.
> +
> +   This implements resource isolation among each group. This works very well
> +   when you want to use strict resource isolation.
> +
> +   Usage Hint:
> +   This seems for static resource assignment but dynamic resource re-assignment
> +   can be done by resetting "limit" of groups. When you consider "limit" as
> +   the amount of allowed _current_ resource, a sophisticated resource management
> +   system based on strict resource isolation can be implemented.
> +
> +c. Soft-wall Hierarchy
> +   TBD.
> +
> Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
> +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
> @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
>  	counter->limit = (unsigned long long)LLONG_MAX;
>  }
> 
> +void res_counter_init_hierarchy(struct res_counter *counter,
> +		struct res_counter *parent)
> +{
> +	spin_lock_init(&counter->lock);
> +	counter->limit = 0;
> +	counter->parent = parent;
> +}
> +
>  int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
>  {
>  	if (counter->usage + val > counter->limit) {
> @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
>  		return &counter->limit;
>  	case RES_FAILCNT:
>  		return &counter->failcnt;
> +	case RES_FOR_CHILDREN:
> +		return &counter->for_children;
>  	};
> 
>  	BUG();
> @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
> 
>  ssize_t res_counter_write(struct res_counter *counter, int member,
>  		const char __user *userbuf, size_t nbytes, loff_t *pos,
> -		int (*write_strategy)(char *st_buf, unsigned long long *val))
> +		int (*write_strategy)(char *st_buf, unsigned long long *val),
> +		int (*set_strategy)(struct res_counter *res,
> +			unsigned long long val, int what))
>  {
>  	int ret;
>  	char *buf, *end;
> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
>  		if (*end != '\0')
>  			goto out_free;
>  	}
> -	spin_lock_irqsave(&counter->lock, flags);
> -	val = res_counter_member(counter, member);
> -	*val = tmp;
> -	spin_unlock_irqrestore(&counter->lock, flags);
> -	ret = nbytes;
> +	if (set_strategy) {
> +		ret = set_strategy(res, tmp, member);


I'm afraid, I don't understand the set_strategy and it's purpose.

> +		if (!ret)
> +			ret = nbytes;
> +	} else {
> +		spin_lock_irqsave(&counter->lock, flags);
> +		val = res_counter_member(counter, member);
> +		*val = tmp;
> +		spin_unlock_irqrestore(&counter->lock, flags);
> +		ret = nbytes;
> +	}
>  out_free:
>  	kfree(buf);
>  out:
>  	return ret;
>  }
> +
> +
> +int res_counter_move_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{
> +	struct res_counter *parent = child->parent;
> +	unsigned long flags;
> +
> +	BUG_ON(!parent);
> +
> +	while (1) {
> +		spin_lock_irqsave(&parent->lock, flags);
> +		if (parent->usage + val < parent->limit) {
> +			parent->for_children += val;
> +			parent->usage += val;
> +			break;
> +		}
> +		spin_unlock_irqrestore(&parent->lock, flags);
> +
> +		if (!retry || !callback)
> +			goto failed;
> +		/* -1 means  infinite loop */
> +		if (retry != -1)
> +			--retry;

I don't like the idea of spinning in an infinite loop, I would prefer to fail
things instead of burning CPU cycles.

> +		yield();
> +		callback(parent, val);

This code is not very understandable. Why do we yield before callback?

> +	}
> +	spin_unlock_irqrestore(&parent->lock, flags);
> +
> +	spin_lock_irqsave(&child->lock, flags);
> +	child->limit += val;
> +	spin_unlock_irqrestore(&child->lock, flags);
> +	return 0;
> +fail:
> +	return 1;
> +}
> +
> +
> +int res_counter_return_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{
> +	unsigned long flags;
> +	struct res_counter *parent = child->parent;
> +
> +	BUG_ON(!parent);
> +
> +	while (1) {
> +		spin_lock_irqsave(&child->lock, flags);
> +		if (val == (unsigned long long) -1) {
> +			val = child->limit;
> +			child->limit = 0;
> +			break;
> +		} else if (child->usage <= child->limit - val) {
> +			child->limit -= val;
> +			break;
> +		}
> +		spin_unlock_irqrestore(&child->lock, flags);
> +
> +		if (!retry)
> +			goto fail;
> +		/* -1 means infinite loop */
> +		if (retry != -1)
> +			--retry;
> +		yield();
> +		callback(parent, val);

Ditto comments as above.

> +	}
> +	spin_unlock_irqrestore(&child->lock, flags);
> +
> +	spin_lock_irqsave(&parent->lock, flags);
> +	BUG_ON(parent->for_children < val);
> +	BUG_ON(parent->usage < val);
> +	parent->for_children -= val;
> +	parent->usage -= val;
> +	spin_unlock_irqrestore(&parent->lock, flags);
> +	return 0;
> +fail:
> +	return 1;
> +}
> 

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Mon, 09 Jun 2008 15:18:47 +0530
Balbir Singh <balbir@linux.vnet.ibm.com> wrote:

> KAMEZAWA Hiroyuki wrote:
> > A simple hard-wall hierarhcy support for res_counter.
> > 
> > Changelog v2->v3
> >  - changed the name and arguments of functions.
> >  - rewrote to be read easily.
> >  - named as HardWall hierarchy.
> > 
> > This implements following model
> >  - A cgroup's tree means hierarchy of resource.
> >  - All child's resource is moved from its parents.
> >  - The resource moved to children is charged as parent's usage.
> >  - The resource moves when child->limit is changed.
> >  - The sum of resource for children and its own usage is limited by "limit".
> > 
> > This implies
> >  - No dynamic automatic hierarhcy balancing in the kernel.
> >  - Each resource is isolated completely.
> >  - The kernel just supports resource-move-at-change-in-limit.
> >  - The user (middle-ware) is responsible to make hierarhcy balanced well.
> 
> We'd definitely like to see a user level tool/application as a demo of how this
> can be achieved.
> 
I don't have one, now. I'll write one when I have time. Need now ?
Hmm...maybe I(we) need some more patches to implement useful statistics,
notifier to middlewares.



> >    Good balance can be achieved by changing limit from user land.
> > 
> > 
> > Background:
> >  Recently, there are popular resource isolation technique widely used,
> >  i.e. Hardware-Virtualization. We can do hierarchical resource isolation
> >  by using cgroup on it. But supporting hierarchy management in croups
> >  has some advantages of performance, unity and costs of management.
> > 
> >  There are good resource management in other OSs, they support some kind of
> >  hierarchical resource management. We wonder what kind of hierarchy policy
> >  is good for Linux. And there is an another point. Hierarchical system can be
> >  implemented by the kernel and user-land co-operation.  So, there are various
> >  choices to do in the kernel. Doing all in the kernel or export some proper
> >  interfaces to the user-land. Middle-wares are tend to be used for management.
> >  I hope there will be Open Source one.
> > 
> >  At supporting hierarchy in cgroup, several aspects of characteristics of
> >  policy of hierarchy can be considered. Some needs automatic balancing
> >  between several groups.
> > 
> >   - fairness    ... how fairness is kept under policy
> > 
> >   - performance ... should be _fast_. multi-level resource balancing tend
> >                  to use much amount of CPU and can cause soft lockup.
> > 
> >   - predictability ... resource management are usually used for resource
> >                  isolation. the kernel must not break the isolation and
> >                  predictability of users against application's progress.
> > 
> >   - flexibility ... some sophisticated dynamic resource balancing with
> >  		 soft-limit is welcomed when the user doesn't want strict
> > 		 resource isolation or when the user cannot estimate how much
> > 		 they want correctly.
> 
> Soft limits has been on my plate for a while now. I'll take a crack at it. At
> the moment the statistics is a bit of a worry, since users/administrators need
> good statistics to take further action.
> 
Yes, statistics is not enough now.



> > 
> > Hard Wall Hierarchy.
> > 
> >  This patch implements a hard-wall model of hierarchy for resources.
> >  Works well for users who want strict resource isolation.
> > 
> >  This model allows the move of resource only between a parent and its children.
> >  The resource is moved to a child when it declares the amount of resources to be
> >  used. (by limit)
> 
> The other reason for preferring a shares based approach is that, the it will be
> more in line with the CPU controllers interfaces.
> 

You have to think of the major difference of tha nature of CPU and Memory.
We have to reclaim the resource with some feedbacks among sevral cgroups.
But ok, if it's can be implemented in simple way.
I have no objections if cost is very low. My concern is only performance.
(and maintenance)


> > Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
> > +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> > @@ -38,6 +38,16 @@ struct res_counter {
> >  	 * the number of unsuccessful attempts to consume the resource
> >  	 */
> >  	unsigned long long failcnt;
> > +
> > +	/*
> > +	 * hierarchy support: the parent of this resource.
> > +	 */
> > +	struct res_counter *parent;
> > +	/*
> > +	 * the amount of resources assigned to children.
> > +	 */
> > +	unsigned long long for_children;
> > +
> 
> I would prefer to use a better name, lent_out? reserved_for_children?
> borrowed_by_children?
> 
ok. use other names.



> >  	/*
> >  	 * the lock to protect all of the above.
> >  	 * the routines below consider this to be IRQ-safe
> > @@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
> >  ssize_t res_counter_read(struct res_counter *counter, int member,
> >  		const char __user *buf, size_t nbytes, loff_t *pos,
> >  		int (*read_strategy)(unsigned long long val, char *s));
> > +
> > +/*
> > + * An interface for setting res_counter's member (ex. limit)
> > + * the new parameter is passed by *buf and translated by write_strategy().
> > + * Then, it is applied to member under the control of set_strategy().
> > + * If write_strategy() and set_strategy() can be NULL. see res_counter.c
> > + */
> > +
> >  ssize_t res_counter_write(struct res_counter *counter, int member,
> > -		const char __user *buf, size_t nbytes, loff_t *pos,
> > -		int (*write_strategy)(char *buf, unsigned long long *val));
> > +	const char __user *buf, size_t nbytes, loff_t *pos,
> > +        int (*write_strategy)(char *buf, unsigned long long *val),
> > +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
> > +			    int what),
> > +	);
> > 
> >  /*
> >   * the field descriptors. one for each member of res_counter
> > @@ -76,15 +97,33 @@ enum {
> >  	RES_MAX_USAGE,
> >  	RES_LIMIT,
> >  	RES_FAILCNT,
> > +	RES_FOR_CHILDREN,
> 
> RES_BORROWED? RES_BORROWED_BY_CHILDREN?
> 
ok, again.

> >  };
> > 
> >  /*
> >   * helpers for accounting
> >   */
> > 
> > +/*
> > + * initialize res_counter.
> > + * @counter : the counter
> > + *
> > + * initialize res_counter and set default limit to very big value(unlimited)
> > + */
> > +
> >  void res_counter_init(struct res_counter *counter);
> > 
> >  /*
> > + * initialize res_counter under hierarchy.
> > + * @counter : the counter
> > + * @parent : the parent of the counter
> > + *
> > + * initialize res_counter and set default limit to 0. and set "parent".
> > + */
> > +void res_counter_init_hierarchy(struct res_counter *counter,
> > +				struct res_counter *parent);
> > +
> > +/*
> >   * charge - try to consume more resource.
> >   *
> >   * @counter: the counter
> > @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
> >  	cnt->failcnt = 0;
> >  	spin_unlock_irqrestore(&cnt->lock, flags);
> >  }
> > +
> > +/**
> > + * Move resources from a parent to a child.
> > + * At success,
> > + *           parent->usage += val.
> > + *           parent->for_children += val.
> > + *           child->limit += val.
> > + *
> > + * @child:    an entity to set res->limit. The parent is child->parent.
> > + * @val:      the amount of resource to be moved.
> > + * @callback: called when the parent's free resource is not enough to be moved.
> > + *            this can be NULL if no callback is necessary.
> > + * @retry:    limit for the number of trying to callback.
> > + *            -1 means infinite loop. At each retry, yield() is called.
> > + * Returns 0 at success, !0 at failure.
> > + *
> > + * The callback returns 0 at success, !0 at failure.
> > + *
> > + */
> > +
> > +int res_counter_move_resource(struct res_counter *child,
> > +	unsigned long long val,
> > +        int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry);
> > +
> > +
> > +/**
> > + * Return resource to its parent.
> > + * At success,
> > + *           parent->usage  -= val.
> > + *           parent->for_children -= val.
> > + *           child->limit -= val.
> > + *
> > + * @child:   entry to resize. The parent is child->parent.
> > + * @val  :   How much does child repay to parent ? -1 means 'all'
> > + * @callback: A callback for decreasing resource usage of child before
> > + *            returning. If NULL, just deceases child's limit.
> > + * @retry:   # of retries at calling callback for freeing resource.
> > + *            -1 means infinite loop. At each retry, yield() is called.
> > + * Returns 0 at success.
> > + */
> > +
> > +int res_counter_return_resource(struct res_counter *child,
> > +	unsigned long long val,
> > +	int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry);
> > +
> >  #endif
> > Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> > +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > @@ -44,6 +44,13 @@ to work with it.
> >   	Protects changes of the above values.
> > 
> > 
> > + f. struct res_counter *parent
> > +
> > +	Parent res_counter under hierarchy.
> > +
> > + g. unsigned long long for_children
> > +
> > +	Resources assigned to children. This is included in usage.
> > 
> >  2. Basic accounting routines
> > 
> > @@ -179,3 +186,37 @@ counter fields. They are recommended to 
> >      still can help with it).
> > 
> >   c. Compile and run :)
> > +
> > +
> > +6. Hierarchy
> > + a. No Hierarchy
> > +   each cgroup can use its own private resource.
> > +
> > + b. Hard-wall Hierarhcy
> > +   A simple hierarchical tree system for resource isolation.
> > +   Allows moving resources only between a parent and its children.
> > +   A parent can move its resource to children and remember the amount to
> > +   for_children member. A child can get new resource only from its parent.
> > +   Limit of a child is the amount of resource which is moved from its parent.
> > +
> 
> OK, after reading this I am totally sure I want a shares based interface. Limits
> are not shared like this.
> 
> A child and a parent should both be capable of having a limit of 1G, but they
> could use different shares factors to govern, how much each children will get.
> Doing it this way, breaks limit semantics.
> 
Not easy to use in my point of view. Can we use 'share' in proper way 
on no-swap machine ?


> 
> > +   When add "val" to a child,
> > +	parent->usage += val
> > +	parent->for_children += val
> > +	child->limit += val
> > +   When a child returns its resource
> > +	parent->usage -= val
> > +	parent->for_children -= val
> > +	child->limit -= val.
> > +
> > +   This implements resource isolation among each group. This works very well
> > +   when you want to use strict resource isolation.
> > +
> > +   Usage Hint:
> > +   This seems for static resource assignment but dynamic resource re-assignment
> > +   can be done by resetting "limit" of groups. When you consider "limit" as
> > +   the amount of allowed _current_ resource, a sophisticated resource management
> > +   system based on strict resource isolation can be implemented.
> > +
> > +c. Soft-wall Hierarchy
> > +   TBD.
> > +
> > Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
> > +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
> > @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
> >  	counter->limit = (unsigned long long)LLONG_MAX;
> >  }
> > 
> > +void res_counter_init_hierarchy(struct res_counter *counter,
> > +		struct res_counter *parent)
> > +{
> > +	spin_lock_init(&counter->lock);
> > +	counter->limit = 0;
> > +	counter->parent = parent;
> > +}
> > +
> >  int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
> >  {
> >  	if (counter->usage + val > counter->limit) {
> > @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
> >  		return &counter->limit;
> >  	case RES_FAILCNT:
> >  		return &counter->failcnt;
> > +	case RES_FOR_CHILDREN:
> > +		return &counter->for_children;
> >  	};
> > 
> >  	BUG();
> > @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
> > 
> >  ssize_t res_counter_write(struct res_counter *counter, int member,
> >  		const char __user *userbuf, size_t nbytes, loff_t *pos,
> > -		int (*write_strategy)(char *st_buf, unsigned long long *val))
> > +		int (*write_strategy)(char *st_buf, unsigned long long *val),
> > +		int (*set_strategy)(struct res_counter *res,
> > +			unsigned long long val, int what))
> >  {
> >  	int ret;
> >  	char *buf, *end;
> > @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
> >  		if (*end != '\0')
> >  			goto out_free;
> >  	}
> > -	spin_lock_irqsave(&counter->lock, flags);
> > -	val = res_counter_member(counter, member);
> > -	*val = tmp;
> > -	spin_unlock_irqrestore(&counter->lock, flags);
> > -	ret = nbytes;
> > +	if (set_strategy) {
> > +		ret = set_strategy(res, tmp, member);
> 
> 
> I'm afraid, I don't understand the set_strategy and it's purpose.
> 
Sorry. I'm now rewritten and removed this.



> > +		if (!ret)
> > +			ret = nbytes;
> > +	} else {
> > +		spin_lock_irqsave(&counter->lock, flags);
> > +		val = res_counter_member(counter, member);
> > +		*val = tmp;
> > +		spin_unlock_irqrestore(&counter->lock, flags);
> > +		ret = nbytes;
> > +	}
> >  out_free:
> >  	kfree(buf);
> >  out:
> >  	return ret;
> >  }
> > +
> > +
> > +int res_counter_move_resource(struct res_counter *child,
> > +				unsigned long long val,
> > +	int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry)
> > +{
> > +	struct res_counter *parent = child->parent;
> > +	unsigned long flags;
> > +
> > +	BUG_ON(!parent);
> > +
> > +	while (1) {
> > +		spin_lock_irqsave(&parent->lock, flags);
> > +		if (parent->usage + val < parent->limit) {
> > +			parent->for_children += val;
> > +			parent->usage += val;
> > +			break;
> > +		}
> > +		spin_unlock_irqrestore(&parent->lock, flags);
> > +
> > +		if (!retry || !callback)
> > +			goto failed;
> > +		/* -1 means  infinite loop */
> > +		if (retry != -1)
> > +			--retry;
> 
> I don't like the idea of spinning in an infinite loop, I would prefer to fail
> things instead of burning CPU cycles.
> 
ok, will remove "-1" case.




> > +		yield();
> > +		callback(parent, val);
> 
> This code is not very understandable. Why do we yield before callback?
> 

yield() after callback() means that res_counter's state will be
far different from the state after callback.
So, we have to yield before call back and check res_coutner sooner.

Thanks,
-Kame

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Mon, 09 Jun 2008 15:18:47 +0530
Balbir Singh <balbir@linux.vnet.ibm.com> wrote:

> KAMEZAWA Hiroyuki wrote:
> > A simple hard-wall hierarhcy support for res_counter.
> > 
> > Changelog v2->v3
> >  - changed the name and arguments of functions.
> >  - rewrote to be read easily.
> >  - named as HardWall hierarchy.
> > 
> > This implements following model
> >  - A cgroup's tree means hierarchy of resource.
> >  - All child's resource is moved from its parents.
> >  - The resource moved to children is charged as parent's usage.
> >  - The resource moves when child->limit is changed.
> >  - The sum of resource for children and its own usage is limited by "limit".
> > 
> > This implies
> >  - No dynamic automatic hierarhcy balancing in the kernel.
> >  - Each resource is isolated completely.
> >  - The kernel just supports resource-move-at-change-in-limit.
> >  - The user (middle-ware) is responsible to make hierarhcy balanced well.
> 
> We'd definitely like to see a user level tool/application as a demo of how this
> can be achieved.
> 
I don't have one, now. I'll write one when I have time. Need now ?
Hmm...maybe I(we) need some more patches to implement useful statistics,
notifier to middlewares.



> >    Good balance can be achieved by changing limit from user land.
> > 
> > 
> > Background:
> >  Recently, there are popular resource isolation technique widely used,
> >  i.e. Hardware-Virtualization. We can do hierarchical resource isolation
> >  by using cgroup on it. But supporting hierarchy management in croups
> >  has some advantages of performance, unity and costs of management.
> > 
> >  There are good resource management in other OSs, they support some kind of
> >  hierarchical resource management. We wonder what kind of hierarchy policy
> >  is good for Linux. And there is an another point. Hierarchical system can be
> >  implemented by the kernel and user-land co-operation.  So, there are various
> >  choices to do in the kernel. Doing all in the kernel or export some proper
> >  interfaces to the user-land. Middle-wares are tend to be used for management.
> >  I hope there will be Open Source one.
> > 
> >  At supporting hierarchy in cgroup, several aspects of characteristics of
> >  policy of hierarchy can be considered. Some needs automatic balancing
> >  between several groups.
> > 
> >   - fairness    ... how fairness is kept under policy
> > 
> >   - performance ... should be _fast_. multi-level resource balancing tend
> >                  to use much amount of CPU and can cause soft lockup.
> > 
> >   - predictability ... resource management are usually used for resource
> >                  isolation. the kernel must not break the isolation and
> >                  predictability of users against application's progress.
> > 
> >   - flexibility ... some sophisticated dynamic resource balancing with
> >  		 soft-limit is welcomed when the user doesn't want strict
> > 		 resource isolation or when the user cannot estimate how much
> > 		 they want correctly.
> 
> Soft limits has been on my plate for a while now. I'll take a crack at it. At
> the moment the statistics is a bit of a worry, since users/administrators need
> good statistics to take further action.
> 
Yes, statistics is not enough now.



> > 
> > Hard Wall Hierarchy.
> > 
> >  This patch implements a hard-wall model of hierarchy for resources.
> >  Works well for users who want strict resource isolation.
> > 
> >  This model allows the move of resource only between a parent and its children.
> >  The resource is moved to a child when it declares the amount of resources to be
> >  used. (by limit)
> 
> The other reason for preferring a shares based approach is that, the it will be
> more in line with the CPU controllers interfaces.
> 

You have to think of the major difference of tha nature of CPU and Memory.
We have to reclaim the resource with some feedbacks among sevral cgroups.
But ok, if it's can be implemented in simple way.
I have no objections if cost is very low. My concern is only performance.
(and maintenance)


> > Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
> > +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> > @@ -38,6 +38,16 @@ struct res_counter {
> >  	 * the number of unsuccessful attempts to consume the resource
> >  	 */
> >  	unsigned long long failcnt;
> > +
> > +	/*
> > +	 * hierarchy support: the parent of this resource.
> > +	 */
> > +	struct res_counter *parent;
> > +	/*
> > +	 * the amount of resources assigned to children.
> > +	 */
> > +	unsigned long long for_children;
> > +
> 
> I would prefer to use a better name, lent_out? reserved_for_children?
> borrowed_by_children?
> 
ok. use other names.



> >  	/*
> >  	 * the lock to protect all of the above.
> >  	 * the routines below consider this to be IRQ-safe
> > @@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
> >  ssize_t res_counter_read(struct res_counter *counter, int member,
> >  		const char __user *buf, size_t nbytes, loff_t *pos,
> >  		int (*read_strategy)(unsigned long long val, char *s));
> > +
> > +/*
> > + * An interface for setting res_counter's member (ex. limit)
> > + * the new parameter is passed by *buf and translated by write_strategy().
> > + * Then, it is applied to member under the control of set_strategy().
> > + * If write_strategy() and set_strategy() can be NULL. see res_counter.c
> > + */
> > +
> >  ssize_t res_counter_write(struct res_counter *counter, int member,
> > -		const char __user *buf, size_t nbytes, loff_t *pos,
> > -		int (*write_strategy)(char *buf, unsigned long long *val));
> > +	const char __user *buf, size_t nbytes, loff_t *pos,
> > +        int (*write_strategy)(char *buf, unsigned long long *val),
> > +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
> > +			    int what),
> > +	);
> > 
> >  /*
> >   * the field descriptors. one for each member of res_counter
> > @@ -76,15 +97,33 @@ enum {
> >  	RES_MAX_USAGE,
> >  	RES_LIMIT,
> >  	RES_FAILCNT,
> > +	RES_FOR_CHILDREN,
> 
> RES_BORROWED? RES_BORROWED_BY_CHILDREN?
> 
ok, again.

> >  };
> > 
> >  /*
> >   * helpers for accounting
> >   */
> > 
> > +/*
> > + * initialize res_counter.
> > + * @counter : the counter
> > + *
> > + * initialize res_counter and set default limit to very big value(unlimited)
> > + */
> > +
> >  void res_counter_init(struct res_counter *counter);
> > 
> >  /*
> > + * initialize res_counter under hierarchy.
> > + * @counter : the counter
> > + * @parent : the parent of the counter
> > + *
> > + * initialize res_counter and set default limit to 0. and set "parent".
> > + */
> > +void res_counter_init_hierarchy(struct res_counter *counter,
> > +				struct res_counter *parent);
> > +
> > +/*
> >   * charge - try to consume more resource.
> >   *
> >   * @counter: the counter
> > @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
> >  	cnt->failcnt = 0;
> >  	spin_unlock_irqrestore(&cnt->lock, flags);
> >  }
> > +
> > +/**
> > + * Move resources from a parent to a child.
> > + * At success,
> > + *           parent->usage += val.
> > + *           parent->for_children += val.
> > + *           child->limit += val.
> > + *
> > + * @child:    an entity to set res->limit. The parent is child->parent.
> > + * @val:      the amount of resource to be moved.
> > + * @callback: called when the parent's free resource is not enough to be moved.
> > + *            this can be NULL if no callback is necessary.
> > + * @retry:    limit for the number of trying to callback.
> > + *            -1 means infinite loop. At each retry, yield() is called.
> > + * Returns 0 at success, !0 at failure.
> > + *
> > + * The callback returns 0 at success, !0 at failure.
> > + *
> > + */
> > +
> > +int res_counter_move_resource(struct res_counter *child,
> > +	unsigned long long val,
> > +        int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry);
> > +
> > +
> > +/**
> > + * Return resource to its parent.
> > + * At success,
> > + *           parent->usage  -= val.
> > + *           parent->for_children -= val.
> > + *           child->limit -= val.
> > + *
> > + * @child:   entry to resize. The parent is child->parent.
> > + * @val  :   How much does child repay to parent ? -1 means 'all'
> > + * @callback: A callback for decreasing resource usage of child before
> > + *            returning. If NULL, just deceases child's limit.
> > + * @retry:   # of retries at calling callback for freeing resource.
> > + *            -1 means infinite loop. At each retry, yield() is called.
> > + * Returns 0 at success.
> > + */
> > +
> > +int res_counter_return_resource(struct res_counter *child,
> > +	unsigned long long val,
> > +	int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry);
> > +
> >  #endif
> > Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> > +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > @@ -44,6 +44,13 @@ to work with it.
> >   	Protects changes of the above values.
> > 
> > 
> > + f. struct res_counter *parent
> > +
> > +	Parent res_counter under hierarchy.
> > +
> > + g. unsigned long long for_children
> > +
> > +	Resources assigned to children. This is included in usage.
> > 
> >  2. Basic accounting routines
> > 
> > @@ -179,3 +186,37 @@ counter fields. They are recommended to 
> >      still can help with it).
> > 
> >   c. Compile and run :)
> > +
> > +
> > +6. Hierarchy
> > + a. No Hierarchy
> > +   each cgroup can use its own private resource.
> > +
> > + b. Hard-wall Hierarhcy
> > +   A simple hierarchical tree system for resource isolation.
> > +   Allows moving resources only between a parent and its children.
> > +   A parent can move its resource to children and remember the amount to
> > +   for_children member. A child can get new resource only from its parent.
> > +   Limit of a child is the amount of resource which is moved from its parent.
> > +
> 
> OK, after reading this I am totally sure I want a shares based interface. Limits
> are not shared like this.
> 
> A child and a parent should both be capable of having a limit of 1G, but they
> could use different shares factors to govern, how much each children will get.
> Doing it this way, breaks limit semantics.
> 
Not easy to use in my point of view. Can we use 'share' in proper way 
on no-swap machine ?


> 
> > +   When add "val" to a child,
> > +	parent->usage += val
> > +	parent->for_children += val
> > +	child->limit += val
> > +   When a child returns its resource
> > +	parent->usage -= val
> > +	parent->for_children -= val
> > +	child->limit -= val.
> > +
> > +   This implements resource isolation among each group. This works very well
> > +   when you want to use strict resource isolation.
> > +
> > +   Usage Hint:
> > +   This seems for static resource assignment but dynamic resource re-assignment
> > +   can be done by resetting "limit" of groups. When you consider "limit" as
> > +   the amount of allowed _current_ resource, a sophisticated resource management
> > +   system based on strict resource isolation can be implemented.
> > +
> > +c. Soft-wall Hierarchy
> > +   TBD.
> > +
> > Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
> > +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
> > @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
> >  	counter->limit = (unsigned long long)LLONG_MAX;
> >  }
> > 
> > +void res_counter_init_hierarchy(struct res_counter *counter,
> > +		struct res_counter *parent)
> > +{
> > +	spin_lock_init(&counter->lock);
> > +	counter->limit = 0;
> > +	counter->parent = parent;
> > +}
> > +
> >  int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
> >  {
> >  	if (counter->usage + val > counter->limit) {
> > @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
> >  		return &counter->limit;
> >  	case RES_FAILCNT:
> >  		return &counter->failcnt;
> > +	case RES_FOR_CHILDREN:
> > +		return &counter->for_children;
> >  	};
> > 
> >  	BUG();
> > @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
> > 
> >  ssize_t res_counter_write(struct res_counter *counter, int member,
> >  		const char __user *userbuf, size_t nbytes, loff_t *pos,
> > -		int (*write_strategy)(char *st_buf, unsigned long long *val))
> > +		int (*write_strategy)(char *st_buf, unsigned long long *val),
> > +		int (*set_strategy)(struct res_counter *res,
> > +			unsigned long long val, int what))
> >  {
> >  	int ret;
> >  	char *buf, *end;
> > @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
> >  		if (*end != '\0')
> >  			goto out_free;
> >  	}
> > -	spin_lock_irqsave(&counter->lock, flags);
> > -	val = res_counter_member(counter, member);
> > -	*val = tmp;
> > -	spin_unlock_irqrestore(&counter->lock, flags);
> > -	ret = nbytes;
> > +	if (set_strategy) {
> > +		ret = set_strategy(res, tmp, member);
> 
> 
> I'm afraid, I don't understand the set_strategy and it's purpose.
> 
Sorry. I'm now rewritten and removed this.



> > +		if (!ret)
> > +			ret = nbytes;
> > +	} else {
> > +		spin_lock_irqsave(&counter->lock, flags);
> > +		val = res_counter_member(counter, member);
> > +		*val = tmp;
> > +		spin_unlock_irqrestore(&counter->lock, flags);
> > +		ret = nbytes;
> > +	}
> >  out_free:
> >  	kfree(buf);
> >  out:
> >  	return ret;
> >  }
> > +
> > +
> > +int res_counter_move_resource(struct res_counter *child,
> > +				unsigned long long val,
> > +	int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry)
> > +{
> > +	struct res_counter *parent = child->parent;
> > +	unsigned long flags;
> > +
> > +	BUG_ON(!parent);
> > +
> > +	while (1) {
> > +		spin_lock_irqsave(&parent->lock, flags);
> > +		if (parent->usage + val < parent->limit) {
> > +			parent->for_children += val;
> > +			parent->usage += val;
> > +			break;
> > +		}
> > +		spin_unlock_irqrestore(&parent->lock, flags);
> > +
> > +		if (!retry || !callback)
> > +			goto failed;
> > +		/* -1 means  infinite loop */
> > +		if (retry != -1)
> > +			--retry;
> 
> I don't like the idea of spinning in an infinite loop, I would prefer to fail
> things instead of burning CPU cycles.
> 
ok, will remove "-1" case.




> > +		yield();
> > +		callback(parent, val);
> 
> This code is not very understandable. Why do we yield before callback?
> 

yield() after callback() means that res_counter's state will be
far different from the state after callback.
So, we have to yield before call back and check res_coutner sooner.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

KAMEZAWA Hiroyuki wrote:
> On Mon, 09 Jun 2008 15:18:47 +0530
> Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
> 
>> KAMEZAWA Hiroyuki wrote:
>>> A simple hard-wall hierarhcy support for res_counter.
>>>
>>> Changelog v2->v3
>>>  - changed the name and arguments of functions.
>>>  - rewrote to be read easily.
>>>  - named as HardWall hierarchy.
>>>
>>> This implements following model
>>>  - A cgroup's tree means hierarchy of resource.
>>>  - All child's resource is moved from its parents.
>>>  - The resource moved to children is charged as parent's usage.
>>>  - The resource moves when child->limit is changed.
>>>  - The sum of resource for children and its own usage is limited by "limit".
>>>
>>> This implies
>>>  - No dynamic automatic hierarhcy balancing in the kernel.
>>>  - Each resource is isolated completely.
>>>  - The kernel just supports resource-move-at-change-in-limit.
>>>  - The user (middle-ware) is responsible to make hierarhcy balanced well.
>> We'd definitely like to see a user level tool/application as a demo of how this
>> can be achieved.
>>
> I don't have one, now. I'll write one when I have time. Need now ?
> Hmm...maybe I(we) need some more patches to implement useful statistics,
> notifier to middlewares.
> 

Yes, we need more useful statistics.

> 
> 
>>>    Good balance can be achieved by changing limit from user land.
>>>
>>>
>>> Background:
>>>  Recently, there are popular resource isolation technique widely used,
>>>  i.e. Hardware-Virtualization. We can do hierarchical resource isolation
>>>  by using cgroup on it. But supporting hierarchy management in croups
>>>  has some advantages of performance, unity and costs of management.
>>>
>>>  There are good resource management in other OSs, they support some kind of
>>>  hierarchical resource management. We wonder what kind of hierarchy policy
>>>  is good for Linux. And there is an another point. Hierarchical system can be
>>>  implemented by the kernel and user-land co-operation.  So, there are various
>>>  choices to do in the kernel. Doing all in the kernel or export some proper
>>>  interfaces to the user-land. Middle-wares are tend to be used for management.
>>>  I hope there will be Open Source one.
>>>
>>>  At supporting hierarchy in cgroup, several aspects of characteristics of
>>>  policy of hierarchy can be considered. Some needs automatic balancing
>>>  between several groups.
>>>
>>>   - fairness    ... how fairness is kept under policy
>>>
>>>   - performance ... should be _fast_. multi-level resource balancing tend
>>>                  to use much amount of CPU and can cause soft lockup.
>>>
>>>   - predictability ... resource management are usually used for resource
>>>                  isolation. the kernel must not break the isolation and
>>>                  predictability of users against application's progress.
>>>
>>>   - flexibility ... some sophisticated dynamic resource balancing with
>>>  		 soft-limit is welcomed when the user doesn't want strict
>>> 		 resource isolation or when the user cannot estimate how much
>>> 		 they want correctly.
>> Soft limits has been on my plate for a while now. I'll take a crack at it. At
>> the moment the statistics is a bit of a worry, since users/administrators need
>> good statistics to take further action.
>>
> Yes, statistics is not enough now.
> 
> 
> 
>>> Hard Wall Hierarchy.
>>>
>>>  This patch implements a hard-wall model of hierarchy for resources.
>>>  Works well for users who want strict resource isolation.
>>>
>>>  This model allows the move of resource only between a parent and its children.
>>>  The resource is moved to a child when it declares the amount of resources to be
>>>  used. (by limit)
>> The other reason for preferring a shares based approach is that, the it will be
>> more in line with the CPU controllers interfaces.
>>
> 
> You have to think of the major difference of tha nature of CPU and Memory.
> We have to reclaim the resource with some feedbacks among sevral cgroups.
> But ok, if it's can be implemented in simple way.
> I have no objections if cost is very low. My concern is only performance.
> (and maintenance)
> 

True, I don't see hierarchy as adding too much additional cost.

> 
>>> Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
>>> ===================================================================
>>> --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
>>> +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
>>> @@ -38,6 +38,16 @@ struct res_counter {
>>>  	 * the number of unsuccessful attempts to consume the resource
>>>  	 */
>>>  	unsigned long long failcnt;
>>> +
>>> +	/*
>>> +	 * hierarchy support: the parent of this resource.
>>> +	 */
>>> +	struct res_counter *parent;
>>> +	/*
>>> +	 * the amount of resources assigned to children.
>>> +	 */
>>> +	unsigned long long for_children;
>>> +
>> I would prefer to use a better name, lent_out? reserved_for_children?
>> borrowed_by_children?
>>
> ok. use other names.
> 
> 
> 
>>>  	/*
>>>  	 * the lock to protect all of the above.
>>>  	 * the routines below consider this to be IRQ-safe
>>> @@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
>>>  ssize_t res_counter_read(struct res_counter *counter, int member,
>>>  		const char __user *buf, size_t nbytes, loff_t *pos,
>>>  		int (*read_strategy)(unsigned long long val, char *s));
>>> +
>>> +/*
>>> + * An interface for setting res_counter's member (ex. limit)
>>> + * the new parameter is passed by *buf and translated by write_strategy().
>>> + * Then, it is applied to member under the control of set_strategy().
>>> + * If write_strategy() and set_strategy() can be NULL. see res_counter.c
>>> + */
>>> +
>>>  ssize_t res_counter_write(struct res_counter *counter, int member,
>>> -		const char __user *buf, size_t nbytes, loff_t *pos,
>>> -		int (*write_strategy)(char *buf, unsigned long long *val));
>>> +	const char __user *buf, size_t nbytes, loff_t *pos,
>>> +        int (*write_strategy)(char *buf, unsigned long long *val),
>>> +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
>>> +			    int what),
>>> +	);
>>>
>>>  /*
>>>   * the field descriptors. one for each member of res_counter
>>> @@ -76,15 +97,33 @@ enum {
>>>  	RES_MAX_USAGE,
>>>  	RES_LIMIT,
>>>  	RES_FAILCNT,
>>> +	RES_FOR_CHILDREN,
>> RES_BORROWED? RES_BORROWED_BY_CHILDREN?
>>
> ok, again.
> 
>>>  };
>>>
>>>  /*
>>>   * helpers for accounting
>>>   */
>>>
>>> +/*
>>> + * initialize res_counter.
>>> + * @counter : the counter
>>> + *
>>> + * initialize res_counter and set default limit to very big value(unlimited)
>>> + */
>>> +
>>>  void res_counter_init(struct res_counter *counter);
>>>
>>>  /*
>>> + * initialize res_counter under hierarchy.
>>> + * @counter : the counter
>>> + * @parent : the parent of the counter
>>> + *
>>> + * initialize res_counter and set default limit to 0. and set "parent".
>>> + */
>>> +void res_counter_init_hierarchy(struct res_counter *counter,
>>> +				struct res_counter *parent);
>>> +
>>> +/*
>>>   * charge - try to consume more resource.
>>>   *
>>>   * @counter: the counter
>>> @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
>>>  	cnt->failcnt = 0;
>>>  	spin_unlock_irqrestore(&cnt->lock, flags);
>>>  }
>>> +
>>> +/**
>>> + * Move resources from a parent to a child.
>>> + * At success,
>>> + *           parent->usage += val.
>>> + *           parent->for_children += val.
>>> + *           child->limit += val.
>>> + *
>>> + * @child:    an entity to set res->limit. The parent is child->parent.
>>> + * @val:      the amount of resource to be moved.
>>> + * @callback: called when the parent's free resource is not enough to be moved.
>>> + *            this can be NULL if no callback is necessary.
>>> + * @retry:    limit for the number of trying to callback.
>>> + *            -1 means infinite loop. At each retry, yield() is called.
>>> + * Returns 0 at success, !0 at failure.
>>> + *
>>> + * The callback returns 0 at success, !0 at failure.
>>> + *
>>> + */
>>> +
>>> +int res_counter_move_resource(struct res_counter *child,
>>> +	unsigned long long val,
>>> +        int (*callback)(struct res_counter *res, unsigned long long val),
>>> +	int retry);
>>> +
>>> +
>>> +/**
>>> + * Return resource to its parent.
>>> + * At success,
>>> + *           parent->usage  -= val.
>>> + *           parent->for_children -= val.
>>> + *           child->limit -= val.
>>> + *
>>> + * @child:   entry to resize. The parent is child->parent.
>>> + * @val  :   How much does child repay to parent ? -1 means 'all'
>>> + * @callback: A callback for decreasing resource usage of child before
>>> + *            returning. If NULL, just deceases child's limit.
>>> + * @retry:   # of retries at calling callback for freeing resource.
>>> + *            -1 means infinite loop. At each retry, yield() is called.
>>> + * Returns 0 at success.
>>> + */
>>> +
>>> +int res_counter_return_resource(struct res_counter *child,
>>> +	unsigned long long val,
>>> +	int (*callback)(struct res_counter *res, unsigned long long val),
>>> +	int retry);
>>> +
>>>  #endif
>>> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
>>> ===================================================================
>>> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
>>> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
>>> @@ -44,6 +44,13 @@ to work with it.
>>>   	Protects changes of the above values.
>>>
>>>
>>> + f. struct res_counter *parent
>>> +
>>> +	Parent res_counter under hierarchy.
>>> +
>>> + g. unsigned long long for_children
>>> +
>>> +	Resources assigned to children. This is included in usage.
>>>
>>>  2. Basic accounting routines
>>>
>>> @@ -179,3 +186,37 @@ counter fields. They are recommended to 
>>>      still can help with it).
>>>
>>>   c. Compile and run :)
>>> +
>>> +
>>> +6. Hierarchy
>>> + a. No Hierarchy
>>> +   each cgroup can use its own private resource.
>>> +
>>> + b. Hard-wall Hierarhcy
>>> +   A simple hierarchical tree system for resource isolation.
>>> +   Allows moving resources only between a parent and its children.
>>> +   A parent can move its resource to children and remember the amount to
>>> +   for_children member. A child can get new resource only from its parent.
>>> +   Limit of a child is the amount of resource which is moved from its parent.
>>> +
>> OK, after reading this I am totally sure I want a shares based interface. Limits
>> are not shared like this.
>>
>> A child and a parent should both be capable of having a limit of 1G, but they
>> could use different shares factors to govern, how much each children will get.
>> Doing it this way, breaks limit semantics.
>>
> Not easy to use in my point of view. Can we use 'share' in proper way 
> on no-swap machine ?
> 

Not sure I understand your question. Share represents the share of available
resources.

> 
>>> +   When add "val" to a child,
>>> +	parent->usage += val
>>> +	parent->for_children += val
>>> +	child->limit += val
>>> +   When a child returns its resource
>>> +	parent->usage -= val
>>> +	parent->for_children -= val
>>> +	child->limit -= val.
>>> +
>>> +   This implements resource isolation among each group. This works very well
>>> +   when you want to use strict resource isolation.
>>> +
>>> +   Usage Hint:
>>> +   This seems for static resource assignment but dynamic resource re-assignment
>>> +   can be done by resetting "limit" of groups. When you consider "limit" as
>>> +   the amount of allowed _current_ resource, a sophisticated resource management
>>> +   system based on strict resource isolation can be implemented.
>>> +
>>> +c. Soft-wall Hierarchy
>>> +   TBD.
>>> +
>>> Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
>>> ===================================================================
>>> --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
>>> +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
>>> @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
>>>  	counter->limit = (unsigned long long)LLONG_MAX;
>>>  }
>>>
>>> +void res_counter_init_hierarchy(struct res_counter *counter,
>>> +		struct res_counter *parent)
>>> +{
>>> +	spin_lock_init(&counter->lock);
>>> +	counter->limit = 0;
>>> +	counter->parent = parent;
>>> +}
>>> +
>>>  int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
>>>  {
>>>  	if (counter->usage + val > counter->limit) {
>>> @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
>>>  		return &counter->limit;
>>>  	case RES_FAILCNT:
>>>  		return &counter->failcnt;
>>> +	case RES_FOR_CHILDREN:
>>> +		return &counter->for_children;
>>>  	};
>>>
>>>  	BUG();
>>> @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
>>>
>>>  ssize_t res_counter_write(struct res_counter *counter, int member,
>>>  		const char __user *userbuf, size_t nbytes, loff_t *pos,
>>> -		int (*write_strategy)(char *st_buf, unsigned long long *val))
>>> +		int (*write_strategy)(char *st_buf, unsigned long long *val),
>>> +		int (*set_strategy)(struct res_counter *res,
>>> +			unsigned long long val, int what))
>>>  {
>>>  	int ret;
>>>  	char *buf, *end;
>>> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
>>>  		if (*end != '\0')
>>>  			goto out_free;
>>>  	}
>>> -	spin_lock_irqsave(&counter->lock, flags);
>>> -	val = res_counter_member(counter, member);
>>> -	*val = tmp;
>>> -	spin_unlock_irqrestore(&counter->lock, flags);
>>> -	ret = nbytes;
>>> +	if (set_strategy) {
>>> +		ret = set_strategy(res, tmp, member);
>>
>> I'm afraid, I don't understand the set_strategy and it's purpose.
>>
> Sorry. I'm now rewritten and removed this.
> 
> 

OK

> 
>>> +		if (!ret)
>>> +			ret = nbytes;
>>> +	} else {
>>> +		spin_lock_irqsave(&counter->lock, flags);
>>> +		val = res_counter_member(counter, member);
>>> +		*val = tmp;
>>> +		spin_unlock_irqrestore(&counter->lock, flags);
>>> +		ret = nbytes;
>>> +	}
>>>  out_free:
>>>  	kfree(buf);
>>>  out:
>>>  	return ret;
>>>  }
>>> +
>>> +
>>> +int res_counter_move_resource(struct res_counter *child,
>>> +				unsigned long long val,
>>> +	int (*callback)(struct res_counter *res, unsigned long long val),
>>> +	int retry)
>>> +{
>>> +	struct res_counter *parent = child->parent;
>>> +	unsigned long flags;
>>> +
>>> +	BUG_ON(!parent);
>>> +
>>> +	while (1) {
>>> +		spin_lock_irqsave(&parent->lock, flags);
>>> +		if (parent->usage + val < parent->limit) {
>>> +			parent->for_children += val;
>>> +			parent->usage += val;
>>> +			break;
>>> +		}
>>> +		spin_unlock_irqrestore(&parent->lock, flags);
>>> +
>>> +		if (!retry || !callback)
>>> +			goto failed;
>>> +		/* -1 means  infinite loop */
>>> +		if (retry != -1)
>>> +			--retry;
>> I don't like the idea of spinning in an infinite loop, I would prefer to fail
>> things instead of burning CPU cycles.
>>
> ok, will remove "-1" case.
> 
> 
> 
> 
>>> +		yield();
>>> +		callback(parent, val);
>> This code is not very understandable. Why do we yield before callback?
>>
> 
> yield() after callback() means that res_counter's state will be
> far different from the state after callback.
> So, we have to yield before call back and check res_coutner sooner.
> 

But does yield() get us any guarantees of seeing the state change?

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Balbir Singh]

KAMEZAWA Hiroyuki wrote:
> On Mon, 09 Jun 2008 15:18:47 +0530
> Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
> 
>> KAMEZAWA Hiroyuki wrote:
>>> A simple hard-wall hierarhcy support for res_counter.
>>>
>>> Changelog v2->v3
>>>  - changed the name and arguments of functions.
>>>  - rewrote to be read easily.
>>>  - named as HardWall hierarchy.
>>>
>>> This implements following model
>>>  - A cgroup's tree means hierarchy of resource.
>>>  - All child's resource is moved from its parents.
>>>  - The resource moved to children is charged as parent's usage.
>>>  - The resource moves when child->limit is changed.
>>>  - The sum of resource for children and its own usage is limited by "limit".
>>>
>>> This implies
>>>  - No dynamic automatic hierarhcy balancing in the kernel.
>>>  - Each resource is isolated completely.
>>>  - The kernel just supports resource-move-at-change-in-limit.
>>>  - The user (middle-ware) is responsible to make hierarhcy balanced well.
>> We'd definitely like to see a user level tool/application as a demo of how this
>> can be achieved.
>>
> I don't have one, now. I'll write one when I have time. Need now ?
> Hmm...maybe I(we) need some more patches to implement useful statistics,
> notifier to middlewares.
> 

Yes, we need more useful statistics.

> 
> 
>>>    Good balance can be achieved by changing limit from user land.
>>>
>>>
>>> Background:
>>>  Recently, there are popular resource isolation technique widely used,
>>>  i.e. Hardware-Virtualization. We can do hierarchical resource isolation
>>>  by using cgroup on it. But supporting hierarchy management in croups
>>>  has some advantages of performance, unity and costs of management.
>>>
>>>  There are good resource management in other OSs, they support some kind of
>>>  hierarchical resource management. We wonder what kind of hierarchy policy
>>>  is good for Linux. And there is an another point. Hierarchical system can be
>>>  implemented by the kernel and user-land co-operation.  So, there are various
>>>  choices to do in the kernel. Doing all in the kernel or export some proper
>>>  interfaces to the user-land. Middle-wares are tend to be used for management.
>>>  I hope there will be Open Source one.
>>>
>>>  At supporting hierarchy in cgroup, several aspects of characteristics of
>>>  policy of hierarchy can be considered. Some needs automatic balancing
>>>  between several groups.
>>>
>>>   - fairness    ... how fairness is kept under policy
>>>
>>>   - performance ... should be _fast_. multi-level resource balancing tend
>>>                  to use much amount of CPU and can cause soft lockup.
>>>
>>>   - predictability ... resource management are usually used for resource
>>>                  isolation. the kernel must not break the isolation and
>>>                  predictability of users against application's progress.
>>>
>>>   - flexibility ... some sophisticated dynamic resource balancing with
>>>  		 soft-limit is welcomed when the user doesn't want strict
>>> 		 resource isolation or when the user cannot estimate how much
>>> 		 they want correctly.
>> Soft limits has been on my plate for a while now. I'll take a crack at it. At
>> the moment the statistics is a bit of a worry, since users/administrators need
>> good statistics to take further action.
>>
> Yes, statistics is not enough now.
> 
> 
> 
>>> Hard Wall Hierarchy.
>>>
>>>  This patch implements a hard-wall model of hierarchy for resources.
>>>  Works well for users who want strict resource isolation.
>>>
>>>  This model allows the move of resource only between a parent and its children.
>>>  The resource is moved to a child when it declares the amount of resources to be
>>>  used. (by limit)
>> The other reason for preferring a shares based approach is that, the it will be
>> more in line with the CPU controllers interfaces.
>>
> 
> You have to think of the major difference of tha nature of CPU and Memory.
> We have to reclaim the resource with some feedbacks among sevral cgroups.
> But ok, if it's can be implemented in simple way.
> I have no objections if cost is very low. My concern is only performance.
> (and maintenance)
> 

True, I don't see hierarchy as adding too much additional cost.

> 
>>> Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
>>> ===================================================================
>>> --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
>>> +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
>>> @@ -38,6 +38,16 @@ struct res_counter {
>>>  	 * the number of unsuccessful attempts to consume the resource
>>>  	 */
>>>  	unsigned long long failcnt;
>>> +
>>> +	/*
>>> +	 * hierarchy support: the parent of this resource.
>>> +	 */
>>> +	struct res_counter *parent;
>>> +	/*
>>> +	 * the amount of resources assigned to children.
>>> +	 */
>>> +	unsigned long long for_children;
>>> +
>> I would prefer to use a better name, lent_out? reserved_for_children?
>> borrowed_by_children?
>>
> ok. use other names.
> 
> 
> 
>>>  	/*
>>>  	 * the lock to protect all of the above.
>>>  	 * the routines below consider this to be IRQ-safe
>>> @@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
>>>  ssize_t res_counter_read(struct res_counter *counter, int member,
>>>  		const char __user *buf, size_t nbytes, loff_t *pos,
>>>  		int (*read_strategy)(unsigned long long val, char *s));
>>> +
>>> +/*
>>> + * An interface for setting res_counter's member (ex. limit)
>>> + * the new parameter is passed by *buf and translated by write_strategy().
>>> + * Then, it is applied to member under the control of set_strategy().
>>> + * If write_strategy() and set_strategy() can be NULL. see res_counter.c
>>> + */
>>> +
>>>  ssize_t res_counter_write(struct res_counter *counter, int member,
>>> -		const char __user *buf, size_t nbytes, loff_t *pos,
>>> -		int (*write_strategy)(char *buf, unsigned long long *val));
>>> +	const char __user *buf, size_t nbytes, loff_t *pos,
>>> +        int (*write_strategy)(char *buf, unsigned long long *val),
>>> +	int (*set_strategy)(struct res_counter *res, unsigned long long val,
>>> +			    int what),
>>> +	);
>>>
>>>  /*
>>>   * the field descriptors. one for each member of res_counter
>>> @@ -76,15 +97,33 @@ enum {
>>>  	RES_MAX_USAGE,
>>>  	RES_LIMIT,
>>>  	RES_FAILCNT,
>>> +	RES_FOR_CHILDREN,
>> RES_BORROWED? RES_BORROWED_BY_CHILDREN?
>>
> ok, again.
> 
>>>  };
>>>
>>>  /*
>>>   * helpers for accounting
>>>   */
>>>
>>> +/*
>>> + * initialize res_counter.
>>> + * @counter : the counter
>>> + *
>>> + * initialize res_counter and set default limit to very big value(unlimited)
>>> + */
>>> +
>>>  void res_counter_init(struct res_counter *counter);
>>>
>>>  /*
>>> + * initialize res_counter under hierarchy.
>>> + * @counter : the counter
>>> + * @parent : the parent of the counter
>>> + *
>>> + * initialize res_counter and set default limit to 0. and set "parent".
>>> + */
>>> +void res_counter_init_hierarchy(struct res_counter *counter,
>>> +				struct res_counter *parent);
>>> +
>>> +/*
>>>   * charge - try to consume more resource.
>>>   *
>>>   * @counter: the counter
>>> @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
>>>  	cnt->failcnt = 0;
>>>  	spin_unlock_irqrestore(&cnt->lock, flags);
>>>  }
>>> +
>>> +/**
>>> + * Move resources from a parent to a child.
>>> + * At success,
>>> + *           parent->usage += val.
>>> + *           parent->for_children += val.
>>> + *           child->limit += val.
>>> + *
>>> + * @child:    an entity to set res->limit. The parent is child->parent.
>>> + * @val:      the amount of resource to be moved.
>>> + * @callback: called when the parent's free resource is not enough to be moved.
>>> + *            this can be NULL if no callback is necessary.
>>> + * @retry:    limit for the number of trying to callback.
>>> + *            -1 means infinite loop. At each retry, yield() is called.
>>> + * Returns 0 at success, !0 at failure.
>>> + *
>>> + * The callback returns 0 at success, !0 at failure.
>>> + *
>>> + */
>>> +
>>> +int res_counter_move_resource(struct res_counter *child,
>>> +	unsigned long long val,
>>> +        int (*callback)(struct res_counter *res, unsigned long long val),
>>> +	int retry);
>>> +
>>> +
>>> +/**
>>> + * Return resource to its parent.
>>> + * At success,
>>> + *           parent->usage  -= val.
>>> + *           parent->for_children -= val.
>>> + *           child->limit -= val.
>>> + *
>>> + * @child:   entry to resize. The parent is child->parent.
>>> + * @val  :   How much does child repay to parent ? -1 means 'all'
>>> + * @callback: A callback for decreasing resource usage of child before
>>> + *            returning. If NULL, just deceases child's limit.
>>> + * @retry:   # of retries at calling callback for freeing resource.
>>> + *            -1 means infinite loop. At each retry, yield() is called.
>>> + * Returns 0 at success.
>>> + */
>>> +
>>> +int res_counter_return_resource(struct res_counter *child,
>>> +	unsigned long long val,
>>> +	int (*callback)(struct res_counter *res, unsigned long long val),
>>> +	int retry);
>>> +
>>>  #endif
>>> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
>>> ===================================================================
>>> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
>>> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
>>> @@ -44,6 +44,13 @@ to work with it.
>>>   	Protects changes of the above values.
>>>
>>>
>>> + f. struct res_counter *parent
>>> +
>>> +	Parent res_counter under hierarchy.
>>> +
>>> + g. unsigned long long for_children
>>> +
>>> +	Resources assigned to children. This is included in usage.
>>>
>>>  2. Basic accounting routines
>>>
>>> @@ -179,3 +186,37 @@ counter fields. They are recommended to 
>>>      still can help with it).
>>>
>>>   c. Compile and run :)
>>> +
>>> +
>>> +6. Hierarchy
>>> + a. No Hierarchy
>>> +   each cgroup can use its own private resource.
>>> +
>>> + b. Hard-wall Hierarhcy
>>> +   A simple hierarchical tree system for resource isolation.
>>> +   Allows moving resources only between a parent and its children.
>>> +   A parent can move its resource to children and remember the amount to
>>> +   for_children member. A child can get new resource only from its parent.
>>> +   Limit of a child is the amount of resource which is moved from its parent.
>>> +
>> OK, after reading this I am totally sure I want a shares based interface. Limits
>> are not shared like this.
>>
>> A child and a parent should both be capable of having a limit of 1G, but they
>> could use different shares factors to govern, how much each children will get.
>> Doing it this way, breaks limit semantics.
>>
> Not easy to use in my point of view. Can we use 'share' in proper way 
> on no-swap machine ?
> 

Not sure I understand your question. Share represents the share of available
resources.

> 
>>> +   When add "val" to a child,
>>> +	parent->usage += val
>>> +	parent->for_children += val
>>> +	child->limit += val
>>> +   When a child returns its resource
>>> +	parent->usage -= val
>>> +	parent->for_children -= val
>>> +	child->limit -= val.
>>> +
>>> +   This implements resource isolation among each group. This works very well
>>> +   when you want to use strict resource isolation.
>>> +
>>> +   Usage Hint:
>>> +   This seems for static resource assignment but dynamic resource re-assignment
>>> +   can be done by resetting "limit" of groups. When you consider "limit" as
>>> +   the amount of allowed _current_ resource, a sophisticated resource management
>>> +   system based on strict resource isolation can be implemented.
>>> +
>>> +c. Soft-wall Hierarchy
>>> +   TBD.
>>> +
>>> Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
>>> ===================================================================
>>> --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
>>> +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
>>> @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
>>>  	counter->limit = (unsigned long long)LLONG_MAX;
>>>  }
>>>
>>> +void res_counter_init_hierarchy(struct res_counter *counter,
>>> +		struct res_counter *parent)
>>> +{
>>> +	spin_lock_init(&counter->lock);
>>> +	counter->limit = 0;
>>> +	counter->parent = parent;
>>> +}
>>> +
>>>  int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
>>>  {
>>>  	if (counter->usage + val > counter->limit) {
>>> @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
>>>  		return &counter->limit;
>>>  	case RES_FAILCNT:
>>>  		return &counter->failcnt;
>>> +	case RES_FOR_CHILDREN:
>>> +		return &counter->for_children;
>>>  	};
>>>
>>>  	BUG();
>>> @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
>>>
>>>  ssize_t res_counter_write(struct res_counter *counter, int member,
>>>  		const char __user *userbuf, size_t nbytes, loff_t *pos,
>>> -		int (*write_strategy)(char *st_buf, unsigned long long *val))
>>> +		int (*write_strategy)(char *st_buf, unsigned long long *val),
>>> +		int (*set_strategy)(struct res_counter *res,
>>> +			unsigned long long val, int what))
>>>  {
>>>  	int ret;
>>>  	char *buf, *end;
>>> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
>>>  		if (*end != '\0')
>>>  			goto out_free;
>>>  	}
>>> -	spin_lock_irqsave(&counter->lock, flags);
>>> -	val = res_counter_member(counter, member);
>>> -	*val = tmp;
>>> -	spin_unlock_irqrestore(&counter->lock, flags);
>>> -	ret = nbytes;
>>> +	if (set_strategy) {
>>> +		ret = set_strategy(res, tmp, member);
>>
>> I'm afraid, I don't understand the set_strategy and it's purpose.
>>
> Sorry. I'm now rewritten and removed this.
> 
> 

OK

> 
>>> +		if (!ret)
>>> +			ret = nbytes;
>>> +	} else {
>>> +		spin_lock_irqsave(&counter->lock, flags);
>>> +		val = res_counter_member(counter, member);
>>> +		*val = tmp;
>>> +		spin_unlock_irqrestore(&counter->lock, flags);
>>> +		ret = nbytes;
>>> +	}
>>>  out_free:
>>>  	kfree(buf);
>>>  out:
>>>  	return ret;
>>>  }
>>> +
>>> +
>>> +int res_counter_move_resource(struct res_counter *child,
>>> +				unsigned long long val,
>>> +	int (*callback)(struct res_counter *res, unsigned long long val),
>>> +	int retry)
>>> +{
>>> +	struct res_counter *parent = child->parent;
>>> +	unsigned long flags;
>>> +
>>> +	BUG_ON(!parent);
>>> +
>>> +	while (1) {
>>> +		spin_lock_irqsave(&parent->lock, flags);
>>> +		if (parent->usage + val < parent->limit) {
>>> +			parent->for_children += val;
>>> +			parent->usage += val;
>>> +			break;
>>> +		}
>>> +		spin_unlock_irqrestore(&parent->lock, flags);
>>> +
>>> +		if (!retry || !callback)
>>> +			goto failed;
>>> +		/* -1 means  infinite loop */
>>> +		if (retry != -1)
>>> +			--retry;
>> I don't like the idea of spinning in an infinite loop, I would prefer to fail
>> things instead of burning CPU cycles.
>>
> ok, will remove "-1" case.
> 
> 
> 
> 
>>> +		yield();
>>> +		callback(parent, val);
>> This code is not very understandable. Why do we yield before callback?
>>
> 
> yield() after callback() means that res_counter's state will be
> far different from the state after callback.
> So, we have to yield before call back and check res_coutner sooner.
> 

But does yield() get us any guarantees of seeing the state change?

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Randy Dunlap]

On Wed, 4 Jun 2008 14:01:53 +0900 KAMEZAWA Hiroyuki wrote:

> A simple hard-wall hierarhcy support for res_counter.
> 
> Changelog v2->v3
>  - changed the name and arguments of functions.
>  - rewrote to be read easily.
>  - named as HardWall hierarchy.

> ---
>  Documentation/controllers/resource_counter.txt |   41 +++++++++
>  include/linux/res_counter.h                    |   90 +++++++++++++++++++-
>  kernel/res_counter.c                           |  112 +++++++++++++++++++++++--
>  3 files changed, 235 insertions(+), 8 deletions(-)
> 
> Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
> +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> @@ -76,15 +97,33 @@ enum {
>  	RES_MAX_USAGE,
>  	RES_LIMIT,
>  	RES_FAILCNT,
> +	RES_FOR_CHILDREN,
>  };
>  
>  /*
>   * helpers for accounting
>   */
>  
> +/*
> + * initialize res_counter.
> + * @counter : the counter
> + *
> + * initialize res_counter and set default limit to very big value(unlimited)
> + */
> +
>  void res_counter_init(struct res_counter *counter);

For these non-static (non-private) functions, please use kernel-doc notation
(see Documentation/kernel-doc-nano-HOWTO.txt and/or examples in other source files).
Also, we prefer for the function documentation to be above its definition (implementation)
rather than above its declaration, so the kernel-doc should be moved to .c files
instead of living in .h files.


>  
>  /*
> + * initialize res_counter under hierarchy.
> + * @counter : the counter
> + * @parent : the parent of the counter
> + *
> + * initialize res_counter and set default limit to 0. and set "parent".
> + */
> +void res_counter_init_hierarchy(struct res_counter *counter,
> +				struct res_counter *parent);
> +
> +/*
>   * charge - try to consume more resource.
>   *
>   * @counter: the counter
> @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
>  	cnt->failcnt = 0;
>  	spin_unlock_irqrestore(&cnt->lock, flags);
>  }
> +
> +/**
> + * Move resources from a parent to a child.
> + * At success,
> + *           parent->usage += val.
> + *           parent->for_children += val.
> + *           child->limit += val.
> + *
> + * @child:    an entity to set res->limit. The parent is child->parent.
> + * @val:      the amount of resource to be moved.
> + * @callback: called when the parent's free resource is not enough to be moved.
> + *            this can be NULL if no callback is necessary.
> + * @retry:    limit for the number of trying to callback.
> + *            -1 means infinite loop. At each retry, yield() is called.
> + * Returns 0 at success, !0 at failure.
> + *
> + * The callback returns 0 at success, !0 at failure.
> + *
> + */
> +
> +int res_counter_move_resource(struct res_counter *child,
> +	unsigned long long val,
> +        int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry);
> +
> +
> +/**
> + * Return resource to its parent.
> + * At success,
> + *           parent->usage  -= val.
> + *           parent->for_children -= val.
> + *           child->limit -= val.
> + *
> + * @child:   entry to resize. The parent is child->parent.
> + * @val  :   How much does child repay to parent ? -1 means 'all'
> + * @callback: A callback for decreasing resource usage of child before
> + *            returning. If NULL, just deceases child's limit.
> + * @retry:   # of retries at calling callback for freeing resource.
> + *            -1 means infinite loop. At each retry, yield() is called.
> + * Returns 0 at success.
> + */
> +
> +int res_counter_return_resource(struct res_counter *child,
> +	unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry);
> +
>  #endif
> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> @@ -179,3 +186,37 @@ counter fields. They are recommended to 
>      still can help with it).
>  
>   c. Compile and run :)
> +
> +
> +6. Hierarchy
> + a. No Hierarchy
> +   each cgroup can use its own private resource.
> +
> + b. Hard-wall Hierarhcy
> +   A simple hierarchical tree system for resource isolation.
> +   Allows moving resources only between a parent and its children.
> +   A parent can move its resource to children and remember the amount to
> +   for_children member. A child can get new resource only from its parent.
> +   Limit of a child is the amount of resource which is moved from its parent.
> +
> +   When add "val" to a child,
> +	parent->usage += val
> +	parent->for_children += val
> +	child->limit += val
> +   When a child returns its resource
> +	parent->usage -= val
> +	parent->for_children -= val
> +	child->limit -= val.
> +
> +   This implements resource isolation among each group. This works very well
> +   when you want to use strict resource isolation.
> +
> +   Usage Hint:
> +   This seems for static resource assignment but dynamic resource re-assignment

           seems to be?

> +   can be done by resetting "limit" of groups. When you consider "limit" as
> +   the amount of allowed _current_ resource, a sophisticated resource management
> +   system based on strict resource isolation can be implemented.
> +
> +c. Soft-wall Hierarchy
> +   TBD.
> +
> Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
> +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
> @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
>  	counter->limit = (unsigned long long)LLONG_MAX;
>  }
>  
> +void res_counter_init_hierarchy(struct res_counter *counter,
> +		struct res_counter *parent)
> +{
> +	spin_lock_init(&counter->lock);
> +	counter->limit = 0;
> +	counter->parent = parent;
> +}
> +
>  int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
>  {
>  	if (counter->usage + val > counter->limit) {
> @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
>  		return &counter->limit;
>  	case RES_FAILCNT:
>  		return &counter->failcnt;
> +	case RES_FOR_CHILDREN:
> +		return &counter->for_children;
>  	};
>  
>  	BUG();
> @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
>  
>  ssize_t res_counter_write(struct res_counter *counter, int member,
>  		const char __user *userbuf, size_t nbytes, loff_t *pos,
> -		int (*write_strategy)(char *st_buf, unsigned long long *val))
> +		int (*write_strategy)(char *st_buf, unsigned long long *val),
> +		int (*set_strategy)(struct res_counter *res,
> +			unsigned long long val, int what))
>  {
>  	int ret;
>  	char *buf, *end;
> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
>  		if (*end != '\0')
>  			goto out_free;
>  	}
> -	spin_lock_irqsave(&counter->lock, flags);
> -	val = res_counter_member(counter, member);
> -	*val = tmp;
> -	spin_unlock_irqrestore(&counter->lock, flags);
> -	ret = nbytes;
> +	if (set_strategy) {
> +		ret = set_strategy(res, tmp, member);
> +		if (!ret)
> +			ret = nbytes;
> +	} else {
> +		spin_lock_irqsave(&counter->lock, flags);
> +		val = res_counter_member(counter, member);
> +		*val = tmp;
> +		spin_unlock_irqrestore(&counter->lock, flags);
> +		ret = nbytes;
> +	}
>  out_free:
>  	kfree(buf);
>  out:
>  	return ret;
>  }
> +
> +
> +int res_counter_move_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{
> +	struct res_counter *parent = child->parent;
> +	unsigned long flags;
> +
> +	BUG_ON(!parent);
> +
> +	while (1) {
> +		spin_lock_irqsave(&parent->lock, flags);
> +		if (parent->usage + val < parent->limit) {
> +			parent->for_children += val;
> +			parent->usage += val;
> +			break;
> +		}
> +		spin_unlock_irqrestore(&parent->lock, flags);
> +
> +		if (!retry || !callback)
> +			goto failed;
> +		/* -1 means  infinite loop */
> +		if (retry != -1)
> +			--retry;
> +		yield();
> +		callback(parent, val);
> +	}
> +	spin_unlock_irqrestore(&parent->lock, flags);
> +
> +	spin_lock_irqsave(&child->lock, flags);
> +	child->limit += val;
> +	spin_unlock_irqrestore(&child->lock, flags);
> +	return 0;
> +fail:
> +	return 1;
> +}
> +
> +
> +int res_counter_return_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{
> +	unsigned long flags;
> +	struct res_counter *parent = child->parent;
> +
> +	BUG_ON(!parent);
> +
> +	while (1) {
> +		spin_lock_irqsave(&child->lock, flags);
> +		if (val == (unsigned long long) -1) {
> +			val = child->limit;
> +			child->limit = 0;
> +			break;
> +		} else if (child->usage <= child->limit - val) {
> +			child->limit -= val;
> +			break;
> +		}
> +		spin_unlock_irqrestore(&child->lock, flags);
> +
> +		if (!retry)
> +			goto fail;
> +		/* -1 means infinite loop */
> +		if (retry != -1)
> +			--retry;
> +		yield();
> +		callback(parent, val);
> +	}
> +	spin_unlock_irqrestore(&child->lock, flags);
> +
> +	spin_lock_irqsave(&parent->lock, flags);
> +	BUG_ON(parent->for_children < val);
> +	BUG_ON(parent->usage < val);
> +	parent->for_children -= val;
> +	parent->usage -= val;
> +	spin_unlock_irqrestore(&parent->lock, flags);
> +	return 0;
> +fail:
> +	return 1;
> +}
> --


---
~Randy
'"Daemon' is an old piece of jargon from the UNIX operating system,
where it referred to a piece of low-level utility software, a
fundamental part of the operating system."

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[Randy Dunlap]

On Wed, 4 Jun 2008 14:01:53 +0900 KAMEZAWA Hiroyuki wrote:

> A simple hard-wall hierarhcy support for res_counter.
> 
> Changelog v2->v3
>  - changed the name and arguments of functions.
>  - rewrote to be read easily.
>  - named as HardWall hierarchy.

> ---
>  Documentation/controllers/resource_counter.txt |   41 +++++++++
>  include/linux/res_counter.h                    |   90 +++++++++++++++++++-
>  kernel/res_counter.c                           |  112 +++++++++++++++++++++++--
>  3 files changed, 235 insertions(+), 8 deletions(-)
> 
> Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
> +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
> @@ -76,15 +97,33 @@ enum {
>  	RES_MAX_USAGE,
>  	RES_LIMIT,
>  	RES_FAILCNT,
> +	RES_FOR_CHILDREN,
>  };
>  
>  /*
>   * helpers for accounting
>   */
>  
> +/*
> + * initialize res_counter.
> + * @counter : the counter
> + *
> + * initialize res_counter and set default limit to very big value(unlimited)
> + */
> +
>  void res_counter_init(struct res_counter *counter);

For these non-static (non-private) functions, please use kernel-doc notation
(see Documentation/kernel-doc-nano-HOWTO.txt and/or examples in other source files).
Also, we prefer for the function documentation to be above its definition (implementation)
rather than above its declaration, so the kernel-doc should be moved to .c files
instead of living in .h files.


>  
>  /*
> + * initialize res_counter under hierarchy.
> + * @counter : the counter
> + * @parent : the parent of the counter
> + *
> + * initialize res_counter and set default limit to 0. and set "parent".
> + */
> +void res_counter_init_hierarchy(struct res_counter *counter,
> +				struct res_counter *parent);
> +
> +/*
>   * charge - try to consume more resource.
>   *
>   * @counter: the counter
> @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
>  	cnt->failcnt = 0;
>  	spin_unlock_irqrestore(&cnt->lock, flags);
>  }
> +
> +/**
> + * Move resources from a parent to a child.
> + * At success,
> + *           parent->usage += val.
> + *           parent->for_children += val.
> + *           child->limit += val.
> + *
> + * @child:    an entity to set res->limit. The parent is child->parent.
> + * @val:      the amount of resource to be moved.
> + * @callback: called when the parent's free resource is not enough to be moved.
> + *            this can be NULL if no callback is necessary.
> + * @retry:    limit for the number of trying to callback.
> + *            -1 means infinite loop. At each retry, yield() is called.
> + * Returns 0 at success, !0 at failure.
> + *
> + * The callback returns 0 at success, !0 at failure.
> + *
> + */
> +
> +int res_counter_move_resource(struct res_counter *child,
> +	unsigned long long val,
> +        int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry);
> +
> +
> +/**
> + * Return resource to its parent.
> + * At success,
> + *           parent->usage  -= val.
> + *           parent->for_children -= val.
> + *           child->limit -= val.
> + *
> + * @child:   entry to resize. The parent is child->parent.
> + * @val  :   How much does child repay to parent ? -1 means 'all'
> + * @callback: A callback for decreasing resource usage of child before
> + *            returning. If NULL, just deceases child's limit.
> + * @retry:   # of retries at calling callback for freeing resource.
> + *            -1 means infinite loop. At each retry, yield() is called.
> + * Returns 0 at success.
> + */
> +
> +int res_counter_return_resource(struct res_counter *child,
> +	unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry);
> +
>  #endif
> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> @@ -179,3 +186,37 @@ counter fields. They are recommended to 
>      still can help with it).
>  
>   c. Compile and run :)
> +
> +
> +6. Hierarchy
> + a. No Hierarchy
> +   each cgroup can use its own private resource.
> +
> + b. Hard-wall Hierarhcy
> +   A simple hierarchical tree system for resource isolation.
> +   Allows moving resources only between a parent and its children.
> +   A parent can move its resource to children and remember the amount to
> +   for_children member. A child can get new resource only from its parent.
> +   Limit of a child is the amount of resource which is moved from its parent.
> +
> +   When add "val" to a child,
> +	parent->usage += val
> +	parent->for_children += val
> +	child->limit += val
> +   When a child returns its resource
> +	parent->usage -= val
> +	parent->for_children -= val
> +	child->limit -= val.
> +
> +   This implements resource isolation among each group. This works very well
> +   when you want to use strict resource isolation.
> +
> +   Usage Hint:
> +   This seems for static resource assignment but dynamic resource re-assignment

           seems to be?

> +   can be done by resetting "limit" of groups. When you consider "limit" as
> +   the amount of allowed _current_ resource, a sophisticated resource management
> +   system based on strict resource isolation can be implemented.
> +
> +c. Soft-wall Hierarchy
> +   TBD.
> +
> Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
> ===================================================================
> --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
> +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
> @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
>  	counter->limit = (unsigned long long)LLONG_MAX;
>  }
>  
> +void res_counter_init_hierarchy(struct res_counter *counter,
> +		struct res_counter *parent)
> +{
> +	spin_lock_init(&counter->lock);
> +	counter->limit = 0;
> +	counter->parent = parent;
> +}
> +
>  int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
>  {
>  	if (counter->usage + val > counter->limit) {
> @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
>  		return &counter->limit;
>  	case RES_FAILCNT:
>  		return &counter->failcnt;
> +	case RES_FOR_CHILDREN:
> +		return &counter->for_children;
>  	};
>  
>  	BUG();
> @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
>  
>  ssize_t res_counter_write(struct res_counter *counter, int member,
>  		const char __user *userbuf, size_t nbytes, loff_t *pos,
> -		int (*write_strategy)(char *st_buf, unsigned long long *val))
> +		int (*write_strategy)(char *st_buf, unsigned long long *val),
> +		int (*set_strategy)(struct res_counter *res,
> +			unsigned long long val, int what))
>  {
>  	int ret;
>  	char *buf, *end;
> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
>  		if (*end != '\0')
>  			goto out_free;
>  	}
> -	spin_lock_irqsave(&counter->lock, flags);
> -	val = res_counter_member(counter, member);
> -	*val = tmp;
> -	spin_unlock_irqrestore(&counter->lock, flags);
> -	ret = nbytes;
> +	if (set_strategy) {
> +		ret = set_strategy(res, tmp, member);
> +		if (!ret)
> +			ret = nbytes;
> +	} else {
> +		spin_lock_irqsave(&counter->lock, flags);
> +		val = res_counter_member(counter, member);
> +		*val = tmp;
> +		spin_unlock_irqrestore(&counter->lock, flags);
> +		ret = nbytes;
> +	}
>  out_free:
>  	kfree(buf);
>  out:
>  	return ret;
>  }
> +
> +
> +int res_counter_move_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{
> +	struct res_counter *parent = child->parent;
> +	unsigned long flags;
> +
> +	BUG_ON(!parent);
> +
> +	while (1) {
> +		spin_lock_irqsave(&parent->lock, flags);
> +		if (parent->usage + val < parent->limit) {
> +			parent->for_children += val;
> +			parent->usage += val;
> +			break;
> +		}
> +		spin_unlock_irqrestore(&parent->lock, flags);
> +
> +		if (!retry || !callback)
> +			goto failed;
> +		/* -1 means  infinite loop */
> +		if (retry != -1)
> +			--retry;
> +		yield();
> +		callback(parent, val);
> +	}
> +	spin_unlock_irqrestore(&parent->lock, flags);
> +
> +	spin_lock_irqsave(&child->lock, flags);
> +	child->limit += val;
> +	spin_unlock_irqrestore(&child->lock, flags);
> +	return 0;
> +fail:
> +	return 1;
> +}
> +
> +
> +int res_counter_return_resource(struct res_counter *child,
> +				unsigned long long val,
> +	int (*callback)(struct res_counter *res, unsigned long long val),
> +	int retry)
> +{
> +	unsigned long flags;
> +	struct res_counter *parent = child->parent;
> +
> +	BUG_ON(!parent);
> +
> +	while (1) {
> +		spin_lock_irqsave(&child->lock, flags);
> +		if (val == (unsigned long long) -1) {
> +			val = child->limit;
> +			child->limit = 0;
> +			break;
> +		} else if (child->usage <= child->limit - val) {
> +			child->limit -= val;
> +			break;
> +		}
> +		spin_unlock_irqrestore(&child->lock, flags);
> +
> +		if (!retry)
> +			goto fail;
> +		/* -1 means infinite loop */
> +		if (retry != -1)
> +			--retry;
> +		yield();
> +		callback(parent, val);
> +	}
> +	spin_unlock_irqrestore(&child->lock, flags);
> +
> +	spin_lock_irqsave(&parent->lock, flags);
> +	BUG_ON(parent->for_children < val);
> +	BUG_ON(parent->usage < val);
> +	parent->for_children -= val;
> +	parent->usage -= val;
> +	spin_unlock_irqrestore(&parent->lock, flags);
> +	return 0;
> +fail:
> +	return 1;
> +}
> --


---
~Randy
'"Daemon' is an old piece of jargon from the UNIX operating system,
where it referred to a piece of low-level utility software, a
fundamental part of the operating system."

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Wed, 11 Jun 2008 16:24:27 -0700
Randy Dunlap <randy.dunlap@oracle.com> wrote:
> >  /*
> >   * helpers for accounting
> >   */
> >  
> > +/*
> > + * initialize res_counter.
> > + * @counter : the counter
> > + *
> > + * initialize res_counter and set default limit to very big value(unlimited)
> > + */
> > +
> >  void res_counter_init(struct res_counter *counter);
> 
> For these non-static (non-private) functions, please use kernel-doc notation
> (see Documentation/kernel-doc-nano-HOWTO.txt and/or examples in other source files).
> Also, we prefer for the function documentation to be above its definition (implementation)
> rather than above its declaration, so the kernel-doc should be moved to .c files
> instead of living in .h files.
> 
Ah, sorry. I'll do so in the next version. Maybe I should move other comments
in res_counter.h to res_counter.c


> 
> >  
> >  /*
> > + * initialize res_counter under hierarchy.
> > + * @counter : the counter
> > + * @parent : the parent of the counter
> > + *
> > + * initialize res_counter and set default limit to 0. and set "parent".
> > + */
> > +void res_counter_init_hierarchy(struct res_counter *counter,
> > +				struct res_counter *parent);
> > +
> > +/*
> >   * charge - try to consume more resource.
> >   *
> >   * @counter: the counter
> > @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
> >  	cnt->failcnt = 0;
> >  	spin_unlock_irqrestore(&cnt->lock, flags);
> >  }
> > +
> > +/**
> > + * Move resources from a parent to a child.
> > + * At success,
> > + *           parent->usage += val.
> > + *           parent->for_children += val.
> > + *           child->limit += val.
> > + *
> > + * @child:    an entity to set res->limit. The parent is child->parent.
> > + * @val:      the amount of resource to be moved.
> > + * @callback: called when the parent's free resource is not enough to be moved.
> > + *            this can be NULL if no callback is necessary.
> > + * @retry:    limit for the number of trying to callback.
> > + *            -1 means infinite loop. At each retry, yield() is called.
> > + * Returns 0 at success, !0 at failure.
> > + *
> > + * The callback returns 0 at success, !0 at failure.
> > + *
> > + */
> > +
> > +int res_counter_move_resource(struct res_counter *child,
> > +	unsigned long long val,
> > +        int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry);
> > +
> > +
> > +/**
> > + * Return resource to its parent.
> > + * At success,
> > + *           parent->usage  -= val.
> > + *           parent->for_children -= val.
> > + *           child->limit -= val.
> > + *
> > + * @child:   entry to resize. The parent is child->parent.
> > + * @val  :   How much does child repay to parent ? -1 means 'all'
> > + * @callback: A callback for decreasing resource usage of child before
> > + *            returning. If NULL, just deceases child's limit.
> > + * @retry:   # of retries at calling callback for freeing resource.
> > + *            -1 means infinite loop. At each retry, yield() is called.
> > + * Returns 0 at success.
> > + */
> > +
> > +int res_counter_return_resource(struct res_counter *child,
> > +	unsigned long long val,
> > +	int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry);
> > +
> >  #endif
> > Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> > +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > @@ -179,3 +186,37 @@ counter fields. They are recommended to 
> >      still can help with it).
> >  
> >   c. Compile and run :)
> > +
> > +
> > +6. Hierarchy
> > + a. No Hierarchy
> > +   each cgroup can use its own private resource.
> > +
> > + b. Hard-wall Hierarhcy
> > +   A simple hierarchical tree system for resource isolation.
> > +   Allows moving resources only between a parent and its children.
> > +   A parent can move its resource to children and remember the amount to
> > +   for_children member. A child can get new resource only from its parent.
> > +   Limit of a child is the amount of resource which is moved from its parent.
> > +
> > +   When add "val" to a child,
> > +	parent->usage += val
> > +	parent->for_children += val
> > +	child->limit += val
> > +   When a child returns its resource
> > +	parent->usage -= val
> > +	parent->for_children -= val
> > +	child->limit -= val.
> > +
> > +   This implements resource isolation among each group. This works very well
> > +   when you want to use strict resource isolation.
> > +
> > +   Usage Hint:
> > +   This seems for static resource assignment but dynamic resource re-assignment
> 
>            seems to be?
> 
will fix.

Thank you for review!

Thanks,
-Kame

Re: [RFC][PATCH 1/2] memcg: res_counter hierarchy

[KAMEZAWA Hiroyuki]

On Wed, 11 Jun 2008 16:24:27 -0700
Randy Dunlap <randy.dunlap@oracle.com> wrote:
> >  /*
> >   * helpers for accounting
> >   */
> >  
> > +/*
> > + * initialize res_counter.
> > + * @counter : the counter
> > + *
> > + * initialize res_counter and set default limit to very big value(unlimited)
> > + */
> > +
> >  void res_counter_init(struct res_counter *counter);
> 
> For these non-static (non-private) functions, please use kernel-doc notation
> (see Documentation/kernel-doc-nano-HOWTO.txt and/or examples in other source files).
> Also, we prefer for the function documentation to be above its definition (implementation)
> rather than above its declaration, so the kernel-doc should be moved to .c files
> instead of living in .h files.
> 
Ah, sorry. I'll do so in the next version. Maybe I should move other comments
in res_counter.h to res_counter.c


> 
> >  
> >  /*
> > + * initialize res_counter under hierarchy.
> > + * @counter : the counter
> > + * @parent : the parent of the counter
> > + *
> > + * initialize res_counter and set default limit to 0. and set "parent".
> > + */
> > +void res_counter_init_hierarchy(struct res_counter *counter,
> > +				struct res_counter *parent);
> > +
> > +/*
> >   * charge - try to consume more resource.
> >   *
> >   * @counter: the counter
> > @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
> >  	cnt->failcnt = 0;
> >  	spin_unlock_irqrestore(&cnt->lock, flags);
> >  }
> > +
> > +/**
> > + * Move resources from a parent to a child.
> > + * At success,
> > + *           parent->usage += val.
> > + *           parent->for_children += val.
> > + *           child->limit += val.
> > + *
> > + * @child:    an entity to set res->limit. The parent is child->parent.
> > + * @val:      the amount of resource to be moved.
> > + * @callback: called when the parent's free resource is not enough to be moved.
> > + *            this can be NULL if no callback is necessary.
> > + * @retry:    limit for the number of trying to callback.
> > + *            -1 means infinite loop. At each retry, yield() is called.
> > + * Returns 0 at success, !0 at failure.
> > + *
> > + * The callback returns 0 at success, !0 at failure.
> > + *
> > + */
> > +
> > +int res_counter_move_resource(struct res_counter *child,
> > +	unsigned long long val,
> > +        int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry);
> > +
> > +
> > +/**
> > + * Return resource to its parent.
> > + * At success,
> > + *           parent->usage  -= val.
> > + *           parent->for_children -= val.
> > + *           child->limit -= val.
> > + *
> > + * @child:   entry to resize. The parent is child->parent.
> > + * @val  :   How much does child repay to parent ? -1 means 'all'
> > + * @callback: A callback for decreasing resource usage of child before
> > + *            returning. If NULL, just deceases child's limit.
> > + * @retry:   # of retries at calling callback for freeing resource.
> > + *            -1 means infinite loop. At each retry, yield() is called.
> > + * Returns 0 at success.
> > + */
> > +
> > +int res_counter_return_resource(struct res_counter *child,
> > +	unsigned long long val,
> > +	int (*callback)(struct res_counter *res, unsigned long long val),
> > +	int retry);
> > +
> >  #endif
> > Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > ===================================================================
> > --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
> > +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
> > @@ -179,3 +186,37 @@ counter fields. They are recommended to 
> >      still can help with it).
> >  
> >   c. Compile and run :)
> > +
> > +
> > +6. Hierarchy
> > + a. No Hierarchy
> > +   each cgroup can use its own private resource.
> > +
> > + b. Hard-wall Hierarhcy
> > +   A simple hierarchical tree system for resource isolation.
> > +   Allows moving resources only between a parent and its children.
> > +   A parent can move its resource to children and remember the amount to
> > +   for_children member. A child can get new resource only from its parent.
> > +   Limit of a child is the amount of resource which is moved from its parent.
> > +
> > +   When add "val" to a child,
> > +	parent->usage += val
> > +	parent->for_children += val
> > +	child->limit += val
> > +   When a child returns its resource
> > +	parent->usage -= val
> > +	parent->for_children -= val
> > +	child->limit -= val.
> > +
> > +   This implements resource isolation among each group. This works very well
> > +   when you want to use strict resource isolation.
> > +
> > +   Usage Hint:
> > +   This seems for static resource assignment but dynamic resource re-assignment
> 
>            seems to be?
> 
will fix.

Thank you for review!

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>