No limitation in physical memory in cgroups

No limitation in physical memory in cgroups

[anqin]

Hi all,

I would like to bind given physical memory to specific task group,
however it does not take effect in my experiments. Are there
something wrong in my experiments.

My experiment is done under the kernel 2.6.29.3 and I constructed
my task group by following commands:

a) In shell #1, prepare a bash :

# bash
# echo $$
2253

b) In shell #2, prepare the memory control via cgroupfs:

# mount -t cgroup cgroup /mnt/mycgrp
# cd /mn/mycgrp
# mkdir mycontainer
# echo 0 > mycontainer/cpuset.mems
# echo 0-1 > mycontainer/cpuset.cpus
# echo 2252 > mycontainer/tasks

# cat mycontainer/memory.usage_in_bytes
2875392
# echo 3000000 > mycontainer/memory.max_usage_in_bytes
# cat mycontainer/memory.max_usage_in_bytes
3002368


c) In Shell #1, run a memory consumer (in which, malloc() is called
to allocate memory and not free until program is existed) to allocate
500M memory:

# /tmp/memoy_consumer_program  500

In Shell #2, the used memory ascends from start point 2875392 when
program begins (from number presented in memory.usage_in_bytes),
but it return to start point when it touches the maximal boundary.

On the other hand, I also run the top to watch the memory hold by
memoy_consumer_program. In top, the memory (both virtual and rss
memory ) is always growing without any limitation.

Is this phenomenon the correct behaviors of memory cgroups?


Best Regards,

Anqin

Re: No limitation in physical memory in cgroups

[Daisuke Nishimura]

Hi.

Thank you for using memory cgroup :)

On Wed, 20 May 2009 13:52:29 +0800, anqin <anqin.qin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi all,
> 
> I would like to bind given physical memory to specific task group,
> however it does not take effect in my experiments. Are there
> something wrong in my experiments.
> 
> My experiment is done under the kernel 2.6.29.3 and I constructed
> my task group by following commands:
> 
> a) In shell #1, prepare a bash :
> 
> # bash
> # echo $$
> 2253
> 
> b) In shell #2, prepare the memory control via cgroupfs:
> 
> # mount -t cgroup cgroup /mnt/mycgrp
> # cd /mn/mycgrp
> # mkdir mycontainer
> # echo 0 > mycontainer/cpuset.mems
> # echo 0-1 > mycontainer/cpuset.cpus
> # echo 2252 > mycontainer/tasks
> 
> # cat mycontainer/memory.usage_in_bytes
> 2875392
> # echo 3000000 > mycontainer/memory.max_usage_in_bytes
> # cat mycontainer/memory.max_usage_in_bytes
> 3002368
> 
You need to write the limit to memory.limit_in_bytes.
max_usage_in_bytes means the max of the usage so far(you can reset it
by writing to it).

See Documentation/cgroups/memory.txt(it might be a bit old).


Thanks,
Daisuke Nishimura.

> 
> c) In Shell #1, run a memory consumer (in which, malloc() is called
> to allocate memory and not free until program is existed) to allocate
> 500M memory:
> 
> # /tmp/memoy_consumer_program  500
> 
> In Shell #2, the used memory ascends from start point 2875392 when
> program begins (from number presented in memory.usage_in_bytes),
> but it return to start point when it touches the maximal boundary.
> 
> On the other hand, I also run the top to watch the memory hold by
> memoy_consumer_program. In top, the memory (both virtual and rss
> memory ) is always growing without any limitation.
> 
> Is this phenomenon the correct behaviors of memory cgroups?
> 
> 
> Best Regards,
> 
> Anqin
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linux-foundation.org/mailman/listinfo/containers

Re: No limitation in physical memory in cgroups

[anqin]

>> b) In shell #2, prepare the memory control via cgroupfs:
>>
>> # mount -t cgroup cgroup /mnt/mycgrp
>> # cd /mn/mycgrp
>> # mkdir mycontainer
>> # echo 0 > mycontainer/cpuset.mems
>> # echo 0-1 > mycontainer/cpuset.cpus
>> # echo 2252 > mycontainer/tasks
>>
>> # cat mycontainer/memory.usage_in_bytes
>> 2875392
>> # echo 3000000 > mycontainer/memory.max_usage_in_bytes
>> # cat mycontainer/memory.max_usage_in_bytes
>> 3002368
>>
> You need to write the limit to memory.limit_in_bytes.
> max_usage_in_bytes means the max of the usage so far(you can reset it
> by writing to it).
>

Sorry for missing the information in above statement. In above experiment,
the memory.limit_in_bytes has also been set to same quota as memory.
max_usage_in_bytes. In other words,

# cat mycontainer/memory.limit_in_bytes
3002368
# cat mycontainer/memory.max_usage_in_bytes
3002368


Under these setting, I got the result as mentioned in last email.

Any help?



Best,

Anqin

Re: No limitation in physical memory in cgroups

[Li Zefan]

anqin wrote:
> Hi all,
> 
> I would like to bind given physical memory to specific task group,
> however it does not take effect in my experiments. Are there
> something wrong in my experiments.
> 
> My experiment is done under the kernel 2.6.29.3 and I constructed
> my task group by following commands:
> 
> a) In shell #1, prepare a bash :
> 
> # bash
> # echo $$
> 2253
> 

The pid of shell #1 is 2253.

> b) In shell #2, prepare the memory control via cgroupfs:
> 
> # mount -t cgroup cgroup /mnt/mycgrp
> # cd /mn/mycgrp
> # mkdir mycontainer
> # echo 0 > mycontainer/cpuset.mems
> # echo 0-1 > mycontainer/cpuset.cpus
> # echo 2252 > mycontainer/tasks
> 

but this is 2252 !!

> # cat mycontainer/memory.usage_in_bytes
> 2875392
> # echo 3000000 > mycontainer/memory.max_usage_in_bytes
> # cat mycontainer/memory.max_usage_in_bytes
> 3002368
> 
> 
> c) In Shell #1, run a memory consumer (in which, malloc() is called
> to allocate memory and not free until program is existed) to allocate
> 500M memory:
> 
> # /tmp/memoy_consumer_program  500
> 
> In Shell #2, the used memory ascends from start point 2875392 when
> program begins (from number presented in memory.usage_in_bytes),
> but it return to start point when it touches the maximal boundary.
> 
> On the other hand, I also run the top to watch the memory hold by
> memoy_consumer_program. In top, the memory (both virtual and rss
> memory ) is always growing without any limitation.
> 
> Is this phenomenon the correct behaviors of memory cgroups?
> 
> 
> Best Regards,

Re: No limitation in physical memory in cgroups

[anqin]

Thank you for answering.

>> # bash
>> # echo $$
>> 2253
>>
>
I have checked again the experiment.
And, the "2253" is typos, should be
"2252" as the one written as follows.

I have retried the experiment and same
result is given.

Any comment?  Does the limitation of
physical memory not support in 2.6.29.3?
Or, something missing maybe?


Thanks again,

Anqin

Re: No limitation in physical memory in cgroups

[KAMEZAWA Hiroyuki]

On Wed, 20 May 2009 18:35:34 +0800
anqin <anqin.qin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Thank you for answering.
> 
> >> # bash
> >> # echo $$
> >> 2253
> >>
> >
> I have checked again the experiment.
> And, the "2253" is typos, should be
> "2252" as the one written as follows.
> 
> I have retried the experiment and same
> result is given.
> 
> Any comment?  Does the limitation of
> physical memory not support in 2.6.29.3?
> Or, something missing maybe?
> 
> 
I wonder....

max_usage_in_bytes is not _reseted_ when you specified limit.
Then, please set limit before attaching task.

Sane sequence is.
1. mount
2. mkdir
3. set limit
4. attach.


Thanks,
-Kame

Re: No limitation in physical memory in cgroups

[anqin]

> I wonder....
>
> max_usage_in_bytes is not _reseted_ when you specified limit.
> Then, please set limit before attaching task.
>
> Sane sequence is.
> 1. mount
> 2. mkdir
> 3. set limit
> 4. attach.
>
>

I have tried again the experiment based on the above rules,
but I get the same result.

Again, my operations are as follows: (I have prepared three shell, #1,
#2 and #3)

a) In shell #1, prepare a bash :

# bash
# echo $$
11026

b) In shell #2, prepare the memory control via cgroupfs:

# mount -t cgroup cgroup /mnt/mycgrp2
# cd /mn/mycgrp2
# mkdir mycontainer2
# cd mycontainer2
# echo 0 > cpuset.mems
# echo 0-1 > cpuset.cpus
# echo 350000 > memory.limit_in_bytes

# cat memory.usage_in_bytes
0
# cat memory.max_usage_in_bytes
0
# cat memory.limit_in_bytes
352256

# echo 11026 > tasks
# cat tasks
11026


c) In Shell #1, run a memory consumer (in which, malloc() is called
to allocate memory and not free until program is existed) to allocate
500M memory:


# ./memconsume  500
Begin allocate block ... OK
allocate No.2 block ... OK
...

d) In Shell #3, run the top and see the memory consumption of memconsume:

# ps -ef | grep memconsume
root     11667 11647  0 00:40 pts/0    00:00:00 ./memconsume 500
root     11671 10536  0 00:56 pts/2    00:00:00 grep memconsume
# top -p 11667

e) In Sehll #2, I cat the usage_in_bytes to see the ascending
change of memory:

# cat memory.usage_in_bytes
143360
# cat memory.usage_in_bytes
159744
# cat memory.usage_in_bytes
233472
...
# cat memory.usage_in_bytes   //<--  hit the max limited value
352256
# cat memory.usage_in_bytes
258048
# cat memory.usage_in_bytes
282624
...
# cat memory.usage_in_bytes   // <--  hit the max limited value
348160
# cat memory.usage_in_bytes
122880
...

On the other hand, the top shown in Shell #3 that the virtual
memory is growing up to 450M, but RES is fluctuating between
615 and 497 and the %MEM always shows 0.2  and 0.3 in the whole
experiment.

Does the phenomena conform with the behaviors of memroy cgroups?
Currently, the memory cgroups can not kill the processes or limit
their further allocation of memory when the processes call for more
memory, right?

Maybe, something wrong in my test codes. The following is the
code of memconsume.c

------------------------  memconsume.c ----------------------------

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
	int i, t_total;
	char *p = NULL;

	if (argc != 2) {
		printf("Usage: %s  number (block = 1M) \n", argv[0]);
		return -1;
	}

	i = t_total = 0;

	t_total = atoi(argv[1]);
	if (t_total == 0) {
		printf("Usage: %s number (kb) \n", argv[0]);
		printf("\t- number is integer (kb)\n");
		return -1;
	}

	printf("Begin allocate memory ...");

	while (1) {
		p = NULL;
		p = (char *) malloc(1024 * 1024);
		if (p) {
			printf("OK\n");
			i++;
			if (i >= t_total)
				break;
		} else
			printf("fail\n");
		sleep(1);
		printf("allocate No.%d block ...", i + 1);
	}


	printf("End the consumption, total memory = %d * 1024 * 1024 bytes\n",  i);
	return 0;
}


------------------------------------------------------------------------------

Thanks again,

Anqin

Re: No limitation in physical memory in cgroups

[KAMEZAWA Hiroyuki]

On Thu, 21 May 2009 14:47:17 +0800
anqin <anqin.qin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> > I wonder....
> >
> > max_usage_in_bytes is not _reseted_ when you specified limit.
> > Then, please set limit before attaching task.
> >
> > Sane sequence is.
> > 1. mount
> > 2. mkdir
> > 3. set limit
> > 4. attach.
> >
> >
> 
> I have tried again the experiment based on the above rules,
> but I get the same result.
> 
> Again, my operations are as follows: (I have prepared three shell, #1,
> #2 and #3)
> 
> a) In shell #1, prepare a bash :
> 
> # bash
> # echo $$
> 11026
> 
> b) In shell #2, prepare the memory control via cgroupfs:
> 
> # mount -t cgroup cgroup /mnt/mycgrp2
> # cd /mn/mycgrp2
> # mkdir mycontainer2
> # cd mycontainer2
> # echo 0 > cpuset.mems
> # echo 0-1 > cpuset.cpus
> # echo 350000 > memory.limit_in_bytes
> 
> # cat memory.usage_in_bytes
> 0
> # cat memory.max_usage_in_bytes
> 0
> # cat memory.limit_in_bytes
> 352256
> 
> # echo 11026 > tasks
> # cat tasks
> 11026
> 
> 
> c) In Shell #1, run a memory consumer (in which, malloc() is called
> to allocate memory and not free until program is existed) to allocate
> 500M memory:
> 
> 
> # ./memconsume  500
> Begin allocate block ... OK
> allocate No.2 block ... OK
> ...
> 
> d) In Shell #3, run the top and see the memory consumption of memconsume:
> 
> # ps -ef | grep memconsume
> root     11667 11647  0 00:40 pts/0    00:00:00 ./memconsume 500
> root     11671 10536  0 00:56 pts/2    00:00:00 grep memconsume
> # top -p 11667
> 
> e) In Sehll #2, I cat the usage_in_bytes to see the ascending
> change of memory:
> 
> # cat memory.usage_in_bytes
> 143360
> # cat memory.usage_in_bytes
> 159744
> # cat memory.usage_in_bytes
> 233472
> ...
> # cat memory.usage_in_bytes   //<--  hit the max limited value
> 352256
> # cat memory.usage_in_bytes
> 258048
> # cat memory.usage_in_bytes
> 282624
> ...
> # cat memory.usage_in_bytes   // <--  hit the max limited value
> 348160
> # cat memory.usage_in_bytes
> 122880
> ...
> 
Ok, them, usage_in_bytes doesn't exceed limit. As expected.



> On the other hand, the top shown in Shell #3 that the virtual
> memory is growing up to 450M, but RES is fluctuating between
> 615 and 497 and the %MEM always shows 0.2  and 0.3 in the whole
> experiment.
> 
Ok.

> Does the phenomena conform with the behaviors of memroy cgroups?
> Currently, the memory cgroups can not kill the processes or limit
> their further allocation of memory when the processes call for more
> memory, right?
> 
right. memcg doesn't prevent malloc or mmap. but controls usage of "physical"
memory.

> Maybe, something wrong in my test codes. The following is the
> code of memconsume.c
> 




> ------------------------  memconsume.c ----------------------------
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> 
> int main(int argc, char *argv[])
> {
> 	int i, t_total;
> 	char *p = NULL;
> 
> 	if (argc != 2) {
> 		printf("Usage: %s  number (block = 1M) \n", argv[0]);
> 		return -1;
> 	}
> 
> 	i = t_total = 0;
> 
> 	t_total = atoi(argv[1]);
> 	if (t_total == 0) {
> 		printf("Usage: %s number (kb) \n", argv[0]);
> 		printf("\t- number is integer (kb)\n");
> 		return -1;
> 	}
> 
> 	printf("Begin allocate memory ...");
> 
> 	while (1) {
> 		p = NULL;
> 		p = (char *) malloc(1024 * 1024);
> 		if (p) {
> 			printf("OK\n");
> 			i++;
> 			if (i >= t_total)
> 				break;
> 		} else
> 			printf("fail\n");
> 		sleep(1);
> 		printf("allocate No.%d block ...", i + 1);
> 	}

You need to call memset() or some to use "physical" memory by page fault.
malloc just allocates virtual address. (and this program will start to use
swap.)


BTW, Try
# echo 300M > .../memory.memsw.limit_in_bytes 
If you config CGROUP_MEM_RES_CTRL_SWAP. You can limit mem+swap usage.

Thanks,
-Kame

> 
> 
> 	printf("End the consumption, total memory = %d * 1024 * 1024 bytes\n",  i);
> 	return 0;
> }
> 
> 
> ------------------------------------------------------------------------------
> 
> Thanks again,
> 
> Anqin
> 

Re: No limitation in physical memory in cgroups

[anqin]

> right. memcg doesn't prevent malloc or mmap. but controls usage of "physical"
> memory.
>

ok.

> You need to call memset() or some to use "physical" memory by page fault.
> malloc just allocates virtual address. (and this program will start to use
> swap.)
>
>
> BTW, Try
> # echo 300M > .../memory.memsw.limit_in_bytes
> If you config CGROUP_MEM_RES_CTRL_SWAP. You can limit mem+swap usage.
>

hmm...  is the functionality  one of those in kernel 2.6.29.3?
I have compiled the kernel after marked with all cgroup-related
options, including the

[*] Resource counters
[*]   Memory Resource Controller for Control Cgroups
   [*]   Memory Resource Controller Swap Extension (EXPERIMENTAL)

but I can not find the memory.memsw.limit_in_bytes exists in cgroupfs.
It seems the only file related to swap is memory.swappiness.

I grep the created .config file and find the

CGROUP_MEM_RES_CTRL_SWAP=y

I have searched the Document/cgroups but found nothing.
Any idea?


Thanks again,

Anqin

Re: No limitation in physical memory in cgroups

[Balbir Singh]

* anqin <anqin.qin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> [2009-05-21 19:56:56]:

> > right. memcg doesn't prevent malloc or mmap. but controls usage of "physical"
> > memory.
> >
> 
> ok.
> 
> > You need to call memset() or some to use "physical" memory by page fault.
> > malloc just allocates virtual address. (and this program will start to use
> > swap.)
> >
> >
> > BTW, Try
> > # echo 300M > .../memory.memsw.limit_in_bytes
> > If you config CGROUP_MEM_RES_CTRL_SWAP. You can limit mem+swap usage.
> >
> 
> hmm...  is the functionality  one of those in kernel 2.6.29.3?
> I have compiled the kernel after marked with all cgroup-related
> options, including the
> 
> [*] Resource counters
> [*]   Memory Resource Controller for Control Cgroups
>    [*]   Memory Resource Controller Swap Extension (EXPERIMENTAL)
> 
> but I can not find the memory.memsw.limit_in_bytes exists in cgroupfs.
> It seems the only file related to swap is memory.swappiness.
> 
> I grep the created .config file and find the
> 
> CGROUP_MEM_RES_CTRL_SWAP=y
> 
> I have searched the Document/cgroups but found nothing.
> Any idea?

Are you booting with noswapaccount as a boot parameter?

-- 
	Balbir

Re: No limitation in physical memory in cgroups

[anqin]

> Are you booting with noswapaccount as a boot parameter?
>

hmm....  I am not sure. I just compiled the kernel at default.
Where can I check the boot parameter? In /boot/grub/menu.list,
no the parameter yet.



Thanks,

Anqin

Re: No limitation in physical memory in cgroups

[KAMEZAWA Hiroyuki]

On Fri, 22 May 2009 10:31:39 +0800
anqin <anqin.qin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> > Are you booting with noswapaccount as a boot parameter?
> >
> 
> hmm....  I am not sure. I just compiled the kernel at default.
> Where can I check the boot parameter? In /boot/grub/menu.list,
> no the parameter yet.
> 
Hmm. When 

CONFIG_CGROUP_MEM_RES_CTLR=y
CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

And no special boot option, the memory cgroup directory will be.
==
[root@bluextal cgroup]# ls /cgroups/A/
memory.failcnt                   memory.memsw.usage_in_bytes
memory.force_empty               memory.stat
memory.limit_in_bytes            memory.swappiness
memory.max_usage_in_bytes        memory.usage_in_bytes
memory.memsw.failcnt             memory.use_hierarchy
memory.memsw.limit_in_bytes      notify_on_release
memory.memsw.max_usage_in_bytes  tasks
==

You can see memory.memsw.* for limiting mem+swap usage.
This memory.memsw. files disappears when you set "noswapaccount" boot option...

I can only say "counld you confirm again ?"
 config, the kernel version etc...

Sorry for no help.

Bye.
-Kame

Re: No limitation in physical memory in cgroups

[anqin]

I have recompiled the kernel and tested again.
When I set the limitation of swap, the test program
is killed when it want to apply more memory.

Thanks.


>> hmm....  I am not sure. I just compiled the kernel at default.
>> Where can I check the boot parameter? In /boot/grub/menu.list,
>> no the parameter yet.
>>
> Hmm. When
>
> CONFIG_CGROUP_MEM_RES_CTLR=y
> CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
>
> And no special boot option, the memory cgroup directory will be.
> ==
> [root@bluextal cgroup]# ls /cgroups/A/
> memory.failcnt                   memory.memsw.usage_in_bytes
> memory.force_empty               memory.stat
> memory.limit_in_bytes            memory.swappiness
> memory.max_usage_in_bytes        memory.usage_in_bytes
> memory.memsw.failcnt             memory.use_hierarchy
> memory.memsw.limit_in_bytes      notify_on_release
> memory.memsw.max_usage_in_bytes  tasks
> ==
>
> You can see memory.memsw.* for limiting mem+swap usage.
> This memory.memsw. files disappears when you set "noswapaccount" boot option...
>
> I can only say "counld you confirm again ?"
>  config, the kernel version etc...
>
> Sorry for no help.
>
> Bye.
> -Kame
>
>
>

Re: No limitation in physical memory in cgroups

[Balbir Singh]

* anqin <anqin.qin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> [2009-05-20 13:52:29]:

> Hi all,
> 
> I would like to bind given physical memory to specific task group,
> however it does not take effect in my experiments. Are there
> something wrong in my experiments.
> 
> My experiment is done under the kernel 2.6.29.3 and I constructed
> my task group by following commands:
> 
> a) In shell #1, prepare a bash :
> 
> # bash
> # echo $$
> 2253
> 
> b) In shell #2, prepare the memory control via cgroupfs:
> 
> # mount -t cgroup cgroup /mnt/mycgrp
> # cd /mn/mycgrp
> # mkdir mycontainer
> # echo 0 > mycontainer/cpuset.mems
> # echo 0-1 > mycontainer/cpuset.cpus
> # echo 2252 > mycontainer/tasks

OK, so you setup cpusets

> 
> # cat mycontainer/memory.usage_in_bytes
> 2875392
> # echo 3000000 > mycontainer/memory.max_usage_in_bytes
> # cat mycontainer/memory.max_usage_in_bytes
> 3002368

Where did you setup memory.limit_in_bytes?

> 
> 
> c) In Shell #1, run a memory consumer (in which, malloc() is called
> to allocate memory and not free until program is existed) to allocate
> 500M memory:
> 
> # /tmp/memoy_consumer_program  500
> 
> In Shell #2, the used memory ascends from start point 2875392 when
> program begins (from number presented in memory.usage_in_bytes),
> but it return to start point when it touches the maximal boundary.
> 
> On the other hand, I also run the top to watch the memory hold by
> memoy_consumer_program. In top, the memory (both virtual and rss
> memory ) is always growing without any limitation.
> 
> Is this phenomenon the correct behaviors of memory cgroups?
> 
> 
> Best Regards,
> 
> Anqin
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linux-foundation.org/mailman/listinfo/containers

-- 
	Balbir

Re: No limitation in physical memory in cgroups

[anqin]

>>
>> # cat mycontainer/memory.usage_in_bytes
>> 2875392
>> # echo 3000000 > mycontainer/memory.max_usage_in_bytes
>> # cat mycontainer/memory.max_usage_in_bytes
>> 3002368
>
> Where did you setup memory.limit_in_bytes?
>

Missing a command here. I have already run the command:

# echo 3000000 > mycontainer/memory.limit_in_bytes


as mentioned in previous email.