Re: cgroup memory limit problems

["Arkadiusz Miśkiewicz" <arekm-evZBlRFTdvA@public.gmane.org>]

On Wednesday 09 of November 2011, Balbir Singh wrote:
> 2011/11/10 Arkadiusz Miśkiewicz <arekm@maven.pl>:
> > Hi,
> > 
> > I have a machine with 6GB of ram and a cgroup for apache processes
> > limited to memory.limit_in_bytes = "5100M";
> >                memory.soft_limit_in_bytes = "5000M";
> 
> Can you please send the output of memory.stat?

Right now there is small number of users using this server (night here) so:

# cat /dev/cgroup/memory/somecgroup/httpd/memory.stat
cache 511426560
rss 2637795328
mapped_file 2129920
pgpgin 32233755
pgpgout 31509870
inactive_anon 720896
active_anon 2637271040
inactive_file 255959040
active_file 255234048
unevictable 0
hierarchical_memory_limit 5347737600
total_cache 511426560
total_rss 2637795328
total_mapped_file 2129920
total_pgpgin 32233755
total_pgpgout 31509870
total_inactive_anon 720896
total_active_anon 2637271040
total_inactive_file 255959040
total_active_file 255234048
total_unevictable 0

> 
> > Unfortunately when apache processes ate all ram assigned to their cgroup
> > load on whole machine jumps the roof.
> > 
> > cgroup aware OOM kicks in, kills one process and that doesn't help.
> > 
> > If I'm fast enough I notice and then apache processes require tons of
> > kill -9 (I'm doing "killall -9 apache" in a while (true) loop for
> > 20-30s) to get killed (and that not always succeeds - sometimes I'm
> > unable to kill these and I'm just doing sysrq u, s, b after few
> > minutes.. if I'm lucky. Sometimes I cannot do any command).
> > 
> > This all happens on 2.6.38.8 kernel.
> > http://ixion.pld-linux.org/~arekm/cgroup-eaten-memory-failure-1.txt
> > for kernel log. It ends with reboot of the machine.
> 
> From the logs
> 
> "Nov  9 20:48:53 tm2 kernel: [18300.349106] Task in /somecgroup/httpd
> killed as a result of limit of /somecgroup/httpd
> Nov  9 20:48:53 tm2 kernel: [18300.349110] memory: usage 5222400kB,
> limit 5222400kB, failcnt 282869
> Nov  9 20:48:53 tm2 kernel: [18300.349113] memory+swap: usage 0kB,
> limit 9007199254740991kB, failcnt 0"
> 
> It seems like you've disabled swap, is that correct?

Yes, this machine has no swap but the kernel has 

# zcat /proc/config.gz |grep CGROUP_MEM
CONFIG_CGROUP_MEM_RES_CTLR=y
CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
# CONFIG_CGROUP_MEM_RES_CTLR_SWAP_ENABLED is not set

+ ctlr swap wasn't enabled at boot.

> Balbir

ps. vger.kernel.org doesn't like my address (no idea why and postmaster@ is 
silent), so likely my reply won't reach cgroup@
-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/containers