* memcg with kmem limit doesn't recover after disk i/o causes limit to be hit @ 2014-04-16 15:46 Richard Davies 2014-04-18 15:59 ` Michal Hocko 0 siblings, 1 reply; 38+ messages in thread From: Richard Davies @ 2014-04-16 15:46 UTC (permalink / raw) To: cgroups, linux-mm Hi all, I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers. I feel there are probably two bugs here - the disk i/o is not successfully managed within the kmem limit - the cgroup never recovers, despite the untar i/o process exiting I'm happy to help with any further debugging or try patches. To replicate on Linux 3.14.0, run the following 6 commands: # mkdir -p /sys/fs/cgroup/test/ # cat /sys/fs/cgroup/cpuset.cpus > /sys/fs/cgroup/test/cpuset.cpus # cat /sys/fs/cgroup/cpuset.mems > /sys/fs/cgroup/test/cpuset.mems # echo $((1<<26)) >/sys/fs/cgroup/test/memory.kmem.limit_in_bytes # echo $$ > /sys/fs/cgroup/test/tasks # tar xfz linux-3.14.1.tar.gz Part way through the untar, the tar command exits after many errors: ... linux-3.14.1/include/linux/ima.h: Can't create 'linux-3.14.1/include/linux/ima.h' linux-3.14.1/include/linux/in.h: Can't create 'linux-3.14.1/include/linux/in.h' linux-3.14.1/include/linux/in6.h: Can't create 'linux-3.14.1/include/linux/in6.h' ... At the same time, many errors are logged in the kernel log, of the form: 14:58:45 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 14:58:45 kernel: cache: radix_tree_node(3:test2), object size: 560, buffer size: 568, default order: 2, min order: 0 14:58:45 kernel: node 0: slabs: 140, objs: 3920, free: 0 14:58:45 kernel: node 1: slabs: 464, objs: 12992, free: 0 and 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0 and 15:31:15 kernel: ENOMEM in journal_alloc_journal_head, retrying. 15:31:20 kernel: __slab_alloc: 6946855 callbacks suppressed 15:31:20 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8050) 15:31:20 kernel: cache: jbd2_journal_head(2:test), object size: 112, buffer size: 112, default order: 0, min order: 0 15:31:20 kernel: node 0: slabs: 80, objs: 2880, free: 0 15:31:20 kernel: node 1: slabs: 0, objs: 0, free: 0 After this, the bash prompt in the cgroup is unusable: # ls -bash: fork: Cannot allocate memory But typically the system outside the cgroup continues to work. All of the above happens every time that I run these 6 commands. I am happy to help with extra information on kernel configuration, but I hope that the above is sufficient for others to replicate. I'm also happy to try suggestions and patches. In addition, sometimes these 6 commands trigger more serious issues beyond the cgroup. I have copied a long kernel log of such an incident below. Thanks in advance for your help, Richard. 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff811017ad>] ? rcu_eqs_enter+0x70/0x83 15:11:36 kernel: [<ffffffff811017ce>] ? rcu_user_enter+0xe/0x10 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 0 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 24 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 54 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 7 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 3 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 80 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 183 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 46 15:11:36 kernel: active_anon:20251 inactive_anon:3312 isolated_anon:0 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 15:11:36 kernel: free:7963301 slab_reclaimable:19802 slab_unreclaimable:7992 15:11:36 kernel: mapped:7165 shmem:1531 pagetables:1575 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3390912kB min:9572kB low:11964kB high:14356kB active_anon:7148kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4836kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12257092kB min:35416kB low:44268kB high:53124kB active_anon:28044kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:12072kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16189300kB min:45076kB low:56344kB high:67612kB active_anon:45812kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 49*4kB (UEM) 62*8kB (U) 30*16kB (U) 12*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 2*2048kB (MR) 826*4096kB (MR) = 3390740kB 15:11:36 kernel: Node 0 Normal: 112*4kB (UE) 57*8kB (UE) 12*16kB (UEM) 32*32kB (UEM) 15*64kB (UEM) 5*128kB (UM) 3*256kB (UEM) 3*512kB (UEM) 2*1024kB (UM) 1*2048kB (M) 2990*4096kB (MR) = 12257160kB 15:11:36 kernel: Node 1 Normal: 95*4kB (UM) 8*8kB (U) 0*16kB 8*32kB (U) 6*64kB (UM) 10*128kB (UM) 2*256kB (EM) 2*512kB (UE) 0*1024kB 1*2048kB (M) 3951*4096kB (MR) = 16189244kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 183020 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe 15:11:36 kernel: [ 4353] 104 4353 88747 10609 59 0 0 mysqld 15:11:36 kernel: [ 4354] -2 4354 1023 158 8 0 0 logger 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 627 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: Out of memory: Kill process 4353 (mysqld) score 0 or sacrifice child 15:11:36 kernel: Killed process 4353 (mysqld) total-vm:354988kB, anon-rss:35704kB, file-rss:6732kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 45 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 57 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 131 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 174 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 113 15:11:36 kernel: active_anon:11418 inactive_anon:3312 isolated_anon:0 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 15:11:36 kernel: free:7971790 slab_reclaimable:19802 slab_unreclaimable:7992 15:11:36 kernel: mapped:5661 shmem:1531 pagetables:1478 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:6640kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16219896kB min:45076kB low:56344kB high:67612kB active_anon:15112kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:1780kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB 15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 55*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258116kB 15:11:36 kernel: Node 1 Normal: 245*4kB (UEM) 101*8kB (UEM) 21*16kB (UEM) 23*32kB (UEM) 19*64kB (UEM) 18*128kB (UEM) 9*256kB (M) 7*512kB (UEM) 2*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16220140kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 183020 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: Out of memory: Kill process 3988 (named) score 0 or sacrifice child 15:11:36 kernel: Killed process 3988 (named) total-vm:165728kB, anon-rss:15848kB, file-rss:2172kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 33 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 53 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 151 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 44 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 137 15:11:36 kernel: active_anon:11808 inactive_anon:3312 isolated_anon:0 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 15:11:36 kernel: free:7971490 slab_reclaimable:19802 slab_unreclaimable:7992 15:11:36 kernel: mapped:6534 shmem:1531 pagetables:1575 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10132kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16218696kB min:45076kB low:56344kB high:67612kB active_anon:16672kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB 15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 58*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258140kB 15:11:36 kernel: Node 1 Normal: 90*4kB (UEM) 64*8kB (UEM) 7*16kB (UE) 12*32kB (UEM) 15*64kB (UEM) 17*128kB (UEM) 7*256kB (M) 7*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16218776kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 183020 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5197] -2 5197 12302 1440 25 0 0 mysqld 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger 15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child 15:11:36 kernel: Killed process 4394 (postgres) total-vm:99840kB, anon-rss:1844kB, file-rss:1004kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 30 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 51 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 47 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173 15:11:36 kernel: active_anon:10570 inactive_anon:2937 isolated_anon:0 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 15:11:36 kernel: free:7972794 slab_reclaimable:19802 slab_unreclaimable:7927 15:11:36 kernel: mapped:6267 shmem:1531 pagetables:1575 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3395828kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:2592kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:3960kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12270500kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:5904kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10520kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16208948kB min:45076kB low:56344kB high:67612kB active_anon:26032kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 137*4kB (UEM) 92*8kB (UM) 49*16kB (UM) 26*32kB (UEM) 12*64kB (UM) 11*128kB (UEM) 3*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3395796kB 15:11:36 kernel: Node 0 Normal: 337*4kB (UM) 138*8kB (UEM) 47*16kB (UEM) 47*32kB (UEM) 22*64kB (UEM) 10*128kB (UM) 2*256kB (UE) 3*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12270820kB 15:11:36 kernel: Node 1 Normal: 44*4kB (UEM) 70*8kB (UE) 9*16kB (UEM) 9*32kB (U) 6*64kB (U) 10*128kB (UE) 0*256kB 3*512kB (UEM) 1*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16209168kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 183117 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 452 36 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5197] 104 5197 57326 4132 35 0 0 mysqld 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger 15:11:36 kernel: Out of memory: Kill process 5197 (mysqld) score 0 or sacrifice child 15:11:36 kernel: Killed process 5197 (mysqld) total-vm:231548kB, anon-rss:17736kB, file-rss:5256kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 67 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 197 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 82 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 2 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 156 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 164 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173 15:11:36 kernel: active_anon:7704 inactive_anon:1830 isolated_anon:0 15:11:36 kernel: active_file:29305 inactive_file:151670 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 15:11:36 kernel: free:7976547 slab_reclaimable:19802 slab_unreclaimable:7927 15:11:36 kernel: mapped:4180 shmem:220 pagetables:1478 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3397200kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:1024kB active_file:15260kB inactive_file:79780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2208kB shmem:112kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12273064kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:3304kB active_file:63984kB inactive_file:434284kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:3924kB shmem:448kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16220024kB min:45076kB low:56344kB high:67612kB active_anon:14568kB inactive_anon:2992kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:320kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 117*8kB (UEM) 56*16kB (UEM) 29*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397532kB 15:11:36 kernel: Node 0 Normal: 314*4kB (UEM) 156*8kB (UEM) 55*16kB (UEM) 51*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273176kB 15:11:36 kernel: Node 1 Normal: 276*4kB (UEM) 126*8kB (UEM) 31*16kB (UEM) 28*32kB (UEM) 20*64kB (UM) 20*128kB (UEM) 7*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16220592kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 181806 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 1271 35 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5202] -2 5202 2336 77 7 0 0 mysqld_safe 15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child 15:11:36 kernel: Killed process 4357 (postgres) total-vm:99116kB, anon-rss:1232kB, file-rss:3852kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 98 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 196 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 79 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 179 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 156 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 15:11:36 kernel: active_anon:6997 inactive_anon:1812 isolated_anon:0 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 15:11:36 kernel: free:7976969 slab_reclaimable:19834 slab_unreclaimable:7983 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12273056kB min:35416kB low:44268kB high:53124kB active_anon:13500kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16221640kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB 15:11:36 kernel: Node 0 Normal: 357*4kB (UEM) 164*8kB (UEM) 64*16kB (UEM) 52*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273588kB 15:11:36 kernel: Node 1 Normal: 408*4kB (UEM) 141*8kB (UEM) 45*16kB (UEM) 28*32kB (UEM) 19*64kB (UM) 22*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16221912kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 181749 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: Out of memory: Kill process 1999 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1999 (httpd) total-vm:329268kB, anon-rss:3272kB, file-rss:1208kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 75 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 159 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 15:11:36 kernel: active_anon:6430 inactive_anon:1812 isolated_anon:0 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 15:11:36 kernel: free:7977670 slab_reclaimable:19834 slab_unreclaimable:7983 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12275340kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16222160kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB 15:11:36 kernel: Node 0 Normal: 390*4kB (UEM) 183*8kB (UEM) 68*16kB (UEM) 56*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276112kB 15:11:36 kernel: Node 1 Normal: 431*4kB (UEM) 150*8kB (UEM) 46*16kB (UEM) 30*32kB (UEM) 19*64kB (UM) 23*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16222284kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 181749 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: Out of memory: Kill process 2001 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 2001 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1212kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 72 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 181 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 15:11:36 kernel: active_anon:5804 inactive_anon:1812 isolated_anon:0 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 15:11:36 kernel: free:7978396 slab_reclaimable:19834 slab_unreclaimable:7983 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12275600kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16224804kB min:45076kB low:56344kB high:67612kB active_anon:10240kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB 15:11:36 kernel: Node 0 Normal: 393*4kB (UEM) 180*8kB (UEM) 71*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276340kB 15:11:36 kernel: Node 1 Normal: 506*4kB (UEM) 161*8kB (UEM) 49*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16224960kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 181749 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: Out of memory: Kill process 2000 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 2000 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1208kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 68 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 183 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 15:11:36 kernel: active_anon:5143 inactive_anon:1812 isolated_anon:0 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10972kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB 15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB 15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 182*8kB (UEM) 50*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227520kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 181749 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1997 (elastic-poolio) total-vm:19684kB, anon-rss:276kB, file-rss:1212kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace: 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 15:11:36 kernel: Mem-Info: 15:11:36 kernel: Node 0 DMA per-cpu: 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 159 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 15:11:36 kernel: active_anon:5078 inactive_anon:1812 isolated_anon:0 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0 15:11:36 kernel: free_cma:0 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10712kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB 15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB 15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 198*8kB (UEM) 55*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227728kB 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 15:11:36 kernel: 181749 total pagecache pages 15:11:36 kernel: 0 pages in swap cache 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 15:11:36 kernel: Free swap = 16507900kB 15:11:36 kernel: Total swap = 16507900kB 15:11:36 kernel: 8388414 pages RAM 15:11:36 kernel: 0 pages HighMem/MovableOnly 15:11:36 kernel: 65682 pages reserved 15:11:36 kernel: 0 pages hwpoisoned 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1995 (httpd) total-vm:31888kB, anon-rss:952kB, file-rss:2344kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4551 (apache2) total-vm:299432kB, anon-rss:1896kB, file-rss:760kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4550 (apache2) total-vm:299440kB, anon-rss:1896kB, file-rss:756kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4548 (apache2) total-vm:75736kB, anon-rss:1648kB, file-rss:572kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4540 (apache2) total-vm:76004kB, anon-rss:1680kB, file-rss:1492kB 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child 15:11:37 kernel: Killed process 4386 (bash) total-vm:19872kB, anon-rss:484kB, file-rss:1472kB 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit 2014-04-16 15:46 memcg with kmem limit doesn't recover after disk i/o causes limit to be hit Richard Davies @ 2014-04-18 15:59 ` Michal Hocko 2014-04-18 17:57 ` Vladimir Davydov 0 siblings, 1 reply; 38+ messages in thread From: Michal Hocko @ 2014-04-18 15:59 UTC (permalink / raw) To: Richard Davies; +Cc: cgroups, linux-mm, Vladimir Davydov [CC Vladimir] On Wed 16-04-14 16:46:50, Richard Davies wrote: > Hi all, > > I have a simple reproducible test case in which untar in a memcg with a kmem > limit gets into trouble during heavy disk i/o (on ext3) and never properly > recovers. This is simplified from real world problems with heavy disk i/o > inside containers. > > I feel there are probably two bugs here > - the disk i/o is not successfully managed within the kmem limit > - the cgroup never recovers, despite the untar i/o process exiting > > I'm happy to help with any further debugging or try patches. > > > To replicate on Linux 3.14.0, run the following 6 commands: > > # mkdir -p /sys/fs/cgroup/test/ > # cat /sys/fs/cgroup/cpuset.cpus > /sys/fs/cgroup/test/cpuset.cpus > # cat /sys/fs/cgroup/cpuset.mems > /sys/fs/cgroup/test/cpuset.mems > # echo $((1<<26)) >/sys/fs/cgroup/test/memory.kmem.limit_in_bytes > # echo $$ > /sys/fs/cgroup/test/tasks > # tar xfz linux-3.14.1.tar.gz > > Part way through the untar, the tar command exits after many errors: > > ... > linux-3.14.1/include/linux/ima.h: Can't create 'linux-3.14.1/include/linux/ima.h' > linux-3.14.1/include/linux/in.h: Can't create 'linux-3.14.1/include/linux/in.h' > linux-3.14.1/include/linux/in6.h: Can't create 'linux-3.14.1/include/linux/in6.h' > ... > > At the same time, many errors are logged in the kernel log, of the form: > > 14:58:45 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) > 14:58:45 kernel: cache: radix_tree_node(3:test2), object size: 560, buffer size: 568, default order: 2, min order: 0 > 14:58:45 kernel: node 0: slabs: 140, objs: 3920, free: 0 > 14:58:45 kernel: node 1: slabs: 464, objs: 12992, free: 0 > > and > > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) > 15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0 > 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0 > 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0 > > and > > 15:31:15 kernel: ENOMEM in journal_alloc_journal_head, retrying. > 15:31:20 kernel: __slab_alloc: 6946855 callbacks suppressed > 15:31:20 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8050) > 15:31:20 kernel: cache: jbd2_journal_head(2:test), object size: 112, buffer size: 112, default order: 0, min order: 0 > 15:31:20 kernel: node 0: slabs: 80, objs: 2880, free: 0 > 15:31:20 kernel: node 1: slabs: 0, objs: 0, free: 0 > > > After this, the bash prompt in the cgroup is unusable: > > # ls > -bash: fork: Cannot allocate memory > > But typically the system outside the cgroup continues to work. > > > All of the above happens every time that I run these 6 commands. > > I am happy to help with extra information on kernel configuration, but I > hope that the above is sufficient for others to replicate. I'm also happy to > try suggestions and patches. > > > In addition, sometimes these 6 commands trigger more serious issues beyond > the cgroup. I have copied a long kernel log of such an incident below. > > Thanks in advance for your help, > > Richard. > > > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) > 15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0 > 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0 > 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) > 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 > 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 > 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff811017ad>] ? rcu_eqs_enter+0x70/0x83 > 15:11:36 kernel: [<ffffffff811017ce>] ? rcu_user_enter+0xe/0x10 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 24 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 54 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 178 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 7 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 3 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 1 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 80 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 167 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 183 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 46 > 15:11:36 kernel: active_anon:20251 inactive_anon:3312 isolated_anon:0 > 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 > 15:11:36 kernel: free:7963301 slab_reclaimable:19802 slab_unreclaimable:7992 > 15:11:36 kernel: mapped:7165 shmem:1531 pagetables:1575 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3390912kB min:9572kB low:11964kB high:14356kB active_anon:7148kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4836kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12257092kB min:35416kB low:44268kB high:53124kB active_anon:28044kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:12072kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16189300kB min:45076kB low:56344kB high:67612kB active_anon:45812kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 49*4kB (UEM) 62*8kB (U) 30*16kB (U) 12*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 2*2048kB (MR) 826*4096kB (MR) = 3390740kB > 15:11:36 kernel: Node 0 Normal: 112*4kB (UE) 57*8kB (UE) 12*16kB (UEM) 32*32kB (UEM) 15*64kB (UEM) 5*128kB (UM) 3*256kB (UEM) 3*512kB (UEM) 2*1024kB (UM) 1*2048kB (M) 2990*4096kB (MR) = 12257160kB > 15:11:36 kernel: Node 1 Normal: 95*4kB (UM) 8*8kB (U) 0*16kB 8*32kB (U) 6*64kB (UM) 10*128kB (UM) 2*256kB (EM) 2*512kB (UE) 0*1024kB 1*2048kB (M) 3951*4096kB (MR) = 16189244kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 183020 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd > 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe > 15:11:36 kernel: [ 4353] 104 4353 88747 10609 59 0 0 mysqld > 15:11:36 kernel: [ 4354] -2 4354 1023 158 8 0 0 logger > 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres > 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres > 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres > 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 627 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: Out of memory: Kill process 4353 (mysqld) score 0 or sacrifice child > 15:11:36 kernel: Killed process 4353 (mysqld) total-vm:354988kB, anon-rss:35704kB, file-rss:6732kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14 > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 45 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 57 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 150 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 12 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 131 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 174 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 113 > 15:11:36 kernel: active_anon:11418 inactive_anon:3312 isolated_anon:0 > 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 > 15:11:36 kernel: free:7971790 slab_reclaimable:19802 slab_unreclaimable:7992 > 15:11:36 kernel: mapped:5661 shmem:1531 pagetables:1478 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:6640kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16219896kB min:45076kB low:56344kB high:67612kB active_anon:15112kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:1780kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB > 15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 55*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258116kB > 15:11:36 kernel: Node 1 Normal: 245*4kB (UEM) 101*8kB (UEM) 21*16kB (UEM) 23*32kB (UEM) 19*64kB (UEM) 18*128kB (UEM) 9*256kB (M) 7*512kB (UEM) 2*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16220140kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 183020 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd > 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe > 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres > 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres > 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres > 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: Out of memory: Kill process 3988 (named) score 0 or sacrifice child > 15:11:36 kernel: Killed process 3988 (named) total-vm:165728kB, anon-rss:15848kB, file-rss:2172kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14 > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 33 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 53 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 151 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 100 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 44 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 137 > 15:11:36 kernel: active_anon:11808 inactive_anon:3312 isolated_anon:0 > 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 > 15:11:36 kernel: free:7971490 slab_reclaimable:19802 slab_unreclaimable:7992 > 15:11:36 kernel: mapped:6534 shmem:1531 pagetables:1575 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10132kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16218696kB min:45076kB low:56344kB high:67612kB active_anon:16672kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB > 15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 58*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258140kB > 15:11:36 kernel: Node 1 Normal: 90*4kB (UEM) 64*8kB (UEM) 7*16kB (UE) 12*32kB (UEM) 15*64kB (UEM) 17*128kB (UEM) 7*256kB (M) 7*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16218776kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 183020 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd > 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe > 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres > 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres > 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres > 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: [ 5197] -2 5197 12302 1440 25 0 0 mysqld > 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger > 15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child > 15:11:36 kernel: Killed process 4394 (postgres) total-vm:99840kB, anon-rss:1844kB, file-rss:1004kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14 > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 30 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 51 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 1 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 47 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173 > 15:11:36 kernel: active_anon:10570 inactive_anon:2937 isolated_anon:0 > 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 > 15:11:36 kernel: free:7972794 slab_reclaimable:19802 slab_unreclaimable:7927 > 15:11:36 kernel: mapped:6267 shmem:1531 pagetables:1575 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3395828kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:2592kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:3960kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12270500kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:5904kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10520kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16208948kB min:45076kB low:56344kB high:67612kB active_anon:26032kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 137*4kB (UEM) 92*8kB (UM) 49*16kB (UM) 26*32kB (UEM) 12*64kB (UM) 11*128kB (UEM) 3*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3395796kB > 15:11:36 kernel: Node 0 Normal: 337*4kB (UM) 138*8kB (UEM) 47*16kB (UEM) 47*32kB (UEM) 22*64kB (UEM) 10*128kB (UM) 2*256kB (UE) 3*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12270820kB > 15:11:36 kernel: Node 1 Normal: 44*4kB (UEM) 70*8kB (UE) 9*16kB (UEM) 9*32kB (U) 6*64kB (U) 10*128kB (UE) 0*256kB 3*512kB (UEM) 1*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16209168kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 183117 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd > 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe > 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres > 15:11:36 kernel: [ 4393] 103 4393 24775 452 36 0 0 postgres > 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: [ 5197] 104 5197 57326 4132 35 0 0 mysqld > 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger > 15:11:36 kernel: Out of memory: Kill process 5197 (mysqld) score 0 or sacrifice child > 15:11:36 kernel: Killed process 5197 (mysqld) total-vm:231548kB, anon-rss:17736kB, file-rss:5256kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 67 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 197 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 82 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 2 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 156 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 164 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173 > 15:11:36 kernel: active_anon:7704 inactive_anon:1830 isolated_anon:0 > 15:11:36 kernel: active_file:29305 inactive_file:151670 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0 > 15:11:36 kernel: free:7976547 slab_reclaimable:19802 slab_unreclaimable:7927 > 15:11:36 kernel: mapped:4180 shmem:220 pagetables:1478 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3397200kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:1024kB active_file:15260kB inactive_file:79780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2208kB shmem:112kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12273064kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:3304kB active_file:63984kB inactive_file:434284kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:3924kB shmem:448kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16220024kB min:45076kB low:56344kB high:67612kB active_anon:14568kB inactive_anon:2992kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:320kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 117*8kB (UEM) 56*16kB (UEM) 29*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397532kB > 15:11:36 kernel: Node 0 Normal: 314*4kB (UEM) 156*8kB (UEM) 55*16kB (UEM) 51*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273176kB > 15:11:36 kernel: Node 1 Normal: 276*4kB (UEM) 126*8kB (UEM) 31*16kB (UEM) 28*32kB (UEM) 20*64kB (UM) 20*128kB (UEM) 7*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16220592kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 181806 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd > 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe > 15:11:36 kernel: [ 4357] 103 4357 24779 1271 35 0 0 postgres > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: [ 5202] -2 5202 2336 77 7 0 0 mysqld_safe > 15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child > 15:11:36 kernel: Killed process 4357 (postgres) total-vm:99116kB, anon-rss:1232kB, file-rss:3852kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 98 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 196 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 79 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 179 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 156 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 > 15:11:36 kernel: active_anon:6997 inactive_anon:1812 isolated_anon:0 > 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 > 15:11:36 kernel: free:7976969 slab_reclaimable:19834 slab_unreclaimable:7983 > 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12273056kB min:35416kB low:44268kB high:53124kB active_anon:13500kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16221640kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB > 15:11:36 kernel: Node 0 Normal: 357*4kB (UEM) 164*8kB (UEM) 64*16kB (UEM) 52*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273588kB > 15:11:36 kernel: Node 1 Normal: 408*4kB (UEM) 141*8kB (UEM) 45*16kB (UEM) 28*32kB (UEM) 19*64kB (UM) 22*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16221912kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 181749 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd > 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: Out of memory: Kill process 1999 (httpd) score 0 or sacrifice child > 15:11:36 kernel: Killed process 1999 (httpd) total-vm:329268kB, anon-rss:3272kB, file-rss:1208kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 75 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 159 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 180 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 > 15:11:36 kernel: active_anon:6430 inactive_anon:1812 isolated_anon:0 > 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 > 15:11:36 kernel: free:7977670 slab_reclaimable:19834 slab_unreclaimable:7983 > 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12275340kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16222160kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB > 15:11:36 kernel: Node 0 Normal: 390*4kB (UEM) 183*8kB (UEM) 68*16kB (UEM) 56*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276112kB > 15:11:36 kernel: Node 1 Normal: 431*4kB (UEM) 150*8kB (UEM) 46*16kB (UEM) 30*32kB (UEM) 19*64kB (UM) 23*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16222284kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 181749 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd > 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: Out of memory: Kill process 2001 (httpd) score 0 or sacrifice child > 15:11:36 kernel: Killed process 2001 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1212kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 72 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 168 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 181 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 > 15:11:36 kernel: active_anon:5804 inactive_anon:1812 isolated_anon:0 > 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 > 15:11:36 kernel: free:7978396 slab_reclaimable:19834 slab_unreclaimable:7983 > 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12275600kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16224804kB min:45076kB low:56344kB high:67612kB active_anon:10240kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB > 15:11:36 kernel: Node 0 Normal: 393*4kB (UEM) 180*8kB (UEM) 71*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276340kB > 15:11:36 kernel: Node 1 Normal: 506*4kB (UEM) 161*8kB (UEM) 49*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16224960kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 181749 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: Out of memory: Kill process 2000 (httpd) score 0 or sacrifice child > 15:11:36 kernel: Killed process 2000 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1208kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 68 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 183 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 > 15:11:36 kernel: active_anon:5143 inactive_anon:1812 isolated_anon:0 > 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 > 15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983 > 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10972kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB > 15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB > 15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 182*8kB (UEM) 50*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227520kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 181749 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child > 15:11:36 kernel: Killed process 1997 (elastic-poolio) total-vm:19684kB, anon-rss:276kB, file-rss:1212kB > 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 > 15:11:36 kernel: tar cpuset=test mems_allowed=0-1 > 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 > 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 > 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 > 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 > 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 > 15:11:36 kernel: Call Trace: > 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77 > 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208 > 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f > 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24 > 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c > 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104 > 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b > 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d > 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 > 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176 > 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d > 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 > 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 > 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61 > 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30 > 15:11:36 kernel: Mem-Info: > 15:11:36 kernel: Node 0 DMA per-cpu: > 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > 15:11:36 kernel: Node 0 DMA32 per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 > 15:11:36 kernel: Node 0 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 159 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 > 15:11:36 kernel: Node 1 Normal per-cpu: > 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 > 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 > 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 > 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 > 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 > 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182 > 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 > 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106 > 15:11:36 kernel: active_anon:5078 inactive_anon:1812 isolated_anon:0 > 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0 > 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0 > 15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983 > 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0 > 15:11:36 kernel: free_cma:0 > 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097 > 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671 > 15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10712kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0 > 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB > 15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB > 15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 198*8kB (UEM) 55*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227728kB > 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 15:11:36 kernel: 181749 total pagecache pages > 15:11:36 kernel: 0 pages in swap cache > 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0 > 15:11:36 kernel: Free swap = 16507900kB > 15:11:36 kernel: Total swap = 16507900kB > 15:11:36 kernel: 8388414 pages RAM > 15:11:36 kernel: 0 pages HighMem/MovableOnly > 15:11:36 kernel: 65682 pages reserved > 15:11:36 kernel: 0 pages hwpoisoned > 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name > 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty > 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty > 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty > 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup > 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty > 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup > 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd > 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd > 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount > 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid > 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid > 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq > 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd > 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd > 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd > 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd > 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd > 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod > 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd > 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa > 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic > 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic > 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain > 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init > 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain > 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init > 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient > 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd > 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron > 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd > 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd > 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd > 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty > 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash > 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 > 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 > 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 > 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 > 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron > 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty > 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd > 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash > 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail > 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar > 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep > 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd > 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty > 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty > 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash > 15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child > 15:11:36 kernel: Killed process 1995 (httpd) total-vm:31888kB, anon-rss:952kB, file-rss:2344kB > 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child > 15:11:36 kernel: Killed process 4551 (apache2) total-vm:299432kB, anon-rss:1896kB, file-rss:760kB > 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child > 15:11:36 kernel: Killed process 4550 (apache2) total-vm:299440kB, anon-rss:1896kB, file-rss:756kB > 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child > 15:11:36 kernel: Killed process 4548 (apache2) total-vm:75736kB, anon-rss:1648kB, file-rss:572kB > 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child > 15:11:36 kernel: Killed process 4540 (apache2) total-vm:76004kB, anon-rss:1680kB, file-rss:1492kB > 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child > 15:11:37 kernel: Killed process 4386 (bash) total-vm:19872kB, anon-rss:484kB, file-rss:1472kB > 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit 2014-04-18 15:59 ` Michal Hocko @ 2014-04-18 17:57 ` Vladimir Davydov 2014-04-18 18:20 ` Michal Hocko 2014-04-20 14:28 ` Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] Richard Davies 0 siblings, 2 replies; 38+ messages in thread From: Vladimir Davydov @ 2014-04-18 17:57 UTC (permalink / raw) To: Richard Davies; +Cc: Michal Hocko, cgroups, linux-mm Hi Richard, 18.04.2014 19:59, Michal Hocko: > [CC Vladimir] > > On Wed 16-04-14 16:46:50, Richard Davies wrote: >> Hi all, >> >> I have a simple reproducible test case in which untar in a memcg with a kmem >> limit gets into trouble during heavy disk i/o (on ext3) and never properly >> recovers. This is simplified from real world problems with heavy disk i/o >> inside containers. >> >> I feel there are probably two bugs here >> - the disk i/o is not successfully managed within the kmem limit >> - the cgroup never recovers, despite the untar i/o process exiting Unfortunately, work on per cgroup kmem limits is not completed yet. Currently it lacks kmem reclaim on per cgroup memory pressure, which is vital for using kmem limits in real life. Basically that means that if a process inside a memory cgroup reaches its kmem limit, it will be returned ENOMEM on any allocation attempt, and no attempt will be made to reclaim old cached data. In your case untar consumes all kmem available to the cgroup by allocating memory for storing fs metadata (inodes, dentries). Those metadata are left cached in memory after untar dies, because they can be potentially used by other processes. As a result, any further attempt to allocate kmem (e.g. to create a process) will fail. It should try to reclaim the cached metadata instead, but this functionality is not implemented yet. In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though. Thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit 2014-04-18 17:57 ` Vladimir Davydov @ 2014-04-18 18:20 ` Michal Hocko 2014-04-18 18:37 ` Vladimir Davydov 2014-04-20 14:28 ` Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] Richard Davies 1 sibling, 1 reply; 38+ messages in thread From: Michal Hocko @ 2014-04-18 18:20 UTC (permalink / raw) To: Vladimir Davydov; +Cc: Richard Davies, cgroups, linux-mm On Fri 18-04-14 21:57:51, Vladimir Davydov wrote: > In short, kmem limiting for memory cgroups is currently broken. Do not > use it. We are working on making it usable though. Maybe we should make this explicit in both Documentation/cgroups/memory.txt and config MEMCG_KMEM help text. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit 2014-04-18 18:20 ` Michal Hocko @ 2014-04-18 18:37 ` Vladimir Davydov 0 siblings, 0 replies; 38+ messages in thread From: Vladimir Davydov @ 2014-04-18 18:37 UTC (permalink / raw) To: Michal Hocko; +Cc: Richard Davies, cgroups, linux-mm 18.04.2014 22:20, Michal Hocko: > On Fri 18-04-14 21:57:51, Vladimir Davydov wrote: >> In short, kmem limiting for memory cgroups is currently broken. Do not >> use it. We are working on making it usable though. > > Maybe we should make this explicit in both > Documentation/cgroups/memory.txt and config MEMCG_KMEM help text. Yeah, definitely. I'll send the patch. Thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-18 17:57 ` Vladimir Davydov 2014-04-18 18:20 ` Michal Hocko @ 2014-04-20 14:28 ` Richard Davies 2014-04-20 18:35 ` Tim Hockin 2014-04-22 18:39 ` Dwight Engen 1 sibling, 2 replies; 38+ messages in thread From: Richard Davies @ 2014-04-20 14:28 UTC (permalink / raw) To: Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange Cc: cgroups, linux-mm, containers Vladimir Davydov wrote: > Richard Davies wrote: > > I have a simple reproducible test case in which untar in a memcg with a > > kmem limit gets into trouble during heavy disk i/o (on ext3) and never > > properly recovers. This is simplified from real world problems with > > heavy disk i/o inside containers. > > Unfortunately, work on per cgroup kmem limits is not completed yet. > Currently it lacks kmem reclaim on per cgroup memory pressure, which is > vital for using kmem limits in real life. ... > In short, kmem limiting for memory cgroups is currently broken. Do not > use it. We are working on making it usable though. Thanks for explaining the strange errors I got. My motivation is to prevent a fork bomb in a container from affecting other processes outside that container. kmem limits were the preferred mechanism in several previous discussions about two years ago (I'm copying in participants from those previous discussions and give links below). So I tried kmem first but found bugs. What is the best mechanism available today, until kmem limits mature? RLIMIT_NPROC exists but is per-user, not per-container. Perhaps there is an up-to-date task counter patchset or similar? Thank you all, Richard. Some references to previous discussions: Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for memcg: stripped down version http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372 Re: [PATCH 00/10] cgroups: Task counter subsystem v8 http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310 [RFD] Merge task counter into memcg http://thread.gmane.org/gmane.linux.kernel/1280302 Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186 [PATCH] new cgroup controller "fork" http://thread.gmane.org/gmane.linux.kernel/1210878 Re: Process Limit cgroups http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369 Re: [lxc-devel] process number limit https://www.mail-archive.com/lxc-devel@lists.sourceforge.net/msg03309.html -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-20 14:28 ` Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] Richard Davies @ 2014-04-20 18:35 ` Tim Hockin 2014-04-22 18:39 ` Dwight Engen 1 sibling, 0 replies; 38+ messages in thread From: Tim Hockin @ 2014-04-20 18:35 UTC (permalink / raw) To: Richard Davies Cc: Michal Hocko, Vladimir Davydov, Frederic Weisbecker, Johannes Weiner, containers, Glauber Costa, Tejun Heo, David Rientjes, linux-mm, Daniel Walsh, William Dauchy, Max Kellermann, cgroups, Daniel Berrange [-- Attachment #1: Type: text/plain, Size: 2325 bytes --] I would still be in strong support of a cgroup replacement for NPROC rlimit. On Apr 20, 2014 7:29 AM, "Richard Davies" <richard@arachsys.com> wrote: > Vladimir Davydov wrote: > > Richard Davies wrote: > > > I have a simple reproducible test case in which untar in a memcg with a > > > kmem limit gets into trouble during heavy disk i/o (on ext3) and never > > > properly recovers. This is simplified from real world problems with > > > heavy disk i/o inside containers. > > > > Unfortunately, work on per cgroup kmem limits is not completed yet. > > Currently it lacks kmem reclaim on per cgroup memory pressure, which is > > vital for using kmem limits in real life. > ... > > In short, kmem limiting for memory cgroups is currently broken. Do not > > use it. We are working on making it usable though. > > Thanks for explaining the strange errors I got. > > > My motivation is to prevent a fork bomb in a container from affecting other > processes outside that container. > > kmem limits were the preferred mechanism in several previous discussions > about two years ago (I'm copying in participants from those previous > discussions and give links below). So I tried kmem first but found bugs. > > > What is the best mechanism available today, until kmem limits mature? > > RLIMIT_NPROC exists but is per-user, not per-container. > > Perhaps there is an up-to-date task counter patchset or similar? > > > Thank you all, > > Richard. > > > > Some references to previous discussions: > > Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for > memcg: stripped down version > http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372 > > Re: [PATCH 00/10] cgroups: Task counter subsystem v8 > http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310 > > [RFD] Merge task counter into memcg > http://thread.gmane.org/gmane.linux.kernel/1280302 > > Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic > http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186 > > [PATCH] new cgroup controller "fork" > http://thread.gmane.org/gmane.linux.kernel/1210878 > > Re: Process Limit cgroups > http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369 > > Re: [lxc-devel] process number limit > https://www.mail-archive.com/lxc-devel@lists.sourceforge.net/msg03309.html > [-- Attachment #2: Type: text/html, Size: 3477 bytes --] ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-20 14:28 ` Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] Richard Davies 2014-04-20 18:35 ` Tim Hockin @ 2014-04-22 18:39 ` Dwight Engen 2014-04-22 20:05 ` Richard Davies 1 sibling, 1 reply; 38+ messages in thread From: Dwight Engen @ 2014-04-22 18:39 UTC (permalink / raw) To: Richard Davies Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm On Sun, 20 Apr 2014 15:28:30 +0100 Richard Davies <richard@arachsys.com> wrote: > Vladimir Davydov wrote: > > Richard Davies wrote: > > > I have a simple reproducible test case in which untar in a memcg > > > with a kmem limit gets into trouble during heavy disk i/o (on > > > ext3) and never properly recovers. This is simplified from real > > > world problems with heavy disk i/o inside containers. > > > > Unfortunately, work on per cgroup kmem limits is not completed yet. > > Currently it lacks kmem reclaim on per cgroup memory pressure, > > which is vital for using kmem limits in real life. > ... > > In short, kmem limiting for memory cgroups is currently broken. Do > > not use it. We are working on making it usable though. > > Thanks for explaining the strange errors I got. > > > My motivation is to prevent a fork bomb in a container from affecting > other processes outside that container. > > kmem limits were the preferred mechanism in several previous > discussions about two years ago (I'm copying in participants from > those previous discussions and give links below). So I tried kmem > first but found bugs. > > > What is the best mechanism available today, until kmem limits mature? > > RLIMIT_NPROC exists but is per-user, not per-container. > > Perhaps there is an up-to-date task counter patchset or similar? I updated Frederic's task counter patches and included Max Kellermann's fork limiter here: http://thread.gmane.org/gmane.linux.kernel.containers/27212 I can send you a more recent patchset (against 3.13.10) if you would find it useful. > Thank you all, > > Richard. > > > > Some references to previous discussions: > > Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller > for memcg: stripped down version > http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372 > > Re: [PATCH 00/10] cgroups: Task counter subsystem v8 > http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310 > > [RFD] Merge task counter into memcg > http://thread.gmane.org/gmane.linux.kernel/1280302 > > Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic > http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186 > > [PATCH] new cgroup controller "fork" > http://thread.gmane.org/gmane.linux.kernel/1210878 > > Re: Process Limit cgroups > http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369 > > Re: [lxc-devel] process number limit > https://www.mail-archive.com/lxc-devel@lists.sourceforge.net/msg03309.html > _______________________________________________ > Containers mailing list > Containers@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/containers -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-22 18:39 ` Dwight Engen @ 2014-04-22 20:05 ` Richard Davies 2014-04-22 20:13 ` Tim Hockin ` (2 more replies) 0 siblings, 3 replies; 38+ messages in thread From: Richard Davies @ 2014-04-22 20:05 UTC (permalink / raw) To: Dwight Engen Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm Dwight Engen wrote: > Richard Davies wrote: > > Vladimir Davydov wrote: > > > In short, kmem limiting for memory cgroups is currently broken. Do > > > not use it. We are working on making it usable though. ... > > What is the best mechanism available today, until kmem limits mature? > > > > RLIMIT_NPROC exists but is per-user, not per-container. > > > > Perhaps there is an up-to-date task counter patchset or similar? > > I updated Frederic's task counter patches and included Max Kellermann's > fork limiter here: > > http://thread.gmane.org/gmane.linux.kernel.containers/27212 > > I can send you a more recent patchset (against 3.13.10) if you would > find it useful. Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too. Thanks, Richard. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-22 20:05 ` Richard Davies @ 2014-04-22 20:13 ` Tim Hockin 2014-04-23 6:07 ` Marian Marinov 2014-06-10 12:18 ` Alin Dobre 2 siblings, 0 replies; 38+ messages in thread From: Tim Hockin @ 2014-04-22 20:13 UTC (permalink / raw) To: Richard Davies Cc: Dwight Engen, Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm@kvack.org Who in kernel-land still needs to be convinced of the utility of this idea? On Tue, Apr 22, 2014 at 1:05 PM, Richard Davies <richard@arachsys.com> wrote: > Dwight Engen wrote: >> Richard Davies wrote: >> > Vladimir Davydov wrote: >> > > In short, kmem limiting for memory cgroups is currently broken. Do >> > > not use it. We are working on making it usable though. > ... >> > What is the best mechanism available today, until kmem limits mature? >> > >> > RLIMIT_NPROC exists but is per-user, not per-container. >> > >> > Perhaps there is an up-to-date task counter patchset or similar? >> >> I updated Frederic's task counter patches and included Max Kellermann's >> fork limiter here: >> >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 >> >> I can send you a more recent patchset (against 3.13.10) if you would >> find it useful. > > Yes please, I would be interested in that. Ideally even against 3.14.1 if > you have that too. > > Thanks, > > Richard. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-22 20:05 ` Richard Davies 2014-04-22 20:13 ` Tim Hockin @ 2014-04-23 6:07 ` Marian Marinov 2014-04-23 12:49 ` Dwight Engen 2014-06-10 12:18 ` Alin Dobre 2 siblings, 1 reply; 38+ messages in thread From: Marian Marinov @ 2014-04-23 6:07 UTC (permalink / raw) To: Richard Davies, Dwight Engen Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm On 04/22/2014 11:05 PM, Richard Davies wrote: > Dwight Engen wrote: >> Richard Davies wrote: >>> Vladimir Davydov wrote: >>>> In short, kmem limiting for memory cgroups is currently broken. Do >>>> not use it. We are working on making it usable though. > ... >>> What is the best mechanism available today, until kmem limits mature? >>> >>> RLIMIT_NPROC exists but is per-user, not per-container. >>> >>> Perhaps there is an up-to-date task counter patchset or similar? >> >> I updated Frederic's task counter patches and included Max Kellermann's >> fork limiter here: >> >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 >> >> I can send you a more recent patchset (against 3.13.10) if you would >> find it useful. > > Yes please, I would be interested in that. Ideally even against 3.14.1 if > you have that too. Dwight, do you have these patches in any public repo? I would like to test them also. Marian > > Thanks, > > Richard. > -- > To unsubscribe from this list: send the line "unsubscribe cgroups" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-23 6:07 ` Marian Marinov @ 2014-04-23 12:49 ` Dwight Engen 2014-04-28 18:00 ` Serge Hallyn ` (2 more replies) 0 siblings, 3 replies; 38+ messages in thread From: Dwight Engen @ 2014-04-23 12:49 UTC (permalink / raw) To: Marian Marinov Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov <mm@yuhu.biz> wrote: > On 04/22/2014 11:05 PM, Richard Davies wrote: > > Dwight Engen wrote: > >> Richard Davies wrote: > >>> Vladimir Davydov wrote: > >>>> In short, kmem limiting for memory cgroups is currently broken. > >>>> Do not use it. We are working on making it usable though. > > ... > >>> What is the best mechanism available today, until kmem limits > >>> mature? > >>> > >>> RLIMIT_NPROC exists but is per-user, not per-container. > >>> > >>> Perhaps there is an up-to-date task counter patchset or similar? > >> > >> I updated Frederic's task counter patches and included Max > >> Kellermann's fork limiter here: > >> > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 > >> > >> I can send you a more recent patchset (against 3.13.10) if you > >> would find it useful. > > > > Yes please, I would be interested in that. Ideally even against > > 3.14.1 if you have that too. > > Dwight, do you have these patches in any public repo? > > I would like to test them also. Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 > Marian > > > > > Thanks, > > > > Richard. > > -- > > To unsubscribe from this list: send the line "unsubscribe cgroups" > > in the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-23 12:49 ` Dwight Engen @ 2014-04-28 18:00 ` Serge Hallyn 2014-04-29 7:25 ` Michal Hocko 2014-05-06 11:40 ` Marian Marinov 2014-06-10 14:50 ` Marian Marinov 2 siblings, 1 reply; 38+ messages in thread From: Serge Hallyn @ 2014-04-28 18:00 UTC (permalink / raw) To: Dwight Engen Cc: Marian Marinov, Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Johannes Weiner, Glauber Costa, Michal Hocko, linux-mm, William Dauchy, David Rientjes, Tejun Heo, cgroups Quoting Dwight Engen (dwight.engen@oracle.com): > On Wed, 23 Apr 2014 09:07:28 +0300 > Marian Marinov <mm@yuhu.biz> wrote: > > > On 04/22/2014 11:05 PM, Richard Davies wrote: > > > Dwight Engen wrote: > > >> Richard Davies wrote: > > >>> Vladimir Davydov wrote: > > >>>> In short, kmem limiting for memory cgroups is currently broken. > > >>>> Do not use it. We are working on making it usable though. > > > ... > > >>> What is the best mechanism available today, until kmem limits > > >>> mature? > > >>> > > >>> RLIMIT_NPROC exists but is per-user, not per-container. > > >>> > > >>> Perhaps there is an up-to-date task counter patchset or similar? > > >> > > >> I updated Frederic's task counter patches and included Max > > >> Kellermann's fork limiter here: > > >> > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 > > >> > > >> I can send you a more recent patchset (against 3.13.10) if you > > >> would find it useful. > > > > > > Yes please, I would be interested in that. Ideally even against > > > 3.14.1 if you have that too. > > > > Dwight, do you have these patches in any public repo? > > > > I would like to test them also. > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute. -serge -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-28 18:00 ` Serge Hallyn @ 2014-04-29 7:25 ` Michal Hocko 2014-04-29 13:03 ` Serge Hallyn 0 siblings, 1 reply; 38+ messages in thread From: Michal Hocko @ 2014-04-29 7:25 UTC (permalink / raw) To: Serge Hallyn Cc: Dwight Engen, Marian Marinov, Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Johannes Weiner, Glauber Costa, linux-mm, William Dauchy, David Rientjes, Tejun Heo, cgroups On Mon 28-04-14 18:00:25, Serge Hallyn wrote: > Quoting Dwight Engen (dwight.engen@oracle.com): > > On Wed, 23 Apr 2014 09:07:28 +0300 > > Marian Marinov <mm@yuhu.biz> wrote: > > > > > On 04/22/2014 11:05 PM, Richard Davies wrote: > > > > Dwight Engen wrote: > > > >> Richard Davies wrote: > > > >>> Vladimir Davydov wrote: > > > >>>> In short, kmem limiting for memory cgroups is currently broken. > > > >>>> Do not use it. We are working on making it usable though. > > > > ... > > > >>> What is the best mechanism available today, until kmem limits > > > >>> mature? > > > >>> > > > >>> RLIMIT_NPROC exists but is per-user, not per-container. > > > >>> > > > >>> Perhaps there is an up-to-date task counter patchset or similar? > > > >> > > > >> I updated Frederic's task counter patches and included Max > > > >> Kellermann's fork limiter here: > > > >> > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 > > > >> > > > >> I can send you a more recent patchset (against 3.13.10) if you > > > >> would find it useful. > > > > > > > > Yes please, I would be interested in that. Ideally even against > > > > 3.14.1 if you have that too. > > > > > > Dwight, do you have these patches in any public repo? > > > > > > I would like to test them also. > > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian > that a task limit would be a proper cgroup extension, and specifically > that approximating that with a kmem limit is not a reasonable substitute. The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 7:25 ` Michal Hocko @ 2014-04-29 13:03 ` Serge Hallyn 2014-04-29 13:57 ` Marian Marinov ` (2 more replies) 0 siblings, 3 replies; 38+ messages in thread From: Serge Hallyn @ 2014-04-29 13:03 UTC (permalink / raw) To: Michal Hocko Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Daniel Walsh, cgroups, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes Quoting Michal Hocko (mhocko@suse.cz): > On Mon 28-04-14 18:00:25, Serge Hallyn wrote: > > Quoting Dwight Engen (dwight.engen@oracle.com): > > > On Wed, 23 Apr 2014 09:07:28 +0300 > > > Marian Marinov <mm@yuhu.biz> wrote: > > > > > > > On 04/22/2014 11:05 PM, Richard Davies wrote: > > > > > Dwight Engen wrote: > > > > >> Richard Davies wrote: > > > > >>> Vladimir Davydov wrote: > > > > >>>> In short, kmem limiting for memory cgroups is currently broken. > > > > >>>> Do not use it. We are working on making it usable though. > > > > > ... > > > > >>> What is the best mechanism available today, until kmem limits > > > > >>> mature? > > > > >>> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container. > > > > >>> > > > > >>> Perhaps there is an up-to-date task counter patchset or similar? > > > > >> > > > > >> I updated Frederic's task counter patches and included Max > > > > >> Kellermann's fork limiter here: > > > > >> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 > > > > >> > > > > >> I can send you a more recent patchset (against 3.13.10) if you > > > > >> would find it useful. > > > > > > > > > > Yes please, I would be interested in that. Ideally even against > > > > > 3.14.1 if you have that too. > > > > > > > > Dwight, do you have these patches in any public repo? > > > > > > > > I would like to test them also. > > > > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > > > > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 > > > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian > > that a task limit would be a proper cgroup extension, and specifically > > that approximating that with a kmem limit is not a reasonable substitute. > > The current state of the kmem limit, which is improving a lot thanks to > Vladimir, is not a reason for a new extension/controller. We are just > not yet there. It has nothing to do with the state of the limit. I simply don't believe that emulating RLIMIT_NPROC by controlling stack size is a good idea. -serge -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 13:03 ` Serge Hallyn @ 2014-04-29 13:57 ` Marian Marinov 2014-04-29 14:04 ` Tim Hockin 2014-04-29 15:43 ` Michal Hocko 2 siblings, 0 replies; 38+ messages in thread From: Marian Marinov @ 2014-04-29 13:57 UTC (permalink / raw) To: Serge Hallyn, Michal Hocko Cc: Richard Davies, Vladimir Davydov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Daniel Walsh, cgroups, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes On 04/29/2014 04:03 PM, Serge Hallyn wrote: > Quoting Michal Hocko (mhocko@suse.cz): >> On Mon 28-04-14 18:00:25, Serge Hallyn wrote: >>> Quoting Dwight Engen (dwight.engen@oracle.com): >>>> On Wed, 23 Apr 2014 09:07:28 +0300 >>>> Marian Marinov <mm@yuhu.biz> wrote: >>>> >>>>> On 04/22/2014 11:05 PM, Richard Davies wrote: >>>>>> Dwight Engen wrote: >>>>>>> Richard Davies wrote: >>>>>>>> Vladimir Davydov wrote: >>>>>>>>> In short, kmem limiting for memory cgroups is currently broken. >>>>>>>>> Do not use it. We are working on making it usable though. >>>>>> ... >>>>>>>> What is the best mechanism available today, until kmem limits >>>>>>>> mature? >>>>>>>> >>>>>>>> RLIMIT_NPROC exists but is per-user, not per-container. >>>>>>>> >>>>>>>> Perhaps there is an up-to-date task counter patchset or similar? >>>>>>> >>>>>>> I updated Frederic's task counter patches and included Max >>>>>>> Kellermann's fork limiter here: >>>>>>> >>>>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212 >>>>>>> >>>>>>> I can send you a more recent patchset (against 3.13.10) if you >>>>>>> would find it useful. >>>>>> >>>>>> Yes please, I would be interested in that. Ideally even against >>>>>> 3.14.1 if you have that too. >>>>> >>>>> Dwight, do you have these patches in any public repo? >>>>> >>>>> I would like to test them also. >>>> >>>> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: >>>> >>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 >>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 >>> >>> Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian >>> that a task limit would be a proper cgroup extension, and specifically >>> that approximating that with a kmem limit is not a reasonable substitute. >> >> The current state of the kmem limit, which is improving a lot thanks to >> Vladimir, is not a reason for a new extension/controller. We are just >> not yet there. > > It has nothing to do with the state of the limit. I simply don't > believe that emulating RLIMIT_NPROC by controlling stack size is a > good idea. > > -serge I think that having a limit on the number of processes allowed in a cgroup is a lot better then relaying on the kmem limit. The problem that task-limit tries to solve is degradation of system performance caused by too many processes in a certain cgroup. I'm currently testing the patches with 3.12.16. -hackman -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 13:03 ` Serge Hallyn 2014-04-29 13:57 ` Marian Marinov @ 2014-04-29 14:04 ` Tim Hockin 2014-04-29 15:43 ` Michal Hocko 2 siblings, 0 replies; 38+ messages in thread From: Tim Hockin @ 2014-04-29 14:04 UTC (permalink / raw) To: Serge Hallyn Cc: Michal Hocko, Vladimir Davydov, Marian Marinov, Frederic Weisbecker, Johannes Weiner, containers, Tim Hockin, Glauber Costa, Tejun Heo, David Rientjes, linux-mm, Daniel Walsh, William Dauchy, Max Kellermann, cgroups, Richard Davies [-- Attachment #1: Type: text/plain, Size: 2464 bytes --] Thank you. These are two different things. They may have a relationship but they ate not the same, and pretending they are is a bad experience. On Apr 29, 2014 6:04 AM, "Serge Hallyn" <serge.hallyn@ubuntu.com> wrote: > Quoting Michal Hocko (mhocko@suse.cz): > > On Mon 28-04-14 18:00:25, Serge Hallyn wrote: > > > Quoting Dwight Engen (dwight.engen@oracle.com): > > > > On Wed, 23 Apr 2014 09:07:28 +0300 > > > > Marian Marinov <mm@yuhu.biz> wrote: > > > > > > > > > On 04/22/2014 11:05 PM, Richard Davies wrote: > > > > > > Dwight Engen wrote: > > > > > >> Richard Davies wrote: > > > > > >>> Vladimir Davydov wrote: > > > > > >>>> In short, kmem limiting for memory cgroups is currently > broken. > > > > > >>>> Do not use it. We are working on making it usable though. > > > > > > ... > > > > > >>> What is the best mechanism available today, until kmem limits > > > > > >>> mature? > > > > > >>> > > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container. > > > > > >>> > > > > > >>> Perhaps there is an up-to-date task counter patchset or > similar? > > > > > >> > > > > > >> I updated Frederic's task counter patches and included Max > > > > > >> Kellermann's fork limiter here: > > > > > >> > > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 > > > > > >> > > > > > >> I can send you a more recent patchset (against 3.13.10) if you > > > > > >> would find it useful. > > > > > > > > > > > > Yes please, I would be interested in that. Ideally even against > > > > > > 3.14.1 if you have that too. > > > > > > > > > > Dwight, do you have these patches in any public repo? > > > > > > > > > > I would like to test them also. > > > > > > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > > > > > > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 > > > > > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian > > > that a task limit would be a proper cgroup extension, and specifically > > > that approximating that with a kmem limit is not a reasonable > substitute. > > > > The current state of the kmem limit, which is improving a lot thanks to > > Vladimir, is not a reason for a new extension/controller. We are just > > not yet there. > > It has nothing to do with the state of the limit. I simply don't > believe that emulating RLIMIT_NPROC by controlling stack size is a > good idea. > > -serge > [-- Attachment #2: Type: text/html, Size: 3834 bytes --] ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 13:03 ` Serge Hallyn 2014-04-29 13:57 ` Marian Marinov 2014-04-29 14:04 ` Tim Hockin @ 2014-04-29 15:43 ` Michal Hocko 2014-04-29 16:06 ` Tim Hockin 2 siblings, 1 reply; 38+ messages in thread From: Michal Hocko @ 2014-04-29 15:43 UTC (permalink / raw) To: Serge Hallyn Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Daniel Walsh, cgroups, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes On Tue 29-04-14 13:03:53, Serge Hallyn wrote: > Quoting Michal Hocko (mhocko@suse.cz): > > On Mon 28-04-14 18:00:25, Serge Hallyn wrote: > > > Quoting Dwight Engen (dwight.engen@oracle.com): > > > > On Wed, 23 Apr 2014 09:07:28 +0300 > > > > Marian Marinov <mm@yuhu.biz> wrote: > > > > > > > > > On 04/22/2014 11:05 PM, Richard Davies wrote: > > > > > > Dwight Engen wrote: > > > > > >> Richard Davies wrote: > > > > > >>> Vladimir Davydov wrote: > > > > > >>>> In short, kmem limiting for memory cgroups is currently broken. > > > > > >>>> Do not use it. We are working on making it usable though. > > > > > > ... > > > > > >>> What is the best mechanism available today, until kmem limits > > > > > >>> mature? > > > > > >>> > > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container. > > > > > >>> > > > > > >>> Perhaps there is an up-to-date task counter patchset or similar? > > > > > >> > > > > > >> I updated Frederic's task counter patches and included Max > > > > > >> Kellermann's fork limiter here: > > > > > >> > > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 > > > > > >> > > > > > >> I can send you a more recent patchset (against 3.13.10) if you > > > > > >> would find it useful. > > > > > > > > > > > > Yes please, I would be interested in that. Ideally even against > > > > > > 3.14.1 if you have that too. > > > > > > > > > > Dwight, do you have these patches in any public repo? > > > > > > > > > > I would like to test them also. > > > > > > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > > > > > > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 > > > > > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian > > > that a task limit would be a proper cgroup extension, and specifically > > > that approximating that with a kmem limit is not a reasonable substitute. > > > > The current state of the kmem limit, which is improving a lot thanks to > > Vladimir, is not a reason for a new extension/controller. We are just > > not yet there. > > It has nothing to do with the state of the limit. I simply don't > believe that emulating RLIMIT_NPROC by controlling stack size is a > good idea. I was not the one who decided that the kmem extension of memory controller should cover also the task number as a side effect but still the decision sounds plausible to me because the kmem approach is more generic. Btw. if this is a problem them please go ahead and continue the original discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the other people involved. I do not see any new arguments here, except that the kmem implementation is not ready yet. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 15:43 ` Michal Hocko @ 2014-04-29 16:06 ` Tim Hockin 2014-04-29 16:51 ` Frederic Weisbecker 0 siblings, 1 reply; 38+ messages in thread From: Tim Hockin @ 2014-04-29 16:06 UTC (permalink / raw) To: Michal Hocko Cc: Serge Hallyn, Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve? Is there some specific technical reason why you think this is a bad idea? I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously). On Tue, Apr 29, 2014 at 8:43 AM, Michal Hocko <mhocko@suse.cz> wrote: > On Tue 29-04-14 13:03:53, Serge Hallyn wrote: >> Quoting Michal Hocko (mhocko@suse.cz): >> > On Mon 28-04-14 18:00:25, Serge Hallyn wrote: >> > > Quoting Dwight Engen (dwight.engen@oracle.com): >> > > > On Wed, 23 Apr 2014 09:07:28 +0300 >> > > > Marian Marinov <mm@yuhu.biz> wrote: >> > > > >> > > > > On 04/22/2014 11:05 PM, Richard Davies wrote: >> > > > > > Dwight Engen wrote: >> > > > > >> Richard Davies wrote: >> > > > > >>> Vladimir Davydov wrote: >> > > > > >>>> In short, kmem limiting for memory cgroups is currently broken. >> > > > > >>>> Do not use it. We are working on making it usable though. >> > > > > > ... >> > > > > >>> What is the best mechanism available today, until kmem limits >> > > > > >>> mature? >> > > > > >>> >> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container. >> > > > > >>> >> > > > > >>> Perhaps there is an up-to-date task counter patchset or similar? >> > > > > >> >> > > > > >> I updated Frederic's task counter patches and included Max >> > > > > >> Kellermann's fork limiter here: >> > > > > >> >> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 >> > > > > >> >> > > > > >> I can send you a more recent patchset (against 3.13.10) if you >> > > > > >> would find it useful. >> > > > > > >> > > > > > Yes please, I would be interested in that. Ideally even against >> > > > > > 3.14.1 if you have that too. >> > > > > >> > > > > Dwight, do you have these patches in any public repo? >> > > > > >> > > > > I would like to test them also. >> > > > >> > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: >> > > > >> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 >> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 >> > > >> > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian >> > > that a task limit would be a proper cgroup extension, and specifically >> > > that approximating that with a kmem limit is not a reasonable substitute. >> > >> > The current state of the kmem limit, which is improving a lot thanks to >> > Vladimir, is not a reason for a new extension/controller. We are just >> > not yet there. >> >> It has nothing to do with the state of the limit. I simply don't >> believe that emulating RLIMIT_NPROC by controlling stack size is a >> good idea. > > I was not the one who decided that the kmem extension of memory > controller should cover also the task number as a side effect but still > the decision sounds plausible to me because the kmem approach is more > generic. > > Btw. if this is a problem them please go ahead and continue the original > discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the > other people involved. > > I do not see any new arguments here, except that the kmem implementation > is not ready yet. > -- > Michal Hocko > SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 16:06 ` Tim Hockin @ 2014-04-29 16:51 ` Frederic Weisbecker 2014-04-29 16:59 ` Tim Hockin 0 siblings, 1 reply; 38+ messages in thread From: Frederic Weisbecker @ 2014-04-29 16:51 UTC (permalink / raw) To: Tim Hockin Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, containers, Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote: > Why the insistence that we manage something that REALLY IS a > first-class concept (hey, it has it's own RLIMIT) as a side effect of > something that doesn't quite capture what we want to achieve? It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers. Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it. Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet. > > Is there some specific technical reason why you think this is a bad > idea? > I would think, especially in a more unified hierarchy world, > that more cgroup controllers with smaller sets of responsibility would > make for more manageable code (within limits, obviously). Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 16:51 ` Frederic Weisbecker @ 2014-04-29 16:59 ` Tim Hockin 2014-04-29 17:06 ` Michal Hocko 2014-04-29 21:44 ` Frederic Weisbecker 0 siblings, 2 replies; 38+ messages in thread From: Tim Hockin @ 2014-04-29 16:59 UTC (permalink / raw) To: Frederic Weisbecker Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, containers, Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work. You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID. On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker <fweisbec@gmail.com> wrote: > On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote: >> Why the insistence that we manage something that REALLY IS a >> first-class concept (hey, it has it's own RLIMIT) as a side effect of >> something that doesn't quite capture what we want to achieve? > > It's not a side effect, the kmem task stack control was partly > motivated to solve forkbomb issues in containers. > > Also in general if we can reuse existing features and code to solve > a problem without disturbing side issues, we just do it. > > Now if kmem doesn't solve the issue for you for any reason, or it does > but it brings other problems that aren't fixable in kmem itself, we can > certainly reconsider this cgroup subsystem. But I haven't yet seen > argument of this kind yet. > >> >> Is there some specific technical reason why you think this is a bad >> idea? >> I would think, especially in a more unified hierarchy world, >> that more cgroup controllers with smaller sets of responsibility would >> make for more manageable code (within limits, obviously). > > Because it's core code and it adds complications and overhead in the > fork/exit path. We just don't add new core code just for the sake of > slightly prettier interfaces. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 16:59 ` Tim Hockin @ 2014-04-29 17:06 ` Michal Hocko 2014-04-29 17:30 ` Dwight Engen 2014-04-29 21:44 ` Frederic Weisbecker 1 sibling, 1 reply; 38+ messages in thread From: Michal Hocko @ 2014-04-29 17:06 UTC (permalink / raw) To: Tim Hockin Cc: Frederic Weisbecker, Serge Hallyn, Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, containers, Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes On Tue 29-04-14 09:59:30, Tim Hockin wrote: > Here's the reason it doesn't work for us: It doesn't work. There is a "simple" solution for that. Help us to fix it. > It was something like 2 YEARS since we first wanted this, and it STILL > does not work. My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there. > You're postponing a pretty simple request indefinitely in > favor of a much more complex feature, which still doesn't really give > me what I want. But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs. > What I want is an API that works like rlimit but per-cgroup, rather > than per-UID. You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them. > On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker <fweisbec@gmail.com> wrote: > > On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote: > >> Why the insistence that we manage something that REALLY IS a > >> first-class concept (hey, it has it's own RLIMIT) as a side effect of > >> something that doesn't quite capture what we want to achieve? > > > > It's not a side effect, the kmem task stack control was partly > > motivated to solve forkbomb issues in containers. > > > > Also in general if we can reuse existing features and code to solve > > a problem without disturbing side issues, we just do it. > > > > Now if kmem doesn't solve the issue for you for any reason, or it does > > but it brings other problems that aren't fixable in kmem itself, we can > > certainly reconsider this cgroup subsystem. But I haven't yet seen > > argument of this kind yet. > > > >> > >> Is there some specific technical reason why you think this is a bad > >> idea? > >> I would think, especially in a more unified hierarchy world, > >> that more cgroup controllers with smaller sets of responsibility would > >> make for more manageable code (within limits, obviously). > > > > Because it's core code and it adds complications and overhead in the > > fork/exit path. We just don't add new core code just for the sake of > > slightly prettier interfaces. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 17:06 ` Michal Hocko @ 2014-04-29 17:30 ` Dwight Engen 2014-04-29 18:09 ` Richard Davies 0 siblings, 1 reply; 38+ messages in thread From: Dwight Engen @ 2014-04-29 17:30 UTC (permalink / raw) To: Michal Hocko Cc: Tim Hockin, Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Serge Hallyn, Glauber Costa, linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh On Tue, 29 Apr 2014 19:06:39 +0200 Michal Hocko <mhocko@suse.cz> wrote: > On Tue 29-04-14 09:59:30, Tim Hockin wrote: > > Here's the reason it doesn't work for us: It doesn't work. > > There is a "simple" solution for that. Help us to fix it. > > > It was something like 2 YEARS since we first wanted this, and it > > STILL does not work. > > My recollection is that it was primarily Parallels and Google asking > for the kmem accounting. The reason why I didn't fight against > inclusion although the implementation at the time didn't have a > proper slab shrinking implemented was that that would happen later. > Well, that later hasn't happened yet and we are slowly getting there. > > > You're postponing a pretty simple request indefinitely in > > favor of a much more complex feature, which still doesn't really > > give me what I want. > > But we cannot simply add a new interface that will have to be > maintained for ever just because something else that is supposed to > workaround bugs. > > > What I want is an API that works like rlimit but per-cgroup, rather > > than per-UID. > > You can use an out-of-tree patchset for the time being or help to get > kmem into shape. If there are principal reasons why kmem cannot be > used then you better articulate them. Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together. > > On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker > > <fweisbec@gmail.com> wrote: > > > On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote: > > >> Why the insistence that we manage something that REALLY IS a > > >> first-class concept (hey, it has it's own RLIMIT) as a side > > >> effect of something that doesn't quite capture what we want to > > >> achieve? > > > > > > It's not a side effect, the kmem task stack control was partly > > > motivated to solve forkbomb issues in containers. > > > > > > Also in general if we can reuse existing features and code to > > > solve a problem without disturbing side issues, we just do it. > > > > > > Now if kmem doesn't solve the issue for you for any reason, or it > > > does but it brings other problems that aren't fixable in kmem > > > itself, we can certainly reconsider this cgroup subsystem. But I > > > haven't yet seen argument of this kind yet. > > > > > >> > > >> Is there some specific technical reason why you think this is a > > >> bad idea? > > >> I would think, especially in a more unified hierarchy world, > > >> that more cgroup controllers with smaller sets of responsibility > > >> would make for more manageable code (within limits, obviously). > > > > > > Because it's core code and it adds complications and overhead in > > > the fork/exit path. We just don't add new core code just for the > > > sake of slightly prettier interfaces. > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 17:30 ` Dwight Engen @ 2014-04-29 18:09 ` Richard Davies 2014-04-29 18:27 ` Michal Hocko 0 siblings, 1 reply; 38+ messages in thread From: Richard Davies @ 2014-04-29 18:09 UTC (permalink / raw) To: Dwight Engen Cc: Michal Hocko, Tim Hockin, Vladimir Davydov, David Rientjes, Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh Dwight Engen wrote: > Michal Hocko wrote: > > Tim Hockin wrote: > > > Here's the reason it doesn't work for us: It doesn't work. > > > > There is a "simple" solution for that. Help us to fix it. > > > > > It was something like 2 YEARS since we first wanted this, and it > > > STILL does not work. > > > > My recollection is that it was primarily Parallels and Google asking > > for the kmem accounting. The reason why I didn't fight against > > inclusion although the implementation at the time didn't have a > > proper slab shrinking implemented was that that would happen later. > > Well, that later hasn't happened yet and we are slowly getting there. > > > > > You're postponing a pretty simple request indefinitely in > > > favor of a much more complex feature, which still doesn't really > > > give me what I want. > > > > But we cannot simply add a new interface that will have to be > > maintained for ever just because something else that is supposed to > > workaround bugs. > > > > > What I want is an API that works like rlimit but per-cgroup, rather > > > than per-UID. > > > > You can use an out-of-tree patchset for the time being or help to get > > kmem into shape. If there are principal reasons why kmem cannot be > > used then you better articulate them. > > Is there a plan to separately account/limit stack pages vs kmem in > general? Richard would have to verify, but I suspect kmem is not currently > viable as a process limiter for him because icache/dcache/stack is all > accounted together. Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers. In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching. So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache. I am also somewhat worried that normal software use could legitimately go above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing just by booting a distro with a few daemons in a container. Richard. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 18:09 ` Richard Davies @ 2014-04-29 18:27 ` Michal Hocko 2014-04-29 18:39 ` Richard Davies 2014-04-29 21:36 ` Marian Marinov 0 siblings, 2 replies; 38+ messages in thread From: Michal Hocko @ 2014-04-29 18:27 UTC (permalink / raw) To: Richard Davies Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes, Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh On Tue 29-04-14 19:09:27, Richard Davies wrote: > Dwight Engen wrote: > > Michal Hocko wrote: > > > Tim Hockin wrote: > > > > Here's the reason it doesn't work for us: It doesn't work. > > > > > > There is a "simple" solution for that. Help us to fix it. > > > > > > > It was something like 2 YEARS since we first wanted this, and it > > > > STILL does not work. > > > > > > My recollection is that it was primarily Parallels and Google asking > > > for the kmem accounting. The reason why I didn't fight against > > > inclusion although the implementation at the time didn't have a > > > proper slab shrinking implemented was that that would happen later. > > > Well, that later hasn't happened yet and we are slowly getting there. > > > > > > > You're postponing a pretty simple request indefinitely in > > > > favor of a much more complex feature, which still doesn't really > > > > give me what I want. > > > > > > But we cannot simply add a new interface that will have to be > > > maintained for ever just because something else that is supposed to > > > workaround bugs. > > > > > > > What I want is an API that works like rlimit but per-cgroup, rather > > > > than per-UID. > > > > > > You can use an out-of-tree patchset for the time being or help to get > > > kmem into shape. If there are principal reasons why kmem cannot be > > > used then you better articulate them. > > > > Is there a plan to separately account/limit stack pages vs kmem in > > general? Richard would have to verify, but I suspect kmem is not currently > > viable as a process limiter for him because icache/dcache/stack is all > > accounted together. > > Certainly I would like to be able to limit container fork-bombs without > limiting the amount of disk IO caching for processes in those containers. > > In my testing with of kmem limits, I needed a limit of 256MB or lower to > catch fork bombs early enough. I would definitely like more than 256MB of > disk caching. > > So if we go the "working kmem" route, I would like to be able to specify a > limit excluding disk cache. Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one. Kmem accounting refers to kernel internal allocations - slab memory and per process kernel stack. You can see how much memory is allocated per container by memory.kmem.usage_in_bytes or have a look at /proc/slabinfo to see what kind of memory kernel allocates globally and might be accounted for a container as well. The primary problem with the kmem accounting right now is that such a memory is not "reclaimed" and so if the kmem limit is reached all the further kmem allocations fail. The biggest user of the kmem allocations on many systems is dentry and inode chache which is reclaimable easily. When this is implemented the kmem limit will be usable to both prevent forkbombs but also other DOS scenarios when the kernel is pushed to allocate a huge amount of memory. HTH > I am also somewhat worried that normal software use could legitimately go > above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing > just by booting a distro with a few daemons in a container. > > Richard. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 18:27 ` Michal Hocko @ 2014-04-29 18:39 ` Richard Davies 2014-04-29 19:03 ` Michal Hocko 2014-04-29 21:36 ` Marian Marinov 1 sibling, 1 reply; 38+ messages in thread From: Richard Davies @ 2014-04-29 18:39 UTC (permalink / raw) To: Michal Hocko Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes, Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh Michal Hocko wrote: > Richard Davies wrote: > > Dwight Engen wrote: > > > Is there a plan to separately account/limit stack pages vs kmem in > > > general? Richard would have to verify, but I suspect kmem is not > > > currently viable as a process limiter for him because > > > icache/dcache/stack is all accounted together. > > > > Certainly I would like to be able to limit container fork-bombs without > > limiting the amount of disk IO caching for processes in those containers. > > > > In my testing with of kmem limits, I needed a limit of 256MB or lower to > > catch fork bombs early enough. I would definitely like more than 256MB of > > disk caching. > > > > So if we go the "working kmem" route, I would like to be able to specify a > > limit excluding disk cache. > > Page cache (which is what you mean by disk cache probably) is a > userspace accounted memory with the memory cgroup controller. And you > do not have to limit that one. OK, that's helpful - thanks. As an aside, with the normal (non-kmem) cgroup controller, is there a way for me to exclude page cache and only limit the equivalent of the rss line in memory.stat? e.g. say I have a 256GB physical machine, running 200 containers, each with 1GB normal-mem limit (for running software) and 256MB kmem limit (to stop fork-bombs). The physical disk IO bandwidth is a shared resource between all the containers, so ideally I would like the kernel to used the 56GB of RAM as shared page cache however it best reduces physical IOPs, rather than having a per-container limit. Thanks, Richard. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 18:39 ` Richard Davies @ 2014-04-29 19:03 ` Michal Hocko 0 siblings, 0 replies; 38+ messages in thread From: Michal Hocko @ 2014-04-29 19:03 UTC (permalink / raw) To: Richard Davies Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes, Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh On Tue 29-04-14 19:39:28, Richard Davies wrote: > Michal Hocko wrote: > > Richard Davies wrote: > > > Dwight Engen wrote: > > > > Is there a plan to separately account/limit stack pages vs kmem in > > > > general? Richard would have to verify, but I suspect kmem is not > > > > currently viable as a process limiter for him because > > > > icache/dcache/stack is all accounted together. > > > > > > Certainly I would like to be able to limit container fork-bombs without > > > limiting the amount of disk IO caching for processes in those containers. > > > > > > In my testing with of kmem limits, I needed a limit of 256MB or lower to > > > catch fork bombs early enough. I would definitely like more than 256MB of > > > disk caching. > > > > > > So if we go the "working kmem" route, I would like to be able to specify a > > > limit excluding disk cache. > > > > Page cache (which is what you mean by disk cache probably) is a > > userspace accounted memory with the memory cgroup controller. And you > > do not have to limit that one. > > OK, that's helpful - thanks. > > As an aside, with the normal (non-kmem) cgroup controller, is there a way > for me to exclude page cache and only limit the equivalent of the rss line > in memory.stat? No > e.g. say I have a 256GB physical machine, running 200 containers, each with > 1GB normal-mem limit (for running software) and 256MB kmem limit (to stop > fork-bombs). > > The physical disk IO bandwidth is a shared resource between all the > containers, so ideally I would like the kernel to used the 56GB of RAM as > shared page cache however it best reduces physical IOPs, rather than having > a per-container limit. Then do not use any memory.limit_in_bytes and if there is a memory pressure then the global reclaim will shrink all the containers proportionally and the page cache will be the #1 target for the reclaim (but we are getting off-topic here I am afraid). -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 18:27 ` Michal Hocko 2014-04-29 18:39 ` Richard Davies @ 2014-04-29 21:36 ` Marian Marinov 2014-04-30 13:31 ` Michal Hocko 1 sibling, 1 reply; 38+ messages in thread From: Marian Marinov @ 2014-04-29 21:36 UTC (permalink / raw) To: Michal Hocko, Richard Davies Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh On 04/29/2014 09:27 PM, Michal Hocko wrote: > On Tue 29-04-14 19:09:27, Richard Davies wrote: >> Dwight Engen wrote: >>> Michal Hocko wrote: >>>> Tim Hockin wrote: >>>>> Here's the reason it doesn't work for us: It doesn't work. >>>> >>>> There is a "simple" solution for that. Help us to fix it. >>>> >>>>> It was something like 2 YEARS since we first wanted this, and it >>>>> STILL does not work. >>>> >>>> My recollection is that it was primarily Parallels and Google asking >>>> for the kmem accounting. The reason why I didn't fight against >>>> inclusion although the implementation at the time didn't have a >>>> proper slab shrinking implemented was that that would happen later. >>>> Well, that later hasn't happened yet and we are slowly getting there. >>>> >>>>> You're postponing a pretty simple request indefinitely in >>>>> favor of a much more complex feature, which still doesn't really >>>>> give me what I want. >>>> >>>> But we cannot simply add a new interface that will have to be >>>> maintained for ever just because something else that is supposed to >>>> workaround bugs. >>>> >>>>> What I want is an API that works like rlimit but per-cgroup, rather >>>>> than per-UID. >>>> >>>> You can use an out-of-tree patchset for the time being or help to get >>>> kmem into shape. If there are principal reasons why kmem cannot be >>>> used then you better articulate them. >>> >>> Is there a plan to separately account/limit stack pages vs kmem in >>> general? Richard would have to verify, but I suspect kmem is not currently >>> viable as a process limiter for him because icache/dcache/stack is all >>> accounted together. >> >> Certainly I would like to be able to limit container fork-bombs without >> limiting the amount of disk IO caching for processes in those containers. >> >> In my testing with of kmem limits, I needed a limit of 256MB or lower to >> catch fork bombs early enough. I would definitely like more than 256MB of >> disk caching. >> >> So if we go the "working kmem" route, I would like to be able to specify a >> limit excluding disk cache. > > Page cache (which is what you mean by disk cache probably) is a > userspace accounted memory with the memory cgroup controller. And you > do not have to limit that one. Kmem accounting refers to kernel internal > allocations - slab memory and per process kernel stack. You can see how > much memory is allocated per container by memory.kmem.usage_in_bytes or > have a look at /proc/slabinfo to see what kind of memory kernel > allocates globally and might be accounted for a container as well. > > The primary problem with the kmem accounting right now is that such a > memory is not "reclaimed" and so if the kmem limit is reached all the > further kmem allocations fail. The biggest user of the kmem allocations > on many systems is dentry and inode chache which is reclaimable easily. > When this is implemented the kmem limit will be usable to both prevent > forkbombs but also other DOS scenarios when the kernel is pushed to > allocate a huge amount of memory. I would have to disagree here. If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem. The simple scheduling of so many processes generates have load on the machine. Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes. Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure on the kmem, but will would not produce many processes. On the other hand, forking thousands of processes with low memory footprint will hit the scheduler a lot faster then hitting the kmem limit. Kmem limit is something that we need! But firmly believe that we need a simple NPROC limit for cgroups. -hackman > > HTH > >> I am also somewhat worried that normal software use could legitimately go >> above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing >> just by booting a distro with a few daemons in a container. >> >> Richard. > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 21:36 ` Marian Marinov @ 2014-04-30 13:31 ` Michal Hocko 0 siblings, 0 replies; 38+ messages in thread From: Michal Hocko @ 2014-04-30 13:31 UTC (permalink / raw) To: Marian Marinov Cc: Richard Davies, Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh On Wed 30-04-14 00:36:40, Marian Marinov wrote: > On 04/29/2014 09:27 PM, Michal Hocko wrote: > >On Tue 29-04-14 19:09:27, Richard Davies wrote: > >>Dwight Engen wrote: > >>>Michal Hocko wrote: > >>>>Tim Hockin wrote: > >>>>>Here's the reason it doesn't work for us: It doesn't work. > >>>> > >>>>There is a "simple" solution for that. Help us to fix it. > >>>> > >>>>>It was something like 2 YEARS since we first wanted this, and it > >>>>>STILL does not work. > >>>> > >>>>My recollection is that it was primarily Parallels and Google asking > >>>>for the kmem accounting. The reason why I didn't fight against > >>>>inclusion although the implementation at the time didn't have a > >>>>proper slab shrinking implemented was that that would happen later. > >>>>Well, that later hasn't happened yet and we are slowly getting there. > >>>> > >>>>>You're postponing a pretty simple request indefinitely in > >>>>>favor of a much more complex feature, which still doesn't really > >>>>>give me what I want. > >>>> > >>>>But we cannot simply add a new interface that will have to be > >>>>maintained for ever just because something else that is supposed to > >>>>workaround bugs. > >>>> > >>>>>What I want is an API that works like rlimit but per-cgroup, rather > >>>>>than per-UID. > >>>> > >>>>You can use an out-of-tree patchset for the time being or help to get > >>>>kmem into shape. If there are principal reasons why kmem cannot be > >>>>used then you better articulate them. > >>> > >>>Is there a plan to separately account/limit stack pages vs kmem in > >>>general? Richard would have to verify, but I suspect kmem is not currently > >>>viable as a process limiter for him because icache/dcache/stack is all > >>>accounted together. > >> > >>Certainly I would like to be able to limit container fork-bombs without > >>limiting the amount of disk IO caching for processes in those containers. > >> > >>In my testing with of kmem limits, I needed a limit of 256MB or lower to > >>catch fork bombs early enough. I would definitely like more than 256MB of > >>disk caching. > >> > >>So if we go the "working kmem" route, I would like to be able to specify a > >>limit excluding disk cache. > > > >Page cache (which is what you mean by disk cache probably) is a > >userspace accounted memory with the memory cgroup controller. And you > >do not have to limit that one. Kmem accounting refers to kernel internal > >allocations - slab memory and per process kernel stack. You can see how > >much memory is allocated per container by memory.kmem.usage_in_bytes or > >have a look at /proc/slabinfo to see what kind of memory kernel > >allocates globally and might be accounted for a container as well. > > > >The primary problem with the kmem accounting right now is that such a > >memory is not "reclaimed" and so if the kmem limit is reached all the > >further kmem allocations fail. The biggest user of the kmem allocations > >on many systems is dentry and inode chache which is reclaimable easily. > >When this is implemented the kmem limit will be usable to both prevent > >forkbombs but also other DOS scenarios when the kernel is pushed to > >allocate a huge amount of memory. > > I would have to disagree here. > If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem. > The simple scheduling of so many processes generates have load on the machine. > Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes. What prevents you from setting the kmem limit to NR_PROC * 8K + slab_pillow? > Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure > on the kmem, but will would not produce many processes. Once we have a proper slab reclaim implementation this shouldn't be a problem. > On the other hand, forking thousands of processes with low memory footprint > will hit the scheduler a lot faster then hitting the kmem limit. > > Kmem limit is something that we need! But firmly believe that we need > a simple NPROC limit for cgroups. Once again. If you feel that your usecase is not covered by the kmem limit follow up on the original email thread I have referenced earlier in the thread. Splitting up the discussion doesn't help at all. > -hackman > > > > >HTH > > > >>I am also somewhat worried that normal software use could legitimately go > >>above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing > >>just by booting a distro with a few daemons in a container. > >> > >>Richard. > > > -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 16:59 ` Tim Hockin 2014-04-29 17:06 ` Michal Hocko @ 2014-04-29 21:44 ` Frederic Weisbecker 2014-04-30 13:12 ` Daniel J Walsh 1 sibling, 1 reply; 38+ messages in thread From: Frederic Weisbecker @ 2014-04-29 21:44 UTC (permalink / raw) To: Tim Hockin Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, containers, Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote: > Here's the reason it doesn't work for us: It doesn't work. It was > something like 2 YEARS since we first wanted this, and it STILL does > not work. When I was working on the task counter cgroup subsystem 2 years ago, the patches were actually pushed back by google people, in favour of task stack kmem cgroup subsystem. The reason was that expressing the forkbomb issue in terms of number of tasks as a resource is awkward and that the real resource in the game comes from kernel memory exhaustion due to task stack being allocated over and over, swap ping-pong and stuffs... And that was a pretty good argument. I still agree with that. Especially since that could solve others people issues at the same time. kmem cgroup has a quite large domain of application. > You're postponing a pretty simple request indefinitely in > favor of a much more complex feature, which still doesn't really give > me what I want. What I want is an API that works like rlimit but > per-cgroup, rather than per-UID. The request is simple but I don't think that adding the task counter cgroup subsystem is simpler than extending the kmem code to apply limits to only task stack. Especially in terms of maintainance. Also you guys have very good mm kernel developers who are already familiar with this. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-29 21:44 ` Frederic Weisbecker @ 2014-04-30 13:12 ` Daniel J Walsh 2014-04-30 13:28 ` Frederic Weisbecker 0 siblings, 1 reply; 38+ messages in thread From: Daniel J Walsh @ 2014-04-30 13:12 UTC (permalink / raw) To: Frederic Weisbecker, Tim Hockin Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, containers, cgroups, Glauber Costa, linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes On 04/29/2014 05:44 PM, Frederic Weisbecker wrote: > On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote: >> Here's the reason it doesn't work for us: It doesn't work. It was >> something like 2 YEARS since we first wanted this, and it STILL does >> not work. > When I was working on the task counter cgroup subsystem 2 years > ago, the patches were actually pushed back by google people, in favour > of task stack kmem cgroup subsystem. > > The reason was that expressing the forkbomb issue in terms of > number of tasks as a resource is awkward and that the real resource > in the game comes from kernel memory exhaustion due to task stack being > allocated over and over, swap ping-pong and stuffs... > > And that was a pretty good argument. I still agree with that. Especially > since that could solve others people issues at the same time. kmem > cgroup has a quite large domain of application. > >> You're postponing a pretty simple request indefinitely in >> favor of a much more complex feature, which still doesn't really give >> me what I want. What I want is an API that works like rlimit but >> per-cgroup, rather than per-UID. > The request is simple but I don't think that adding the task counter > cgroup subsystem is simpler than extending the kmem code to apply limits > to only task stack. Especially in terms of maintainance. > > Also you guys have very good mm kernel developers who are already > familiar with this. I would look at this from a Usability point of view. It is a lot easier to understand number of processes then the mount of KMEM those processes will need. Setting something like ProcessLimit=1000 in a systemd unit file is easy to explain. Now if systemd has the ability to translate this into something that makes sense in terms of kmem cgroup, then my argument goes away. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-30 13:12 ` Daniel J Walsh @ 2014-04-30 13:28 ` Frederic Weisbecker 0 siblings, 0 replies; 38+ messages in thread From: Frederic Weisbecker @ 2014-04-30 13:28 UTC (permalink / raw) To: Daniel J Walsh Cc: Tim Hockin, Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin, containers, cgroups, Glauber Costa, linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes On Wed, Apr 30, 2014 at 09:12:20AM -0400, Daniel J Walsh wrote: > > On 04/29/2014 05:44 PM, Frederic Weisbecker wrote: > > On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote: > >> Here's the reason it doesn't work for us: It doesn't work. It was > >> something like 2 YEARS since we first wanted this, and it STILL does > >> not work. > > When I was working on the task counter cgroup subsystem 2 years > > ago, the patches were actually pushed back by google people, in favour > > of task stack kmem cgroup subsystem. > > > > The reason was that expressing the forkbomb issue in terms of > > number of tasks as a resource is awkward and that the real resource > > in the game comes from kernel memory exhaustion due to task stack being > > allocated over and over, swap ping-pong and stuffs... > > > > And that was a pretty good argument. I still agree with that. Especially > > since that could solve others people issues at the same time. kmem > > cgroup has a quite large domain of application. > > > >> You're postponing a pretty simple request indefinitely in > >> favor of a much more complex feature, which still doesn't really give > >> me what I want. What I want is an API that works like rlimit but > >> per-cgroup, rather than per-UID. > > The request is simple but I don't think that adding the task counter > > cgroup subsystem is simpler than extending the kmem code to apply limits > > to only task stack. Especially in terms of maintainance. > > > > Also you guys have very good mm kernel developers who are already > > familiar with this. > I would look at this from a Usability point of view. It is a lot easier > to understand number of processes then the mount of KMEM those processes > will need. Setting something like > ProcessLimit=1000 in a systemd unit file is easy to explain. Yeah that's a fair point. > Now if systemd has the ability to translate this into something that makes > sense in terms of kmem cgroup, then my argument goes away. Yeah if we keep the kmem direction, this can be a place where we do the mapping. Now I just hope the amount of stack memory allocated doesn't differ too much per arch. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-23 12:49 ` Dwight Engen 2014-04-28 18:00 ` Serge Hallyn @ 2014-05-06 11:40 ` Marian Marinov 2014-05-07 17:15 ` Dwight Engen 2014-06-10 14:50 ` Marian Marinov 2 siblings, 1 reply; 38+ messages in thread From: Marian Marinov @ 2014-05-06 11:40 UTC (permalink / raw) To: Dwight Engen Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm On 04/23/2014 03:49 PM, Dwight Engen wrote: > On Wed, 23 Apr 2014 09:07:28 +0300 > Marian Marinov <mm@yuhu.biz> wrote: > >> On 04/22/2014 11:05 PM, Richard Davies wrote: >>> Dwight Engen wrote: >>>> Richard Davies wrote: >>>>> Vladimir Davydov wrote: >>>>>> In short, kmem limiting for memory cgroups is currently broken. >>>>>> Do not use it. We are working on making it usable though. >>> ... >>>>> What is the best mechanism available today, until kmem limits >>>>> mature? >>>>> >>>>> RLIMIT_NPROC exists but is per-user, not per-container. >>>>> >>>>> Perhaps there is an up-to-date task counter patchset or similar? >>>> >>>> I updated Frederic's task counter patches and included Max >>>> Kellermann's fork limiter here: >>>> >>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212 >>>> >>>> I can send you a more recent patchset (against 3.13.10) if you >>>> would find it useful. >>> >>> Yes please, I would be interested in that. Ideally even against >>> 3.14.1 if you have that too. >> >> Dwight, do you have these patches in any public repo? >> >> I would like to test them also. > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 > Guys I tested the patches with 3.12.16. However I see a problem with them. Trying to set the limit to a cgroup which already have processes in it does not work: [root@sp2 lxc]# echo 50 > cpuacct.task_limit -bash: echo: write error: Device or resource busy [root@sp2 lxc]# echo 0 > cpuacct.task_limit -bash: echo: write error: Device or resource busy [root@sp2 lxc]# I have even tried to remove this check: + if (cgroup_task_count(cgrp) || !list_empty(&cgrp->children)) + return -EBUSY; But still give me 'Device or resource busy'. Any pointers of why is this happening ? Marian >> Marian >> >>> >>> Thanks, >>> >>> Richard. >>> -- >>> To unsubscribe from this list: send the line "unsubscribe cgroups" >>> in the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >> > > -- > To unsubscribe from this list: send the line "unsubscribe cgroups" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-05-06 11:40 ` Marian Marinov @ 2014-05-07 17:15 ` Dwight Engen 2014-05-07 22:39 ` Marian Marinov 0 siblings, 1 reply; 38+ messages in thread From: Dwight Engen @ 2014-05-07 17:15 UTC (permalink / raw) To: Marian Marinov Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm On Tue, 06 May 2014 14:40:55 +0300 Marian Marinov <mm@yuhu.biz> wrote: > On 04/23/2014 03:49 PM, Dwight Engen wrote: > > On Wed, 23 Apr 2014 09:07:28 +0300 > > Marian Marinov <mm@yuhu.biz> wrote: > > > >> On 04/22/2014 11:05 PM, Richard Davies wrote: > >>> Dwight Engen wrote: > >>>> Richard Davies wrote: > >>>>> Vladimir Davydov wrote: > >>>>>> In short, kmem limiting for memory cgroups is currently broken. > >>>>>> Do not use it. We are working on making it usable though. > >>> ... > >>>>> What is the best mechanism available today, until kmem limits > >>>>> mature? > >>>>> > >>>>> RLIMIT_NPROC exists but is per-user, not per-container. > >>>>> > >>>>> Perhaps there is an up-to-date task counter patchset or similar? > >>>> > >>>> I updated Frederic's task counter patches and included Max > >>>> Kellermann's fork limiter here: > >>>> > >>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212 > >>>> > >>>> I can send you a more recent patchset (against 3.13.10) if you > >>>> would find it useful. > >>> > >>> Yes please, I would be interested in that. Ideally even against > >>> 3.14.1 if you have that too. > >> > >> Dwight, do you have these patches in any public repo? > >> > >> I would like to test them also. > > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 > > > Guys I tested the patches with 3.12.16. However I see a problem with > them. > > Trying to set the limit to a cgroup which already have processes in > it does not work: This is a similar check/limitation to the one for kmem in memcg, and is done here to keep the res_counters consistent and from going negative. It could probably be relaxed slightly by using res_counter_set_limit() instead, but you would still need to initially set a limit before adding tasks to the group. > [root@sp2 lxc]# echo 50 > cpuacct.task_limit > -bash: echo: write error: Device or resource busy > [root@sp2 lxc]# echo 0 > cpuacct.task_limit > -bash: echo: write error: Device or resource busy > [root@sp2 lxc]# > > I have even tried to remove this check: > + if (cgroup_task_count(cgrp) > || !list_empty(&cgrp->children)) > + return -EBUSY; > But still give me 'Device or resource busy'. > > Any pointers of why is this happening ? > > Marian > > >> Marian > >> > >>> > >>> Thanks, > >>> > >>> Richard. > >>> -- > >>> To unsubscribe from this list: send the line "unsubscribe cgroups" > >>> in the body of a message to majordomo@vger.kernel.org > >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>> > >>> > >> > > > > -- > > To unsubscribe from this list: send the line "unsubscribe cgroups" > > in the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-05-07 17:15 ` Dwight Engen @ 2014-05-07 22:39 ` Marian Marinov 2014-05-08 15:25 ` Richard Davies 0 siblings, 1 reply; 38+ messages in thread From: Marian Marinov @ 2014-05-07 22:39 UTC (permalink / raw) To: Dwight Engen, Marian Marinov Cc: Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Johannes Weiner, Glauber Costa, Michal Hocko, linux-mm, William Dauchy, David Rientjes, Tejun Heo, cgroups On 05/07/2014 08:15 PM, Dwight Engen wrote: > On Tue, 06 May 2014 14:40:55 +0300 > Marian Marinov <mm@yuhu.biz> wrote: > >> On 04/23/2014 03:49 PM, Dwight Engen wrote: >>> On Wed, 23 Apr 2014 09:07:28 +0300 >>> Marian Marinov <mm@yuhu.biz> wrote: >>> >>>> On 04/22/2014 11:05 PM, Richard Davies wrote: >>>>> Dwight Engen wrote: >>>>>> Richard Davies wrote: >>>>>>> Vladimir Davydov wrote: >>>>>>>> In short, kmem limiting for memory cgroups is currently broken. >>>>>>>> Do not use it. We are working on making it usable though. >>>>> ... >>>>>>> What is the best mechanism available today, until kmem limits >>>>>>> mature? >>>>>>> >>>>>>> RLIMIT_NPROC exists but is per-user, not per-container. >>>>>>> >>>>>>> Perhaps there is an up-to-date task counter patchset or similar? >>>>>> >>>>>> I updated Frederic's task counter patches and included Max >>>>>> Kellermann's fork limiter here: >>>>>> >>>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212 >>>>>> >>>>>> I can send you a more recent patchset (against 3.13.10) if you >>>>>> would find it useful. >>>>> >>>>> Yes please, I would be interested in that. Ideally even against >>>>> 3.14.1 if you have that too. >>>> >>>> Dwight, do you have these patches in any public repo? >>>> >>>> I would like to test them also. >>> >>> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: >>> >>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 >>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 >>> >> Guys I tested the patches with 3.12.16. However I see a problem with >> them. >> >> Trying to set the limit to a cgroup which already have processes in >> it does not work: > > This is a similar check/limitation to the one for kmem in memcg, and is > done here to keep the res_counters consistent and from going negative. > It could probably be relaxed slightly by using res_counter_set_limit() > instead, but you would still need to initially set a limit before > adding tasks to the group. I have removed the check entirely and still receive the EBUSY... I just don't understand what is returning it. If you have any pointers, I would be happy to take a look. I'll look at set_limit(), thanks for pointing that one. What I'm proposing is the following checks: if (val > RES_COUNTER_MAX || val < 0) return -EBUSY; if (val != 0 && val <= cgroup_task_count(cgrp)) return -EBUSY; res_counter_write_u64(&ca->task_limit, type, val); This way we ensure that val is within the limits > 0 and < RES_COUNTER_MAX. And also allow only values of 0 or greater then the current task count. Marian > >> [root@sp2 lxc]# echo 50 > cpuacct.task_limit >> -bash: echo: write error: Device or resource busy >> [root@sp2 lxc]# echo 0 > cpuacct.task_limit >> -bash: echo: write error: Device or resource busy >> [root@sp2 lxc]# >> >> I have even tried to remove this check: >> + if (cgroup_task_count(cgrp) >> || !list_empty(&cgrp->children)) >> + return -EBUSY; >> But still give me 'Device or resource busy'. >> >> Any pointers of why is this happening ? >> >> Marian >> >>>> Marian >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Richard. >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe cgroups" >>>>> in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>> >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe cgroups" >>> in the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >> > > _______________________________________________ > Containers mailing list > Containers@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/containers > -- Marian Marinov Founder & CEO of 1H Ltd. Jabber/GTalk: hackman@jabber.org ICQ: 7556201 Mobile: +359 886 660 270 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-05-07 22:39 ` Marian Marinov @ 2014-05-08 15:25 ` Richard Davies 0 siblings, 0 replies; 38+ messages in thread From: Richard Davies @ 2014-05-08 15:25 UTC (permalink / raw) To: Marian Marinov Cc: Dwight Engen, Marian Marinov, Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin, Frederic Weisbecker, containers, Johannes Weiner, Glauber Costa, Michal Hocko, linux-mm, William Dauchy, David Rientjes, Tejun Heo, cgroups Marian Marinov wrote: > On 05/07/2014 08:15 PM, Dwight Engen wrote: > >On Tue, 06 May 2014 14:40:55 +0300 > >Marian Marinov <mm@yuhu.biz> wrote: > > > >>On 04/23/2014 03:49 PM, Dwight Engen wrote: > >>>On Wed, 23 Apr 2014 09:07:28 +0300 > >>>Marian Marinov <mm@yuhu.biz> wrote: > >>> > >>>>On 04/22/2014 11:05 PM, Richard Davies wrote: > >>>>>Dwight Engen wrote: > >>>>>>Richard Davies wrote: > >>>>>>>Vladimir Davydov wrote: > >>>>>>>>In short, kmem limiting for memory cgroups is currently broken. > >>>>>>>>Do not use it. We are working on making it usable though. > >>>>>... > >>>>>>>What is the best mechanism available today, until kmem limits > >>>>>>>mature? > >>>>>>> > >>>>>>>RLIMIT_NPROC exists but is per-user, not per-container. > >>>>>>> > >>>>>>>Perhaps there is an up-to-date task counter patchset or similar? > >>>>>> > >>>>>>I updated Frederic's task counter patches and included Max > >>>>>>Kellermann's fork limiter here: > >>>>>> > >>>>>>http://thread.gmane.org/gmane.linux.kernel.containers/27212 > >>>>>> > >>>>>>I can send you a more recent patchset (against 3.13.10) if you > >>>>>>would find it useful. > >>>>> > >>>>>Yes please, I would be interested in that. Ideally even against > >>>>>3.14.1 if you have that too. > >>>> > >>>>Dwight, do you have these patches in any public repo? > >>>> > >>>>I would like to test them also. > >>> > >>>Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > >>> > >>>git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > >>>git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 > >>> > >>Guys I tested the patches with 3.12.16. However I see a problem with > >>them. > >> > >>Trying to set the limit to a cgroup which already have processes in > >>it does not work: > > > >This is a similar check/limitation to the one for kmem in memcg, and is > >done here to keep the res_counters consistent and from going negative. > >It could probably be relaxed slightly by using res_counter_set_limit() > >instead, but you would still need to initially set a limit before > >adding tasks to the group. > > I have removed the check entirely and still receive the EBUSY... I > just don't understand what is returning it. If you have any > pointers, I would be happy to take a look. > > I'll look at set_limit(), thanks for pointing that one. > > What I'm proposing is the following checks: > > if (val > RES_COUNTER_MAX || val < 0) > return -EBUSY; > if (val != 0 && val <= cgroup_task_count(cgrp)) > return -EBUSY; > > res_counter_write_u64(&ca->task_limit, type, val); > > This way we ensure that val is within the limits > 0 and < > RES_COUNTER_MAX. And also allow only values of 0 or greater then the > current task count. I have also noticed that I can't change many different cgroup limits while there are tasks running in the cgroup - not just cpuacct.task_limit, but also kmem and even normal memory.limit_in_bytes I would like to be able to change all of these limits, as long as the new limit is greater than the actual current use. Could a method like this be used for all of the others too? Richard. > >>[root@sp2 lxc]# echo 50 > cpuacct.task_limit > >>-bash: echo: write error: Device or resource busy > >>[root@sp2 lxc]# echo 0 > cpuacct.task_limit > >>-bash: echo: write error: Device or resource busy > >>[root@sp2 lxc]# > >> > >>I have even tried to remove this check: > >>+ if (cgroup_task_count(cgrp) > >>|| !list_empty(&cgrp->children)) > >>+ return -EBUSY; > >>But still give me 'Device or resource busy'. > >> > >>Any pointers of why is this happening ? > >> > >>Marian -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-23 12:49 ` Dwight Engen 2014-04-28 18:00 ` Serge Hallyn 2014-05-06 11:40 ` Marian Marinov @ 2014-06-10 14:50 ` Marian Marinov 2 siblings, 0 replies; 38+ messages in thread From: Marian Marinov @ 2014-06-10 14:50 UTC (permalink / raw) To: Dwight Engen Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm [-- Attachment #1: Type: text/plain, Size: 2238 bytes --] On 04/23/2014 03:49 PM, Dwight Engen wrote: > On Wed, 23 Apr 2014 09:07:28 +0300 > Marian Marinov <mm@yuhu.biz> wrote: > >> On 04/22/2014 11:05 PM, Richard Davies wrote: >>> Dwight Engen wrote: >>>> Richard Davies wrote: >>>>> Vladimir Davydov wrote: >>>>>> In short, kmem limiting for memory cgroups is currently broken. >>>>>> Do not use it. We are working on making it usable though. >>> ... >>>>> What is the best mechanism available today, until kmem limits >>>>> mature? >>>>> >>>>> RLIMIT_NPROC exists but is per-user, not per-container. >>>>> >>>>> Perhaps there is an up-to-date task counter patchset or similar? >>>> >>>> I updated Frederic's task counter patches and included Max >>>> Kellermann's fork limiter here: >>>> >>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212 >>>> >>>> I can send you a more recent patchset (against 3.13.10) if you >>>> would find it useful. >>> >>> Yes please, I would be interested in that. Ideally even against >>> 3.14.1 if you have that too. >> >> Dwight, do you have these patches in any public repo? >> >> I would like to test them also. > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at: > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13 > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14 I did a backport of the patches to 3.12.16 and forward ported them to 3.12.20. I'm very happy with how they work. I used the patches on machines with 10-20k processes and it worked perfectly when some of the containers spawned 100s of processes. It really saved us when one of the containers was attacked :) The only thing that I'm going to add is on the fly change of the limit. Marian > >> Marian >> >>> >>> Thanks, >>> >>> Richard. >>> -- >>> To unsubscribe from this list: send the line "unsubscribe cgroups" >>> in the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >> > > -- > To unsubscribe from this list: send the line "unsubscribe cgroups" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] 2014-04-22 20:05 ` Richard Davies 2014-04-22 20:13 ` Tim Hockin 2014-04-23 6:07 ` Marian Marinov @ 2014-06-10 12:18 ` Alin Dobre 2 siblings, 0 replies; 38+ messages in thread From: Alin Dobre @ 2014-06-10 12:18 UTC (permalink / raw) To: Richard Davies, Dwight Engen Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm On 22/04/14 21:05, Richard Davies wrote: > Dwight Engen wrote: >> Richard Davies wrote: >>> Vladimir Davydov wrote: >>>> In short, kmem limiting for memory cgroups is currently broken. Do >>>> not use it. We are working on making it usable though. > ... >>> What is the best mechanism available today, until kmem limits mature? >>> >>> RLIMIT_NPROC exists but is per-user, not per-container. >>> >>> Perhaps there is an up-to-date task counter patchset or similar? >> >> I updated Frederic's task counter patches and included Max Kellermann's >> fork limiter here: >> >> http://thread.gmane.org/gmane.linux.kernel.containers/27212 >> >> I can send you a more recent patchset (against 3.13.10) if you would >> find it useful. > > Yes please, I would be interested in that. Ideally even against 3.14.1 if > you have that too. Any chance for a 3.15 rebase, since the changes from cgroup_fork() makes the operation no longer trivial. Cheers, Alin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 38+ messages in thread
end of thread, other threads:[~2014-06-10 14:50 UTC | newest] Thread overview: 38+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-04-16 15:46 memcg with kmem limit doesn't recover after disk i/o causes limit to be hit Richard Davies 2014-04-18 15:59 ` Michal Hocko 2014-04-18 17:57 ` Vladimir Davydov 2014-04-18 18:20 ` Michal Hocko 2014-04-18 18:37 ` Vladimir Davydov 2014-04-20 14:28 ` Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] Richard Davies 2014-04-20 18:35 ` Tim Hockin 2014-04-22 18:39 ` Dwight Engen 2014-04-22 20:05 ` Richard Davies 2014-04-22 20:13 ` Tim Hockin 2014-04-23 6:07 ` Marian Marinov 2014-04-23 12:49 ` Dwight Engen 2014-04-28 18:00 ` Serge Hallyn 2014-04-29 7:25 ` Michal Hocko 2014-04-29 13:03 ` Serge Hallyn 2014-04-29 13:57 ` Marian Marinov 2014-04-29 14:04 ` Tim Hockin 2014-04-29 15:43 ` Michal Hocko 2014-04-29 16:06 ` Tim Hockin 2014-04-29 16:51 ` Frederic Weisbecker 2014-04-29 16:59 ` Tim Hockin 2014-04-29 17:06 ` Michal Hocko 2014-04-29 17:30 ` Dwight Engen 2014-04-29 18:09 ` Richard Davies 2014-04-29 18:27 ` Michal Hocko 2014-04-29 18:39 ` Richard Davies 2014-04-29 19:03 ` Michal Hocko 2014-04-29 21:36 ` Marian Marinov 2014-04-30 13:31 ` Michal Hocko 2014-04-29 21:44 ` Frederic Weisbecker 2014-04-30 13:12 ` Daniel J Walsh 2014-04-30 13:28 ` Frederic Weisbecker 2014-05-06 11:40 ` Marian Marinov 2014-05-07 17:15 ` Dwight Engen 2014-05-07 22:39 ` Marian Marinov 2014-05-08 15:25 ` Richard Davies 2014-06-10 14:50 ` Marian Marinov 2014-06-10 12:18 ` Alin Dobre
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).