From: Olivier Bonvalet <xen.list@daevel.fr>
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: xen-users@lists.xen.org, xen-devel@lists.xen.org
Subject: Re: [Xen-users] unexpected Out Of Memory (OOM)
Date: Wed, 07 Aug 2013 10:58:14 +0200 [thread overview]
Message-ID: <1375865894.13572.7.camel@localhost> (raw)
In-Reply-To: <1375864197.619.8.camel@kazak.uk.xensource.com>
Le mercredi 07 août 2013 à 09:29 +0100, Ian Campbell a écrit :
> On Wed, 2013-08-07 at 02:02 +0200, Olivier Bonvalet wrote:
> > Hi,
> >
> > I have a memory problem on several PV domU, I *suppose* since the
> > upgrade to 3.10.4 kernel (and still occur with 3.10.5).
> > I use Xen 4.2.2 (from Debian sid), and the dom0 run under Debian Wheezy
> > with a vanilla 3.9.11 kernel.
> >
> > The problem : once a domU throw oom-killer, then it loop until all
> > processes will be killed, then the kernel abort and shutdown the VM.
> >
> > I'm not sure how to read the OOM-killer messages about memory state, so
> > I copy/paste a list of them here, about a 2GB RAM domU :
>
> Does /proc/meminfo on the VM show that it actually has 2GB?
>
Now yes. I will follow it, for the next time.
> > [56313.921381] cron invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
> > [56313.921407] CPU: 0 PID: 7001 Comm: cron Not tainted 2.6.50-dae-xen #2
>
> This appears to show a 2.6.50 kernel, whatever that is, and not 3.10.x?
>
Yes it's a 3.10.5 here, it was renamed for compatibility with some old
code.
> > [56313.921413] 0000000000000000 ffffffff8134ab12 ffff880079cd2120 0000000000000000
> > [56313.921424] 00000000000280da ffffffff81089ad6 ffffffff8134f09a ffff88007773bba0
> > [56313.921433] ffffffff814d3d48 0000000000000000 00000000000280da 000000000000000b
> > [56313.921443] Call Trace:
> > [56313.921454] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56313.921462] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56313.921470] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56313.921480] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56313.921486] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56313.921493] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56313.921502] [<ffffffff810da1db>] ? alloc_pages_vma+0xd4/0xfc
> > [56313.921510] [<ffffffff810cf31f>] ? anon_vma_alloc+0x11/0x21
> > [56313.921516] [<ffffffff810c7ff1>] ? handle_pte_fault+0x60f/0x6ed
> > [56313.921538] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56313.921547] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56313.921560] [<ffffffff81072f4b>] ? lg_local_lock+0x11/0x14
> > [56313.921566] [<ffffffff810fee0e>] ? mntput_no_expire+0x2d/0x133
> > [56313.921572] [<ffffffff81072f5f>] ? lg_local_unlock+0x11/0x14
> > [56313.921579] [<ffffffff810f3dca>] ? do_last+0x8eb/0x9c3
> > [56313.921587] [<ffffffff811b0401>] ? string.isra.5+0x37/0x9b
> > [56313.921593] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56313.921600] [<ffffffff81101c75>] ? seq_read+0xb8/0x31a
> > [56313.921606] [<ffffffff811b185d>] ? copy_user_generic_string+0x2d/0x40
> > [56313.921612] [<ffffffff81101e60>] ? seq_read+0x2a3/0x31a
> > [56313.921632] [<ffffffff81129c95>] ? proc_reg_read+0x54/0x74
> > [56313.921644] [<ffffffff810e977f>] ? vfs_read+0x8e/0xf0
> > [56313.921650] [<ffffffff810e9e38>] ? SyS_read+0x41/0x70
> > [56313.921656] [<ffffffff8134fc69>] ? system_call_fastpath+0x16/0x1b
> > [56313.921662] Mem-Info:
> > [56313.921671] Node 0 DMA per-cpu:
> > [56313.921676] CPU 0: hi: 0, btch: 1 usd: 0
> > [56313.921680] Node 0 DMA32 per-cpu:
> > [56313.921684] CPU 0: hi: 186, btch: 31 usd: 83
> > [56313.921693] active_anon:22546 inactive_anon:1033 isolated_anon:1
> > [56313.921693] active_file:153 inactive_file:849 isolated_file:0
> > [56313.921693] unevictable:0 dirty:0 writeback:0 unstable:0
> > [56313.921693] free:3392 slab_reclaimable:101577 slab_unreclaimable:232545
> > [56313.921693] mapped:3613 shmem:8327 pagetables:2848 bounce:0
> > [56313.921693] free_cma:0
> > [56313.921708] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [56313.921726] lowmem_reserve[]: 0 1973 1973 1973
> > [56313.921732] Node 0 DMA32 free:5632kB min:5660kB low:7072kB high:8488kB active_anon:90052kB inactive_anon:4132kB active_file:612kB inactive_file:3396kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:14452kB shmem:33308kB slab_reclaimable:404740kB slab_unreclaimable:926360kB kernel_stack:572928kB pagetables:11032kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2541 all_unreclaimable? yes
> > [56313.921749] lowmem_reserve[]: 0 0 0 0
> > [56313.921757] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB
> > [56313.921775] Node 0 DMA32: 204*4kB (EM) 492*8kB (EM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5632kB
> > [56313.921791] 9341 total pagecache pages
> > [56313.921794] 0 pages in swap cache
> > [56313.921797] Swap cache stats: add 0, delete 0, find 0/0
> > [56313.921799] Free swap = 0kB
> > [56313.921802] Total swap = 0kB
> > [56313.927968] 524287 pages RAM
> > [56313.927976] 12090 pages reserved
> > [56313.927978] 307132 pages shared
> > [56313.927980] 430905 pages non-shared
> > [56313.927983] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56313.927996] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56313.928001] [ 815] 0 815 13018 114 20 0 0 rsyslogd
> > [56313.928006] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56313.928011] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56313.928015] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56313.928020] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd
> > [56313.928025] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56313.928029] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56313.928034] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56313.928038] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56313.928043] [ 1016] 0 1016 86937 2033 155 0 0 apache2
> > [56313.928048] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56313.928057] [ 1054] 0 1054 5747 69 16 0 0 cron
> > [56313.928062] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56313.928067] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [56313.928072] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56313.928077] [ 1130] 0 1130 18989 374 35 0 0 searchd
> > [56313.928081] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56313.928086] [ 1152] 105 1152 10265 1088 24 0 0 unbound
> > [56313.928090] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56313.928150] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56313.928157] [31557] 33 31557 88162 5277 161 0 0 apache2
> > [56313.928163] [31998] 33 31998 88044 5689 162 0 0 apache2
> > [56313.928168] [ 1031] 33 1031 87879 4726 161 0 0 apache2
> > [56313.928172] [ 1579] 33 1579 87493 5027 158 0 0 apache2
> > [56313.928177] [ 2901] 33 2901 87622 4707 158 0 0 apache2
> > [56313.928182] [ 3447] 33 3447 82447 4550 150 0 0 apache2
> > [56313.928186] [ 4559] 33 4559 87714 4945 160 0 0 apache2
> > [56313.928191] [ 4560] 33 4560 82914 4906 150 0 0 apache2
> > [56313.928195] [ 4561] 33 4561 87859 3840 159 0 0 apache2
> > [56313.928200] [ 6205] 33 6205 87422 3053 154 0 0 apache2
> > [56313.928204] [ 6776] 0 6776 10460 89 25 0 0 cron
> > [56313.928229] [ 6780] 0 6780 1002 23 8 0 0 sh
> > [56313.928236] [ 6784] 0 6784 7322 109 20 0 0 wget
> > [56313.928241] [ 6847] 0 6847 10460 89 25 0 0 cron
> > [56313.928246] [ 6856] 0 6856 1002 23 8 0 0 sh
> > [56313.928253] [ 6862] 0 6862 7322 108 20 0 0 wget
> > [56313.928258] [ 6924] 0 6924 10460 89 25 0 0 cron
> > [56313.928263] [ 6928] 0 6928 1002 24 8 0 0 sh
> > [56313.928267] [ 6932] 0 6932 7322 107 20 0 0 wget
> > [56313.928272] [ 6948] 33 6948 82020 2026 135 0 0 apache2
> > [56313.928277] [ 6991] 33 6991 86937 1966 137 0 0 apache2
> > [56313.928281] [ 6992] 33 6992 86937 1966 137 0 0 apache2
> > [56313.928285] [ 6993] 0 6993 10460 106 25 0 0 cron
> > [56313.928290] [ 6994] 0 6994 10460 106 25 0 0 cron
> > [56313.928294] [ 6995] 0 6995 10460 106 25 0 0 cron
> > [56313.928298] [ 6999] 1337 6999 10459 183 23 0 0 cron
> > [56313.928302] [ 7000] 0 7000 10459 182 23 0 0 cron
> > [56313.928306] [ 7001] 0 7001 10460 182 23 0 0 cron
> > [56313.928311] Out of memory: Kill process 31998 (apache2) score 11 or sacrifice child
> > [56313.928315] Killed process 31998 (apache2) total-vm:352176kB, anon-rss:10848kB, file-rss:11908kB
> > [56317.522091] searchd invoked oom-killer: gfp_mask=0x2840d0, order=0, oom_score_adj=0
> > [56317.522109] CPU: 0 PID: 1130 Comm: searchd Not tainted 2.6.50-dae-xen #2
> > [56317.522115] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff8800793078e8
> > [56317.522125] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001802120
> > [56317.522133] ffffffff814d3d48 0000000000000000 00000000002840d0 000000000000000a
> > [56317.522142] Call Trace:
> > [56317.522153] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56317.522161] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56317.522168] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56317.522177] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56317.522184] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56317.522190] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56317.522197] [<ffffffff810deff1>] ? new_slab+0x6e/0x1d5
> > [56317.522203] [<ffffffff8134b905>] ? __slab_alloc+0x28f/0x38b
> > [56317.522211] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
> > [56317.522219] [<ffffffff81111258>] ? mpage_alloc+0x25/0x7d
> > [56317.522226] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
> > [56317.522232] [<ffffffff810e00b4>] ? kmem_cache_alloc+0x3d/0x8f
> > [56317.522238] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
> > [56317.522245] [<ffffffff810ae675>] ? add_to_page_cache_locked+0x1b/0x92
> > [56317.522251] [<ffffffff810ae6fa>] ? add_to_page_cache_lru+0xe/0x31
> > [56317.522258] [<ffffffff8111185f>] ? mpage_readpages+0x9c/0x105
> > [56317.522264] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
> > [56317.522271] [<ffffffff810b6aeb>] ? __do_page_cache_readahead+0x112/0x1ab
> > [56317.522277] [<ffffffff810b6ddb>] ? ra_submit+0x18/0x1b
> > [56317.522283] [<ffffffff810b0309>] ? filemap_fault+0x173/0x35f
> > [56317.522289] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [56317.522296] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [56317.522304] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [56317.522310] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [56317.522440] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56317.522449] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56317.522457] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> > [56317.522464] [<ffffffff81350efb>] ? xen_hypervisor_callback+0x1b/0x20
> > [56317.522471] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
> > [56317.522479] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31
> > [56317.522498] [<ffffffff8107f778>] ? ktime_get_ts+0x35/0x72
> > [56317.522503] [<ffffffff810f6d92>] ? poll_select_copy_remaining+0xde/0xf3
> > [56317.522508] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
> > [56317.522512] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31
> > [56317.522517] [<ffffffff8107f8e7>] ? __getnstimeofday+0x1b/0x55
> > [56317.522522] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56317.522526] Mem-Info:
> > [56317.522529] Node 0 DMA per-cpu:
> > [56317.522540] CPU 0: hi: 0, btch: 1 usd: 0
> > [56317.522544] Node 0 DMA32 per-cpu:
> > [56317.522549] CPU 0: hi: 186, btch: 31 usd: 90
> > [56317.522556] active_anon:22815 inactive_anon:1033 isolated_anon:0
> > [56317.522556] active_file:95 inactive_file:793 isolated_file:0
> > [56317.522556] unevictable:0 dirty:0 writeback:0 unstable:0
> > [56317.522556] free:3378 slab_reclaimable:101712 slab_unreclaimable:232523
> > [56317.522556] mapped:3140 shmem:8327 pagetables:2478 bounce:0
> > [56317.522556] free_cma:0
> > [56317.522572] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [56317.522590] lowmem_reserve[]: 0 1973 1973 1973
> > [56317.522599] Node 0 DMA32 free:5576kB min:5660kB low:7072kB high:8488kB active_anon:91128kB inactive_anon:4132kB active_file:380kB inactive_file:3172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12560kB shmem:33308kB slab_reclaimable:405280kB slab_unreclaimable:926272kB kernel_stack:572976kB pagetables:9552kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3288 all_unreclaimable? yes
> > [56317.522621] lowmem_reserve[]: 0 0 0 0
> > [56317.522626] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB
> > [56317.522644] Node 0 DMA32: 254*4kB (UEM) 460*8kB (UEM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5576kB
> > [56317.522660] 9223 total pagecache pages
> > [56317.522663] 0 pages in swap cache
> > [56317.522668] Swap cache stats: add 0, delete 0, find 0/0
> > [56317.522671] Free swap = 0kB
> > [56317.522673] Total swap = 0kB
> > [56317.529054] 524287 pages RAM
> > [56317.529064] 12090 pages reserved
> > [56317.529068] 294744 pages shared
> > [56317.529071] 431526 pages non-shared
> > [56317.529075] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56317.529090] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56317.529098] [ 815] 0 815 13018 110 20 0 0 rsyslogd
> > [56317.529105] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56317.529112] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56317.529119] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56317.529126] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd
> > [56317.529133] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56317.529139] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56317.529146] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56317.529152] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56317.529159] [ 1016] 0 1016 86937 2043 155 0 0 apache2
> > [56317.529166] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56317.529177] [ 1054] 0 1054 5747 69 16 0 0 cron
> > [56317.529184] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56317.529190] [ 1104] 106 1104 9630 153 23 0 0 ntpd
> > [56317.529197] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56317.529203] [ 1130] 0 1130 18989 395 35 0 0 searchd
> > [56317.529211] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56317.529220] [ 1152] 105 1152 10265 1088 24 0 0 unbound
> > [56317.529228] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56317.529237] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56317.529245] [31557] 33 31557 88162 5277 161 0 0 apache2
> > [56317.529252] [ 1031] 33 1031 87879 4726 161 0 0 apache2
> > [56317.529261] [ 1579] 33 1579 87493 5027 158 0 0 apache2
> > [56317.529267] [ 2901] 33 2901 87622 4707 158 0 0 apache2
> > [56317.529274] [ 3447] 33 3447 35706 2055 71 0 0 apache2
> > [56317.529281] [ 4559] 33 4559 87714 4945 160 0 0 apache2
> > [56317.529287] [ 4560] 33 4560 35648 1981 71 0 0 apache2
> > [56317.529296] [ 4561] 33 4561 87859 3840 159 0 0 apache2
> > [56317.529304] [ 6205] 33 6205 87422 3053 154 0 0 apache2
> > [56317.529311] [ 6776] 0 6776 10460 89 25 0 0 cron
> > [56317.529317] [ 6780] 0 6780 1002 23 8 0 0 sh
> > [56317.529324] [ 6784] 0 6784 7322 109 20 0 0 wget
> > [56317.529331] [ 6847] 0 6847 10460 89 25 0 0 cron
> > [56317.529338] [ 6856] 0 6856 1002 23 8 0 0 sh
> > [56317.529345] [ 6862] 0 6862 7322 108 20 0 0 wget
> > [56317.529351] [ 6924] 0 6924 10460 89 25 0 0 cron
> > [56317.529361] [ 6928] 0 6928 1002 24 8 0 0 sh
> > [56317.529367] [ 6932] 0 6932 7322 107 20 0 0 wget
> > [56317.529374] [ 6948] 33 6948 35279 1532 70 0 0 apache2
> > [56317.529381] [ 6991] 33 6991 86937 1966 137 0 0 apache2
> > [56317.529388] [ 6992] 33 6992 86937 1966 137 0 0 apache2
> > [56317.529394] [ 6993] 0 6993 10460 88 25 0 0 cron
> > [56317.529403] [ 6994] 0 6994 10460 88 25 0 0 cron
> > [56317.529410] [ 6995] 0 6995 10460 88 25 0 0 cron
> > [56317.529416] [ 6999] 1337 6999 1002 23 8 0 0 sh
> > [56317.529423] [ 7000] 0 7000 1002 24 8 0 0 sh
> > [56317.529429] [ 7001] 0 7001 1002 23 8 0 0 sh
> > [56317.529436] [ 7002] 1337 7002 1481 21 9 0 0 sleep
> > [56317.529445] [ 7003] 0 7003 7322 110 20 0 0 wget
> > [56317.529452] [ 7004] 0 7004 7322 110 20 0 0 wget
> > [56317.529458] Out of memory: Kill process 31557 (apache2) score 10 or sacrifice child
> > [56317.529464] Killed process 31557 (apache2) total-vm:352648kB, anon-rss:11004kB, file-rss:10104kB
> > [56680.628586] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> > [56680.628601] CPU: 0 PID: 1579 Comm: apache2 Not tainted 2.6.50-dae-xen #2
> > [56680.628607] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006e3ffbb0
> > [56680.628636] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001a8c8e0
> > [56680.628645] ffffffff814d3d48 0000000000000000 00000000000201da 000000000000000b
> > [56680.628654] Call Trace:
> > [56680.628666] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56680.628675] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56680.628682] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56680.628692] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56680.628699] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56680.628706] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56680.628713] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> > [56680.628720] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [56680.628727] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [56680.628735] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [56680.628741] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [56680.628748] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56680.628756] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56680.628763] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56680.628768] Mem-Info:
> > [56680.628771] Node 0 DMA per-cpu:
> > [56680.628776] CPU 0: hi: 0, btch: 1 usd: 0
> > [56680.628782] Node 0 DMA32 per-cpu:
> > [56680.628788] CPU 0: hi: 186, btch: 31 usd: 115
> > [56680.628796] active_anon:19702 inactive_anon:1033 isolated_anon:0
> > [56680.628796] active_file:156 inactive_file:1004 isolated_file:0
> > [56680.628796] unevictable:0 dirty:0 writeback:0 unstable:0
> > [56680.628796] free:3382 slab_reclaimable:103079 slab_unreclaimable:233653
> > [56680.628796] mapped:3205 shmem:8327 pagetables:2067 bounce:0
> > [56680.628796] free_cma:0
> > [56680.628814] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1632kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [56680.628848] lowmem_reserve[]: 0 1973 1973 1973
> > [56680.628856] Node 0 DMA32 free:5600kB min:5660kB low:7072kB high:8488kB active_anon:78780kB inactive_anon:4132kB active_file:624kB inactive_file:4016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12820kB shmem:33308kB slab_reclaimable:410684kB slab_unreclaimable:930456kB kernel_stack:576528kB pagetables:8264kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2646 all_unreclaimable? yes
> > [56680.628883] lowmem_reserve[]: 0 0 0 0
> > [56680.628890] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 10*32kB (EM) 24*64kB (EM) 9*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> > [56680.628921] Node 0 DMA32: 0*4kB 564*8kB (UEM) 26*16kB (MR) 1*32kB (R) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5600kB
> > [56680.628945] 9494 total pagecache pages
> > [56680.628949] 0 pages in swap cache
> > [56680.628957] Swap cache stats: add 0, delete 0, find 0/0
> > [56680.628962] Free swap = 0kB
> > [56680.628965] Total swap = 0kB
> > [56680.634908] 524287 pages RAM
> > [56680.634916] 12090 pages reserved
> > [56680.634919] 291027 pages shared
> > [56680.634921] 431125 pages non-shared
> > [56680.634924] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56680.634937] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56680.634943] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> > [56680.634948] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56680.634953] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56680.634957] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56680.634961] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd
> > [56680.634966] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56680.634971] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56680.634975] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56680.634980] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56680.634985] [ 1016] 0 1016 86937 2014 155 0 0 apache2
> > [56680.634990] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56680.634995] [ 1054] 0 1054 5747 69 16 0 0 cron
> > [56680.634999] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56680.635004] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [56680.635008] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56680.635013] [ 1130] 0 1130 18989 400 35 0 0 searchd
> > [56680.635018] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56680.635022] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> > [56680.635027] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56680.635144] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56680.635150] [ 1031] 33 1031 87879 4814 161 0 0 apache2
> > [56680.635155] [ 1579] 33 1579 87941 5548 159 0 0 apache2
> > [56680.635160] [ 2901] 33 2901 87622 4692 158 0 0 apache2
> > [56680.635164] [ 4559] 33 4559 87714 4947 160 0 0 apache2
> > [56680.635169] [ 4561] 33 4561 87859 3852 159 0 0 apache2
> > [56680.635174] [ 6205] 33 6205 87294 3684 155 0 0 apache2
> > [56680.635179] [ 6991] 33 6991 87434 3395 152 0 0 apache2
> > [56680.635183] [ 6992] 33 6992 87593 3185 148 0 0 apache2
> > [56680.635188] [ 7289] 0 7289 10460 89 25 0 0 cron
> > [56680.635192] [ 7297] 0 7297 1002 23 7 0 0 sh
> > [56680.635197] [ 7303] 0 7303 7322 110 20 0 0 wget
> > [56680.635202] [ 7363] 0 7363 10460 89 25 0 0 cron
> > [56680.635206] [ 7374] 0 7374 1002 25 7 0 0 sh
> > [56680.635213] [ 7378] 0 7378 7322 108 19 0 0 wget
> > [56680.635218] [ 7437] 0 7437 10460 89 25 0 0 cron
> > [56680.635222] [ 7438] 0 7438 10460 89 25 0 0 cron
> > [56680.635227] [ 7439] 0 7439 10460 89 25 0 0 cron
> > [56680.635231] [ 7443] 0 7443 1002 23 8 0 0 sh
> > [56680.635235] [ 7444] 0 7444 1002 23 8 0 0 sh
> > [56680.635239] [ 7445] 1337 7445 1002 24 7 0 0 sh
> > [56680.635244] [ 7446] 1337 7446 1481 22 8 0 0 sleep
> > [56680.635248] [ 7447] 0 7447 7322 109 19 0 0 wget
> > [56680.635252] [ 7448] 0 7448 7322 107 19 0 0 wget
> > [56680.635257] Out of memory: Kill process 1579 (apache2) score 11 or sacrifice child
> > [56680.635262] Killed process 1579 (apache2) total-vm:351764kB, anon-rss:12092kB, file-rss:10100kB
> > [56734.778939] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> > [56734.778957] CPU: 0 PID: 7524 Comm: wget Not tainted 2.6.50-dae-xen #2
> > [56734.778963] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006d133bb0
> > [56734.778973] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800634686a0
> > [56734.778982] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009
> > [56734.779011] Call Trace:
> > [56734.779020] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56734.779026] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56734.779031] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56734.779037] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56734.779042] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56734.779046] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56734.779051] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> > [56734.779055] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [56734.779060] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [56734.779065] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [56734.779069] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [56734.779073] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56734.779079] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56734.779085] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> > [56734.779089] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
> > [56734.779093] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
> > [56734.779097] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56734.779100] Mem-Info:
> > [56734.779102] Node 0 DMA per-cpu:
> > [56734.779106] CPU 0: hi: 0, btch: 1 usd: 0
> > [56734.779108] Node 0 DMA32 per-cpu:
> > [56734.779112] CPU 0: hi: 186, btch: 31 usd: 62
> > [56734.779117] active_anon:18885 inactive_anon:1033 isolated_anon:0
> > [56734.779117] active_file:190 inactive_file:1109 isolated_file:0
> > [56734.779117] unevictable:0 dirty:2 writeback:0 unstable:0
> > [56734.779117] free:3395 slab_reclaimable:103429 slab_unreclaimable:233857
> > [56734.779117] mapped:3186 shmem:8327 pagetables:2038 bounce:0
> > [56734.779117] free_cma:0
> > [56734.779234] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [56734.779256] lowmem_reserve[]: 0 1973 1973 1973
> > [56734.779267] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:75512kB inactive_anon:4132kB active_file:760kB inactive_file:4424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:8kB writeback:0kB mapped:12744kB shmem:33308kB slab_reclaimable:412052kB slab_unreclaimable:931272kB kernel_stack:577184kB pagetables:8148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3285 all_unreclaimable? yes
> > [56734.779293] lowmem_reserve[]: 0 0 0 0
> > [56734.779300] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> > [56734.779327] Node 0 DMA32: 99*4kB (E) 521*8kB (EM) 16*16kB (ER) 6*32kB (UMR) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
> > [56734.779351] 9645 total pagecache pages
> > [56734.779359] 0 pages in swap cache
> > [56734.779363] Swap cache stats: add 0, delete 0, find 0/0
> > [56734.779367] Free swap = 0kB
> > [56734.779370] Total swap = 0kB
> > [56734.785510] 524287 pages RAM
> > [56734.785519] 12090 pages reserved
> > [56734.785523] 286620 pages shared
> > [56734.785526] 431108 pages non-shared
> > [56734.785529] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56734.785544] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56734.785553] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> > [56734.785560] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56734.785567] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56734.785573] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56734.785579] [ 911] 104 911 12487 208 27 0 0 zabbix_agentd
> > [56734.785586] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56734.785593] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56734.785600] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56734.785607] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56734.785614] [ 1016] 0 1016 86937 2024 155 0 0 apache2
> > [56734.785620] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56734.785627] [ 1054] 0 1054 5747 70 16 0 0 cron
> > [56734.785634] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56734.785641] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [56734.785648] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56734.785654] [ 1130] 0 1130 18989 395 35 0 0 searchd
> > [56734.785661] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56734.785665] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> > [56734.785670] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56734.785712] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56734.785718] [ 1031] 33 1031 87879 4814 161 0 0 apache2
> > [56734.785722] [ 2901] 33 2901 87622 4692 158 0 0 apache2
> > [56734.785727] [ 4559] 33 4559 87714 4954 160 0 0 apache2
> > [56734.785731] [ 4561] 33 4561 87859 3815 159 0 0 apache2
> > [56734.785737] [ 6205] 33 6205 87294 3674 155 0 0 apache2
> > [56734.785741] [ 6991] 33 6991 87434 3398 152 0 0 apache2
> > [56734.785745] [ 6992] 33 6992 87593 3228 150 0 0 apache2
> > [56734.785750] [ 7363] 0 7363 10460 89 25 0 0 cron
> > [56734.785754] [ 7374] 0 7374 1002 25 7 0 0 sh
> > [56734.785760] [ 7378] 0 7378 7322 108 19 0 0 wget
> > [56734.785765] [ 7505] 0 7505 10460 89 25 0 0 cron
> > [56734.785769] [ 7506] 0 7506 10460 89 25 0 0 cron
> > [56734.785773] [ 7507] 0 7507 10460 89 25 0 0 cron
> > [56734.785777] [ 7508] 0 7508 10460 89 25 0 0 cron
> > [56734.785782] [ 7509] 0 7509 10460 89 25 0 0 cron
> > [56734.785786] [ 7510] 0 7510 10460 89 25 0 0 cron
> > [56734.785790] [ 7517] 0 7517 1002 24 8 0 0 sh
> > [56734.785794] [ 7518] 0 7518 1002 24 8 0 0 sh
> > [56734.785800] [ 7519] 0 7519 1002 24 8 0 0 sh
> > [56734.785804] [ 7520] 0 7520 1002 23 8 0 0 sh
> > [56734.785808] [ 7521] 1337 7521 1002 24 8 0 0 sh
> > [56734.785813] [ 7522] 0 7522 1002 23 7 0 0 sh
> > [56734.785817] [ 7523] 1337 7523 1481 21 8 0 0 sleep
> > [56734.785822] [ 7524] 0 7524 7322 148 20 0 0 wget
> > [56734.785826] [ 7525] 0 7525 7322 111 20 0 0 wget
> > [56734.785830] [ 7526] 0 7526 7322 110 19 0 0 wget
> > [56734.785834] [ 7527] 0 7527 7322 110 20 0 0 wget
> > [56734.785839] [ 7528] 0 7528 1002 24 7 0 0 maxlifetime
> > [56734.785846] [ 7530] 0 7530 16356 291 37 0 0 php5
> > [56734.785850] Out of memory: Kill process 4559 (apache2) score 9 or sacrifice child
> > [56734.785854] Killed process 4559 (apache2) total-vm:350856kB, anon-rss:11596kB, file-rss:8220kB
> > [56793.525886] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> > [56793.525921] CPU: 0 PID: 7614 Comm: wget Not tainted 2.6.50-dae-xen #2
> > [56793.525928] 0000000000000000 ffffffff8134ab12 ffffffff8134f478 00000000000007c5
> > [56793.525938] 0000000000000010 ffffffff81089ad6 ffffffff8134f09a ffff880001a127c0
> > [56793.525946] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009
> > [56793.525976] Call Trace:
> > [56793.525986] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56793.526012] [<ffffffff8134f478>] ? retint_restore_args+0x5/0x6
> > [56793.526042] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56793.526048] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56793.526057] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56793.526064] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56793.526070] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56793.526082] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> > [56793.526090] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [56793.526097] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [56793.526104] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [56793.526111] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [56793.526117] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56793.526125] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56793.526133] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> > [56793.526142] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
> > [56793.526149] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
> > [56793.526155] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56793.526160] Mem-Info:
> > [56793.526163] Node 0 DMA per-cpu:
> > [56793.526171] CPU 0: hi: 0, btch: 1 usd: 0
> > [56793.526175] Node 0 DMA32 per-cpu:
> > [56793.526180] CPU 0: hi: 186, btch: 31 usd: 150
> > [56793.526188] active_anon:17848 inactive_anon:1033 isolated_anon:5
> > [56793.526188] active_file:78 inactive_file:778 isolated_file:0
> > [56793.526188] unevictable:0 dirty:0 writeback:0 unstable:0
> > [56793.526188] free:3395 slab_reclaimable:104385 slab_unreclaimable:234096
> > [56793.526188] mapped:2974 shmem:8327 pagetables:2047 bounce:0
> > [56793.526188] free_cma:0
> > [56793.526213] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes
> > [56793.526238] lowmem_reserve[]: 0 1973 1973 1973
> > [56793.526247] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:71364kB inactive_anon:4132kB active_file:312kB inactive_file:3100kB unevictable:0kB isolated(anon):20kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:11896kB shmem:33308kB slab_reclaimable:415876kB slab_unreclaimable:932228kB kernel_stack:577872kB pagetables:8184kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1295 all_unreclaimable? yes
> > [56793.526277] lowmem_reserve[]: 0 0 0 0
> > [56793.526284] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> > [56793.526316] Node 0 DMA32: 3*4kB (UEM) 621*8kB (UEM) 2*16kB (UR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
> > [56793.526343] 9197 total pagecache pages
> > [56793.526347] 0 pages in swap cache
> > [56793.526352] Swap cache stats: add 0, delete 0, find 0/0
> > [56793.526356] Free swap = 0kB
> > [56793.526359] Total swap = 0kB
> > [56793.532348] 524287 pages RAM
> > [56793.532355] 12090 pages reserved
> > [56793.532358] 284892 pages shared
> > [56793.532360] 430988 pages non-shared
> > [56793.532363] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56793.532376] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56793.532381] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> > [56793.532386] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56793.532391] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56793.532396] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56793.532400] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd
> > [56793.532405] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56793.532409] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56793.532414] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56793.532418] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56793.532424] [ 1016] 0 1016 86937 2028 155 0 0 apache2
> > [56793.532428] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56793.532433] [ 1054] 0 1054 5747 70 16 0 0 cron
> > [56793.532437] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56793.532442] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [56793.532447] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56793.532451] [ 1130] 0 1130 18989 374 35 0 0 searchd
> > [56793.532456] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56793.532461] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> > [56793.532467] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56793.532484] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56793.532491] [ 1031] 33 1031 87879 4791 161 0 0 apache2
> > [56793.532497] [ 2901] 33 2901 87622 4723 158 0 0 apache2
> > [56793.532504] [ 4561] 33 4561 87859 3866 159 0 0 apache2
> > [56793.532510] [ 6205] 33 6205 87294 3663 155 0 0 apache2
> > [56793.532515] [ 6991] 33 6991 87434 3398 152 0 0 apache2
> > [56793.532522] [ 6992] 33 6992 87593 3228 150 0 0 apache2
> > [56793.532526] [ 7363] 0 7363 10460 89 25 0 0 cron
> > [56793.532531] [ 7374] 0 7374 1002 25 7 0 0 sh
> > [56793.532537] [ 7378] 0 7378 7322 108 19 0 0 wget
> > [56793.532547] [ 7555] 33 7555 86937 1967 137 0 0 apache2
> > [56793.532553] [ 7593] 0 7593 10460 88 25 0 0 cron
> > [56793.532559] [ 7594] 0 7594 10460 88 25 0 0 cron
> > [56793.532564] [ 7595] 0 7595 10460 88 25 0 0 cron
> > [56793.532568] [ 7596] 0 7596 10460 88 25 0 0 cron
> > [56793.532573] [ 7597] 0 7597 10460 88 25 0 0 cron
> > [56793.532580] [ 7598] 0 7598 10460 88 25 0 0 cron
> > [56793.532584] [ 7599] 0 7599 10460 88 25 0 0 cron
> > [56793.532588] [ 7607] 0 7607 1002 25 8 0 0 sh
> > [56793.532593] [ 7608] 1337 7608 1002 24 8 0 0 sh
> > [56793.532597] [ 7609] 0 7609 1002 26 7 0 0 sh
> > [56793.532601] [ 7610] 0 7610 1002 24 7 0 0 sh
> > [56793.532606] [ 7611] 0 7611 1002 25 7 0 0 sh
> > [56793.532611] [ 7612] 0 7612 1002 23 8 0 0 sh
> > [56793.532615] [ 7613] 0 7613 1002 25 7 0 0 sh
> > [56793.532620] [ 7614] 0 7614 7322 156 20 0 0 wget
> > [56793.532624] [ 7615] 1337 7615 1481 19 8 0 0 sleep
> > [56793.532628] [ 7616] 0 7616 7322 110 19 0 0 wget
> > [56793.532632] [ 7617] 0 7617 7322 110 20 0 0 wget
> > [56793.532637] [ 7618] 0 7618 7322 109 19 0 0 wget
> > [56793.532641] [ 7619] 0 7619 7322 109 20 0 0 wget
> > [56793.532646] [ 7620] 0 7620 7322 109 20 0 0 wget
> > [56793.532652] Out of memory: Kill process 1031 (apache2) score 9 or sacrifice child
> > [56793.532656] Killed process 1031 (apache2) total-vm:351516kB, anon-rss:9624kB, file-rss:9540kB
> >
> > ...
> >
> > [57131.763804] zabbix_sender invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> > [57131.763840] CPU: 0 PID: 8158 Comm: zabbix_sender Not tainted 2.6.50-dae-xen #2
> > [57131.763847] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff880073e21bb0
> > [57131.763858] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800788aea00
> > [57131.763867] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000004
> > [57131.763885] Call Trace:
> > [57131.763896] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [57131.763906] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [57131.763913] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [57131.763921] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [57131.763947] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [57131.763955] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [57131.763962] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> > [57131.763969] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [57131.763977] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [57131.763985] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [57131.763991] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [57131.763997] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [57131.764006] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [57131.764017] [<ffffffff810cbcbc>] ? __split_vma.isra.25+0x11a/0x182
> > [57131.764023] [<ffffffff810ce298>] ? mprotect_fixup+0x1bd/0x1f7
> > [57131.764030] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [57131.764034] Mem-Info:
> > [57131.764037] Node 0 DMA per-cpu:
> > [57131.764041] CPU 0: hi: 0, btch: 1 usd: 0
> > [57131.764045] Node 0 DMA32 per-cpu:
> > [57131.764049] CPU 0: hi: 186, btch: 31 usd: 62
> > [57131.764076] active_anon:11743 inactive_anon:1026 isolated_anon:0
> > [57131.764076] active_file:288 inactive_file:1121 isolated_file:0
> > [57131.764076] unevictable:0 dirty:0 writeback:0 unstable:0
> > [57131.764076] free:3369 slab_reclaimable:109088 slab_unreclaimable:235420
> > [57131.764076] mapped:358 shmem:8328 pagetables:821 bounce:0
> > [57131.764076] free_cma:0
> > [57131.764092] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1760kB slab_unreclaimable:4152kB kernel_stack:2056kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [57131.764115] lowmem_reserve[]: 0 1973 1973 1973
> > [57131.764122] Node 0 DMA32 free:5548kB min:5660kB low:7072kB high:8488kB active_anon:46972kB inactive_anon:4104kB active_file:1152kB inactive_file:4472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:1432kB shmem:33312kB slab_reclaimable:434592kB slab_unreclaimable:937528kB kernel_stack:582224kB pagetables:3284kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4240 all_unreclaimable? yes
> > [57131.764151] lowmem_reserve[]: 0 0 0 0
> > [57131.764158] Node 0 DMA: 64*4kB (UE) 33*8kB (UEM) 13*16kB (UEM) 11*32kB (UEM) 19*64kB (EM) 8*128kB (M) 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> > [57131.764189] Node 0 DMA32: 383*4kB (UEM) 164*8kB (UEM) 79*16kB (UMR) 17*32kB (UEMR) 4*64kB (EM) 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5548kB
> > [57131.764242] 9744 total pagecache pages
> > [57131.764253] 0 pages in swap cache
> > [57131.764256] Swap cache stats: add 0, delete 0, find 0/0
> > [57131.764263] Free swap = 0kB
> > [57131.764267] Total swap = 0kB
> > [57131.770252] 524287 pages RAM
> > [57131.770260] 12090 pages reserved
> > [57131.770262] 265440 pages shared
> > [57131.770265] 433233 pages non-shared
> > [57131.770267] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [57131.770280] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [57131.770285] [ 815] 0 815 13048 150 20 0 0 rsyslogd
> > [57131.770290] [ 853] 104 853 12486 99 27 0 0 zabbix_agentd
> > [57131.770295] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [57131.770299] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [57131.770305] [ 1016] 0 1016 86937 2031 155 0 0 apache2
> > [57131.770310] [ 1054] 0 1054 5747 69 16 0 0 cron
> > [57131.770314] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [57131.770319] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [57131.770323] [ 1130] 0 1130 18989 374 35 0 0 searchd
> > [57131.770328] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [57131.770334] [ 7363] 0 7363 10460 89 25 0 0 cron
> > [57131.770339] [ 7374] 0 7374 1002 25 7 0 0 sh
> > [57131.770343] [ 7378] 0 7378 7322 108 19 0 0 wget
> > [57131.770348] [ 8003] 0 8003 10460 88 25 0 0 cron
> > [57131.770352] [ 8006] 0 8006 10460 88 25 0 0 cron
> > [57131.770357] [ 8009] 0 8009 10460 88 25 0 0 cron
> > [57131.770361] [ 8020] 1337 8020 1002 23 7 0 0 sh
> > [57131.770366] [ 8023] 0 8023 1002 23 6 0 0 sh
> > [57131.770370] [ 8024] 0 8024 1002 24 7 0 0 sh
> > [57131.770399] [ 8030] 0 8030 7322 108 19 0 0 wget
> > [57131.770406] [ 8035] 0 8035 7322 109 20 0 0 wget
> > [57131.770410] [ 8051] 0 8051 1002 41 8 0 0 rc
> > [57131.770414] [ 8054] 0 8054 3085 60 11 0 0 startpar
> > [57131.770419] [ 8055] 1337 8055 1002 23 8 0 0 zabbix-launch-t
> > [57131.770424] [ 8057] 0 8057 1002 40 7 0 0 apache2
> > [57131.770430] [ 8067] 1337 8067 2841 76 9 0 0 apache
> > [57131.770436] [ 8084] 0 8084 1002 24 8 0 0 apache2ctl
> > [57131.770443] [ 8092] 0 8092 26561 288 48 0 0 apache2
> > [57131.770450] [ 8115] 0 8115 1002 22 8 0 0 zabbix-agent
> > [57131.770457] [ 8116] 0 8116 980 32 8 0 0 start-stop-daem
> > [57131.770461] [ 8158] 1337 8158 10130 168 22 0 0 zabbix_sender
> > [57131.770466] [ 8161] 33 8161 86937 1967 137 0 0 apache2
> > [57131.770471] Out of memory: Kill process 8161 (apache2) score 4 or sacrifice child
> > [57131.770475] Killed process 8161 (apache2) total-vm:347748kB, anon-rss:7868kB, file-rss:0kB
> >
> >
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xen.org
> > http://lists.xen.org/xen-users
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2013-08-07 8:58 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-07 0:02 unexpected Out Of Memory (OOM) Olivier Bonvalet
2013-08-07 8:29 ` [Xen-users] " Ian Campbell
2013-08-07 8:58 ` Olivier Bonvalet [this message]
2013-08-07 9:35 ` Olivier Bonvalet
2013-08-07 9:46 ` Olivier Bonvalet
2013-08-07 10:16 ` Ian Campbell
2013-08-07 11:17 ` Olivier Bonvalet
2013-08-07 13:36 ` Ian Campbell
2013-08-07 14:26 ` Olivier Bonvalet
2013-08-07 21:37 ` Olivier Bonvalet
2013-08-08 8:58 ` Ian Campbell
2013-08-08 10:10 ` Olivier Bonvalet
2013-08-08 10:18 ` Ian Campbell
2013-08-08 11:43 ` Olivier Bonvalet
2013-08-08 13:25 ` Wei Liu
2013-08-08 14:17 ` Olivier Bonvalet
2013-08-09 12:53 ` Konrad Rzeszutek Wilk
2013-08-07 14:48 ` Wei Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1375865894.13572.7.camel@localhost \
--to=xen.list@daevel.fr \
--cc=Ian.Campbell@citrix.com \
--cc=xen-devel@lists.xen.org \
--cc=xen-users@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.